Module 5

Predictive Analytics II - Text, Web, and Social Media Analytics:

Lecture Notes: Text Analytics and Text Mining Overview

1. Introduction

• The Information Age is defined by the massive growth of data — much of which
is unstructured text.

• 85% of all corporate data are stored in unstructured form (Merrill Lynch &
Gartner, 2005).

• Unstructured data are doubling every 18 months.

• Businesses that effectively analyze textual data can derive valuable insights
and gain competitive advantage.

2. What Are Text Analytics and Text Mining?

2.1 Purpose

Both aim to:

• Convert unstructured textual data into actionable knowledge.

• Use Natural Language Processing (NLP) and data analytics techniques.

2.2 Definitions

Term Description

A broader concept including information retrieval, information extraction,

Text
data mining, and web mining. Focuses on analyzing and understanding
Analytics
text data.

Focused on discovering new and useful knowledge from textual data.

Text Mining
Often considered a subset of text analytics.

2.3 Relationship Between Text Analytics and Text Mining

• Text Analytics = Information Retrieval + Information Extraction + Data Mining

+ Web Mining
or simply,
Text Analytics = Information Retrieval + Text Mining
 • Text Analytics → More business-oriented term.

• Text Mining → Commonly used in academic and research contexts.

• In practice, the two terms are often used interchangeably.

3. Related Application Areas and Foundational Disciplines

3.1 Application Areas (as per Figure 6.2)

• Text Analytics Components:

o Document matching

o Link analysis

o Search engines

o Part-of-speech (POS) tagging

o Lemmatization

o Word disambiguation

o Information retrieval

• Text Mining Components:

 o Knowledge discovery from textual data

o Classification

o Clustering

o Association analysis

• Web Mining Components:

o Web content mining

o Web structure mining

o Web usage mining

3.2 Foundational Disciplines

• Statistics

• Machine Learning

• Data Mining

• Natural Language Processing (NLP)

• Artificial Intelligence

• Computer Science

• Management Science

These disciplines form the foundation (“the base of the house”) of text analytics and
mining.

4. Text Mining: Process and Characteristics

4.1 Definition

Text Mining (also called Text Data Mining or Knowledge Discovery in Textual Databases)
→ Semi-automated process of extracting patterns and useful information from large
volumes of unstructured textual data.

4.2 Comparison with Data Mining

Feature Data Mining Text Mining

Structured (numerical, Unstructured (text documents, PDFs,

Data Type
categorical) XML, etc.)
Feature Data Mining Text Mining

Input
Tables, databases Free-form text
Format

Discover patterns in structured

Goal Discover patterns and knowledge in text
data

Apply algorithms on structured Impose structure + apply mining

Process
data algorithms

4.3 Steps in Text Mining Process

1. Text Preprocessing (Imposing Structure):

o Tokenization, stopword removal, stemming, lemmatization.

o Transform text into structured formats (e.g., term-document matrices).

2. Knowledge Extraction:

o Apply data mining and NLP techniques (classification, clustering,

association, etc.).

5. Applications of Text Mining

5.1 Key Domains

• Law: Analyzing court orders, legal documents.

• Academia: Mining research articles and literature.

• Finance: Analyzing quarterly reports and financial statements.

• Medicine: Processing discharge summaries, medical records.

• Biology: Studying molecular interactions.

• Technology: Mining patent files and technical reports.

• Marketing & Customer Analytics: Analyzing reviews, complaints, and

feedback.

5.2 Example

• Mining customer feedback and warranty claims → Identify recurring issues,

improve product quality.
 • Analyzing e-mails → Spam filtering, prioritization, auto-responses (Weng & Liu,
2004).

6. Major Application Areas in Text Mining

Application Description

Identifying key phrases, entities, and relationships using

Information Extraction
pattern matching.

Predicting documents of interest based on user profiles and

Topic Tracking
viewed content.

Automatically condensing documents to key points for

Summarization
quick understanding.

Categorization Assigning documents to predefined categories based on

(Classification) content.

Clustering Grouping similar documents without predefined labels.

Connecting documents that share related concepts to

Concept Linking
improve information discovery.

Automatically finding the best answer to a user’s question

Question Answering
using knowledge-based matching.

7. Importance and Benefits

• Enables knowledge discovery from vast unstructured text.

• Improves decision-making and competitive advantage.

• Supports automation in customer support, email management, and content

analysis.

• Facilitates insight generation from diverse sources like research papers, legal
documents, and social media.

8. Summary

• Text Analytics and Text Mining transform unstructured text into actionable
insights.
 • They leverage techniques from NLP, machine learning, and data mining.

• Applications span across business, healthcare, science, and technology.

• As unstructured data continues to grow exponentially, text mining is becoming

a key enabler for intelligent decision-making.

Lecture Notes: 6.3 – Natural Language Processing (NLP)

1. Introduction

• Natural Language Processing (NLP) is a core component of text mining.

• It lies at the intersection of artificial intelligence (AI) and computational

linguistics.

• NLP focuses on understanding, interpreting, and generating human language

so that computers can process it meaningfully.

Key Idea:

NLP converts natural human language into formal representations (numeric or

symbolic data) that computers can manipulate and analyze.

2. Early Text Mining and the Bag-of-Words Model

2.1 The Bag-of-Words Concept

• One of the earliest text-mining methods for introducing structure into text.

• Represents text (sentence, paragraph, or document) as a collection of

individual words, ignoring grammar and word order.

Example: Spam Filtering

• Two word “bags”:

o One bag: words common in spam emails (e.g., “Viagra,” “buy,” “stock”).

o Another bag: words common in legitimate emails (e.g., “meeting,”

“project”).

• The new email is compared to both bags → classified based on word frequency
and match score.

2.2 Limitations

• Disregards semantics (meaning) and syntax (structure).

 • Only counts word occurrences — cannot capture context or relationships
between words.

3. The Need for NLP

• Humans communicate with structured and contextual meaning.

• Therefore, automated systems must go beyond word counting and incorporate:

o Grammar

o Semantics

o Contextual relationships

Case Example: Evidence-Based Medicine (University of Maryland)

• Used bag-of-words on over 500,000 MEDLINE research articles for information

appraisal.

• Even with advanced classifiers, results were no better than random guessing.

• Conclusion: Bag-of-words fails to capture complex medical language and

relationships → NLP is needed for richer semantic understanding.

4. Definition and Scope of NLP

Definition

NLP is the field of AI that studies methods for processing and understanding natural
human language, aiming to convert it into a form suitable for computational analysis.

Goal

• Move beyond syntax-driven word manipulation toward true understanding of:

o Grammar

o Semantics

o Context

Reality Check

• True human-like understanding by computers remains limited due to:

o Ambiguity in human language

o Context-dependence
 o Cultural and pragmatic factors

5. Challenges in NLP

Challenge Description Example / Issue

“Book” can be a noun (“a

Part-of-Speech Identifying whether a word is a
book”) or verb (“to book a
(POS) Tagging noun, verb, adjective, etc.
ticket”).

Languages like Chinese or

Splitting text into meaningful
Text Segmentation Japanese lack clear word
units (words or sentences).
boundaries.

Word Sense Determining the correct meaning “Bank” (riverbank vs.

Disambiguation of a word with multiple senses. financial institution).

“I saw the man with the

A sentence may have multiple
Syntactic Ambiguity telescope.” (Who has the
grammatical interpretations.
telescope?)

Errors, typos, slang, accents, or

Imperfect / Irregular
speech variations complicate “u r gr8” (you are great).
Input
parsing.

Sentences can express actions, “Can you pass the salt?” = a

Speech Acts
not just information. request, not a question.

6. Advances in NLP Research

6.1 Automated Knowledge Extraction

• Stanford University NLP Lab developed algorithms that:

o Parse large text corpora to identify concepts and relationships.

o Automatically generate knowledge bases.

6.2 WordNet

• A manually built lexical database of English:

o Contains words, definitions, synonyms, and semantic relations.

• Expensive and time-consuming to create manually.

 • NLP algorithms can now automatically enrich WordNet with new knowledge at
lower cost.

7. Application Highlight: NLP in Customer Relationship Management (CRM)

7.1 Sentiment Analysis

• NLP is used to detect positive or negative opinions about products/services

from textual data (reviews, feedback, social media).

• Helps businesses understand customer perception and improve loyalty.

8. Analytics in Action 6.1 – Deliver Innovation by Understanding Customer

Sentiments

Case Study: Think Big Analytics (Teradata Company)

• Objective: Analyze automobile safety issue notes and customer feedback

using NLP and text clustering.

8.1 Data Sources

• Customer complaints, call center notes, emails, social media posts, etc.

• Text is typically unstructured, domain-specific, and full of slang or

abbreviations.

8.2 Analytical Techniques Used

Technique Purpose

TF-IDF (Term Frequency–Inverse Measures how important a word is in a

Document Frequency) document relative to others.

Quantifies similarity between two text

Cosine Similarity
documents (e.g., complaint notes).

• Combined, these techniques identify common themes or issues across large

text corpora.

8.3 Process

1. Cluster related words/phrases to isolate common issues.

2. Apply NLP sentiment analysis to determine tone (positive, negative, neutral).

 3. Identify recurring themes (e.g., brake problems, power steering leaks, factory
issues).

4. Take corrective actions based on findings.

8.4 Business Impact

• Helped the automaker:

o Detect specific safety issues.

o Link issues to particular factories.

o Prioritize high-value customers for service or compensation.

o Segment customers geographically (e.g., California vs. Nebraska

regulations).

8.5 Cross-Industry Applications

• Healthcare: Analyze patient notes or medical transcripts.

• Retail: Monitor social media feedback on new products.

• Finance: Review emails for compliance.

• Utilities: Detect safety or grid issues.

Key Takeaway

Text clustering + NLP enables organizations to transform unstructured customer data

into actionable innovation and enhanced decision-making.

9. Popular NLP Tasks and Applications

Task Description

Automatically answers questions posed in natural

Question Answering (QA)
language.

Automatic Summarization Creates concise summaries of long documents.

Natural Language Generation Converts database information into human-readable

(NLG) text.

Natural Language Converts human language into machine-processable

Understanding (NLU) formal representations.
Task Description

Machine Translation (MT) Automatically translates text between languages.

Foreign Language Reading Helps non-native speakers read text with

Assistance pronunciation aids.

Foreign Language Writing

Assists in writing correctly in another language.
Assistance

Speech Recognition Converts spoken words to text.

Text-to-Speech (TTS) Converts text into audible speech.

Text Proofing Detects and corrects grammatical or spelling errors.

Optical Character Converts scanned or handwritten text images into

Recognition (OCR) editable text.

10. The Role of NLP in Text Mining

Relationship:

Text Mining = NLP + Data Mining

Explanation:

• NLP → Extracts features and meaning from unstructured text.

• Data Mining → Applies analytical methods (classification, clustering,

association) on the extracted features to discover patterns and insights.

11. Summary

• Early bag-of-words approaches were simple but lacked contextual

understanding.

• NLP enables semantic and syntactic analysis, leading to better text mining
outcomes.

• NLP faces challenges like ambiguity, segmentation, and context dependence.

• Applications of NLP are vast — from sentiment analysis and machine

translation to speech recognition and automated summarization.

• The future of text mining relies heavily on advancements in NLP for better
accuracy and deeper understanding.
Lecture Notes – TEXT MINING APPLICATIONS

1. Introduction

• Definition:
Text mining refers to the process of deriving high-quality information from text.
It involves structuring unstructured text data, identifying patterns, and extracting
meaningful knowledge.

• Growing Importance:

o Explosion of unstructured data (emails, documents, web content, social

media).

o Organizations are realizing the value of text mining tools for decision
support, marketing insights, and security intelligence.

• Key Application Areas:

1. Marketing and Customer Relationship Management (CRM)

2. Security and Law Enforcement

3. Biomedical and Healthcare Research

4. Academic and Publishing Sectors

2. MARKETING APPLICATIONS

2.1 Customer Relationship Management (CRM)

• Purpose: Understand customer sentiments, predict behaviors, and improve

retention.

• Data Sources:

o Call center logs and transcribed conversations

o Customer reviews, blogs, and discussion boards

o Social media feedback

• Use Cases:

o Cross-selling and Up-selling:

Text mining helps identify customer needs and preferences.

o Customer Churn Prediction:

 ▪ Coussement & Van den Poel (2009) improved churn prediction
accuracy using text mining.

o Product Attribute Extraction:

▪ Ghani et al. (2006) developed systems that infer implicit and

explicit product features using semi-supervised learning.

▪ Helps in demand forecasting, recommendation systems, and

supplier selection.

3. SECURITY APPLICATIONS

3.1 Surveillance and Intelligence Systems

• ECHELON System:

o A rumored global surveillance network for intercepting and analyzing

communications (calls, emails, faxes).

o Uses text mining to identify intelligence patterns.

• EUROPOL’s OASIS System (2007):

o Integrates structured and unstructured data for tracking organized crime.

o Uses advanced text and data mining technologies to support

international law enforcement.

• U.S. Homeland Security Initiative:

o Joint FBI–CIA system creating a large data warehouse with data/text

mining modules for national security intelligence.

4. BIOMEDICAL APPLICATIONS

4.1 Importance:

• Biomedical literature is structured, standardized, and rich, making it ideal for

text mining.

• Growth of open-source journals and bioinformatics data increases the need

for automated analysis.

4.2 Key Applications:

1. Gene & Protein Analysis:

 o Experimental data (DNA microarrays, proteomics) must be analyzed with
existing literature.

o Text mining helps interpret and validate experimental findings.

2. Protein Location Prediction:

o Shatkay et al. (2007) combined sequence- and text-based features.

o Improved accuracy in predicting protein locations within cells.

3. Disease–Gene Relationship Extraction:

o Chun et al. (2006) extracted disease–gene links using MEDLINE data.

o Used Named Entity Recognition (NER) to reduce false positives.

o Improved relation extraction precision by 26.7%.

4. Gene/Protein Interaction Discovery (Figure 6.4):

o Text processed using tokenization, POS tagging, and ontology mapping.

o Helps decode complex relationships in the Human Genome Project.

5. ACADEMIC APPLICATIONS

5.1 For Publishers

• Publishers (e.g., Nature, NIH) develop open text mining interfaces.

• Aim: Improve indexing, retrieval, and semantic understanding of scholarly

content.

5.2 For Research Institutions

• National Centre for Text Mining (UK):

o Collaboration between Manchester and Liverpool Universities.

o Provides customized tools for biological and social science research.

• BioText Project (UC Berkeley):

o Helps bioscience researchers mine and analyze scientific text data.

6. SUMMARY

Domain Purpose / Outcome

Customer insights, churn prediction, product attribute

Marketing
discovery

Security Surveillance, crime tracking, deception detection

Gene–protein interaction, disease–gene mapping, protein

Biomedical
localization

Indexing scientific databases, supporting open text mining

Academic
initiatives

7. Key Takeaways

• Text mining transforms unstructured text into actionable knowledge.

• Applications span multiple sectors—from commerce to healthcare to national

security.

• Integration of text mining and machine learning significantly enhances

predictive accuracy.

• Future trend: Cross-domain integration (e.g., combining biomedical text mining

with AI-driven knowledge graphs).
Lecture Notes: TEXT MINING PROCESS

Overview

• Text mining projects require a structured methodology similar to CRISP-DM

(Cross Industry Standard Process for Data Mining).

• However, text mining emphasizes data preprocessing due to unstructured

textual inputs.

• The goal: extract meaningful, actionable knowledge from unstructured text

data for decision-making.

1. Context Diagram of the Text Mining Process

Key Components:

• Input:

o Unstructured data (text, XML, HTML, etc.)

o Structured data (from databases)

• Output:
 o Context-specific knowledge for decision-making

• Controls (Constraints):

o Software/hardware limitations

o Privacy and linguistic issues

• Mechanisms:

o Tools, techniques, and domain expertise

Purpose:
To process unstructured and structured data to extract useful patterns and
insights.

2. The Three-Step Text Mining Process

Step Task Input Output

Collection of
Establish the Unstructured
1 domain-specific
Corpus data
documents

Create the Term-

TDM with term
2 Document Matrix Corpus
frequencies
(TDM)

Discovered
Extract
3 TDM patterns and
Knowledge
models

Feedback loops allow going back to earlier tasks if expected outputs are not
achieved.
 Task 1: Establish the Corpus

Purpose:

• Collect and organize all documents relevant to the domain.

Data Sources:

• Textual documents, XML files, e-mails, web pages, notes

• Transcribed voice recordings using speech recognition

Processing Steps:

1. Convert documents into a uniform format (e.g., ASCII text).

2. Organize them into a digital corpus (folder or database).

3. Use text mining tools to convert corpus into a flat file ready for analysis.

Task 2: Create the Term–Document Matrix (TDM)

Definition:

• A matrix representation where:

o Rows = documents

o Columns = terms

o Cells = frequency of each term in each document

 Example:

Key Steps:

1. Filtering Unimportant Terms

o Remove stop words (e.g., “the,” “is,” “and”) that have no discriminative
power.

o Use domain-specific stop word lists.

2. Use of Dictionaries

o Create a predefined list of terms for indexing.

o Define synonyms and phrases (e.g., “Eiffel Tower”).

3. Stemming

o Reduce words to their root form.

o Example: modeling, modeled → model

4. Handling Large Matrices

o TDMs are often large and sparse.

o Dimensionality reduction is needed for efficient processing.

 Representing the Indices

Raw term frequencies can be transformed or normalized to better reflect

importance:

Representation Description

Reduces impact of extremely frequent

Log frequencies
words

Binary frequencies 1 = term present, 0 = absent

Inverse Document Reduces weight of common terms

Frequency (IDF) across documents

Dimensionality Reduction

Goal: Manage large, sparse matrices.

Methods:

1. Manual elimination of irrelevant terms (expert review).

2. Automatic filtering of rare terms.

3. SVD (Singular Value Decomposition):

o Related to Principal Component Analysis (PCA).

o Reduces dimensions to reveal latent semantic structure (hidden

meaning).

o Helps find relationships between terms and documents.

Task 3: Extract Knowledge

Once TDM is structured, apply data mining techniques to identify patterns and
relationships.

Main Methods:

1. Classification

2. Clustering

3. Association

4. Trend Analysis
 1. Classification

Purpose: Assign documents to predefined categories (topics, genres, etc.).

Examples:

• Spam vs. non-spam e-mails

• Automatic topic tagging

• Web page categorization

Approaches:

• Knowledge Engineering:

o Expert-defined rules and categories.

• Machine Learning:

o Model learns from labeled examples.

o More scalable and popular for large text sets.

2. Clustering

Purpose:
Group similar documents without predefined labels (unsupervised learning).

Applications:

• Document retrieval and browsing

• Web content organization

Benefits:

• Improved recall: Returns all documents in similar clusters.

• Improved precision: Narrows search results to the most relevant clusters.

Methods:

• Scatter/Gather: Dynamic grouping for browsing without specific query.

• Query-specific clustering: Hierarchical approach ranking clusters by relevance.

 3. Association

Definition:
Discover relationships among terms or concepts within documents.

Example:

• “Software Implementation Failure” often occurs with “ERP” and “CRM.”

o Support = 4% (in all documents)

o Confidence = 55% (within “Software Implementation Failure” docs)

Applications:

• Identifying co-occurring terms in literature (e.g., tracking disease spread).

4. Trend Analysis

Purpose:
Identify changes in concept frequency or relationships over time.

Example:

• Tracking evolution of research topics in academic journals.

• Comparing term distributions across different time periods.

Application:
Delen & Crossland (2008) analyzed IS research trends across top journals.

Summary Table:

Method Type Goal Example

Assign
documents to Spam
Classification Supervised
known detection
categories

Group similar Web page

Clustering Unsupervised
documents grouping

Pattern Find related ERP ↔

Association
discovery terms CRM
 Method Type Goal Example

Trend Detect concept Topic shift

Temporal
Analysis evolution in journals

Key Takeaways

• Text mining follows a structured three-step process:

Corpus creation → TDM formation → Knowledge extraction.

• Preprocessing (cleaning, stemming, stop-word removal) is critical.

• Dimensionality reduction (SVD) improves efficiency and meaning extraction.

• Final goal: Convert unstructured text into structured, decision-support

knowledge.
Lecture Notes: Overview of Sentiment Analysis
Sentiment Analysis (also called opinion mining) is a subfield of text analytics and NLP that
focuses on detecting opinions, emotions, and attitudes expressed in textual data.
It helps answer:
“What do people feel about a certain topic?”

With massive growth in social media, blogs, and review platforms, sentiment analysis allows
organizations to automatically analyze millions of text posts to understand public opinion
about products, brands, or policies.

Conceptual Background
• Sentiment vs. Opinion:
Sentiment refers to feelings or emotions expressed toward something, while opinion
can be more factual or belief-based.
• Polarity:
The degree or direction of sentiment — typically positive vs. negative, sometimes
extended to neutral or intensity-based (e.g., strongly positive).
• Explicit vs. Implicit Sentiment:
o Explicit: Direct expression — “The movie was fantastic.”
o Implicit: Indirect expression — “The movie put me to sleep.”

Major Applications
1. Voice of the Customer (VOC):
Understanding customer satisfaction and feedback to improve products/services.
2. Voice of the Market (VOM):
Detecting market trends and competitor insights.
3. Voice of the Employee (VOE):
Analyzing employee feedback to improve engagement.
4. Brand Management:
Monitoring and protecting a company’s online reputation.
5. Financial Markets:
Analyzing news, blogs, and tweets to forecast stock or market trends.
6. Politics:
Gauging voter opinions and predicting election outcomes.
7. Government Intelligence:
Monitoring sentiment for security or policy insights.
8. Other Uses:
Ad placement, e-commerce personalization, email triage, and citation analysis.
Sentiment Analysis Process (Unsupervised/Lexicon-Based
Approach)
A 4-step process (see Figure 6.9 in the text):

1. Sentiment Detection:
Identify whether a text is objective (fact) or subjective (opinion).
→ Compute Objectivity–Subjectivity (O–S) Polarity.
2. Polarity Classification (N–P Polarity):
Determine if the opinion is positive or negative, and its strength.
3. Target Identification:
Identify what or whom the opinion is about (e.g., a product, person, event).
 4. Collection and Aggregation:
Combine all identified sentiments into an overall score for the document or dataset.

Methods for Polarity Identification

Two main techniques:

1. Lexicon-Based (Unsupervised)

• Uses predefined word dictionaries (lexicons) with associated polarity scores.

• Example lexicons:
o WordNet
o SentiWordNet (assigns positivity, negativity, and objectivity scores)
o WordNet-Affect (adds emotional categories like anger, joy, etc.)
• Advantage: No labeled data required.
• Limitation: May not adapt well to domain-specific contexts (e.g., “cold” in medicine
vs. hospitality).

2. Machine Learning (Supervised)

• Uses labeled training data (e.g., positive/negative movie reviews) to train models.
• Example datasets:
o Cornell Movie Review Data
o Stanford Large Movie Review Dataset
o MPQA Corpus
o Restaurant Reviews Dataset
• Models learn patterns (words, phrases, structures) that correlate with sentiment
classes.

Example Tools & Datasets

Dataset Source Description
1,000 positive and 1,000
Cornell Movie Review Pang & Lee
negative reviews
25,000 labeled reviews
Stanford IMDB Reviews Stanford AI
(train/test)
Multi-perspective Question News articles with opinion
MPQA Corpus
Answering labels
Multiple-Aspect Restaurant Reviews rated on food,
Snyder & Barzilay
Reviews ambiance, service, etc.
Congressional Floor-Debate
Thomas et al. Political speech sentiment
Transcripts
 Dataset Source Description
Merchant feedback from
Economining NYU Stern
Amazon

Case Study: Wimbledon (IBM & AELTC)

IBM used sentiment analysis, real-time data analytics, and NLP to enhance Wimbledon’s
digital experience:

• Real-time match data (3.4M+ data points) analyzed for trends and narratives.
• IBM Watson mined unstructured data for historical and contextual insights.
• Cloud analytics and security ensured performance and data protection.
• Result: 71 million visits, 542 million page views — record engagement.

Key Takeaways
• Sentiment analysis bridges data mining, NLP, and social analytics.
• Enables real-time, large-scale opinion tracking for smarter decision-making.
• Techniques vary between rule-based lexicon models and machine learning
approaches.
• Widely applicable in marketing, finance, governance, and media.
• When combined with topic modeling, it provides context + emotion, enhancing
understanding of what people talk about and how they feel about it.
Lecture Notes – Web Mining Overview

1. Introduction
• The Internet has revolutionized business operations by creating:
o Opportunities – global customer reach, new markets.
o Challenges – intense global competition, fast-changing environments.
• Having a strong digital presence is now a business necessity, not a choice.
• Customers today:
o Expect online products and services.
o Share opinions, reviews, and experiences publicly via social media.

2. Importance of Web Data

• The Web serves as the world’s largest data repository, containing:
o Textual content (HTML/XML pages)
o Hyperlinks (connections between pages)
o Usage data (visitor behavior, clicks, transactions)
• Every action online (clicks, searches, transactions) generates valuable data.
• Analyzing this data helps:
o Improve websites and user experience.
o Enhance business relationships.
o Gain insights into customer behavior.

3. Challenges in Web Mining (Han & Kamber, 2006)

Challenge Explanation
The Web’s size and growth make complete data collection
1. The Web is too big
impossible.
2. The Web is too complex Web pages lack structure; styles and formats vary widely.
3. The Web is too dynamic Content is updated constantly (news, stock, blogs, etc.).
4. The Web is not domain- Users have diverse backgrounds and needs; searches are often
specific inefficient.
Most information is irrelevant to a specific user — finding
5. The Web has everything
relevant data is difficult.
4. From Web Search to Web Mining
• Search Engines index web pages using keywords, but have limitations:
o Return too many (often irrelevant) results.
o Miss relevant documents lacking exact keywords.
• Web Mining goes beyond keyword search, uncovering:
o Patterns, relationships, and authoritative sources.
o Classifies and ranks web documents intelligently.

5. Definition of Web Mining

Web Mining (or Web Data Mining) is the process of discovering intrinsic patterns and
relationships from Web data — expressed in textual, linkage, or usage form.
(First introduced by Etzioni, 1996)

Goal:

Transform vast web data (transactions, interactions, logs) into actionable knowledge for
better decision-making.

6. Web Mining vs. Web Analytics

Aspect Web Mining Web Analytics
Discovery of new patterns and
Focus Measurement of predefined metrics
relationships
Includes content, structure, and usage Mainly usage and site performance
Scope
data data
Approach Predictive / Prescriptive analytics Descriptive analytics
Outcome Generates new knowledge Describes “what happened”
Relation Web analytics is a subset of Web mining –
7. Taxonomy of Web Mining

Web Mining is divided into three main areas:

1. Web Content Mining

• Source: Unstructured text/content from web pages (HTML format).

• Focus: Extracting meaningful data from web documents.
• Techniques/Applications:
o Web crawlers (spiders): Automatically collect page data.
o Competitive intelligence: Monitor competitors’ products/services.
o Opinion and news mining: Extract sentiment or summarize reviews.
o Automated data collection: For predictive modeling.

Example:

• Movie success prediction: Data about movies (cast, budget, release info) is
automatically collected from multiple web sources using crawlers.

2. Web Structure Mining

• Source: Hyperlinks (URL links) between web pages.

• Focus: Discovering relationships and authority/hub structures.
• Concepts:
o Authority Page: A highly referenced or “endorsed” page.
 o Hub Page: A page that links to many authorities (e.g., resource lists).
• Relationship:
A good hub links to many good authorities;
a good authority is linked by many good hubs.

Algorithms:

• HITS (Hyperlink-Induced Topic Search) – Kleinberg, 1999

o Calculates hub and authority scores recursively.
o Forms the basis for link analysis in modern search engines.
• PageRank (Google) – Uses link structure to rank importance.

Applications:

• Improve search engine ranking and relevance.

• Understand community or topic-based clusters of web pages.

3. Web Usage Mining

• Source: Web logs and user interaction data (clickstreams, sessions).

• Focus: Analyzing user behavior and site performance.
• Applications:
o Web analytics
o Clickstream analysis
o Customer journey mapping
o Marketing attribution
o Social media and customer sentiment tracking

8. Integration with Other Mining Domains

• Web Mining integrates with:
o Data Mining (structured data analysis)
o Text Mining (unstructured text extraction)
o Social Network Analysis
o Marketing and Customer Analytics

These collectively provide a 360° customer view — combining content, behavior, and
sentiment.

9. Summary
• Web mining transforms the massive, unstructured web into business intelligence.
• It addresses challenges of size, complexity, and dynamism.
 • Three main branches:
1. Content Mining → What is on the web?
2. Structure Mining → How is the web connected?
3. Usage Mining → How do users interact with it?
• Tools and algorithms like HITS and PageRank enable identifying authoritative
sources and user behavior patterns.
• Outcome: Smarter search, improved marketing, better customer experience, and data-
driven decisions.

Key Terms

• Web Crawler / Spider: Automated software to collect web data.

• Authority Page: Page highly endorsed by others (many incoming links).
• Hub Page: Page linking to multiple authorities.
• HITS Algorithm: Calculates authority and hub scores for link analysis.
• PageRank: Algorithm that ranks web pages based on link popularity.
• Clickstream: Sequence of clicks or navigation paths by users.
• Web Analytics: Metrics-based study of web traffic and behavior.
Lecture Notes: Search Engines
1. Introduction
• Definition: A search engine is a software system designed to search for information
(documents, websites, files) on the Internet based on user-entered keywords or
phrases.
• Purpose: Helps users find relevant content quickly from the vast and complex World
Wide Web.
• Example: Google, Bing, Yahoo, Baidu, DuckDuckGo.
• Importance: Central to most Internet-based transactions, research, and everyday
activities.

2. Working Principle
• Search engines locate, organize, and rank web content.
• Two key metrics:
1. Effectiveness (Quality): Accuracy of retrieved results.
2. Efficiency (Speed): Time taken to return results.
• Challenge: Improving one metric often compromises the other.
• Functionally, they are both searching and finding engines.

3. Anatomy of a Search Engine

A search engine consists of two major cycles:

A. Development Cycle (Offline / Back-end Process)

Objective: Build and maintain a massive, indexed database of web pages.

Main Components:

1. Web Crawler (Spider):

o Automatically browses the web to find and fetch web pages.
o Starts from a list of URLs (called seeds).
o Follows hyperlinks and adds discovered URLs to the scheduler.
o Policies determine crawling priority and frequency.
2. Document Indexer:
o Processes fetched pages into a searchable structure.
o Steps:
▪ Step 1: Preprocessing
 ▪ Convert pages into a standardized format (text, hyperlinks,
images separated).
▪ Step 2: Parsing
▪ Apply NLP/text mining techniques:
▪ Tokenization
▪ Stop-word removal
▪ Spelling correction
▪ Stemming (reduce words to root form)
▪ Synonym & homonym detection (WordNet, lexicons)
▪ Step 3: Creating Term-by-Document Matrix
▪ Create relationships between terms and documents.
▪ Assign weights using TF–IDF (Term Frequency / Inverse
Document Frequency).
▪ Produces an indexed database for fast retrieval.

B. Responding Cycle (Online / Front-end Process)

Objective: Respond to user queries with the most relevant and ranked results.
Main Components:

1. Query Analyzer:
o Converts user query into the same structure used by the indexer.
o Performs tokenization, stop-word removal, stemming, and disambiguation.
o Ensures compatibility with document database.
2. Document Matcher / Ranker:
o Matches the processed query with indexed documents.
o Ranks results by relevance.
o Ranking algorithms determine the order of displayed results.
o Example:
▪ Early systems used keyword frequency and matching.
▪ Google’s PageRank Algorithm (1997) revolutionized ranking by
evaluating the importance and relevance of pages.
o Search engines also monitor user click behavior to refine ranking algorithms
continuously.
4. Search Engine Optimization (SEO)
Definition:
SEO is the practice of improving a website’s visibility in organic (non-paid) search engine
results.

Objectives:

• Increase ranking position.

• Drive more organic traffic.
• Improve website visibility and conversion rate.

Factors Considered:

• How search engines work.

• Keywords people use.
• Preferred search engines of the target audience.

5. SEO Process
1. Content Optimization
o Use relevant keywords naturally.
o Maintain quality and originality of content.
2. HTML Optimization
o Improve title tags, meta descriptions, headings, and URLs.
3. Link Building
o Increase the number of backlinks (external sites linking to your content).
4. Regular Updates
o Fresh and relevant content attracts frequent crawling.
5. URL Normalization
o Ensure all versions of a URL direct to one canonical version for consistency.

6. Evolution of SEO
• Early Days:
o Webmasters submitted URLs manually for indexing.
o Search engines used spiders to crawl and store content.
• Modern Approach:
o Automatic, continuous crawling and indexing.
o Advanced algorithms detect manipulation and spam.
7. SEO Methods
A. White-Hat SEO (Ethical)

• Follows search engine guidelines.

• Focuses on user experience.
• Practices include:
o Quality content creation.
o Proper metadata usage.
o Accessible site structure.

B. Black-Hat SEO (Unethical)

• Violates search engine rules to gain higher rankings quickly.

• Practices include:
o Hidden text or links.
o Cloaking (showing different content to users and crawlers).
o Keyword stuffing.
• Consequences: Penalization or permanent ban from search listings.
o Example: BMW Germany and Ricoh Germany (banned by Google in 2006).

8. SEO Challenges and Risks

• Frequent algorithm updates by search engines.
o Google made 500+ changes in 2010 (~1.5 per day).
• No guarantee of stability or consistent ranking.
• Heavy dependence on search engine traffic is risky.

Strategies to Mitigate Risk:

1. Hire specialized SEO firms.

2. Use paid listings or sponsored results.
3. Diversify marketing beyond search engines.

9. Top Search Engines

(As per 2016 eBizMBA Rank)

Rank Search Engine Estimated Monthly Visitors

1 Google 1.6 billion
2 Bing 400 million
3 Yahoo! Search 300 million
Rank Search Engine Estimated Monthly Visitors
4 Ask 245 million
5 AOL Search 125 million
... ... ...

(As per 2022 ReliableSoft Update)

1. Google
2. Microsoft Bing
3. Yahoo
4. Baidu (China)
5. Yandex (Russia)
6. DuckDuckGo
7. [Link]
8. Ecosia
9. [Link]
10. Internet Archive

10. Summary
• Search engines are complex information retrieval systems.
• Operate via two main cycles: Development and Responding.
• Crawling, indexing, ranking, and query analysis are core components.
• SEO ensures better visibility and ranking through ethical (white-hat) or unethical
(black-hat) practices.
• Continuous improvement in algorithms ensures better relevance, accuracy, and
fairness in results.

Key Terms:

• Crawler/Spider – Program that browses the web for indexing.

• Indexing – Process of organizing content for retrieval.
• TF–IDF – Statistical measure of term relevance.
• PageRank – Google’s algorithm for ranking web pages.
• White-Hat SEO – Ethical optimization techniques.
• Black-Hat SEO – Manipulative, disapproved methods.
Lecture Notes: Web Usage Mining (Web
Analytics)

1. Introduction
Definition:

• Web Usage Mining (WUM), also known as Web Analytics, is the process of
extracting useful insights from data generated by web visits, clicks, and
transactions.
• It helps organizations understand user behavior and interaction patterns on their
websites.

Purpose:

• To analyze clickstream data — the sequence of web pages visited by users — for
improving website design, marketing, and customer experience.

Example:

• If 60% of visitors who searched “hotels in Maui” also searched for “airfares to Maui,”
the company can cross-promote flight and hotel deals.
• If 70% of downloads occur between 7–11 p.m., companies can allocate more
bandwidth or support staff during those hours.

2. Process of Web Usage Mining

The general process (as in Figure 6.14) involves:
A. Data Collection

• Data is gathered from web logs and other sources.

B. Data Preprocessing

• Merging logs from multiple servers.

• Cleaning (removing bots, duplicates, errors).
• Structuring the data by:
o Identifying users.
o Identifying sessions.
o Identifying page views and visits.

C. Knowledge Extraction

• Applying data mining techniques to derive:

o Usage patterns
o User profiles
o Page profiles
o Visit profiles
o Customer value

D. Using the Knowledge

• To improve the website, enhance data quality, and increase customer value.

3. Web Analytics Technologies

Definition:

• Web analytics technologies measure, collect, and analyze web data to understand
and optimize website performance.

Applications:

• Measure web traffic and user engagement.

• Evaluate marketing campaign effectiveness.
• Support e-business and market research.
• Assess ROI of online and offline promotions.

Examples of Tools:

• Google Analytics (most widely used)

• Yahoo! Analytics
• Microsoft Clarity
• Other modern tools (e.g., Matomo, Adobe Analytics)
4. Types of Web Analytics
A. Off-Site Web Analytics

• Measures website performance outside your own site.

• Focuses on:
o Potential audience size (reach).
o Share of voice (visibility across the web).
o Buzz or sentiment (public opinion and mentions).

B. On-Site Web Analytics

• Measures visitor behavior on your website.

• Tracks:
o Traffic sources.
o Page visits, conversions, and user flow.
o Effectiveness of campaigns and landing pages.
• Tools often combine both off-site and on-site features.

5. Data Collection Methods

Two main technical methods for on-site analytics:

1. Server Log File Analysis

• Web servers automatically record all user requests.

• Advantages: Historical, raw data.
• Disadvantages: Limited detail on user actions beyond requests.

2. Page Tagging

• Involves JavaScript code embedded in web pages.

• When a page loads or a click occurs, a signal is sent to an analytics server.
• Provides real-time, event-level tracking.

Additional Data Sources:

• Email campaigns
• Social media engagement
• Sales and leads database
• Direct marketing data
6. Web Analytics Metrics
Purpose:

• Provide actionable insights for improving website usability, traffic acquisition,

customer segmentation, and conversion rates.

Four Main Categories (The Westover Group, 2013):

1. Website Usability – How visitors use your site.

2. Traffic Sources – Where visitors come from.
3. Visitor Profiles – Who your visitors are.
4. Conversion Statistics – What actions visitors take and what it means for business.

7. Website Usability Metrics

Metric Description & Use
Avg. pages viewed per visitor; low values may indicate poor site design or
Page Views
irrelevant content.
Time on Longer time indicates engagement; must be checked with page views to ensure
Site users aren’t lost.
Track engagement with downloadable resources (e.g., PDFs, videos).
Downloads
Example: If 60% of demo viewers buy, promote the demo more.
Click Map Shows where users click most; helps identify important or ignored elements.
Click Paths Traces user navigation routes; identifies where users drop off or lose interest.

8. Traffic Sources Metrics

Traffic Source Description
Referral Other sites that send traffic via links. Measure conversions from each
Websites referral.
Search Engines Includes paid and organic searches; analyze top-performing keywords.
Visitors typing your URL directly or using bookmarks (from print ads,
Direct Traffic
brochures, etc.).
Offline Measureable using dedicated URLs (e.g., [Link]/offer50)
Campaigns in print/radio ads.
Online Track banner ads, email marketing, and PPC campaigns via tagged
Campaigns URLs.

9. Visitor Profiles
Segmentation Variables:

Aspect Insights Gained

Keywords Identify visitor intent—awareness vs. problem-solving.
Content
Track activity by product/service category.
Groupings
Geography Determine regional or city-wise engagement.
Identify peak browsing or purchase hours for support or marketing
Time of Day
timing.
Landing Pages Measure effectiveness of campaign-specific landing pages.

10. Conversion Statistics

Metric Meaning & Use
New Visitors Measure reach and visibility growth.
Returning Visitors Indicates loyalty or long purchase cycles.
Completed forms or registrations; low completion = form design
Leads
issue.
Sales/Conversions Completed purchases or actions; monitor for trends.
Abandonment/Exit Identify where users drop off or leave; high exits may indicate
Rates content or UX mismatch.

Example:
If many users leave after the pricing page, review its layout, clarity, or call-to-action.

11. Web Analytics Dashboard

• Dashboards display key performance indicators (KPIs) such as:
o Page views, sessions, bounce rates, conversions, traffic sources.
o Can be generated using tools like Google Analytics Dashboard (see Figure
6.15).
• Used for weekly/monthly reviews to monitor performance and ROI.

12. Benefits of Web Analytics

Understand user behavior & preferences.
Optimize website design and navigation.
Measure effectiveness of marketing campaigns.
 Improve customer engagement and conversion rates.
Provide data-driven decision support for business growth.

13. Summary
• Web Usage Mining helps convert raw web interaction data into actionable business
intelligence.
• Core steps: Collect → Preprocess → Extract Knowledge → Apply Insights.
• Web analytics tools provide near real-time performance metrics.
• Key focus areas include usability, traffic, visitor segmentation, and conversion
optimization.
• When used effectively, web analytics drives higher ROI, better user experience,
and informed strategic decisions.

Key Terms:

• Clickstream Data – Sequence of user interactions on a site.

• Page Tagging – Embedding JavaScript code for tracking.
• Conversion – Desired user action (purchase, signup, download).
• Bounce Rate – Percentage of visitors leaving after one page.
• KPI – Key Performance Indicator for website success.

Lecture Notes: Social Analytics

1. Introduction
• Definition (General Meaning):
The term social analytics originally referred to a philosophical concept by Lars-
Henrik Schmidt (1980s) focusing on “socius” — commonness that is neither
universal nor communal.
• Contemporary (Business/IT) Definition:
According to Gartner,

“Social analytics is the monitoring, analyzing, measuring, and interpreting of digital

interactions and relationships of people, topics, ideas, and content.”

• Purpose:
o To gain insights into customer behavior, preferences, and opinions.
 o To analyze how people interact and influence each other through digital
platforms.

2. Branches of Social Analytics

Social analytics can be divided into two interrelated branches:

1. Social Network Analysis (SNA)

2. Social Media Analytics (SMA)

3. Social Network Analysis (SNA)

Definition

• SNA studies the structure and dynamics of relationships between individuals,

groups, or organizations.
• Based on network theory, where:
o Nodes: Individuals or organizations.
o Ties (Edges): Relationships or connections (e.g., friendship, communication,
transactions).

Origins

• Interdisciplinary field combining sociology, psychology, statistics, and graph

theory.
• Mathematical foundation developed in 1950s, and formal theories in 1980s (Scott &
Davis, 2003).

Applications

• Business analytics and marketing.

• Consumer intelligence.
• Criminology, sociology, and innovation studies.
 4.

Types of Social Networks (Relevant to Business)

Type Description / Application
Represent information flow among individuals (e.g., telecom data,
Communication
email networks). Used by telecom companies to improve customer
Networks
relationships.
Traditionally local, now include online communities via social
Community
networking platforms. Used to analyze user interactions and
Networks
preferences.
Analyze relationships among criminals or gangs for law enforcement
Criminal Networks
and crime prevention.
Innovation Study how ideas and innovations spread among members; helps
Networks identify early adopters and innovation influencers.

5. SNA Metrics (Measurement Categories)

SNA metrics are grouped into three main categories:

A. Connections (Relational Characteristics)

Metric Description
Tendency for similar individuals to connect (by gender, age,
Homophily
status, etc.).
Number of different types of relationships between two people
Multiplexity
(e.g., co-workers and friends = 2).
Mutuality / Reciprocity Degree to which connections are reciprocated.
 Metric Description
Network Closure /
Extent to which one’s friends are also friends with each other.
Transitivity
Tendency to form ties with geographically or socially close
Propinquity
individuals.

B. Distributions (Structural Properties)

Metric Description
Bridge A node connecting two otherwise separate groups (fills a “structural hole”).
Centrality Indicates importance or influence of a node. Types:

• Degree Centrality – number of direct connections.

• Closeness Centrality – distance to all other nodes.
• Betweenness Centrality – control over information flow.
• Eigenvector Centrality – influence based on neighbors’ importance. |
| Density | Ratio of existing ties to all possible ties in the network. |
| Distance | Minimum number of connections between two nodes. |
| Structural Holes | Gaps where connections are missing; filling them gives
competitive advantage. |
| Tie Strength | Combination of time, emotional intensity, intimacy, and reciprocity
(strong vs weak ties). |

C. Segmentation (Group-level Structures)

Metric Description
Cliques / Social Groups where everyone is connected to everyone (cliques) or loosely
Circles connected (social circles).
Clustering
Likelihood that two associates of a node are connected to each other.
Coefficient
Degree of connectedness; how many members can be removed before
Cohesion
the network breaks apart.

6. Social Media Analytics (SMA)

Definition

• Refers to collecting and analyzing data from social media platforms to understand
behaviors, opinions, and trends.
• Combines text analytics, network analysis, and sentiment analysis.
Key Features of Social Media

• Built on Web 2.0 technologies enabling user-generated content and interaction.

• Examples: Facebook, Twitter, YouTube, LinkedIn, Instagram, Blogs, Wikis.

Types of Social Media (Kaplan & Haenlein, 2010)

Type Example
Collaborative Projects Wikipedia
Blogs & Microblogs Twitter
Content Communities YouTube
Social Networking Sites Facebook
Virtual Game Worlds World of Warcraft
Virtual Social Worlds Second Life

7. Difference Between Social Media and Industrial Media

Characteristic Social Media Industrial Media
Quality Varies widely (from high to low) Controlled and consistent
Reach Global, decentralized Global, but centralized
Frequency Rapid and continuous Infrequent and scheduled
Accessibility Open and low cost Costly, corporate/government-owned
Usability Easy for anyone to use Requires professional skills
Immediacy Instant feedback Slow publishing cycle
Updatability Can be edited anytime Fixed once published

8. Levels of Social Media Engagement (Brogan & Bastone,

2011)
Level (from least to most engaged)
Inactives
Spectators
Collectors
Joiners
Critics
Creators

• Trend: Over 82% of online adults now participate actively in at least one upper
engagement category.
9. Example Case Study – Sport England (KNIME + Power
BI)
• Goal: Automate reporting of social media campaign performance.
• Solution:
o Used KNIME Analytics Platform + Azure SQL + Power BI.
o Collected data from Twitter, Facebook, Instagram, YouTube, and Medium.
o Performed sentiment analysis using Azure Cognitive Services.
o Reports auto-generated daily instead of monthly.
• Benefits:
o Automated, branded, and real-time analytics.
o Reduced manual effort and improved responsiveness.
o Enabled data-driven campaign evaluation (“This Girl Can” campaign).

10. Measuring Social Media Impact

Analytics Type Purpose
Descriptive Analytics Track activity and trends (followers, posts, likes, etc.).
Social Network
Identify connections, influencers, and influence patterns.
Analysis
Predictive and text analytics to detect sentiment, topics, and hidden
Advanced Analytics
insights.

11. Best Practices in Social Media Analytics (Paine &

Chaves, 2012)
Practice Key Idea
1. Measurement as Use analytics to guide strategy, not to reward/punish. Focus on
Guidance what works.
2. Track Sentiment Measure customer emotions (positive, negative, neutral).
Accurately Separate mixed-polarity statements for clearer action.
3. Improve Text Analysis Refine NLP models and vocabularies regularly to enhance
Continuously accuracy.
Study how messages spread — what content goes viral and
4. Analyze Ripple Effect
why.
Track overall industry and customer conversations, not just
5. Look Beyond the Brand
brand mentions.
Detect individuals shaping opinions and trends — not just
6. Identify Influencers
brand advocates.
 Practice Key Idea
7. Verify Analytics Evaluate tool accuracy (typically 60–90%, depending on data
Accuracy type).
8. Integrate Insights into Use analytics findings in strategic and operational decision-
Planning making cycles.

12. Summary
• Social Analytics integrates Social Network Analysis and Social Media Analytics to
understand relationships, influence, and sentiment in online environments.
• SNA explores how entities connect, communicate, and influence each other.
• SMA mines user-generated content to extract actionable business insights.
• Best practices emphasize accurate measurement, influencer tracking, continuous
model tuning, and integration into business strategy.

Key Terms Recap

• SNA: Study of connections among social entities.

• SMA: Analysis of social media data.
• Centrality: Node importance in a network.
• Sentiment Analysis: Identifying opinions and emotions in text.
• Homophily: Tendency to connect with similar individuals.
• Structural Hole: Gap between disconnected groups offering strategic advantage.