Text Mining/Analytics
in R
Exploratory text analysis
Text mining
Extracting relevant information or knowledge from text data,
Not always sure what we are looking for (until we find it)!
Exploratory text analysis
• Gain an initial understanding of the text data
• Clean and preprocess the texts
• Identify patterns and data characteristics
Exploratory text analysis serves as a first step towards
further statistical analysis (e. g. sentiment analysis,
text classification, . . . )
Text Analytics
• Analyzing Text Data to Extract Meaning and Insights
• Text analytics in data science involves using
statistical and machine learning techniques to extract
insights from unstructured text data.
Source
Text data can come from various sources:
• Websites
• Books
• Social media
• Databases
• Digital scans of printed materials
Typically in unstructured format (data without a pre-defined
data model)
Oracle said that Approximately 90% of the world’s data is held
in unstructured formats.
Text Data
• Texts are stored as raw character strings.
• Text string contains tokens, which is a semantically
meaningful unit of text.
• Tokens can be words, sentences, paragraphs, etc.
Token Type Count
Document 1
Paragraphs 4464
Sentences 6044
Words 47707
Goal is to transform the raw string into tokens to
perform meaningful text analysis
Various Text Data Types
• Shorter Text: This includes tweets, comments, and
chat messages, which are typically shorter than 140
characters.
• Longer Text: This includes articles, reports, and
essays, which are typically longer than 140 characters.
• Semi-structured Text: This includes emails, support
tickets, and product reviews, which have a basic
structure but also contain unstructured information.
• Multilingual Text: This includes text data in different
languages and requires specialized techniques for
analysis.
Preprocessing steps
1. Text cleaning: Involves removing irrelevant information such as stop words, punctuation, and
special characters.
Stop words: These are common words that add little meaning to text analysis, such as "is", "the",
"and", "in", etc.
Example:
Original: “The cat is on the mat.”
Without stop words: “cat mat”
Punctuation: These are symbols like . , ! ? ; : that don't add semantic meaning in many NLP tasks.
Example:
Original: “Hello, world! r language is great.”
Without punctuation: “Hello world r language is great”
Special characters: These are symbols like @ # $ % ^ & * which are generally irrelevant unless
contextually important (e.g., in tweets or passwords).
Example:
Original: “Get 50% off now!!! #Sale @Amazon”
Without special characters: “Get 50 off now Sale Amazon”
2. Text normalization: This involves converting text to a standard format,
including converting all text to lowercase, stemming, and lemmatization.
Lowercasing
Converts all characters to lowercase to ensure uniformity and reduce duplicates.
Before: “Apple is not apple.”
After: “apple is not apple.”
Stemming
Reduces words to their root form by removing suffixes. It may not always produce
real words.
Before: “running”, “runner”, “ran”
After (Stemmed): “run”, “runner”, “ran” → often becomes “run”
Lemmatization
Also reduces words to their base form, but ensures the output is a valid word
using vocabulary and context (like part of speech).
Before: “running”, “better”
After (Lemmatized): “run”, “good”
3. Text tokenization: This involves breaking down the text into
smaller parts, such as words or phrases.
Type Input Text Output Tokens
Word Tokenization "AI is powerful!" ['AI', 'is', 'powerful', '!']
"AI is powerful. It changes ['AI is powerful.', 'It changes
Sentence Tokenization
the world." the world.']
4. Text encoding: This involves converting text into numerical
representations that machine learning algorithms can process.
One-Hot Encoding: Each word is represented as a binary
vector with one "hot" (1) value and the rest "cold" (0).
Vocabulary: ["apple", "banana", "orange"]
Word One-Hot Vector
apple [1, 0, 0]
banana [0, 1, 0]
orange [0, 0, 1]
Label Encoding:
Assigns each word an integer label.
Word Label
apple 0
banana 1
orange 2
TF-IDF (Term Frequency-Inverse Document Frequency):
Weights words by their importance in a document relative to a
collection of documents.
"cat sat on the mat“
"dog sat on the log“
TF-IDF gives higher weight to unique words like "cat" and "dog"
and lower to common ones like "on", "sat".
Extracting knowledge from text data
• Define The Problem: questions you want to ask with text
analytics like customer sentiment, emerging trends etc.
• Collect the data: from various sources, or source
• Preprocessing
• Apply text analytics techniques: such as sentiment analysis,
text classification, text summarizing etc.
Text analytics and NLP
Natural language processing (NLP) uses computational
methods to analyze and comprehend natural language data, is
a subfield that includes text analytics.