Module 5

This is a single, concatenated file, suitable for printing or saving as a PDF for offline viewing. Please
note that some animations or images may not work.

Module 5
April 8 - April 14

Topics: Lecture 9: Association Rules

Lecture 10: Text Mining

Readings: Lecture 9:
Lecture Notes
Shmueli et al., Chapter 14

Lecture 10:
Lecture Notes
Shmueli et al., Chapter 20

Tutorials: Lecture 9:
No tutorials this lecture

Lecture 10:
Text Mining

Discussions: Discussion 5

Initial post due Thursday, April 10 at 11:59 PM ET

Respond to at least two of your classmates' posts by Monday, April 14 at
11:59 PM ET

Assignments: Team Project Executive Briefings (per team)

Assessments: None this week

Live Classroom: Module 5 Live Classroom Session: Monday, April 14 at 8:00pm ET

Lecture 9: Association Rules

Learning Objectives

After successfully completing the lesson, students will be able to:

1. Describe the general concept behind association rules, along with potential business uses for such rules.
2. Identify association rules as a form of unsupervised learning.
3. Given a particular set of transactions, identify the support, confidence, and lift for various itemsets.
4. Differentiate between user-based collaborative filtering and item-based collaborative filtering.

Association Rules

Association rules are developed through an unsupervised learning process that helps us understand what sorts of products tend to be consumed along with other products.
Association rules are used to help drive many modern recommendation systems, including the ones that you see on sites such as Amazon and Netflix.

In some cases, association rules will simply identify relationships that might have made intuitive sense anyway — for example, if someone tells you that ketchup and mustard
tend to be purchased with hot dog rolls, your (polite) response might be something along the lines of “Okay, now please tell me something I don’t already know.”

Association rules become more interesting, and more valuable to a business, when they reveal relationships that are less immediately obvious. What if, for example, a

supermarket’s data shows that consumers who buy diapers are more likely to also purchase beer?1 What if Amazon shoppers who buy certain types of novels are more likely to
buy dark-colored toasters than light-colored toasters? When we uncover patterns such as these, we can have fun speculating about the “why” behind such relationships, but
most experienced data analysts will insist that the “why” isn’t really important — the important thing to focus on is the “what” — the knowledge of the relationship itself.

In R, we commonly use a transactions database, also known as a sparse matrix, to analyze data in order to discover association rules. To learn more about why this data
structure is ideal for transactions analysis, please view “Association Rules and the Sparse Matrix” in the AD699 video library on the R Videos page.

The section below will cover three common components of association rules — support, confidence, and lift. To illustrate the concepts, we’ll use a hypothetical set of 10
transactions at a fast food restaurant. This data will be presented in a binary matrix format (see Figure 11.1), with “1” indicating that the consumer purchased the particular item,
and “0” indicating that the consumer did not purchase the item. Assume that no consumer purchased more than one of any particular item.

1
This is one of the iconic association rules examples. Some have claimed that the diapers-beer relationship is apocryphal.

Support

The first association rule that we will explore here is that of support. Support can be expressed either as a whole number or as a percentage, and it indicates how much of a
particular itemset appears in a larger set of transactions (in these lecture notes and throughout AD699, we will express support as a decimal or fractional value). Consider the
following hypothetical transactions data from a fast-food restaurant:

Figure 11.1: AD699 Fast Food Transactions Matrix

Transaction Apple
Burger Fries Coke Shake Sundae
# Pie

1 1 1 1 0 0 0

2 0 0 0 1 0 0

3 1 0 1 0 1 0

4 1 1 0 1 0 0

5 0 0 0 0 0 1

6 1 0 1 0 0 0

7 1 1 1 1 0 0

8 0 0 1 0 1 1

9 1 1 1 0 0 1

10 0 0 0 1 0 1

In the chart above, the support for {Burger} is 6/10, or 0.6. To look at another single-item itemset, the support for {Shake} is 4/10, or 0.4. To find the support for {Burger,
Sundae}, we need to find the number of transactions that include each of these items, and then divide by the total number of transactions. Since there is only one such
transaction, the support for {Burger, Sundae} is 0.1. Note that we use the term “itemset” regardless of the number of items in the set — we can have a single-item itemset, such
as {Shake}, a two-item itemset {Burger, Sundae}, a three-item itemset {Burger, Sundae, Fries} and so on.

Support might seem too basic to be very useful, but it does have some important purposes. Chiefly, support is used to establish cutoff values. For example, let’s say we were
looking at a set of grocery store purchases, with 10,000 total transactions and 586 unique items. If we want to identify frequent itemsets, we might use a cutoff of 5%, or 0.05.
We can use something called the a priori algorithm to help us identify these itemsets; first, it would comb through all the transactions and eliminate any single itemsets that
occured fewer than 500 times. That would quickly weed out any items that we deemed too rare to really matter in our analysis. Then, it would iterate through all the two-item

itemsets the same way, and so on.2

Test Yourself
Based on the transactions data in Figure 11.1, what is the support for {Burger, Coke, Apple Pie}?

0.6

This is false.
 0.1

This is true.

1.0

This is false.

0.75

This is false.

Answer: b. Since exactly 1 transaction out of 10 contains this entire itemset, the support for {Burger,
Coke, Apple Pie} is 0.1.

2
To find frequent two-item itemsets, the algorithm would not need to re-check the items that were discarded on the first iteration. If anything did not occur 500 or more times as a
single itemset, it could not have occurred 500 or more times as part of a two-item itemset.

Confidence

The next relationship that we’ll explore is confidence. The confidence of a particular rule can be found by taking the consequent of the rule (right-hand side) and dividing it by
the antecedent of the rule (left-hand side). Another way to state that is: GIVEN the antecedent, what is the probability of the consequent occurring? If you prefer to think of it as
a conditional probability problem (which it certainly is!) just remember to put the antecedent in the denominator and the consequent in the numerator. Remember also that the
antecedent must be completely disjoint from the consequent — in other words, the two cannot overlap at all.

To use an example from Figure 11.1, let’s find the confidence for IF {Burger} THEN {Coke}. Here, burger is our antecedent, and Coke is our consequent. There are six total
transactions with the antecedent; of these, five also include the consequent. The confidence for this rule is 5 , or 0.83.
6

One problem with interpreting confidence is the possibility that the two itemsets are just both frequently purchased. A supermarket might find, for example, that there is a high
confidence percentage for the rule IF {Eggs} THEN {Pasta}. That might not necessarily reflect any real relationship between the two products, but simply the fact that many
consumers buy both of them. For that reason, lift can often be a more meaningful metric (lift will be described in detail in the following section).

Another point worth mentioning about confidence rules is that they sometimes require human review before going into product recommendations. Headphones are frequently
purchased with smartphones, so a confidence rule would imply a strong relationship, regardless of which item was used as the antecedent or the consequent. While it would
seem appropriate to recommend headphones to someone who has just bought a smartphone, it might seem strange to recommend a smartphone purchase to someone who
has just bought headphones.

Test Yourself
What is the confidence for the rule IF {Burger, Coke} then {Fries}?

0.75

This is false.

0.30

This is false.

0.60

This is true.

1.00

This is false.

Answer: c. There are five total transactions that contain the antecedent itemset. Of those five, three
also contain the consequent itemset, so the confidence for this rule is ⅗, or 0.60.

Lift

Lift, or lift ratio, compares the probability that a person will purchase a consequent itemset, given their purchase of an antecedent itemset, with the probability that a random
consumer from the transactions database will buy the consequent itemset. Lift ratio can be found by dividing the confidence of a rule by the benchmark confidence for the
consequent itemset. “The benchmark confidence for the consequent itemset” sounds like quite a mouthful of jargon, but it’s really just the support for the consequent. To take it
from abstraction into reality, let’s find the lift ratio for a particular rule using Figure 11.1.

3
Suppose we want to find the lift for IF {Burger, Fries} THEN {Coke}. The first thing we need is the confidence for this rule. That is 3 , or .75. There are four transactions that
4
include the antecedent itsemset; of these four, three also contain the consequent itemset. Now we need the benchmark confidence for {Coke} — in other words, how likely is
any consumer from this group to buy a Coke? That is 6 , or 0.6. Dividing the confidence of the rule by the benchmark confidence gives us .75 = 1.25.
10 .6

A potential pitfall associated with lift ratio interpretation involves the role of random chance. Particularly for very tiny datasets — the 10-transaction set used here certainly fits
that description — it is very hard to draw any meaningful conclusions from the lift ratio of any two itemsets.

Test Yourself
Based on the transaction data in Figure 11.1, what is the lift ratio for IF {Sundae} THEN {Apple Pie}?

0.85

This is false.

2.12

This is false.

1.00

This is false.

1.25

This is true.

Answer: d. First, we need to find the confidence for this rule. Since there are only two transactions
that contain the antecedent, and only one of these contains the consequent, the confidence is 0.5.
The benchmark confidence is 4 , or 0.4. The lift ratio for IF {Sundae} THEN {Apple Pie} is therefore
10
.5
, or 1.25.
.4

Wrap-Up of Support, Confidence, and Lift

Let’s take a look at the first row here (again, all of this comes from the audioscrobbler data set, which is part of the nutshell package). Rule #1 tells us that nearly 14 percent of
all audioscrobbler subscribers listened to both Green Day and the Red Hot Chili Peppers. Among people who listened to Green Day, 48 percent also listened to the Red Hot
Chili Peppers. Someone who listened to Green Day was twice as likely as a random person from the dataset to be a Red Hot Chili Peppers listener.

Collaborative Filtering

Collaborative filtering fuels most online product recommendation systems. It describes a system that will make recommendations to users based on the actions and preferences
of other users. There are different ways in which it can be set up — for example, an online video service could recommend products based simply on what other people viewed
or could instead make those recommendations based on what other people had offered ratings and/or written reviews for.

A major advantage of collaborative filtering — as opposed to content-based filtering — is that it is product-agnostic. Unlike the content-based filtering model that we saw with
Pandora’s Music Genome Product, it does not require a hugely manpower-intensive set-up process. Some of the results that it generates might be non-obvious or even
counter-intuitive, but, as stated in the introduction to this lecture, that's actually a positive thing.

One drawback to collaborative filtering is known as the “cold start” problem — without initial user data, a collaborative filtering system has nothing to get started with. However,
this can quickly be overcome as users of a system begin to make purchases and/or recommendations.

Item-based Collaborative Filtering

Item-based collaborative filtering can be done quite effectively with the association rules from the earlier part of this lecture. Most Amazon customers are familiar with this type
of system — this is what fuels the “Customers who bought X also bought Y” messages that you see as you build your online shopping cart through Amazon. These sorts of
recommendations can be quite powerful, as they seem to suggest to customers what else they might want, but not in an overly salesy way. Instead, the statement “also bought
Y” is presented as neutral and data-driven.

User-based Collaborative Filtering

With user-based collaborative filtering, product recommendations are made based on the similarity among the users themselves, rather than on any characteristics of the
products. For example, suppose that I distributed the seven-question survey shown below to all of the students in AD699. Students would be asked to just check one category
for each of the seven statements. Then, using one of the distance metrics that we covered in the section on k-nearest neighbors, I could determine which of the students in the
class are most similar to one another.

Strongly
Statements Disagree Neutral Agree Strongly Agree
Disagree

When I go on vacation, I want to do as little as possible — I’d

rather sit on the beach than explore museums and ancient ruins
all day.

Once my car loan is paid off, I immediately begin looking to

purchase a new car.

I would rather watch a horror movie than a comedy.

If it’s in the 70s or 80s outside, I’d rather open the window than
blast the A/C.

If I’m meeting someone at 11:30, I’m almost certain to show up

by 11:15.

I would rather cook a meal myself than order take-out.

I still keep in close contact with at least two friends from high
school.

Now, suppose that the survey results, along with the subsequent number-crunching, showed that the three students who are most similar to Adam are Brian, Claire, and
Denise. Let’s also imagine that we are running a streaming movie service called AD699Flix. If we notice that Adam gives a five-star rating to a new movie called Sharkstorm 7,
then if we’re using a user-based collaborative filtering, it would make sense for us to recommend this movie to Brian, Claire, and Denise, too. Likewise, we would recommend
movies that Brian rated highly to Adam, Claire, and Denise.

Summary

Relationships are not immediately obvious or intuitive. A collaborative filtering system is “product agnostic” — that is to say, it is concerned only with the relationships that it finds
among items and involves no analyst judgements or opinions about why certain items might go together. As mentioned at the beginning of this section, the AD699 Video Library
contains “Association Rules & the Sparse Matrix,” which includes analysis of support, confidence, and lift rules using the Groceries dataset from the arules package.

Lecture 10: Text Mining

Learning Objectives

After successfully completing the lesson, students will be able to:

1. Describe what text mining is and articulate why it is so useful and so prevalent today.
2. Identify what is meant by the term “bag-of-words,” and explain why such an approach generally makes a model simpler in design and easier to use.
3. Explain the concept of sentiment analysis, and identify why sentiment analysis is such a challenging problem in data mining.
So far in our course we have been working with structured quantitative data: numerical, binary (yes/no) and categorical. Here we will look at unstructured text.

What is Text Mining?

The Oxford English Dictionary defines text mining as "the process or practice of examining large collections of written resources in order to generate new information, typically
using specialized computer software." Text mining is a specific area of the data mining.

In short, text mining is the process of extracting meaningful information from text data.

The evolution of the field of text mining has been boosted by the wide availability of large amounts of text data, such as social media data.

We can formulate three different text mining goals:

1. 1. To label (classify) a document as belonging to a class or to cluster similar documents, with no attempt to extract the overall meaning from a single document. This is an
extension of predictive modeling and will be our focus in this lecture.
2. To extract an author’s emotional intent from text. This process is called sentiment analysis.
3. To extract more detailed meaning from a document. This is called “Natural Language Processing” (NLP). It is an ambitious undertaking and won’t be our focus in this
lecture. NLP is outside the scope of AD699.

The first goal requires a large collection of documents, the ability to extract predictor variables from documents, and for the classification task, lots of pre-labeled documents to
train a model. The models that are used, are the predictive models that we have already dealt with for numerical and categorical data.

The second and third goals might involve a single document.

For the third goal, the computer must learn at least some version of the complex “algorithms” that make up human language comprehension: grammar, syntax, punctuation, etc.
Or it must process a natural (non-computer) language to understand documents in that language.

Our focus will remain with the overall focus of our textbook and the first goal. The lecture will also give you a basic concept of sentiment analysis.

Text Mining Workflow

The text mining workflow is the transition process that turns unorganized text data into organized text data. This process breaks down into six steps:

The initial step is the definition of a problem and the final step is reaching an insight. Each step is important for a smooth transition.

Problem definition & specific goals

Identification of a problem is the initial step. The problem definition is fundamental for a text mining project.

Identify text to be collected

After the problem is defined and specific goals are set, then the next step is to identify text data to be collected. Understanding of the medium and data integrity is crucial for
project success when the analytical project deals with text mining.

Text organization
The next step is to organize the text data being collected. This could be done by source, by author, chronologically, or by any other significant criterion.

Feature extraction
Once the text data is organized, the next step is feature extraction. The exact method for feature extraction depends on the purpose of the text mining initiative or on the
selected text mining approach. Feature extraction could involve extracting word tokens or scoring sentiment.

Analysis
The fifth step is to apply analysis. In this lecture we will cover basic analytical methods applicable to text data.

Reach an insight, recommendation or output

The final and sixth step should give answers to the problems initially identified. Depending on the analytical task, these answers could include reaching an insight or
recommendation, or — when predicative modeling is in place — providing an output.

Syntactic Parsing vs. Bag-of-Words

Syntactic parsing and "bag-of-words" are two different text mining approaches.

Syntactic parsing extracts various features by considering word types and word order. For feature extraction, semantic parsing breaks up the text continually following a tree
structure. As a result, the words are tagged first as parts of the sentence, then as sub-parts, and then as a verbs, articles, adjectives or nouns (furthermore, a word can be
tagged as a proper noun or as a named entity). This means that a single word can produce multiple features and this makes the semantic parsing feature-rich.

In contrast, the "bag-of-words" approach doesn't consider either word types or word order. This method treats the words as tokens - attributes of the document. Each token is a
single feature.

In this lecture we are going to look deeper at the "bag-of-words" approach.

Here are the main characteristics of the "bag-of words" approach:

This method ignores grammar, syntax, punctuation and word order

The method treats the document being analyzed as a “bag of words”
This approach is effective for labeling (classifying) a document as belonging to a class, or clustering similar documents
The supervised machine learning is the quintessential application of the “bag-of-words” method
The method requires a large number of documents - a corpus*
The model doesn’t need 100% accuracy to be efficient for classification or clustering

Corpus
*"Corpus" can refer to a preselected standard set of documents which could be reused for

development, improvement and optimization of text mining algorithms.

Text Mining with a Bag of Words in R

Let’s see how the “bag-of-words” approach is utilized in R.

Creating a VCorpus Object

Text mining begins with loading text data into R. After the text data is loaded, the next step is to convert it to a corpus. In R, a “corpus” is defined as a data type for storing and
manipulating sets (collections) of text documents.

The corpus data type could be permanent corpus, PCorpus, or volatile corpus, VCorpus. The difference between these two kinds is the medium where the text data is
being stored. The PCorpus is saved to disk, while the VCorpus resides in the computer’s RAM and enables faster computing.
tm Package in R – Small Example
Given the following sentences:

S1. this is the first sentence.

S2. this is a second sentence.
S3. the third sentence is here.

To enter this text as a corpus:

# load tm package
library(tm)

# define vector of sentences ("docs")

text <- c("this is the first sentence",
"this is a second sentence",
"the third sentence is here")
# convert sentences into a corpus
crp <- VCorpus(VectorSource(text))

# examine the first document's content

content(crp[[1]])

Output:

> library(tm)
Loading required package: NLP
> text <- c("this is the first sentence",
+ "this is a second sentence",
+ "the third sentence is here")
> crp <- VCorpus(VectorSource(text))
> content(crp[[1]])
[1] "this is the first sentence"
>

To convert text data into a corpus, R must distinguish the elements in a text source and interpret each element as a document. For that purpose, the tm package provides
functions called source functions. The output of these functions is called a source object.
Creating a Larger VCorpus Object

# read zip file into a volatile corpus

corp <- VCorpus(ZipSoruce("[Link]", resursive = T))

# [Link] is a collection of 2,000 newsgroup documents

# partitioned in 2 different newsgroups
# corresponging to 2 different topics

# "recursive = T" stands to reach into subdirectories

# while getting documents

# examine the first document's content

content(corp[[1]])

# examine the one thousand and first document's content

content(corp[[1001]])

Output:

> content(corp[[1001]])
[1] "Newsgroups: [Link]"
[2] "Path: [Link]!rochester!udel!gatech!us
[3] "From: et@[Link] (Eric H. Taylor)"
[4] "Subject: Re: HELP_WITH_TRACKING_DEVICE"
[5] "Message-ID: <C4ruoI.B2z@[Link]>
[6] "Followup-To: [Link]"
[7] "Summary: underground and underwater wireless methods"
[8] "Keywords: Rogers, Tesla, Hertz, underground, underwater,
[9] "Sender: Eric H. Taylor"
[10] "Nntp-Posting-Host: [Link]"
[11] "Organization: 4-L Laboratories"
[12] "References: <00969FBA.E640FF10@[Link]>"
[13] "Date: Wed, 31 Mar 1993 21:07:28 GMT"
[14] "Expires: Fri, 30 Apr 1993 06:00:00 GMT"
[15] "Lines: 36"
[16] ""
[17] "In article <00969FBA.E640FF10@[Link]> mcdonal
[18] ">[...]"
[19] ">There are a variety of water-proof housings I could use
[20] ">of the problem is the electronics...hence this posting.
[21] ">transmission would be reliable underwater, in murky or
[22] ">conditions? I'm not sure if sound is feasible given th
[23] ">water...obviously direction would have to be accurate b
[24] ">relatively short (I imaging 2 or 3 hundred yards would)
[25] ">"

The source function called VectorSource() interprets text data contained in a vector, while ZipSource() reads a zipped file. Other source functions are provided
by the tm package as well. A popular and specific one is the DataframeSource() (with the tm package loaded, enter ?DataframeSource in the R console
to see details about this function).

Document-Term and Term-Document Matrices

In a spreadsheet model of text, columns are terms and rows are documents, or vice versa. Each cell shows the presence, absence or frequency of a particular term in a
particular document.

Let's consider the following two sentences:

S1. First, we consider the spreadsheet model

S2. Then we consider another model

The spreadsheet below is the document-term matrix showing the presence or absence of terms in each document (sentence):

first we consider the spreadsheet model then another

S1 1 1 1 1 1 1 0 0

S2 0 1 1 0 0 1 1 1

The document-term matrix represents each document as a row. This can be instrumental for comparison of authors within rows or to keep the order of the terms within the
documents (e.g. to preserve time series).

The term-document matrix is a transposition of the document-term matrix. In this case, the matrix has terms in the first column and documents across the top as individual
column names.
The term-document matrix is applicable for language analysis, where the terms are usually more significant than the documents. It is natural to deal with more rows than
columns in a spreadsheet.

The [Link]() function turns the term-document matrix into a simple matrix, enabling further analysis of the information.

Making a Term-Document Matrix

To make a Term-Document Matrix, R needs a corpus object. Remember that the sentences S1.“this is the first sentence,” S2.“this is a second sentence,” and S3.“the third
sentence is here” are stored into a VCorpus named crp (the one we created previously).

# simple Term Document Matrix example

tdm <- TermDocumentMatrix(crp)
inspect(tdm)

Output:

> tdm <- TermDocumentMatrix(crp)

> inspect(tdm)
<<TermDocumentMatrix (terms: 7, documents: 3)>>
Non-/sparse entries: 11/10
Sparsity : 48%
Maximal term length: 8
Weighting : term frequency (tf)
Sample :
Docs
Terms 1 2 3
first 1 0 0
here 0 0 1
second 0 1 0
sentence 1 1 1
the 1 0 1
third 0 0 1
this 1 1 0 ⟵
⟵ The term "this" does not occur in document 3
>

Preprocessing Text
For the examples that we’ve just looked at, making a matrix is a simple process. We had just words, spaces, and periods. Each word is preceded or followed by a delimiter — a
space or period. The real text is much more complex (as our corpus containing 2,000 newsgroup documents). There are lots of other things to process like digits, email
addresses, URLs, proper nouns, etc.

The goal of preprocessing is vocabulary reduction of text without losing meaning and/or predictive power (e.g., ignoring case, numbers, etc.).

Tokenization

The raw text needs to be transformed to useful predictor information. The first thing to do “is to separate out and identify individual terms. The process by which you identify
delimiters and use them to separate terms is called tokenization. The resulting terms are also called tokens” (Shmueli, Bruce, Yahav, Patel, & Lichtendahl, Jr., 2018).

Removing Stop Words

All the delimiters can be removed. Punctuation characters and additional white spaces can be also removed. Terms that are on a stop list (stop words) can be removed as well.
This eliminates very common terms without any predictive power reducing the size of the document. This is illustrated with the default “English” stop list that is included in the
R’s tm package:

> stopwords("english")
 [1] "i" "me" "my" "myself" "we" "our" "ours" "ourselves" "you" "your"
[11] "yours" "yourself" "yourselves" "me" "him" "his" "himself" "she" "her" "hers"
[21] "herself" "it" "its" "itself" "they" "them" "their" "theirs" "themselves" "what"
[31] "which" "who" "whom" "this" "that" "these" "those" "am" "is" "are"
[41] "was" "were" "be" "been" "being" "have" "has" "had" "having" "do"
[51] "does" "did" "doing" "would" "should" "could" "ought" "i'm" "you're" "he's"
[61] "she's" "it's" "we're" "they're" "i've" "you've" "we've" "they've" "i'd" "you'd"
[71] "he'd" "she'd" "we'd" "they'd" "i'll" "you'll" "he'll" "she'll" "we'll" "they'll"
[81] "isn't" "aren't" "wasn't" "weren't" "hasn't" "haven't" "hadn't" "doesn't" "don't" "didn't"
[91] "won't" "wouldn't" "shan't" "shouldn't" "can't" "cannot" "couldn't" "mustn't" "let's" "that's"
[101] "who's" "what's" "here's" "there's" "when's" "where's" "why's" "how's" "a" "an"
[111] "the" "and" "but" "if" "or" "because" "as" "until" "while" "of"
[121] "at" "by" "for" "with" "about" "against" "between" "into" "through" "during"
[131] "before" "after" "above" "below" "to" "from" "up" "down" "in" "out"
[141] "on" "off" "over" "under" "again" "further" "then" "once" "here" "there"
[151] "when" "where" "why" "how" "all" "any" "both" "each" "few" "more"
[161] "most" "other" "some" "such" "no" "nor" "not" "only" "own" "same"
[171] "so" "than" "too" "very"

Word Stemming

Stemming is a reduction of “multiple variants of a word to a common core” (Shmueli, Bruce, Yahav, Patel, & Lichtendahl, Jr., 2018). For example:

Stem, stemming, stemmed, etc. → stem

Normalization

“Normalization [is used] when the presence of a type of term might be important but we don’t need the specific term” (Shmueli, Bruce, Yahav, Patel, & Lichtendahl, Jr., 2018).
For example:

Replace john@[Link] with “email token”

Replace [Link] with “url token”

Preprocessing: Simple Example

# simple text preprocessing example

text <- c("this is the first sentence!!",
"this is a second Sentence :)",
"the third sentence, is here",
"fourth of all sentences")
crp <- VCorpus(VectorSource(text))
tdm <- TermDocumentMatrix(crp)

# tokenization
crp <- tm_map(crp, stripWhitespace)
crp <- tm_map(crp, removePunctuation)
tdm1 <- TermDocumentMatrix(crp)

# removing stop words

crp <- tm_map(crp, removeWords, stopwords("english"))

# stemming
crp <- tm_map(crp, stemDocument)
tdm2 <- TermDocumentMatrix(crp)

# to compare intermediate and final results

inspect(tdm)
inspect(tdm1)
inspect(tdm2)

Output:

> inspect(tdm)
<<TermDocumentMatrix (terms: 12, documents: 4)>>
Non-/sparse entries: 14/34
Sparsity : 71%
Maximal term length: 10
Weighting : term frequency (tf)
Sample :
Docs
Terms 1 2 3 4
all 0 0 0 1
first 1 0 0 0
fourth 0 0 0 1
here 0 0 1 0
second 0 1 0 0
sentence 0 1 0 0
sentence!! 1 0 0 0
sentence, 0 0 1 0
the 1 0 1 0
this 1 0 0 0
 > inspect(tdm1)
<<TermDocumentMatrix (terms: 10, documents: 4)>>
Non-/sparse entries: 14/26
Sparsity : 65%
Maximal term length: 9
Weighting : term frequency (tf)
Sample :
Docs
Terms 1 2 3 4
all 0 0 0 1
first 1 0 0 0
fourth 0 0 0 1
here 0 0 1 0
second 0 1 0 0
sentence 1 1 1 0
sentences 0 0 0 1
the 1 0 1 0
third 0 0 1 0
this 1 1 0 0
> inspect(tdm2)
<<TermDocumentMatrix (terms: 5, documents: 4)>>
Non-/sparse entries: 8/12
Sparsity : 60%
Maximal term length: 7
Weighting : term frequency (tf)
Sample :
Docs
Terms 1 2 3 4
first 1 0 0 0
fourth 0 0 0 1
second 0 1 0 0
sentence 1 1 1 1
third 0 0 1 0

The tm_map() function provided in the tm package applies cleaning functions to a corpus. To scale the cleaning steps, these functions are being mapped to an entire
corpus with the help of tm_map().

Changing Frequency Weights

So far, we have made Term-Document Matrices using the term frequency to weight terms. There are more efficient term weights, such as TF-IDF (term frequency-inverse
document frequency).

The TF-IDF score takes into account the frequency of term appearance in a single document but also considers how often the term appears among all documents.

The terms that appear most frequently are most likely to be important. The term frequency part of TF-IDF represents this. TF is normalized by the length of the document. A
term which appears in all documents is not likely to be important or to bring insights. The inverse document frequency (IDF) part reflects this in the TF-IDF weight.

The mathematical explanation behind the score can be found in your textbook (Chapter 20), but the bottom line is:

TF-IDF is high where a rare term is present or frequent in a document

TF-IDF is near zero where a term is absent from a document, or abundant across all documents

TF-IDF, Simple Example

# simple TF-IDF example

tfidf <- weightTfIdf(tdm2)
inspect(tdm2)
inspect(tfidf)

Output:

> inspect(tdm2)
<<TermDocumentMatrix (terms: 5, documents: 4)>>
Non-/sparse entries: 8/12
Sparsity : 60%
Maximal term length: 7
Weighting : term frequency (tf)
Sample :
Docs
Terms 1 2 3 4
first 1 0 0 0
forth 0 0 0 1
second 0 1 0 0
sentenc 1 1 1 1 ⟵ ⟵
third 0 0 1 0
> inspect(tfidf)
<<TermDocumentMatrix (terms: 5, documents: 4)>>
Non-/sparse entries: 4/16
Sparsity : 80%
 Maximal term length: 7
Weighting : term frequency - inverse document frequency (normalized) (tf-idf)
Sample :
Docs
Terms 1 2 3 4
first 1 0 0 0
forth 0 0 0 1
second 0 1 0 0
sentenc 0 0 0 0 ⟵ ⟵ The frequency weight of the term "sentenc" which is present
in all documents is changed to zero.
third 0 0 1 0
>

Apply Preprocessing Steps to a Corpus

When you are performing an exploratory analysis, it is likely to apply the same preprocessing steps on multiple text datasets (i.e. on different corpora). A good practice in such a
case is to describe and save the preprocessing steps into a predefined reusable custom function. See the example below. The clean_corpus() function applies a
series of preprocessing steps (executed by other functions in order) to its argument “corpus”, and then returns an end result of them. That way you will save time and your code
will be cleaner and easier to maintain.

Let’s apply preprocessing steps to our large corpus (“corp”) using the shown custom function, and then compute the term frequency and create a term-document matrix:

# text preprocessing custom function definition

clean_corpus <- function(corpus) {

# tokenization
corpus <- tm_map(corpus, stripWhitespace)
corpus <- tm_map(corpus, removePunctuation)
corpus <- tm_map(corpus, removeNumbers)

# removing stop words

corpus <- tm_map(corpus, removeWords, stopwords("english"))

# stemming
corp <- tm_map(corp, stemDocument)
return(corpus)

# apply preprocessing steps to the "corp" corpus

corp <- clean_corpus(corp)

# compute term frequency anc create term document matrix

tdm <- TermDocumentMatrix(corp)

Text Data Visualization

A sound business reason to create text mining visuals is the fact that they help decision makers come to quick conclusions.

A popular text data visual is the word cloud. In a word cloud, the size of the characters used to display a word shows the frequency of that particular term's occurrence. The
bigger the terms are, the higher their occurrence in the text data is. Different colors could be used to show another metric. In the example below we use a single color and word
size illustrates the individual term's frequency.

# visualize word cloud on tdm

mtx <- [Link](tdm)
term_frequency <- rowSums(mtx)
term_frequency <- sort(term_frequency, decreasing = TRUE)
word_freqs <- [Link](term = names(term_frequency), num = term_frequency)
library(wordcloud)
wordcloud(word_freqs$term, word_freqs$num, [Link] = 50, colors = "red")

Output:
Terms like newsgroups, organization and subject are not very informative, are they?

Now let’s compute TF-IDF to change the frequency weights, and then visualize the word cloud again on the resulting post-reduction Term Document Matrix.

compute TF-IDF to change frequency weights

tfidf <- weightTfIdf(tdm)

# visualize word cloud on tfidf

mtx <- [Link](tfidf)
term_frequency <- rowSums(mtx)
term_frequency <- sort(term_frequency, decreasing = TRUE)
word_freqs <- [Link](term = names(term_frequency), num = term_frequency)
wordcloud(word_freqs$term, word_freqs$num, [Link] = 50, colors = "green")

Output:

This is much more informative.

From Terms to Concepts — Latent Semantic Indexing

For large text datasets, the post-reduction term-document matrix is still too big and heavy for processing. Recall how, with principal components, we derived a small set of
synthetic predictor variables, each of which was a linear combination of “like-minded” original variables. Latent semantic indexing, LSI (or latent semantic analysis, LSA), does
something similar for text data – it maps multiple terms to a small set of concepts.

"For example: if we inspected our document collection, we might find that each time the term
“alternator” appeared in an automobile document, the document also included the terms “battery” and
“headlights”. Or each time the term “brake” appeared in an automobile document, the terms “pads”
and “squeaky” also appeared. However, there is no detectable pattern regarding the use of the terms
“alternator” and “brake” together. Documents including “alternator” might or might not include “brake”
and documents including “brake” might or might not include “alternator.” Our four terms, battery,
headlights, pads, and squeaky describe two different automobile repair issues: failing brakes and a
 bad alternator."

Source: Analytic Solver Platform, XLMiner Platform. (2014). Data Mining User Guide. Frontline Systems, p. 245.

It is important to note that concepts could be used to identify subjects in documents and clusters of documents with a common subject, but this approach is not universal.
Sometimes it’s not possible to map concepts to clear subjects. Concepts are most effective for reduction of vocabulary and for reduction of term-document matrices from
thousands of columns to a dozen or two.

Let’s apply LSA on our post-reduction term-documents Matrix to extract 20 concepts:

#apply LSA to extract (20) concepts

library(lsa)
[Link] <- lsa(tdidf, dim = 20)

Run a Standard Predictive Model on a Concept Matrix

Now that we have a clean, structured dataset, our task is reduced to what we have previously done in the numerical data mining section:

Identification of class labels for training, or

Identification of numerical predictors

Example: Classify posts as auto-related or electronics-related. Here’s a sample post:

From: smith@[Link] (Tom Smith) Subject: Ford Explorer 4WD - do I need

performance axle?
We’re considering getting a Ford Explorer XLT with 4WD and we have the following
questions (All we would do is go skiing - no off-roading):
1. With 4WD, do we need the "performance axle" - (limited slip axle)? Its purpose
is to allow the tires to act independently when the tires are on different
terrain.
2. Do we need the all-terrain tires (P235/75X15) or will the all-season
(P225/70X15) be good enough for us at Lake Tahoe?
Thanks,
Tom
-
================================================
Tom Smith Silicon Graphics smith@[Link] 2011 N. Shoreline Rd. MS
8U-815 415-962-0494 (fax) Mountain View, CA 94043
================================================

Now let’s run a standard predictive model on the Concept Matrix:

# create an array of records labels

label <- c(rep(1, 1000), rep(0, 1000))

# 1000 1's which will be used to label the first 1000 documents,
# which we know are auto, and 1000 0's to label the remainder,
# which are electronic

# sample 60% training data

training <- sample(c(1:2000), 0.6*2000)

# convert to data frame

[Link] <- [Link]([Link]([Link]$dk))

# run logistic model on training

trainData = cbind(label = label[training], [Link][training,])
reg <- glm(label ~ ., data = trainData, family = "binomial")

# compute accuracy on validation set

validData = cbind(label = label[-training], [Link][-training,])
pred <- predict(reg, newdata = validData, type = "response")

# produce confusion matrix

library(caret)
confusionMatrix([Link](ifelse(pred>0.5, 1, 0)), [Link](label[-training]))

# if the result is an Error: "package e1071 is required",

# then "[Link]("e1071")" shall be executed
# before the execution of the confusionMatrix() function

To evaluate the model performance let’s look at the confusion matrix:

 > confusionMatrix([Link](ifelse(pred>0.5, 1, 0)), [Link](label[-training]))
Confusion Matrix and Statistics

Reference
Prediction 0 1
0 399 10
1 16 375

Accuracy : 0.9675 ⟵⟵ The high accuracy shows that the posts are very separable
95% CI : (0.9527, 0.9787)
No Information Rate : 0.5188
P-Value [Acc > NIR] : <2e-16

Kappa : 0.9349
Mcnemar's Test P-Value : 0.3268

Sensitivity : 0.9614
Specificity : 0.9740
Pos Pred Value : 0.9756
Neg Pred Value : 0.9591
Prevalence : 0.5188
Detection Rate : 0.4988
Detection Prevalence : 0.5112
Balanced Accuracy : 0.9677

'Positive' Class : 0

Sentiment Analysis — Basic Concepts

What Is Sentiment Analysis?

Sentiment analysis is the process of extracting an author’s emotional intent from text.

Application and Benefits

Sentiment analysis has various applications and can bring many benefits to the enterprise. The most general business application of sentiment analysis is to get insights into
how customers (or potential customers) feel about a product or service.

Sentiment analysis is most often applied to social media data. It is used to identify potential opportunities for or threats to a business by uncovering tendencies in sentiment
concerning the company’s portfolio, enabling decision makers to take proactive measures.

"Sentiment analysis could also be applied to your corporate network, for example, by applying it to
your email server, emails could be monitored for their general “tone.” For example, Tone Detector is
an Outlook add-in that determines the “tone” of your email as you type. Like an emotional spell
checker for all of your outgoing email."

-Mikaela Berman in Sentiment Analysis: Overview, Applications and Benefits

Basic Sentiment Analysis in R

The qdap package provides a so-called polarity function:

> library(qdap)
> polarity(text_df$text_column)
> polarity(text_df$column, text_df$factor_or_author_grouping)

The polarity() function returns a “polarity” object with positive and negative scores.

> library(qdap)
> library(magrittr)

> text_df %$% polarity(text)

# The magrittr package's dollar pipe operator %$%
 # forwards the data frame into polarity()
# and you declare a text column name and
# a grouping variable (opt.) without quotes.
# This is the so called "syntax sugar" ☺

A subjectivity lexicon is a predefined list of words paired with their emotional context such as positive/negative or specific emotions like “frustration” or “joy.”

An example subjectivity lexicon:

Word Polarity

Amazing Positive

Bad Negative

Good Positive

… …

Wonderful Positive

The package's qdap’s polarity() function uses a lexicon from hash_sentiment_huliu.

Preprocessing: Simple Example

# load "hp_tweets.txt" which contains 1,000 tweets about Harry Potter

hp_tweets <- [Link]("~/hp_tweets.txt", quote = "", header = FALSE, stringAsFactors = FALSE)

# take a look at the structure of the resulting R object and first few tweets
head(hp_tweets)

Output:

> head(ph_tweets)
V1
1 Is it just me, or does Harry Potter suck?...
2 This quiz sucks and Harry Potter sucks ok bye..
3 I would like a Harry Potter type scarf for Christmas..
4 Is it just me, or does Harry Potter suck?...
5 I love Harry Potter.
6 I am going to start reading the Harry Potter series again because that is one awesome story.

# load qdap and magrittr packages

library(qdap)
library(magrittr)

# calculate polarity scores for the tweets

hp_polarity <- hp_tweets %$% polarity(v1)

# print the resulting polarity object

hp_polarity

Output:

> hp_polarity
all [Link] [Link] [Link] [Link] [Link]
1 all 1000 14270 0.138 0.526 0.262

# plot the tweets' polarity

 plot(hp_polarity)

Output:

# extract the polarity scores

hp_scored_tweets <- counts(hp_polarity)

# view the head of the resulted data frame

head(hp_scored_tweets)

Output:

> head(hp_scored_tweets)
all wc polarity [Link] [Link] [Link]
1 all 9 -0.333 - suck Is it just me, or does Harry Potter suck?...
2 all 9 -0.333 - sucks, sucks This quiz sucks and Harry Potter sucks ok bye..
3 all 9 -0.333 like - I would like a Harry Potter type scarf for Christmas..
4 all 9 -0.333 - suck Is it just me, or does Harry Potter suck?...
5 all 9 -0.333 love - I love Harry Potter.
6 all 9 -0.333 awesome - I am going to start reading the Harry Potter series again because that is one awesome story.

The above data frame is ready for further analysis and exploration. For further enrichment, students can also check the text mining individual exercises.

References
Shmueli, G., Bruce, P. C., Yahav, I., Patel, R. N., & Lichtendahl, Jr., K. C. (2018). Data Mining for Business Analytics: Concepts, Techniques, and Applications in R. Hoboken,
NJ: Wiley.

Module 5 Practice Questions

The following are some review questions for you to practice. Please read each question, think
carefully, figure out your own answer first, and then click "Show Answer" to compare your answer to
the suggested answer.

Test Yourself 5.01

Which of the following statements would be true for a text mining approach that uses the "bag-of-
words" approach to analyze the sentence below (in italics)?

"Bill Gates considered the historic importance of Windows."

a. The sentence "Bill Gates considered the historic importance of Windows" would be treated the
exact same way as "Gates historic importance the considered Bill Windows of."

This is true.

b. Because three words in the sentence start with capital letters, the model would identify those
words as proper nouns.

This is False.

c. The sentence could not be handled by such a model, as the model would require a baseline of at
least 10,000 data points before it could process new sentences.

This is false.

Suggested Answer: Answer a is correct. With a "bag of words" approach, the text data is simply
analyzed word-for-word. Context, grammar, and syntax are all ignored. While such an approach
ignores a lot of the nuance that comes with language, it is computationally efficient and can be
surprisingly effective.

Test Yourself 5.02

Why might someone include a stop word list during the preprocessing stage of text mining?

a. A stopword list could be built in a way that instructs the model to stop when it encounters a
specific word – that way, it will be flagged for analysis later on.

This is False.

b. A stopword list could be used to automatically eliminate words that are not likely to be helpful or
useful for the model.

This is true.

c. A stopword list can eliminate word endings, so that a verb like "run" can be treated the same way
as "runner" or "running."

This is false.

Suggested Answer: Answer b is correct. Common stopwords in English include words such as "did",
"have", "me", and "here." Someone building a model can add or remove terms from the generic list.
Removing stopwords from the analysis of text reduces the number of terms that the model needs to
analyze, and shifts the focus towards words that are more likely to be of value/interest.