GBUS515 –Business Intelligence and Information Systems
Chapter 20 - Text Mining
Instructor – Dr. Sunita Goel
Adapted from Shmueli, Bruce & Patel, Data Mining for Business Analytics, 3e
© Galit Shmueli and Peter Bruce 2010
Text data
Up to now we have been dealing with structured
quantitative data
Numerical
Binary (yes/no)
Multicategory
Now we turn to unstructured text
Applications of Text Mining
Insurance fraud – notes in claim forms can be
mined and transformed into predictor variables for a
predictive model
The model is trained on prior claims in two classes –
found to be fraudulent, and not found to be
fraudulent
The model is then applied to new claims
Applications, cont.
Maintenance or support
tickets often contain text
fields
These fields could be mined
to classify ticket in several
ways:
How urgent?
How much time to fix?
What category of technician
is needed to fix?
Applications, cont.
Medical triage/diagnosis
Clinics could use patient online appointment request
forms to route requests
Admin asst.
Nurse
Doctor
What exactly is text mining?
Classify (label) thousands of documents
Extension of predictive modeling (our focus)
Extract meaning from a single document – interpreting
it like a human reads language
“Natural language processing” (very ambitious, not predictive modeling, not
our focus)
Classification (labeling) and
clustering
No attempt to extract overall document meaning
from a single document
Focus is on assigning a label or class to numerous
documents
As with numerical data mining, the goal is to do
better than guessing
“Bag-of-words”
Grammar, syntax, punctuation, word order are ignored
The document is considered as a “bag of words”
This approach is, nonetheless, effective when the goal is to
decide which category or cluster a document falls in
A typical application is supervised learning
Requires lots of documents (a corpus)*
Do not need 100% accuracy
*“Corpus” often refers to a fixed standard set of documents that many researchers
can use to develop and tune text mining algorithms.
The spreadsheet model of text
Columns are terms
Rows are documents
Cells indicate presence/absence (or frequency) of
terms in documents
Consider the two sentences:
S1 First we consider the spreadsheet model
S2 Then we consider another model
Here is the resulting spreadsheet, using
presence/absence:
firs t we cons ider the s preads heet model then another
S1 1 1 1 1 1 1 0 0
S2 0 1 1 0 0 1 1 0
Need to turn text into a matrix
For the two documents (sentences S1 and S2) that
we looked at earlier, the process of producing a
matrix is simple. We had
Words
Spaces
Periods
Each word is preceded or followed by a space or
period – a delimiter.
Real text is more complicated
Lots of things to process besides
words…
Numbers (including dates, percents, monetary
amounts), e.g. from Google Annual Report 2014:
We considered the historical trends in currency exchange rates and determined that it was reasonably possible that
changes in exchange rates of 20% could be experienced in the near term. If the U.S. dollar weakened by 20% at
December 31, 2013 and 2014, the amount recorded in AOCI related to our foreign exchange options before tax
effect would have been approximately $4 million and $686 million lower at December 31, 2013 and December
31, 2014, and the total amount of expense recorded as interest and other income, net, would have been
approximately $123 million and $90 million higher in the years ended December 31, 2013 and December 31,
2014. If the U.S. dollar strengthened by 20% at December 31, 20013 and December 31, 2014, the amount
recorded in accumulated AOCI related to our foreign exchange options before tax effect would have been
approximately $1.7 billion and $2.5 billion higher at December 31, 2013 and December 31, 2014, and the total
amount of expense recorded as interest and other income, net, would have been approximately $120 million and
$164 million higher in the years ended December 31, 2013 and December 31, 2014.
Email addresses, url’s, stray characters
introduced by file conversions, …
Sender: Distribution list for statistical items of interest <WSS-ELECTRONIC-MAIL-LIST@[Link]>
From: "Massimini, Vince" <svm@[Link]> Subject: Comparing the Maximal Procedure to Permuted
Blocks Randomization To: <WSS-ELECTRONIC-MAIL-LIST@[Link]> Precedence: list List-Help:
<[Link]
For more WSS events, see [Link] WSS Public Health/Biostatistics Section and NCI Division of
Cancer Preventi= on Jointly Sponsored Event: =20
SPEAKER: Vance W. Berger, PhD National Cancer Institute and University of = Maryland Baltimore County
and Klejda Bejleri, BS Biometry and Statistics, D= epartment of Biological Statistics and Computational
Biology, Cornell Unive= rsity, Ithaca, NY 14853 =20 TITLE: Comparing the Maximal Procedure to
Permuted Blocks Randomization
TIME AND PLACE: Monday, June 8th NCI Shady Grove, 9609 Medical Center Drive= , Rockville MD Room
5E30/32. Bring photo ID, allow time to get through secu= rity
Proper nouns & terms specific to a
particular field
From Techsmith corporate information: All-In-One Capture, Camtasia, Camtasia Studio,
Camtasia Relay, Coach's Eye, DubIt, EnSharpen, Enterprise Wide, Expressshow, Jing,
Morae, Rich Recording Technology (RRT), Snagit, [Link], ScreenChomp, Show
The World, SmartFocus, TechSmith, TechSmith and T Design logo, TechSmith Fuse,
TechSmith Relay, TSCC, and UserVue are marks or registered marks of TechSmith
Corporation.
From medical journal: Eight hundred elderly women and men from the population-based
Framingham Osteoporosis Study had BMD assessed in 1988-1989 and again in 1992-
1993. BMD was measured at femoral neck, trochanter, Ward's area, radial shaft,
ultradistal radius, and lumbar spine using Lunar densitometers. (Risk Factors for Longitudinal Bone
Loss in Elderly Men and Women: The Framingham Osteoporosis Study, Journal of Bone and Mineral Research Volume
15, Issue 4, pages 710–720, April 2000)
Tokenization
We need to move from a mass of text to useful
predictor information
The first step is to separate out and identify
individual terms
The process by which you identify delimiters and use
them to separate terms is called tokenization. The
resulting terms are also called tokens.
Preprocessing
Goal – reduction of text (also called vocabulary
reduction) without losing meaning or predictive
power
Stemming
Reducing multiple variants of a word to a common
core
Travel, traveling, traveled, etc. -> travel
Ignore case
Frequency filters can eliminate terms that
Appear in nearly all documents
Appear in hardly any documents
Preprocessing, cont.
All punctuation characters treated as delimiters
Punctuation characters are sometimes also considered
terms of length 1, but not in XLMiner
Default in XLMiner is to drop all terms <=2 characters
long
Default is also to drop all terms that are on a stoplist
XLMiner comes with a default stoplist
User can edit it
Preprocessing, cont.
Frequency vs. presence/absence
Normalization, when the presence of a type of term
might be important but we don’t need the specific
term. For example
Replace john@[Link] with “email token”
Replace [Link] with “url token”
Post-reduction matrix
Rows are still records, columns are the reduced set of terms
Options for cell entries:
0/1 (presence absence)
Frequency count
TF-IDF (term frequency – inverse document frequency)
TF = frequency of term
IDF = log of inverse of the frequency with which documents have
that term
There are varying definitions of both TF and IDF, hence of TF-IDF
Bottom line:
TF-IDF is high where a rare term is present or frequent in a document
TF-IDF is near zero where a term is absent from a document, or abundant
across all documents
From terms to concepts – Latent
Semantic Indexing (LSI)
The post-reduction term/document matrix is often still
huge – too big for easy processing
Recall how, with principal components, we derived a
small set of synthetic predictor variables, each of which
was a linear combination of “like-minded” original
variables.
Latent semantic indexing does something similar for
text – it maps multiple terms to a small set of concepts.
Intuitive explanation - LSI
For example: if we inspected our document collection, we might find that each
time the term “alternator” appeared in an automobile document, the
document also included the terms “battery” and “headlights.” Or each time
the term “brake" appeared in an automobile document, the terms “pads”
and “squeaky” also appeared. However, there is no detectable pattern
regarding the use of the terms “alternator” and “brake” together.
Documents including “alternator” might or might not include “brake” and
documents including “brake” might or might not include “alternator.” Our
four terms, battery, headlights, pads, and squeaky describe two different
automobile repair issues: failing brakes and a bad alternator.
Analytic Solver Platform, XLMiner Platform, Data Mining User Guide, 2014,
Frontline Systems, p. 245}
Extracting Meaning?
It may be possible to use the concepts to identify
themes in the document corpus, and clusters of
documents sharing those themes.
Often, however, the concepts do not map in obvious
fashion to meaningful themes.
Their key contribution is simply reducing the
vocabulary – instead of a matrix with thousands of
columns, we can deal with just a dozen or two.
The concept document matrix
A small portion of the document-concept matrix. The concepts may or
may not carry meaning that we can figure out; in any case they are
mathematical constructs from hundreds, perhaps thousands, of terms,
and can be used as the variables in a predictive model.
A predictive model
Now we have a clean, structured dataset similar to
what we have used in our numerical data mining:
Class identifications (labels) for training
Numerical predictors
Chapter Exercises
(Updated in Canvas)