Data Mining:

Concepts and Techniques

Mining Text Data
 Mining Text and Web Data

 Text mining, natural language processing and

information extraction: An Introduction
 Text categorization methods
 Mining Text Data: An Introduction
Data Mining / Knowledge Discovery

Structured Data Multimedia Free Text Hypertext

HomeLoan ( Frank Rizzo bought <a href>Frank Rizzo
Loanee: Frank Rizzo his home from Lake </a> Bought
Lender: MWF View Real Estate in <a hef>this home</a>
Agency: Lake View 1992. from <a href>Lake
Amount: $200,000 He paid $200,000 View Real Estate</a>
Term: 15 years under a15-year loan In <b>1992</b>.
) Loans($200K,[map],...) from MW Financial. <p>...
 Bag-of-Tokens Approaches
Documents Token Sets

Four score and seven nation – 5

years ago our fathers brought civil - 1
forth on this continent, a new war – 2
nation, conceived in Liberty, Feature men – 2
and dedicated to the Extraction died – 4
proposition that all men are people – 5
created equal. Liberty – 1
Now we are engaged in a God – 1
great civil war, testing …
whether that nation, or …

Loses all order-specific information!

Severely limits context!
 Natural Language Processing
A dog is chasing a boy on the playground Lexical
Det Noun Aux Verb Det Noun Prep Det Noun analysis
(part-of-speech
Noun Phrase tagging)
Noun Phrase Complex Verb Noun Phrase

Prep Phrase
Semantic analysis Verb Phrase
Syntactic analysis
Dog(d1). (Parsing)
Boy(b1).
Playground(p1). Verb Phrase
Chasing(d1,b1,p1).
Sentence
+
Scared(x) if Chasing(_,x,_).
A person saying this may
be reminding another person to
get the dog back…
Scared(b1)
Inference Pragmatic analysis
(speech act)

(Taken from ChengXiang Zhai, CS 397cxz – Fall 2003)

 General NLP—Too Difficult!
 Word-level ambiguity
 “design” can be a noun or a verb (Ambiguous POS)

 “root” has multiple meanings (Ambiguous sense)

 Syntactic ambiguity
 “natural language processing” (Modification)

 “A man saw a boy with a telescope.” (PP Attachment)

 Anaphora resolution
 “John persuaded Bill to buy a TV for himself.”

(himself = John or Bill?)

 Presupposition
 “He has quit smoking.” implies that he smoked before.

Humans rely on context to interpret (when possible).

This context may extend beyond a given document!
(Taken from ChengXiang Zhai, CS 397cxz – Fall 2003)
 Shallow Linguistics

Progress on Useful Sub-Goals:

• English Lexicon
• Part-of-Speech Tagging
• Word Sense Disambiguation
• Phrase Detection / Parsing
 WordNet
An extensive lexical network for the English language
• Contains over 138,838 words.
• Several graphs, one for each part-of-speech.
• Synsets (synonym sets), each defining a semantic sense.
• Relationship information (antonym, hyponym, meronym …)
• Downloadable for free (UNIX, Windows)
• Expanding to other languages (Global WordNet Association)
• Funded >$3 million, mainly government (translation interest)
• Founder George Miller, National Medal of Science, 1991.
watery parched

moist wet dry arid

synonym

antonym
damp anhydrous
 Part-of-Speech Tagging
Training data (Annotated text)
This sentence serves as an example of annotated text…
Det N V1 P Det N P V2 N

This is a new sentence.

“This is a new sentence.” POS Tagger
Det Aux Det Adj N

Pick the most

p( w1 likely
,..., wk , ttag sequence.
1 ,..., tk )

 p(t1 | w1 )... p(tk | wk ) p( w1 )... p( wk )


p( w1 ,..., wk , t1 ,..., tk )   k
 p( wi | ti ) p(ti | ti 1 )
Independent assignment
 p(t1 | w1 )... p(tk | wk ) p(iw11 )... p( wk ) Most common tag

 k
 p( wi | ti ) p(ti | ti 1 )
 i 1 Partial dependency
(HMM)
 Word Sense Disambiguation
?
“The difficulties of computational linguistics are rooted in ambiguity.”
N Aux V P N
Supervised Learning
Features:
• Neighboring POS tags (N Aux V P N)
• Neighboring words (linguistics are rooted in ambiguity)
• Stemmed form (root)
• Dictionary/Thesaurus entries of neighboring words
• High co-occurrence words (plant, tree, origin,…)
• Other senses of word within discourse
Algorithms:
• Rule-based Learning (e.g. IG guided)
• Statistical Learning (i.e. Naïve Bayes)
• Unsupervised Learning (i.e. Nearest Neighbor)
 Parsing
Choose most likely parse tree… S Probability of this tree=0.000015

Probabilistic CFG NP VP

S NP VP 1.0 Det BNP VP PP

NP  Det BNP 0.3
NP  BNP 0.4 A N Aux V NP P NP
NP NP PP 0.3
Grammar BNP N … dog is chasing on
VP  V a boy
VP  Aux V NP .. the playground
VP  VP PP
… . S Probability of this tree=0.000011
PP  P NP 1.0
NP VP
V  chasing 0.01 Det NP
Aux is BNP Aux V
N  dog PP
0.003 A
N  boy N is chasing NP
Lexicon P NP
N playground …
Det the dog a boy on
…
Det a
P  on the playground
 Mining Text and Web Data

 Text mining, natural language processing and

information extraction: An Introduction
 Text information system and information
retrieval
 Text categorization methods
 Mining Web linkage structures
 Summary
 Text Databases and IR

 Text databases (document databases)

 Large collections of documents from various sources:
news articles, research papers, books, digital libraries,
e-mail messages, and Web pages, library database, etc.
 Data stored is usually semi-structured

 Traditional information retrieval techniques become

inadequate for the increasingly vast amounts of text
data
 Information retrieval
 A field developed in parallel with database systems

 Information is organized into (a large number of)

documents
 Information retrieval problem: locating relevant
documents based on user input, such as keywords or
example documents
 Information Retrieval
 Typical IR systems
 Online library catalogs
 Online document management systems
 Information retrieval vs. database systems
 Some DB problems are not present in IR, e.g., update,
transaction management, complex objects
 Some IR problems are not addressed well in DBMS,
e.g., unstructured documents, approximate search
using keywords and relevance
 Basic Measures for Text Retrieval

Relevant Relevant &

Retrieved Retrieved

All Documents

 Precision: the percentage of retrieved documents that are in fact relevant

to the query (i.e., “correct” responses)
| {Relevant}  {Retrieved } |
precision 
| {Retrieved } |
 Recall: the percentage of documents that are relevant to the query and
were, in fact, retrieved
| {Relevant}  {Retrieved} |
Recall 
| {Relevant} |
 Information Retrieval Techniques

 Basic Concepts
 A document can be described by a set of

representative keywords called index terms.

 Different index terms have varying relevance when

used to describe document contents.

 This effect is captured through the assignment of

numerical weights to each index term of a document.

(e.g.: frequency, tf-idf)
 DBMS Analogy
 Index Terms  Attributes

 Weights  Attribute Values

 Information Retrieval Techniques
 Index Terms (Attribute) Selection:
 Stop list

 Word stem

 Index terms weighting methods

 Terms  Documents Frequency Matrices

 Information Retrieval Models:
 Boolean Model

 Vector Model

 Probabilistic Model
 Boolean Model

 Consider that index terms are either present or

absent in a document
 As a result, the index term weights are assumed to
be all binaries
 A query is composed of index terms linked by three
connectives: not, and, and or
 e.g.: car and repair, plane or airplane
 The Boolean model predicts that each document is
either relevant or non-relevant based on the match of
a document to the query
 Keyword-Based Retrieval

 A document is represented by a string, which can be

identified by a set of keywords
 Queries may use expressions of keywords
 E.g., car and repair shop, tea or coffee, DBMS but not
Oracle
 Queries and retrieval should consider synonyms, e.g.,
repair and maintenance
 Major difficulties of the model
 Synonymy: A keyword T does not appear anywhere in
the document, even though the document is closely
related to T, e.g., data mining
 Polysemy: The same keyword may mean different
things in different contexts, e.g., mining
Similarity-Based Retrieval in Text Data

 Finds similar documents based on a set of common

keywords
 Answer should be based on the degree of relevance based
on the nearness of the keywords, relative frequency of the
keywords, etc.
 Basic techniques
 Stop list
 Set of words that are deemed “irrelevant”, even

though they may appear frequently

 E.g., a, the, of, for, to, with, etc.

 Stop lists may vary when document set varies

Similarity-Based Retrieval in Text Data

 Word stem
 Several words are small syntactic variants of each

other since they share a common word stem

 E.g., drug, drugs, drugged

 A term frequency table

 Each entry frequent_table(i, j) = # of occurrences

of the word ti in document di

 Usually, the ratio instead of the absolute number of

occurrences is used
 Similarity metrics: measure the closeness of a document
to a query (a set of keywords)
 Relative term occurrences v1  v2
sim(v1 , v2 ) 
 Cosine distance: | v1 || v2 |
 Indexing Techniques

 Inverted index
 Maintains two hash- or B+-tree indexed tables:

 document_table: a set of document records <doc_id,

postings_list>
 term_table: a set of term records, <term, postings_list>

 Answer query: Find all docs associated with one or a set of terms

 + easy to implement

 – do not handle well synonymy and polysemy, and posting lists could

be too long (storage could be very large)

 Signature file
 Associate a signature with each document

 A signature is a representation of an ordered list of terms that

describe the document

 Order is obtained by frequency analysis, stemming and stop lists
 Types of Text Data Mining
 Keyword-based association analysis
 Automatic document classification
 Similarity detection
 Cluster documents by a common author

 Cluster documents containing information from a

common source
 Link analysis: unusual correlation between entities
 Sequence analysis: predicting a recurring event
 Anomaly detection: find information that violates usual
patterns
 Hypertext analysis
 Patterns in anchors/links

 Anchor text correlations with linked objects

Keyword-Based Association Analysis
 Motivation
 Collect sets of keywords or terms that occur frequently together and
then find the association or correlation relationships among them
 Association Analysis Process
 Preprocess the text data by parsing, stemming, removing stop
words, etc.
 Evoke association mining algorithms
 Consider each document as a transaction
 View a set of keywords in the document as a set of items in the
transaction
 Term level association mining
 No need for human effort in tagging documents
 The number of meaningless results and the execution time is greatly
reduced
 Text Classification

 Motivation
 Automatic classification for the large number of on-line text

documents (Web pages, e-mails, corporate intranets, etc.)

 Classification Process
 Data preprocessing

 Definition of training set and test sets

 Creation of the classification model using the selected

classification algorithm
 Classification model validation

 Classification of new/unknown text documents

 Text document classification differs from the classification of

relational data
 Document databases are not structured according to attribute-

value pairs
 Text Classification(2)

 Classification Algorithms:
 Support Vector Machines

 K-Nearest Neighbors

 Naïve Bayes

 Neural Networks

 Decision Trees

 Association rule-based

 Boosting
 Document Clustering

 Motivation
 Automatically group related documents based on their
contents
 No predetermined training sets or taxonomies

 Generate a taxonomy at runtime

 Clustering Process
 Data preprocessing: remove stop words, stem, feature
extraction, lexical analysis, etc.
 Hierarchical clustering: compute similarities applying
clustering algorithms.
 Model-Based clustering (Neural Network Approach):
clusters are represented by “exemplars”. (e.g.: SOM)
 Text Categorization

 Pre-given categories and labeled document

examples (Categories may form hierarchy)
 Classify new documents
 A standard classification (supervised learning )
problem
Sports
Categorization
Business
System
Education
… …
Sports
Science
Business

Education
 Applications

 News article classification

 Automatic email filtering
 Webpage classification
 Word sense disambiguation
 ……
 Categorization Methods
 Manual: Typically rule-based
 Does not scale up (labor-intensive, rule inconsistency)

 May be appropriate for special data on a particular

domain
 Automatic: Typically exploiting machine learning techniques
 Vector space model based

 Prototype-based (Rocchio)
 K-nearest neighbor (KNN)
 Decision-tree (learn rules)
 Neural Networks (learn non-linear classifier)
 Support Vector Machines (SVM)
 Probabilistic or generative model based
 Naïve Bayes classifier
 Vector Space Model

 Represent a doc by a term vector

 Term: basic concept, e.g., word or phrase
 Each term defines one dimension
 N terms define a N-dimensional space
 Element of vector corresponds to term weight
 E.g., d = (x1,…,xN), xi is “importance” of term i
 New document is assigned to the most likely category
based on vector similarity.
 VS Model: Illustration

Starbucks
C2 Category 2

Category 3
C3

Java
new doc

Microsoft C1 Category 1
 What VS Model Does Not Specify

 How to select terms to capture “basic concepts”

 Word stopping

 e.g. “a”, “the”, “always”, “along”

 Word stemming

 e.g. “computer”, “computing”, “computerize” =>

“compute”
 Latent semantic indexing

 How to assign weights

 Not all words are equally important: Some are more
indicative than others
 e.g. “algebra” vs. “science”

 How to measure the similarity

 How to Assign Weights

 Two-fold heuristics based on frequency

 TF (Term frequency)

 More frequent within a document  more relevant

to semantics
 e.g., “query” vs. “commercial”

 IDF (Inverse document frequency)

 Less frequent among documents  more
discriminative
 e.g. “algebra” vs. “science”
 TF Weighting

 Weighting:
 More frequent => more relevant to topic
 e.g. “query” vs. “commercial”
 Raw TF= f(t,d): how many times term t appears in
doc d
 Normalization:
 Document length varies => relative frequency preferred
 e.g., Maximum frequency normalization
 How to Measure Similarity?

 Given two document

 Similarity definition
 dot product

 normalized dot product (or cosine)

 Illustrative Example

text
doc1 mining Sim(newdoc,doc1)=4.8*2.4+4.5*4.5
search
engine
text Sim(newdoc,doc2)=2.4*2.4
To whom is newdoc
more similar?
travel
text
Sim(newdoc,doc3)=0
map
doc2 travel

text mining travel map search engine govern president congress

IDF(faked) 2.4 4.5 2.8 3.3 2.1 5.4 2.2 3.2 4.3

government doc1 2(4.8) 1(4.5) 1(2.1) 1(5.4)

president doc2 1(2.4 ) 2 (5.6) 1(3.3)
congress
doc3 doc3 1 (2.2) 1(3.2) 1(4.3)

newdoc 1(2.4) 1(4.5)

……