SUPERVISED LEARNING:
CLASSIFICATION
Probabilistic Learning: Using Naive
Bayes
Understanding Naive Bayes
• When a meteorologist provides a weather forecast,
precipitation is typically described with terms such
as "70 percent chance of rain." Such forecasts are
known as probability of precipitation reports.
• The Naive Bayes algorithm describes a simple
method to apply Bayes' theorem to classification
problems.
• The Naive Bayes algorithm is named as such
because it makes some "naive"
• assumptions about the data.
• In particular, Naive Bayes assumes that all of
the features in the dataset are equally
important and independent.
• These assumptions are rarely true in most
real-world applications.
Using numeric features with Naive
Bayes
• Naive Bayes uses frequency tables to learn the data,
each feature must be categorical in order to create the
combinations of class and feature values comprising of
the matrix.
• Since numeric features do not have categories of
values, the preceding algorithm does not work directly
with numeric data.
• One easy and effective solution is to discretize numeric
features, which simply means
• that the numbers are put into categories known as bins.
• Discretization is also sometimes called binning.
• There are several different ways to discretize a
numeric feature. Perhaps the most common is
to explore the data for natural categories or cut
points in the distribution of data.
• For example, suppose that you added a feature
to the spam dataset that recorded the time of
night or day the e-mail was sent, from 0 to 24
hours past midnight.
Filtering mobile phone spam
with the Naive Bayes algorithm
• As the worldwide use of mobile phones has
grown, a new avenue for electronic junk mail has
opened for disreputable marketers.
• To develop the Naive Bayes classifier, we will use
data adapted from the SMS Spam Collection.
• This dataset includes the text of SMS messages
along with a label indicating whether the message
is unwanted. Junk messages are labeled spam,
while legitimate messages are labeled ham.
Air-Traffic Data
Days Season Fog Rain Class
Weekday Spring None None On Time
Weekday Winter None Slight On Time
Weekday Winter None None On Time
Holiday Winter High heavy Late
Saturday Summer Normal None On Time
Weekday Autumn Normal None Very Late
Holiday Summer High Slight On Time
Sunday Summer Normal None On Time
Weekday Winter High Heavy Very Late
Weekday Summer None Slight On Time
Cond. to next slide…
10
Air-Traffic Data
Cond. from previous slide…
Days Season Fog Rain Class
Saturday Spring High Heavy Cancelled
Weekday Summer High Slight On Time
Weekday Winter Normal None Late
Weekday Summer High None On Time
Weekday Winter Normal Heavy Very Late
Saturday Autumn High Slight On Time
Weekday Autumn None Heavy On Time
Holiday Spring Normal Slight On Time
Weekday Spring Normal slight On Time
Weekday Spring Normal slight On Time
11
Naïve Bayesian Classifier
• Example: With reference to the Air Traffic Dataset mentioned earlier, let us
tabulate all the posterior and prior probabilities as shown below.
12
Naïve Bayesian Classifier
13
Naïve Bayesian Classifier
Instance:
Week Day Winter High Heavy ???
Case1: Class = On Time : 0.70 × 0.64 × 0.14 × 0.29 × 0.07 = 0.0013
Case2: Class = Late : 0.10 × 0.50 × 1.0 × 0.50 × 0.50 = 0.0125
Case3: Class = Very Late : 0.15 × 1.0 × 0.67 × 0.33 × 0.67 = 0.0222
Case4: Class = Cancelled : 0.05 × 0.0 × 0.0 × 1.0 × 1.0 = 0.0000
Case3 is the strongest; Hence correct classification is Very Late
14
A Practice Example
age income studentcredit_rating
buys_compu
Example 8.4 <=30 high no fair no
<=30 high no excellent no
Class: 31…40 high no fair yes
C1:buys_computer = >40 medium no fair yes
‘yes’
>40 low yes fair yes
C2:buys_computer = ‘no’
>40 low yes excellent no
31…40 low yes excellent yes
Data instance
<=30 medium no fair no
X = (age <=30,
Income = medium, <=30 low yes fair yes
Student = yes >40 medium yes fair yes
Credit_rating = fair) <=30 medium yes excellent yes
31…40 medium no excellent yes
31…40 high yes fair yes
>40 medium no excellent no
15
A Practice Example
P(C ): P(buys_computer = “yes”) = 9/14 = 0.643
i
P(buys_computer = “no”) = 5/14= 0.357
Compute P(X|C ) for each class
i
P(age = “<=30” | buys_computer = “yes”) = 2/9 = 0.222
P(age = “<= 30” | buys_computer = “no”) = 3/5 = 0.6
P(income = “medium” | buys_computer = “yes”) = 4/9 = 0.444
P(income = “medium” | buys_computer = “no”) = 2/5 = 0.4
P(student = “yes” | buys_computer = “yes) = 6/9 = 0.667
P(student = “yes” | buys_computer = “no”) = 1/5 = 0.2
P(credit_rating = “fair” | buys_computer = “yes”) = 6/9 = 0.667
P(credit_rating = “fair” | buys_computer = “no”) = 2/5 = 0.4
X = (age <= 30 , income = medium, student = yes, credit_rating = fair)
P(X|Ci) : P(X|buys_computer = “yes”) = 0.222 × 0.444 × 0.667 × 0.667 = 0.044
P(X|buys_computer = “no”) = 0.6 × 0.4 × 0.2 × 0.4 = 0.019
P(X|Ci)*P(Ci) : P(X|buys_computer = “yes”) * P(buys_computer = “yes”) = 0.028
P(X|buys_computer = “no”) * P(buys_computer = “no”) = 0.007
Therefore, X belongs to class (“buys_computer = yes”)
16
• We'll begin by importing the CSV data and
saving it in a data frame:
> sms_raw <- [Link]("sms_spam.csv",
stringsAsFactors = FALSE)
• Using the str() function, we see that the
sms_raw data frame includes 5,559 total SMS
messages with two features: type and text.
> str(sms_raw)
• The type element is currently a character vector.
Since this is a categorical variable, it would be better
to convert it into a factor, as shown in the following
code:
> sms_raw$type <- factor(sms_raw$type)
• Examining this with the str() and table() functions, we
see that type has now been appropriately recoded as
a factor. Additionally, we see that 747 (about 13
percent) of SMS messages in our data were labeled
as spam, while the others were labeled as ham:
• SMS messages are strings of text composed of
words, spaces, numbers, and punctuation.
• Handling this type of complex data takes a lot
of thought and effort.
• This functionality has been provided by the
members of the R community in a text mining
package titled tm.
> [Link]("tm")
> library(tm)
• The first step in processing text data involves creating
a corpus, which is a collection of text documents.
• Since we already loaded the SMS message text into R,
we'll use the VectorSource() reader function to create
a source object from the existing sms_raw$text vector,
which can then be supplied to VCorpus() as follows:
• > sms_corpus <-
VCorpus(VectorSource(sms_raw$text))
• By printing the corpus, we see that it contains
documents for each of the 5,559 SMS
messages in the training data:
> print(sms_corpus)
• To view the actual message text, the
[Link]() function must be applied to the
desired messages.
• > [Link](sms_corpus[[1]])
• To view multiple documents, we'll need to use
[Link]() on several items in the
sms_corpus object. To do so, we'll use the
lapply() function.
> lapply(sms_corpus[1:2], [Link])
• the corpus contains the raw text of 5,559 text
messages. In order to perform our analysis,
we need to divide these messages into
individual words.
• The tm_map() function provides a method to
apply a transformation (also known as mapping)
to a tm corpus.
• Our first order of business will be to standardize
the messages to use only lowercase characters.
• We need to use the tm wrapper function
content_transformer() to treat tolower() as a
• transformation function that can be used to
access the corpus.
> sms_corpus_clean <- tm_map(sms_corpus,
content_transformer(tolower))
• Let's inspect the first message in the original
corpus and compare it to the same in the
transformed corpus:
> [Link](sms_corpus[[1]])
> [Link](sms_corpus_clean[[1]])
• Let's continue our cleanup by removing numbers
from the SMS messages.
> sms_corpus_clean <-
tm_map(sms_corpus_clean, removeNumbers)
• Our next task is to remove filler words such as to,
and, but, and or from our SMS messages. These
terms are known as stop words.
• The stop words alone are not a useful
transformation. What we need is a way to remove
any words that appear in the stop words list. The
solution lies in the removeWords() function
> sms_corpus_clean <-
tm_map(sms_corpus_clean,
removeWords, stopwords())
• We can also eliminate any punctuation from the
text messages using the built-in
removePunctuation() transformation:
> sms_corpus_clean <-
tm_map(sms_corpus_clean,
removePunctuation)
• Another common standardization for text data
involves reducing words to their root form in a
process called stemming.
• The tm package provides stemming
functionality via integration with the
SnowballC package.
> [Link]("SnowballC")
> library(SnowballC)
• The SnowballC package provides a wordStem()
function, which for a character vector, returns
the same vector of terms in its root form.
• In order to apply the wordStem() function to
an entire corpus of text documents, the tm
package includes a stemDocument()
transformation.
• We apply this to our corpus with the
tm_map() function exactly as done earlier
> sms_corpus_clean <-
tm_map(sms_corpus_clean, stemDocument)
• The final step in our text cleanup process is to
remove additional whitespace, using the built-
in stripWhitespace() transformation:
> sms_corpus_clean <-
tm_map(sms_corpus_clean,
stripWhitespace)
• The final step is to split the messages into individual
components through a process called tokenization.
• The DocumentTermMatrix() function will take a
corpus and create a data structure called a
Document Term Matrix (DTM) in which rows
indicate documents (SMS messages) and columns
indicate terms (words).
> sms_dtm <-
DocumentTermMatrix(sms_corpus_clean)
• With our data prepared for analysis, we now
need to split the data into training and test
datasets, so that once our spam classifier is
built, it can be evaluated on data it has not
previously seen.
• We'll divide the data into two portions: 75
percent for training and 25 percent for testing.
• We can simply take the first 4,169 for training
and leave the remaining 1,390 for testing.
> sms_dtm_train <- sms_dtm[1:4169, ]
> sms_dtm_test <- sms_dtm[4170:5559, ]
• The labels are not stored in the DTM, so we
would need to pull them from the original
sms_raw data frame:
> sms_train_labels <- sms_raw[1:4169, ]$type
> sms_test_labels <- sms_raw[4170:5559, ]
$type
• Let's compare the proportion of spam in the
training and test data frames:
> [Link](table(sms_train_labels))
> [Link](table(sms_test_labels))
• A word cloud is a way to visually depict the
frequency at which words appear in text data.
• Words appearing more often in the text are
shown in a larger font, while less common terms
are shown in smaller fonts.
• The wordcloud package provides a simple R function to
create this type of diagrams.
> [Link]("wordcloud")
> library(wordcloud)
• This will create a word cloud from our prepared SMS
corpus. Since we specified [Link] = FALSE, the
cloud will be arranged in a nonrandom order with higher
• frequency words placed closer to the center
• > wordcloud(sms_corpus_clean, [Link] = 50,
[Link] = FALSE)
• The [Link] parameter specifies the number of
times a word must appear in the corpus before it will
be displayed in the cloud.
• The final step in the data preparation process is to
transform the sparse matrix into a data structure that
can be used to train a Naive Bayes classifier.
• To reduce the number of features, we will eliminate
any word that appear in less than five SMS messages,
or in less than about 0.1 percent of the records in the
training data.
• Finding frequent words requires use of the
findFreqTerms() function in the tm package.
> sms_freq_words <- findFreqTerms(sms_dtm_train,
5)
> str(sms_freq_words)
• We now need to filter our DTM to include only the
terms appearing in a specified vector.
• we want all the rows, but only the columns
representing the words in the sms_freq_words vector,
our commands are:
> sms_dtm_freq_train<- sms_dtm_train[ ,
sms_freq_words]
> sms_dtm_freq_test <- sms_dtm_test[ ,
sms_freq_words]
• The Naive Bayes classifier is typically trained
on data with categorical features.
• The following defines a convert_counts()
function to convert counts to Yes/No strings:
> convert_counts <- function(x) {
x <- ifelse(x > 0, "Yes", "No")
}
• We now need to apply convert_counts() to each
of the columns in our sparse matrix.
• The apply() function allows a function to be
used on each of the rows or columns in a
matrix. It uses a MARGIN parameter to specify
either rows or columns.
> sms_train <- apply(sms_dtm_freq_train,
MARGIN = 2, convert_counts)
> sms_test <- apply(sms_dtm_freq_test,
MARGIN = 2, convert_counts)
• It is time to apply the Naive Bayes algorithm.
• The algorithm will use the presence or
absence of words to estimate the probability
that a given SMS message is spam.
• The Naive Bayes implementation we will
employ is in the e1071 package.
> [Link]("e1071")
> library(e1071)
• To build our model on the sms_train matrix,
we'll use the following command:
> sms_classifier <- naiveBayes(sms_train,
sms_train_labels)
• To evaluate the SMS classifier, we need to test
its predictions on unseen messages in the test
data.
• The classifier that we trained has been named
sms_classifier.
• The predict() function is used to make the
predictions.
> sms_test_pred <- predict(sms_classifier,
sms_test)
• To compare the predictions to the true values,
we'll use the CrossTable() function in the
gmodels package.
> library(gmodels)
> CrossTable(sms_test_pred,sms_test_labels,
[Link] = FALSE, prop.t = FALSE, dnn =
c('predicted', 'actual'))