Effective Methods for Improving Naive Bayes
Text Classifiers
Sang-Bum Kim1 , Hae-Chang Rim1 , DongSuk Yook1 , and Heui-Seok Lim2
1
Dept. of CSE, Korea University
Anam-dong 5 ka, SungPuk-gu
SEOUL, 136-701, KOREA
2
Dept. of Info&Comm, Chonan University
Anseo-Dong, Chonan
ChungChong-NamDo, 330-180, Korea
Abstract. Though naive Bayes text classifiers are widely used because
of its simplicity, the techniques for improving performances of these clas-
sifiers have been rarely studied. In this paper, we propose and evaluate
some general and effective techniques for improving performance of the
naive Bayes text classifier. We suggest document model based parameter
estimation and document length normalization to alleviate the problems
in the traditional multinomial approach for text classification. In addi-
tion, Mutual-Information-weighted naive Bayes text classifier is proposed
to increase the effect of highly informative words. Our techniques are
evaluated on the Reuters21578 and 20 Newsgroups collections, and sig-
nificant improvements are obtained over the existing multinomial naive
Bayes approach.
1 Introduction
Text categorization is the problem of assigning predefined categories to free
text documents. There are several similar tasks such as text filtering and rout-
ing. All of these tasks require text classifiers that decide which class is more
relevant to a given document or which document is more relevant to a fixed
user interest. Thus, text classifiers should be able to rank categories given a
document and rank documents given a class. A growing number of statistical
learning methods have been applied to these problems in recent years, includ-
ing nearest neighbor classifiers[7], naive Bayes classifiers[5], and support vector
machines[3], etc. Among these methods, naive Bayes text classifiers have been
widely used because of its simplicity although they have been reported as one
of poor-performing classfiers in text categorization task[8,2]. Since several stud-
ies show that naive Bayes performs surprisingly well in many other domains[1],
it is worth of clarifying the reason that naive Bayes fails in the text classifica-
tion tasks and improving them. This paper describes the problems in traditional
naive Bayes text classification approach and suggests some effective techniques
for improving them.
M. Ishizuka and A. Sattar (Eds.): PRICAI 2002, LNAI 2417, pp. 414–423, 2002.
c Springer-Verlag Berlin Heidelberg 2002
Effective Methods for Improving Naive Bayes Text Classifiers 415
2 Multinomial Naive Bayes for Text Classifier
For text categorization task, a naive Bayes is used to calculate probabilities of
a class given a document. Similar to the traditional probabilistic information
retrieval model, what we are concerned about is not the probability itself calcu-
lated by the naive Bayes formula but the ability to give higher scores to more
relevant documents given a class or to more relevant classes given a document.
For this reason, one can use the log-odds ratio as relevance score rather than
the probability itself. Thus, the relevance score of document di for class cj is
calculated with Bayes’ rule as follows:
P (cj |di ) P (di |cj ) P (cj )
Relevance(di , cj ) = log = log + log (1)
P (c¯j |di ) P (di |c¯j ) P (c¯j )
and, we can estimate the priors as follows:
number of documents belong to cj
P (cj ) =
total number of documents
Now, the only parameters we should estimate are P (di |cj ) and P (di |c¯j ).
There have been two approaches to estimate them; multivariate approach
and multinomial approach. In the multivariate approach, a document is consid-
ered as a |V |-dimensional vector D = (w1 , w2 , · · · w|V | ) which is the result of
|V | independent Bernoulli trials, where |V | is the vocabulary size and wk is a
binary variable representing the occurrence or non-occurrence of the k-th word
in the vocabulary. The most serious problem in this approach is that there is no
way to reflect the information about term frequencies. This weak point of the
multivariate approach was experimentally surveyed by [5].
On the other hand, the multinomial approach specifies that a document is
represented by the set of word occurrences from the document. The order of the
words is lost, but the number of occurrences of each word in the document is
captured. When calculating the probability of a document, one multiplies the
probability of the words that occur. In the multinomial approach, the first part
of formula (1) is calculated as follows:
|V | P (w |c )T Fik
Pn (di |cj ) P (|di |)|di |! k=1 n TkFikj !
log = log |V | P (w |c¯ )T Fik
Pn (di |c¯j ) P (|di |)|di |! k=1 n TkFikj !
|V |
Pn (wk |cj )
= T Fik · log (2)
Pn (wk |c¯j )
k=1,wk ∈di
In formula (2), T Fik represents the number of wk in the document di . The
parameters Pn (wk |cj ) and Pn (wk |c¯j ) are estimated as follows:
|D|
θ + i=1 T Fik P (yi = cj |di )
Pn (wk |cj ) = |V | |D| (3)
θ · |V | + s=1 i=1 T Fis P (yi = cj |di )
|D|
θ + i=1 T Fik P (yi = cj |di )
Pn (wk |c¯j ) = |V | |D| (4)
θ · |V | + s=1 i=1 T Fis P (yi = cj |di )