I.J.

Modern Education and Computer Science, 2013, 12, 31-39

Published Online December 2013 in MECS ([Link]
DOI: 10.5815/ijmecs.2013.12.05

Spam Mail Detection through Data Mining – A

Comparative Performance Analysis
Megha Rathi
Department of computer Science Engineering of Jaypee Institute of Information Technology, Noida, India
E-mail: [Link]@[Link]

Vikas Pareek
Department of Computer Science of Banasthali University, Banasthali, India
E-mail: er_pareekvikas@[Link]

Abstract — As web is expanding day by day and people user. These kind of unwanted mails are better known as
generally rely on web for communication so e-mails are Spam Mails. Spam email is the practice of frequently
the fastest way to send information from one place to sending unwanted data or bulk data in a large quantity to
another. Now a day’s all the transactions all the some email accounts. Spam Mail is a subset of
communication whether general or of business taking electronic spam involving nearly identical messages sent
place through e-mails. E-mail is an effective tool for to various recipients by email. Spam mails also include
communication as it saves a lot of time and cost. But e- malware as scripts or other executable file attachment.
mails are also affected by attacks which include Spam There are two main types of spam and they have
Mails. Spam is the use of electronic messaging systems different affects on Internet users. Cancellable Usenet
to send bulk data. Spam is flooding the Internet with spam is a single message sent to 20 or more Usenet
many copies of the same message, in an attempt to force groups. Usenet spams aims at “lurkers”, people who
the message on people who would not otherwise choose read newsgroups but rarely or never post and give their
to receive it. In this study, we analyze various data address away. Usenet spam subverts the ability of
mining approach to spam dataset in order to find out the system administrator to manage the topics they accept
best classifier for email classification. In this paper we on their systems. Another type of Email spam targets
analyze the performance of various classifiers with individual users with direct mail messages. Email spam
feature selection algorithm and without feature selection list are created by scanning Usenet postings, stealing
algorithm. Initially we experiment with the entire dataset Internet mailing list. Email spam is any email that meets
without selecting the features and apply classifiers one the following three criteria:
by one and check the results. Then we apply Best-First 1) Anonymity: The address and identity of the sender
feature selection algorithm in order to select the desired are concealed.
features and then apply various classifiers for 2) Mass Mailing: The email is sent to large group of
classification. In this study it has been found that results people.
are improved in terms of accuracy when we embed 3) Unsolicited: The email is not requested by
feature selection process in the experiment. Finally we recipients.
found Random Tree as best classifier for spam mail Spam Mail has become an increasing problem in
classification with accuracy = 99.72%. Still none of the recent years. It has been estimated that around 70% of
algorithm achieves 100% accuracy in classifying spam all emails are spam. As the usage of web expanding,
emails but Random Tree is very nearby to that. problem of spam mails are also expanding. According to
[1] it has been found that on an average 10 days per year
Index Terms — Classifier, Feature Selection, E-mails, waste on dealing with spam mails only. Spam is an
Spam Mails. expensive problem that costs billion of dollars per year
to service providers for lost of bandwidth. Spam is a
major problem that attacks the existence of electronic
I. INTRODUCTION messages. So it is very essential to distinguish emails
E-Mail is an effective way of communication as it from spam mails, many methods have been proposed for
saves a lot of time and money this makes it as a classification of email messages as spam mail or
favourite means of communication in personal as well as legitimate mail and it has been found that machine
in professional communication. E-mails provide a way learning algorithm success ratio for classification is very
for internet users to easily transfer information globally. high [2].
But there is also a case when your e-mails are affected Several algorithms are used for classification of spam
by attacks whether active or passive. Sometimes we mails which are extensively utilize and analyze out of
receive e-mail from unknown source and also e-mail which support vector machine, Naïve Bayes, Decision
comprised of contents which is of no importance to the Tree, Neural network classifiers are well known
classifiers. In this paper we experiment our data set with

Copyright © 2013 MECS I.J. Modern Education and Computer Science, 2013, 12, 31-39
32 Spam Mail Detection through Data Mining – A Comparative Performance Analysis

these given algorithms: Naïve Bayes, Bayes Net, 2) Clusters: Data items are placed in a group
Support vector machine (SVM), function Tree (FT), J48, according to logical relationships. For example, data can
Random Forest and Random Tree. Initially we be mined to identify market segments.
experiment on entire data set which consists of total 58 3) Associations: Data mining is applied to data set to
attributes and total number of instances is 4601. We find out the associations.
apply above mentioned algorithm one by one on the data 4) Sequential Patterns: Data is mined to anticipate
set and check the result and it is retrieved from the study behavior patterns and trends.
that out of all these classifiers Random Forest and Basically Data mining involves listed five elements:
Random Tree works well and gives accuracy better than
1) Extract, transform, and load data on data
other classifiers in detection of spam mails. In order to
warehouse system.
compare the result that classifiers works well with some
2) Store and manage data in multidimensional
attributes selected or not, then we apply Feature
database system.
selection algorithm on the same dataset (the algorithm
3) Provide data access in an easier manner to business
we used here is Best First Search algorithm) and apply
analyst and technical professionals.
the same classifiers with features selected. Out of 58
4) Analyze data by existing tool/application software.
features only 15 features are selected and apply the same
5) Make data in format which is useful to concerned
above mentioned algorithm on this reduced dataset.
user such as graph or tables.
From this study it is found that all classifier’s accuracy
improved when we select features through Best-First Sometimes we treat data mining as a synonym for
algorithm. Again when compared with all classifiers another known term, Knowledge discovery from
which we experimented on this reduced data set Random databases (KDD), because data mining is necessary step
Tree shows better results in context of accuracy. in the process of knowledge discovery from the database.
This paper is organized as follows: Section 2 Knowledge discovery is a combination of all these steps
comprised of Background study, Section 3 presents shown in fig.1.
related work, Section 4 presents the Experimental work
and results, Section 5 presents Experimental Results and Steps of knowledge discovery
Section 6 presents conclusion and future work.

II. BACKGROUND STUDY Data Data Data Data Data Pattern Knowledge
Cleaning Integration Selection Transformation Mining Evaluation Presentation
This section presents an overview of what is Data
Mining, different algorithm of data mining, explains
Figure. 1: Knowledge Discovery steps
Feature selection and most of the terms that we used in
this paper. Data mining involves many different algorithms to
A. Data Mining achieve the desired tasks. All of these algorithms try to
fit a model, the algorithm examine the data and find out
Data Mining is basically the discovery of knowledge
the model that is closest to the characteristics of the data
from the large database. It is a technique that attempts to
being examined. Data mining algorithms characterized
find out new patterns in huge data sets. It is mixture of
based on the purpose of the algorithm to fit a model to
various fields like Artificial Intelligence, Machine
the data, based on Preference, and all algorithms require
Learning, statistics, and Database systems. The main
some approach for searching. Fig.2 shows the model
objective of data mining approach is to extract
than can be either predictive or descriptive.
information from a data set and transform it into and
understandable form for further use. The actual data
mining task is the automatic or semi-automatic analysis Data Mining
of large quantities of data to extract previously
undetermined interesting patterns. Data Mining is the
process of analyzing data from different perspective and
summarizing it into useful information and this
information can be used to increase revenue, cut costs, Predictive Descriptive
for classification, prediction etc. It is the process of
finding correlations in large relational databases. While
large-scale information technology has been evolving Classification Sequence
Time Series Association
separate transactions and analytical systems, data mining Rules Discovery
Analysis
provides the link between the two approaches. Data
mining software analyzes relationships in stored data Regression
based on end user queries. In general these 4 types of Prediction
Summarization Clustering
relationships are sought:
1) Classes: Class is used to place the data in Figure. 2: Data Mining Models and tasks
predetermined groups.

Copyright © 2013 MECS I.J. Modern Education and Computer Science, 2013, 12, 31-39
 Spam Mail Detection through Data Mining – A Comparative Performance Analysis 33

If we consider X to be an object to be classified with

B. Algorithm used in this study
the probabilities of belonging to one of the classes
2.2.1 Support Vector Machine C1,C2,C3 etc. by calculating P(Ci/X). Once these
probabilities have been computed for all the classes, we
Support vector machines are supervised learning simply assign X to the class that has highest probability.
models with associated learning models that analyze
data and are mainly used for classification purpose. P (Ci/X) = [P(X/Ci) P (Ci)] / P(X) (2)
Support vector machine (SVM) takes a set of input data
Where P (Ci/X) is the probability of the object X
and output the prediction that data lies in one of the two
belonging to a class Ci, P(X/Ci) is the probability of
categories i.e. it classify the data into two possible
obtaining attribute values X if we know that it belongs to
classes. Given a set of training examples, each marked
class Ci. P (Ci) is the probability of any object belonging
as belonging to one of the two classes, an SVM training
to a class Ci without any other information, and P(X) is
algorithm build a model that assign new data in one
the probability of obtaining attribute values X whatever
class or the other. Basically SVM is a representation of
class the object belongs to.
the examples as points in space, mapped so that new
examples of the separate classes are clearly classified as 2.2.3 Decision Tree
belonging to one of the two categories. A support vector A decision tree is a classification method that results
machine performs classification by constructing an N- in a flow-chart like tree structure where each node
dimensional hyper plane that optimally categorizes the denotes a test on attribute value and each branch
data in two categories. SVM are set of related supervised represents an outcome of the test. The tree leaves
learning methods used for classification and regression represents the classes. Decision tree is model that is both
[3]. SVM map input vector to a higher dimensional predictive and descriptive; it represents relationships
plane where a maximal separating hyper plane is found in training data. The tree consists of zero or more
constructed. Two parallel hyper planes are constructed internal nodes and one or more leaf nodes with each
on each side of the hyper plane that separates the data. internal node being a decision node having two or more
The separating hyper plane is the hyper plane that child nodes. Decision tree use divide and conquer
maximizes the distance between the two hyper planes. method to split the problem search space into subsets.
Larger the margin or distance better the generalization Decision tree is constructed to model the classification
error of the classifier. process. Once the tree is built it is applied to each tuple
2.2.2 Naïve Bayes in the database and results in a classification for that
tuple. There are two basic steps in this technique:
A naïve Bayes classifier is a simple probabilistic
building the tree and applying the tree to the dataset. The
classifier with strong independence assumptions. In
decision tree approach to classification is to divide the
simple terms, a naïve bayes classifier assumes that the
search space into rectangular regions. A tuple is
presence/absence of a particular feature of a class is
classified based on the region into which it falls.
unrelated to the presence/absence of any other feature,
Given a database D = {t1, t2,….., tn} where ti =
given the class variable depending on the nature of
{ ti1 ,……, tih} and the database schema contains the
probability model, naïve bayes classifier can be trained
following attributes {A1, A2, ……, Ah}. Also given is a
in supervised learning setting. An advantage of the naïve
set of classes C = {C1,……, Cm }. A Decision tree is a
bayes classifier is that it only requires a small amount of
tree associated with D that has the following properties:
training data to estimate the parameters required for
1) Each internal node is labeled with an attribute Ai.
classification. In Bayesian classification we have a
2) Each edge is labeled with a predicate that can be
hypothesis that the given data belongs to a particular
applied to the attribute associated with the parent.
class. We then calculate the probability for the
3) Each leaf node is labeled with a class Cj.
hypothesis to be true. Bayesian classifiers are basically
statistical classifiers i.e. they can predict the class 2.2.4 Feature Selection
membership probabilities, such as the probability that a Feature Selection also known as feature reduction,
given sample data belongs to a particular class. attribute selection is the technique of selecting a subset
The naïve Bayes technique is based on Bayesian of relevant features for building the learning models.
approach hence it is a simple, clear and fast classifier [4]. Feature selection is very important step in analyzing the
Before reaching to the main term of Baye’s theorem we data, by removing irrelevant and redundant features
will first analyze some terms used in the theorem. P (A) from the data. Feature selection overall improves the
is the probability that event A will occur. P (A/B) is the performance of learning model by:
probability that event A will happen given that event B 1) Alleviating the effect of curse of dimensionality.
has already happened or we may define it as the 2) Enhancing generalization capability.
conditional probability of A based on the condition that 3) Speeding up learning process.
B has already happened. Bayes theorem is defined in 4) Improving model interpretability.
equation 1. Feature Selection helps in gaining the better
P (A/B) = P (B/A) P (A) P (B) (1) understanding of the data by telling which are the
important attributes or features and how they are related
with each other. It is the process of selecting a subset of

Copyright © 2013 MECS I.J. Modern Education and Computer Science, 2013, 12, 31-39
34 Spam Mail Detection through Data Mining – A Comparative Performance Analysis

the terms occurring in the training set and using only this
subset as features in classification. It serves two main Training Data Classification
Algorithm
purposes: First, it makes training and applying a
classifier more efficient by decreasing the size of data
set. Second, feature selection enhances accuracy of
classifier by eliminating extra features from the data set.
A Feature selection algorithm is a computational Classification rules
solution which is motivated by certain rules of relevance. If age = “31---40” and
An irrelevant feature is not useful for induction, but it income = high then
also not essential that all relevant features are used for credit rating = excellent
induction [5]. Feature Selection algorithm can be
classified according to the kind of output they produce:
(1) algorithms that produce a linear order of features and Name Age Income Credit rating
(2) algorithms that produce a subset of original features. Sandy <=30 low fair
In the study [6, 7, 8] characterization of Feature
Bill <=30 low excellent
selection algorithm is described. In this context it is
possible to describe this characterization as a search Susan >40 medium fair
problem as follows:
Figure. 3: Learning and Training of classifier
1) Search Organization. This technique is related to
the portion of hypothesis investigated with respect to In the second step as shown in fig. 4, the model is
their total number. used for classification. First the predictive accuracy of
2) Generation of Successors. This technique defines the model is estimated. The accuracy of a model on a
by which possible variants of the current hypothesis are given test data set is the percentage of test set samples
proposed. that are correctly classified by the model. For each test
3) Evaluation Measure. Method, by which variants or sample the known class label is compared with the
successor candidates are evaluated, allowing to compare learned model’s class prediction for that sample.
different hypothesis to supervise the search process.
In Feature Selection Algorithm we select a subset of
features. Subset selection evaluates a subset of features Classification Rules
and these algorithms can be broken into Wrappers,
filters and Embedded. Wrappers use a search algorithm
to find out the space of possible features. Wrappers are
Test Data New Data
computationally expensive and have a risk of over fitting
the model. Filters are same as Wrappers in context of
search space, but instead of evaluating against a model,
a simpler filter is evaluated. Embedded approach is (John,31…40,high,credit rating?)
embedded in and specific to a model.
Following are some extensively used Feature selection
algorithms: (1) Best First (2) Simulated Annealing (3) Excellent
Genetic algorithm (4) Scatter Search and (5) Greedy
forward selection etc. Name Age Income Credit rating
Frank > 40 high fair
2.2.5 Classification and Prediction
Crest <=30 low fair
Classification is the separation of objects into classes. Annee 31…40 high excellent
If the classes are created without looking at the data then
the classification is known as apriori classification. If Figure. 4: Classification
classes are created by looking at the data then the
classification method is known as posterior classification. Prediction can be viewed as the construction and use
On classification it is assumed that the classes have been of a model to assess the class of an unlabeled sample, or
deemed apriori and classification then consists of to assess the value ranges of an attribute that a given
training the system so that when a new object is sample is likely to have. In this context classification
introduced to the trained system it is able to assign the and regression are the two major types of prediction
object to one of the existing classes. This approach is problems, where classification is used to predict discrete
better known as supervised learning. Data Classification or nominal values, while regression is used to predict
is a two step process as shown in fig. 3. In the first step, continuous or ordered values.
model is built describing a predetermined set of data
classes. The model is constructed by analyzing database
III. RELATED WORK
tuples described by the attributes. Each tuple is assumed
to belong to one of the existing class, as determined by Spam Mails are one of the major problematic areas in
the class label attribute. The data tuples analyzed to the Internet world which can bring financial loss to
build the model collectively form the training set. organizations and also bring damage to individual users

Copyright © 2013 MECS I.J. Modern Education and Computer Science, 2013, 12, 31-39
 Spam Mail Detection through Data Mining – A Comparative Performance Analysis 35

as well. Email spam also known as junk mails which are help of existing machine learning techniques. According
sent to a group of recipients who have not requested it. to the study [15] clusters of spam emails are created with
Spam is a serious problem that threatens the existence of the help of criterion function. Criterion function is
e-mail services. As it involves no cost so it is quiet defined as the maximization of similarity between
cheap to send bulk e-mail to a group of users. It messages in clusters and this similarity is calculated
consumes a lot of time to delete or sort these spam using k-nearest neighbour algorithm.
emails and also introduces a risk of deleting normal Symbiotic Data Mining is a distributed data mining
mails by mistake. In the study [9] Rambow et al. approach which unifies content based filtering with
applied machine learning techniques for email collaborative filtering is described in [16]. The main
summarization. In this study, RIPPER classifier is used objective is to make use of local filters again in order to
for the determination of sentences which should be improve personalized filtering in context of privacy. In
included in a summary. Learning model use features study [17] email classifiers based on the approach of
such as Linguistic feature, email features, and threading feed forward back propagation neural network and
structure. This approach requires positive examples in Bayesian classifiers are evaluated. From this study it is
huge quantity and it is also found that summaries are not found that feed forward back propagation neural
produced for varying length based on user interest. network classifier provides very high accuracy as
There are so many existing techniques for detection of compared to other existing classifiers. In the paper [18]
these spam emails. These approaches come mainly from Bayesian approach is applied for the problem of
the area of Artificial Intelligence, Data Mining, or classification and clustering using model based on the
Machine Learning. Machine learning techniques are assumptions like: population, subject, latent variable,
more varied and used extensively for spam mail and sampling scheme.
classification. Decision tree classify spam mails using According to [19] content filtering was one of the first
previous data [10]. But it is costly to calculate and types of anti spam filter. These types of filters make use
recalculate as spammers change technique. In the study of hard coded rules which has an associated score and is
[11] Bayesian networks found as the very popular updated periodically. One main example of such type of
technique for spam mail detection. But with this filter is Spam Assassin [20] which works by scanning
approach it is quiet difficult to scale up on many features the text document of the e-mail against each rule and
to come out with the judgement. add score for all matching rules. According to the study
In [12] fuzzy clustering approach is used. In this paper [21] if total score of the e-mail exceeds some set
author evaluated the use of fuzzy clustering and text threshold score then that message falls into spam mail
mining for spam filtering. Fuzzy clustering is scalable category. In order to generate these score a single
and easy to update approach. This study deals with the perceptron is used where the inputs to the perceptron
examination of use of fuzzy clustering algorithm to build indicate whether a rule was matched and the weight for
a spam filter. Classifier has been tested on different data the corresponding input indicates the score for each rule.
sets and after testing Fuzzy C-Means using In the paper [22] spam is detected using artificial
Heterogeneous Value Difference Metric with variable neural network. In this paper author designed the
percentages of spam and used a standard model of artificial neural network spam detector using the
assessment for the problem of spam mail classification. perceptron learning rule. Perceptron employs a
This paper makes use of text mining and fuzzy stochastic gradient method for training, where the true
clustering as an anti-spam technique. If each email that gradient is evaluated on a single training example and
comes in is used as part of the data pool to make the weights are adjusted accordingly until a stopping
decisions about future emails, spam trends will be criterion is met. At each iteration an error weight
detected. It is found that there is not large cost of adjustment value are computed by comparing the actual
calculation and recalculation that would occur with output value with the expected output value. Testing
decision tree, or with some rule-based filters. phase was done by subjecting the Artificial Neural
We all were aware about the fact that Spam mails Network to messages that were not used in training
create a lot of problem in today’s world. So various without adjusting the weights.
approaches are developed to stop spam mails. The main
objective in spam filtering is to rule out the unwanted
emails automatically from user inbox. These unwanted
are root cause for the problems like filling mailboxes, IV. PROPOSED WORK
engulfing important personal mail, wasting a lot of In this study we detect spam mails using various
network bandwidth also causes congestion problem, classifiers. The whole experiment comprised of two
time and energy loss to the users while sorting these parts. First we will apply various classifiers for spam
unwanted mails [13]. In the study [14] two methods are mail classification and check the results in terms of
described for classification. First is done with some rules accuracy for each classifier. Here we use the entire data
that are defined manually, like rule based expert system. set and apply algorithm one by one without selecting
This technique of classification is applied when classes any feature. In the second part we detect spam mails by
are static, and their components are easily separated in not using the entire data set instead we apply feature
accordance with the features. Second is done with the selection algorithm first, the algorithm which we use

Copyright © 2013 MECS I.J. Modern Education and Computer Science, 2013, 12, 31-39
36 Spam Mail Detection through Data Mining – A Comparative Performance Analysis

here is Best-First Feature Selection algorithm then on (RMSE), Relative Absolute Error (RAE), and Root
the reduced data set with selected features we will apply Relative Squared Error (RRSE) for all the classifiers and
all the classifiers one by one and check the results. It is compare the results based on all these statistics. Same
found that classifier’s accuracy improved when we characteristics are also compared for all classifiers when
embed feature selection algorithm in the process. These we use feature selection in the process.
are some of the classifiers that we use in this study: (1) The dataset which we use is from UCI Machine
Naïve Bayes (2) Bayesian Net (3) Support Vector Learning Repository [23] which consists of 58 attributes
Machine (SVM) (4) Function Tree (FT) (5) J48 (6) where 57 continuous attribute and 1 nominal class label
Random Forest (7) Random Tree and (8) Simple Cart. attribute and total number of instances is 4601. Table I
We find out accuracy, Kappa statistics (KS), Mean presents the entire dataset with attribute description.
Absolute Error (MAE), Root Mean Squared Error

TABLE 1: Dataset attribute Description

Attribute Number Attribute Type Attribute Description

A1 to A48 char_freq_CHAR Percentage of characters in the e-mail that match
CHAR.
A49 to A54 capital_run_length_average Average length of uninterrupted sequences of
capital letters.
A55 capital_run_length_longest Length of longest uninterrupted sequence of
capital letters.
A56 capital_run_length_longest Length of longest uninterrupted sequence of
capital letters.
A57 capital_run_length_total Total number of capital letters in the e-mail.
A58 Class Attribute Denotes whether e-mail was considered as spam
with class label (1) and not spam with class label
(0).

The overall design of the proposed system is depicted tasks like data cleaning, data integration, data
in Fig.5 for classification of e-mail as spam without transformation, or data reduction. Before applying any
taken into consideration the feature selection approach data mining techniques to the data set we first normalize
and Fig.6 for classification of e-mail as spam with taken the entire data set in order to yield good results. Up to
into consideration the Feature Selection Approach. this step both the proposed system works similarly then
Below shown (Fig. 5 and Fig.6) is the overall after as per architecture I shown in fig.5 we apply
architecture of proposed system. In this architecture first classifiers one by one to the entire data set and evaluate
we train the spam data set which comprised of 58 the performance of classifier. Then test the data using
attributes with total 4601 instances. Then we apply the classifiers and classify mails as spam and non spam.
Preprocessing, as we all know that real world data As per proposed architecture II shown in Fig. 6 after the
contains missing values or noisy values so in order to pre-processing step we first apply Feature selection
produce good results from the data set we need to mine algorithm, the algorithm which we deploy here is Best-
data. As quality decision depends on good quality data, First Feature Selection algorithm. Table II reflects the
pre-processing is crucial step before applying any view of selected features after applying the algorithm to
classifier to the data set. Pre-Processing involves the the data set.

TABLE 2: Selected Attributes after Feature Selection

Attribute number Attribute type Attribute Description

4,5,7,16,21,23,24, char_freq_CHAR Percentage of characters in the e-mail
25,27,42,44,46 that match CHAR

52,53 Capital_run_length_average Average length of uninterrupted

sequences of capital letters.
55 capital_run_length_longest Length of longest uninterrupted
sequence of capital letters.

Copyright © 2013 MECS I.J. Modern Education and Computer Science, 2013, 12, 31-39
 Spam Mail Detection through Data Mining – A Comparative Performance Analysis 37

Total 15 attributes are selected out of 58 attributes. Training

Attribute number 4, 5, 7, 16, 21, 23, 24, 25, 27, 42, 44, Preprocessing
Spam
46, 52, 53 and 55are the selected attributes. Data set
Classification algorithms are applied one by one on all
these selected 15 attributes and results which we are
getting are more promising than without applying Feature selection algorithm (Best First)
feature selection approach in the entire process of
detection of spam mails. Feature Selection has been an Reduced data set with selected features
active and fruitful field of research in machine learning,
statistics and data mining. The main aim of this
approach is to select a subset of data sets by eliminating
features or attributes which is of no use, or eliminating
the redundant data from the data set. Feature Selection Classifier 1 Classifier 2 Classifier n
improves efficiency, and also accuracy of the classifier
improved after applying feature selection algorithm.
Feature Selection in supervised learning has main
objective of finding a feature subset that enhances the
classifier accuracy.
Performance Evaluation

Training Spam
Preprocessin
Data set
g Test spam
data Best Classifier

Class Label

Figure. 6: architectural view of proposed system with feature

Classifier 1 Classifier 2 Classifier n selection

V. EXPERIMENT AND ANALYSIS

A. Experiment I
Performance Evaluation
In order to validate the proposed scheme for spam
mail detection, we conduct several experiments. The
main objective is to find out the best classifier whose
accuracy is better than the rest of the classifiers. The
Test spam dataset which we use is Spambase dataset consisting of
data Best Classifier
57 attributes with one target attribute in discrete format.
Following classification are applied one by one on the
dataset: (1) Naïve Bayes (2) Bayesian Net (3) Support
Vector Machine (SVM) (4) Function Tree (FT) (5) J48
(6) Random Forest (7) Random Tree and (8) Simple
Class Label Cart. And it is found form this study that out of all
classifiers investigated on the given data set Random
forest achieves highest accuracy that is 94.82%. Table
III presented the result of entire classifiers in terms of
accuracy, Kappa statistics (KS), Mean Absolute Error
Figure. 5: Overall architecture of proposed system (MAE), Root Mean Squared Error (RMSE), Relative
Absolute Error (RAE), and Root Relative Squared Error
(RRSE). We computed all these listed statistics and
prepare this comparative table from the result. After
analyzing the data presented in table III , Random Forest
is found to be the best classifier for spam mail
classification with accuracy= 94.82%, then second
highest accuracy is achieved by FT whose accuracy is
93.34% and so on. So from this study it is found that tree
like classifier performs well in case of classification of
spam mails.

Copyright © 2013 MECS I.J. Modern Education and Computer Science, 2013, 12, 31-39
38 Spam Mail Detection through Data Mining – A Comparative Performance Analysis

TABLE 3: Result Set (without selecting the features)

Algorithm Accuracy (%) KS MAE RMSE RAE RRSE
Naïve Bayes 79.2871 0.5965 0.2066 0.4527 43.2668 92.64
Bayes Net 89.8066 0.7838 0.1026 0.2995 21.4795 61.08
SVM 90.4151 0.7954 0.0958 0.3096 20.0706 63.35
FT 93.34 0.861 0.0742 0.2468 15.5274 50.49
J48 92.97 0.8528 0.0892 0.2562 18.6861 52.43
Random Forest 94.82 0.8908 0.0961 0.2064 20.1225 42.23
Random Tree 90.93 0.8108 0.0903 0.3001 18.9101 61.40
Simple Cart 92.43 0.8410 0.1055 0.2606 22.0913 53.33

approach produce above 95% accurate results for four

B. Experiment II
classifiers (FT, J48, Random Forest, Random Tree) and
In this experiment we first applied Best-First Feature above 90% accurate results for two classifiers (Simple
selection algorithm for selecting a subset of features Cart, Bayes Net). And highest accuracy is achieved by
from the given data set. Initially total 58 attributes was Random Tree which is equal to 99.7175% and second
present in the given Spambase data set, but after highest is achieved by Random Forest which is equal to
applying Best-First algorithm on the given data total 15 99.54%.However it is quiet difficult to achieve 100%
attributes are selected. Then we apply Classifiers on this accuracy but these two classifiers (Random Tree and
reduced data set for the detection of spam mail. Table Random Forest) are very nearby to that.
IV gives the summary of the result. Best first filtering

TABLE 4: Result Set after filtering with Best-First

Algorithm Accuracy (%) KS MAE RMSE RAE RRSE
Naïve Bayes 78.9394 0.5935 0.2010 0.4235 42.0966 86.6669
Bayes Net 92.719 0.8456 0.0825 0.2444 17.2669 50.0103
SVM 86.54 0.7091 0.1345 0.3668 28.1717 75.0630
FT 95.54 0.9064 0.0550 0.1987 11.5181 40.6682
J48 95.65 0.9083 0.0772 0.1965 16.1653 40.20
Random Forest 99.54 0.9904 0.0252 0.0827 5.271 16.9271
Random Tree 99.7175 0.9941 0.0040 0.045 0.8478 9.2079
Simple Cart 93.9361 0.8721 0.1070 0.2313 22.4038 47.33

here in this study we achieve highest accuracy =

VI. CONCLUSION 99.715%. As we all know that it is very difficult to
achieve 100% accuracy but Random Tree and Random
In Spam mail classification is major area of concern Forest (accuracy>99%) is very nearby to that. Therefore
these days as it helps in the detection of unwanted e- it is find that tree like classifiers works well in spam
mails and threats. So now a day’s most of the mail detection and accuracy improved incredibly when
researchers are working in this area in order to find out we first apply feature selection algorithm into the entire
the best classifier for detecting the spam mails. So a process.
filter is required with high accuracy to filter the
unwanted mails or spam mails. In this paper we focussed
on finding the best classifier for spam mail classification
REFERENCES
using Data Mining techniques. So we applied various
classification algorithms on the given input data set and [1] Nie N, Simpser A, Stepanikova I, and Zheng [Link]
check the results. From this study we analyze that years after the birth of Internet, how do Americans
classifiers works well when we embed feature selection use the internet in their daily lives[R]. Technical
approach in the classification process that is the report, Stanford University, 2004.
accuracy improved drastically when classifiers are [2] Almeida T, Yamakami A, Almeida J. Evaluation of
applied on the reduced data set instead of the entire data approaches for dimensionality reduction applied
set. The results gained were promising Accuracy of the with NaïveBayes anti-spam filters [C]. In the
classifier Random Tree is 99.715% with best-first Proceedings of the 8th IEEE International
feature selection algorithm and accuracy is 90.93% only
when we don’t apply this subset selection algorithm. So,

Copyright © 2013 MECS I.J. Modern Education and Computer Science, 2013, 12, 31-39
 Spam Mail Detection through Data Mining – A Comparative Performance Analysis 39

conference on machine learning and applications, [18] Erosheva E A and Fienberg S E. Bayesian mixed
Miami, FL, USA,2009, 517-522. membership models for soft clustering and
[3] Vapnik V N. Statistical learning theory [M]. John classification [J]. Proceedings of National
W iley &Sons, NewYork, N Y, 1998. Academy of Sciences, 2004, 97(22):11885-11892.
[4] Ian H, Witten and Eibe [Link] Mining: [19] Crawford E, Kay J, McCreath E. Automatic
Practical machine learning tools and techniques”, induction of rules for e-mail classification [C]. In
2nd Edition. San Fransisco: Morgan Kaufmann; 6th Australian Document Computing symposium,
2005. Coffs Harbour, Australia, 2001, 13-20.
[5] Caruana R.A. and Freitag D. How useful is [20] Spam Assassin. The Apache Spam Assassin
Relevance? Technical Report [A]. AAAI Project. [Link]
Symposium on Relevance, New Orleans, 1994. [21] Stern H. Fast Spam Assassin Score Learning tool
[6] Blum A.L. and Langley P. Selection of Relevant [Link]
Features and Examples in Machine Learning [C]. ssin 3.0.3/masses/[Link],2004.
In International Symposium on Artificial [22] Kufandirimbwa O, Gotora R. Spam detection using
Intelligence on Relevance, 1997, 245-271. Artificial Neural Networks [J]. In Online Journal of
[7] Doak J. An Evaluation of Feature Selection Physical and Environmental Science Research,
Methods and their Application to computer 2012, 1:22-29.
Security [R]. Technical Report CSE-92-18, Davis, [23] UCI – Machine Learning Repository – Spambase
Ca: University of California, Department of [Link]://[Link]/ml/datasets/Spam
computer Science, 1992. base.
[8] Liu H and Motoda H, and Dash M. A Monotonic
Measure for Optimal Feature Selection [C]. In Proc.
Of the European Conf. on Machine Learning,
Springer Verlag, 1998, 101-106.
Authors’ Profiles
[9] Ducheneaut N and Bellotti V. E-mail as habitat: an
exploration of embedded personal information Megha Rathi: She is Assistant
management [A]. Interactions ACM, 2001, 8: 30- Professor (Grade II) at Jaypee
38. Institute of Information Technology,
[10] Carreras X, and Marquez L. Boosting trees for anti India. She holds a Masters of
spam filtering [C]. In International conference on Technology and a Bachelor of
Recent Advances in Natural Language Processing. , Engineering degree in Computer
2001 160-167. Science and Engineering. Currently
[11] Sahami M, Dumasi S, Heckerman D, and Horvitz she is pursuing her PhD in Computer Science and
E. A Bayesian approach to filtering junk e-mail: In Engineering. Her areas of interest are Database systems,
Learning for text categorization [A]. Papers from Software Engineering, Software Testing and Artificial
the 1998 Workshop, Madison, Wisconsin, 1998. Intelligence.
[12] Mohammad N.T.A Fuzzy clustering approach to
filter spam E-mail [A].Proceedings of World Vikas Pareek: He is Associate
Congress on Engineering, vol. 3, WCE-2011. Professor at Banasthali University,
[13] Ahmed K. An overview of content-based spam Rajasthan, India. He obtained his
filtering techniques [A]. Informatica, 2007, 31(3): Doctorate in the area of Cryptography.
269-277. He also holds a Bachelor of
[14] Biro I, Szabo J, Benczur A, and Siklosi D. Linked Engineering degree in Computer
Latent Dirichlet Allocation in Web Spam Filtering Science and Engineering. His areas of
[A].In Proceedings of the 4th International interest are Cryptography, Algorithms, Data Structures,
Workshop on Adversarial Information Retrieval on and Electronic Commerce. He has many publications in
the Web (AIR Web), Madrid, Spain, 2009. international journals and conferences to his credit.
[15] Perkins A. The classification of search engine spam.
[Link] [Link]/white
papers/spam classification, 2001.
[16] Paulo C, Clotilde L, Pedro S. Symniotic data
mining for personalized spam filtering [C]. In the
Proceedings of the International Conference on
Web Intelligence and Intelligent Agent Technology,
2009, 149-156.
[17] Rasim M A, Ramiz M A, and Saadat A N.
Classification of Textual E-mail spam using Data
Mining Techniques [J]. In the Journal of Applied
Computational Intelligence and Soft Computing,
2011.

Copyright © 2013 MECS I.J. Modern Education and Computer Science, 2013, 12, 31-39