UNIT-II CLASSIFICATION METHODS

Classification – Lazy Learner - K-Nearest Neighbor – diagnosing breast cancer with kNN
algorithm – Probabilistic Learner – Naïve Bayes – filtering mobile phone spam with
naïve bayes algorithm – Divide and Conquer – Decision Trees and Rules – Understanding
decision trees – identifying risky bank loan using C5.0 – Understanding classification
rules –.identifying poisonous mushrooms with rule learners.

2.1 CLASSIFICATION
Classification in machine learning is a supervised learning task where a model is trained
on a set of labelled data to predict the class of new data points. The model learns to
identify the patterns that distinguish between different classes of data and then uses
those patterns to classify new data points.
There are many different types of classification algorithms, each with its own strengths
and weaknesses. Some of the most common classification algorithms include:
 Logistic regression: This is a linear model that can be used for binary
classification problems.
 Support vector machines (SVMs): SVMs are non-linear models that can be used
for both binary and multi-class classification problems.
 Decision trees: Decision trees are a simple and intuitive type of classifier that can
be used for both binary and multi-class classification problems.
 Random forests: Random forests are an ensemble method that combines
multiple decision trees to improve accuracy.
 Naive Bayes: Naive Bayes is a simple probabilistic classifier that can be used for
both binary and multi-class classification problems.
The best algorithm for a particular classification task will depend on the characteristics
of the data and the desired level of accuracy.
Here are some examples of classification tasks that can be solved using machine
learning:
 Spam filtering: This is a binary classification problem where the goal is to classify
emails as either spam or not spam.
 Fraud detection: This is a binary classification problem where the goal is to
identify fraudulent transactions.
 Image classification: This is a multi-class classification problem where the goal is
to classify images into different categories, such as cats, dogs, or cars.
 Text classification: This is a multi-class classification problem where the goal is to
classify text into different categories, such as news articles, product reviews, or
social media posts.

1
Classification is a powerful tool that can be used to solve a wide variety of problems. By
using machine learning algorithms, businesses can automate tasks, improve decision
making, and gain valuable insights into their data.
2.2 LEARNERS IN CLASSIFICATION PROBLEMS
There are two types of learners in machine learning
Eager Learners
Early learning, also known as eager learning, refers to machine learning algorithms that
try to learn a model from the training data as soon as possible. These algorithms focus
on generalizing patterns from the training data during the learning phase. They aim to
find a model that can quickly capture the relationships in the training data and be ready
for predictions without significant delay. Before obtaining a test dataset, eager learners
build a classification model using a training dataset. They spend more time studying and
less time predicting.
Examples of early learning algorithms include decision trees, naive Bayes classifiers, and
some types of neural networks. These algorithms often create explicit models during
training, which can be used directly for predictions without extensive computation
during inference.
Advantages:
 Fast training and inference times.
 Suitable for datasets with smaller sizes and relatively simple patterns.
Disadvantages:
 May overfit if the training data contains noise or outliers.
 Less adaptable to changing data distributions.

Lazy Learners
Lazy learning, also known as instance-based learning or memory-based learning, refers
to machine learning algorithms that postpone learning a model until a query or
prediction is made. Instead of building an explicit model during the training phase, lazy
learning algorithms store the training data instances and their labels. When a prediction
is needed, these algorithms search for similar instances in the training data and make
predictions based on the labels of those similar instances. In other words, It first stores
the training dataset before waiting for the test dataset to arrive. When using a lazy
learner, the classification is carried out using the training dataset's most appropriate
data. Less time is spent on training, but more time is spent on predictions.
The k-nearest neighbors (k-NN) algorithm is a classic example of a lazy learning
algorithm. It doesn't generalize from the training data but rather memorizes it and uses
it directly for making predictions.
Advantages:
 Can adapt to changes in data distribution over time.
 Can handle noisy data and complex relationships.
Disadvantages:
 Slower prediction times, as similarity search can be computationally intensive.

2
  Requires careful selection of similarity metrics and handling of high-dimensional
data.

2.3 K-Nearest Neighbours Algorithm

The k-nearest neighbours algorithm, also known as KNN or k-NN, is a non-parametric,
supervised learning classifier, which uses proximity to make classifications or predictions about
the grouping of an individual data point. K-nearest neighbour (KNN) algorithm can be used for
both classification and regression tasks. The KNN algorithm is based on finding the k-nearest
data points to a given query point in a feature space, and then predicting the output based on
the majority class or average value of those k neighbours.
The basic steps involved in the KNN algorithm are:
1. Choose the number of nearest neighbours (k) and a distance metric.
2. Calculate the distance between the test data point and all the training data points using
the chosen distance metric.
3. Select the k-nearest neighbours based on the shortest distance to the test data point.
4. For classification, determine the majority class among the k-nearest neighbours and
assign that class to the test data point. For regression, calculate the average value of the
target variable for the k-nearest neighbours and assign that value to the test data point.
5. Repeat steps 2-4 for all test data points.

CALCULATING DISTANCE
The first step is to calculate the distance between the new point and each training point. The
choice of the distance metric and the value of k are important hyperparameters in the KNN
algorithm. There are various methods for calculating this distance, of which the most known
methods are — Euclidian, Manhattan (for continuous) and Hamming distance (for categorical).
The most commonly used distance metrics are Euclidean distance and Manhattan distance
Euclidean Distance: Euclidean distance is calculated as the square root of the sum of the
squared differences between a new point (x) and an existing point (y).

3
Manhattan Distance: Manhattan distance is a metric that tells you the distance travelled by a
particular object rather than calculating the difference between two points. This is the distance
between real vectors using the sum of their absolute difference.

Minkowski distance: This distance measure is the generalized form of Euclidean and
Manhattan distance metrics. The parameter, p, in the formula below, allows for the creation of
other distance metrics. Euclidean distance is represented by this formula when p is equal to two,
and Manhattan distance is denoted with p equal to one.

Hamming Distance: Hamming distance is used to compare two binary arrays of data, by
calculating the difference between the bits positions of two strings. It is used to calculate
distance between two new words, that are fixed in length. It is used for categorical variables. If
the value (x) and the value (y) are the same, the distance D will be equal to 0 . Otherwise D=1.

CHOOSING PARAMETERS IN KNN ALGORITHM

Choosing the right parameters for the KNN algorithm is critical to achieving good results.
The main parameter to tune is the value of k, which is the number of neighbours to consider
when making a prediction. The following are some factors to consider when choosing the k
value:
1. Size of the dataset: The larger the dataset, the larger the value of k should be to avoid
overfitting.

4
 2. Nature of the data: The choice of k may depend on the distribution of the data. If the data
is dense, a smaller k value can be used. If the data is sparse, a larger k value may be more
appropriate.
3. Distance metric: The choice of distance metric used to calculate the distance between
data points can also affect the choice of k value. For example, if the data is in a high-
dimensional space, a smaller k value may be more appropriate to avoid the "curse of
dimensionality."
4. Cross-validation: Cross-validation can be used to evaluate the performance of the KNN
algorithm with different k values. A common approach is to use k-fold cross-validation,
where the data is divided into k subsets, and the algorithm is trained and evaluated k
times using different subsets as the validation set.
Choosing the right k value for the KNN algorithm involves considering the size and nature of
the dataset, the distance metric, and using cross-validation to evaluate performance.
PRACTICAL APPLICATIONS OF KNN
K-nearest neighbor (KNN) algorithm has practical applications in various domains such as:
1. Image recognition: KNN can be used for image recognition tasks, such as identifying
handwritten digits or recognizing objects in images. The algorithm can be trained on a
dataset of labeled images and then used to classify new images based on the nearest
neighbours.
2. Recommender systems: KNN can be used in recommender systems to recommend
products or services based on the user's previous behavior or preferences. The
algorithm can find the k nearest neighbours to the user based on their past behavior and
recommend products or services based on the majority choice of those neighbours.
3. Text classification: KNN can be used for text classification tasks, such as sentiment
analysis or spam filtering. The algorithm can be trained on a dataset of labeled text data
and then used to classify new text data based on the nearest neighbours.
4. Anomaly detection: KNN can be used for anomaly detection in data, such as detecting
fraudulent transactions or identifying unusual behavior in sensors. The algorithm can
find the k nearest neighbours to a data point and determine whether it is an outlier
based on the distance from those neighbours.
5. Medical diagnosis: KNN can be used for medical diagnosis tasks, such as identifying
diseases based on patient symptoms or medical records. The algorithm can be trained
on a dataset of labeled medical data and then used to diagnose new patients based on
the nearest neighbours.
Pros of KNN
1. Simple to implement
2. Flexible to feature/distance choices
3. Naturally handles multi-class cases
4. Can do well in practice with enough representative data
Cons of KNN
1. Need to determine the value of parameter K (number of nearest neighbors)

5
 2. Computation cost is quite high because we need to compute the distance of each query
instance to all training samples.
3. Storage of data
4. Must know we have a meaningful distance function.

2.4 Diagnosing Breast Cancer with KNN Algorithm

The Wisconsin Breast Cancer dataset is a classic machine learning dataset that is often used for
binary classification tasks. The dataset contains 569 data points, each of which represents a
breast cancer tumor. Each data point has 9 features, which are:
lump thickness: This is a measure of the amount of clumping of cells in the tumor.
Uniformity of cell size: This is a measure of how similar the size of the cells in the tumor are.
Uniformity of cell shape: This is a measure of how similar the shape of the cells in the tumor are.
Marginal adhesion: This is a measure of how well the cells in the tumor are attached to each
other.
Single epithelial cell size: This is a measure of the size of individual cells in the tumor.
Bare nuclei: This is a measure of how many cells in the tumor have nuclei that are not covered
by cytoplasm.
Bland chromatin: This is a measure of how uniform the chromatin (DNA and protein material) is
in the cells of the tumor.
Normal nuclei: This is a measure of how many cells in the tumor have nuclei that look normal.
Mitoses: This is a measure of how many cells in the tumor are dividing.
The target variable for the dataset is diagnosis: This is a binary variable that indicates whether
the tumor is benign (0) or malignant (1).
SOURCE CODE
> wbcd <- [Link]("C:/Users/Dell/Downloads/archive (9)/[Link]")
> View(wbcd)

> str(wbcd)
'[Link]': 569 obs. of 33 variables:
$ id : int 842302 842517 84300903 84348301 84358402 843786 844359 8445820
2 844981 84501001 ...
$ diagnosis : chr "M" "M" "M" "M" ...
$ radius_mean : num 18 20.6 19.7 11.4 20.3 ...
$ texture_mean : num 10.4 17.8 21.2 20.4 14.3 ...
$ perimeter_mean : num 122.8 132.9 130 77.6 135.1 ...
$ area_mean : num 1001 1326 1203 386 1297 ...
$ smoothness_mean : num 0.1184 0.0847 0.1096 0.1425 0.1003 ...
$ compactness_mean : num 0.2776 0.0786 0.1599 0.2839 0.1328 ...
$ concavity_mean : num 0.3001 0.0869 0.1974 0.2414 0.198 ...
$ concave.points_mean : num 0.1471 0.0702 0.1279 0.1052 0.1043 ...
$ symmetry_mean : num 0.242 0.181 0.207 0.26 0.181 ...
$ fractal_dimension_mean : num 0.0787 0.0567 0.06 0.0974 0.0588 ...
$ radius_se : num 1.095 0.543 0.746 0.496 0.757 ...
$ texture_se : num 0.905 0.734 0.787 1.156 0.781 ...

6
$ perimeter_se : num 8.59 3.4 4.58 3.44 5.44 ...
$ area_se : num 153.4 74.1 94 27.2 94.4 ...
$ smoothness_se : num 0.0064 0.00522 0.00615 0.00911 0.01149 ...
$ compactness_se : num 0.049 0.0131 0.0401 0.0746 0.0246 ...
$ concavity_se : num 0.0537 0.0186 0.0383 0.0566 0.0569 ...
$ concave.points_se : num 0.0159 0.0134 0.0206 0.0187 0.0188 ...
$ symmetry_se : num 0.03 0.0139 0.0225 0.0596 0.0176 ...
$ fractal_dimension_se : num 0.00619 0.00353 0.00457 0.00921 0.00511 ...
$ radius_worst : num 25.4 25 23.6 14.9 22.5 ...
$ texture_worst : num 17.3 23.4 25.5 26.5 16.7 ...
$ perimeter_worst : num 184.6 158.8 152.5 98.9 152.2 ...
$ area_worst : num 2019 1956 1709 568 1575 ...
$ smoothness_worst : num 0.162 0.124 0.144 0.21 0.137 ...
$ compactness_worst : num 0.666 0.187 0.424 0.866 0.205 ...
$ concavity_worst : num 0.712 0.242 0.45 0.687 0.4 ...
$ concave.points_worst : num 0.265 0.186 0.243 0.258 0.163 ...
$ symmetry_worst : num 0.46 0.275 0.361 0.664 0.236 ...
$ fractal_dimension_worst: num 0.1189 0.089 0.0876 0.173 0.0768 ...
$X : logi NA NA NA NA NA NA ...

> wbcd <- wbcd[-1]

> table(wbcd$diagnosis)

B M
357 212

> wbcd$diagnosis<- factor(wbcd$diagnosis, levels = c("B", "M"),labels = c("Benign", "Malignant"

))

> round([Link](table(wbcd$diagnosis)) * 100, digits = 1)

Benign Malignant
62.7 37.3

> wbcd$diagnosis<- factor(wbcd$diagnosis, levels = c("B", "M"),labels = c("Benign", "Malignant"

))

> summary(wbcd[c("radius_mean", "area_mean", "smoothness_mean")])

radius_mean area_mean smoothness_mean
Min. : 6.981 Min. : 143.5 Min. :0.05263
1st Qu.:11.700 1st Qu.: 420.3 1st Qu.:0.08637
Median :13.370 Median : 551.1 Median :0.09587
Mean :14.127 Mean : 654.9 Mean :0.09636
3rd Qu.:15.780 3rd Qu.: 782.7 3rd Qu.:0.10530
Max. :28.110 Max. :2501.0 Max. :0.16340

> normalize <- function(x) {

+ return ((x - min(x)) / (max(x) - min(x)))
+ }

> wbcd_n <- [Link](lapply(wbcd[2:31], normalize))

> summary(wbcd_n$area_mean)
Min. 1st Qu. Median Mean 3rd Qu. Max.
0.0 0.1174 0.1729 0.2169 0.2711 1.0000
7
> wbcd_train <- wbcd_n[1:469, ]
> wbcd_test <- wbcd_n[470:569, ]

> wbcd_train_labels <- wbcd[1:469, 1]

> wbcd_test_labels <- wbcd[470:569, 1]

> [Link]("class")
package ‘class’ successfully unpacked and MD5 sums checked

The downloaded binary packages are in

C:\Users\Dell\AppData\Local\Temp\RtmpsjFVSf\downloaded_packages
> library(class)

> wbcd_test_pred <- knn(train = wbcd_train, test = wbcd_test,cl = wbcd_train_labels, k = 21)

> [Link]("gmodels")
package ‘gmodels’ successfully unpacked and MD5 sums checked

The downloaded binary packages are in

C:\Users\Dell\AppData\Local\Temp\RtmpsjFVSf\downloaded_packages

> library(gmodels)

> CrossTable(x = wbcd_test_labels, y = wbcd_test_pred,[Link]=FALSE)

Cell Contents
|--------------------------|
| N |
| N / Row Total |
| N / Col Total |
| N / Table Total |
|------------------------- |

Total Observations in Table: 100

8
2.5 PROBABLISTIC LEARNER
A probabilistic learner refers to a type of machine learning model that makes predictions or
decisions based on probability distributions. Instead of providing deterministic outputs,
probabilistic learners provide probabilistic outputs, which indicate the likelihood or
probability of different outcomes. Probabilistic learners are often used in situations where
there is uncertainty about the data or the environment. For example, a probabilistic learner
could be used to predict the weather, where the exact outcome is not known in advance.

A probabilistic learner learning is a model that learns from data by assigning probabilities to
different outcomes. This contrasts with a deterministic learner, which simply predicts a
single outcome for each input. They can be used to handle uncertainty, improve accuracy,
and make decisions. However, they can also be more complex and require more data than
deterministic learners.
Some common types of probabilistic learners include:
 Bayesian networks: These models represent uncertainty using a network of nodes and
edges. Each node represents a variable, and the edges represent the relationships
between the variables.
 Hidden Markov models: These models are used to model sequences of data. The hidden
states of the model are not directly observable, but they can be inferred from the
observed data.
 Gaussian processes: These models are used to model continuous data. They assume that
the data is generated from a Gaussian distribution, which is a bell-shaped curve.
Probabilistic learners can be more accurate than deterministic learners in situations where
there is uncertainty. However, they can also be more complex and require more data to train.
Advantages of using probabilistic learners in machine learning:
 They can handle uncertainty: Probabilistic learners can handle uncertainty in the data or
the environment. This makes them more robust to noise and outliers.
 They can be more accurate: Probabilistic learners can often be more accurate than
deterministic learners, especially in situations where there is a lot of uncertainty.
 They can be used for decision making: Probabilistic learners can be used to make
decisions about the best course of action, given the uncertainty in the data.
Disadvantages of using probabilistic learners in machine learning:
 They can be more complex: Probabilistic learners can be more complex than
deterministic learners. This can make them more difficult to train and understand.
 They require more data: Probabilistic learners require more data to train than
deterministic learners. This is because they need to learn the probability distributions of
the data.
 They can be less interpretable: The predictions of probabilistic learners can be less
interpretable than the predictions of deterministic learners. This is because they are
based on probability distributions, which can be difficult to understand.

2.6 NAÏVE BAYES ALGORITHM

The Naïve Bayes classifier is a popular supervised machine learning algorithm used for
classification tasks such as text classification. It belongs to the family of generative learning
algorithms, which means that it models the distribution of inputs for a given class or
9
 category. It is a classification technique based on Bayes’ Theorem with an independence
assumption among predictors. This approach is based on the assumption that the features of
the input data are conditionally independent given the class, allowing the algorithm to make
predictions quickly and accurately.

In statistics, naive Bayes classifiers are considered as simple probabilistic classifiers that
apply Bayes’ theorem. This theorem is based on the probability of a hypothesis, given the
data and some prior knowledge. The naive Bayes classifier assumes that all features in the
input data are independent of each other, which is often not true in real-world scenarios.
However, despite this simplifying assumption, the naive Bayes classifier is widely used
because of its efficiency and good performance in many real-world applications.

Naive Bayes classifiers are among the simplest Bayesian network models, yet they can
achieve high accuracy levels when coupled with kernel density estimation. This technique
involves using a kernel function to estimate the probability density function of the input
data, allowing the classifier to improve its performance in complex scenarios where the data
distribution is not well-defined. As a result, the naive Bayes classifier is a powerful tool in
machine learning, particularly in text classification, spam filtering, and sentiment analysis,
among others.

For example, a fruit may be considered to be an apple if it is red, round, and about 3 inches
in diameter. Even if these features depend on each other or upon the existence of the other
features, all of these properties independently contribute to the probability that this fruit is
an apple and that is why it is known as ‘Naive’.

An NB model is easy to build and particularly useful for very large data sets. Along with
simplicity, Naive Bayes is known to outperform even highly sophisticated classification
methods.

 P(H|E) denotes how event H happens when event E takes place.

 P(E|H) represents how often event E happens when event H takes place first.
 P(H) represents the probability of event X happening on its own.
 P(E) represents the probability of event Y happening on its own.

Conditional Probability
Conditional probability is a subset of probability. It reduces the probability of becoming
dependent on a single event. You can compute the conditional probability for two or more
occurrences.
When you take events X and Y, the conditional probability of event Y is defined as the probability
that the event occurs when event X is already over. It is written as P(Y|X). The mathematical
formula for this is as follows:
P(Y|A) = P(X and Y) /P(X)
Bayesian Probability

10
Bayesian Probability allows to calculate the conditional probabilities. It enables to use of partial
knowledge for calculating the probability of the occurrence of a specific event. This algorithm is
used for developing models for prediction and classification problems like Naive Bayes.
The Bayesian Rule is used in probability theory for computing - conditional probabilities. What
is important is that you cannot discover just how the evidence will impact the probability of an
event occurring, but you can find the exact probability
How Do Naive Bayes Algorithms Work?
Let’s understand it using an example. Below is the training data set of weather and
corresponding target variable ‘Play’ (suggesting possibilities of playing). Now, we need to
classify whether players will play or not based on weather condition. Let’s follow the below
steps to perform it.
1. Convert the data set into a frequency table
In this first step data set is converted into a frequency table
2. Create Likelihood table by finding the probabilities
Create Likelihood table by finding the probabilities like Overcast probability = 0.29 and
probability of playing is 0.64.

3. Use Naive Bayesian equation to calculate the posterior probability

Now, use Naive Bayesian equation to calculate the posterior probability for each class. The class
with the highest posterior probability is the outcome of the prediction.
Problem: Players will play if the weather is sunny. Is this statement correct?
We can solve it using the above-discussed method of posterior probability.
P(Yes | Sunny) = P( Sunny | Yes) * P(Yes) / P (Sunny)
Here P( Sunny | Yes) * P(Yes) is in the numerator, and P (Sunny) is in the denominator.
Here we have P (Sunny |Yes) = 3/9 = 0.33, P(Sunny) = 5/14 = 0.36, P( Yes)= 9/14 = 0.64
Now, P (Yes | Sunny) = 0.33 * 0.64 / 0.36 = 0.60, which has higher probability.
The Naive Bayes uses a similar method to predict the probability of different class based on
various attributes. This algorithm is mostly used in text classification (nlp) and with problems
having multiple classes.
Pros:

11
  It is easy and fast to predict class of test data set. It also perform well in multi class
prediction
 When assumption of independence holds, the classifier performs better compared to
other machine learning models like logistic regression or decision tree, and requires less
training data.
 It perform well in case of categorical input variables compared to numerical variable(s).
For numerical variable, normal distribution is assumed (bell curve, which is a strong
assumption).
Cons:
 If categorical variable has a category (in test data set), which was not observed in
training data set, then model will assign a 0 (zero) probability and will be unable to
make a prediction. This is often known as “Zero Frequency”. To solve this, we can use the
smoothing technique. One of the simplest smoothing techniques is called Laplace
estimation.
 On the other side, Naive Bayes is also known as a bad estimator, so the probability
outputs from predict_proba are not to be taken too seriously.
 Another limitation of this algorithm is the assumption of independent predictors. In real
life, it is almost impossible that we get a set of predictors which are completely
independent.
Applications of Naive Bayes Algorithms
 Real-time Prediction: Naive Bayesian classifier is an eager learning classifier and it is
super fast. Thus, it could be used for making predictions in real time.
 Multi-class Prediction: This algorithm is also well known for multi class prediction
feature. Here we can predict the probability of multiple classes of target variable.
 Text classification/ Spam Filtering/ Sentiment Analysis: Naive Bayesian classifiers
mostly used in text classification (due to better result in multi class problems and
independence rule) have higher success rate as compared to other algorithms. As a
result, it is widely used in Spam filtering (identify spam e-mail) and Sentiment
Analysis (in social media analysis, to identify positive and negative customer sentiments)
 Recommendation System: Naive Bayes Classifier and Collaborative Filtering together
builds a Recommendation System that uses machine learning and data mining
techniques to filter unseen information and predict whether a user would like a given
resource or not.

2.7 Program to implement Filtering Mobile phone spam using Naïve Bayes
> spam <- [Link]("C:/Users/Dell/Downloads/archive (10)/[Link]")
> View(spam)
> str(spam)
'[Link]': 5572 obs. of 5 variables:
$ v1 : chr "ham" "ham" "spam" "ham" ...
$ v2 : chr "Go until jurong point, crazy.. Available only in bugis n great world la e buffet... Cine t
12
here got amore wat..." "Ok lar... Joking wif u oni..." "Free entry in 2 a wkly comp to win FA Cup fin
al tkts 21st May 2005. Text FA to 87121 to receive entry question(std txt rate)T&C "U dun say s
o early hor... U c already then say..." ...
$ X : chr "" "" "" "" ...
$ X.1: chr "" "" "" "" ...
$ X.2: chr "" "" "" "" ...
> spam<-spam[-3:-5]
> spam$v1 <- factor(spam$v1)
> library(stringr)
> spam$v2 = str_replace_all(spam$v2, "[^[:alnum:]]", " ") %>% str_replace_all(.,"[ ]+", " ")
> [Link]("NLP")
package ‘NLP’ successfully unpacked and MD5 sums checked
The downloaded binary packages are in
C:\Users\Dell\AppData\Local\Temp\RtmpMtIB2u\downloaded_packages
> library(NLP)
> library(tm)
> spam_corpus<- VCorpus(VectorSource(spam$v2))
> print(spam_corpus)
<<VCorpus>>
Metadata: corpus specific: 0, document level (indexed): 0
Content: documents: 5572
> print([Link](spam_corpus[[6]]))
[1] "FreeMsg Hey there darling it s been 3 week s now and no word back I d like some fun you u
p for it still Tb ok XxX std chgs to send 1 50 to rcv"
> spam_dtm<- DocumentTermMatrix(spam_corpus, control = list (tolower=TRUE,removeNumb
ers=TRUE,stopwords=TRUE,removePunctuations=TRUE,stemming=TRUE))
> [Link]("SnowballC")
package ‘SnowballC’ successfully unpacked and MD5 sums checked
The downloaded binary packages are in
C:\Users\Dell\AppData\Local\Temp\RtmpMtIB2u\downloaded_packages
> library(SnowballC)
> spam_dtm<- DocumentTermMatrix(spam_corpus, control = list (tolower=TRUE,removeNumb
ers=TRUE,stopwords=TRUE,removePunctuations=TRUE,stemming=TRUE))

13
> x_train<- spam_dtm[1:4169, ]
> x_test<- spam_dtm[4170:5572, ]
> y_train<- spam[1:4169, ]$v1
> y_test<- spam[4170:5572, ]$v1
> spam_freq_word_train<- findFreqTerms(x_train, 5)
> spam_freq_word_test<- findFreqTerms(x_test, 5)
> x_train<- x_train[ , spam_freq_word_train]
> x_test<- x_test[ , spam_freq_word_test]
> convert_counts<- function(x) {x <- ifelse(x >0, "Yes", "No")}
> x_train<- apply(x_train, MARGIN = 2,convert_counts)
> x_test<- apply(x_test, MARGIN = 2,convert_counts)
> library(e1071)
> model <- naiveBayes(x_train, y_train,laplace=1)
> y_pred<- predict(model, x_test)
> cm = table(y_pred, y_test)
> print(cm)
> acc = sum(diag(cm))/sum(cm)
> print(paste("Accuracy: ",acc*100,"%"))
[1] "Accuracy: 98.1468282252316 %"
> library(gmodels)
> CrossTable(x = y_test, y = y_pred,[Link]=FALSE

14
2.8 Decision Trees
A decision tree is a non-parametric tree based model that takes features as internal nodes and
decision criteria i.e. the outcome of those features as decision nodes and terminal nodes
representing the outcome of the dataset.
A decision tree is a supervised machine learning algorithm that can be utilized for classification
and regression tasks. Tree models where the target variable can take a discrete set of values
are called classification trees. Decision trees where the target variable can take continuous
values (typically real numbers) are called regression trees. Classification And Regression Tree
(CART) is general term for this.
The decision tree is recursively split into smaller sub trees until the terminal node contains a
homogeneous sub class or any of the stopping criteria is met.
Stopping criteria for Decision Tree:
The splitting of the decision tree is done until one of the following
1. The resultant terminal node is homogenous subset or pure class.
2. All the features have been utilized and no more features is left for splitting.
3. The decision tree has hit a pre-defined level of height.
Terminology related to Decision Trees

1. Root Node: It represents the entire population or sample and this further gets divided
into two or more homogeneous sets.
2. Splitting: It is a process of dividing a node into two or more sub-nodes.
3. Decision Node: When a sub-node splits into further sub-nodes, then it is called the
decision node.
4. Leaf / Terminal Node: Nodes do not split is called Leaf or Terminal node.

15
 5. Pruning: When we remove sub-nodes of a decision node, this process is called pruning.
You can say the opposite process of splitting.
6. Branch / Sub-Tree: A subsection of the entire tree is called branch or sub-tree.
7. Parent and Child Node: A node, which is divided into sub-nodes is called a parent node
of sub-nodes whereas sub-nodes are the child of a parent node.

Pruning
Pruning is another method that can help us avoid overfitting. It helps in improving the
performance of the tree by cutting the nodes or sub-nodes which are not significant.
Additionally, it removes the branches which have very low importance.
There are mainly 2 ways for pruning:
 Pre-pruning – we can stop growing the tree earlier, which means we can
prune/remove/cut a node if it has low importance while growing the tree.
 Post-pruning – once our tree is built to its depth, we can start pruning the nodes
based on their significance.
Working of Decision Trees
1. It begins with the original set S as the root node.
2. On each iteration of the algorithm, it iterates through the very unused attribute of the set
S and calculates Entropy(H) and Information gain(IG) of this attribute.
3. It then selects the attribute which has the smallest Entropy or Largest Information gain.
4. The set S is then split by the selected attribute to produce a subset of the data.
5. The algorithm continues to recur on each subset, considering only attributes never
selected before.

Attribute Selection Measure

If the dataset consists of N attributes then deciding which attribute to place at the root or at
different levels of the tree as internal nodes is a complicated step. By just randomly selecting any

16
node to be the root can’t solve the issue. If we follow a random approach, it may give us bad
results with low accuracy. Some of the popular attribute selection measures are
Entropy,
Information gain,
Gini index,
Gain Ratio,
Reduction in Variance
Chi-Square
Entropy
Entropy is a measure of the randomness in the information being processed. Entropy varies
between 0 to 1. The higher the entropy, the harder it is to draw any conclusions from that
information. A branch with an entropy of zero is a leaf node and A brach with entropy
more than zero needs further splitting.

Information Gain
Information gain or IG is a statistical property that measures how well a given attribute
separates the training examples according to their target classification. Constructing a decision
tree is all about finding an attribute that returns the highest information gain and the smallest
entropy.

Example of Decision Tree

17
Let’s understand decision trees with the help of an example:

Decision trees are upside down which means the root is at the top and then this root is split into
various several nodes. Decision trees are nothing but a bunch of if-else statements in layman
terms. It checks if the condition is true and if it is then it goes to the next node attached to that
decision.

18
19
20
21
2.9 Identifying Risky Bank Loans using Decision Trees
>credit <- [Link]("C:/Users/Dell/Downloads/[Link]")

> View(credit)

> str(credit)
'[Link]': 1000 obs. of 21 variables:
$ checking_balance : chr "< 0 DM" "1 - 200 DM" "unknown" "< 0 DM" ...
$ months_loan_duration: int 6 48 12 42 24 36 24 36 12 30 ...
$ credit_history : chr "critical" "repaid" "critical" "repaid" ...
$ purpose : chr "radio/tv" "radio/tv" "education" "furniture" ...
$ amount : int 1169 5951 2096 7882 4870 9055 2835 6948 3059 5234 ..
.
$ savings_balance : chr "unknown" "< 100 DM" "< 100 DM" "< 100 DM" ...
$ employment_length : chr "> 7 yrs" "1 - 4 yrs" "4 - 7 yrs" "4 - 7 yrs" ...
$ installment_rate : int 4 2 2 2 3 2 3 2 2 4 ...
$ personal_status : chr "single male" "female" "single male" "single male" ...
$ other_debtors : chr "none" "none" "none" "guarantor" ...
$ residence_history : int 4 2 3 4 4 4 4 2 4 2 ...
$ property : chr "real estate" "real estate" "real estate" "building society savi
ngs" ...
$ age : int 67 22 49 45 53 35 53 35 61 28 ...
$ installment_plan : chr "none" "none" "none" "none" ...
$ housing : chr "own" "own" "own" "for free" ...
$ existing_credits : int 2 1 1 1 2 1 1 1 1 2 ...
$ default : int 1 2 1 1 2 1 1 1 1 2 ...
$ dependents : int 1 1 2 2 2 2 1 1 1 1 ...
$ telephone : chr "yes" "none" "none" "none" ...
$ foreign_worker : chr "yes" "yes" "yes" "yes" ...
$ job : chr "skilled employee" "skilled employee" "unskilled resident" "ski
lled employee" ...

> table(credit$checking_balance)

< 0 DM > 200 DM 1 - 200 DM unknown

274 63 269 394

> table(credit$savings_balance)

< 100 DM > 1000 DM 101 - 500 DM 501 - 1000 DM unknown

603 48 103 63 183

> credit$default = [Link](credit$default)

> levels(credit$default)[levels(credit$default) == "1"] <- "No"

> levels(credit$default)[levels(credit$default) == "2"] <- "Yes"

> str(credit$default)
Factor w/ 2 levels "No","Yes": 1 2 1 1 2 1 1 1 1 2 ...

> credit$checking_balance <- [Link](credit$checking_balance)

> levels(credit$checking_balance)[levels(credit$checking_balance) == "1"] <- "< 0 DM

"

22
> levels(credit$checking_balance)[levels(credit$checking_balance) == "2"] <- " > 200
DM"
> levels(credit$checking_balance)[levels(credit$checking_balance) == "3"] <- "1 - 200
DM"
> levels(credit$checking_balance)[levels(credit$checking_balance) == "4"] <- "unkno
wn"

> str(credit$checking_balance)
Factor w/ 4 levels "< 0 DM","> 200 DM",..: 1 3 4 1 1 4 4 3 4 3 ...

> summary(credit$months_loan_duration) ##Summary of the loan duration

Min. 1st Qu. Median Mean 3rd Qu. Max.
4.0 12.0 18.0 20.9 24.0 72.0

> summary(credit$amount) ##Summary of the amount of loan

Min. 1st Qu. Median Mean 3rd Qu. Max.
250 1366 2320 3271 3972 18424

> table(credit$default)

No Yes
700 300

> table(credit$checking_balance)

< 0 DM > 200 DM 1 - 200 DM unknown

274 63 269 394

> [Link](12345)
> credit_rand <- credit[order(runif(1000)), ]

> summary(credit$amount)
Min. 1st Qu. Median Mean 3rd Qu. Max.
250 1366 2320 3271 3972 18424
> summary(credit_rand$amount)
Min. 1st Qu. Median Mean 3rd Qu. Max.
250 1366 2320 3271 3972 18424

> head(credit$amount)
[1] 1169 5951 2096 7882 4870 9055

> head(credit_rand$amount)
[1] 1199 2576 1103 4020 1501 1568

> credit_train <- credit_rand[1:900, ]

> credit_test <- credit_rand[901:1000, ]

> [Link](table(credit_train$default))
No Yes
0.7022222 0.2977778

> [Link](table(credit_test$default))
No Yes
0.68 0.32
> library(C50)
> credit_model <- C5.0(default ~ ., credit_train)
23
> credit_model

Call:
[Link](formula = default ~ ., data = credit_train)

Classification Tree
Number of samples: 900
Number of predictors: 20

Tree size: 57

Non-standard options: attempt to group attributes

> summary(credit_model)

Call:
[Link](formula = default ~ ., data = credit_train)

C5.0 [Release 2.07 GPL Edition] Tue Aug 29 17:44:42 2023

-------------------------------

Class specified by attribute `outcome'

Read 900 cases (21 attributes) from [Link]

Decision tree:

checking_balance = unknown: No (358/44)

checking_balance in {< 0 DM,> 200 DM,1 - 200 DM}:
:...foreign_worker = no:
:...installment_plan in {none,stores}: No (17/1)
: installment_plan = bank:
: :...residence_history <= 3: Yes (2)
: residence_history > 3: No (2)
foreign_worker = yes:
:...credit_history in {fully repaid,
: fully repaid this bank}: Yes (61/20)
credit_history in {critical,repaid,delayed}:
:...months_loan_duration <= 11: No (76/13)
months_loan_duration > 11:
:...savings_balance = > 1000 DM: No (13)
savings_balance in {< 100 DM,101 - 500 DM,501 - 1000 DM,
: unknown}:
:...checking_balance = > 200 DM:
:...dependents > 1: Yes (3)
: dependents <= 1:
: :...credit_history in {repaid,delayed}: No (23/3)
: credit_history = critical:
: :...amount <= 2337: Yes (3)
: amount > 2337: No (6)
checking_balance = < 0 DM:
:...other_debtors = guarantor:
: :...credit_history = critical: Yes (1)
: : credit_history in {repaid,delayed}: No (11/1)
: other_debtors in {none,co-applicant}:
24
 : :...job = mangement self-employed: No (26/6)
: job in {unskilled resident,skilled employee,
: : unemployed non-resident}:
: :...purpose in {radio/tv,others,repairs,
: : domestic appliances,
: : retraining}: Yes (33/10)
: purpose = education: [S1]
: purpose = business:
: :...job in {unskilled resident,
: : : unemployed non-resident}: No (3)
: : job = skilled employee: Yes (3)
: purpose = car (new): [S2]
: purpose = car (used):
: :...amount > 6229: Yes (5)
: : amount <= 6229: [S3]
: purpose = furniture:
: :...months_loan_duration > 27: Yes (9/1)
: months_loan_duration <= 27: [S4]
checking_balance = 1 - 200 DM:
:...savings_balance = unknown: No (34/6)
savings_balance in {< 100 DM,101 - 500 DM,
: 501 - 1000 DM}:
:...months_loan_duration > 45: Yes (11/1)
months_loan_duration <= 45:
:...installment_plan = stores:
:...age <= 35: Yes (4)
: age > 35: No (2)
installment_plan = bank:
:...residence_history <= 1: No (3)
: residence_history > 1:
: :...existing_credits <= 1: Yes (5)
: existing_credits > 1:
: :...installment_rate > 2: Yes (3)
: installment_rate <= 2: [S5]
installment_plan = none:
:...other_debtors = guarantor: No (7/1)
other_debtors = co-applicant: Yes (3/1)
other_debtors = none:
:...employment_length = 4 - 7 yrs:
:...age <= 41: No (16)
: age > 41: Yes (3/1)
employment_length in {> 7 yrs,
: 1 - 4 yrs,
: 0 - 1 yrs,
: unemployed}:
:...amount > 7980: Yes (7)
amount <= 7980:
:...amount > 4746: No (10)
amount <= 4746: [S6]

SubTree [S1]

savings_balance in {< 100 DM,101 - 500 DM,501 - 1000 DM}: Yes (6)
savings_balance = unknown: No (2)

SubTree [S2]
25
savings_balance = 101 - 500 DM: No (1)
savings_balance in {501 - 1000 DM,unknown}: Yes (4)
savings_balance = < 100 DM:
:...personal_status in {single male,female,divorced male}: Yes (29/6)
personal_status = married male: No (2)

SubTree [S3]

job = unskilled resident: Yes (1)

job in {skilled employee,unemployed non-resident}: No (8/1)

SubTree [S4]

employment_length in {> 7 yrs,4 - 7 yrs}: No (7/1)

employment_length = unemployed: Yes (2)
employment_length = 0 - 1 yrs:
:...job = unskilled resident: Yes (1)
: job in {skilled employee,unemployed non-resident}: No (4)
employment_length = 1 - 4 yrs:
:...property in {building society savings,unknown/none}: No (5)
property in {other,real estate}:
:...residence_history <= 2: No (4/1)
residence_history > 2: Yes (5)

SubTree [S5]

other_debtors in {none,guarantor}: No (3)

other_debtors = co-applicant: Yes (1)

SubTree [S6]

housing = for free: No (2)

housing = rent:
:...credit_history = critical: No (1)
: credit_history in {repaid,delayed}: Yes (10/2)
housing = own:
:...savings_balance = 101 - 500 DM: No (6)
savings_balance in {< 100 DM,501 - 1000 DM}:
:...residence_history <= 1: No (8/1)
residence_history > 1:
:...installment_rate <= 1: No (2)
installment_rate > 1:
:...employment_length in {> 7 yrs,unemployed}: No (13/6)
employment_length in {1 - 4 yrs,0 - 1 yrs}: Yes (10)

Evaluation on training data (900 cases):

Decision Tree
----------------
Size Errors

57 127(14.1%) <<

(a) (b) <-classified as

26
 ---- ----
590 42 (a): class No
85 183 (b): class Yes

Attribute usage:

100.00% checking_balance
60.22% foreign_worker
57.89% credit_history
51.11% months_loan_duration
42.67% savings_balance
30.44% other_debtors
17.78% job
15.56% installment_plan
14.89% purpose
12.89% employment_length
10.22% amount
6.78% residence_history
5.78% housing
3.89% dependents
3.56% installment_rate
3.44% personal_status
2.78% age
1.56% property
1.33% existing_credits

Time: 0.0 secs

> credit_pred <- predict(credit_model, credit_test)

> library(gmodels)
> CrossTable(credit_test$default, credit_pred,
+ [Link] = FALSE, prop.c = FALSE, prop.r = FALSE,
+ dnn = c('actual default', 'predicted default'))

27
2.10 Understanding Classification Rules
Classification rules are a way to represent knowledge in the form of logical if-else
statements that assign classes to unlabeled examples. These rules consist of an
antecedent (conditions) and a consequent (class assignment) and describe situations
like "if this happens, then that happens." Rule learners are often used similarly to
decision tree learners and have applications such as identifying hardware failure
conditions, customer segmentation, and predicting stock market price changes.
Rule learners have advantages over decision trees in that they are more straightforward
to interpret, as rules are presented as propositions. Additionally, rule learners can result
in simpler and more direct knowledge representations compared to decision trees.
Rule learners are particularly suitable for problems with primarily nominal features and
excel at identifying rare events, even if these events depend on specific interactions
among feature values.
2.11 1RIPPER ALGORITHM
The Ripper algorithm, also known as IREP (Incremental Reduced Error Pruning), is a machine
learning algorithm used for constructing decision trees. It was designed to improve upon the
C4.5 algorithm by addressing some of its limitations and providing a more accurate and compact
representation of the decision tree.
Here are the key features and steps of the Ripper algorithm in decision tree construction:

 Rule-Based Approach: Ripper primarily focuses on creating a set of rules as its final model
rather than a traditional tree structure. Each rule is essentially an if-else statement that
makes a classification decision.

28
  Incremental Learning: Ripper builds the decision tree incrementally, starting with an
empty set of rules and adding rules one by one. At each step, it selects the most
informative rule that reduces the classification error on the training data.
 Rule Generation: The algorithm generates candidate rules by considering various
attribute-value combinations. These candidate rules are evaluated based on their ability to
classify instances accurately.
 Pruning: After generating a rule, Ripper employs pruning techniques to remove conditions
from the rule that do not significantly improve classification accuracy. This helps in
simplifying the rule while maintaining its predictive power.
 Rule Set Optimization: Ripper aims to find a set of rules that collectively provide accurate
and comprehensive classification. It uses a technique called "beam search" to explore the
space of possible rules efficiently.
 Handling Continuous Attributes: Unlike some other decision tree algorithms, Ripper can
handle both nominal and continuous attributes. It discretizes continuous attributes when
necessary to create rules.
 Evaluation: Ripper uses metrics like accuracy, error rate, or other criteria to evaluate the
quality of rules and their impact on classification accuracy.
 Stop Criteria: The algorithm stops when a predefined stopping criterion is met, such as
reaching a certain level of accuracy on the training data or when further rule generation is
unlikely to improve accuracy.
Ripper is particularly useful for datasets with mixed data types, including both categorical and
continuous attributes. It is known for producing concise and interpretable rule-based models,
making it a popular choice in applications where human interpretability of the model is
essential.
In summary, the Ripper algorithm is an incremental rule-based approach to decision tree
construction that aims to create a compact set of rules for accurate classification while
addressing some limitations of traditional decision tree algorithms like C4.5.

2.12 Identifying Poisonous Mushrooms With Rule Learners.

mushrooms<-[Link]("C:/Users/Lenovo/Downloads/archive/[Link]",
stringsAsFactors=TRUE)
> View(mushrooms)
> str(mushrooms)
'[Link]': 8124 obs. of 23 variables:
$ class : Factor w/ 2 levels "e","p": 2 1 1 2 1 1 1 1 2 1 ...
$ [Link] : Factor w/ 6 levels "b","c","f","k",..: 6 6 1 6 6 6 1 1 6 1 ...
$ [Link] : Factor w/ 4 levels "f","g","s","y": 3 3 3 4 3 4 3 4 4 3 ...
$ [Link] : Factor w/ 10 levels "b","c","e","g",..: 5 10 9 9 4 10 9 9 9 10 ...
$ bruises : Factor w/ 2 levels "f","t": 2 2 2 2 1 2 2 2 2 2 ...
$ odor : Factor w/ 9 levels "a","c","f","l",..: 7 1 4 7 6 1 1 4 7 1 ...
$ [Link] : Factor w/ 2 levels "a","f": 2 2 2 2 2 2 2 2 2 2 ...
$ [Link] : Factor w/ 2 levels "c","w": 1 1 1 1 2 1 1 1 1 1 ...
$ [Link] : Factor w/ 2 levels "b","n": 2 1 1 2 1 1 1 1 2 1 ...

29
$ [Link] : Factor w/ 12 levels "b","e","g","h",..: 5 5 6 6 5 6 3 6 8 3 ...
$ [Link] : Factor w/ 2 levels "e","t": 1 1 1 1 2 1 1 1 1 1 ...
$ [Link] : Factor w/ 5 levels "?","b","c","e",..: 4 3 3 4 4 3 3 3 4 3 ...
$ [Link]: Factor w/ 4 levels "f","k","s","y": 3 3 3 3 3 3 3 3 3 3 ...
$ [Link]: Factor w/ 4 levels "f","k","s","y": 3 3 3 3 3 3 3 3 3 3 ...
$ [Link] : Factor w/ 9 levels "b","c","e","g",..: 8 8 8 8 8 8 8 8 8 8 ...
$ [Link] : Factor w/ 9 levels "b","c","e","g",..: 8 8 8 8 8 8 8 8 8 8 ...
$ [Link] : Factor w/ 1 level "p": 1 1 1 1 1 1 1 1 1 1 ...
$ [Link] : Factor w/ 4 levels "n","o","w","y": 3 3 3 3 3 3 3 3 3 3 ...
$ [Link] : Factor w/ 3 levels "n","o","t": 2 2 2 2 2 2 2 2 2 2 ...
$ [Link] : Factor w/ 5 levels "e","f","l","n",..: 5 5 5 5 1 5 5 5 5 5 ...
$ [Link] : Factor w/ 9 levels "b","h","k","n",..: 3 4 4 3 4 3 3 4 3 3 ...
$ population : Factor w/ 6 levels "a","c","n","s",..: 4 3 3 4 1 3 3 4 5 4 ...
$ habitat : Factor w/ 7 levels "d","g","l","m",..: 6 2 4 6 2 2 4 4 2 4 ...

> mushrooms$[Link] <- NULL

> table(mushrooms$class)
e p #NOTE: E—Edible and P – Poisonous Mushrooms#
4208 3916
> [Link]("RWeka")
package ‘RWekajars’ successfully unpacked and MD5 sums checked
package ‘rJava’ successfully unpacked and MD5 sums checked
package ‘RWeka’ successfully unpacked and MD5 sums checked
The downloaded binary packages are in
C:\Users\Lenovo\AppData\Local\Temp\RtmpwXjlXs\downloaded_packages
> library(RWeka)
> mushroom_1R <- OneR(class ~., data = mushrooms)
> mushroom_1R
odor:
a -> e
c -> p
f -> p
l -> e

30
 m -> p
n -> e
p -> p
s -> p
y -> p
(8004/8124 instances correct)
OUTPUT

31