PS0002 Introduction to DS & AI LMXiang

Notes for Tutorial 8 - ML for classification: kNN

Besides logistic regression, an alternative method for classification covered in Lecture 06 is
the k-nearest neighbors (kNN). In this tutorial, we demonstrate

• the idea of the kNN for classification

• application and implementation of the kNN
• discuss comparison between kNN and logistic regression.

Unlike logistic regression, the kNN method has no model formation, and thus no we cannot
interpret the effects of features in the corresponding classifier.

Packages “mlbench", “dplyr” and “class" are required in this tutorial.

1. K-nearest neighbors for classification

KNN was introduced as a nonparametric method in the lecture 05 for prediction. It can also
be applied for classification. To classify a new input x, the distance from x to every point in
the training set is measured. The k smallest distances are identified, and the most represented
class by these k nearest neighbours is considered the output class label of x.

In particular, it can be illustrated as follows:

• Suppose we have two classes of data:

class A (squares) and class B (triangles)
• We want to assign the new input data
point (circle) to one of the two classes by
using the kNN algorithm.
• If k = 3, the nearest neighbors include 2
squares and 1 triangle. So, the new data
point to class A.

Different k values may lead to different classification results. In practice, we need to choose
an optimal value of k when implementing the kNN method.

• Pros
o Nonparametric approach- no distribution assumptions on data
o Simple algorithm — to explain and understand/interpret
• Cons:
o Computationally expensive — it stores all of the training data
o Sensitive to irrelevant features and the scale of the data (affect distance)

1
 PS0002 Introduction to DS & AI LMXiang

2. Implementation of kNN for classification

Similar to logistic regression, the following steps are required for performing the KNN:

i) Preparing and exploring the data.

o Understanding data structure
o Selecting variables (features) of interest
o Data normalization
o Splitting training and test data sets
ii) Training a model on data
iii) Evaluating the model performance.
iv) Improve the performance of model.

Function knn() in package “class” is used to perform kNN classification, with syntax below:
knn(train, test, cl, k = 1, …)

where the arguments “train” and “test” are matrices or dataframes of predictors in the
training and test sets (excluding the outcome variable). “cl” is the factor of true classifications
(i.e., true outcome variable) of training set. “k” is the number of neighbors considered
(default value=1).

• Example 1: revisit BreastCancer data

BreastCancer dataset available in package “mlbench” was analysed in Tutorial 7 using logistic
regression. Now let’s perform kNN to classifying the patients.

data(BreastCancer,package="mlbench")
bc <-BreastCancer[[Link](BreastCancer),]
bc[,2:4]<- sapply(bc[,2:4], [Link])
bc<-bc%>%mutate(y=factor(ifelse(Class=="malignant", 1,0)))%>%
+ select([Link]:[Link], y)
#Normalize numeric variables
nor <-function(x) { (x -min(x))/(max(x)-min(x)) }
bc[,1:3] <- sapply(bc[,1:3], nor)
#split data
[Link](100)
[Link] <- sample(1: nrow(bc), size=nrow(bc)*0.8)
[Link] <-bc[[Link], ]
[Link] <- bc[-[Link], ]
#kNN classification
library(class)
[Link](101)
knn1<-knn([Link][,1:3], [Link][,1:3], cl=[Link]$y, k=2)
mean(knn1 ==[Link]$y)
[1] 0.9635036

table(knn1,actual=[Link]$y)
actual
knn1 0 1
0 84 3
1 2 48

Using kNN classification, 96% of patients in the test data set are correctly classified. In
comparison with the accuracy 94% of the logistic regression classification, both methods

2
 PS0002 Introduction to DS & AI LMXiang

perform very well in classification and kNN produces slightly better accuracy in this
example,

• Example 2: loan application

A data scientist in a bank was given a bank credit dataset and asked by his/her manager the
following questions: Should the bank give a loan to an individual? Would an individual
default on his/her loan? Is that person closer in characteristics to people who defaulted or did
not default on their loans?
Study aim: In general, to minimize risk and maximize profit on behalf of the bank.
Specifically, to study the dataset and build a ML model that predicts whether an applicant’s
loan can be approved or not based on his/her socio-economic profile.
Data: The bank credit dataset contains information about 1000 of applicants, including 21
variables (7 numeric), such as their account balance, credit amount, age, occupation, loan
records, etc. By using this data, we are going to predict whether or not to approve the loan of
an applicant, in other words, to classify an applicant into approval or disapproval class.

gc <-[Link]("[Link]", header = TRUE, sep=",")

dim(gc)
[1] 1000 21
str(gc)
'[Link]': 1000 obs. of 21 variables:
$ Default : int 0 1 0 0 1 0 0 0 0 1 ...
$ checkingstatus1: Factor w/ 4 levels "A11","A12","A13",..: 1 2 4 1 1 ...
$ duration : int 6 48 12 42 24 36 24 36 12 30 ...
$ history : Factor w/ 5 levels "A30","A31","A32",..: 5 3 5 3 4 ...
$ purpose : Factor w/ 10 levels "A40","A41","A410",..: 5 5 8 4 ...
$ amount : int 1169 5951 2096 7882 4870 9055 2835 6948 5234 ...
$ savings : Factor w/ 5 levels "A61","A62","A63",..: 5 1 1 1 1 ...
$ employ : Factor w/ 5 levels "A71","A72","A73",..: 5 3 4 4 3 ...
$ installment : int 4 2 2 2 3 2 3 2 2 4 ...
$ status : Factor w/ 4 levels "A91","A92","A93",..: 3 2 3 3 3 ...
$ others : Factor w/ 3 levels "A101","A102",..: 1 1 1 3 1 1 1 ...
$ residence : int 4 2 3 4 4 4 4 2 4 2 ...
$ property : Factor w/ 4 levels "A121","A122",..: 1 1 1 2 4 4 2 ...
$ age : int 67 22 49 45 53 35 53 35 61 28 ...
$ otherplans : Factor w/ 3 levels "A141","A142",..: 3 3 3 3 3 3 3 ...
$ housing : Factor w/ 3 levels "A151","A152",..: 2 2 2 3 3 3 2 ...
$ cards : int 2 1 1 1 2 1 1 1 1 2 ...
$ job : Factor w/ 4 levels "A171","A172",..: 3 3 2 3 3 2 3 ...
$ liable : int 1 1 2 2 2 2 1 1 1 1 ...
$ tele : Factor w/ 2 levels "A191","A192": 2 1 1 1 1 2 1 2 ...
$ foreign : Factor w/ 2 levels "A201","A202": 1 1 1 1 1 1 1 11 ...

#convert outcome default to factor(categorical variable)

gc$Default <- factor(gc$Default)
#Find those numeric variables (T/F)
numvar<- sapply(gc, [Link])
#Normalize 7 numeric variables
nor <-function(x) { (x -min(x))/(max(x)-min(x)) }
gc[,numvar] <- sapply(gc[,numvar], nor)
#For a simple illustration, select only first 3 numeric variables
gc1 <- gc%>%select("Default","duration", "amount", "installment")

#split data
[Link](100)
[Link] <- sample(1: nrow(gc1), size=nrow(gc1)*0.8)

3
 PS0002 Introduction to DS & AI LMXiang

[Link] <-gc1[[Link], ]
[Link] <- gc1[-[Link], ]

#classification with k=3

library(class)
[Link](101)
knn1<-knn([Link][, 2:4], [Link][,2:4], [Link]$Default, k=3)
mean(knn1 ==[Link]$Default )
[1] 0.61

In order to find an optimal value of k, we do the following loop.

#try different k to find the best classfier
ac<-rep(0, 30)
for(i in 1:30){
[Link](101)
knn.i<-knn([Link][,2:4], [Link][,2:4], cl=[Link]$Defaul, k=i)
ac[i]<-mean(knn.i ==[Link]$Default)
cat("k=", i, " accuracy=", ac[i], "\n")
}
#Accuracy plot
plot(ac, type="b", xlab="K",ylab="Accuracy")

The above calculation and plot indicate that k=14 results in highest accuracy 70.5% , i.w., 70.5
% of the points in the test set are correctly classified. We compute its confusion matrix
[Link](101)
knn2<-knn([Link][,2:4], [Link][,2:4], cl=[Link]$Default, k=14)
> mean(knn2 ==[Link]$Default)
[1] 0.705
table(knn2,actual=[Link]$Default)
actual
knn2 0 1
0 127 48
1 11 14

We can compare the kNN with logistic regression classification.

mlogit <- glm(Default~., data = [Link], family = "binomial")
#predicted probability P(Y=1)
Pred.p <-predict(mlogit, newdata =[Link], type = "response")
y_pred_num <-ifelse(Pred.p > 0.5, 1, 0)
y_pred <-factor(y_pred_num, levels=c(0, 1))
mean(y_pred ==[Link]$Default )

4
 PS0002 Introduction to DS & AI LMXiang

[1] 0.68

table(y_pred,actual=[Link]$Default )
actual
y_pred 0 1
0 132 58
1 6 4

We can see that both methods can correctly classify around 70% of the points in the test set.
From the confusion matrices, kNN yields more false positive, while logistic regression results
in more false negative in this example.

Note: 70% is a reasonably good accuracy in this example. Unlike the classification for breast
cancer patients in Example 1, where the benign and malignant classes are well distinguished
by those biological features, whether a loan applicant defaults on his/her loan or not is often
affected by many other factors (might be qualitative).