PS0002 Introduction to DS & AI LMXiang

Notes for Tutorial 9 - ML for classification: SVM

The 3rd method for classification covered in Lecture 06 is the support vector machine
(SVM). In this tutorial, we aim to

• develop intuition (rather than rigor) for understanding of rationale behind SVM in
classification
• demonstrate application and implementation of SVM
• compare SVM with the other two methods: logistic regression and kNN.

The following packages are required in this tutorial:

• “e1071” which contains function svm()
• “mlbench” for getting dataset : BreastCancer
• “dplyr” for preparing data

1. Support vector machine

Unlike the previous two classification methods logistic regression and kNN, the basic
principle behind SVM is to formally define a hyperplane (in two dimensional space it is
simply a line), which acts like a decision boundary between two classes and best separates the
classes.

• How to draw a hyperplane that best separates classes?

To explain this, we first define some terms used in SVM.

Suppose we draw a random hyperplane and

then check the distance between the Class A
hyperplane and the closest data points from
each class.
o These closest data points to the
hyperplane are known as support
vectors.
o The optimum hyperplane will have a
maximum distance from each of the
support vectors. This distance between
the hyperplane and the support vectors
Support vectors
is known as the margin.

Class B

Margin

o Such a linear hyperplane can be mathematically defined by

𝐰 𝑇 𝐱 𝑖 + b,
th
where 𝐱 𝑖 is the 𝑖 data point and 𝐰 is the hyperplane normal vector.

1
 PS0002 Introduction to DS & AI LMXiang

o The goal of SMV becomes to find 𝐰 and b such that the margin is maximized and
all training data are correctly classify. That is, solve 𝐰 and b to maximize margin 𝑀
(the width of the separation between support vectors of classes)
𝟐
𝑀 = ||𝐰||, ||𝐰||2 = 𝐰 𝑇 𝐰
or equivalently minimize a quadratic target function
𝐰 𝑇 𝐰/𝟐
subject to y𝑖 (w 𝐱 𝑖 +b) ≥ 1 for any 𝑖 ,
T

where y𝑖 is the class label of data point 𝐱 𝑖 with y𝑖 =1 for class A and -1 for class B,

Refer to Slides 51-53 of Lecture Notes 06 for more details of deriving M, while the
methods for solving this constrained quadratic optimization problem given in Slides
54-60 are not required.

• How SVM classifies a new data point?

o After obtaining the solutions of 𝐰 and b from the training data set, the linear SVM
classifier is then given by
𝑓(𝐱) = sign(𝐰 𝑇 𝐱 + b)
for any new data point 𝐱.
o If 𝑓(𝐱) = 1, then classify data point 𝐱 to Class A, otherwise to Class B.

• Extension to non-linear SVM classifier

In practice, not all classes can be well separated by drawing a straight line (or linear
hyperplane).

φ(x) Expanded feature space

Feature space
𝑥2

Kernel Φ:
x → φ(x)

𝑥1
𝑥2

Non-linear hyperplane 𝑥1 Linear hyperplane

How can we do if the data show like this? Non-linear SVM is implemented using a
kernel to transform 𝐱 𝑖 into 𝜑(𝐱 𝑖 ) in a
A simple trick would be transformation.
higher dimensional space that has a clear
(Refer to Slides 62-67 in Lecture Notes 06)
(linear) dividing margin between classes
After kernel transformation, a linear hyperplane can then be established to separate classes.

2
 PS0002 Introduction to DS & AI LMXiang

• Pros:
o Effective in the higher dimensional cases
o Best algorithm when classes are separable
o The hyperplane is affected by only the support vectors thus outliers have less impact.
• Cons:
o Computationally expensive for larger dataset
o May not perform well in case of overlapped classes.
o Selecting the appropriate kernel function can be tricky.

2. Application and implementation of SVM for classification

• Implementation in R
Function svm() in package “e1071” is used to train a SVM for classification, with syntax:

svm(formula, data = NULL, ..., kernel=, [Link] =[Link], scale = TRUE)

where the arguments “formula” is a symbolic description of the model to be fit, same as used in
regression, and “data” specifies the dataframe containing the variables in the model, “kernel”
specifies the kernel type used in training and predicting, with possible choices linear,
polynomial, radial basis and sigmoid.

svm() returns an object containing the fitted model, including: SV (the resulting support
vectors), index (the index of the resulting support vectors in the data matrix) etc.
See details at [Link]

Relevant functions in the package:

o predict() – Using this method, we obtain predictions from the model, as well as
decision values from the binary classifiers. (Note that function fitted() used in the
lecture notes for a simple illustration of implementation of SVM without splitting
training and test datasets.)
o plot() – Visualizing data, support vectors and decision boundaries, if provided.

• Example: revisit BreastCancer data

BreastCancer dataset available in package “mlbench” was analysed in Tutorials 7-8 using
logistic regression and kNN, respectlively. Now we apply SVM to classifying the patients into
benign or malignant class.

data(BreastCancer,package="mlbench")
bc <-BreastCancer[[Link](BreastCancer),]
bc[,2:4]<- sapply(bc[,2:4], [Link])
bc<-bc%>%mutate(y=factor(ifelse(Class=="malignant", 1,0)))%>%
+ select([Link]:[Link], y)
#split data
[Link](100)
[Link] <- sample(1: nrow(bc), size=nrow(bc)*0.8)
[Link] <-bc[[Link], ]
[Link] <- bc[-[Link], ]
# SVM classification, install package e1071
library(e1071)
[Link]<-svm(y~[Link]+[Link]+[Link], data = [Link],
+ kernel = "linear")

3
 PS0002 Introduction to DS & AI LMXiang

summary([Link])
Call:
svm(formula = y ~ [Link] + [Link] + [Link], data = [Link],
kernel = "linear")

Parameters: Target function with Lagrange parameter C

SVM-Type: C-classification
SVM-Kernel: linear
cost: 1 Value of C
Number of Support Vectors: 67
( 33 34 )
Number of Classes: 2
Levels:
01
#predict newdata in test set
[Link] <- predict([Link], newdata=[Link][,1:3])
#evaluate classification performance and check accuracy
table([Link], [Link]$y)

[Link] 0 1
0 82 4
1 4 47
mean([Link] ==[Link]$y)
[1] 0.9416058

SVM with linear kernel performs well in classification similar to previous two methods:
accuracy 96% of kNN classification and 94% of the logistic regression classification in this
example.

• Improve classification performance using other kernel functions

Try other kernel types to see if the performance can be improved, say the radial kernel.
function [Link]() tunes best parameters for the other nonlinear kernel functions in SVM.
The tuning process requires cross validation which randomly splits data into several folds
(e.g., 10-fold by default). It also requires ranges of parameters, such as

o cost: cost of constraints violation (default: 1)---it is the term ‘C’ in the Lagrange
formulation, C>0, we set a range of (10-1 , 102) in this example.
o gamma: parameter needed for all kernels except linear, default: 1/(data
dimension). Set several values <1 and ≥1.
#Set a seed for reproducing results
[Link](123)
[Link]<-[Link](y~., data=[Link], kernel="radial",
+ cost=10^(-1:2), gamma=c(.1,.5,1,2))
summary([Link])
Parameter tuning of ‘svm’:
- sampling method: 10-fold cross validation
- best parameters:
gamma cost
1 1
- best performance: 0.04212121

- Detailed performance results:

gamma cost error dispersion
1 0.1 0.1 0.04942761 0.02581506
2 0.5 0.1 0.04579125 0.02615015
……

4
 PS0002 Introduction to DS & AI LMXiang

#visualize results of parameter tuning

plot([Link])
#confusion matrix and accuracy
[Link] = [Link]$[Link]
[Link] = predict([Link], newdata=[Link][,1:3])
table([Link], [Link]$y)
[Link] 0 1
0 83 2
1 3 49
mean([Link] ==[Link]$y)
[1] 0.9635036

In the output above, the value of “best performance” is the average cross-validation error
rate calculated by the 10-fold cross-validation for the optimal parameter combination
(gamma=1, cost=1) identified during the tuning. Here, the cross-validation error rate refers to
the proportion of misclassified observations in the classification task for each fold during the
10-fold cross-validation. The value 0.04212121 means that the best performing svm with a
radial kernel achieved an approximate 4.2% cross-validation error rate.

A lower best performance value indicates a better SVM model. You can also tune an SVM
with linear kernel.
[Link](123)
[Link].tune0<-[Link](y~.,data=[Link],kernel="linear",cost=10^(-1:2))
summary([Link].tune0)

Using a radial kernel, the tuned SVM shows improved classification performance over the
linear SVM in terms of best performance and classification accuracy. It correctly classifies
over 96% of patients versus 94% for the linear SVM.

• Final remarks for classification

1) Key concepts in confusion matrix
• Recall (Sensitivity) = TP/(TP+FN): measures the
proportion of actual positives correctly identified.
- High recall is essential when false negatives are costly
(e.g., cancer detection).

• False Positive Rate (FPR) = FP/(FP+TN): measures

the proportion of actual negatives incorrectly identified as
positive.
- Low FPR is desired to reduce false alarms

2) Multi-class classification
We have introduced three methods for binary classification only.
o kNN() for both binary and multi-class classification,
o logistic regression glm() for binary classification only,
o svm() for both binary and multi-class classification.