MAST90083 Computational Statistics & Data Mining SVM
Tutorial & Practical 8: Support Vector Machines
Question 1
d
Show that the value M = 2
of the margin for the maximum margin hyperplane is given by
n
1 X
= µi
M2 i=1
where {µi }, i = 1, ..., n are given maximizing
n n n
X 1 XX
LD = µi − µi µj yi yj (x>
i xj )
i=1
2 i=1 j=1
subject to
n
X
µi ≥ 0, i = 1, ..., n, and µi yi = 0.
i=1
Question 2
In this question, we learn how to fit a support vector machine (SVM) model to data by using
the e1071 package in R. We first generate two classes of (X1 , X2 ) vector values and then use
the svm function in library(e1071) to find the nonlinear decision boundary.
1. First, we generate two classes of (X1 , X2 ) vector values (i.e. p = 2 predictors) of size
n = 100 from two different probability distributions respectively. The distribution for
the first class is a mixture of 10 bivariate Gaussians with the 10 means being sampled
from an N ((1, 0)T , I2 ) distribution, and with the 10 variance matrices all being I2 . The
distribution for the second class is the same except that the 10 means are sampled from
N ((0, 1)T , I2 ). Data points of each class are generated by first randomly picking one of
the ten bivariate Gaussians, and then sampling from that bivariate Gaussian with the
associated mean and the covariance 0.2I2 . Plot your generated dataset.
2. The true decision boundary for these simulated data can be obtained by finding where
the two mixture density functions are equal. The boundary can be visually displayed
as a level-0 contour of the differences between the two densities on a grid of points.
Compute the true decision boundary and plot this boundary together with the dataset.
3. Create a vector of response factor Y values of {−1, 1} corresponding to the two classes
of predictors values generated above. Call function svm from library(e1071). Type
?svm to see the description of the arguments in this function (e.g., cost, gamma, kernel,
scale, degree and coef). First, we look at polynomial kernel for SVM. we can first fit
a linear decision boundary by using a polynomial kernel of degree one, make predictions
on a grid and plot the resulting decision boundary. How good a job does the linear
boundary do in separating the two classes? You also can change the values of degree
to see predictions from the polynomial kernel with higher order degree.
1
MAST90083 Computational Statistics & Data Mining SVM
4. Now we consider the radial basis functions to make a nonlinear decision boundary. We
can first try the function svm(kernel="radial",cost=1.5,gamma=1). Plot the decision
boundary and is it close to the true boundary? Then we can try different values of
cost = 1500, 0.0015 and gamma to check how these arguments will influence the estimated
decision boundary. To find the parameters in this svm function properly, we can use the
tune function. This function will “tune” the SVM by testing a range of choices for both
gamma and cost and compute the cross-validated misclassification rates for the range
of parameters submitted. Once reasonable estimates for gamma and cost have been
obtained, fit the SVM and plot the decision boundary.