MACHINE LEARNING

SUPPORT VECTOR MACHINE

ALGORITHM
 Support Vector Machine
 Support Vector Machine is a supervised machine learning
algorithm used for classification as well as regression algorithms
 The goal of the SVM algorithm is to create the best line or
decision boundary that can separate n-dimensional space (where
n is the number of features we have) into classes so that we can
correctly put the new data point in the correct category/class
 This best decision boundary is called a hyperplane
 Given a set of labeled data points, an SVM tries to separate the
data points into different output classes
 Why is it known as SVM?
 Because it chooses extreme vectors or support vectors to
create the hyperplane, that’s why it is named as SVM
 Support Vector Machine
SVM Hyperplane
 There may be multiple lines/decision boundaries to segregate the
classes in n-dimensional space
 Still, we want to search out the simplest decision boundary that helps
to classify the information points
 This best boundary is considered to be the hyperplane of SVM
 The dimensions of the hyperplane rely on the features present within
the dataset
 These features suggest if there are 2 target labels in the dataset, then
the hyperplane is going to be a line
 And if there are 3 target labels, then the hyperplane is going to be a
2-dimension plane
 We always create a hyperplane that provides maximum margin
 This margin simply means there should be a maximum
distance between the data points
 Support Vector Machine
SVM Support Vectors
 Support vectors are defined as the data points, which
are closest to the hyperplane/decision boundary and
have some effect on its position
 As these vectors are supporting the hyperplane,
therefore named as Support vectors
 They are the most important points because they
determine the position and orientation of the
boundary
 Machine refers to the algorithm or model that learns
from data to find the best separating hyperplane
Linear Support Vector Machine
Linear Support Vector Machine
Linear Support Vector Machine
Linear Support Vector Machine
Linear Support Vector Machine (What
could be a good separator for this data?)
Linear Support Vector Machine
Linear Support Vector Machine
Linear Support Vector Machine
How can we identify the right hyper-plane?
Identify the right hyper-plane: Here, we have three hyper-planes (A, B, and C). Now, identify
the right hyper-plane to classify stars and circles.
How can we identify the right hyper-plane?
Identify the right hyper-plane: Here, we have three hyper-planes (A, B, and C), and all
segregate the classes well. Now, How can we identify the right hyper-plane?

If we select a hyper-plane having a low margin, then there is a high

chance of misclassification.
How can we identify the right hyper-plane?
Identify the right hyper-plane: Hint: Use the rules as discussed in the previous section to
identify the right hyper-plane.
Non-linearly separable data
 A linear boundary might be too simple to capture
the data. What could be the solution?
 Option 1: Relax the constraints and allow some
points to be misclassified by the margin.
 Option 2: Allow a nonlinear decision boundary in
the input space by finding a linear decision
boundary in an expanded space (similar to adding
polynomial terms in linear regression.)
Kernel Methods (Making the Non-
Linear Linear)
 How to separate the following data with straight
line?
Kernel Methods (Making the Non-
Linear Linear)
How can we identify the right hyper-plane?
Find the hyper-plane to segregate to classes: In the scenario below, we can’t have a linear
hyper-plane between the two classes, so how does SVM classify these two classes?
• SVM can solve this problem by introducing additional features. Here, we will add a new
feature, z=x2 +y2

• All values for z would always be positive because z is the squared sum of both x and y
• In the original plot, red circles appear close to the origin of the x and y axes, leading to
a lower value of z. The star is relatively away from the original results due to the higher
value of z.
Kernel Methods (Making the Non-
Linear Linear)
Kernel Methods (Making the Non-
Linear Linear)
 The Kernel Trick

• The Kernel Trick is a technique used to handle non-linearly separable data by

transforming it into a higher-dimensional space where it becomes linearly
separable
• The SVM kernel could be a function that takes low dimensional input space and
transforms it into a better dimensional space, i.e., it converts non-separable
problems to separable problems
• It helps us to deal with non-linear separation problems
• It does some extremely complex data transformations, then finds out the method
to separate the data points based on the target classes you’ve defined
 Different Types of Kernels
Linear Kernel
• It is the most basic type of kernel, usually one dimensional in nature
• It proves to be the best function when there are lots of features
• The linear kernel is mostly preferred for text-classification
problems as most of these kinds of classification problems can be
linearly separated
• Linear kernel functions are faster than other functions

Polynomial Kernel
• It is a more generalized representation of the linear kernel
• It is not as preferred as other kernel functions as it is less
efficient and accurate
 Different Types of Kernels
Gaussian Radial Basis Function (RBF)
• It is one of the most preferred and used kernel functions in SVM
• It is usually chosen for non-linear data
• It helps to make proper separation when there is no prior knowledge
of data
Sigmoid Kernel
• It is mostly preferred for neural networks
• This kernel function is similar to a two-layer perceptron model of the
neural network, which works as an activation function for neurons
Different Types of Kernels
SVMs versus Logistic Regression
Tuning the hyper-parameters for SVM

 Kernel
 Gamma
 C
Tuning the hyper-parameters for SVM
 Gamma
 Intuitively, the gamma parameter defines how far the
influence of a single training example reaches, with low
values meaning ‘far’ and high values meaning ‘close’.
 The gamma parameters can be seen as the inverse of
the radius of influence of samples selected by the
model as support vectors.
 A lower value of Gamma will loosely fit the training
dataset, whereas a higher value of gamma will exactly
fit the training dataset, which causes?
 over-fitting.
Tuning the hyper-parameters for SVM
 Tuning the hyper-parameters for SVM

 C (also called regularization parameter)

 The C parameter controls how much you want to

punish your model for each misclassified point for a

given curve.
Large values of C Small Values of C
Large effect of noisy points. Low effect of noisy points.

A plane with very few Planes that separate the

misclassifications points well will be found,
will be given precedence. even if there are some
misclassifications
Tuning the hyper-parameters for SVM
Tuning the hyper-parameters for SVM
 Conclusion
 Advantages
 SVM Classifiers offer good accuracy and perform faster prediction
 They also use less memory because they use a subset of training points
in the decision phase
 SVM works well with a clear margin of separation and with high
dimensional space
 SVM’s are very good when we have no idea on the data
 Works well with even unstructured and semi structured data like text
and Images
 SVM models have generalization in practice, the risk of over-fitting is
less in SVM
 Conclusion
 Disadvantages
 SVM is not suitable for large datasets because of its
high training time
 It works poorly with overlapping classes and is also
sensitive to the type of kernel used
 Sensitive to outliers, which can shift the decision
boundary
 Choosing a “good” kernel function is not easy
 Regularization parameter (C) and gamma (γ) need
careful tuning to avoid overfitting or underfitting
 Not Ideal for Multi-Class Classification