Support
Vector
Machine
Module checklist
■ Task
❏ Hyperplane (or decision boundary)
❏ Kernel trick
■ Learning Methodology
❏ Quadratic optimization
■ Performance Measure
❏ Confusion matrix
❏ Gamma
e.g sentiment analysis uses structured data
to classify emotional polarity of text
Unstructured data Structured data Supervised classification
Interpretable to a human Interpretable to both a Output classifying the
but not a computer human and computer sentiment of text, voice
program program recording, or facial
Data representation Sentiment analysis expression
algorithms
text facial images voice
or videos
We will use our now familiar framework to
introduce linear support vector machines
What is the problem we want our
Task
model to solve?
ML algorithms can be supervised or
Learning
unsupervised. This determines the
Methodology
learning methodology.
Performance Quantitative measure we use to
Measure evaluate the model’s performance.
Source: Deep Learning Book - Chapter 5: Introduction to Machine Learning
What is the
Task problem we
want to solve?
SVMs are a form of classification that draw boundaries to classify data
How might sentiment analysis apply to the
graph above?
What is the
Task problem we
want to solve?
Areas divided by the SVM boundaries define the predictions for sentiment
☹ For instance red data points
represent text with predicted
negative sentiment...
…while blue data points are
Note, SVM may
misclassify some of
☺ positive sentiment
the points. Boundaries
are rarely perfect
Feature
Task engineering &
selection
In reality, a graph representing sentiment analysis would be across more
than two dimensions
Graph Sentiment analysis equivalent
Data points Document or text to be classified
● Red ● Classified as negative
● Green ● Classified as positive
Axis: x1 Frequency of word x1 in the text
Axis: x2 Frequency of word x2 in the text
Axis: x3 Frequency of word x3 in the text
... Each word adds another
dimension (difficult to visualize)
Task Defining f(x)
The line dividing the the data is called a decision boundary or hyperplane
Like decision trees, a SVM has data points
represented by values across various
features and a classification outcome
Our f(x) is the optimal “hyperplane” dividing
the class outcomes
In the example to the left, the solid
hyperplane is a better boundary than the
dotted line
Two examples of
decision boundaries
Is our f(x)
Task correct for this
problem?
Sometimes the hyperplane is not linear but rather a funky shape
So far, our examples of support
vector machine data points can be
split with a linear hyperplane
This is called a data set that is
linearly separable
However, what about data sets
that look more like the example on
the left?
Is our f(x)
Task correct for this
problem?
To create a non-linear hyperlane, we can use a math shortcut called kernels
Theoretically, if we find the right
combinations of features, we can
find a mapping of the points in a
space that is linearly separable
This is computationally exhaustive.
The “kernel trick” is a
mathematical shortcut that SVMs
take to create non-linear
boundaries
How does our
Learning
ML model
Methodology
learn?
Not all hyperplanes are created equal. Some are better than others
Earlier we mentioned that the solid
hyperplane or decision boundary is a
better boundary than the dotted line.
Why?
Intuitively, the solid line more clearly
divides the data.
When we add the red dot for class A, the
dotted line would have incorrectly
classified it as class B.
How does our
Learning
ML model
Methodology
learn?
What are these red lines we’ve drawn
onto the example graph we know?
They are the lines parallel to the
hyperplane or decision boundary that touch
a point in each class
The point the lines are touching are called
the support vectors
Which red line (dotted or solid) looks like the better hyperplane to you?
Learning What is our
Methodology loss function?
The solid line is optimal.
Notice how there is much more
distance between the two red,
solid lines versus the two red,
dotted lines
Learning What is our
Methodology loss function?
If you’re curious, mathematically this can be
expressed as a quadratic optimization problem Note: the equation
displayed is a
Minimize the distance simplified version
between the decision
boundaries In reality, SVMs
allow for:
● Some data
points to be
classified
incorrectly
● More than
two classes
Such that all data points (not just -1
are classified correctly as and 1)
-1 or 1
Flexibility of
Performance
the model
You can adjust your SVM models with gamma, a sensitivity input. The higher the
gamma, the less influence a single training data point has on the model predictions
Low gamma High gamma
How does gamma relate to bias and variance?
Flexibility of
Performance
the model
It turns out high gamma can lead to overfitting, or high variance.
High bias and High variance
low gamma and gamma
Model cheat sheet
Pros Cons
● Can be used with sparse or ● Data should be linearly separable
imbalanced data for “hard-margin” SVMs
● Finds the optimal decision ● Additional assumptions are used
when data is not linearly
boundary to separate data points
separable
(compared to other classifiers like ● Provides deterministic
perceptrons) classification (no probabilistic
● Does not take up a lot of memory estimate)
to store
● Computationally easy to add
features due to the kernel trick
sources we recommend:
Support vector machines
● *****ESPECIALLY REVIEW THE TWO LINKS BELOW IN GREEN FOR
COURSE RELATED MATERIAL****
●
● [Link]
Beginners
● [Link]
●
● Introduction to Machine Learning with Python (O’Reilly Media) Chapter 2,
Linear Models
● An Idiot’s Guide to Support Vector Machines (R. Berwick)
● Sentiment analysis using Support Vector Machine (Nurulhuda Zainuddin)