Support Vector Machine (SVM)
A Support Vector Machine (SVM) is a supervised learning model that identifies the hyperplane
that maximizes the margin between different classes in a dataset.
1. The margin is the distance between the hyperplane and the nearest data points from each
class.
2. These nearest points are called support vectors.
3. The optimal hyperplane is the one with the maximum margin, ensuring better
generalization.
Classification of SVM: SVM can be classified based on data separability and learning objective.
1. Based on Data Separability:
a. Linear SVM: Used when data is linearly separable. A straight line (2D) or plane
(higher dimensions) separates classes.
b. Non-Linear SVM: Used when data is not linearly separable. It uses the kernel
trick to transform data into higher dimensions. Linear kernel, Polynomial kernel,
Radial Basis Function (RBF) kernel are some kernels used in Non-Linear SVM.
2. Based on Learning Task:
a. Support Vector Classification (SVC): Separates data into two or more
categories. It can handle binary classification or multi-class classification using
one-vs-one or one-vs-rest strategies
b. Support Vector Regression: It predicts continuous values. This approach tries to
fit data within a defined margin of tolerance.
Advantages of SVM:
1. Effective in high-dimensional spaces
2. Works well with small datasets
3. Memory efficient (uses support vectors only)
4. Flexible due to kernel functions
Disadvantages of SVM:
1. Computationally expensive for large datasets
2. Choosing the right kernel can be difficult
3. Not ideal for very noisy data
Prerequisites:
Line: A line is a one-dimensional affine subspace of a vector space.
Plane: A plane is a two-dimensional affine subspace of a vector space.
Hyperplane: A hyperplane in ℝⁿ is an (𝑛 − 1) dimensional affine subspace.
Line (ℝ²) Plane Hyperplane
𝑎𝑥 + 𝑏𝑦 + 𝑐 = 0 𝑎𝑥 + 𝑏𝑦 + 𝑐𝑧 + 𝑑 = 0 𝑎1 𝑥1 + 𝑎2 𝑥2 + ⋯ + 𝑎𝑛 𝑥𝑛 + 𝑏
=0
Euclidean
1. a, b are not both zero. 1. a,b,c are not all zero 1. 𝑎1 … 𝑎𝑛 ≠ 0
2. x, y are variables 2. (a,b,c) is the normal vector 2. b is a constant
3. c is a constant
𝑥 = 𝑝 + 𝑡𝑣 𝑥 = 𝑝 + 𝑠𝑉1 + 𝑡𝑉2 𝑎⋅𝑥+𝑏 =0
1. 𝑝 = 𝑎 fixed point on the 1. p = point on plane 1. a = normal vector
Vector
line 2. 𝑉1 , 𝑉2 linearly independent 2. x = variable vector
2. 𝑣 ≠ 0 = direction vectors
𝑑𝑖𝑟𝑒𝑐𝑡𝑖𝑜𝑛 𝑣𝑒𝑐𝑡𝑜𝑟 3. 𝑠, 𝑡 ∈ 𝑅
3. 𝑡 ∈ 𝑅
1. Has dimension 1. 1. Has dimension 2 1. Has dimension n – 1
Explanation
2. Extends infinitely in two 2. Is determined by: 2. Divides ℝⁿ into two half-
directions. a) A point spaces
3. Is determined by a point b) Two independent directions 3. Is determined by
and a direction. (normal vector) a) normal vector
b) A scalar offset
Visualization not possible
Visualization
Sample x₁ x₂ Class (y)
A 1 2 -1
B 2 3 -1
C 3 3 +1
D 5 6 +1
E 6 7 +1
Plot the points:
From This image, we can see, B(2,3) and C(3,3) point are the closest point between two
classes, therefore B and C are the support vectors.
As we know, SVM draw plane between classes,
Equation of Hyperplane: 𝒘𝟏 𝒙𝟏 + 𝒘𝟐 𝒙𝟐 + 𝒃 = 𝟎
𝟐+𝟑 𝟑+𝟑
Mid-point between support vectors: 𝑴 = ( , ) = (𝟐. 𝟓, 𝟑)
𝟐 𝟐
Direction between Support Vectors: 𝑪 − 𝑩 = (𝟑 − 𝟐, 𝟐 − 𝟐 ) = (𝟏, 𝟎)
Therefore, (𝒘𝟏 , 𝒘𝟐 ) = (𝟏, 𝟎)
So, hyperplane is: 𝒙𝟏 + 𝒃 = 𝟎
Since the hyperplane pass through mid-point, so 𝒙𝟏 = 𝟐. 𝟓, so b = -2.5
Decision boundary: 𝐟(𝐱) = 𝒙𝟏 − 𝟐. 𝟓
Classification rule: If 𝒙𝟏 > 𝟐. 𝟓 = +𝟏, 𝒙𝟏 < 𝟐. 𝟓 = +𝟏
𝟐 𝟐 𝟐
Margin formula: 𝜸 = = = =𝟐
‖𝒘‖ √𝒘𝟏 𝟐 +𝒘𝟐 𝟐 √𝟏𝟐 +𝟎𝟐
𝜸
Effective Margin: 𝒎𝒂𝒓𝒈𝒊𝒏 𝟐 = 𝟏