Aight brada, let’s break down Support Vector Machines (SVM) step by step with formulas.
Step 0: SVM Model
• SVM is a classifier that finds the best separating hyperplane between two classes.
• For linear SVM, the decision boundary is:
f (x) = wT x + b
- Predict class:
y^ = sign(f (x))
Where: - w = weight vector (normal to hyperplane) - b = bias (intercept)
Step 1: Margin
• SVM finds the hyperplane with the maximum margin between the two classes.
• Margin = distance between closest points of each class (support vectors) and the hyperplane.
• Constraint for all points:
yi (wT xi + b) ≥ 1, i = 1, ..., n
Where yi ∈ {−1, 1} .
Step 2: Optimization Problem
• Maximize margin is equivalent to minimizing ∣∣w∣∣2 :
1
min ∣∣w∣∣2
w,b 2
Subject to:
yi (wT xi + b) ≥ 1
• This is a convex quadratic optimization problem.
Step 3: Soft Margin (for non-separable data)
• Introduce slack variables ξi to allow some misclassification:
n
1
min ∣∣w∣∣2 + C ∑ ξi
w,b 2
i=1
1
Subject to:
yi (wT xi + b) ≥ 1 − ξi , ξi ≥ 0
- C controls trade-off between margin size and classification error.
Step 4: Kernel Trick (for non-linear SVM)
• Map input features to higher-dimensional space ϕ(x) and apply linear SVM there:
f (x) = wT ϕ(x) + b
- Common kernels: - Linear: K(xi , xj )
= xTi xj - Polynomial: K(xi , xj ) = (xTi xj + c)d - RBF
(Gaussian): K(xi , xj ) = exp(−γ∣∣xi − xj ∣∣2 )
Step 5: Prediction
• For a new sample x:
n
y^ = sign (∑ αi yi K(xi , x) + b)
i=1
Where αi are Lagrange multipliers from optimization problem.
✅ Summary of Formulas
1. Decision function: f (x) = wT x + b
2. Constraint: yi (w T xi + b) ≥ 1
3. Optimization (hard margin): minw,b 12 ∣∣w∣∣2 subject to constraint
4. Soft margin: minw,b 12 ∣∣w∣∣2 + C ∑ ξi with constraints
5. Kernel trick: f (x) = w T ϕ(x) + b
6. Prediction with kernel: y^ = sign(∑ αi yi K(xi , x) + b)