Support Vector Machine (SVM)
A detailed tutorial covering maximum-margin classification, support vectors, soft margins,
hinge loss, kernels, RBF, C, gamma, SVR, Python, tuning, and practical workflow.
1. What Is SVM?
Support Vector Machine is a supervised-learning algorithm primarily used for classification, with Support
Vector Regression (SVR) for regression. Its central idea is to find a decision boundary that maximizes the
margin between classes.
• The closest observations are support vectors.
• The margin measures distance from the boundary to the nearest observations.
• Soft-margin SVM allows violations.
• Kernels allow nonlinear decision boundaries.
2. Linear Decision Boundary
wᵀx + b = 0
w determines the orientation of the hyperplane and b determines its position. The sign of the decision
function determines the predicted side.
3. Maximum Margin
Margin width = 2 / ||w||
Maximizing the margin is equivalent to minimizing the magnitude of w under the classification constraints.
4. Hard-Margin SVM
yᵢ(wᵀxᵢ + b) ≥ 1
This assumes perfectly separable training data and is sensitive to outliers.
5. Soft-Margin SVM
min 1/2 ||w||² + C Σᵢ ξᵢ
yᵢ(wᵀxᵢ+b) ≥ 1−ξᵢ, ξᵢ ≥ 0
Slack variables allow observations to violate the margin. C controls the trade-off between a wide margin
and violations.
6. Understanding C
C Typical effect
Small C More tolerance for margin violations; stronger regularization; smoother boundary.
Large C Stronger penalty for violations; tighter fit to training data; potentially more overfitting.
7. Hinge Loss
Support Vector Machine (SVM) — Detailed Tutorial Page 1
L = max(0, 1 − y f(x))
Correct predictions with sufficient margin have zero hinge loss. Points inside the margin or misclassified
receive positive loss.
8. Support Vectors
Support vectors are the observations that critically influence the maximum-margin boundary. Observations
far away from the boundary usually have much less direct influence.
9. Feature Scaling
Scaling is usually essential for SVM, especially RBF kernels. Distances and dot products can otherwise be
dominated by large-scale features.
• Use StandardScaler in a Pipeline.
• Fit scaling only on training data.
• Do not fit preprocessing on the full dataset before cross-validation.
10. Kernel Trick
A kernel computes similarity in an implicit feature space, allowing SVM to learn nonlinear boundaries
without explicitly constructing every transformed feature.
10.1 Linear Kernel
K(x,z) = xᵀz
10.2 Polynomial Kernel
K(x,z) = (γ xᵀz + r)^d
10.3 RBF Kernel
K(x,z) = exp(−γ ||x−z||²)
RBF is one of the most commonly used nonlinear SVM kernels.
11. Understanding Gamma
For RBF SVM, gamma controls how quickly similarity falls as distance increases.
Gamma Typical behavior
Small gamma Broader influence; smoother decision boundary; possible underfitting.
Large gamma Local influence; more flexible boundary; possible overfitting.
12. C and Gamma Together
• High C + high gamma can create a very flexible boundary.
• Low C + low gamma produces stronger regularization and a smoother boundary.
• Use cross-validation to select these parameters.
13. Classification in Python
from [Link] import Pipeline
from [Link] import StandardScaler
Support Vector Machine (SVM) — Detailed Tutorial Page 2
from [Link] import SVC
model = Pipeline([
("scaler", StandardScaler()),
("svc", SVC(kernel="rbf", C=1.0, gamma="scale"))
])
[Link](X_train, y_train)
pred = [Link](X_test)
14. Probability Estimates
SVC does not produce probabilities by default. Set probability=True when probability estimates are
required; this adds calibration-related computation.
model = Pipeline([
("scaler", StandardScaler()),
("svc", SVC(kernel="rbf", probability=True))
])
[Link](X_train, y_train)
proba = model.predict_proba(X_test)[:, 1]
15. Support Vector Regression
SVR extends the margin idea to numerical prediction. It uses an epsilon-insensitive tube: errors inside the
tube do not contribute to the epsilon-insensitive loss.
|yᵢ − f(xᵢ)| ≤ ε
from [Link] import SVR
from [Link] import Pipeline
from [Link] import StandardScaler
model = Pipeline([
("scaler", StandardScaler()),
("svr", SVR(kernel="rbf", C=10, epsilon=0.1))
])
[Link](X_train, y_train)
pred = [Link](X_test)
16. Important Hyperparameters
Parameter Meaning
C Penalty/regularization trade-off.
kernel Linear, RBF, polynomial, sigmoid, etc.
gamma Kernel locality/flexibility for RBF/poly/sigmoid.
degree Polynomial kernel degree.
epsilon Width of SVR's insensitive tube.
class_weight Class weighting for imbalanced classification.
17. Hyperparameter Tuning
from sklearn.model_selection import GridSearchCV
params = {
"svc__C": [0.1, 1, 10, 100],
"svc__gamma": ["scale", 0.01, 0.1, 1],
"svc__kernel": ["rbf", "linear"]
}
search = GridSearchCV(model, params, cv=5, scoring="f1", n_jobs=-1)
[Link](X_train, y_train)
print(search.best_params_)
Support Vector Machine (SVM) — Detailed Tutorial Page 3
18. SVM vs Logistic Regression
Property Logistic Regression SVM
Core objective Likelihood/log loss Margin maximization
Probability Natural output Needs probability option/calibration
Boundary Linear unless features transformed Linear or nonlinear kernels
Scaling Usually helpful Usually essential
Interpretability Higher for linear model Lower, especially with kernels
19. SVM vs Random Forest
Property SVM Random Forest
Scaling Important Usually unnecessary
Nonlinearity Kernels Tree splits
Large data Kernel SVM can be expensive Often practical
Interactions Kernel-based Captured by trees
Interpretability Moderate/low Moderate/low
20. Class Imbalance
For imbalanced classification, consider class_weight='balanced', appropriate metrics, and threshold
analysis. Accuracy alone can be misleading.
svc = SVC(kernel="rbf", C=1.0, gamma="scale", class_weight="balanced")
21. Strengths
• Strong maximum-margin theory.
• Effective in high-dimensional spaces.
• Kernel trick enables nonlinear boundaries.
• Linear SVM can be excellent for sparse text features.
• Can perform strongly with clear class separation.
22. Limitations
• Kernel SVM can be expensive on very large datasets.
• C and gamma tuning can be sensitive.
• Scaling is usually required.
• Nonlinear models are difficult to interpret.
• Probability estimates are not as direct as logistic regression.
• Performance can degrade with substantial noise and overlap.
23. End-to-End Workflow
Step Action
1 Define target and metric.
Support Vector Machine (SVM) — Detailed Tutorial Page 4
Step Action
2 Audit missing values, categories, outliers, and leakage.
3 Split data appropriately.
4 Scale inside a pipeline.
5 Start with linear SVM.
6 Try RBF if nonlinear structure is plausible.
7 Tune C and gamma with cross-validation.
8 Evaluate on untouched test data.
9 Calibrate probabilities if the application requires them.
24. Common Mistakes
• Using SVM without scaling.
• Choosing C/gamma using the test set.
• Assuming RBF is always best.
• Using accuracy alone for severe class imbalance.
• Ignoring computational cost.
• Fitting preprocessing before cross-validation and causing leakage.
25. Interview Questions
Q1. What is the main idea of SVM?
Find a maximum-margin separating boundary; support vectors are the critical observations.
Q2. What does C control?
The trade-off between margin width and violations. Larger C penalizes violations more strongly.
Q3. What does gamma control?
For RBF, gamma controls how local the influence of observations is. Higher gamma generally creates a
more flexible boundary.
Q4. Why scale features?
SVM relies on distances/dot products, so unequal feature scales can distort the geometry.
Q5. What is the kernel trick?
It computes similarity in an implicit feature space without explicitly creating every transformed feature.
26. Practice Problems
• Explain why maximizing the margin can improve generalization.
• What happens as C becomes very large?
• What happens as RBF gamma becomes very large?
• Why is scaling important?
• Compare linear and RBF SVM.
• Why can kernel SVM be expensive on large datasets?
Support Vector Machine (SVM) — Detailed Tutorial Page 5
• When might logistic regression be preferable?
27. Quick Reference
Concept Remember
SVM Maximum-margin supervised-learning method.
Support vectors Critical observations defining the margin/boundary.
C Margin-violation penalty / regularization trade-off.
Kernel Enables nonlinear decision functions.
RBF gamma Controls locality/flexibility.
Hinge loss max(0, 1 − y f(x)).
SVR Regression extension using an epsilon-insensitive tube.
28. Final Takeaway
SVM is fundamentally a margin-based algorithm. Linear SVM finds a maximum-margin hyperplane; soft
margins use C to tolerate violations; kernels such as RBF enable nonlinear boundaries. In practice, the
critical workflow is scale correctly → choose a kernel → tune C/gamma → cross-validate → evaluate on
untouched data.
Support Vector Machine (SVM) — Detailed Tutorial Page 6