Support Vector Machine(SVM)
Algorithm
Hlua Khiangte
DEFINITION
Support Vector Machine (SVM) is a supervised learning algorithm.
● Classification
● Regression
● Goal: Identify the optimal separating boundary between data classes.
● Known for strong performance on both linearly and non-linearly separable data.
Applied :
● image recognition
● text classification
● bioinformatics
Key Idea of SVM
● Find a hyperplane that best separates the classes.
● The best hyperplane maximizes the margin—distance between the boundary and the nearest data
points.
● These critical data points are called Support Vectors.
Mathematical Foundations of SVM Algorithm
For two-class classification, given a dataset (xi, yi), where xi are feature vectors and yi are class
labels (either +1 or -1), SVM aims to minimize the function:
Hyperplane formula:
w·x + b = 0 where:
Objective:
Maximize margin = 2 / ||w|| ● w represents the weight vector,
● b is the bias term,
Optimization: ● ||w||2 ensures the maximization of the
Minimize 1/2 ||w||² margin.
Subject to:
yi(w·xi + b) ≥ 1 for all training samples.
Kernel Trick : Handling Non-Linearity
Real-world data is often non-linearly separable. SVM uses kernel functions to transform data
into a higher-dimensional space where it becomes separable. Some popular kernels include:
● Linear Kernel: K(x, y) = x · y (used in linear support vector machine)
● Polynomial Kernel: K(x, y) = (x · y + c)d
● Radial Basis Function (RBF) Kernel: K(x, y) = e−γ||x − y||2
Problem Statement:
A company wants to classify emails as spam or not spam based on features such as word frequency and
message length.
import numpy as np
import pandas as pd
from sklearn.model_selection import train_test_split
from sklearn.feature_extraction.text import TfidfVectorizer
from [Link] import SVC
from [Link] import accuracy_score, classification_report
# Sample dataset (emails & labels: 1 for spam, 0 for not spam)
emails = ["Win a lottery now", "Meeting scheduled for tomorrow", "Get discount on medicines",
"Your bank account is updated", "Urgent: Update your password"]
labels = [1, 0, 1, 0, 1] # Spam = 1, Not Spam = 0
# Convert text data into numerical form using TF-IDF Vectorizer
vectorizer = TfidfVectorizer()
X = vectorizer.fit_transform(emails)
# Splitting dataset into training and testing sets
X_train, X_test, y_train, y_test = train_test_split(X, labels, test_size=0.2, random_state=42)
# Train the SVM model
svm_model = SVC(kernel='linear', C=1.0)
svm_model.fit(X_train, y_train)
# Make predictions
y_pred = svm_model.predict(X_test)
# Evaluate performance
accuracy = accuracy_score(y_test, y_pred)
report = classification_report(y_test, y_pred)
print("Model Accuracy:", accuracy)
print("Classification Report:\n", report)