Classification Algorithms in

Machine Learning
A comprehensive guide to supervised learning techniques for [Link] students
Overview of Classification Algorithms
Classification is one of the most fundamental tasks in machine learning, where the goal is to predict categorical labels for new observations based on
patterns learned from training data. These algorithms form the backbone of many real-world applications, from spam detection to medical diagnosis,
image recognition, and credit scoring.

Naïve Bayes K-Nearest Neighbors Support Vector Machine

Probability-based classifier using Bayes' Instance-based learning that classifies Powerful classifier that finds the optimal
theorem with strong independence based on the majority vote of k closest hyperplane maximizing the margin between
assumptions between features training examples classes

Each algorithm has unique characteristics, strengths, and ideal use cases. Understanding when and how to apply each method is crucial for developing
effective machine learning solutions.
Naïve Bayes Algorithm
Core Concept
Naïve Bayes is a family of probabilistic classifiers based on Bayes' theorem with an
assumption of independence between features. Despite its "naïve" assumption, it
performs remarkably well in many real-world situations, particularly in text classification
and spam filtering.

The algorithm calculates the probability of a class given the features using the formula:

P (F eatures#Class) × P (Class)
P (Class#F eatures) =
P (F eatures)

Since calculating P(Features) is computationally expensive, Naïve Bayes assumes feature

independence, simplifying the calculation to:

n
P (Class#F eatures) ? P (Class) × / P (F eaturei #Class)
i=1

01 02

Calculate Prior Probabilities Compute Likelihoods

Determine the probability of each class in the training data Calculate the conditional probability of each feature given each class

03 04

Apply Bayes' Theorem Make Prediction

Combine priors and likelihoods to get posterior probabilities Select the class with highest posterior probability
Naïve Bayes: Implementation Details
Types of Naïve Bayes Classifiers

Depending on the distribution of the data, different variants of Naïve Bayes can be used:

Gaussian Naïve Bayes Multinomial Naïve Bayes Bernoulli Naïve Bayes

Assumes features follow a normal Suitable for discrete data, particularly text Designed for binary/boolean features. Each
distribution. Ideal for continuous data. classification. Features represent word feature is either present or absent. Useful
Uses mean and variance of each feature counts or frequencies. Commonly used in for text classification with binary word
per class to calculate probabilities. document classification and spam occurrence features.
detection.

Advantages and Limitations

Strengths 7 Weaknesses
Extremely fast training and prediction Strong independence assumption rarely holds
Works well with high-dimensional data Can be outperformed by other algorithms
Requires minimal training data Zero frequency problem (solved by smoothing)
Handles missing values gracefully Poor probability estimates
Performs well with irrelevant features
K-Nearest Neighbors (K-NN)

Visual Intuition Distance Metrics

K-NN works by finding the k closest neighbors to a query point and Commonly uses Euclidean, Manhattan, or Minkowski distance to measure
assigning the majority class among those neighbors similarity between data points

How K-NN Works

K-Nearest Neighbors is a non-parametric, instance-based learning algorithm that makes predictions based on the similarity between data points. Unlike
other algorithms that build a model during training, K-NN stores the entire training dataset and makes predictions at runtime by finding the most similar
training examples.

1 2

Training Phase Distance Calculation

Store all training data without building explicit model Compute distance between query point and all training points

3 4

Find Neighbors Majority Vote

Select k points with smallest distances Assign class based on most frequent neighbor class

The choice of k is critical: small k values make the model sensitive to noise, while large k values smooth decision boundaries but may include irrelevant
points. Cross-validation is typically used to find optimal k.
K-NN: Key Considerations
Choosing the Right K
Selecting the optimal value of k is crucial for K-NN performance. The
value of k represents the number of neighbors that vote on the class of a
new observation.

Small k (k=1, 3, 5): More flexible decision boundaries, sensitive to

noise and outliers, higher variance
Large k (k=15, 25, 50): Smoother decision boundaries, less sensitive
to noise, higher bias
Rule of thumb: k = :n where n is the number of training samples
Odd values: Prevent ties in binary classification

Cross-validation helps identify the k value that balances bias and variance
for your specific dataset.

Distance Metrics in Detail

Euclidean Distance Manhattan Distance Minkowski Distance

n n n

d(x, y) = 3(xi 2 yi )2 d(x, y) = 3 #xi 2 yi # d(x, y) = (3 #xi 2 yi #p )1/p

i=1 i=1 i=1

Most common metric, measures straight- Sum of absolute differences, also called L1 Generalization of Euclidean and Manhattan.
line distance. Sensitive to different scales. norm. Less sensitive to outliers. p=2 gives Euclidean, p=1 gives Manhattan.

Important: Always normalize or standardize features before applying K-NN since it's sensitive to the scale of features. Different scales can
cause some features to dominate distance calculations.
Support Vector Machine (SVM)
Core Concept

Support Vector Machine is a powerful supervised learning algorithm that finds the optimal hyperplane that maximally separates different classes. The
key insight is that instead of just finding any separating hyperplane, SVM identifies the one with the largest margin between classes, making it more
robust to new data.

Linear Separation Kernel Trick

Find hyperplane that separates classes with maximum margin Transform data to higher dimensions for non-linear separation

1 2 3

Support Vectors
Identify critical training points closest to decision boundary

Mathematical Foundation

For linearly separable data, SVM finds the hyperplane defined by w·x + b = 0 that maximizes the margin. The optimization problem is formulated as:

1
Minimize: ##w##2 Subject to: yi (w ç xi + b) g 1
2
For non-linearly separable data, slack variables (¿_i) are introduced to allow some misclassifications:

n
1
Minimize: ##w##2 + C 3 ¿i
2
i=1

where C is the regularization parameter controlling the trade-off between maximizing margin and minimizing classification error.
SVM: Kernels and Implementation
The Kernel Trick

One of SVM's most powerful features is its ability to handle non-linearly separable data through kernel functions. Instead of explicitly transforming data
to higher dimensions, kernels compute the dot product in the transformed space directly.

Linear Kernel Polynomial Kernel Radial Basis Function (RBF)

K(xi , xj ) = xTi xj K(xi , xj ) = (xTi xj + c)d K(xi , xj ) = exp(2³##xi 2 xj ##2 )

Standard dot product, suitable for linearly Creates polynomial decision boundaries. d is Most popular kernel, creates complex non-
separable data degree, c is constant linear boundaries

Parameters and Tuning

C Parameter ³ (Gamma) Parameter

Regularization parameter that controls the trade-off between achieving a Kernel coefficient for RBF, polynomial, and sigmoid kernels. Small ³
low training error and a low testing error. Small C creates a wider margin means far influence (smoother decision boundary), large ³ means near
with more misclassifications, large C creates a narrower margin with influence (more complex, potentially overfitting).
fewer misclassifications.

Implementation Tip: Use grid search with cross-validation to find optimal C and ³ values. Start with C in {0.1, 1, 10, 100} and ³ in {0.001, 0.01,
0.1, 1}. Standardize features before training SVM.
Evaluating Classification Model Performance
Model evaluation is critical to understand how well your classifier performs and to compare different algorithms. Different metrics provide different
insights into model behavior.

Confusion Matrix Foundation

Every evaluation metric starts with the confusion matrix, which shows actual vs. predicted classifications:

Predicted: Yes Predicted: No

Actual: Yes True Positive (TP) False Negative (FN)

Actual: No False Positive (FP) True Negative (TN)

TP TN FP FN
True Positives True Negatives False Positives False Negatives
Correctly predicted positive cases Correctly predicted negative cases Incorrectly predicted as positive Incorrectly predicted as negative

Key Evaluation Metrics

1 2

Accuracy Precision

TP + TN TP
Accuracy = Precision =
TP + TN + FP + FN TP + FP
Overall correctness. Best when classes are balanced. Proportion of predicted positives that are actually positive. Important
when FP cost is high.

3 4

Recall (Sensitivity) Specificity

TP TN
Recall = Specificity =
TP + FN TN + FP
Proportion of actual positives correctly identified. Important when FN Proportion of actual negatives correctly identified. Complement of
cost is high. false positive rate.
Comparing Metrics and Final Insights
Understanding the Trade-offs

Choosing the right metric depends on your specific problem and the cost of different types of errors. For example:

Medical diagnosis: High recall is crucial (don't miss any sick patients)
Spam detection: High precision is important (don't misclassify important emails)
General applications: Balance precision and recall using F1-score

Additional Important Metrics

F1-Score ROC Curve & AUC Matthews Correlation

Plots True Positive Rate vs. False Positive
Coefficient
Precision × Recall
F1 = 2 × Rate at various thresholds. AUC summarizes Comprehensive metric for binary
Precision + Recall
overall performance. classification that considers all four
Harmonic mean of precision and recall. Best confusion matrix values.
when seeking balance between both.

Summary: When to Use Each Algorithm

Naïve Bayes K-NN SVM

Text classification tasks Small to medium datasets High-dimensional data
High-dimensional data Local patterns important Clear margin separation
Fast training required No assumptions about data Non-linear patterns
Small training datasets Non-linear boundaries Robust to overfitting
Baseline model Lazy learning acceptable Memory efficient

Final Advice: Always start with a simple algorithm as baseline, then try more complex models. Use cross-validation for reliable performance
estimates. Choose evaluation metrics based on business requirements, not just accuracy.