MACHINE LEARNING · STUDY GUIDE

Supervised Learning
A Detailed Guide to Regression & Classification Algorithms

REGRESSION CLASSIFICATION

1 Linear Regression 1 Logistic Regression

2 Ridge Regression 2 Decision Tree Classifier

3 Lasso Regression 3 Random Forest Classifier

4 Elastic Net 4 XGBoost / LightGBM / CatBoost

5 Decision Tree Regressor 5 Support Vector Machine (SVM)

6 Random Forest Regressor 6 Naive Bayes

7 Gradient Boosting / XGBoost 7 K-Nearest Neighbours (KNN)

14 algorithms · explained in detail Intuition — Math — Hyperparameters — Code

Contents
Foundations of Supervised Learning
What supervised learning is, regression vs. classification, the bias–variance
trade-off, regularisation, and evaluation metrics.

Part I · Regression Algorithms

1 Linear Regression
2 Ridge Regression
3 Lasso Regression
4 Elastic Net
5 Decision Tree Regressor
6 Random Forest Regressor
7 Gradient Boosting / XGBoost

Part II · Classification Algorithms

1 Logistic Regression
2 Decision Tree Classifier
3 Random Forest Classifier
4 XGBoost / LightGBM / CatBoost
5 Support Vector Machine (SVM)
6 Naive Bayes
7 K-Nearest Neighbours (KNN)

Appendix · Cheat-Sheet & Model Selection

At-a-glance comparison tables and a practical workflow for choosing a model.
FOUNDATIONS SUPERVISED LEARNING · DETAILED GUIDE

0 Foundations of Supervised
Learning
The shared vocabulary you need before reading any algorithm chapter

Supervised learning is the branch of machine learning where a model learns from
labelled examples. You hand the algorithm a set of input–output pairs, and it
searches for a function that maps inputs to outputs well enough to generalise to
data it has never seen.

The core setup

Every supervised problem is built from a dataset of m examples. Each example has a
feature vector x (the inputs, also called predictors or independent variables) and a target y
(the label or dependent variable). The goal is to learn a function f such that f(x) ≈ y, not
just on the training data but on future data drawn from the same distribution.

The two families of problems differ only in the nature of the target:

• Regression — the target is a continuous number (house price, temperature,

demand). The model outputs a real value and we measure how far off it is.
• Classification — the target is a discrete category (spam / not-spam, disease type,
digit 0–9). The model outputs a class or a probability over classes.

How a model is trained and judged

Fitting a model means choosing its internal parameters to minimise a loss function — a
number that measures prediction error on the training data. Some models solve this with a
closed-form equation; most use iterative optimisation such as gradient descent, which
repeatedly nudges parameters in the direction that reduces loss.

Because low training error is easy to fake by memorising, we always split the data. A
typical workflow holds out a test set that the model never sees during training, and often a
validation set (or cross-validation) for tuning hyperparameters. Performance on held-out
data is the only honest estimate of how the model will behave in production.

Parameters vs. hyperparameters

Parameters are learned from data (e.g. regression coefficients, tree split points).
Hyperparameters are set by you before training and control how learning happens (e.g.
regularisation strength, tree depth, number of neighbours). Tuning hyperparameters well
is often what separates a mediocre model from a strong one.

The bias–variance trade-off

Generalisation error decomposes into three pieces: bias, variance, and irreducible noise.
Bias is error from overly simple assumptions — the model underfits and misses real
structure. Variance is sensitivity to the particular training sample — the model overfits and

3
FOUNDATIONS SUPERVISED LEARNING · DETAILED GUIDE

memorises noise. Lowering one usually raises the other; the art is finding the sweet spot.

Expected error = (underfitting) + (overfitting) + (irreducible noise)

• Underfitting shows up as high error on both training and test data. The fix is a more
flexible model or better features.
• Overfitting shows up as low training error but high test error. The fix is more data,
regularisation, or a simpler model.

Regularisation and feature scaling

Regularisation adds a penalty for model complexity to the loss, discouraging extreme
parameter values and curbing variance. It is the central idea behind Ridge, Lasso, and
Elastic Net, and it appears as depth limits in trees and as the margin objective in SVMs.
Feature scaling (standardising features to comparable ranges) matters greatly for
distance- and gradient-based methods (KNN, SVM, regularised linear models) but is
irrelevant for tree-based methods, which split on thresholds and are invariant to monotonic
rescaling.

How we score regression models

Metric Meaning Notes

Same units as the target; robust to

MAE Mean absolute error — average of |y − ŷ|.
outliers.

Penalises large errors heavily; basis

MSE Mean squared error — average of (y − ŷ)².
of most optimisation.

Back in target units; still

RMSE Square root of MSE.
outlier-sensitive.

1 is perfect, 0 = no better than the

R² Fraction of variance the model explains.
mean, can go negative.

How we score classification models

Metric Meaning Use when

Classes are balanced and errors

Accuracy Fraction of predictions that are correct.
are equally costly.

Of predicted positives, how many are truly False positives are expensive (e.g.
Precision
positive. spam filters).

False negatives are expensive (e.g.

Recall Of actual positives, how many you caught.
disease screening).

You need a single balanced score

F1 Harmonic mean of precision and recall.
on imbalanced data.

Comparing probabilistic classifiers

ROC-AUC Ranking quality across all thresholds.
threshold-free.

4
FOUNDATIONS SUPERVISED LEARNING · DETAILED GUIDE

Metric Meaning Use when

Well-calibrated probabilities
Log loss Penalises confident wrong probabilities.
matter.

How to read each chapter that follows

Every algorithm is presented the same way: the intuition first, then how it works with
the key equations, the hyperparameters you will actually tune, an honest strengths /
limitations comparison, guidance on when to reach for it, and a minimal scikit-learn
snippet you can adapt.

5
REGRESSION ALGORITHMS SUPERVISED LEARNING · DETAILED GUIDE

PART I

Regression
Predicting a continuous number — from the straight-line
baseline to the boosted-tree champion.

1 Linear Regression
2 Ridge Regression
3 Lasso Regression
4 Elastic Net
5 Decision Tree Regressor
6 Random Forest Regressor
7 Gradient Boosting / XGBoost

6
REGRESSION ALGORITHMS SUPERVISED LEARNING · DETAILED GUIDE

1 Linear Regression
The straight-line baseline every regression model is measured against

Linear regression is the foundation of the entire family. It models the target as a
weighted sum of the features plus an intercept, then chooses the weights that make
the squared prediction errors as small as possible.

Intuition
Imagine a cloud of points and you want the single straight line that passes through them as
snugly as possible. With more than one feature the line becomes a flat plane or hyperplane,
but the idea is unchanged: find the slope(s) and intercept that minimise the total squared
vertical distance between the line and the points.

How it works
The model assumes the prediction is linear in the parameters:

Training minimises the residual sum of squares, equivalently the mean squared error,
over the m training examples. This loss is called Ordinary Least Squares (OLS):

Because the loss is convex and quadratic, there is a unique optimum. It can be found in one
step with the normal equation, or iteratively with gradient descent for very large
datasets:

Key assumptions
• Linearity — the relationship between features and target is genuinely linear.
• Independence of errors and homoscedasticity (constant error variance).
• Little multicollinearity — features are not near-duplicates of each other.
• Roughly normal residuals if you want valid confidence intervals and p-values.

Why these assumptions matter

Violations don't necessarily break prediction, but they invalidate the statistical
interpretation (coefficient significance, intervals). When features are highly correlated,
OLS coefficients become unstable — which is exactly the problem Ridge and Lasso were
built to solve.

Hyperparameters

7
REGRESSION ALGORITHMS SUPERVISED LEARNING · DETAILED GUIDE

Hyperparameter What it controls

Whether to learn an intercept term β₀. Leave on unless data is

fit_intercept
pre-centred.

Plain OLS has no regularisation strength to tune — that is what the

(none for OLS)
next three models add.

✓ Strengths ✗ Limitations
▸ Fast, closed-form solution and trivial to ▸ Can only capture linear relationships.
train. ▸ Sensitive to outliers because errors are
▸ Highly interpretable — each coefficient is a squared.
direct effect size. ▸ Unstable coefficients under
▸ Strong, well-understood statistical theory. multicollinearity.
▸ A natural baseline for any regression task. ▸ Underfits complex, non-linear data.

When to use it
Reach for linear regression as your first baseline, when interpretability matters more than
raw accuracy, or when the relationship really is approximately linear and you have more
rows than features.

from sklearn.linear_model import LinearRegression

from sklearn.model_selection import train_test_split

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2)

model = LinearRegression()
[Link](X_train, y_train)
print(model.coef_, model.intercept_)
print('R2:', [Link](X_test, y_test))

8
REGRESSION ALGORITHMS SUPERVISED LEARNING · DETAILED GUIDE

2 Ridge Regression
Linear regression + L2 penalty for stability under correlated features

Ridge regression is linear regression with an L2 penalty. It keeps every feature but
shrinks the coefficients toward zero, trading a little bias for a large reduction in
variance — especially valuable when features are correlated.

Intuition
When predictors are correlated, OLS can assign wildly large, oscillating coefficients that fit
noise. Ridge adds a cost for large coefficients, so the optimiser prefers small, stable
weights. The result is a model that generalises better even though it fits the training data
slightly worse.

How it works
Ridge adds the squared L2 norm of the coefficients (excluding the intercept) to the OLS
loss, scaled by a strength α (often written λ):

It still has a closed-form solution, and the added αI term guarantees the matrix is always
invertible — curing the instability of pure OLS:

The effect of α
At α = 0 Ridge is identical to OLS. As α grows, all coefficients shrink smoothly toward (but
never exactly to) zero, biasing the model and reducing variance. You tune α with
cross-validation; RidgeCV does this automatically. Always scale features first, since the
penalty is sensitive to their magnitudes.

Hyperparameters
Hyperparameter What it controls

Regularisation strength. Higher = more shrinkage, more bias, less

alpha
variance.

Optimisation method (auto, cholesky, lsqr, sag) — rarely needs

solver
changing.

fit_intercept Whether to fit the intercept; the intercept is never penalised.

9
REGRESSION ALGORITHMS SUPERVISED LEARNING · DETAILED GUIDE

✓ Strengths ✗ Limitations
▸ Tames multicollinearity and stabilises ▸ Never sets coefficients exactly to zero —
coefficients. no feature selection.
▸ Reduces overfitting via a single, intuitive ▸ Keeps all features, so the model stays
knob. dense and less interpretable.
▸ Retains the closed-form, fast solution of ▸ Requires feature scaling to behave
OLS. sensibly.
▸ Works well when many features each ▸ Still fundamentally a linear model.
contribute a little.

When to use it
Use Ridge when you have many correlated features that you believe are all relevant, when
OLS coefficients look unstable, or whenever you want a gentle, reliable reduction in
overfitting without dropping any predictors.

from sklearn.linear_model import RidgeCV

from [Link] import make_pipeline
from [Link] import StandardScaler

model = make_pipeline(
StandardScaler(),
RidgeCV(alphas=[0.1, 1.0, 10.0, 100.0]) # picks best alpha by CV
)
[Link](X_train, y_train)
print('R2:', [Link](X_test, y_test))

10
REGRESSION ALGORITHMS SUPERVISED LEARNING · DETAILED GUIDE

3 Lasso Regression
Linear regression + L1 penalty that performs automatic feature selection

Lasso regression swaps Ridge's squared penalty for an absolute-value (L1) penalty.
That single change lets it drive some coefficients exactly to zero, performing
automatic feature selection.

Intuition
The L1 penalty has sharp corners on the coefficient axes. Geometrically, the optimum
tends to land on those corners, where one or more coefficients are exactly zero. The
practical upshot: Lasso produces sparse models that use only a subset of the features,
which is invaluable when you suspect many features are irrelevant.

How it works
Lasso minimises the OLS loss plus the L1 norm of the coefficients:

Because the absolute value is not differentiable at zero, there is no closed form. scikit-learn
solves it with coordinate descent, optimising one coefficient at a time while holding the
others fixed and repeating until convergence.

Lasso vs. Ridge in one line

Ridge shrinks all coefficients toward zero but keeps them; Lasso shrinks and eliminates
the weak ones. If you want a smaller, more interpretable model, Lasso. If you want to
keep every feature but stabilise it, Ridge.

Hyperparameters
Hyperparameter What it controls

Strength of the L1 penalty. Higher = more coefficients forced to

alpha
zero = sparser model.

Iterations for coordinate descent; raise it if you get a convergence

max_iter
warning.

'cyclic' or 'random' coordinate updates; 'random' can converge

selection
faster.

11
REGRESSION ALGORITHMS SUPERVISED LEARNING · DETAILED GUIDE

✓ Strengths ✗ Limitations
▸ Built-in feature selection — yields sparse, ▸ With correlated features it arbitrarily keeps
interpretable models. one and zeros the rest.
▸ Excellent when only a few features truly ▸ Can be unstable: small data changes flip
matter. which features survive.
▸ Reduces overfitting and model size at ▸ No closed form; slower to fit than Ridge.
once. ▸ May discard useful features if α is too
▸ LassoCV tunes α automatically. aggressive.

When to use it
Reach for Lasso in high-dimensional settings where you expect many features to be useless
and want the model to tell you which ones, or whenever a compact, interpretable set of
predictors is the goal.

from sklearn.linear_model import LassoCV

from [Link] import make_pipeline
from [Link] import StandardScaler

model = make_pipeline(StandardScaler(), LassoCV(cv=5))

[Link](X_train, y_train)

lasso = model.named_steps['lassocv']
print('best alpha:', lasso.alpha_)
print('features kept:', (lasso.coef_ != 0).sum())

12
REGRESSION ALGORITHMS SUPERVISED LEARNING · DETAILED GUIDE

4 Elastic Net
A tunable blend of Lasso and Ridge — sparsity with stability

Elastic Net blends the L1 and L2 penalties, combining Lasso's feature selection with
Ridge's stability. It is the default choice when you have many correlated features
and still want a sparse model.

Intuition
Lasso struggles with groups of correlated features — it picks one and discards the rest,
almost at random. Ridge keeps the whole group but never zeros anything. Elastic Net takes
the best of both: the L1 part still removes irrelevant features, while the L2 part encourages
correlated features to be selected (or dropped) together as a group.

How it works
The loss mixes both penalties, with a ratio that decides how much of each:

Here α sets the overall regularisation strength and ρ (scikit-learn's l1_ratio) sets the
balance: ρ = 1 is pure Lasso, ρ = 0 is pure Ridge, and values in between blend the two.

The grouping effect

Elastic Net's signature behaviour is the grouping effect: strongly correlated predictors get
similar coefficients and tend to enter or leave the model together. This makes it far more
stable than Lasso on real-world data, where correlated features are the norm rather than
the exception.

Hyperparameters
Hyperparameter What it controls

alpha Overall regularisation strength (combined L1 + L2).

Mix between L1 and L2: 0 = Ridge, 1 = Lasso, 0.5 = balanced

l1_ratio
blend.

max_iter Coordinate-descent iterations; increase on convergence warnings.

13
REGRESSION ALGORITHMS SUPERVISED LEARNING · DETAILED GUIDE

✓ Strengths ✗ Limitations
▸ Combines feature selection (L1) with ▸ Two hyperparameters to tune instead of
stability (L2). one.
▸ Handles groups of correlated features ▸ Still a linear model — cannot capture
gracefully. non-linearity.
▸ Usually the safest regularised-linear ▸ Slightly less interpretable than pure Lasso.
default. ▸ Requires feature scaling.
▸ ElasticNetCV tunes both α and l1_ratio
together.

When to use it
Choose Elastic Net when you have high-dimensional data with correlated features and want
both sparsity and stability — it is often the best-performing linear model in practice and a
sensible default over plain Lasso.

from sklearn.linear_model import ElasticNetCV

from [Link] import make_pipeline
from [Link] import StandardScaler

model = make_pipeline(
StandardScaler(),
ElasticNetCV(l1_ratio=[0.1, 0.5, 0.7, 0.9, 1.0], cv=5)
)
[Link](X_train, y_train)

14
REGRESSION ALGORITHMS SUPERVISED LEARNING · DETAILED GUIDE

5 Decision Tree Regressor

Recursive yes/no splits that carve the data into predictable regions

A decision tree regressor splits the feature space into rectangular regions with a
series of yes/no questions, then predicts the average target value within each
region. It captures non-linear relationships with zero assumptions about their shape.

Intuition
Think of a flowchart of questions: Is square footage > 1500? If yes, is the house newer than
1990? Each answer narrows the data down. The leaves at the bottom hold groups of similar
training points, and the prediction for any new point is simply the mean target of the leaf it
lands in.

How it works
The tree is grown greedily. At each node it searches every feature and every possible
threshold to find the split that most reduces the impurity — for regression, the variance
(mean squared error) of the target within the child nodes:

Splitting continues recursively until a stopping rule fires (maximum depth reached, too few
samples to split, or no split improves the loss). Without limits, a tree will grow until each
leaf is nearly pure — which means it memorises the training set.

Trees overfit by default

A single unconstrained tree is a textbook high-variance model: it can fit the training data
almost perfectly and generalise badly. The cure is pruning — limiting depth, requiring a
minimum number of samples per leaf, or using cost-complexity pruning (ccp_alpha). This
high variance is also exactly why ensembles of trees work so well.

Hyperparameters
Hyperparameter What it controls

Maximum number of question levels. The primary control on

max_depth
overfitting.

min_samples_split Minimum samples a node must have to be split further.

min_samples_leaf Minimum samples allowed in a leaf; smooths predictions.

max_features How many features to consider at each split.

Cost-complexity pruning strength; prunes weak branches after

ccp_alpha
growing.

15
REGRESSION ALGORITHMS SUPERVISED LEARNING · DETAILED GUIDE

✓ Strengths ✗ Limitations
▸ Captures non-linear patterns and ▸ High variance — prone to overfitting
interactions automatically. without pruning.
▸ Needs no feature scaling and handles ▸ Predictions are piecewise-constant
mixed data types. (blocky), not smooth.
▸ Highly interpretable — you can read the ▸ Small data changes can produce a very
rules directly. different tree.
▸ Robust to outliers and irrelevant features. ▸ A single tree rarely matches ensemble
accuracy.

When to use it
Use a single tree when interpretability is paramount and you can show the rules to
stakeholders, or as a building block and intuition pump before moving to Random Forests
and Gradient Boosting, which almost always predict better.

from [Link] import DecisionTreeRegressor

model = DecisionTreeRegressor(
max_depth=5,
min_samples_leaf=10,
random_state=42
)
[Link](X_train, y_train)
print('R2:', [Link](X_test, y_test))

16
REGRESSION ALGORITHMS SUPERVISED LEARNING · DETAILED GUIDE

6 Random Forest Regressor

A committee of decorrelated trees averaged for low-variance accuracy

A random forest is an ensemble of many decision trees whose predictions are

averaged. By training each tree on a different random slice of data and features, it
cancels out the high variance of individual trees and produces a far more accurate,
stable model.

Intuition
One tree is a noisy expert. A forest is a committee of hundreds of decorrelated experts:
each makes its own errors, but when you average their votes the errors cancel and the
signal remains. This is the wisdom-of-crowds principle applied to trees.

How it works
Two sources of randomness make the trees different from each other:

1. Bagging (bootstrap aggregating) — each tree is trained on a bootstrap sample: a

random draw of the training rows with replacement.
2. Feature subsampling — at each split, only a random subset of features is
considered, which decorrelates the trees so they don't all key on the same dominant
feature.

For regression, the final prediction is the average over all trees:

Out-of-bag (OOB) validation

Because each tree omits about a third of the rows (the bootstrap leaves them out), those
rows act as a free validation set for that tree. Averaging predictions on the left-out rows
gives an OOB score — a built-in estimate of test error without needing a separate
hold-out set.

Hyperparameters
Hyperparameter What it controls

Number of trees. More is better (and never overfits) until returns

n_estimators
flatten.

Features considered per split — the key knob for decorrelating

max_features
trees.

Depth limit per tree; forests tolerate deep trees better than a lone
max_depth
tree.

min_samples_leaf Minimum samples per leaf; larger values smooth predictions.

17
REGRESSION ALGORITHMS SUPERVISED LEARNING · DETAILED GUIDE

Hyperparameter What it controls

bootstrap Whether to sample rows with replacement (enables OOB scoring).

✓ Strengths ✗ Limitations
▸ Strong accuracy out of the box with little ▸ A black box compared with a single tree.
tuning. ▸ Large models are memory-heavy and
▸ Resists overfitting thanks to averaging. slower to predict.
▸ Handles non-linearity, interactions, and ▸ Can't extrapolate beyond the range seen in
mixed types. training.
▸ Provides feature-importance scores and ▸ Often edged out on accuracy by tuned
OOB estimates. gradient boosting.
▸ Parallelises trivially across cores.

When to use it
Random forest is the dependable workhorse: an excellent default when you want strong
accuracy with minimal tuning, robust behaviour on messy tabular data, and useful feature
importances — all without worrying much about scaling or hyperparameters.

from [Link] import RandomForestRegressor

model = RandomForestRegressor(
n_estimators=300,
max_features='sqrt',
min_samples_leaf=2,
oob_score=True,
n_jobs=-1, random_state=42
)
[Link](X_train, y_train)
print('OOB R2:', model.oob_score_)

18
REGRESSION ALGORITHMS SUPERVISED LEARNING · DETAILED GUIDE

7 Gradient Boosting & XGBoost

Trees built sequentially, each correcting the last — the tabular champion

Gradient boosting builds an ensemble of trees sequentially, where each new tree
is trained to correct the errors of the trees before it. XGBoost, LightGBM and
CatBoost are highly optimised, regularised implementations of this idea — and they
dominate tabular-data competitions.

Intuition
Where a random forest builds many independent trees in parallel and averages them,
boosting builds trees one after another. Each tree focuses on the mistakes still left by the
running ensemble. Add enough small corrective trees, each scaled down by a learning rate,
and the model gradually converges on the target — turning many weak learners into one
strong learner.

How it works
The model is an additive sum of trees. Starting from a constant prediction, each stage fits a
new tree to the negative gradient of the loss (for squared error, this is simply the residuals)
and adds a shrunken version of it:

The learning rate ν (often 0.01–0.3) shrinks each tree's contribution. Smaller rates need
more trees but generalise better. The interplay between learning rate and number of trees
is the heart of tuning a boosted model.

What XGBoost / LightGBM / CatBoost add

All three are gradient boosting under the hood but add speed and accuracy: XGBoost
uses second-order gradients and built-in L1/L2 regularisation; LightGBM grows trees
leaf-wise with histogram binning for blazing speed on large data; CatBoost handles
categorical features natively and resists a subtle form of overfitting called target leakage.
For most tabular problems one of these is the accuracy leader.

Hyperparameters
Hyperparameter What it controls

Number of boosting rounds (trees). Pairs inversely with

n_estimators
learning_rate.

learning_rate Shrinkage per tree; lower is more accurate but needs more trees.

max_depth Tree depth; boosted trees are usually shallow (3–8).

Fraction of rows sampled per tree — stochastic boosting, fights

subsample
overfitting.

colsample_bytree Fraction of features sampled per tree.

19
REGRESSION ALGORITHMS SUPERVISED LEARNING · DETAILED GUIDE

Hyperparameter What it controls

reg_lambda / reg_alpha L2 / L1 regularisation on leaf weights (XGBoost).

✓ Strengths ✗ Limitations
▸ State-of-the-art accuracy on structured / ▸ Many interacting hyperparameters —
tabular data. needs careful tuning.
▸ Captures complex non-linear interactions. ▸ Sequential training is slower and harder to
▸ Handles missing values and gives feature parallelise than forests.
importances. ▸ Easy to overfit if learning rate / trees are
▸ Early stopping and regularisation control mis-set.
overfitting well. ▸ Less interpretable; effectively a black box.

When to use it
Use gradient boosting when accuracy on tabular data is the priority and you are willing to
tune. It is the go-to choice for Kaggle-style problems, ranking, and most high-stakes
structured-data prediction — with early stopping to find the right number of trees
automatically.

from xgboost import XGBRegressor

model = XGBRegressor(
n_estimators=2000, learning_rate=0.03,
max_depth=4, subsample=0.8, colsample_bytree=0.8,
reg_lambda=1.0, early_stopping_rounds=50, random_state=42
)
[Link](X_train, y_train,
eval_set=[(X_test, y_test)], verbose=False)

20
CLASSIFICATION ALGORITHMS SUPERVISED LEARNING · DETAILED GUIDE

PART II

Classification
Predicting a category — from interpretable linear models
to margins, neighbours, and ensembles.

1 Logistic Regression
2 Decision Tree Classifier
3 Random Forest Classifier
4 XGBoost / LightGBM / CatBoost
5 Support Vector Machine (SVM)
6 Naive Bayes
7 K-Nearest Neighbours (KNN)

21
CLASSIFICATION ALGORITHMS SUPERVISED LEARNING · DETAILED GUIDE

1 Logistic Regression
A linear model squashed into probabilities — the classification baseline

Despite its name, logistic regression is a classification algorithm. It models the

probability that an example belongs to a class by passing a linear combination of
the features through the S-shaped sigmoid function.

Intuition
Linear regression can output any number, which is useless for a probability that must live
between 0 and 1. Logistic regression fixes this by squashing the linear output through the
sigmoid, producing a clean probability. You then threshold that probability (typically at 0.5)
to decide the class.

How it works
First compute a linear score, then squash it with the sigmoid to get a probability:

The model is trained by minimising log loss (binary cross-entropy), which heavily penalises
confident wrong predictions:

This loss is convex but has no closed form, so it is solved with gradient descent or
specialised solvers. The decision boundary is linear: the model separates classes with a
straight line, plane, or hyperplane. For more than two classes it generalises to the softmax
(multinomial) form.

Regularisation is built in
scikit-learn's LogisticRegression applies L2 regularisation by default, controlled by C —
the inverse of strength, so smaller C means stronger regularisation. You can switch to L1
for sparsity or Elastic Net. Scaling features helps the solver converge.

Hyperparameters
Hyperparameter What it controls

Inverse regularisation strength. Smaller C = stronger penalty =

C
simpler model.

penalty Type of regularisation: 'l2' (default), 'l1', or 'elasticnet'.

solver Optimiser (lbfgs, liblinear, saga); some pair with specific penalties.

class_weight Set to 'balanced' to counteract class imbalance.

22
CLASSIFICATION ALGORITHMS SUPERVISED LEARNING · DETAILED GUIDE

✓ Strengths ✗ Limitations
▸ Outputs calibrated probabilities, not just ▸ Decision boundary is linear — misses
labels. complex patterns.
▸ Fast, simple, and highly interpretable ▸ Needs feature engineering to model
(coefficients = log-odds). non-linear effects.
▸ Strong, hard-to-beat baseline for ▸ Sensitive to outliers and strong
classification. multicollinearity.
▸ Regularisation guards against overfitting. ▸ Assumes a roughly linear relationship in
the log-odds.

When to use it
Use logistic regression as your first classification baseline, whenever you need
interpretable probabilities (credit scoring, medical risk), or when the classes are close to
linearly separable. It is fast enough to train on very large datasets.

from sklearn.linear_model import LogisticRegression

from [Link] import make_pipeline
from [Link] import StandardScaler

model = make_pipeline(
StandardScaler(),
LogisticRegression(C=1.0, class_weight='balanced', max_iter=1000)
)
[Link](X_train, y_train)
proba = model.predict_proba(X_test)[:, 1]

23
CLASSIFICATION ALGORITHMS SUPERVISED LEARNING · DETAILED GUIDE

2 Decision Tree Classifier

Yes/no splits that sort examples into pure, majority-class leaves

The classification tree works exactly like its regression cousin, but instead of
predicting an average it predicts the majority class of each leaf and chooses splits
that make the leaves as pure as possible.

Intuition
Ask a sequence of yes/no questions that progressively sort examples into groups
dominated by a single class. A new example follows the questions down to a leaf and is
assigned that leaf's majority class — or the class proportions as probabilities.

How it works
At each node the tree picks the split that most reduces impurity. The two common
impurity measures for classification are Gini and entropy:

Here pk is the fraction of class k in the node. A node with one class has impurity 0; a 50/50
mix is maximally impure. The tree greedily chooses the split giving the largest weighted
drop in impurity (the information gain), then recurses.

Gini or entropy?
In practice the two give very similar trees. Gini is slightly faster to compute and is
scikit-learn's default; entropy comes from information theory. Don't agonise over the
choice — depth and leaf-size limits matter far more for performance.

Hyperparameters
Hyperparameter What it controls

criterion 'gini' or 'entropy' (also 'log_loss') — the impurity measure.

max_depth Tree depth; the main lever against overfitting.

min_samples_leaf Minimum samples per leaf; larger values generalise better.

class_weight Weight classes to handle imbalance (e.g. 'balanced').

ccp_alpha Cost-complexity pruning to trim weak branches.

24
CLASSIFICATION ALGORITHMS SUPERVISED LEARNING · DETAILED GUIDE

✓ Strengths ✗ Limitations
▸ Easy to visualise and explain to ▸ Overfits readily without depth or leaf limits.
non-experts. ▸ Unstable — small data changes reshape
▸ Handles non-linear boundaries and feature the tree.
interactions. ▸ Biased toward features with many levels.
▸ No scaling needed; works with mixed ▸ A single tree usually trails ensembles in
feature types. accuracy.
▸ Naturally provides class probabilities.

When to use it
Use a single classification tree when transparency is the priority and you must justify each
decision, for quick exploratory insight into which features drive the outcome, or as the
conceptual stepping stone to forests and boosting.

from [Link] import DecisionTreeClassifier

model = DecisionTreeClassifier(
criterion='gini', max_depth=5,
min_samples_leaf=10, class_weight='balanced',
random_state=42
)
[Link](X_train, y_train)

25
CLASSIFICATION ALGORITHMS SUPERVISED LEARNING · DETAILED GUIDE

3 Random Forest Classifier

Hundreds of decorrelated trees voting for robust accuracy

A random forest classifier trains many decorrelated decision trees and lets them
vote. The majority class wins, and averaging the trees' probabilities gives smooth,
reliable predictions with strong out-of-the-box accuracy.

Intuition
Each individual tree is an overconfident, noisy classifier. Train hundreds of them on
different bootstrap samples and feature subsets, and their idiosyncratic mistakes cancel
when you take a vote. The forest is dramatically more stable and accurate than any single
tree.

How it works
The mechanics mirror the regression forest — bagging plus random feature selection at
each split — but aggregation differs. For a class label the forest takes a majority vote; for
probabilities it averages each tree's class proportions:

Why feature subsampling matters

If one feature is very predictive, every tree would split on it first and the trees would look
nearly identical — averaging clones gains nothing. By forcing each split to consider only a
random subset of features (commonly √n features for classification), the forest
decorrelates its trees, and decorrelation is what makes the averaging powerful.

Hyperparameters
Hyperparameter What it controls

n_estimators Number of trees; more improves stability with diminishing returns.

max_features Features per split (default √n); the key decorrelation knob.

max_depth Per-tree depth limit; often left unbounded in forests.

min_samples_leaf Minimum samples per leaf to control tree complexity.

class_weight 'balanced' or 'balanced_subsample' for skewed classes.

26
CLASSIFICATION ALGORITHMS SUPERVISED LEARNING · DETAILED GUIDE

✓ Strengths ✗ Limitations
▸ High accuracy with minimal tuning — a ▸ Less interpretable than a single tree.
superb default. ▸ Memory- and compute-heavy at prediction
▸ Resists overfitting through averaging. time.
▸ Robust to outliers, noise, and irrelevant ▸ Can be biased on highly imbalanced data
features. without weighting.
▸ Gives feature importances and OOB error ▸ Frequently out-predicted by tuned gradient
estimates. boosting.
▸ Handles many classes and large feature
sets.

When to use it
Random forest is the reliable first choice for most tabular classification tasks: it delivers
strong accuracy with almost no tuning, tolerates messy data, and surfaces feature
importances — ideal when you need a solid model fast.

from [Link] import RandomForestClassifier

model = RandomForestClassifier(
n_estimators=400, max_features='sqrt',
class_weight='balanced', oob_score=True,
n_jobs=-1, random_state=42
)
[Link](X_train, y_train)
print('OOB accuracy:', model.oob_score_)

27
CLASSIFICATION ALGORITHMS SUPERVISED LEARNING · DETAILED GUIDE

4 XGBoost / LightGBM / CatBoost

Sequential boosted trees optimising log loss — the tabular champions

For classification, gradient-boosted trees optimise log loss instead of squared error,
but the engine is identical: trees added one at a time, each correcting the
ensemble's current errors. XGBoost, LightGBM and CatBoost are the dominant
tabular classifiers.

Intuition
Start with a constant guess at the class probabilities, then repeatedly add small trees that
push the predicted probabilities toward the truth. Each tree fits the gradient of the log-loss
— effectively the residual error in probability space — and a learning rate keeps each
correction modest so the model converges smoothly.

How it works
The additive update is the same as in regression, only the loss changes to log loss. The raw
additive output is a log-odds score, converted to a probability by the sigmoid (or softmax
for multi-class):

Picking the right implementation

XGBoost — the proven all-rounder with strong regularisation. LightGBM — fastest on
large datasets thanks to leaf-wise, histogram-based growth; watch overfitting on small
data. CatBoost — best when you have many categorical features, which it encodes
natively with minimal preprocessing. All support early stopping on a validation set to
choose the number of trees.

Hyperparameters
Hyperparameter What it controls

Boosting rounds; tuned together with learning_rate (use early

n_estimators
stopping).

learning_rate Shrinkage; lower needs more trees but generalises better.

max_depth / num_leaves Tree complexity (depth for XGBoost, leaves for LightGBM).

subsample Row sampling per tree for stochastic boosting.

colsample_bytree Feature sampling per tree.

scale_pos_weight Up-weights the positive class for imbalanced data.

28
CLASSIFICATION ALGORITHMS SUPERVISED LEARNING · DETAILED GUIDE

✓ Strengths ✗ Limitations
▸ Best-in-class accuracy on structured / ▸ Many hyperparameters; tuning takes
tabular data. effort.
▸ Models complex non-linear interactions ▸ Sequential training is slower than a forest.
automatically. ▸ Overfits if learning rate / depth are mis-set.
▸ Handles missing values and imbalance ▸ Low interpretability without tools like
well. SHAP.
▸ Early stopping and regularisation control
overfitting.

When to use it
Use boosted trees when classification accuracy on tabular data is what matters most and
you can invest in tuning. They win the majority of structured-data competitions and excel
at fraud detection, churn, ranking, and risk scoring.

from xgboost import XGBClassifier

model = XGBClassifier(
n_estimators=2000, learning_rate=0.03, max_depth=4,
subsample=0.8, colsample_bytree=0.8,
eval_metric='logloss', early_stopping_rounds=50,
random_state=42
)
[Link](X_train, y_train,
eval_set=[(X_test, y_test)], verbose=False)

29
CLASSIFICATION ALGORITHMS SUPERVISED LEARNING · DETAILED GUIDE

5 Support Vector Machine

The maximum-margin separator, kernelised for non-linear boundaries

A Support Vector Machine finds the decision boundary that separates the classes
with the widest possible margin. With the kernel trick it can draw highly
non-linear boundaries while still solving a clean, convex optimisation.

Intuition
Many lines can separate two classes, but SVM picks the one that sits as far as possible from
the nearest points of either class. Those nearest points are the support vectors — they
alone define the boundary. A wide margin means the model is confident and tends to
generalise well.

How it works
SVM maximises the margin, which is equivalent to minimising the norm of the weight
vector subject to classifying points correctly. A slack term (controlled by C) allows some
violations for non-separable data, giving the soft-margin formulation:

The second term is the hinge loss. For non-linear problems the kernel trick implicitly
maps data into a higher-dimensional space where it becomes separable, using a kernel
function such as the RBF (Gaussian) kernel — without ever computing the mapping
explicitly:

The two knobs that define an SVM

C trades margin width against training errors: small C = wider margin, more tolerance,
simpler model; large C = fits training data harder. gamma (RBF) sets how far one
example's influence reaches: large gamma = tight, wiggly boundaries (risk of overfitting);
small gamma = smooth boundaries. Always scale features first.

Hyperparameters
Hyperparameter What it controls

Regularisation: low = wide soft margin, high = fit training data

C
harder.

kernel 'linear', 'rbf', 'poly', 'sigmoid' — shape of the boundary.

Reach of each point for RBF/poly kernels; controls boundary

gamma
curviness.

class_weight 'balanced' to handle skewed classes.

30
CLASSIFICATION ALGORITHMS SUPERVISED LEARNING · DETAILED GUIDE

✓ Strengths ✗ Limitations
▸ Effective in high-dimensional spaces, even ▸ Scales poorly — slow on large datasets
when features > samples. (roughly O(m²–m³)).
▸ Kernel trick captures complex non-linear ▸ Sensitive to C, gamma, and kernel choice;
boundaries. needs tuning.
▸ Margin maximisation gives good ▸ Requires feature scaling to work properly.
generalisation. ▸ Probabilities are not native (need extra
▸ Only support vectors matter — calibration).
memory-efficient model.

When to use it
SVMs shine on small-to-medium datasets with clear margins, especially high-dimensional
ones like text classification or bioinformatics. A linear SVM is a strong, fast choice for sparse
high-dimensional data; the RBF kernel handles non-linear problems when you have the
time to tune.

from [Link] import SVC

from [Link] import make_pipeline
from [Link] import StandardScaler

model = make_pipeline(
StandardScaler(),
SVC(kernel='rbf', C=1.0, gamma='scale',
class_weight='balanced', probability=True)
)
[Link](X_train, y_train)

31
CLASSIFICATION ALGORITHMS SUPERVISED LEARNING · DETAILED GUIDE

6 Naive Bayes
Bayes' theorem plus a bold independence assumption — fast and
surprisingly good

Naive Bayes is a probabilistic classifier built directly on Bayes' theorem, with one
bold simplifying assumption: that all features are conditionally independent given
the class. That 'naive' assumption is rarely true, yet the method works remarkably
well.

Intuition
For a new example, ask: given these feature values, which class is most probable? Bayes'
theorem lets you flip that into quantities you can estimate from training data — how likely
each feature value is within each class. Multiply those likelihoods together with the class's
base rate, and the largest product wins.

How it works
Bayes' theorem says the posterior probability of a class is proportional to the prior times
the likelihood. The naive independence assumption lets the joint likelihood factor into a
simple product over features:

Each P(xj | y) is estimated from the training data; the form depends on the feature type,
which gives the three common variants. Probabilities are multiplied in log-space to avoid
numerical underflow, and Laplace smoothing handles unseen values.

• Gaussian NB — continuous features, modelled with a normal distribution per class.

• Multinomial NB — count data such as word frequencies; the classic text classifier.
• Bernoulli NB — binary present/absent features.

Why 'naive' still works

Real features are usually correlated, violating the independence assumption. But Naive
Bayes only needs to rank the classes correctly, not estimate exact probabilities — and the
ranking is often right even when the assumption is wrong. Its probability estimates,
however, tend to be poorly calibrated (over-confident).

Hyperparameters
Hyperparameter What it controls

Laplace/Lidstone smoothing for Multinomial/Bernoulli; prevents zero

alpha
probabilities.

var_smoothing Stability term added to variances in Gaussian NB.

fit_prior Whether to learn class priors from data or assume them uniform.

32
CLASSIFICATION ALGORITHMS SUPERVISED LEARNING · DETAILED GUIDE

✓ Strengths ✗ Limitations
▸ Extremely fast to train and predict, even ▸ The independence assumption is usually
on huge data. false.
▸ Works well with very high-dimensional ▸ Probability estimates are poorly calibrated.
data (e.g. text). ▸ Correlated features can degrade accuracy.
▸ Needs little training data to estimate ▸ Generally less accurate than discriminative
parameters. models when data is plentiful.
▸ Simple, with no real tuning required.

When to use it
Naive Bayes is the classic choice for text classification — spam filtering, sentiment,
document categorisation — and any high-dimensional, sparse problem where speed
matters and a strong, cheap baseline is welcome.

from sklearn.feature_extraction.text import TfidfVectorizer

from sklearn.naive_bayes import MultinomialNB
from [Link] import make_pipeline

model = make_pipeline(
TfidfVectorizer(),
MultinomialNB(alpha=1.0)
)
[Link](text_train, y_train)

33
CLASSIFICATION ALGORITHMS SUPERVISED LEARNING · DETAILED GUIDE

7 K-Nearest Neighbours
Lazy, instance-based voting by the closest examples

K-Nearest Neighbours is the simplest classifier of all: to label a new point, find the k
closest training points and let them vote. It does no real training — it just memorises
the data and defers all work to prediction time.

Intuition
Things that are close together tend to be alike. To classify a new example, look at its k
nearest neighbours in feature space and assign whichever class is most common among
them. There is no model to fit — the training data is the model, which is why KNN is called a
lazy or instance-based learner.

How it works
Pick a value of k and a distance metric (Euclidean is standard). For a query point, compute
its distance to every training point, take the k smallest, and predict the majority class
among them — optionally weighting closer neighbours more heavily:

Choosing k and scaling features

Small k (e.g. 1) gives a jagged boundary that overfits noise; large k oversmooths and
underfits. Tune k by cross-validation, and prefer odd k to break ties in binary problems.
Because KNN relies on distances, feature scaling is mandatory — otherwise a
large-range feature dominates the metric and swamps the others.

Hyperparameters
Hyperparameter What it controls

n_neighbors The k — how many neighbours vote. The central tuning parameter.

weights 'uniform' (equal votes) or 'distance' (closer neighbours count more).

metric Distance function: Euclidean, Manhattan, Minkowski, etc.

algorithm How neighbours are found: brute, kd_tree, or ball_tree.

34
CLASSIFICATION ALGORITHMS SUPERVISED LEARNING · DETAILED GUIDE

✓ Strengths ✗ Limitations
▸ Dead simple and intuitive, with no training ▸ Prediction is slow — it scans all data each
phase. time.
▸ Naturally handles non-linear, multi-class ▸ Suffers badly from the curse of
boundaries. dimensionality.
▸ Makes no assumptions about the data ▸ Mandatory feature scaling; sensitive to
distribution. irrelevant features.
▸ Adapts instantly as new data is added. ▸ Memory-hungry: the whole training set
must be stored.

When to use it
KNN suits small, low-dimensional datasets with meaningful distances, as a quick baseline,
or for recommendation-style 'find similar items' tasks. Avoid it on large or high-dimensional
data, where prediction becomes slow and distances lose meaning.

from [Link] import KNeighborsClassifier

from [Link] import make_pipeline
from [Link] import StandardScaler

model = make_pipeline(
StandardScaler(),
KNeighborsClassifier(n_neighbors=7, weights='distance')
)
[Link](X_train, y_train)

35
CHEAT-SHEET SUPERVISED LEARNING · DETAILED GUIDE

A Cheat-Sheet & Model Selection

Everything on one page — plus how to decide what to reach for

Use these tables as a fast lookup once you know the algorithms. The golden rule of
applied machine learning still holds: start simple, establish a baseline, and only add
complexity when the data justifies it.

Regression algorithms at a glance

Algorithm Core idea Reach for it when…

Linear You want a fast, interpretable linear

Fit a line by least squares.
Regression baseline.

Linear + penalty on squared Features are correlated and you want

Ridge (L2)
coefficients. all of them, stably.

Linear + penalty that zeros weak You want automatic feature selection
Lasso (L1)
coefficients. and a sparse model.

Many correlated features and you

Elastic Net Blend of L1 and L2 penalties.
want sparsity + stability.

Recursive splits predicting leaf You need a transparent, non-linear

Decision Tree
means. single model.

Average of many decorrelated You want strong accuracy with

Random Forest
trees. minimal tuning.

Gradient Sequential trees correcting Accuracy on tabular data is the priority

Boosting residuals. and you can tune.

Classification algorithms at a glance

Algorithm Core idea Reach for it when…

Logistic You need interpretable probabilities

Linear score through a sigmoid.
Regression and a baseline.

Splits into pure, majority-class Transparency and explainable rules

Decision Tree
leaves. matter most.

You want robust accuracy on tabular

Random Forest Vote of many decorrelated trees.
data fast.

Sequential trees minimising log You want top accuracy and can invest
Boosted Trees
loss. in tuning.

Maximum-margin (kernelised) Small / high-dimensional data with

SVM
boundary. clear margins.

Bayes' rule with feature Text or sparse high-dimensional data;

Naive Bayes
independence. speed matters.

36
CHEAT-SHEET SUPERVISED LEARNING · DETAILED GUIDE

Algorithm Core idea Reach for it when…

Small, low-dimensional data with

KNN Vote of the k closest examples.
meaningful distances.

A practical workflow for choosing a model

1. Start with a baseline. Linear or logistic regression tells you instantly whether the
signal is largely linear and gives a score to beat.
2. Add regularisation (Ridge / Lasso / Elastic Net, or the C in logistic regression) if you
have many features or see overfitting.
3. Move to a Random Forest for a strong, low-effort non-linear model. It usually
beats the linear baseline on tabular data with no scaling and little tuning.
4. Try Gradient Boosting (XGBoost / LightGBM / CatBoost) when you need to squeeze
out the last few points of accuracy and can afford to tune. It is the typical winner on
structured data.
5. Consider SVM for small, high-dimensional problems with clear margins, and Naive
Bayes for fast text classification.
6. Use KNN mainly for small datasets or similarity-style tasks; it rarely scales.

Three habits that matter more than the algorithm

1. Validate honestly — always score on held-out data or cross-validation, never on the
training set. 2. Scale when needed — mandatory for KNN, SVM, and regularised linear
models; irrelevant for tree-based methods. 3. Tune hyperparameters with grid or
randomised search inside cross-validation, and use early stopping for boosting. A
well-validated simple model beats a carelessly evaluated complex one every time.

A closing thought
No algorithm is universally best — this is the famous no free lunch theorem. The right
choice depends on your data size, dimensionality, the importance of interpretability, and
how much accuracy is worth to you. Understanding the trade-offs in this guide is what lets
you make that call quickly and well.

37