Supervised Machine Learning Modules
Module 1: Data Strategy & Evaluation
Understanding data and evaluation methods is key. Bias–variance tradeoff: low-complexity models have
high bias (underfit), high-complexity models have high variance (overfit) 1 2 . The goal is to balance them
(see illustrative curve below). High bias means systematic error; high variance means sensitivity to noise 3
2 .
Feature: Test error vs. model complexity. As complexity increases, bias² drops but variance rises, yielding a U-
shaped total error 1 2 .
• Cross-validation: Use Stratified K-Fold for classification. It splits data into K folds while preserving
class proportions in each fold 4 . This ensures each fold mirrors the original class distribution,
reducing evaluation bias on imbalanced data.
• Imbalanced data: Standard accuracy is misleading when classes are skewed. For example, Kaggle’s
Credit Card Fraud dataset has only 0.17% fraud cases 5 – a dummy model could get 99.8%
accuracy by always predicting non-fraud. Instead, use metrics that account for class imbalance.
• Metrics (classification): Accuracy = (TP+TN)/(all). For imbalanced data, avoid accuracy and prefer
precision, recall, F1 score, or AUC. Precision measures correctness of positive predictions and recall
(sensitivity) measures coverage of actual positives; F1 is their harmonic mean. F1 is preferable to
accuracy on imbalanced classes 6 7 . For example:
• Accuracy: misleading on imbalanced sets (all-majority baseline can be high accuracy) 6 .
• Precision/Recall: measure performance on the minority class (fraud, disease, etc.).
• F1 Score: balances precision and recall; better than accuracy for imbalanced data 6 7 .
• ROC-AUC / PR-AUC: threshold-independent measures of ranking quality (useful too).
• Regression metrics: Use mean squared error (MSE) or R² (explained variance) for regression tasks.
Data strategy examples: Generate synthetic data (e.g. make_blobs ) to demonstrate concepts. For real data,
consider the Kaggle Credit Card Fraud (284,807 samples, 492 frauds = 0.17% positives) 5 . Handle missing
1
values via imputation: e.g. KNNImputer in scikit-learn fills missing entries using the mean of K nearest
neighbors 8 .
Module 2: Linear Models & Gradient Descent
Linear models assume a linear relationship between inputs and output. Ordinary Least Squares (OLS)
Regression fits coefficients to minimize the sum of squared errors between predictions and targets 9 . It
has a closed-form solution via the normal equation for small data, but for high-dimensional or streaming
data, iterative methods are preferred 10 11 .
- Normal Equation: Solves $w = (X^TX)^{-1}X^T y$ exactly. Works if feature matrix is not too large, but
expensive when features or samples are huge 10 .
- Gradient Descent (GD): Iteratively update weights to minimize loss. At each step, move in the negative
gradient direction to reduce error 11 . Batch GD uses the whole dataset per update (slow for big data),
whereas stochastic/mini-batch GD uses subsets. Gradient descent is effective for large-scale linear regression
11 .
Implementation notes: Feature scaling (standardizing inputs) is crucial for GD convergence. Synthetic data
can illustrate both approaches; e.g. fit a line to 2D points. Frameworks (scikit-learn’s LinearRegression )
solve OLS directly, while custom code can demonstrate SGD steps.
Module 3: Advanced Optimization (SGD, Momentum, Adam)
For large datasets or complex models, classic GD is slow. Stochastic Gradient Descent (SGD) updates
parameters using one (or a few) samples at a time, approximating the true gradient 12 . This makes
convergence noisy but much faster per update, allowing online learning 12 .
Enhancements:
- Mini-batch GD: Compromise between batch and SGD: use small batches (e.g. 32–256 samples) for
updates.
- Momentum: Accelerates SGD by adding an inertia term (exponentially weighted average of past
gradients), helping traverse flat regions and damp oscillations 13 .
- Adam: Combines momentum and adaptive learning rates. It keeps moving averages of both gradients
and squared gradients (first and second moments) to adjust learning rates per-parameter 14 13 . Adam
converges faster in practice and is widely used for deep models 14 .
These optimizers form the backbone of training neural nets and other complex models. They reduce
training time and often yield better optima than vanilla GD.
Module 4: Regularization (Ridge, Lasso)
Regularization prevents overfitting by penalizing large weights. L2 (Ridge) regularization adds a penalty
proportional to the sum of squared coefficients; L1 (Lasso) adds a penalty proportional to the sum of
absolute values 15 . Both mitigate overfitting by discouraging complex models 16 .
- Ridge (L2): Shrinks coefficients continuously toward zero but never exactly zero 15 17 . Works well when
many features are correlated (multicollinearity).
- Lasso (L1): Can drive some coefficients exactly to zero, performing feature selection and yielding sparse
2
models 15 18 . Useful when expecting only a few important features.
Both use a hyperparameter (lambda or C in practice) to control strength: larger lambda = stronger
regularization (higher bias, lower variance) 16 . In practice, use cross-validation to tune this penalty.
Module 5: Logistic Regression & Classification Nuances
Logistic regression is a linear model for binary classification. It applies the logistic (sigmoid) function to a
linear combination of features, outputting a probability in [0,1] 19 . This maps any real-valued input to a
probability of class 1. Predictions are made by thresholding (often 0.5) on this probability. Logistic
regression uses a log-loss (cross-entropy) objective and can be regularized (like linear models).
Key points: each example’s probability is modeled as $\sigma(w^T x)$; coefficients are usually fit by
maximum likelihood (iterative optimization). The sigmoid’s S-curve makes it suitable for probabilistic
classification 19 .
Classification tools: Besides logistic regression, simple baseline classifiers include:
- K-Nearest Neighbors (KNN): A non-parametric instance-based method. It classifies a sample based on
the majority class among its k nearest neighbors 20 . KNN makes no strong distributional assumptions,
serving as a useful naive baseline 20 .
- Naive Bayes: A probabilistic classifier assuming feature conditional independence 21 . It’s simple and fast
(closed-form update), but can be less accurate than modern methods 22 .
Choose evaluation metrics as in Module 1: accuracy, precision/recall/F1, ROC-AUC, etc., depending on class
balance. Decision threshold can be tuned to balance precision vs. recall.
Imputation reminder: In preprocessing, note that KNN can also be used for imputation: KNNImputer in
scikit-learn fills missing values using the mean of the nearest neighbors 8 . This is a non-parametric way to
infer missing entries from similar instances.
Module 6: Decision Trees (ID3/CART)
Decision trees partition the feature space into axis-aligned regions by recursively splitting on features. At
each node, the tree picks a feature and threshold that best separates the classes. Common splitting criteria
are Gini impurity or information gain (entropy) 23 . Lower impurity (more homogeneous) splits are preferred
23 .
• Gini impurity: Measures the likelihood of misclassification of a random label; lower is better 24 .
• Entropy (Information Gain): Measures the disorder of class labels; a split that reduces entropy the
most is chosen 25 .
Decision trees can overfit by creating very deep trees that memorize the training data. Limit overfitting via
pruning (pre-prune by max depth or min samples per leaf, or post-prune by cost-complexity). Trees are
interpretable (we can visualize the splits) and can handle mixed data types. However, they are high-variance
learners without regularization.
3
Module 7: Ensembles Part 1 – Bagging & Random Forests
Ensembles combine multiple models to improve accuracy. Bagging (Bootstrap Aggregating): Train many
base learners (often deep decision trees) on different bootstrapped subsets of the data (sampling with
replacement) and average their predictions 26 27 . Bagging reduces variance: averaging uncorrelated
errors leads to a more stable overall model. For instance, Random Forests use bagged trees.
• Bagging idea: Each tree is trained on a random subset of samples; results are averaged (for
regression) or voted (for classification). This “committee” of trees is much more robust than any
single tree 26 27 .
• Random Forest: A refinement of bagging. In addition to bootstrapping samples, at each split it
considers only a random subset of features 28 . This de-correlates the trees (they don’t all pick the
same strong predictor), further reducing variance 28 29 . A Random Forest is an ensemble of
dozens or hundreds of trees whose majority vote is the final prediction.
Ensembles like bagging/Random Forests typically reduce overfitting and improve test accuracy on many
tasks 30 28 . They handle unbalanced data via class weighting or balanced subsampling.
Module 8: Ensembles Part 2 – Boosting (AdaBoost, XGBoost, etc.)
Boosting builds an ensemble sequentially: each new model focuses on correcting the mistakes of the
previous ones. In AdaBoost (Adaptive Boosting), a series of weak learners (often decision stumps) is
trained. After each model, sample weights are increased on misclassified points so that the next learner
focuses on those “hard” cases 31 . Finally, each learner’s prediction is combined via a weighted vote
(learners with lower error get higher weight) 31 . AdaBoost can dramatically improve performance, though
it can be sensitive to noise (outliers get higher weight).
More advanced boosting (e.g. Gradient Boosting, XGBoost) generalizes this idea by optimizing a
differentiable loss via gradient descent in function space. XGBoost (Extreme Gradient Boosting) is a highly
optimized, efficient implementation of gradient-boosted trees 32 33 . It builds trees sequentially, where
each tree fits the residual errors of the previous ensemble. XGBoost uses second-order (Hessian) information
for faster convergence, supports regularization to reduce overfitting, and handles missing data internally
32 33 .
• AdaBoost: First successful boosting method. Focuses on misclassified samples iteratively 31 .
• XGBoost: Implements gradient boosting with decision trees. It adds each new tree to minimize the
overall loss, often with shrinkage (learning rate) and regularization 32 33 . It is renowned for high
predictive performance on tabular data.
Boosting algorithms tend to reduce both bias and variance by combining many weak learners into one
strong model 31 . In practice, these methods (especially XGBoost and its variants) often dominate Kaggle
competitions for structured data due to their power and flexibility.
Summary: Across all modules, the key is understanding tradeoffs: bias vs. variance, complexity vs.
interpretability, and choice of metric vs. data properties. Use synthetic examples (e.g. make_blobs ) to
illustrate algorithms and real datasets (like credit-fraud) to anchor practice. Always validate models with
appropriate metrics and cross-validation to ensure robust, generalizable performance 3 6 .
4
1 2 Bias and Variance in Machine Learning - GeeksforGeeks
[Link]
3 Bias–variance tradeoff - Wikipedia
[Link]
4 StratifiedKFold — scikit-learn 1.8.0 documentation
[Link]
5 Using Symbolic Regression to predict rare events - TuringBot
[Link]
6 Classification: Accuracy, recall, precision, and related metrics | Machine Learning | Google for
7
Developers
[Link]
8 KNNImputer — scikit-learn 1.8.0 documentation
[Link]
9 LinearRegression — scikit-learn 1.8.0 documentation
[Link]
10 11 Gradient Descent in Linear Regression - GeeksforGeeks
[Link]
12 Stochastic gradient descent - Wikipedia
[Link]
13 14 Adam - Cornell University Computational Optimization Open Textbook - Optimization Wiki
[Link]
15 16 17 18 L1 and L2 Regularization Methods, Explained | Built In
[Link]
19 Logistic regression - Wikipedia
[Link]
20 What is the k-nearest neighbors algorithm? | IBM
[Link]
21 22 Naive Bayes classifier - Wikipedia
[Link]
23 24 25 Gini Impurity and Entropy in Decision Tree - GeeksforGeeks
[Link]
26 1.11. Ensembles: Gradient boosting, random forests, bagging, voting, stacking — scikit-learn 1.8.0
documentation
[Link]
27 28 Bagging and Random Forest Ensemble Algorithms for Machine Learning -
30
[Link]
[Link]
29 Random forest - Wikipedia
[Link]
5
31 Boost Your Models with AdaBoost Explained | DigitalOcean
[Link]
32 33 XGBoost - GeeksforGeeks
[Link]