MACHINE LEARNING

Complete Exam Study Guide

Simple English • Proper Notation • Every Formula Worked Out

Paper Pattern Marks Likely Topics

3 Questions (one = 5+5) 10M each = 30M Decision Tree, Regression, Naive Bayes
1 Question 15M Mixed / Analytical
1 Question 5M Cross Validation (most likely)

NO CODE • NO DERIVATIONS • REMEMBER FORMULAE • PROBLEM BASED

UNIT 1 — Supervised Learning I

1. Introduction to Machine Learning

Machine Learning = teaching a computer to learn patterns from data, without being explicitly
programmed for every rule.

The classic 3-part definition (Mitchell, 1997):

• Task (T): What the computer does. E.g., classify email as spam or not spam.
• Performance (P): How well it does it. E.g., 95% accuracy.
• Experience (E): Data it learns from. E.g., 10,000 labelled emails.

Steps to Build an ML Model

1. Explore and preprocess the data (handle missing values, outliers, scale features).
2. Split data: Training set (70%) + Test set (30%).
3. Build / train the model using training data.
4. Evaluate on test data.
5. Calculate performance metrics.
2. Linear Regression
Linear Regression predicts a CONTINUOUS NUMBER (not a category). Example: predict house price
from area, or exam score from study hours.

Notation used in this guide

Symbol Meaning
a y-intercept (where the line crosses the y-axis). Same as b₀ in textbooks.
b slope of the line. Same as b₁ in textbooks.
x̄ (x-bar) mean (average) of all x values
ȳ (y-bar) mean (average) of all y values
y' predicted value of y

The 3 Formulas for Simple Linear Regression

FORMULA 1: Direct formula using correlation (r)

b = r × (Sy / Sx)
a = ȳ − b × x̄
y' = a + b×x
WHEN TO USE: Use this when the question gives you r (correlation coefficient), Sx (std dev of x),
and Sy (std dev of y) directly. OR when you can compute them from the data. This is the most
general formula.

FORMULA 2: Deviation score formula

b = Σ(x − x̄)(y − ȳ) / Σ(x − x̄)²
a = ȳ − b × x̄
WHEN TO USE: Use this when the question gives you a raw dataset (table of x, y values) and does
NOT give r. You compute deviations from the mean yourself. Very common in exams with small
datasets.

FORMULA 3: Raw score (computational) formula

b = [nΣxy − Σx·Σy] / [nΣx² − (Σx)²]
a = (Σy − bΣx) / n
WHEN TO USE: Use this when the question gives you a raw dataset and you want to avoid
computing means and deviations separately. It uses column sums directly. Faster when data has
awkward decimals.

Quick Decision Guide:

Situation Use Formula
Given r, Sx, Sy directly Formula 1 (correlation method)
Given raw x, y data; compute step by step Formula 2 (deviation method)
 Given raw x, y data; want column sums Formula 3 (raw score method)
shortcut
All three give the SAME b and a Pick whichever the question suits

➔ WORKED EXAMPLE: Formula 1 — Given r, Sx, Sy

Dataset: r = 0.95, x̄ = 3, ȳ = 7, Sx = 1.58, Sy = 3.16

Step 1: Find b: b = r × (Sy/Sx) = 0.95 × (3.16/1.58) = 0.95 × 2 = 1.90

Step 2: Find a: a = ȳ − b×x̄ = 7 − 1.90×3 = 7 − 5.7 = 1.3
Step 3: Equation: y' = 1.3 + 1.90x
Step 4: Predict for x=5: y' = 1.3 + 1.90×5 = 1.3 + 9.5 = 10.8

➔ WORKED EXAMPLE: Formula 2 — Deviation method from raw data

x (Study hrs) y (Score)
1 2
2 4
3 5
4 4
5 5

Step 1: n=5. Σx=15, Σy=20 → x̄=15/5=3, ȳ=20/5=4

Step 2: Compute (x−x̄) and (y−ȳ) for each row:
x y x−x̄ y−ȳ (x−x̄)(y−ȳ) (x−x̄)²
1 2 −2 −2 4 4
2 4 −1 0 0 1
3 5 0 1 0 0
4 4 1 0 0 1
5 5 2 1 2 4
SUM 6 10
Step 3: Σ(x−x̄)(y−ȳ) = 6, Σ(x−x̄)² = 10
Step 4: b = 6 / 10 = 0.6
Step 5: a = ȳ − b×x̄ = 4 − 0.6×3 = 4 − 1.8 = 2.2
Step 6: Equation: y' = 2.2 + 0.6x
Step 7: Predict for x=6: y' = 2.2 + 0.6×6 = 2.2 + 3.6 = 5.8

➔ WORKED EXAMPLE: Formula 3 — Raw score (column sum) method — same dataset
x y x² xy
1 2 1 2
2 4 4 8
 3 5 9 15
4 4 16 16
5 5 25 25
SUM: 15 SUM: 20 SUM: 55 SUM: 66
Step 1: n=5, Σx=15, Σy=20, Σx²=55, Σxy=66
Step 2: b = (nΣxy − Σx·Σy) / (nΣx² − (Σx)²)
Step : = (5×66 − 15×20) / (5×55 − 15²)
Step : = (330 − 300) / (275 − 225)
Step : = 30 / 50 = 0.6 ✓ Same answer as Formula 2!
Step 3: a = (Σy − bΣx) / n = (20 − 0.6×15) / 5 = (20 − 9) / 5 = 11/5 = 2.2 ✓
Step 4: Equation: y' = 2.2 + 0.6x (identical result)

EXAM TIP: In the exam, Formula 2 and Formula 3 are most common with a small table. Formula 1
is used when r is given. All three give the same answer.

Error Metrics for Regression

Metric Formula Simple Meaning When to Use
MAE (1/n) Σ|y − y'| Average absolute When all errors should be
error. Easy to treated equally. Robust to
understand. outliers.
MSE (1/n) Σ(y − y')² Average squared When you want to penalise
error. Bigger large errors heavily. Default
mistakes hurt more. choice.
MPE (100%/n) Σ[(y−y')/y] Average When error relative to actual
percentage error. value matters more than
absolute error.
R² 1 − SSres/SStot How well the line Always report this. Tells how
fits. 1=perfect, much variance the model
0=terrible. explains.
Adj R² 1 − (1−R²)(n−1)/(n−p−1) R² adjusted for Use with Multiple Regression.
extra variables. Penalises adding useless
variables.

SSres = Σ(y − y')² | SStot = Σ(y − ȳ)² | n = data points, p = number of independent variables

➔ WORKED EXAMPLE: Error Metrics — using the same dataset (y' = 2.2 + 0.6x)
x y (actual) y' Error (y−y') |Error| Error²
(predicted)
1 2 2.8 −0.8 0.8 0.64
2 4 3.4 0.6 0.6 0.36
3 5 4.0 1.0 1.0 1.00
4 4 4.6 −0.6 0.6 0.36
 5 5 5.2 −0.2 0.2 0.04
SUM=3.2 SUM=2.40
Step MAE: (1/5) × 3.2 = 0.64
Step MSE: (1/5) × 2.40 = 0.48
Step ȳ: = 4. SSres = 2.40 | SStot = (2−4)²+(4−4)²+(5−4)²+(4−4)²+(5−4)² = 4+0+1+0+1 = 6
Step R²: 1 − 2.40/6 = 1 − 0.40 = 0.60 (model explains 60% of variance)

Multiple Linear Regression (MLR)

y' = a + b₁ x₁ + b₂ x₂ + ... + bₙxₙ
WHEN TO USE: Use when you have MORE THAN ONE input variable predicting one output. E.g.,
predicting salary using years_experience AND education_level.

MLR Assumption What it Means How to Test

Linear Relationship Each x relates linearly to y Scatter plot
No Multicollinearity x-variables not correlated with VIF = 1/(1−R²). High VIF → problem.
each other
No Autocorrelation Each row is independent (no Durbin-Watson test
time dependency)
Homoscedasticity Error variance is constant (no Breusch-Pagan test
funnelling)

Polynomial Regression
y' = a + b₁ x + b₂ x² + b₃ x³ + ...
WHEN TO USE: Use when data is CURVED (not a straight line). Plot the data first — if it looks like
a curve, use polynomial. E.g., growth of bacteria over time, speed vs fuel efficiency.

➔ WORKED EXAMPLE: Polynomial vs Linear — Which to pick?

Dataset: Hours of sleep vs alertness score
Hours 4 5 6 7 8 9 10
Alertness 40 60 80 90 80 60 40
This data goes up then comes back down — it's a curve (inverted U). A straight line would fit badly.
Use Polynomial Regression (degree 2: y' = a + bx + cx²).

EXAM TIP: Exam will not make you solve polynomial regression by hand. Know WHEN to use it
(curved data). Know the equation form.
3. Logistic Regression
Logistic Regression is for CLASSIFICATION — predicting a category (Yes/No, 0/1, Spam/Not Spam).
Despite the name, it is NOT regression!

The Key Formula — Sigmoid Function

p = 1 / (1 + e^(−y))
where y = a + b₁ x₁ + b₂ x₂ + ...
WHEN TO USE: Use when the OUTPUT is a CATEGORY (binary: 0 or 1). Input features can be
numbers or categories. E.g., Will patient have heart disease? (Yes/No). Will email be spam?
(Yes/No).

• p > 0.5 → Predict Class 1 (Positive / Yes)

• p ≤ 0.5 → Predict Class 0 (Negative / No)

Log Odds Formula

log(p / (1−p)) = a + b₁ x₁ + b₂ x₂ + ...
• p/(1−p) is the ODDS (how many times more likely is yes than no)
• log of odds = log odds
• Logistic regression finds the best a and b values using Maximum Likelihood Estimation

➔ WORKED EXAMPLE: Logistic Regression — Classify from sigmoid

Given: a = −6, b = 0.5, patient age x = 14
Step 1: Compute y = a + b×x = −6 + 0.5×14 = −6 + 7 = 1
Step 2: Compute p = 1/(1+e^(−1)) = 1/(1+0.368) = 1/1.368 = 0.731
Step 3: p = 0.731 > 0.5 → Predict: YES (positive class)

EXAM TIP: Logistic regression questions may ask you to (1) apply the sigmoid formula, (2) decide
the class. Always show the p value and comparison with 0.5.
4. Naive Bayes
Naive Bayes classifies by asking: given this data, which class is most probable? It uses Bayes
Theorem. 'Naive' = assumes all features are independent of each other.

Bayes Theorem
P(C | X) = P(X | C) × P(C) / P(X)
Term Name Meaning
P(C | X) Posterior Probability of class C given data X — this is what we want
P(X | C) Likelihood Probability of seeing data X if class is C
P(C) Prior Overall probability of class C in training data
P(X) Evidence Constant for all classes — we ignore it for comparison

WHEN TO USE: Use Naive Bayes when: (1) features are categorical, (2) you have multiple
features, (3) dataset is small to medium. Works great for text classification (spam filter), medical
diagnosis.

Step-by-Step Method
6. Count class probabilities: P(C) = count of class / total rows
7. For each feature value, count: P(feature=val | class) = count / class_total
8. Multiply all likelihoods for each class: P(X|C) = P(f1|C) × P(f2|C) × ...
9. Multiply by prior: score(C) = P(X|C) × P(C)
10. Pick class with HIGHEST score

➔ WORKED EXAMPLE: Naive Bayes — Buy Computer dataset (from your PPT)
14 rows. Predict for X = (age≤30, income=medium, student=yes, credit=fair)

Step 1: Class priors

Class Count P(C)
buys=YES 9 9/14 = 0.643
buys=NO 5 5/14 = 0.357

Step 2: Likelihoods for each feature

Feature P(feature | YES) P(feature | NO)
age ≤ 30 2/9 = 0.222 3/5 = 0.600
income = medium 4/9 = 0.444 2/5 = 0.400
student = yes 6/9 = 0.667 1/5 = 0.200
credit = fair 6/9 = 0.667 2/5 = 0.400
Step 3: Multiply likelihoods
Step YES: P(X|YES) = 0.222 × 0.444 × 0.667 × 0.667 = 0.0439
Step NO: P(X|NO) = 0.600 × 0.400 × 0.200 × 0.400 = 0.0192

Step 4: Multiply by prior

Step YES: 0.0439 × 0.643 = 0.0282
Step NO: 0.0192 × 0.357 = 0.0069

Step 5: Compare → YES wins

0.0282 > 0.0069 → Prediction: buys_computer = YES

EXAM TIP: The exam WILL give a similar table and ask you to classify. Show every step. Never
skip the prior multiplication at Step 4!
5. Decision Trees
A Decision Tree splits data at each node by asking a question about a feature. The goal: reach PURE
leaf nodes (all one class). The tricky part: which feature to split on first?
Answer: pick the feature that gives the BEST split. Two ways to measure 'best':

Method A: Information Gain (using Entropy)

Entropy H(S) = −Σ p(i) × log₂ (p(i))
Information Gain IG(S, A) = H(S) − Σ [ (|Sv|/|S|) × H(Sv) ]
Term Meaning
H(S) Entropy of the full set S. Measures uncertainty/impurity.
p(i) Proportion of class i in the set
Sv Subset of S where attribute A has value v
|Sv|/|S| Weight = fraction of total examples in that subset
IG(S,A) How much entropy REDUCES after splitting on attribute A

WHEN TO USE: Use Information Gain when the exam question says 'use entropy' or 'use ID3 /
C4.5 algorithm'. Pick the attribute with the HIGHEST Information Gain as root.

Key facts about entropy:

• If all examples are ONE class → Entropy = 0 (perfectly pure, no uncertainty)
• If examples are 50-50 split → Entropy = 1 (maximum uncertainty)
• log₂ (1) = 0, log₂ (0.5) = −1, log₂ (0.25) = −2

➔ WORKED EXAMPLE: Information Gain — Play Golf dataset (14 rows: 9 Yes, 5 No)
Full dataset entropy first:
H(S) = −(9/14)×log₂ (9/14) − (5/14)×log₂ (5/14)
= −(0.643×(−0.637)) − (0.357×(−1.485))
= 0.409 + 0.530 = 0.940

Now calculate IG for WIND attribute:

Wind=Weak: 8 rows (6 Yes, 2 No) | Wind=Strong: 6 rows (3 Yes, 3 No)
H(Weak) = −(6/8)×log₂ (6/8) − (2/8)×log₂ (2/8) = 0.811
H(Strong) = −(3/6)×log₂ (3/6) − (3/6)×log₂ (3/6) = 1.000
IG(Wind) = 0.940 − [(8/14)×0.811 + (6/14)×1.000]
= 0.940 − [0.463 + 0.429] = 0.940 − 0.892 = 0.048

Do the same for Outlook, Temperature, Humidity. Outlook gives IG ≈ 0.246 (highest).
So OUTLOOK becomes the ROOT NODE.

Method B: Gini Index

Gini(S) = 1 − Σ p(i)²
 Weighted Gini(A) = Σ [ (|Sv|/|S|) × Gini(Sv) ]
WHEN TO USE: Use Gini when the exam question says 'use Gini index' or 'use CART algorithm'.
Pick the attribute with the LOWEST Weighted Gini as root.

• Gini = 0 means perfectly pure (all one class)

• Gini = 0.5 means worst split (50-50 for binary class)

➔ WORKED EXAMPLE: Gini Index — same Play Golf dataset

Overall Gini first:
Gini(S) = 1 − [(9/14)² + (5/14)²] = 1 − [0.413 + 0.128] = 0.459

Gini for WIND: Weak(8 rows: 6Y,2N) | Strong(6 rows: 3Y,3N)

Gini(Weak) = 1 − [(6/8)² + (2/8)²] = 1 − [0.5625+0.0625] = 0.375
Gini(Strong) = 1 − [(3/6)² + (3/6)²] = 1 − [0.25+0.25] = 0.500
Weighted Gini(Wind) = (8/14)×0.375 + (6/14)×0.500 = 0.214 + 0.214 =
0.428

Gini for HUMIDITY: High(7 rows: 3Y,4N) | Normal(7 rows: 6Y,1N)

Gini(High) = 1 − [(3/7)² + (4/7)²] = 1 − [0.184+0.327] = 0.490
Gini(Normal) = 1 − [(6/7)² + (1/7)²] = 1 − [0.735+0.020] = 0.245
Weighted Gini(Humidity) = (7/14)×0.490 + (7/14)×0.245 = 0.368

Compute for all attributes then compare weighted Gini:

Attribute Weighted Gini Rank
Outlook ≈0.344 1st (LOWEST → BEST → Root)
Humidity 0.368 2nd
Wind 0.428 3rd
Temperature ≈0.440 4th

Comparison: Info Gain vs Gini

Information Gain Gini Index

Algorithm ID3, C4.5 CART

Pick attribute with... HIGHEST IG LOWEST Gini
Range 0 to 1 0 (pure) to 0.5
Measures Entropy reduction Impurity (chance of
misclassification)

EXAM TIP: The exam question WILL specify which method to use. Both use the same dataset.
Know both methods cold. Decision Tree is almost certainly a 10M question.
6. Class Imbalance Problem
Class imbalance = one class has far more examples than another. E.g., 99% healthy, 1% sick.

• A model that always predicts 'healthy' gets 99% accuracy — but it’s USELESS for finding sick
patients.
• Accuracy is misleading with imbalanced data.
• Use: ROC-AUC, Precision, Recall, F1 Score instead.

7. Performance Metrics
The Confusion Matrix
Predicted: Positive Predicted: Negative

Actual: Positive TP – True Positive (hit) FN – False Negative (miss!)

Actual: Negative FP – False Positive (false alarm!) TN – True Negative (correct reject)

All Metric Formulae with When to Use

Metric Formula When to Use
Accuracy (TP+TN)/(TP+TN+FP+FN) Balanced dataset. NOT suitable for imbalanced
data.
Precision TP/(TP+FP) When FALSE POSITIVES are costly. E.g., spam
filter (don't mark real email as spam).
Recall (Sensitivity) TP/(TP+FN) When FALSE NEGATIVES are costly. E.g.,
cancer detection (don’t miss a sick person).
Specificity TN/(TN+FP) When correctly identifying negatives matters.
E.g., healthy vs diseased screening.
F1 Score 2×(P×R)/(P+R) When you need BALANCE between Precision
and Recall. Best for imbalanced data.
TPR (=Recall) TP/(TP+FN) Y-axis of ROC curve.
FPR FP/(FP+TN) X-axis of ROC curve. Lower is better.

➔ WORKED EXAMPLE: Confusion Matrix — Cancer Detection

Model results: TP=90, FP=10, FN=5, TN=895
Metric Calculation Result
Accuracy (90+895)/(90+10+5+895) 98.5% — looks great but...
Precision 90/(90+10) 90% — of predicted cancer, 90% really had
it
Recall 90/(90+5) 94.7% — of actual cancer cases, we caught
94.7%
Specificity 895/(895+10) 98.9%
F1 Score 2×(0.90×0.947)/(0.90+0.947) 92.3%
ROC Curve & AUC
• ROC = Receiver Operating Characteristic curve
• Plot: TPR (y-axis) vs FPR (x-axis) at different classification thresholds
• AUC = Area Under the ROC Curve. AUC=1.0 is perfect. AUC=0.5 is random guessing.
• Higher AUC = better model. Use AUC to COMPARE two models.

WHEN TO USE: Use ROC-AUC when: dataset is imbalanced, or when you need to compare two
different models fairly.

EXAM TIP: Exam commonly gives a confusion matrix and asks for all metrics. Memorise all 5
formulae and know which to use when!
UNIT 2 — Supervised Learning II

8. Support Vector Machines (SVM)

SVM finds the BEST separating boundary (hyperplane) between two classes. Not just any boundary —
the one with the MAXIMUM MARGIN.

SVM Term Meaning

Hyperplane The decision boundary. In 2D it’s a line; in 3D a plane; in n-D it’s a
hyperplane.
Support Vectors The data points CLOSEST to the hyperplane. These define the margin.
Margin Gap between the hyperplane and the nearest data points on each side.
SVM maximises this.
Hard Margin SVM No errors allowed. Works only if data is perfectly separable.
Soft Margin SVM Allows some misclassifications. Better for real-world noisy data.
Kernel Trick Maps data to higher dimensions so it becomes linearly separable.

WHEN TO USE: Use SVM when: (1) data has clear margin of separation, (2) high-dimensional data
(text, images), (3) small-medium dataset. Works for both linear and non-linear data (using kernels).

Kernel Functions
Kernel When to Use
Linear Data is LINEARLY SEPARABLE (straight line separates classes)
Polynomial Data has polynomial relationship (curves)
RBF (Gaussian) General purpose; works when you don’t know the shape

➔ WORKED EXAMPLE: SVM — Concept

Points: Class A: (1,2),(2,2),(2,3) Class B: (4,4),(5,4),(5,5)
SVM finds the line that maximises the gap between the closest A and B points (support vectors).
These support vectors are the hardest-to-classify points. The model ONLY depends on them.

EXAM TIP: No calculation-based SVM in exam. Theory only: know support vectors, margin,
hyperplane, kernel trick. 5-10M theory question possible.

9. Overfitting & Underfitting

Overfitting Underfitting

What happens Model MEMORISES training data; Model is too simple; cannot learn the
fails on new data pattern
 Bias LOW HIGH
Variance HIGH LOW
Training accuracy Very HIGH Low
Test accuracy LOW (bad!) Low
Model complexity TOO COMPLEX TOO SIMPLE
Also called High Variance model High Bias model
Solution Regularization, more data, cross- More features, more complex model
validation

➔ WORKED EXAMPLE: Overfitting vs Underfitting — Visual idea

Imagine fitting a curve through 5 data points:
• Straight line (degree 1): misses the pattern → UNDERFITTING
• Perfect squiggle through every point (degree 10): memorises noise → OVERFITTING
• Smooth curve (degree 2-3): captures pattern, generalises well → JUST RIGHT

10. Bias-Variance Tradeoff

Total Error = Bias² + Variance + Irreducible Noise

Term Definition Caused By

Bias Error from WRONG ASSUMPTIONS Too simple model, wrong algorithm
in model. Gap between predicted
and true values on average.
Variance Error from SENSITIVITY to small Too complex model, too few training samples
changes in training data. Model
changes a lot with different data.
Irreducible Random noise in data that cannot Measurement errors, missing variables
Noise be eliminated.

• Decreasing Bias usually INCREASES Variance (and vice versa). This is the TRADEOFF.
• Goal: find the sweet spot of low bias AND low variance.
• Bias = E[f'(x)] − f(x)
• Variance = E[(f'(x) − E[f'(x)])²]

➔ WORKED EXAMPLE: Bias-Variance Tradeoff — Simple Illustration

Model Bias Variance Situation
Predict always the mean ȳ High Zero Useless but stable
Simple linear regression Medium Low Underfitting if real data is curved
Polynomial degree 2 Low Medium Good balance for curved data
Polynomial degree 20 on 10 pts Near zero Extreme Overfitting — memorising noise

EXAM TIP: Bias-Variance tradeoff is a theory question. Know the formula, the definitions, and the
relationship. Draw the concept: as complexity increases, bias goes down but variance goes up.
11. Regularization & Generalization
Regularization adds a PENALTY to the loss function to discourage overly complex models. It forces the
model to be simpler, which improves generalization (performance on unseen data).

L1 Regularization — Lasso
Cost = MSE + λ × Σ|bᵢ|
WHEN TO USE: Use Lasso when you suspect many features are IRRELEVANT. Lasso can shrink
weights to exactly ZERO (automatic feature selection). Good for sparse models.

L2 Regularization — Ridge
Cost = MSE + λ × Σ(bᵢ)²
WHEN TO USE: Use Ridge when ALL features are likely relevant but you want to prevent any
single weight from getting too large. Ridge shrinks weights but rarely to zero.

L1 Lasso L2 Ridge

Penalty term λΣ|bᵢ| (absolute values) λΣbᵢ² (squared values)

Effect on weights Can set weights to ZERO Shrinks weights, rarely zero
Feature selection YES — removes useless features NO — keeps all features (smaller)
When to prefer Many irrelevant features All features matter, prevent large
weights

• λ (lambda): Controls penalty strength. λ=0 means no regularization (pure regression). Higher λ
= stronger penalty = simpler model.
• Generalization: A model generalizes well if it performs well on NEW, UNSEEN data. Good
generalization = no overfitting.

➔ WORKED EXAMPLE: Regularization Effect

Without regularization: weights b = [0.3, 8.7, −15.2, 22.1] → wild, overfitting
With Ridge (λ=1): weights shrink to b = [0.3, 2.1, −3.5, 4.2] → controlled, generalizes better
12. Cross Validation — HIGH LIKELY 5M QUESTION
Cross Validation evaluates how well a model generalizes to UNSEEN data. It helps detect overfitting or
underfitting BEFORE you deploy the model.

Why not just train-test split?

• A single 70-30 split depends heavily on WHICH data ends up in test. You might get lucky or
unlucky.
• Cross validation runs multiple train-test splits and averages them → more reliable estimate.

K-Fold Cross Validation — Most Important

WHEN TO USE: Use K-Fold when: you have moderate-sized data and want a reliable estimate of
model performance. Standard choice in most situations. K=5 or K=10 are most common.

11. Split data into K equal parts (folds)

12. For each fold i from 1 to K: Train on all folds EXCEPT fold i. Test on fold i.
13. Record accuracy for each fold
14. Final score = AVERAGE of all K accuracies

➔ WORKED EXAMPLE: 5-Fold Cross Validation — 100 samples, 5 folds of 20 samples each
Iteration Training Folds Test Fold Accuracy
Round 1 Folds 2,3,4,5 (80 samples) Fold 1 (20 samples) 88%
Round 2 Folds 1,3,4,5 (80 samples) Fold 2 (20 samples) 90%
Round 3 Folds 1,2,4,5 (80 samples) Fold 3 (20 samples) 85%
Round 4 Folds 1,2,3,5 (80 samples) Fold 4 (20 samples) 92%
Round 5 Folds 1,2,3,4 (80 samples) Fold 5 (20 samples) 87%

Final accuracy = (88+90+85+92+87)/5 = 442/5 = 88.4%

Other Cross Validation Methods

Method How It Works When to Use
LOOCV (Leave-One- K = n (each sample is test set Very small datasets. Computationally
Out) once) expensive.
Stratified K-Fold Each fold maintains same class Imbalanced datasets. Recommended
ratio as original data default.
Repeated K-Fold Run K-Fold multiple times with When you need very stable estimates.
different random splits

EXAM TIP: For the 5M cross validation question: (1) explain K-Fold clearly, (2) show the worked
example table, (3) compute average accuracy. That is a full-mark answer!
13. Ensemble Methods — THEORY QUESTION
Ensemble = combine many weak models to build one strong model. The key idea: wisdom of the crowd
beats a single expert.

Method How Goal Models run

Bagging Train models on random subsets Reduce VARIANCE (fix PARALLEL
(bootstrap) overfitting)
Boosting Train models sequentially, each Reduce BIAS (fix SEQUENTIAL
fixing previous errors underfitting)
Stacking Train a meta-model on outputs of General improvement Parallel then meta
base models

Bagging — Bootstrap Aggregation

WHEN TO USE: Use Bagging when your single model is OVERFITTING (high variance). Bagging
reduces variance by averaging many models.

15. Create N bootstrapped datasets (random samples WITH replacement from original training
data)
16. Train a separate Decision Tree on EACH bootstrapped dataset
17. To predict: run all N trees, take MAJORITY VOTE (classification) or AVERAGE (regression)

• Random Forest: Famous bagging method. Like bagging, but also randomly selects a SUBSET
OF FEATURES at each split (not just data). More diverse trees → better.
• Out-of-Bag (OOB) samples: Rows NOT selected in a bootstrap sample. Used as automatic
test set for that tree. Free validation!
• More trees = more accuracy. BUT too many trees = slow + diminishing returns (and can overfit).

➔ WORKED EXAMPLE: Bagging Intuition — 3 trees voting

Sample Tree 1 Tree 2 Tree 3 Vote Prediction
Patient 1 YES YES NO 2 YES, 1 NO YES
Patient 2 NO NO NO 3 NO, 0 YES NO
Patient 3 YES NO YES 2 YES, 1 NO YES

Boosting
WHEN TO USE: Use Boosting when your single model is UNDERFITTING (high bias). Boosting
builds models sequentially, each one learning from the previous model’s mistakes.

18. Train Model 1 on original data

19. Find the examples Model 1 got WRONG. Give them higher weight.
20. Train Model 2 on the same data, but paying MORE attention to those wrong examples
21. Repeat for N rounds
22. Final prediction = WEIGHTED SUM of all model predictions

Boosting Algorithm How It Boosts Key Feature

 AdaBoost Increases sample WEIGHT for Simple, interpretable
misclassified examples
Gradient Boosting Each model fits the RESIDUAL Flexible, very powerful
ERRORS of the previous
XGBoost Gradient Boosting + Fastest, most popular in competitions
regularization + parallel
computation

➔ WORKED EXAMPLE: Boosting Intuition

Iteration 1: Model predicts correctly for 80% of data. 20% wrong.
Iteration 2: New model focuses more on the 20% wrong cases. Gets those right, might miss some of
the 80%.
Iteration 3: Model focuses on whatever is still wrong. Keeps improving.
Final: Weighted combination → better than any single model.

Feature Bagging (Random Forest) Boosting (AdaBoost/XGBoost)

Training PARALLEL (independent) SEQUENTIAL (each uses previous)
Goal Reduce VARIANCE Reduce BIAS
Overfitting risk Low Higher (if too many rounds)
Speed Faster (parallel) Slower (sequential)
Best when model is... OVERFITTING UNDERFITTING

EXAM TIP: Bagging vs Boosting is a guaranteed theory question. Know: parallel vs sequential,
variance vs bias, Random Forest vs AdaBoost/XGBoost. This is a 10M theory question!

14. Hyperparameter Tuning

Hyperparameters are settings chosen BEFORE training (the model does not learn them). Examples: K
in K-Fold, depth of Decision Tree, number of trees in Random Forest, λ in regularization.

Method How When to Use

Grid Search Try EVERY combination of Small number of hyperparameters.
hyperparameters in a defined Thorough but slow.
grid
Random Search Try RANDOM combinations from Many hyperparameters. Faster, often finds
the grid good values.
Bayesian Uses past results to intelligently Complex models, expensive training.
Optimization pick next combination
MASTER FORMULA CHEAT SHEET
Print this page. Memorise everything here.

Topic Formula / Key Fact

Linear Reg Line y' = a + bx
b (Formula 1 — use when r, b = r × (Sy / Sx)
Sx, Sy given)
b (Formula 2 — use with raw b = Σ(x−x̄)(y−ȳ) / Σ(x−x̄)²
data, step-by-step)
b (Formula 3 — use with raw b = [nΣxy − Σx·Σy] / [nΣx² − (Σx)²]
data, column sums)
a (all formulas) a = ȳ − b × x̄ OR a = (Σy − bΣx)/n
MAE (1/n) Σ|y − y'|
MSE (1/n) Σ(y − y')²
R² 1 − SSres/SStot where SSres=Σ(y−y')², SStot=Σ(y−ȳ)²
Adj R² 1 − (1−R²)(n−1)/(n−p−1)
Logistic Sigmoid p = 1/(1+e^(−y)), where y = a+bx
Log Odds log(p/(1−p)) = a + bx
Bayes Theorem P(C|X) = P(X|C) × P(C) / P(X)
Entropy H(S) = −Σ p(i) × log₂ (p(i))
Information Gain IG(S,A) = H(S) − Σ[(|Sv|/|S|)×H(Sv)] → Pick HIGHEST
Gini Index Gini(S) = 1 − Σp(i)² Weighted = Σ(|Sv|/|S|)×Gini(Sv) → Pick
LOWEST
Accuracy (TP+TN)/(TP+TN+FP+FN)
Precision TP/(TP+FP)
Recall (TPR) TP/(TP+FN)
Specificity TN/(TN+FP)
FPR FP/(FP+TN)
F1 Score 2×(P×R)/(P+R)
Bias-Variance Total Error = Bias² + Variance + Irreducible Noise
L1 Lasso Cost = MSE + λΣ|bᵢ| → sets some weights to ZERO
L2 Ridge Cost = MSE + λΣbᵢ² → shrinks all weights
VIF 1/(1−R²) (high VIF = multicollinearity problem)
K-Fold Final Score Average of accuracies across K folds
YOUTUBE STUDY LINKS
StatQuest is your best friend. Watch before sleeping. Review next morning.

Topic Channel Search This on YouTube

ALL ML (start here) StatQuest with Josh StatQuest Machine Learning playlist
Starmer
Linear Regression — all StatQuest StatQuest Linear Regression Clearly
3 formulas Explained
Logistic Regression StatQuest StatQuest Logistic Regression
Naive Bayes StatQuest StatQuest Naive Bayes Clearly Explained
Decision Tree + Entropy StatQuest StatQuest Decision Trees
Gini Index StatQuest StatQuest Gini Impurity Decision Tree
Bias and Variance StatQuest StatQuest Bias and Variance
ROC and AUC StatQuest StatQuest ROC and AUC
Cross Validation StatQuest StatQuest Cross Validation
Random Forest / StatQuest StatQuest Random Forests
Bagging
AdaBoost / Boosting StatQuest StatQuest AdaBoost Clearly Explained
Gradient Boosting StatQuest StatQuest Gradient Boost Clearly Explained
XGBoost StatQuest StatQuest XGBoost
SVM StatQuest StatQuest Support Vector Machines
Regularization StatQuest StatQuest Regularization Ridge Lasso
(Ridge+Lasso)
Quick Revision Krish Naik Krish Naik Machine Learning Full Course
Visual intuition 3Blue1Brown 3Blue1Brown Neural Networks

PRO TIP: StatQuest videos are 10–15 min each. Watch at 1.25x speed. Pause and re-draw the
diagrams yourself. For each topic: Watch → Close laptop → Try to recall the formula from memory
→ Write it on paper. That is how it sticks.
EXAM GAME PLAN

What to Expect Question by Question

Question Likely Topic What to Do
Q1 (10M) Decision Tree (Gini or Info Read QP to see which method. Do FULL calculation
Gain) for all attributes. Build the tree.
Q2 (10M) Naive Bayes OR For NB: show all P values, multiply, compare. For
Regression Reg: show formula, compute b & a, predict.
Q3 (5+5=10M) Performance Metrics + From confusion matrix, calculate all metrics. For
SVM/Ensemble Theory theory: definitions, comparisons.
Q4 (15M) Mixed / Analytical Likely combines regression + metrics OR decision
tree + NB. Show all steps clearly.
Q5 (5M) Cross Validation Explain K-Fold, show the table, compute average.
Mention LOOCV and Stratified briefly.

Day Before Exam Checklist

• Write all formulae from the cheat sheet from MEMORY (no peeking). Repeat until perfect.
• Solve one complete Naive Bayes problem from scratch (different dataset, same method).
• Solve one Decision Tree with Information Gain AND one with Gini Index.
• Given a confusion matrix, calculate Accuracy, Precision, Recall, F1 from scratch.
• Write out the 5-Fold CV table from memory. Compute average.
• Say out loud: Bagging vs Boosting (3 differences). Repeat 3 times.

During the Exam

• CHECK first: For Decision Tree questions, read carefully — does it say Gini or Information
Gain?
• Always WRITE THE FORMULA before substituting numbers. Gets partial marks even if
calculation is wrong.
• For Naive Bayes: always finish with 'P(X|C1)*P(C1) vs P(X|C2)*P(C2)' and state the winner
clearly.
• For regression: compute b first, then a, then the line equation, then the prediction. In order.
• Attempt EVERY question. Even partial steps get partial marks.
• Cross-check: after computing b, does the sign make sense? Positive slope = y goes up as x
goes up.

YOU GOT THIS. Practice the problems. Remember the formulae. Show
every step.