Contents

Machine Learning Mastery Lecture Notes 1

Table of Contents . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1

SECTION 1: Core Foundations & Exploratory Data Analysis (EDA) 2

1.1 Conceptual Blueprint — AI vs ML vs DL vs Data Science . . . . . . . . . . . . . . 2
1.2 Exploratory Data Analysis (EDA) . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
1.3 Feature Scaling — Standardization vs Normalization . . . . . . . . . . . . . . . . 4
1.4 Feature Extraction & Feature Selection . . . . . . . . . . . . . . . . . . . . . . . 5

SECTION 2: Supervised Learning — Regression Deep Dive 6

2.1 Linear Regression Intuition . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6
2.2 Cost Function — Mean Squared Error (MSE) . . . . . . . . . . . . . . . . . . . . 6
2.3 Optimization — Gradient Descent . . . . . . . . . . . . . . . . . . . . . . . . . . 7
2.4 Implementation & Evaluation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8

SECTION 3: Supervised Learning — Classification Algorithms & Real-World

Project 9
3.1 Logistic Regression . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9
3.2 K-Nearest Neighbors (KNN) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10
3.3 Decision Trees . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11
3.4 Support Vector Machines (SVM) . . . . . . . . . . . . . . . . . . . . . . . . . . . 12
3.5 Naive Bayes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12
3.6 Model Validation — K-Fold Cross Validation vs Train-Test Split . . . . . . . . . . . 13
3.7 Project & Deployment Lessons — Heart Disease & Titanic Datasets . . . . . . . . 13

SECTION 4: Model Tuning, Ensemble Methods & Unsupervised Learning 16

4.1 Hyperparameter Optimization . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16
4.2 Ensemble Learning Techniques . . . . . . . . . . . . . . . . . . . . . . . . . . . 17
4.3 Unsupervised Learning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18

Pro Project Tips & Common Mistakes 20

1. Top 5 Common ML Mistakes Beginners Make . . . . . . . . . . . . . . . . . . . . 20
2. Production Pipeline Checklist . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21
3. Model Selection Cheat-Sheet . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21

Machine Learning Mastery Lecture Notes

Based on the Complete Machine Learning Course (4-Part Series) — Sheryians AI
School
Reference Guide: Theory · Math Intuition · Code · Deployment

Table of Contents
1. Core Foundations & Exploratory Data Analysis
2. Supervised Learning — Regression Deep Dive
3. Supervised Learning — Classification & Real-World Project
4. Model Tuning, Ensemble Methods & Unsupervised Learning
5. Pro Project Tips & Common Mistakes

1
SECTION 1: Core Foundations & Exploratory Data Analysis
(EDA)
1.1 Conceptual Blueprint — AI vs ML vs DL vs Data Science

Term Definition Scope

Artificial Intelligence (AI) The broad field of building Umbrella field
systems that mimic
human-like intelligence —
reasoning, perception,
decision-making
Machine Learning (ML) A subset of AI where systems Subset of AI
learn patterns from data
instead of being explicitly
programmed
Deep Learning (DL) A subset of ML using Subset of ML
multi-layered neural
networks to learn
hierarchical representations,
especially effective on
unstructured data (images,
text, audio)
Data Science (DS) An interdisciplinary field Overlaps with all of the
combining statistics, domain above, plus analyt-
knowledge, programming, ics/visualization/business
and ML to extract insights context
and build data products

Nesting relationship:
AI ⊃ ML ⊃ DL
Data Science overlaps AI/ML but also includes analytics, statistics, and business reporting

Traditional Programming vs Machine Learning Paradigm

Traditional (Rule-Based) Programming:
Input + Rules → Output
A developer manually writes the logic (rules). The program executes those rules on input to
produce output. This works well when rules are known, finite, and don’t change — e.g., a tax
calculator, a sorting algorithm.
Machine Learning Paradigm:
Input + Output → Rules (Model)
Instead of hand-coding rules, we give the algorithm many examples of inputs and their correct
outputs. The algorithm infers the underlying rule (function) that maps input to output. This
“rule” is the trained model — essentially a mathematical function with learned parameters.
Why this matters: ML is preferred when: - The rules are too complex or unknown to hand-
code (e.g., recognizing a face, detecting spam). - The rules change frequently with new data
(e.g., fraud patterns, recommendation preferences). - There’s abundant historical data with
known outcomes to learn from.

2
Traditional programming is preferred when: - Rules are deterministic, small in number, and
well understood. - Explainability/auditability is legally mandatory and a black-box model is
unacceptable. - There is no representative dataset available.

1.2 Exploratory Data Analysis (EDA)

EDA is the process of investigating a dataset to summarize its main characteristics, often
using visual methods, before applying any modeling. Goal: understand structure, spot prob-
lems, and form hypotheses.

Step-by-Step EDA Workflow

1. Load & Inspect
import pandas as pd
df = pd.read_csv("[Link]")
[Link]()
[Link]()
[Link]()
[Link]

2. Check Data Types & Structure

• Numerical (continuous/discrete) vs Categorical (nominal/ordinal) vs Datetime vs
Text.
• Mismatched dtypes (e.g., a numeric column stored as object due to stray strings)
are a common silent bug.
3. Handling Missing Values

Strategy When to Use

Drop rows (dropna) Missingness is small (<5%) and random
(MCAR)
Drop column >50–60% of a column is missing and it’s not
critical
Mean/Median imputation Numerical, roughly symmetric (mean) or
skewed (median, robust to outliers)
Mode imputation Categorical features
Forward/Backward fill Time series data
Model-based imputation (KNNImputer, When missingness carries structure and
IterativeImputer) simple imputation would bias results

[Link]().sum() # audit missingness

df['age'].fillna(df['age'].median(), inplace=True)
df['city'].fillna(df['city'].mode()[0], inplace=True)
[Link](subset=['target'], inplace=True) # never impute the target

Key distinction (MCAR / MAR / MNAR):

• MCAR (Missing Completely At Random) — missingness unrelated to any variable.
Safe to drop/impute.
• MAR (Missing At Random) — missingness related to other observed variables.
Model-based imputation helps.

3
 • MNAR (Missing Not At Random) — missingness related to the unobserved value
itself (e.g., high earners not disclosing income). Requires domain-aware handling;
naive imputation introduces bias.
4. Outlier Detection & Treatment
• IQR Method:
𝐼𝑄𝑅 = 𝑄3 − 𝑄1
Lower Bound = 𝑄1 − 1.5 × 𝐼𝑄𝑅 , Upper Bound = 𝑄3 + 1.5 × 𝐼𝑄𝑅
Any point outside these bounds is a candidate outlier.
Q1 = df['salary'].quantile(0.25)
Q3 = df['salary'].quantile(0.75)
IQR = Q3 - Q1
lower, upper = Q1 - 1.5*IQR, Q3 + 1.5*IQR
df_clean = df[(df['salary'] >= lower) & (df['salary'] <= upper)]

• Z-Score Method: flag points where |𝑧| > 3.

𝑥−𝜇
𝑧=
𝜎

• Visual detection: boxplots, scatter plots, histograms.

• Treatment options: remove, cap/clip (winsorization), transform (log/sqrt to reduce
skew), or keep if it’s a genuine signal (e.g., fraud amounts are supposed to be ex-
treme).
5. Understanding Distribution Shifts
• Compare train vs test distributions (covariate shift) using histograms/KDE overlays
or statistical tests (Kolmogorov–Smirnov).
• Skewness check: df['col'].skew(). |skew| > 1 typically signals need for transfor-
mation (log, Box-Cox, Yeo-Johnson).
• Class imbalance check for classification targets: df['target'].value_counts(normalize=True).
6. Univariate, Bivariate, Multivariate Analysis
import seaborn as sns
import [Link] as plt

[Link](df['age'], kde=True) # univariate

[Link](x='class', y='fare', data=df) # bivariate
[Link]([Link](numeric_only=True), annot=True, cmap='coolwarm') # multivariate
[Link](df, hue='target')

1.3 Feature Scaling — Standardization vs Normalization

Feature scaling is critical because many algorithms are sensitive to the magnitude/range of
feature values.

Standardization (Z-score Scaling)

𝑥−𝜇
𝑥′ =
𝜎

4
- Centers data around mean 0 with unit variance (std = 1). - Does not bound values to a fixed
range. - Robust-ish to outliers relative to Min-Max, but still affected since 𝜇, 𝜎 are outlier-
sensitive.
from [Link] import StandardScaler
scaler = StandardScaler()
X_train_scaled = scaler.fit_transform(X_train)
X_test_scaled = [Link](X_test) # NEVER fit on test data

Normalization (Min-Max Scaling)

𝑥 − 𝑥𝑚𝑖𝑛
𝑥′ =
𝑥𝑚𝑎𝑥 − 𝑥𝑚𝑖𝑛
- Rescales values into a fixed range, typically [0, 1]. - Highly sensitive to outliers (a single
extreme value compresses everything else).
from [Link] import MinMaxScaler
scaler = MinMaxScaler()
X_train_scaled = scaler.fit_transform(X_train)

When to Use Which

Situation Preferred Scaling Why

Distance-based models Standardization (usually) or These algorithms compute
(KNN, K-Means, SVM with Min-Max distances — unscaled
RBF kernel) features with large ranges
dominate the distance metric
Gradient Descent-based Standardization Speeds up and stabilizes
models (Linear/Logistic convergence; avoids
Regression, Neural elongated cost function
Networks) contours
Tree-based models No scaling needed Trees split on thresholds per
(Decision Tree, Random feature independently —
Forest, XGBoost, Gradient monotonic transforms don’t
Boosting) change split decisions
PCA Standardization (mandatory) PCA is variance-driven;
unscaled features with
larger units dominate the
principal components
Data with known bounded Normalization Fixed, interpretable bounds
range, no major outliers
(e.g., pixel intensities 0–255)

Golden Rule: Always fit the scaler on the training set only, then transform both train and
test using those fitted parameters. Fitting on the full dataset (before train-test split) leaks
test-set statistics into training — a classic and very common beginner mistake (data leakage).

1.4 Feature Extraction & Feature Selection

• Feature Extraction: Deriving new features from existing raw data — e.g., extracting
day_of_week, hour from a timestamp; creating BMI = weight/height²; text → TF-IDF

5
 vectors; combining cuisine lists into counts.
• Feature Selection: Choosing a subset of the most relevant existing features to reduce
dimensionality and noise.
– Filter methods: correlation threshold, Chi-Square test, ANOVA F-test — statistical,
model-agnostic.
– Wrapper methods: Recursive Feature Elimination (RFE) — uses a model’s perfor-
mance to iteratively drop features.
– Embedded methods: L1 (Lasso) regularization naturally zeroes out irrelevant fea-
ture coefficients; tree-based feature_importances_.
from sklearn.feature_selection import RFE
from sklearn.linear_model import LogisticRegression

model = LogisticRegression(max_iter=1000)
selector = RFE(model, n_features_to_select=5)
[Link](X_train_scaled, y_train)
selected_features = X_train.columns[selector.support_]

Typical Kaggle-style EDA → Feature Engineering pipeline (e.g., Titanic dataset): 1.

Load data, inspect nulls/dtypes. 2. Impute (Age → median grouped by Pclass/Sex; Embarked →
mode). 3. Engineer features (FamilySize = SibSp + Parch + 1, Title extracted from Name).
4. Encode categoricals (Sex, Embarked → one-hot or label encoding). 5. Scale numerical
columns. 6. Drop leaky/irrelevant columns (PassengerId, Ticket, Cabin if too sparse). 7.
Split into train/test before any fitting step that “learns” from data.

SECTION 2: Supervised Learning — Regression Deep Dive

2.1 Linear Regression Intuition
Linear Regression models the relationship between an independent variable (feature, 𝑥) and
a dependent variable (target, 𝑦 ) as a straight line.
Simple Linear Regression:
𝑦 = 𝜃 0 + 𝜃1 𝑥
- 𝜃0 (or 𝑐 /intercept) — the value of 𝑦 when 𝑥 = 0. - 𝜃1 (or 𝑚/slope) — how much 𝑦 changes
per unit change in 𝑥.
Multiple Linear Regression (multiple independent variables):
𝑦 = 𝜃0 + 𝜃1 𝑥1 + 𝜃2 𝑥2 + ⋯ + 𝜃𝑛 𝑥𝑛
• Independent variable(s) (𝑥): the predictor(s)/features — assumed to influence the
outcome.
• Dependent variable (𝑦 ): the target/outcome we’re trying to predict.
The goal of training is to find the values of 𝜃0 , 𝜃1 , … , 𝜃𝑛 that make the line fit the data as
closely as possible — the “best-fit line.”

2.2 Cost Function — Mean Squared Error (MSE)

To measure “how well” a line fits, we need a quantitative error metric.

1 𝑛 1 𝑛 2
𝐽 (𝜃) = ∑(𝑦𝑖 − 𝑦𝑖̂ )2 = ∑ (𝑦𝑖 − (𝜃0 + 𝜃1 𝑥𝑖 ))
𝑛 𝑖=1 𝑛 𝑖=1

6
 1
(Some formulations use 2𝑛 for a cleaner gradient derivative — both are valid; the 21 just
cancels the exponent’s 2 during differentiation.)

Why square the error instead of using absolute value?

Reason Explanation
Differentiability |𝑒| has a sharp, non-differentiable corner at
𝑒 = 0, which breaks gradient-based
optimization. 𝑒2 is smooth and differentiable
everywhere — required for calculating
gradients in Gradient Descent.
Penalizes large errors more Squaring amplifies large deviations
disproportionately, pushing the optimizer to
avoid big misses — often desirable since
large errors are usually more costly in
real-world terms.
Convexity MSE as a function of 𝜃 is a convex
(bowl-shaped) function, guaranteeing a
single global minimum reachable by
Gradient Descent — absolute error’s
optimization landscape is less well-behaved
for gradient methods (though MAE is still
convex, it’s non-smooth at 0).
Mathematical tractability Squared error connects directly to variance
and has a closed-form analytical solution
(Normal Equation) via calculus, unlike MAE
which requires linear programming.

2.3 Optimization — Gradient Descent

Gradient Descent iteratively updates parameters to minimize the cost function 𝐽 (𝜃) by moving
in the direction opposite to the gradient (steepest descent).
Update Rule:
𝜕𝐽 (𝜃)
𝜃𝑛𝑒𝑤 = 𝜃𝑜𝑙𝑑 − 𝛼
𝜕𝜃
For Linear Regression with MSE, the partial derivatives are:

𝜕𝐽 2 𝑛
= − ∑(𝑦𝑖 − 𝑦𝑖̂ )
𝜕𝜃0 𝑛 𝑖=1

𝜕𝐽 2 𝑛
= − ∑(𝑦𝑖 − 𝑦𝑖̂ ) ⋅ 𝑥𝑖
𝜕𝜃1 𝑛 𝑖=1

Step-by-step algorithm: 1. Initialize 𝜃0 , 𝜃1 randomly (or to zero). 2. Compute predictions

𝑦 ̂ using current 𝜃 values. 3. Compute the gradient of 𝐽 (𝜃) with respect to each parameter. 4.
Update each parameter: move opposite to the gradient, scaled by learning rate 𝛼. 5. Repeat
steps 2–4 for a fixed number of epochs, or until 𝐽 (𝜃) converges (stops improving beyond a
tolerance).

7
Learning Rate (𝛼)
The learning rate controls the step size of each update.

Learning Rate Behavior

Too small Convergence is extremely slow — many
epochs needed; risk of getting stuck if
combined with limited iterations
Too large Overshooting — the algorithm jumps past
the minimum, and the cost may oscillate or
diverge (increase instead of decrease)
Well-tuned Smooth, efficient convergence to the
minimum in a reasonable number of steps

Overshooting visualized conceptually: picture a ball rolling down a bowl-shaped curve. A

small 𝛼 = tiny careful steps (slow but safe). A large 𝛼 = huge leaps that can bounce the ball
to the opposite wall of the bowl, potentially escaping the bowl entirely (divergence).
Practical fix: use learning rate schedules (decay 𝛼 over epochs) or adaptive optimizers
(Adam, RMSProp — more relevant in DL, but the same intuition of adapting step-size applies).

Gradient Descent Variants

Variant Description
Batch Gradient Descent Uses the entire dataset per update — stable
but slow for large datasets
Stochastic Gradient Descent (SGD) Uses one random sample per update — fast,
noisy convergence path, can escape local
minima
Mini-Batch Gradient Descent Uses a small batch (e.g., 32/64 samples) per
update — the practical default, balances
speed and stability

2.4 Implementation & Evaluation

from sklearn.model_selection import train_test_split

from sklearn.linear_model import LinearRegression
from [Link] import StandardScaler
from [Link] import mean_absolute_error, mean_squared_error, r2_score
import numpy as np

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

scaler = StandardScaler()
X_train_scaled = scaler.fit_transform(X_train)
X_test_scaled = [Link](X_test)

model = LinearRegression()
[Link](X_train_scaled, y_train)
y_pred = [Link](X_test_scaled)

8
print("Intercept:", model.intercept_)
print("Coefficients:", model.coef_)

Evaluation Metrics

Metric Formula Interpretation

1
MAE (Mean Absolute Error) 𝑛 ∑ |𝑦𝑖 − 𝑦𝑖̂ | Average absolute deviation;
same units as target; robust
to outliers
RMSE (Root Mean Squared √ 𝑛1 ∑(𝑦𝑖 − 𝑦𝑖̂ )2 Penalizes large errors more;
Error) same units as target
𝑆𝑆𝑟𝑒𝑠 ∑(𝑦𝑖 −𝑦𝑖̂ )2
𝑅2 Score 1− 𝑆𝑆𝑡𝑜𝑡 =1− ∑(𝑦𝑖 −𝑦)̄ 2 Proportion of variance in 𝑦
explained by the model; 1.0
= perfect fit, 0 = no better
than predicting the mean
2
Adjusted 𝑅2 1 − [ (1−𝑅 )(𝑛−1)
𝑛−𝑝−1 ] where 𝑝 = Penalizes adding irrelevant
number of predictors features that inflate 𝑅2
without real explanatory
power

mae = mean_absolute_error(y_test, y_pred)

rmse = [Link](mean_squared_error(y_test, y_pred))
r2 = r2_score(y_test, y_pred)

n, p = X_test.shape
adj_r2 = 1 - (1 - r2) * (n - 1) / (n - p - 1)

print(f"MAE: {mae:.3f} RMSE: {rmse:.3f} R2: {r2:.3f} Adj R2: {adj_r2:.3f}")

Interpretation caution: A high 𝑅2 on training data with a much lower 𝑅2 on test data
signals overfitting. Adjusted 𝑅2 is preferred over plain 𝑅2 when comparing models with
different numbers of features.

SECTION 3: Supervised Learning — Classification Algo-

rithms & Real-World Project
3.1 Logistic Regression
Intuition: Despite the name, this is a classification algorithm. It models the probability that
an input belongs to a class, using the Sigmoid function to squash any real-valued output into
the range [0, 1].
Math/Logic:
𝑧 = 𝜃0 + 𝜃1 𝑥1 + ⋯ + 𝜃𝑛 𝑥𝑛
1
𝜎(𝑧) =
1 + 𝑒−𝑧

9
 • If 𝜎(𝑧) ≥ 0.5 → predict class 1; else predict class 0 (threshold is tunable).
• The decision boundary is the surface where 𝜎(𝑧) = 0.5, i.e., where 𝑧 = 0 — for 2D
data this is a line; in higher dimensions, a hyperplane.
Cost Function — Log-Loss (Binary Cross-Entropy):

1 𝑛
𝐽 (𝜃) = − ∑ [𝑦𝑖 log(𝑦𝑖̂ ) + (1 − 𝑦𝑖 ) log(1 − 𝑦𝑖̂ )]
𝑛 𝑖=1

MSE isn’t used here because plugging the sigmoid into MSE creates a non-convex cost surface
(many local minima). Log-loss remains convex for logistic regression, guaranteeing Gradient
Descent converges to the global minimum. It also heavily penalizes confident-but-wrong pre-
dictions (e.g., predicting 0.99 when the true label is 0).
Pros: Fast, interpretable coefficients (log-odds), works well when classes are linearly sepa-
rable, outputs calibrated probabilities. Cons: Assumes a linear decision boundary; struggles
with complex, non-linear relationships unless features are engineered (polynomial terms) or
combined with kernel tricks.
from sklearn.linear_model import LogisticRegression
model = LogisticRegression(max_iter=1000)
[Link](X_train_scaled, y_train)
y_pred = [Link](X_test_scaled)
y_proba = model.predict_proba(X_test_scaled)[:, 1]

3.2 K-Nearest Neighbors (KNN)

Intuition: A lazy, instance-based algorithm — it doesn’t “learn” parameters during training.
To classify a new point, it looks at the 𝑘 closest points in the training set and takes a majority
vote (classification) or average (regression).
Distance Metrics:
𝑛
Euclidean: 𝑑(𝑝, 𝑞) = √∑(𝑝𝑖 − 𝑞𝑖 )2
𝑖=1
𝑛
Manhattan: 𝑑(𝑝, 𝑞) = ∑ |𝑝𝑖 − 𝑞𝑖 |
𝑖=1
• Euclidean = straight-line (“as the crow flies”) distance — sensitive to all dimensions
jointly.
• Manhattan = sum of absolute axis-wise differences (“grid/city-block” distance) — often
more robust in high dimensions or when features represent independent, non-continuous
movements.
Selecting 𝑘: - Small 𝑘 (e.g., 𝑘 = 1) → low bias, high variance → very sensitive to noise, overfits.
- Large 𝑘 → high bias, low variance → smoother decision boundary, may underfit and blur class
distinctions. - Common practice: try odd values of 𝑘 (avoids ties in binary classification) and
use cross-validation to pick the best.
Curse of Dimensionality: As the number of features grows, the volume of the feature space
grows exponentially, so data points become sparse and roughly equidistant from each other —
the notion of “nearest” neighbor becomes meaningless. Mitigation: dimensionality reduction
(PCA), feature selection.

10
Pros: Simple, no training phase, naturally handles multi-class. Cons: Slow at prediction
time (must compute distance to all training points), sensitive to feature scale (scaling is
mandatory), sensitive to irrelevant features and the curse of dimensionality.
from [Link] import KNeighborsClassifier
model = KNeighborsClassifier(n_neighbors=5, metric='minkowski', p=2) # p=2 -> Euclidean
[Link](X_train_scaled, y_train)

3.3 Decision Trees

Intuition: Recursively splits the data on feature thresholds to create a tree of if-else decisions,
ending in leaf nodes that represent class predictions.
Splitting criteria — Entropy & Information Gain:
𝑐
𝐸𝑛𝑡𝑟𝑜𝑝𝑦(𝑆) = − ∑ 𝑝𝑖 log2 (𝑝𝑖 )
𝑖=1

Entropy measures impurity/disorder in a node — 0 when a node is pure (all one class), maximal
when classes are evenly mixed.

|𝑆𝑘 |
Information Gain = 𝐸𝑛𝑡𝑟𝑜𝑝𝑦(𝑝𝑎𝑟𝑒𝑛𝑡) − ∑ 𝐸𝑛𝑡𝑟𝑜𝑝𝑦(𝑆𝑘 )
𝑘
|𝑆|
The tree picks, at each split, the feature/threshold that maximizes information gain (i.e., pro-
duces the purest child nodes).
Gini Impurity (alternative to entropy, used by CART/scikit-learn default):
𝑐
𝐺𝑖𝑛𝑖(𝑆) = 1 − ∑ 𝑝𝑖2
𝑖=1

Computationally cheaper than entropy (no log), generally gives similar results.
Overfitting mitigation: - Pruning: Pre-pruning (stop growing early via max_depth,
min_samples_split, min_samples_leaf) vs Post-pruning (grow fully, then trim back branches
that don’t improve validation performance, e.g., cost-complexity pruning ccp_alpha). -
max_depth: caps how deep the tree can grow — deeper trees memorize training data (high
variance).
Pros: Highly interpretable (visualizable), handles non-linear relationships, no scaling re-
quired, handles mixed feature types. Cons: Prone to overfitting if unconstrained, unstable
(small data changes can produce a very different tree), biased toward features with more
levels.
from [Link] import DecisionTreeClassifier
model = DecisionTreeClassifier(criterion='gini', max_depth=5, min_samples_leaf=10, random_state
[Link](X_train, y_train) # no scaling needed

11
3.4 Support Vector Machines (SVM)
Intuition: Finds the hyperplane that best separates classes by maximizing the margin —
the distance between the hyperplane and the nearest data points of each class (the support
vectors).
2
Math/Logic: - Hyperplane: 𝑤⋅𝑥+𝑏 = 0 - Margin width: ‖𝑤‖ — SVM’s objective is to maximize
this, equivalent to minimizing ‖𝑤‖ subject to correctly classifying all points (hard margin) or
allowing some slack for misclassification (soft margin, controlled by hyperparameter 𝐶 ). - 𝐶
(regularization): small 𝐶 → wider margin, tolerates more misclassification (simpler, more
regularized model); large 𝐶 → narrower margin, tries hard to classify every point correctly
(risk of overfitting).
Kernel Trick: When data isn’t linearly separable in its original space, SVM implicitly maps it
into a higher-dimensional space where a linear separator does exist — without explicitly com-
puting the transformation (computational efficiency via the kernel function). - RBF (Radial
2
Basis Function) Kernel: 𝐾(𝑥𝑖 , 𝑥𝑗 ) = 𝑒−𝛾‖𝑥𝑖 −𝑥𝑗 ‖ — creates flexible, non-linear boundaries;
𝛾 controls the influence radius of a single training point (high 𝛾 = tight, complex boundary →
overfitting risk). - Polynomial Kernel: 𝐾(𝑥𝑖 , 𝑥𝑗 ) = (𝑥𝑖 ⋅𝑥𝑗 +𝑐)𝑑 — captures polynomial-order
interactions.
Pros: Effective in high-dimensional spaces, robust to overfitting when margin maximization
is well-regularized, versatile via kernels. Cons: Computationally expensive on large datasets,
less interpretable, requires careful tuning of C and gamma, requires feature scaling.
from [Link] import SVC
model = SVC(kernel='rbf', C=1.0, gamma='scale')
[Link](X_train_scaled, y_train)

3.5 Naive Bayes

Intuition: A probabilistic classifier based on Bayes’ Theorem, with the “naive” assumption
that all features are conditionally independent given the class.
Bayes’ Theorem:
𝑃 (𝐵|𝐴) 𝑃 (𝐴)
𝑃 (𝐴|𝐵) =
𝑃 (𝐵)
Applied to classification (predict class 𝑦 given features 𝑥1 , … , 𝑥𝑛 ):
𝑛
𝑃 (𝑦|𝑥1 , … , 𝑥𝑛 ) ∝ 𝑃 (𝑦) ∏ 𝑃 (𝑥𝑖 |𝑦)
𝑖=1

• 𝑃 (𝑦) — prior probability of the class (from training data frequency).

• 𝑃 (𝑥𝑖 |𝑦) — likelihood of each feature given the class (estimated via Gaussian distribution
for continuous features, or frequency counts for categorical/text features).
• Conditional Independence Assumption: features are assumed independent given the
class — rarely true in reality, but the model still performs surprisingly well in practice
(especially text classification, spam filtering).
Variants: GaussianNB (continuous features), MultinomialNB (word counts, text), BernoulliNB
(binary features).

12
Pros: Extremely fast to train, works well with high-dimensional data (text), performs well
even with the independence assumption violated, needs relatively little training data. Cons:
The independence assumption can hurt accuracy when features are strongly correlated; prob-
ability estimates can be poorly calibrated even when class predictions are correct.
from sklearn.naive_bayes import GaussianNB
model = GaussianNB()
[Link](X_train, y_train)

3.6 Model Validation — K-Fold Cross Validation vs Train-Test Split

Simple Train-Test Split divides data once into (typically) 80% train / 20% test. Problem:
performance estimate depends heavily on which rows happened to land in the test set — a
single “lucky” or “unlucky” split can over- or under-estimate true model performance (high
variance in the evaluation itself). This risks both: - Bias if the split isn’t representative (e.g.,
class imbalance skewed into one split). - Data leakage risk if preprocessing (scaling, impu-
tation, feature selection) is fit on the full dataset before the split, letting test-set information
leak into training.
K-Fold Cross Validation: 1. Split data into 𝑘 equal folds. 2. Train on 𝑘 − 1 folds, validate
on the remaining fold. 3. Repeat 𝑘 times, rotating which fold is held out. 4. Average the 𝑘
performance scores for a more robust, lower-variance estimate.
from sklearn.model_selection import cross_val_score, StratifiedKFold

skf = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)

scores = cross_val_score(model, X_train_scaled, y_train, cv=skf, scoring='accuracy')
print("Mean CV Accuracy:", [Link](), "± ", [Link]())

Stratified K-Fold preserves class proportions in each fold — essential for imbalanced classi-
fication problems.
Why K-Fold is preferred: every data point gets used for both training and validation exactly
once across the folds, giving a much more reliable estimate of how the model generalizes, and
reducing the risk that results are an artifact of one particular split.

3.7 Project & Deployment Lessons — Heart Disease & Titanic Datasets
End-to-End Workflow
Raw Data
│
▼
Data Cleaning (nulls, duplicates, dtypes)
│
▼
Feature Encoding (categorical → numeric)
│
▼
Train-Test Split
│
▼
Feature Scaling (fit on train, transform both)

13
 │
▼
Model Training + K-Fold CV
│
▼
Model Evaluation & Comparison (accuracy, precision, recall, F1, ROC-AUC)
│
▼
Model Selection
│
▼
Model Persistence (pickle/joblib)
│
▼
Deployment (Flask/Streamlit/Django API)

Feature Encoding

# One-Hot Encoding (nominal, no order — e.g., chest pain type)

df = pd.get_dummies(df, columns=['cp', 'thal'], drop_first=True)

# Label Encoding (ordinal, has order — e.g., education level)

from [Link] import LabelEncoder
le = LabelEncoder()
df['education'] = le.fit_transform(df['education'])

Model Evaluation & Selection Logic

For the Heart Disease dataset (binary classification, moderate feature count, mixed nu-
meric/categorical), a typical comparison:
from [Link] import classification_report, confusion_matrix, roc_auc_score

models = {
'Logistic Regression': LogisticRegression(max_iter=1000),
'KNN': KNeighborsClassifier(n_neighbors=7),
'Decision Tree': DecisionTreeClassifier(max_depth=5),
'SVM': SVC(probability=True),
'Naive Bayes': GaussianNB()
}

for name, m in [Link]():

[Link](X_train_scaled, y_train)
preds = [Link](X_test_scaled)
print(name, "Accuracy:", (preds == y_test).mean())

Why KNN/SVM are often chosen for datasets like Heart Disease: these datasets are
moderate-sized, mostly numeric after encoding, and classes are often not perfectly linearly
separable — KNN captures local patterns well, and SVM (RBF kernel) captures non-linear
boundaries robustly, often outperforming plain Logistic Regression while remaining less
prone to overfitting than an unconstrained Decision Tree.

14
Model Persistence

import pickle

# Save
with open('heart_disease_model.pkl', 'wb') as f:
[Link](model, f)

# Also save the scaler! Predictions will be wrong without matching preprocessing.
with open('[Link]', 'wb') as f:
[Link](scaler, f)

# Load
with open('heart_disease_model.pkl', 'rb') as f:
loaded_model = [Link](f)

Alternative with joblib (preferred for large numpy-heavy models, e.g., Random Forests):
import joblib
[Link](model, 'heart_disease_model.joblib')
loaded_model = [Link]('heart_disease_model.joblib')

Deployment Architecture (Flask example)

from flask import Flask, request, jsonify

import pickle
import numpy as np

app = Flask(__name__)
model = [Link](open('heart_disease_model.pkl', 'rb'))
scaler = [Link](open('[Link]', 'rb'))

@[Link]('/predict', methods=['POST'])
def predict():
data = request.get_json()
features = [Link](data['features']).reshape(1, -1)
features_scaled = [Link](features)
prediction = [Link](features_scaled)
probability = model.predict_proba(features_scaled)[0][1]
return jsonify({'prediction': int(prediction[0]), 'probability': float(probability)})

if __name__ == '__main__':
[Link](debug=True)

Deployment architecture concept: the trained .pkl model is a static artifact — it’s loaded
once when the server starts, then reused for every incoming request. The UI (web form,
Streamlit app, or mobile client) sends raw feature values via an HTTP request → the API
applies the same preprocessing (scaler) used in training → feeds it to the loaded model →
returns a prediction as JSON. This separation (train once offline, serve many times online) is
the standard “training pipeline” vs “inference pipeline” split in production ML systems.

15
SECTION 4: Model Tuning, Ensemble Methods & Unsuper-
vised Learning
4.1 Hyperparameter Optimization
Parameters vs Hyperparameters:

Parameters Hyperparameters
Definition Learned automatically from Set manually before training
data during training begins
Examples Coefficients 𝜃 in Linear Learning rate 𝛼, k in KNN,
Regression, split thresholds C/gamma in SVM, max_depth in
in a Decision Tree, support Trees, number of trees in
vectors in SVM Random Forest
Who sets it The optimization algorithm The practitioner (or a search
(e.g., Gradient Descent) algorithm like
GridSearchCV)

GridSearchCV
Exhaustively tries every combination of specified hyperparameter values, using cross-
validation to score each combination, and returns the best-performing set.
from sklearn.model_selection import GridSearchCV
from [Link] import SVC

param_grid = {
'C': [0.1, 1, 10, 100],
'gamma': ['scale', 0.01, 0.1, 1],
'kernel': ['rbf']
}

grid = GridSearchCV(SVC(), param_grid, cv=5, scoring='accuracy', n_jobs=-1)

[Link](X_train_scaled, y_train)

print("Best Params:", grid.best_params_)

print("Best CV Score:", grid.best_score_)
best_model = grid.best_estimator_

For KNN:
param_grid_knn = {'n_neighbors': list(range(1, 21)), 'metric': ['euclidean', 'manhattan']}
grid_knn = GridSearchCV(KNeighborsClassifier(), param_grid_knn, cv=5)
grid_knn.fit(X_train_scaled, y_train)

RandomSearchCV: Instead of trying every combination, samples a fixed number of random

combinations from specified distributions — much faster for large search spaces, and often
finds near-optimal results with a fraction of the compute.
from sklearn.model_selection import RandomizedSearchCV
from [Link] import uniform, randint

param_dist = {'C': uniform(0.1, 100), 'gamma': uniform(0.001, 1)}

16
random_search = RandomizedSearchCV(SVC(kernel='rbf'), param_dist, n_iter=20, cv=5, random_state
random_search.fit(X_train_scaled, y_train)

GridSearchCV vs RandomSearchCV — when to use which: GridSearchCV is exhaustive

and guarantees finding the best combo within the grid, but scales poorly (combinatorial ex-
plosion) — use it for small search spaces with 2–3 hyperparameters. RandomSearchCV scales
much better for large/continuous search spaces and is the practical default when tuning 4+
hyperparameters or continuous ranges.

4.2 Ensemble Learning Techniques

Ensemble methods combine multiple “weaker” models to produce a stronger, more robust
predictor than any single model alone.

Bagging (Bootstrap Aggregating)

Core idea: Train multiple instances of the same algorithm on different bootstrap samples
(random samples drawn with replacement) of the training data, then aggregate their predic-
tions (majority vote for classification, average for regression). This reduces variance.
Random Forest — the canonical bagging algorithm: - Builds many Decision Trees, each on
a bootstrap sample. - At each split, only considers a random subset of features (adds further
decorrelation between trees, beyond just bootstrapping rows). - Final prediction = majority
vote (classification) / average (regression) across all trees.
Out-of-Bag (OOB) Score: Since each tree is trained on a bootstrap sample (~63.2% of
unique rows on average, due to sampling with replacement), the remaining ~36.8% “out-of-
bag” rows for that tree can be used as a free validation set — no need for a separate holdout.
from [Link] import RandomForestClassifier
rf = RandomForestClassifier(n_estimators=200, max_depth=10, oob_score=True, random_state=42)
[Link](X_train, y_train)
print("OOB Score:", rf.oob_score_)
print("Feature Importances:", rf.feature_importances_)

Boosting
Core idea: Train models sequentially, where each new model focuses on correcting the er-
rors (misclassified/high-residual points) made by the previous models. Combines many weak
learners (models barely better than random guessing, typically shallow trees/“stumps”) into
one strong learner. This reduces bias.

Algorithm Mechanism
AdaBoost Increases the sample weight of misclassified
points after each round, so the next weak
learner focuses harder on them; final
prediction is a weighted vote of all learners
Gradient Boosting Each new tree is trained to predict the
residual errors (gradient of the loss function)
of the combined ensemble so far,
progressively minimizing loss

17
Algorithm Mechanism
XGBoost An optimized, regularized, parallelized
implementation of Gradient Boosting — adds
L1/L2 regularization on tree weights,
handles missing values natively, and is
significantly faster; industry standard for
tabular data competitions

from [Link] import AdaBoostClassifier, GradientBoostingClassifier

ada = AdaBoostClassifier(n_estimators=100, learning_rate=0.5, random_state=42)

[Link](X_train, y_train)

gb = GradientBoostingClassifier(n_estimators=100, learning_rate=0.1, max_depth=3, random_state=

[Link](X_train, y_train)

# XGBoost (separate library)

# from xgboost import XGBClassifier
# xgb = XGBClassifier(n_estimators=200, learning_rate=0.1, max_depth=4)
# [Link](X_train, y_train)

Bagging vs Boosting — key difference: Bagging trains models independently and in par-
allel (variance reduction, e.g., Random Forest); Boosting trains models sequentially, each
depending on the last (bias reduction, e.g., XGBoost). Boosting typically achieves higher ac-
curacy but is more prone to overfitting and more sensitive to hyperparameters (learning rate,
number of estimators) than bagging.

4.3 Unsupervised Learning

K-Means Clustering
Intuition: Partitions data into 𝑘 clusters by iteratively assigning points to the nearest cluster
centroid, then recomputing centroids as the mean of assigned points.
Algorithm: 1. Choose 𝑘 (number of clusters) and initialize 𝑘 centroids (often via k-means++
for smarter initialization). 2. Assignment step: assign each point to its nearest centroid
(typically Euclidean distance). 3. Update step: recompute each centroid as the mean of all
points assigned to it. 4. Repeat steps 2–3 until centroids stabilize (convergence) or a max
iteration limit is reached.
Choosing 𝑘 — Elbow Method (WCSS):
𝐾
𝑊 𝐶𝑆𝑆 = ∑ ∑ ‖𝑥𝑖 − 𝜇𝑘 ‖2
𝑘=1 𝑥𝑖 ∈𝐶𝑘

(Within-Cluster Sum of Squares — total squared distance of points from their assigned cen-
troid.) Plot WCSS against different values of 𝑘; WCSS always decreases as 𝑘 increases, but
look for the “elbow” point where the rate of decrease sharply slows — that’s the point of
diminishing returns for adding more clusters.
from [Link] import KMeans
import [Link] as plt

18
wcss = []
for k in range(1, 11):
km = KMeans(n_clusters=k, init='k-means++', random_state=42, n_init=10)
[Link](X_scaled)
[Link](km.inertia_)

[Link](range(1, 11), wcss, marker='o')

[Link]('k'); [Link]('WCSS'); [Link]('Elbow Method')
[Link]()

Silhouette Score: A more rigorous complement to the elbow method, measuring how similar
a point is to its own cluster versus other clusters:

𝑏(𝑖) − 𝑎(𝑖)
𝑠(𝑖) =
max(𝑎(𝑖), 𝑏(𝑖))

where 𝑎(𝑖) = average distance to points in the same cluster, 𝑏(𝑖) = average distance to points
in the nearest other cluster. Ranges from -1 (poor clustering) to +1 (well-separated, dense
clusters).
from [Link] import silhouette_score
score = silhouette_score(X_scaled, km.labels_)

Dimensionality Reduction — PCA (Principal Component Analysis)

Intuition: PCA transforms correlated, high-dimensional features into a smaller set of uncor-
related “principal components” that capture the maximum possible variance in the data, in
decreasing order of importance.
Core concepts: - Variance Preservation: each principal component is chosen to capture
as much of the remaining variance in the data as possible, subject to being orthogonal (un-
correlated) to all previous components. - Eigenvalues & Eigenvectors: PCA computes the
covariance matrix of the (standardized) features, then finds its eigenvectors and eigenvalues.
- Eigenvectors define the direction of each principal component (the new axes). - Eigen-
values indicate how much variance is captured along that eigenvector’s direction — larger
eigenvalue = more important component. - The first principal component (PC1) is the eigen-
vector with the largest eigenvalue (captures the most variance); PC2 is orthogonal to PC1
and captures the next-largest remaining variance, and so on.
from [Link] import PCA

pca = PCA(n_components=0.95) # retain 95% of total variance

X_pca = pca.fit_transform(X_scaled) # MUST scale before PCA

print("Number of components chosen:", pca.n_components_)

print("Explained variance ratio:", pca.explained_variance_ratio_)

[Link]([Link](pca.explained_variance_ratio_))
[Link]('Number of Components'); [Link]('Cumulative Explained Variance')
[Link]()

Why scaling is mandatory before PCA: PCA identifies directions of maximum variance. If
features are on different scales (e.g., income in the thousands vs age in single/double digits),
the feature with the larger numeric range will dominate the variance calculation purely due

19
to units, not genuine importance — distorting the principal components. Standardization
ensures every feature contributes on equal footing.
When to use PCA: reducing training time/memory for high-dimensional data, mitigating mul-
ticollinearity before Linear/Logistic Regression, visualizing high-dimensional data in 2D/3D,
noise reduction. Trade-off: principal components are linear combinations of original features
and lose direct interpretability — a real cost when explainability matters (e.g., regulated do-
mains).

Pro Project Tips & Common Mistakes

1. Top 5 Common ML Mistakes Beginners Make

# Mistake Why It Hurts Fix

1 Scaling before Test-set statistics Always
Train-Test Split (mean, std, min, max) train_test_split
(fitting Standard- leak into training, first, then fit the
Scaler/MinMaxScaler inflating validation scaler only on
on the full dataset) performance in a X_train, and
way that won’t hold transform (not
in production — fit_transform) on
data leakage X_test
2 Ignoring class A model can achieve Check
imbalance 95% accuracy on a value_counts(normalize=True)
dataset that’s 95% on the target early;
one class by always use
predicting that class precision/recall/F1/ROC-
— accuracy looks AUC instead of
great but the model accuracy alone;
is useless for consider
detecting the class_weight='balanced',
minority class (e.g., SMOTE
fraud, disease) oversampling, or
stratified sampling
3 Not using A single split’s Use K-Fold (or
cross-validation / performance Stratified K-Fold for
relying on a single estimate has high classification)
train-test split variance — you cross-validation for a
might get lucky or robust performance
unlucky with which estimate before
rows land in the test finalizing a model
set

20
# Mistake Why It Hurts Fix
4 Blindly trusting Producing plots and After every major
model output metrics without result, write 1–2
without interpreting what sentences of
sanity-checking / they mean misses plain-language
writing bugs (e.g., leakage interpretation;
conclusions giving suspiciously question any metric
perfect scores) and that looks “too good”
produces reports
with no actionable
insight
5 Overfitting via Model memorizes Use regularization
unconstrained training data noise (L1/L2), constrain
model complexity instead of learning complexity
(deep trees, no generalizable (max_depth,
regularization, too patterns — great min_samples_leaf),
many features train score, poor monitor train vs
relative to data size) real-world validation gap, use
performance cross-validation

2. Production Pipeline Checklist

[ ] 1. Data Cleaning — handle nulls, duplicates, incorrect dtypes, outliers
[ ] 2. Encoding — categorical → numeric (one-hot / label / target encoding)
[ ] 3. Train-Test Split — BEFORE any fitting-based preprocessing step
[ ] 4. Scaling — fit on train only, transform train + test
[ ] 5. Cross-Validation — K-Fold / Stratified K-Fold for robust performance estimate
[ ] 6. Hyperparameter Tuning— GridSearchCV / RandomSearchCV using the CV pipeline
[ ] 7. Final Evaluation — on a held-out test set the model has never seen, using
task-appropriate metrics (not just accuracy)
[ ] 8. Pickling — save both the model AND the fitted preprocessing objects
(scaler, encoder) — inference must mirror training exactly
[ ] 9. Deployment — wrap in an API (Flask/FastAPI/Django) or app (Streamlit),
load pickled artifacts once at startup, serve predictions
[ ] 10. Monitoring — track live prediction distributions vs training distribution
to catch data/concept drift over time

3. Model Selection Cheat-Sheet

Scenario Recommended Starting Point Reasoning

Small dataset, need Linear/Logistic Fast, interpretable
interpretability, linear Regression coefficients, low variance on
relationship suspected small data
Small-to-medium dataset, Decision Tree (shallow, Visualizable rules, no scaling
mixed feature types, need pruned) needed, handles
interpretability non-linearity
Medium dataset, non-linear KNN or SVM (RBF kernel) Captures non-linear
boundary, numeric features, patterns; KNN if
scaling feasible simplicity/no training time
matters, SVM if margin
robustness matters

21
Scenario Recommended Starting Point Reasoning
Larger tabular dataset, Random Forest (bagging) Strong out-of-the-box
accuracy is the priority, performance, resistant to
interpretability secondary overfitting, provides feature
importances
Large tabular dataset, Gradient Boosting / State-of-the-art on
squeezing out maximum XGBoost (boosting) structured/tabular data,
accuracy, handles missing values,
competition/production- highly tunable
grade
High-dimensional sparse Naive Bayes Extremely fast, performs
data (text, spam) well despite the
independence assumption,
standard baseline for text
Need to reduce PCA (as a preprocessing Preserves variance,
dimensionality / visualize / step, not a classifier) decorrelates features,
remove multicollinearity first speeds up downstream
models
Dataset size < features (p » Regularized Regularization controls
n), risk of overfitting linear Linear/Logistic variance; embedded feature
models Regression (Lasso/Ridge) selection via L1
or tree ensembles
No labels available, K-Means Clustering Standard unsupervised
exploring natural groupings baseline; use Elbow +
Silhouette to pick k

Rule of thumb summary: - Interpretability matters → Linear/Logistic Regression, shallow

Decision Trees. - Accuracy matters, data is tabular → Random Forest → Gradient Boost-
ing/XGBoost (try both, compare via CV). - Data is text/high-dimensional-sparse → Naive
Bayes, or Logistic Regression with TF-IDF. - Non-linear but modest data size → SVM (RBF)
or KNN. - No labels → K-Means (clustering) or PCA (structure/visualization).

End of Mastery Lecture Notes — Complete Machine Learning Course (4-Part Series).

22