PYTHON PROGRAMMING

MASTERY GUIDE
From Zero to Professional Developer

PART 13
Machine Learning

Fundamentals • Scikit-learn • Supervised Learning • Unsupervised

Model Evaluation • Cross-Validation • Pipelines • Real-World Projects
PART 13: MACHINE LEARNING
Machine Learning (ML) is the discipline of teaching computers to learn patterns from data and make
predictions or decisions without being explicitly programmed for every scenario. It is the technology
behind Netflix recommendations, spam filters, fraud detection, voice assistants, medical diagnosis
systems, and self-driving cars.

Python — through the Scikit-learn library — is the most widely used language for practical machine
learning. In this part you will understand the core ML concepts, implement algorithms from scratch
conceptually, then use Scikit-learn to build, evaluate, and optimise real models on real data.

🎯 What You Will Learn in Part 13

13.1 Machine Learning Fundamentals — types, terminology, the ML workflow

13.2 Scikit-learn API — the consistent fit/predict/score interface
13.3 Data Preparation for ML — feature engineering, scaling, encoding
13.4 Supervised Learning: Regression — linear, polynomial, ridge, lasso
13.5 Supervised Learning: Classification — logistic regression, KNN, SVM, naive Bayes
13.6 Tree-Based Models — decision trees, random forests, gradient boosting
13.7 Unsupervised Learning — K-means clustering, PCA dimensionality reduction
13.8 Model Evaluation — metrics, confusion matrix, ROC-AUC, cross-validation
13.9 Hyperparameter Tuning — GridSearchCV, RandomizedSearchCV
13.10 Pipelines — building robust, reproducible ML workflows
13.11 Real-World ML Projects
13.12 Exercises, Knowledge Check, Common Mistakes, Professional Tips

13.1 Machine Learning Fundamentals

Types of Machine Learning

Type Description Examples Scikit-learn
Supervised Learn from labelled data Spam detection, price LinearRegressio
Learning (input-output pairs) prediction, disease diagnosis n,
RandomForest,
SVC
Unsupervised Find patterns in unlabelled Customer segmentation, KMeans,
Learning data anomaly detection, topic DBSCAN, PCA
modelling
Semi-supervised Small amount of labelled + Image classification with few LabelPropagatio
large unlabelled labels n
Reinforcement Agent learns by trial and error Game playing, robotics, (gym, stable-
Learning with rewards trading baselines3)
Core ML Terminology
Term Definition Example
Feature (X) Input variable used to make predictions Age, salary, number of rooms
Target (y) Output variable we want to predict House price, spam/not-spam,
disease yes/no
Sample / Instance One row of data — one observation One house listing, one email,
one patient record
Training set Data used to train (fit) the model 80% of your dataset
Test set Held-out data to evaluate final model 20% of your dataset
performance
Validation set Data used to tune hyperparameters during Subset of training, or use cross-
development validation
Overfitting Model learns noise in training data — poor High training accuracy, low test
generalisation accuracy
Underfitting Model is too simple — misses real Low training AND test accuracy
patterns
Hyperparameter Parameter set BEFORE training (not Max depth of a tree, learning
learned from data) rate, k in KNN
Bias Error from wrong assumptions — Linear model on non-linear data
underfitting
Variance Error from sensitivity to training data — Complex model that memorises
overfitting training data
Feature Engineering Creating new features from existing ones year-month from a date column

The Machine Learning Workflow

# The ML workflow
# The standard ML workflow (every project follows this)

# Step 1: Define the problem

# What are we predicting? What type of problem is it?

# Step 2: Collect and explore data (EDA — covered in Part 12)

# Load data, check shape, missing values, distributions

# Step 3: Prepare the data

# Clean, engineer features, encode categoricals, scale numerics

# Step 4: Split data

from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test = train_test_split(
X, y, test_size=0.2, random_state=42
)
 # Step 5: Choose and train a model
from [Link] import RandomForestClassifier
model = RandomForestClassifier(n_estimators=100, random_state=42)
[Link](X_train, y_train)

# Step 6: Evaluate the model

score = [Link](X_test, y_test) # accuracy
print(f'Accuracy: {score:.4f}')

# Step 7: Tune hyperparameters

# (GridSearchCV, RandomizedSearchCV — covered in section 13.9)

# Step 8: Make predictions

predictions = [Link](X_test)
probabilities = model.predict_proba(X_test)

13.2 The Scikit-learn API

Scikit-learn's greatest strength is its consistent, unified API. Every algorithm — regardless of how
different it is mathematically — follows the same interface. Once you learn it for one algorithm, you
know it for all of them.

# Scikit-learn API
# The Scikit-learn Estimator API
#
# estimator = AlgorithmClass(hyperparameter1=val, hyperparameter2=val)
# [Link](X_train, y_train) — learn from training data
# predictions = [Link](X) — make predictions
# score = [Link](X_test, y_test) — evaluate performance

# SAME API for ALL algorithms:

from sklearn.linear_model import LinearRegression, LogisticRegression

from [Link] import DecisionTreeClassifier
from [Link] import RandomForestClassifier,
GradientBoostingClassifier
from [Link] import SVC, SVR
from [Link] import KNeighborsClassifier
from sklearn.naive_bayes import GaussianNB
from [Link] import KMeans
from [Link] import PCA

# All follow the same pattern:

model = RandomForestClassifier(n_estimators=100)
[Link](X_train, y_train)
y_pred = [Link](X_test)
print([Link](X_test, y_test))
 # Key attributes after fitting:
# model.coef_ — learned coefficients (linear models)
# model.feature_importances_ — feature importance (tree models)
# model.classes_ — class labels (classifiers)
# model.n_features_in_ — number of features seen during fit

13.3 Data Preparation for ML

Raw data is almost never suitable for machine learning directly. Feature scaling, encoding categorical
variables, and handling missing values are essential preprocessing steps that can have a dramatic
impact on model performance.

Feature Scaling
# Feature scaling
from [Link] import StandardScaler, MinMaxScaler,
RobustScaler
import numpy as np

# StandardScaler — mean=0, std=1 (Z-score normalisation)

# Use when: data is roughly normally distributed
# Required by: SVM, KNN, logistic regression, neural networks
scaler = StandardScaler()
X_train_scaled = scaler.fit_transform(X_train) # fit AND transform train
X_test_scaled = [Link](X_test) # ONLY transform test
(no fit!)

# MinMaxScaler — scales to [0, 1] range

# Use when: you need values in a bounded range
mms = MinMaxScaler()
X_scaled = mms.fit_transform(X)

# RobustScaler — uses median and IQR (robust to outliers)

# Use when: your data has significant outliers
rs = RobustScaler()
X_robust = rs.fit_transform(X)

# KEY RULE: Always fit the scaler on TRAINING data only,

# then use transform() on BOTH train and test.
# Fitting on all data causes data leakage!

Encoding Categorical Variables

# Encoding categorical variables
from [Link] import LabelEncoder, OneHotEncoder
from [Link] import ColumnTransformer
import pandas as pd
 # LabelEncoder — converts categories to integers 0, 1, 2 ...
# Use ONLY for the target variable (y), not features!
le = LabelEncoder()
y_encoded = le.fit_transform(['cat','dog','cat','bird','dog'])
print(y_encoded) # [1 2 1 0 2]
print(le.classes_) # ['bird' 'cat' 'dog']

# OneHotEncoder — creates binary columns for each category

# Use for: nominal (unordered) categorical FEATURES
ohe = OneHotEncoder(sparse_output=False, drop='first')
colours = [['red'],['green'],['blue'],['red'],['blue']]
encoded = ohe.fit_transform(colours)
print(encoded) # [[0,1],[1,0],[0,0],[0,1],[0,0]] (drop='first' drops
'blue')

# OrdinalEncoder — ordered categories (small < medium < large)

from [Link] import OrdinalEncoder
oe = OrdinalEncoder(categories=[['small','medium','large']])

# ColumnTransformer — apply different transformers to different columns

from [Link] import ColumnTransformer

numeric_features = ['age', 'salary', 'experience']

categoric_features= ['department', 'city']

preprocessor = ColumnTransformer(transformers=[
('num', StandardScaler(), numeric_features),
('cat', OneHotEncoder(drop='first'), categoric_features)
])

Handling Missing Values in ML

# Missing value imputation
from [Link] import SimpleImputer, KNNImputer

# SimpleImputer — fill with mean, median, most_frequent, or constant

imputer = SimpleImputer(strategy='median')
X_imputed = imputer.fit_transform(X)

# KNNImputer — fill using K nearest neighbours (more accurate)

knn_imp = KNNImputer(n_neighbors=5)
X_knn = knn_imp.fit_transform(X)

13.4 Supervised Learning: Regression

Regression predicts a continuous numerical value. Examples: predicting house prices, forecasting
sales, estimating patient recovery time.
Linear Regression
# Linear regression
import numpy as np
import pandas as pd
import [Link] as plt
from sklearn.linear_model import LinearRegression
from sklearn.model_selection import train_test_split
from [Link] import mean_squared_error, mean_absolute_error,
r2_score
from [Link] import fetch_california_housing

# Load dataset
housing = fetch_california_housing(as_frame=True)
X, y = [Link], [Link]

# Split
X_train, X_test, y_train, y_test = train_test_split(
X, y, test_size=0.2, random_state=42
)

# Scale features
from [Link] import StandardScaler
scaler = StandardScaler()
X_train_s = scaler.fit_transform(X_train)
X_test_s = [Link](X_test)

# Train
lr = LinearRegression()
[Link](X_train_s, y_train)

# Evaluate
y_pred = [Link](X_test_s)
mse = mean_squared_error(y_test, y_pred)
mae = mean_absolute_error(y_test, y_pred)
r2 = r2_score(y_test, y_pred)
rmse = [Link](mse)

print(f'MAE: {mae:.4f}') # Mean Absolute Error

print(f'RMSE: {rmse:.4f}') # Root Mean Squared Error
print(f'R²: {r2:.4f}') # R-squared (1.0 = perfect)

# Coefficients
coef_df = [Link]({
'Feature': [Link],
'Coefficient': lr.coef_
}).sort_values('Coefficient', ascending=False)
print(coef_df)
Regularised Regression — Ridge, Lasso, ElasticNet
# Ridge, Lasso, ElasticNet
from sklearn.linear_model import Ridge, Lasso, ElasticNet

# Ridge (L2 regularisation) — shrinks coefficients, keeps all features

# Use when: many features, all possibly relevant
ridge = Ridge(alpha=1.0) # alpha controls regularisation strength
[Link](X_train_s, y_train)
print(f'Ridge R²: {[Link](X_test_s, y_test):.4f}')

# Lasso (L1 regularisation) — can shrink coefficients to 0 (feature

selection)
# Use when: you suspect many features are irrelevant
lasso = Lasso(alpha=0.1)
[Link](X_train_s, y_train)
n_zero = [Link](lasso.coef_ == 0)
print(f'Lasso R²: {[Link](X_test_s, y_test):.4f}, zeros: {n_zero}')

# ElasticNet — combination of L1 and L2

en = ElasticNet(alpha=0.1, l1_ratio=0.5)
[Link](X_train_s, y_train)

# Compare
models = {'LinearReg': lr, 'Ridge': ridge, 'Lasso': lasso, 'ElasticNet':
en}
for name, model in [Link]():
r2 = [Link](X_test_s, y_test)
print(f'{name:<12}: R² = {r2:.4f}')

Polynomial Regression
# Polynomial regression
from [Link] import PolynomialFeatures
from [Link] import Pipeline

# Polynomial features transform x -> [1, x, x^2, x^3, ...]

# Combined with linear regression, models non-linear relationships

degrees = [1, 2, 3, 4]
fig, axes = [Link](1, 4, figsize=(18, 4))

X_sample = [Link]([Link](80, 1), axis=0)

y_sample = [Link](2 * [Link] * X_sample).ravel() + [Link](80) *
0.1
X_samp_tr, X_samp_te, y_samp_tr, y_samp_te = train_test_split(
X_sample, y_sample, test_size=0.2, random_state=42)

for ax, degree in zip(axes, degrees):

model = Pipeline([
('poly', PolynomialFeatures(degree=degree)),
 ('scaler', StandardScaler()),
('regression', LinearRegression())
])
[Link](X_samp_tr, y_samp_tr)
train_r2 = [Link](X_samp_tr, y_samp_tr)
test_r2 = [Link](X_samp_te, y_samp_te)

X_plot = [Link](0, 1, 100).reshape(-1, 1)

y_plot = [Link](X_plot)

[Link](X_sample, y_sample, alpha=0.5, s=20)

[Link](X_plot, y_plot, 'r-', linewidth=2)
ax.set_title(f'Degree {degree}\nTrain R²={train_r2:.2f}, Test
R²={test_r2:.2f}')

plt.tight_layout()

13.5 Supervised Learning: Classification

Classification predicts which category an item belongs to. Examples: spam/not-spam, disease
diagnosis, customer churn, sentiment analysis (positive/negative).

Logistic Regression
# Logistic regression
from sklearn.linear_model import LogisticRegression
from [Link] import load_breast_cancer
from sklearn.model_selection import train_test_split
from [Link] import StandardScaler
from [Link] import classification_report, confusion_matrix

# Load binary classification dataset

data = load_breast_cancer()
X, y = [Link], [Link]

X_train, X_test, y_train, y_test = train_test_split(

X, y, test_size=0.2, random_state=42, stratify=y # stratify preserves
class ratio
)

scaler = StandardScaler()
X_tr_s = scaler.fit_transform(X_train)
X_te_s = [Link](X_test)

log_reg = LogisticRegression(max_iter=1000, random_state=42)

log_reg.fit(X_tr_s, y_train)

y_pred = log_reg.predict(X_te_s)
 y_proba = log_reg.predict_proba(X_te_s)[:, 1] # probability of class 1

print(f'Accuracy: {log_reg.score(X_te_s, y_test):.4f}')

print()
print(classification_report(y_test, y_pred,
target_names=data.target_names))

K-Nearest Neighbours (KNN)

# K-Nearest Neighbours
from [Link] import KNeighborsClassifier

# KNN predicts class by voting among the K nearest training points

# K is a critical hyperparameter — too small = overfitting, too large =
underfitting

# Find optimal K
train_scores, test_scores = [], []
k_range = range(1, 31)

for k in k_range:
knn = KNeighborsClassifier(n_neighbors=k)
[Link](X_tr_s, y_train)
train_scores.append([Link](X_tr_s, y_train))
test_scores.append([Link](X_te_s, y_test))

best_k = k_range[test_scores.index(max(test_scores))]
print(f'Best k: {best_k}, Test accuracy: {max(test_scores):.4f}')

# Train with best K

knn = KNeighborsClassifier(n_neighbors=best_k)
[Link](X_tr_s, y_train)
print(f'KNN Accuracy: {[Link](X_te_s, y_test):.4f}')

Support Vector Machine (SVM)

# Support Vector Machine
from [Link] import SVC

# SVM finds the hyperplane that maximally separates classes

# kernel: 'linear', 'rbf' (radial basis function), 'poly', 'sigmoid'
# C: regularisation — small C = wider margin, more misclassification
allowed
# gamma: kernel coefficient for 'rbf' — controls influence radius

svm = SVC(kernel='rbf', C=1.0, gamma='scale', probability=True,

random_state=42)
[Link](X_tr_s, y_train)
print(f'SVM Accuracy: {[Link](X_te_s, y_test):.4f}')
Naive Bayes
# Naive Bayes
from sklearn.naive_bayes import GaussianNB, MultinomialNB

# GaussianNB — for continuous features

gnb = GaussianNB()
[Link](X_tr_s, y_train)
print(f'GaussianNB Accuracy: {[Link](X_te_s, y_test):.4f}')

# MultinomialNB — for count data (text classification, word frequencies)

# from sklearn.naive_bayes import MultinomialNB
# nb = MultinomialNB()
# Excellent for text classification tasks

13.6 Tree-Based Models

Tree-based models are among the most powerful and widely used ML algorithms in industry. They
require minimal preprocessing (no scaling needed), handle mixed data types naturally, and are highly
interpretable. Ensemble methods (Random Forest, Gradient Boosting) consistently win machine
learning competitions.

Decision Tree
# Decision tree
from [Link] import DecisionTreeClassifier, export_text, plot_tree
import [Link] as plt

# Decision tree splits data by asking yes/no questions at each node

# max_depth controls complexity — deeper = more complex = more overfitting
risk

dt = DecisionTreeClassifier(max_depth=4, random_state=42)
[Link](X_train, y_train)

print(f'Train accuracy: {[Link](X_train, y_train):.4f}')

print(f'Test accuracy: {[Link](X_test, y_test):.4f}')

# Feature importance
importances = [Link]({
'Feature': data.feature_names,
'Importance': dt.feature_importances_
}).sort_values('Importance', ascending=False)
print('\nTop 5 Features:')
print([Link]())

# Visualise the tree

fig, ax = [Link](figsize=(20, 8))
 plot_tree(dt, feature_names=data.feature_names,
class_names=data.target_names,
filled=True, ax=ax)
plt.tight_layout()

Random Forest
# Random Forest
from [Link] import RandomForestClassifier

# Random Forest: ensemble of decision trees

# Each tree trained on a random subset of data and features
# Final prediction: majority vote of all trees
# Benefits: robust to overfitting, handles high dimensions, built-in
feature importance

rf = RandomForestClassifier(
n_estimators=200, # number of trees (more = better, up to
diminishing returns)
max_depth=None, # let trees grow fully (forest handles
overfitting via averaging)
max_features='sqrt', # consider sqrt(n_features) at each split
min_samples_split=2,
n_jobs=-1, # use all CPU cores
random_state=42
)
[Link](X_train, y_train)

print(f'RF Train: {[Link](X_train, y_train):.4f}')

print(f'RF Test: {[Link](X_test, y_test):.4f}')

# Feature importance plot

importances = [Link](rf.feature_importances_, index=data.feature_names)
top_features = [Link](10)

fig, ax = [Link](figsize=(10, 6))

top_features.sort_values().plot(kind='barh', color='steelblue', ax=ax)
ax.set_title('Random Forest Feature Importances')
plt.tight_layout()

Gradient Boosting — XGBoost Style

# Gradient boosting
from [Link] import GradientBoostingClassifier,
HistGradientBoostingClassifier

# Gradient Boosting builds trees SEQUENTIALLY

# Each tree corrects the errors of the previous one
# HistGradientBoosting is the faster, modern version
 gb = HistGradientBoostingClassifier(
max_iter=200, # number of boosting rounds
learning_rate=0.05, # step size (smaller = slower but more precise)
max_depth=4,
min_samples_leaf=20,
random_state=42
)
[Link](X_train, y_train)
print(f'GradBoost Test: {[Link](X_test, y_test):.4f}')

# XGBoost (install separately: pip install xgboost)

# from xgboost import XGBClassifier
# xgb = XGBClassifier(n_estimators=200, learning_rate=0.05,
# max_depth=4, use_label_encoder=False, eval_metric='logloss')
# [Link](X_train, y_train)

# Algorithm comparison
algorithms = {
'Logistic Regression': log_reg,
'KNN': knn,
'SVM': svm,
'Decision Tree': dt,
'Random Forest': rf,
'Gradient Boosting': gb,
}
print('\nAlgorithm Comparison:')
for name, model in [Link]():
# Note: some models need scaled data, others don't
print(f' {name:<22}: {[Link](X_test, y_test):.4f}')

13.7 Unsupervised Learning

Unsupervised learning discovers patterns in data without labels. You don't tell the algorithm what to
look for — it finds structure on its own. The two most common techniques are clustering (grouping
similar items) and dimensionality reduction (compressing features).

K-Means Clustering
# K-Means clustering
from [Link] import KMeans
from [Link] import StandardScaler
from [Link] import make_blobs
import [Link] as plt
import numpy as np

# Generate sample data with 4 natural clusters

X, y_true = make_blobs(n_samples=300, centers=4, cluster_std=0.7,
random_state=42)
 # Scale
scaler = StandardScaler()
X_scaled = scaler.fit_transform(X)

# Finding the optimal K — Elbow Method

inertias = []
k_range = range(1, 11)

for k in k_range:
km = KMeans(n_clusters=k, random_state=42, n_init=10)
[Link](X_scaled)
[Link](km.inertia_)

# Plot elbow curve

[Link](figsize=(8, 4))
[Link](k_range, inertias, 'bo-', linewidth=2)
[Link]('Number of Clusters (K)')
[Link]('Inertia (Within-cluster sum of squares)')
[Link]('Elbow Method — Finding Optimal K')
[Link](k_range)
plt.tight_layout()

# Train with optimal K=4

kmeans = KMeans(n_clusters=4, random_state=42, n_init=10)
labels = kmeans.fit_predict(X_scaled)

# Visualise clusters
[Link](figsize=(10, 5))
[Link](1, 2, 1)
[Link](X[:, 0], X[:, 1], c=y_true, cmap='Set1', alpha=0.7)
[Link]('True Labels')

[Link](1, 2, 2)
[Link](X[:, 0], X[:, 1], c=labels, cmap='Set1', alpha=0.7)
centroids = scaler.inverse_transform(kmeans.cluster_centers_)
[Link](centroids[:,0], centroids[:,1], c='black', marker='X', s=200,
zorder=5)
[Link]('K-Means Clusters (X = centroids)')
plt.tight_layout()

PCA — Principal Component Analysis

# PCA
from [Link] import PCA
from [Link] import load_digits
import [Link] as plt

# PCA reduces dimensionality while preserving maximum variance

# Use for: visualisation, noise reduction, speeding up downstream models
 digits = load_digits()
X_dig = [Link] # 1797 samples, 64 features (8x8 pixels)
y_dig = [Link]

# Reduce 64 dimensions to 2 for visualisation

pca = PCA(n_components=2)
X_pca = pca.fit_transform(X_dig)

print(f'Original shape: {X_dig.shape}') # (1797, 64)

print(f'Reduced shape: {X_pca.shape}') # (1797, 2)
print(f'Variance explained: {pca.explained_variance_ratio_.sum()*100:.1f}
%')

[Link](figsize=(10, 7))
scatter = [Link](X_pca[:,0], X_pca[:,1], c=y_dig,
cmap='tab10', alpha=0.7, s=10)
[Link](scatter, label='Digit')
[Link]('Digits Dataset — PCA to 2 Dimensions')
plt.tight_layout()

# How many components to retain 95% variance?

pca_full = PCA().fit(X_dig)
cumvar = [Link](pca_full.explained_variance_ratio_)
n_95 = [Link](cumvar >= 0.95) + 1
print(f'Components for 95% variance: {n_95}') # e.g., 29 out of 64

13.8 Model Evaluation

Choosing the right evaluation metric is as important as choosing the right algorithm. A model with 99%
accuracy sounds great — but if 99% of the data is one class, a model that always predicts that class
achieves 99% accuracy without learning anything.

Regression Metrics
Metric Formula (conceptual) Interpretation Best Value
MAE — Mean Mean of |y - y_pred| Average prediction error in 0 (lower is
Absolute Error original units better)
MSE — Mean Mean of (y - y_pred)² Penalises large errors more 0 (lower is
Squared Error heavily than MAE better)
RMSE — Root MSE sqrt(MSE) Same units as target — 0 (lower is
most interpretable better)
R² — Coefficient of 1 - SS_res/SS_tot Proportion of variance 1.0 (higher is
Determination explained (1=perfect) better)
MAPE — Mean Mean of |y-y_pred|/|y| * 100 Percentage error — good for 0% (lower is
Absolute % Error comparing across scales better)
Classification Metrics
# Classification metrics
from [Link] import (
accuracy_score, precision_score, recall_score, f1_score,
confusion_matrix, classification_report, roc_auc_score, roc_curve
)
import [Link] as plt
import seaborn as sns

y_pred = [Link](X_test)
y_proba = rf.predict_proba(X_test)[:, 1]

# Core metrics
print(f'Accuracy: {accuracy_score(y_test, y_pred):.4f}')
print(f'Precision: {precision_score(y_test, y_pred):.4f}')
print(f'Recall: {recall_score(y_test, y_pred):.4f}')
print(f'F1 Score: {f1_score(y_test, y_pred):.4f}')
print(f'ROC-AUC: {roc_auc_score(y_test, y_proba):.4f}')

# Full classification report

print(classification_report(y_test, y_pred,
target_names=data.target_names))

# Confusion matrix
cm = confusion_matrix(y_test, y_pred)
fig, axes = [Link](1, 2, figsize=(14, 5))

[Link](cm, annot=True, fmt='d', cmap='Blues',

xticklabels=data.target_names,
yticklabels=data.target_names, ax=axes[0])
axes[0].set_title('Confusion Matrix')
axes[0].set_ylabel('Actual')
axes[0].set_xlabel('Predicted')

# ROC Curve
fpr, tpr, thresholds = roc_curve(y_test, y_proba)
auc = roc_auc_score(y_test, y_proba)
axes[1].plot(fpr, tpr, 'b-', linewidth=2, label=f'AUC = {auc:.4f}')
axes[1].plot([0,1],[0,1],'r--', label='Random classifier')
axes[1].set_xlabel('False Positive Rate')
axes[1].set_ylabel('True Positive Rate')
axes[1].set_title('ROC Curve')
axes[1].legend()
plt.tight_layout()

Metric Cheat Sheet

Metric When to Use
 Accuracy Balanced classes, equal cost for each type of error
Precision Cost of False Positive is high (spam filter — don't miss real emails)
Recall (Sensitivity) Cost of False Negative is high (cancer diagnosis — never miss a case)
F1 Score Imbalanced classes or both FP and FN matter equally
ROC-AUC Comparing models regardless of threshold; probability ranking quality
R² Regression — proportion of variance explained by the model
RMSE Regression — when large errors should be penalised more

Cross-Validation
# Cross-validation
from sklearn.model_selection import cross_val_score, StratifiedKFold

# K-Fold Cross-Validation: split data into K folds,

# train on K-1, test on 1, repeat K times, average the scores

cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)

for name, model in [Link]():

scores = cross_val_score(model, X, y, cv=cv, scoring='accuracy',
n_jobs=-1)
print(f'{name:<22}: {[Link]():.4f} (+/- {[Link]()*2:.4f})')

# Cross-validation gives much more reliable performance estimates

# than a single train/test split

13.9 Hyperparameter Tuning

Hyperparameters are settings you choose before training. Finding the best ones is critical for model
performance. Scikit-learn provides automated search tools to do this systematically.

# Hyperparameter tuning
from sklearn.model_selection import GridSearchCV, RandomizedSearchCV
from [Link] import randint, uniform

# ── GridSearchCV — exhaustive search over specified parameter grid ─

param_grid = {
'n_estimators': [50, 100, 200],
'max_depth': [None, 5, 10, 20],
'max_features': ['sqrt', 'log2']
}

grid_search = GridSearchCV(
RandomForestClassifier(random_state=42),
 param_grid,
cv=5,
scoring='f1',
n_jobs=-1,
verbose=1
)
grid_search.fit(X_train, y_train)

print(f'Best params: {grid_search.best_params_}')

print(f'Best CV F1: {grid_search.best_score_:.4f}')
print(f'Test F1: {f1_score(y_test, grid_search.predict(X_test)):.4f}')

# ── RandomizedSearchCV — random sample from distributions (faster) ─

param_dist = {
'n_estimators': randint(50, 500),
'max_depth': [None, 5, 10, 15, 20],
'max_features': ['sqrt', 'log2', None],
'min_samples_split': randint(2, 20),
'min_samples_leaf': randint(1, 10)
}

random_search = RandomizedSearchCV(
RandomForestClassifier(random_state=42),
param_dist,
n_iter=50, # try 50 random combinations
cv=5,
scoring='f1',
n_jobs=-1,
random_state=42
)
random_search.fit(X_train, y_train)
print(f'Random Search Best: {random_search.best_params_}')

13.10 ML Pipelines
A Pipeline chains multiple processing steps into a single object. This prevents data leakage, makes
code cleaner, enables easy serialisation, and ensures the exact same transformations are applied to
training and production data.

# ML Pipeline
from [Link] import Pipeline
from [Link] import ColumnTransformer
from [Link] import StandardScaler, OneHotEncoder
from [Link] import SimpleImputer
from [Link] import RandomForestClassifier
from sklearn.model_selection import cross_val_score
import pandas as pd
# Example: customer churn prediction with mixed data types
numeric_features = ['age', 'tenure', 'monthly_charges', 'total_charges']
categoric_features= ['gender', 'contract', 'internet_service',
'payment_method']

# Numeric pipeline: impute -> scale

numeric_pipeline = Pipeline([
('imputer', SimpleImputer(strategy='median')),
('scaler', StandardScaler())
])

# Categorical pipeline: impute -> encode

categoric_pipeline = Pipeline([
('imputer', SimpleImputer(strategy='most_frequent')),
('encoder', OneHotEncoder(handle_unknown='ignore', drop='first'))
])

# Combine into ColumnTransformer

preprocessor = ColumnTransformer(transformers=[
('num', numeric_pipeline, numeric_features),
('cat', categoric_pipeline, categoric_features)
])

# Full pipeline: preprocessor + model

full_pipeline = Pipeline([
('preprocessor', preprocessor),
('classifier', RandomForestClassifier(n_estimators=100,
random_state=42))
])

# Train and evaluate — pipeline handles ALL steps automatically

full_pipeline.fit(X_train, y_train)
print(f'Pipeline Accuracy: {full_pipeline.score(X_test, y_test):.4f}')

# Cross-validate the full pipeline

cv_scores = cross_val_score(full_pipeline, X, y, cv=5, scoring='accuracy')
print(f'CV Accuracy: {cv_scores.mean():.4f} (+/-
{cv_scores.std()*2:.4f})')

# Save the pipeline

import joblib
[Link](full_pipeline, 'churn_model.pkl')

# Load and predict in production

loaded = [Link]('churn_model.pkl')
new_customer = [Link]({...}) # new customer data
prediction = [Link](new_customer)
13.11 Real-World Project: Titanic Survival Prediction
# Titanic survival prediction
import pandas as pd
import numpy as np
import [Link] as plt
import seaborn as sns
from sklearn.model_selection import train_test_split, cross_val_score
from [Link] import Pipeline
from [Link] import ColumnTransformer
from [Link] import StandardScaler, OneHotEncoder
from [Link] import SimpleImputer
from [Link] import RandomForestClassifier,
GradientBoostingClassifier
from [Link] import classification_report, roc_auc_score

# Load data
titanic = sns.load_dataset('titanic')

# Feature selection
features = ['pclass', 'sex', 'age', 'sibsp', 'parch', 'fare', 'embarked']
target = 'survived'

X = titanic[features]
y = titanic[target]

# Define column types

numeric_cols = ['age', 'fare', 'sibsp', 'parch']
categoric_cols= ['pclass', 'sex', 'embarked']

# Feature engineering
X = [Link]()
X['family_size'] = X['sibsp'] + X['parch'] + 1
X['is_alone'] = (X['family_size'] == 1).astype(int)
numeric_cols.extend(['family_size', 'is_alone'])

# Pipelines
num_pipe = Pipeline([
('impute', SimpleImputer(strategy='median')),
('scale', StandardScaler())
])
cat_pipe = Pipeline([
('impute', SimpleImputer(strategy='most_frequent')),
('encode', OneHotEncoder(drop='first', sparse_output=False))
])

preprocessor = ColumnTransformer([
('num', num_pipe, numeric_cols),
('cat', cat_pipe, categoric_cols)
])
 # Models to compare
models = {
'Random Forest': RandomForestClassifier(n_estimators=200,
random_state=42),
'Gradient Boost': GradientBoostingClassifier(n_estimators=200,
random_state=42)
}

X_train, X_test, y_train, y_test = train_test_split(

X, y, test_size=0.2, random_state=42, stratify=y
)

for name, clf in [Link]():

pipe = Pipeline([('prep', preprocessor), ('clf', clf)])
[Link](X_train, y_train)
y_pred = [Link](X_test)
y_proba = pipe.predict_proba(X_test)[:,1]
auc = roc_auc_score(y_test, y_proba)
acc = [Link](X_test, y_test)
print(f'{name}: Accuracy={acc:.4f}, AUC={auc:.4f}')
print(classification_report(y_test, y_pred,
target_names=['Died','Survived']))

13.12 Exercises and Projects

Exercise Set A: Regression

1. Load the California Housing dataset ([Link].fetch_california_housing). Build and
compare LinearRegression, Ridge (alpha=0.1, 1, 10), Lasso (alpha=0.01, 0.1, 1), and
RandomForestRegressor. Report MAE, RMSE, and R² for each.
2. Build a polynomial regression model that fits a sine wave. Experiment with degrees 1-8. Plot
training vs test R² against degree to visualise the bias-variance tradeoff.

Exercise Set B: Classification

3. Load the Wine dataset ([Link].load_wine — 3 classes). Train KNN, Decision Tree,
Random Forest, and SVM. Use 5-fold cross-validation. Compare accuracy, precision, recall,
and F1 for each algorithm.
4. Build a complete fraud detection pipeline using an imbalanced dataset. Handle class imbalance
with class_weight='balanced'. Evaluate using precision, recall, F1, and ROC-AUC (do NOT use
accuracy as the main metric).

Exercise Set C: Clustering and Pipelines

5. Apply K-Means clustering to the Iris dataset (ignore labels). Use the Elbow method to find
optimal K. Compare clusters found vs true species labels.
 6. Build a complete end-to-end ML pipeline for a mixed dataset (numeric + categorical). Include
imputation, scaling, encoding, a classifier, and hyperparameter tuning via GridSearchCV. Save
the trained pipeline with joblib.

Project: Customer Churn Prediction

Build a production-ready customer churn prediction system:
7. Generate or use a telecom churn dataset with features: age, tenure, monthly_charges,
total_charges, contract_type, internet_service, payment_method, num_complaints, churn
(target).
8. Full EDA: churn rate by feature, correlations, missing value analysis.
9. Feature engineering: customer_lifetime_value, avg_monthly_spend, complaint_rate.
10. Build pipeline with ColumnTransformer, imputation, scaling, encoding.
11. Train and compare: Logistic Regression, Random Forest, Gradient Boosting.
12. Tune the best model with RandomizedSearchCV.
13. Final evaluation: classification report, confusion matrix, ROC-AUC curve.
14. Save the production pipeline. Write a predict(customer_data) function.

13.13 Knowledge Check

# Question
1 What is the difference between supervised and unsupervised learning?
2 What is overfitting and how do you detect it?
3 Why must you fit the StandardScaler on training data only?
4 When would you use Lasso regression over Ridge?
5 What does stratify=y do in train_test_split?
6 How does a Random Forest differ from a single Decision Tree?
7 What is the Elbow Method in K-Means clustering?
8 When should you use Recall as your primary metric instead of Accuracy?
9 What does a ROC-AUC of 0.5 mean?
10 What is data leakage and how do Pipelines prevent it?
11 What is cross-validation and why is it better than a single train/test split?
12 What is the difference between GridSearchCV and RandomizedSearchCV?

13.14 Common Mistakes

Mistake Problem Fix
Fitting scaler on all data Data leakage — test data Fit only on X_train; transform X_train and
influences the model X_test separately
Using accuracy on 99% accuracy on 99% majority Use F1, Recall, ROC-AUC for imbalanced
imbalanced data class is meaningless datasets
Not scaling for Distance-based models very Always scale before KNN, SVM, or neural
KNN/SVM sensitive to scale networks
Skipping cross- Single split may be lucky or Always use 5-fold or 10-fold cross-validation
validation unlucky
Evaluating on training Overly optimistic — model has Always evaluate on held-out test data
data memorised training data
Too many Exponential search space — Use RandomizedSearchCV or narrow the grid
hyperparameters in too slow
GridSearch
Forgetting to set Results change every run — Set random_state=42 on all stochastic
random_state not reproducible components
Treating clustering as K-Means has no ground truth Use internal metrics: silhouette score, inertia
supervised labels

13.15 Professional Tips

🏆 Industry Best Practices for Machine Learning

1. ALWAYS BUILD A BASELINE FIRST: Before any ML, compute a naive baseline
(always predict the majority class, or the mean for regression). Your ML model
must beat this — if it doesn't, something is wrong.

2. SPLIT BEFORE PREPROCESSING: Create your train/test split FIRST. All

preprocessing
(scaling, imputation, encoding) must be fit on training data only.

3. USE PIPELINES ALWAYS: They prevent leakage, make code cleaner, enable easy
deployment, and ensure reproducibility. Never preprocess outside a Pipeline.

4. START SIMPLE: Begin with Logistic Regression or Linear Regression. Complex models
are harder to debug, explain, and deploy. Simple models often match complex ones.

5. CROSS-VALIDATE EVERYTHING: A single train/test split can mislead you. Always

use at least 5-fold cross-validation for model selection.

6. FEATURE IMPORTANCE IS NOT CAUSATION: A feature being important to a model

does not mean it causes the outcome. Be careful about causal claims.
 7. SAVE YOUR MODELS WITH METADATA: When you [Link]() a model, also save
the training date, dataset version, metrics, and hyperparameters used.

8. MONITOR MODELS IN PRODUCTION: Data distributions shift over time (concept drift).
Schedule regular retraining and monitor prediction distributions.

13.16 Part 13 Summary

📚 What You Learned in Part 13

✓ Supervised learning uses labelled data; unsupervised learning finds patterns without
labels
✓ The ML workflow: problem definition → EDA → prepare → split → train → evaluate →
tune → deploy
✓ Scikit-learn's consistent fit/predict/score API works the same for all algorithms
✓ Always fit preprocessing (scaling, imputation) on training data only — never on test data
✓ Regression metrics: MAE, RMSE, R² — choose based on whether large errors need
extra penalty
✓ Classification metrics: accuracy (balanced), precision/recall (imbalanced), F1, ROC-AUC
✓ Linear regression with L1 (Lasso) or L2 (Ridge) regularisation prevents overfitting
✓ Random Forest: ensemble of trees, robust, built-in feature importance, no scaling
needed
✓ Gradient Boosting: sequential tree building, often highest accuracy, slower to train
✓ K-Means clustering: find K cluster centroids; use Elbow method to choose K
✓ PCA: reduce dimensions while preserving maximum variance; visualise high-
dimensional data
✓ Cross-validation gives reliable performance estimates; GridSearchCV tunes
hyperparameters
✓ Pipelines chain preprocessing and modelling, prevent leakage, and simplify deployment

➡️ Coming Up in Part 14: Web Development

In Part 14 we build web applications and APIs with Python:

• Flask — lightweight web framework for APIs and small apps

• Django — full-featured framework for complex web applications
• FastAPI — modern, high-performance async API framework
• REST API design principles
• Authentication — JWT, sessions, OAuth basics
• Deployment — Heroku, Docker basics, environment variables
 — End of Part 13 —
Python Programming Mastery Guide | Part 13: Machine Learning