Selected Topics 2 in Data

Engineering
Dr. Ibrahim Gomaa
Lecture 6:
Supervised Learning — Classification

An in-depth exploration of classification algorithms, evaluation metrics, model

validation, and best practices for building reliable machine learning models.
Chapter Overview

What We'll Cover

01 02

Foundations of Supervised Learning Classification Types & Evaluation

Labeled data, input features, output labels, and the core learning Binary, multi-class, multi-label classification; confusion matrix; key
objective. metrics.

03 04

Model Validation & Data Leakage Algorithms & Best Practices

Train-test splits, leakage types, prevention strategies, and evaluation Overview of common classifiers, their trade-offs, challenges, and
workflows. recommendations.
Foundations of Supervised Machine
Learning
Supervised machine learning is a paradigm in which a model is trained on a
dataset of labeled examples. Every training instance has two essential
components: a set of input features (X) — the independent variables such as
age, salary, or pixel values — and an output label (Y) — the target the model
must predict, such as a purchase decision or spam status.

The algorithm's core objective is to learn a mapping function f : X → Y that

accurately approximates the relationship between inputs and outputs, so it can
generalize to new, unseen data. In essence, supervised learning is learning from
history where the correct answers are already known.
A Concrete Example: Predicting Customer Purchases
Consider a dataset where historical records include a customer's age and salary alongside whether they made a purchase. The model
learns patterns from these labeled rows to predict outcomes for new customers it has never seen before.

Age Salary (USD) Purchased

25 30,000 No

40 80,000 Yes

35 55,000 Yes

22 24,000 No

The model analyzes the input features (Age, Salary) to predict the target label (Purchased). Once trained, it can classify new, unlabeled
customers by recognizing similar patterns in the data.
Taxonomy of Supervised Learning
Supervised learning problems are broadly categorized based on the nature of the output variable. The two primary branches are
classification and regression.

Classification Regression
Predicts a discrete, categorical output label. The model assigns Predicts a continuous, numerical output value. The model
an input to one of a finite set of classes. estimates a quantity along a spectrum.

• Email: Spam / Not Spam • House price in USD

• Medical: Disease / No Disease • Temperature in °C
• Image: Cat / Dog / Bird • Sales volume in units

This chapter focuses exclusively on classification, the most widely applied category of supervised learning tasks.
Classification: A Deeper Perspective
Classification is a supervised learning task where the model predicts a class label for a given input instance. The structure of the label set
defines the specific type of classification problem at hand.

Binary Classification Multi-Class Classification Multi- Label Classification

The output has exactly two mutually The output has more than two Each instance can simultaneously
exclusive classes. Every prediction is mutually exclusive classes. Each belong to multiple classes. Labels are
one or the other — no ambiguity. instance belongs to exactly one class. not mutually exclusive.

Example: Spam (1) vs. Not Spam (0); Example: Iris species — Setosa, Example: A movie tagged as both
Disease (Yes) vs. No Disease (No). Versicolor, or Virginica. "Action" and "Comedy" and "Drama."
The Three Stages of Classification

Train Model
Algorithm learns
patterns

Input Features Predict Labels

Provide raw labeled Classify unseen
data instances

Every classification workflow follows this progression: structured input data is fed to a learning algorithm, which discovers patterns in the labeled training
examples. The resulting trained model is then deployed to assign class labels to new, previously unseen data points.
The Confusion Matrix
The confusion matrix is the foundational tool for evaluating a classification
model. It provides a tabular summary of every prediction outcome — correct
and incorrect — giving a complete picture of model behavior that a single
number cannot capture.
Reading the Confusion Matrix
Predicted Positive Predicted Negative

Actual Positive True Positive (TP) False Negative (FN)

Actual Negative False Positive (FP) True Negative (TN)

TP — True Positive TN — True Negative FP — False Positive FN — False Negative

Correctly predicted positive Correctly predicted negative Incorrectly flagged as Missed a true positive. Also
cases. The model said "yes" cases. The model said "no" positive. Also known as a known as a Type II error.
and it was right. and it was right. Type I error.
Key Evaluation Metrics
From the confusion matrix, several performance metrics can be derived. Each metric illuminates a different dimension of model behavior
and is suitable for different business contexts. No single metric tells the whole story.

Accuracy Precision
The proportion of all predictions that are correct: (TP + TN) / Of all instances predicted positive, the fraction that truly are:
(TP + TN + FP + FN). Best used when classes are balanced. TP / (TP + FP). Minimizes false positives.

Recall ( Sensitivity) F1- Score

Of all actual positive instances, the fraction correctly The harmonic mean of precision and recall: 2 × (Precision ×
identified: TP / (TP + FN). Minimizes false negatives. Recall) / (Precision + Recall). Balances both concerns.
Metric Formulas at a Glance
Accuracy Recall

Precision F1- Score

The F1-score is especially valuable for imbalanced datasets, where accuracy alone can be deceptive. It forces a balance between avoiding
false positives and not missing true positives.
Selecting the Right Metric: A Business View
The choice of evaluation metric should always be driven by the business context and the relative cost of each type of error. The same
model may be evaluated differently depending on the stakes involved.

Prioritize Precision Prioritize Recall Prioritize Accuracy

Use when false positives are costly. A Use when false negatives are dangerous. Use when classes are balanced and error
spam filter that flags a legitimate email In disease screening, missing a true case can costs are equal. A product
causes real harm — prioritize precision to be life-threatening — prioritize recall to recommendation engine where incorrect
ensure flagged items are truly spam. catch every positive case. predictions have low impact is a good
candidate.
The Imbalanced Data Pitfall

Caution: Accuracy can be a dangerously misleading metric when class distributions are skewed. Never rely on accuracy alone
for imbalanced problems.

The Scenario The Lesson

A fraud detection dataset contains 95% non-fraudulent and only In this scenario, the F1-score or recall for the minority class
5% fraudulent transactions. A naive model that always predicts would correctly reveal the model's failure. High accuracy paired
"non-fraud" achieves 95% accuracy — yet it detects zero fraud with poor recall on the minority class is a red flag that demands
cases and is completely useless for its intended purpose. investigation and appropriate metric selection.
Model Validation: The Train- Test
Split
To reliably assess a model's ability to generalize to new, unseen data, the
available dataset must be partitioned into two distinct subsets before any
training occurs. This separation is a non-negotiable requirement for
trustworthy evaluation.
Training Set vs. Test Set
Training Set ( 70– 80% ) Test Set ( 20– 30% )
The larger partition used to train the model. The algorithm A held-out partition used exclusively for final evaluation. It
sees these labeled examples, learns feature patterns, and simulates real-world unseen data and provides an unbiased
adjusts its internal parameters. This data must never be estimate of the model's performance after training is
withheld from the model during training. complete. Never use test data during training.

The primary purpose of this separation is to verify that the model generalizes well and has not simply memorized the training data — a
problem known as overfitting. A model that performs well on training data but poorly on the test set has failed to learn the true
underlying patterns.
Data Leakage: A Critical Concept

Data leakage is one of the most dangerous and subtle flaws in machine learning
pipelines. It occurs when information from outside the training dataset —
such as the test set or future data — is inadvertently used during model training,
producing an overly optimistic and fundamentally unreliable evaluation.
Why Data Leakage Is Dangerous
Inflated Performance Metrics Deployment Failure Subtle and Hard to Detect
When deployed to production, the Leakage can hide throughout the
The model appears to perform model fails to deliver similar results data pipeline and is often discovered
exceptionally well during testing — because the leaked information is no only after deployment failures occur,
sometimes achieving near-perfect longer available, leading to making it costly to fix after the fact.
scores — creating false confidence in erroneous business decisions and lost
its capabilities. trust.
Two Core Types of Data Leakage
Train- Test Contamination Target Leakage

Occurs when information from the test set influences the Occurs when a feature included in training directly reveals or
training phase. A common example: scaling or normalizing the proxies the target variable. The model learns to "cheat" rather
entire dataset before splitting. The scaler's parameters (mean, than learning genuine patterns. For example, including a patient's
standard deviation) are then influenced by test data, leaking final diagnosis note as a feature when predicting that same
information into the training process. diagnosis.
Leakage in Practice: Three Illustrative Examples
1 2 3

Preprocessing Before Splitting Direct Proxy for the Target Future Information Included
Wrong: Normalizing the full dataset, Scenario: Predicting disease diagnosis. Scenario: Predicting customer churn.
then splitting. Right: Split first, then fit Leaky Feature: "Doctor's Final Leaky Feature: "Account Closure Date"
the scaler on training data only and Diagnosis Note" — this directly reveals — this is only available after churn has
apply it to the test set. the answer and produces useless results already occurred and cannot be used
in practice. for prediction.
Five Strategies to Prevent Data Leakage

Fit Transformations on Training Data Only

Split Data First
Scalers, encoders, and imputers must be fitted on training data,
Always perform the train-test split before any preprocessing or then applied (not re-fitted) on the test set.
feature engineering step.

Leverage ML Pipelines
Scrutinize Leaky Features
Use tools like scikit-learn Pipelines to automate the correct order
Ask: "Would this information be available at prediction time?" If of operations and prevent accidental leakage.
not, remove or re-engineer the feature.

For time-series problems: Always use a chronological split — train on past data, validate on future data. Never shuffle temporal
data before splitting.
Standard Model Evaluation Workflow
A robust and reproducible evaluation process follows a well-defined sequence of steps. Skipping or reordering steps introduces bias and undermines the
reliability of your results.

Data Collection Preprocessing

Partitioning Training Prediction

Following this sequence rigorously ensures that each step operates on the correct data partition, evaluation metrics accurately reflect real-world performance,
and data leakage is structurally prevented from entering the pipeline.
Overview of Common Classification
Algorithms
A rich ecosystem of algorithms exists for classification tasks. Each carries
distinct strengths, limitations, and ideal use cases. Choosing the right algorithm
requires understanding both the nature of your data and your performance
priorities.
Logistic Regression & Decision Trees

Logistic Regression Decision Tree

A linear model that estimates class probabilities using a logistic A tree-structured model that recursively splits data based on
(sigmoid) function. It is simple, fast, and highly interpretable, feature values, following a series of if-then rules. It is highly
making it an excellent baseline and the go-to choice when a linear interpretable and easy to visualize, requires minimal data
decision boundary is sufficient or when interpretability is preprocessing, and handles non-linear relationships naturally.
paramount. Its main limitation is an inability to capture complex, However, it is prone to overfitting unless pruned, and can be
non-linear patterns in data. unstable when data changes slightly.
Random Forest & Support Vector Machine

Random Forest Support Vector Machine ( SVM)

An ensemble of many decision trees, each trained on a Finds the optimal hyperplane that maximizes the margin between
bootstrapped sample and a random feature subset. By averaging classes. Exceptionally effective in high-dimensional spaces and for
predictions across trees, Random Forest achieves high accuracy complex, non-linear boundaries using kernel functions. SVM is
and robustness, handles high-dimensional data well, resists robust to overfitting in high-dimensional settings but can be slow to
overfitting, and provides feature importance rankings. The trade- train on large datasets and is sensitive to feature scaling.
off is reduced interpretability and higher computational cost.
k- NN, Naive Bayes & Gradient Boosting
k-Nearest Neighbors Naive Bayes Gradient Boosting ( XGBoost,
LightGBM)
A non-parametric learner that classifies A probabilistic classifier based on Bayes'
each instance by a majority vote theorem with an assumption of feature Builds models sequentially, each
among its k closest neighbors. Intuitive independence. Extremely fast to train correcting the errors of its predecessor.
and requires no training phase. Best for and predict. Excels at text Delivers state-of-the-art performance
small datasets where instance classification and high-dimensional on structured data, handles mixed data
similarity is a strong predictor. Slow at sparse data (e.g., spam filtering). The types, and often wins competitions.
prediction time on large data and independence assumption is often Requires careful hyperparameter
sensitive to irrelevant features. violated, limiting performance on more tuning and is more computationally
complex tasks. expensive to train.
Neural Networks ( Deep Learning)

Multi-layered, highly parameterized models capable of learning hierarchical

representations from raw data. Neural networks achieve exceptional flexibility
and state-of-the-art results on large-scale, unstructured data such as images,
audio, and natural language. Their main trade-offs are the requirement for very
large datasets, significant computational resources (GPUs), and low
interpretability — they effectively function as "black boxes," making it difficult
to understand why a particular prediction was made.
Algorithm Quick Selection Guide
Choosing the right algorithm depends on your data characteristics, interpretability requirements, and accuracy targets. Use this guide as a starting point before
experimenting.
Common Challenges in Classification

Even with the right algorithm and clean data, classification tasks are rarely
straightforward. Understanding and anticipating these challenges is a hallmark
of a skilled practitioner.
Five Core Challenges Explained
Imbalanced Datasets Overfitting
When one class significantly outnumbers the others (e.g., The model memorizes the training data — including its noise
95% negative, 5% positive), the model tends to be biased — and fails to generalize to new examples. Signs include
toward predicting the majority class. Standard accuracy very high training accuracy paired with significantly lower
metrics become misleading, and the minority class — often test accuracy. Regularization, pruning, and ensemble
the class of greatest interest — is effectively ignored. methods help mitigate this.

Underfitting Noisy Data & Feature Engineering

The model is too simple to capture the underlying structure Errors, outliers, and irrelevant features degrade model
of the data, performing poorly on both training and test sets. performance. Identifying the most predictive input variables
This often results from using a model with too few through careful feature selection and engineering is often
parameters or insufficient training iterations. the most impactful step in the entire modeling process.
Best Practices for Classification
Building reliable, production-ready classification models requires more than selecting the right algorithm. These best practices distinguish
rigorous work from ad hoc experimentation.

Align Metrics with Business Goals Leverage the Confusion Matrix

Select evaluation metrics based on the relative costs of different Always examine the full confusion matrix. It reveals nuances — like
error types — not solely on accuracy. Always ask what type of strong precision paired with poor recall — that a single aggregate
mistake is more costly in your specific context. metric will obscure.

Address Imbalanced Data Prevent Data Leakage

Use techniques such as oversampling the minority class (SMOTE), Adhere to a rigorous workflow: split data first, fit all
undersampling the majority class, or applying class weights in the transformations on training data only, and scrutinize every feature
algorithm to give minority classes appropriate influence. for potential leakage before including it in the model.
Validate on Unseen Data — Always
The Golden Rule Why This Matters

Use a held-out test set to obtain an unbiased estimate of Every time you evaluate on the test set and adjust your model in
real-world performance. The test set must remain response, you are implicitly using test set information in your training
untouched until the final evaluation — treat it like exam decisions — a subtle form of data leakage. A validation set acts as a safe
day data you've never seen. intermediate checkpoint, protecting the integrity of your final evaluation
and giving you a trustworthy measure of how the model will perform
For iterative model development, introduce a separate
when deployed in the real world.
validation set (or use cross-validation) for tuning
hyperparameters, reserving the test set exclusively for
final assessment.
Chapter 2 — Key Takeaways
Supervised Learning Fundamentals Evaluation is Multi-Dimensional
Models learn a mapping f: X → Y from labeled data. Use the confusion matrix as your foundation. Choose accuracy,
Classification predicts discrete labels; regression predicts precision, recall, or F1 based on the business cost of each
continuous values. error type.

Validation Must Be Rigorous Algorithm Selection is Context- Driven

Always split data before preprocessing. Keep the test set truly Match the algorithm to the data size, dimensionality,
held-out. Use pipelines to enforce the correct order of interpretability needs, and accuracy targets. No single
operations and prevent leakage. algorithm dominates all scenarios.
Looking Ahead
With a solid foundation in classification fundamentals, evaluation metrics, and validation
strategies, you are now equipped to approach real-world supervised learning problems with
rigor and confidence. The next chapters will build on these concepts to explore specific
algorithms in greater depth, including how to tune them, interpret their outputs, and deploy
them responsibly in production environments.

Decision Trees In Depth

Ensemble Methods

Deep Learning

Model Deployment