Understanding the Confusion Matrix in Machine Learning
A confusion matrix is a foundational performance measurement tool in machine learning
classification. It is a specific table layout that allows visualization of the performance of an
algorithm by directly comparing its predicted classifications against the actual, true labels.
A binary confusion matrix breaks down the model's predictions into four distinct quadrants:
• True Positives (TP): The model correctly predicted the positive class. (e.g., The model
flagged a cell as faulty, and it actually was faulty).
• True Negatives (TN): The model correctly predicted the negative class. (e.g., The
model labeled a cell as functioning, and it was indeed functioning).
• False Positives (FP): The model incorrectly predicted the positive class. (e.g., The
model flagged a cell as faulty, but it was actually fine—a false alarm).
• False Negatives (FN): The model incorrectly predicted the negative class. (e.g., The
model labeled a cell as functioning, but it was actually faulty—a miss).
Key Metrics Derived from the Matrix
• Raw counts are helpful, but we usually extract specific performance metrics from
these four quadrants to understand how the model is failing.
1. Accuracy The overall proportion of correct predictions. While common, it can be
misleading if your dataset is imbalanced (e.g., if 99% of your logic cells are
functioning, a model that simply guesses "functioning" every time will be 99%
accurate, but entirely useless).
2. Precision Out of all the instances the model predicted as positive, how many were
actually positive? This measures the quality of a positive prediction.
3. Recall (Sensitivity or True Positive Rate) Out of all the actual positive instances in
the data, how many did the model successfully find? This measures the model's ability
to detect the target class.
4. F1-Score The harmonic mean of Precision and Recall. It provides a single score that
balances both concerns, which is especially critical when dealing with uneven class
distributions.
Confusion Matrix Problems
Problem 1: Calculating the Core Metrics
The Scenario: You have developed a binary classification model to identify whether a
machine part is defective (Positive) or normal (Negative). You test the model on 1000 parts
and get the following raw results:
• The model correctly identified 60 defective parts.
• The model correctly identified 880 normal parts.
• The model flagged 20 normal parts as defective.
• The model missed 40 defective parts, labeling them as normal.
The Task: Construct the confusion matrix and calculate the Accuracy, Precision, Recall, and
F1-Score.
The Solution: First, map the raw results to the confusion matrix quadrants:
• True Positives (TP): 60
• True Negatives (TN): 880
• False Positives (FP): 20
• False Negatives (FN): 40
Predicted Predicted
Defective (Pos) Normal (Neg)
Actual 40 (False
60 (True Positive)
Defective (Pos) Negative)
Actual Normal 880 (True
20 (False Positive)
(Neg) Negative)
Now, apply the formulas:
• Accuracy:
𝑇𝑃 + 𝑇𝑁 60 + 880
Accuracy = = = 0.94 (or 94\%)
𝑇𝑃 + 𝑇𝑁 + 𝐹𝑃 + 𝐹𝑁 1000
• Precision:
𝑇𝑃 60
Precision = = = 0.75
𝑇𝑃 + 𝐹𝑃 60 + 20
• Recall:
𝑇𝑃 60
Recall = = = 0.60
𝑇𝑃 + 𝐹𝑁 60 + 40
• F1-Score:
Precision × Recall 0.75 × 0.60
F1 = 2 × = 2× ≈ 0.667
Precision + Recall 0.75 + 0.60
Problem 2:
An IoT-based precision farming system is deployed in an arecanut plantation. It uses soil
moisture sensors and an ML algorithm to predict if a specific sector of trees requires
immediate irrigation (Positive = Needs Water, Negative = Sufficiently Hydrated). Over a
month, it makes 500 daily sector predictions.
Here are the results:
• It correctly triggered the irrigation system 80 times when the soil was actually dry.
• It correctly kept the water off 380 times when the soil was sufficiently moist.
• It incorrectly turned the water on 30 times when the soil was already wet enough.
• It failed to turn the water on 10 times when the trees actually needed it.
The Task:
Map the values and calculate the four core metrics.
The Solution:
• True Positives (TP): 80
• True Negatives (TN): 380
• False Positives (FP): 30
• False Negatives (FN): 10
Predicted Needs Predicted
Water (Pos) Hydrated (Neg)
Actual Needs
80 (True Positive) 10 (False Negative)
Water (Pos)
Predicted Needs Predicted
Water (Pos) Hydrated (Neg)
Actual Hydrated
30 (False Positive) 380 (True Negative)
(Neg)
The calculations:
• Accuracy:
80 + 380 460
Accuracy = = = 0.92
500 500
• Precision:
80 80
Precision = = ≈ 0.727
80 + 30 110
• Recall:
80 80
Recall = = ≈ 0.888
80 + 10 90
• F1-Score:
0.727 × 0.888
F1 = 2 × ≈ 0.799
0.727 + 0.888
Problem 3: Automated LMS Flagging
The Scenario:
A new Learning Management System (LMS) features an automated tool designed to flag
potentially plagiarized assignment submissions for instructor review (Positive = Plagiarized,
Negative = Original Work). You test it against a historical database of 1,000 student
submissions where the true status is already known.
Here are the results:
• The tool flagged 95 submissions in total.
• Upon checking those 95 flags, 45 were actually plagiarized, but 50 were entirely
original work.
• The system correctly ignored 900 original submissions.
• The system failed to catch 5 submissions that were known to be plagiarized.
The Task:
Map the values and calculate the metrics. (Hint: This scenario presents the numbers slightly
differently).
The Solution:
First, carefully extract the quadrants. We know the total flags (Predicted Positives) is 95, and
45 of those were correct, meaning the remaining 50 are false alarms.
• True Positives (TP): 45
• False Positives (FP): 50
• True Negatives (TN): 900
• False Negatives (FN): 5
Predicted Plagiarized Predicted Original
(Pos) (Neg)
Actual Plagiarized
45 (True Positive) 5 (False Negative)
(Pos)
Actual Original
50 (False Positive) 900 (True Negative)
(Neg)
The calculations:
• Accuracy:
45 + 900
Accuracy = = 0.945
1000
• Precision:
45 45
Precision = = ≈ 0.473
45 + 50 95
• Recall:
45 45
Recall = = = 0.90
45 + 5 50
• F1-Score:
0.473 × 0.90
F1 = 2 × ≈ 0.620
0.473 + 0.90
Problem 4: Fraud Detection
The Scenario: An e-commerce platform has deployed a new machine learning model to
detect fraudulent credit card transactions.
• Positive Class: Fraudulent transaction
• Negative Class: Legitimate transaction
The team tests the model on a batch of 9,000 recent transactions. Here are the raw results
from the trial:
• The model successfully caught 120 actual fraudulent transactions.
• The model incorrectly flagged 350 completely legitimate transactions as fraud.
• The model failed to detect 30 fraudulent transactions, letting them slip through.
• The model correctly identified 8,500 transactions as legitimate.
Step 1: Mapping the Confusion Matrix
First, we extract the numbers from the scenario and place them into their correct quadrants.
• True Positives (TP): 120 (The model successfully caught actual fraudulent
transactions).
• True Negatives (TN): 8500 (The model correctly identified transactions as
legitimate).
• False Positives (FP): 350 (The model incorrectly flagged completely legitimate
transactions as fraud).
• False Negatives (FN): 30 (The model failed to detect fraudulent transactions, letting
them slip through).
(Quick check: $120 + 8500 + 350 + 30 = 9000$ total transactions. The math checks out!)
Predicted Fraud (Pos) Predicted Legitimate (Neg)
Actual Fraud (Pos) 120 (True Positive) 30 (False Negative)
Actual Legitimate (Neg) 350 (False Positive) 8500 (True Negative)
Step 2: Calculating the Metrics
Now, we plug those values into the core formulas.
1. Accuracy
How often is the model correct overall?
120 + 8500 8620
𝐴𝑐𝑐𝑢𝑟𝑎𝑐𝑦 = = ≈ 0.9578
9000 9000
Answer: ~95.8%
2. Precision
When the model flags a transaction as fraud, how often is it actually fraud?
120 120
𝑃𝑟𝑒𝑐𝑖𝑠𝑖𝑜𝑛 = = ≈ 0.2553
120 + 350 470
Answer: ~25.5%
3. Recall
Out of all the actual fraudulent transactions, what percentage did the model successfully
catch?
120 120
𝑅𝑒𝑐𝑎𝑙𝑙 = = = 0.80
120 + 30 150
Answer: 80.0%
4. F1-Score
The harmonic mean balancing Precision and Recall.
0.2553 × 0.80 0.2042
𝐹1 = 2 × ≈ 2× ≈ 0.387
0.2553 + 0.80 1.0553
Answer: ~0.387
Interpreting the Results
This is a classic example of why Accuracy can be deceptive. The model is nearly 96%
accurate, which sounds fantastic to executives.
However, looking at the other metrics tells the real story:
• The 80% Recall is decent; the platform is catching the vast majority of the actual
fraud.
• The 25.5% Precision is a major operational issue. It means that roughly 3 out of
every 4 times a customer's credit card is blocked for "fraud," it is a false alarm (False
Positive). This will likely lead to furious customers and a flood of customer support
tickets.