CLASS: X
SECTION: ABEF
SUBJECT: ARTIFICIAL INTELLIGENCE
SESSION: 2026-27
UNIT: 3
EVALUATING MODELS - NOTES
Evaluation: Evaluation is the process of checking how well an AI model works by comparing the model's
predictions with the actual correct answers (Reality). The goal of evaluation is to measure the accuracy of
the model and find areas for improvement.
Prediction: The output or result produced by the AI model after processing the input data.
Reality: The true or correct outcome that is already known. It’s the ground truth that is used to
evaluate the model’s accuracy.
Example: Let’s say we have an AI model designed to identify whether an image shows a dog or a cat.
• Prediction: The model predicts that the image is of a dog.
• Reality: The actual correct answer (Reality) is that the image is a dog.
Since the Prediction matches the Reality, the model made a correct prediction. If the Prediction was
"cat" and Reality was "dog," the model would be incorrect.
Need for Evaluation:
• Helps assess if the model is making accurate predictions.
• Identifies mistakes and areas where the model needs improvement.
• Improves the model’s performance over time.
Understanding how to Evaluate AI Model in a scenario:
Let’s imagine we have an AI model that predicts whether a person is sick or healthy based on medical
data. The model gives its Prediction, which we compare to the actual condition (Reality). Here are the
different cases:
1. True Positive (TP):
Reality: The person is sick.
Prediction: The model correctly predicts the person is sick.
This is a correct prediction, and it matches reality.
2. True Negative (TN):
Reality: The person is healthy.
Prediction: The model correctly predicts the person is healthy.
Another correct prediction, matching reality.
3. False Positive (FP):
Reality: The person is healthy.
Prediction: The model incorrectly predicts the person is sick.
The model raises a false alarm, predicting sickness when the person is actually healthy.
4. False Negative (FN):
Reality: The person is sick.
Prediction: The model incorrectly predicts the person is healthy.
This is a missed diagnosis, where the model fails to detect sickness.
These four cases help evaluate how accurate and reliable the AI model is by comparing its
predictions with real-life outcomes.
QVS Academic Resource Sheet [Link]
Confusion matrix:
A Confusion Matrix is a table that records the results of comparing the AI model's predictions with reality.
It helps us understand the model's performance but is not an evaluation metric itself—just a tool to
organize the outcomes.
Here are the four conditions in a scenario:
• True Positive (TP): The model makes a correct prediction when the actual condition is positive.
• True Negative (TN): The model makes a correct prediction when the actual condition is negative.
• False Positive (FP): The model incorrectly predicts a positive outcome when the actual condition
is negative.
• False Negative (FN): The model incorrectly predicts a negative outcome when the actual condition
is positive.
REALITY
Confusion Matrix
Yes No
Yes True Positive (TP) False Positive (FP)
Prediction No False Negative (FN) True Negative (TN)
Prediction and Reality can be mapped using a confusion matrix.
Evaluation Methods:
After understanding the combinations of Prediction and Reality, let’s explore how these conditions help
us evaluate the performance of an AI model.
1. ACCURACY:
• Definition: Accuracy measures the percentage of correct predictions out of all observations.
• Correct Predictions: Occur in True Positive (TP) and True Negative (TN) cases, where the
prediction matches reality.
• Formula for accuracy
Accuracy helps determine how often the model’s predictions are correct, providing a general measure of
performance.
• Total Observations: Includes all possible outcomes—True Positive (TP), True Negative (TN),
False Positive (FP), and False Negative (FN).
• Meaning of Accuracy: Accuracy measures how often the model's predictions match reality. It
gives an idea of how reliable the model is in making correct predictions.
2. PRECISION
• Definition: Precision measures the percentage of correctly predicted positive cases out of all
cases where the model predicted positive. It focuses on how many of the predicted positives
were actually correct.
• Formula for Precision:
QVS Academic Resource Sheet [Link]
RECALL:
Definition: Recall measures how well the model identifies actual positive cases. It is the proportion
of true positive cases among all actual positive cases.
Formula for Recall:
3. F1 SCORE
The F1 Score measures the balance between Precision and Recall. It provides a single value that
combines both metrics, helping us evaluate the model’s performance effectively.
• Formula for F1 Score:
• Perfect F1 Score:
o A perfect F1 Score of 1 (100%) occurs when both Precision and Recall are 1 (100%).
o The F1 score ranges from 0 to 1, where 1 indicates the best possible performance.
The F1 score is useful when we need a balance between Precision and Recall, especially when both
metrics are equally important.
Precision Vs Recall
Feature Precision Recall
Of all predicted positives, how Of all actual positives, how many
Definition
many are correct? are found?
When Important When false positives are costly When false negatives are costly
Out of emails marked spam, Out of all spam emails, how many
Example (Spam Filter)
how many are really spam? were marked as spam?
Which Metric is Important: Precision or Recall?
The importance of Precision or Recall depends on the scenario:
• Use Recall:
o When missing positive cases (False Negatives) is risky.
o Example: Forest Fire Detection (failing to predict a fire can be catastrophic) and Viral
Outbreaks (missing early detection could spread the virus).
• Use Precision:
o When acting on incorrect predictions (False Positives) is costly.
o Example: Mining (wasting resources on false treasure predictions) and Spam Detection
(marking important emails as spam could cause data loss).
Ethical Concerns in Model Evaluation
While evaluating an AI model, the following ethical concerns need to be kept in mind
QVS Academic Resource Sheet [Link]