Unit – 3 Evaluation metrics
for Classification
What is Classification?
► You go to a supermarket and were given two trolleys
► In one, you have to place the fruits and vegetables; in the other, you must
put the grocery items like bread, oil, egg, etc.
► So basically, you are classifying the items of the supermarket into two
classes:
► fruits and vegetables
► grocery
► Classification usually refers to a problem where a specific type of class label
is the result to be predicted from the given input field of data
► For example, here we are working on a vegetable- grocery-classifier model
that predicts whether the item in the supermarket is a vegetable or a grocery
item
Try Yourself:
Which of this is a classification use case example?
House price prediction Credit card fraud detection
Salary prediction
Try Yourself:
Which of this is a classification use case example?
House price prediction Credit card fraud
detection
Salary prediction
Classification Metrics
► Popular metrics used for classification model
► Confusion matrix
► Classification accuracy
► Precision
► Recall
Confusion matrix
Let’s say, based on some clinical parameters;
you have designed a classifier that predicts
whether a person is infected with a certain
disease or not.
The output is 1 if the person is infected or 0 if
the person is not infected. That is, 1 and 0
signify whether a person is infected or not.
Confusion matrix
The confusion matrix is a handy presentation of
the accuracy of a model with two or more
classes
●The table presents the actual values on the
y-axis and predicted values on the x-axis
●The numbers in each cell represents the number
of predictions made by a machine learning
algorithm that falls into that particular category
For example, a machine learning algorithm can predict
0 or 1 and each prediction may actually have been a 0
or 1.
Activity :- Build
the confusion
matrix from
scratch
Fill the matrix based on the table given here
Count the number of rows having YES in both columns
of the table and put the count in the first cell. Similarly,
number of rows having YES in the first column and NO
in the second column will be shown in the top right cell
of confusion matrix. Number of rows having NO in the
first column and YES in the second column will be
shown in the down left cell of confusion matrix. Lastly,
number of rows having NO in the first column and YES
in the second column will be shown in the downright cell
of confusion matrix.
Activity Reflection
► So, there are 07 correct predictions out of 10 predictions.
► What do you think? How good is your model?
► Now that you know how to construct a Confusion matrix, let’s understand each cell of the
matrix in details.
True positive
True Positive (TP) is the
outcome of the model
correctly predicting
the positive class
•Any class can be assumed
as a positive class, and the
rest can be assumed as
negative
•Let’s say class 1 is assumed
as the positive
class
•Can you tell the TP value
from this matrix?
True positive example
Scenario 1:
►Consider you are watching football world cup.
Scenario 2:
►Consider the earlier example of medical diagnosis of an
infected disease.
True Positive examples
You had predicted that France would win the world cup, and it
won.
In the earlier activity, the cases in which we predicted yes (they
have the disease), and they do have the disease.
True Negative
► True Negative (TN) is the outcome of the model
correctly predicting the negative class.
► Since in the previous example, class 1 is assumed
the positive class, class 0 should be assumed the
negative class.
► Can you tell the TN value from this matrix?
True Negative examples
► You had predicted that Germany would not win, and it lost
► In the earlier activity, the cases in which we predicted No (they don’t have the disease),
and they don’t have the disease
False Positive
► False Positive (FP) is the outcome of the model wrongly
predicting the negative class as positive class.
► Here, when a class 0 is predicted as class 1, it falls into the FP
cell.
► Can you tell the FP value from this matrix?
False Positive examples
► You had predicted that Germany would win, but it lost.
► In the earlier activity, the cases in which we predicted Yes (they have the disease), and
they
► don’t have the disease.
False Negative
► False Negative (FN) is the outcome of the model
wrongly predicting the positive class as the negative
class.
► Here, when class 1 is predicted as class 0, it falls into the
FN cell.
► Can you tell the FN value from this matrix?
False Negative
► You had predicted that France would not win but it won
► In the earlier activity, the cases in which we predicted No (they don’t have the disease),
and they have the disease
Prediction
Confusion
Actual
Matrix
1. Accuracy from Confusion matrix
► Classification accuracy is the
number of correct predictions made
as a ratio of all predictions made.
In cases of unbalanced data, we should use other metrics such as Precision, Recall or F1
score.
2. Precision
► Precision is the ratio of the total number of correctly
classified positive examples and the total number of predicted
positive examples.
► Precision = 0.843 means that when our model predicts a
patient has heart disease, it is correct around 84% of the time.
Precision = Correct positive predictions
Total positive predictions
TP
= TP+FP
Precision: where should we use it?
The metrics Precision is generally used for unbalanced datasets when
dealing with the False Positives become important, and the model
needs to reduce the FPs as much as possible.
Precision use case example
For example, take the case of predicting a good day based on weather
conditions to launch satellite.
•Let’s assume a day with favorable weather condition is considered
Positive class and a day with non-favorable weather condition is
considered as Negative class.
•Missing out on predicting a good weather day is okay (low recall)
but predicting the bad weather day (Negative class) as a good weather
day (Positive class) to launch the satellite can be disastrous.
•So, in this case, the FPs need to be reduced as much as possible.
3. Recall from confusion matrix
► The recall is the measure of our model correctly
identifying True Positives
► Thus, for all the patients who actually have heart disease, Recall = Correct positive predictions
recall tells us how many we correctly identified as having Total actual positive values
a heart disease. Recall = 0.86 tells us that out of the total
TP
patients who have heart disease 86% have been correctly = TP+FN
identified.
► Recall is also called as Sensitivity or True Positive Rate
Recall: Where we should we use it?
► The metrics Recall is generally used for unbalanced dataset when
dealing with the False Negatives become important and the model
needs to reduce the FNs as much as possible.
Recall use case example
For example, for a covid-19 prediction classifier, let’s consider
detection of a covid-19 affected case as positive class and
detection of covid-19 non-affected case as negative class.
Imagine if a covid-19 affected person (Positive) is falsely predicted as
non-affected of Covid-19 (Negative), the person if rely solely on the AI
would not get any treatment and also may end up infecting many other
persons.
•So, in this case, the FNs needs to be reduced as much as possible.
•Hence, Precision is a go-to metrics for this kind of use case.
4. F1 Score
► F1-Score provides a way to combine both precisions and recall into a single measure that
captures both properties
► In those use cases, where the dataset is unbalanced, and we are unable to decide whether
FP is more important or FN, we should use the F1 score as the suitable metric.
F1 Score = 2 x Precision x Recall
Precision + Recall
Activity : Decide the appropriate metric to
evaluate the AI model
Fill the confusion matrix based on the table
given above
Calculate the recall from the confusion matrix based on.
Write the formula for recall:
Calculate recall from the formula: