Confusion Matrix, Precision, Re-
call, Accuracy, ROC curve
Advanced Pattern Recognition - Week 1
2018. 3. 13.
Team E
Nuriev Sirojiddin, Tuong Le, Mobeen
Ahmad, Seung Ju Lee, Won Hee Chung
1
CONTENTS
BINARY CLASSIFIER
CONFUSION MATRIX
ACCURACY
PRECISION
RECALL
ROC CURVE
- HOW TO PLOT ROC CURVE
2
Binary Classifier
A binary classifier produces output with two class values or labels,
such as Yes/No, 1/0, Positive/Negative for given input data
A dataset used for performance evaluation is called a test dataset
Observed labels are used to compare with the predicted labels
for performance evaluation after classification
The predicted labels will be exactly the same
if the performance of a classifier is perfect
But it is uncommon to be able to develop a perfect classifier
3
Confusion Matrix
A confusion matrix is formed from the four outcomes
produced as a result of binary classification
True positive (TP): correct positive prediction
False positive (FP): incorrect positive prediction
True negative (TN): correct negative prediction
False negative (FN): incorrect negative prediction
4
Confusion Matrix
Classifier
Levels : Green / Grey
Confusion Matrix
5
Confusion Matrix
True Positives False Positives
Green examples Gray examples
correctly identified Confusion Matrix falsely identified as
as green green
False Negatives True Negatives
Green examples Gray examples cor-
falsely identified as rectly identified as
gray grey
6
Accuracy
Accuracy is calculated as the number of all correct predictions
divided by the total number of the dataset
The best ACC is 1.0, whereas the worst is 0.0
Accuracy = (9+8) / (9+2+1+8) = 0.85
7
Precision
Precision is calculated as the number of correct positive predictions
divided by the total number of positive predictions
The best precision is 1.0, whereas the worst is 0.0
Precision = 9 / (9+2) = 0.81
8
Recall
Sensitivity = Recall = True Positive Rate
Recall is calculated as the number of correct positive predictions
divided by the total number of positives
The best recall is 1.0, whereas the worst is 0.0
Recall = 9 / (9+1) = 0.9
9
Example 1
Example: The example to classify whether images contain either a dog
or a cat
The training data contains 25000 images of dogs and cats;
The training data 75% of 25000 images; (25000*0.75 = 18750)
Validation data 25% of training data; (25000*0.25 = 6250)
Test Data, 5 cats, 5 dogs
Precision = 2/(2 + 0) * 100% = 100%
Recall = 2/(2 + 3) * 100% = 40%
Accuracy = (2 + 5)/(2 + 0 + 3 + 5) * 100% = 70%
10
Example 2
= = = 170/300 = .556
= = .5
= = .5
= = .667
= 0.556
= = .3
= = .6
= = 0.8
P = 0.556
11
ROC Curve
The Receiver Operating Characteristics(ROC) curve
The ROC curve is a evaluation measure
that is based on two basic evaluation measures
- specificity and sensitivity
Specificity = True Negative Rate
Sensitivity = Recall = True Positive Rate
Specificity
Specificity is calculated as the number of correct negative predictions
divided by the total number of negatives
12
How to Plot ROC Curve
Dynamic cut-off thresh-
olds
Cut-off = 0.020 Cut-off = 0.015 Cut-off = 0.010
13
How to Plot ROC Curve
True positive rate (TPR) = 𝑇𝑃/(𝑇𝑃+𝐹𝑁)
and False positive rate (FPR) = 𝐹𝑃/(𝐹𝑃+𝑇𝑁)
Use different cut-off thresholds (0.00, 0.01, 0.02,…, 1.00),
calculate the TPR and FPR, and plot them into graph.
That is receiver operating characteristic (ROC) curve.
Example
TPR = 0.5 TPR = 1 TPR = 1
FPR = 0 FPR = 0.167 FPR = 0.667
14
How to Plot ROC Curve
A ROC curve is created by connecting ROC points of a classifier
A ROC point is a point with a pair of x and y values
where x is 1-specificity and y is sensitivity
The curve starts at (0.0, 0.0) and ends at (1.0, 1.0)
15
ROC Curve
A classifier with the random performance level always shows a straight
line
Two areas separated by this ROC curve
ROC curves in the area with the top left corner indicate good performance
levels
ROC curves in the other area with the bottom right corner indicate poor per-
formance levels
16
ROC Curve
A classifier with the perfect performance level shows
a combination of two straight lines
It is important to notice that classifiers with meaningful performance
levels
usually lie in the area between the random ROC curve and the perfect
ROC curve
17
ROC Curve
AUC(Area under the ROC curve) score
An advantage of using ROC curve is a single measure called AUC score
As the name indicates, it is an area under the curve calculated in the ROC space
Although the theoretical range of AUC score is between 0 and 1,
the actual scores of meaningful classifiers are greater than 0.5,
which is the AUC score of a random classifier
ROC curves clearly shows classifiers A
outperforms classifier B
18
References
[Link]
[Link]
acteristics-plot/
Evaluation Metrics (Classifiers) / CS229 Section / Anand Avati
19
Thank you.
20