Classification model evaluation methods:
Confusion matrix: A confusion matrix is a matrix that summarizes the performance of a
machine learning model on a set of test data.. It is often used to measure the performance of
classification models, which aim to predict a categorical label for each input instance.
The matrix displays the number of instances produced by the model on the test data.
True positives (TP): occur when the model accurately predicts a positive data point.
True negatives (TN): occur when the model accurately predicts a negative data point.
False positives (FP): occur when the model predicts a positive data point incorrectly.
False negatives (FN): occur when the model mispredicts a negative data point.
Metrics based on Confusion Matrix Data
1. Accuracy: is used to measure the performance of the model. It is the ratio of Total correct
instances to the total instances.
2. Precision: is a measure of how accurate a model’s positive predictions are. It is defined as
the ratio of true positive predictions to the total number of positive predictions made by the
model.
3. Recall (Sensitivity-True positive rate): measures the effectiveness of a classification model
in identifying all relevant instances from a dataset. It is the ratio of the number of true positive
(TP) instances to the sum of true positive and false negative (FN) instances.
4. F1-Score: is used to evaluate the overall performance of a classification model. It is the
harmonic mean of precision and recall,
5. Specificity (True negative rate): It measures the ability of a model to correctly identify
negative instances. Specificity is also known as the True Negative Rate.
6. Type 1 and Type 2 error:
Type 1 error
Type 1 error occurs when the model predicts a positive instance, but it is actually negative.
Precision is affected by false positives, as it is the ratio of true positives to the sum of true
positives and false positives.
For example, in a courtroom scenario, a Type 1 Error, often referred to as a false positive,
occurs when the court mistakenly convicts an individual as guilty when, in truth, they are
innocent of the alleged crime. This grave error can have profound consequences, leading to the
wrongful punishment of an innocent person who did not commit the offense in question.
Preventing Type 1 Errors in legal proceedings is paramount to ensuring that justice is
accurately served and innocent individuals are protected from unwarranted harm and
punishment.
Type 2 error
Type 2 error occurs when the model fails to predict a positive instance. Recall is directly
affected by false negatives, as it is the ratio of true positives to the sum of true positives and
false negatives.
In the context of medical testing, a Type 2 Error, often known as a false negative, occurs when
a diagnostic test fails to detect the presence of a disease in a patient who genuinely has it. The
consequences of such an error are significant, as it may result in a delayed diagnosis and
subsequent treatment.
Positive predictive value is the probability that an observation with a positive predicted
outcome actually has a positive outcome.
Positive predictive value = TP / (TP + FP)
Negative predictive value: In machine learning, the negative predictive value is defined as the
proportion of predicted negatives which are real negatives. It reflects the probability that a
predicted negative is a true negative.
Positive predictive value = TN / (TN + FN)
ROC Curve:
The Receiver Operator Characteristic (ROC) curve is an evaluation metric for binary
classification problems. It is a probability curve that plots the TPR against FPR at various
threshold values and essentially separates the ‘signal’ from the ‘noise.’ In other words, it
shows the performance of a classification model at all classification thresholds. The Area
Under the Curve (AUC) is the measure of the ability of a binary classifier to distinguish
between classes and is used as a summary of the ROC curve.
The higher the AUC, the better the model’s performance at distinguishing between the
positive and negative classes.
When AUC = 1, the classifier can correctly distinguish between all the Positive and the
Negative class points. If, however, the AUC had been 0, then the classifier would predict all
Negatives as Positives and all Positives as Negatives.
When 0.5<AUC<1, there is a high chance that the classifier will be able to distinguish the
positive class values from the negative ones. This is so because the classifier is able to
detect more numbers of True positives and True negatives than False negatives and False
positives.
When AUC=0.5, then the classifier is not able to distinguish between Positive and Negative
class points. Meaning that the classifier either predicts a random class or a constant class for
all the data points.