Module: 6
Model Evaluation
• The bias error is an error from erroneous assumptions in
the learning algorithm. High bias can cause an algorithm to
miss the relevant relations between features and target
outputs (underfitting).
• The variance is an error from sensitivity to small
fluctuations in the training set. High variance may result
from an algorithm modeling the random noise in the
training data (overfitting).
• In machine Learning, Classification is the process of categorizing a given
set of data into different categories. In Machine Learning, To measure
the performance of the classification model we use the confusion
matrix.
Confusion Matrix
• A confusion matrix is a matrix that summarizes the performance of a
machine learning model on a set of test data. It is often used to measure
the performance of classification models, which aim to predict a
categorical label for each input instance. The matrix displays the number
of true positives (TP), true negatives (TN), false positives (FP), and false
negatives (FN) produced by the model on the test data.
• For binary classification, the matrix will be of a 2X2 table, For multi-class
classification, the matrix shape will be equal to the number of classes i.e
for n classes it will be nXn.
• A 2X2 Confusion matrix is shown below for the image recognization
having a Dog image or Not Dog image.
•True Positive (TP): It is the total counts having both predicted and actual
values are Dog.
•True Negative (TN): It is the total counts having both predicted and
actual values are Not Dog.
•False Positive (FP): It is the total counts having prediction is Dog while
actually Not Dog.
•False Negative (FN): It is the total counts having prediction is Not Dog
while actually, it is Dog.
•Actual Dog Counts = 6
•Actual Not Dog Counts = 4
•True Positive Counts = 5
•False Positive Counts = 1
•True Negative Counts = 3
•False Negative Counts = 1
• ROC Curve
ROC stands for Receiver Operating Characteristics, and the ROC curve is the
graphical representation of the effectiveness of the binary classification
model. It plots the true positive rate (TPR) vs the false positive rate (FPR) at
different classification thresholds.
• AUC Curve:
AUC stands for Area Under the Curve, and the AUC curve represents the area
under the ROC curve. It measures the overall performance of the binary
classification model. As both TPR and FPR range between 0 to 1, So, the area
will always lie between 0 and 1, and A greater value of AUC denotes better
model performance. Our main goal is to maximize this area in order to have
the highest TPR and lowest FPR at the given threshold. The AUC measures the
probability that the model will assign a randomly chosen positive instance a
higher predicted probability compared to a randomly chosen negative instance.
It represents the probability with with our model is able to distinguish
between the two classes which are present in our target.
• Regression models are used to quantify the relationship
between one or more predictor variables and s response
variable.
• Whenever we fit a regression model, we want to understand
how well the model is able to use the values of the predictor
variables to predict the value of the response variable.
• Two metrics we often use to quantify how well a model fits
a dataset are the mean squared error (MSE) and the root
mean squared error (RMSE)
RMSE: A metric that tells us the square root of the average
squared difference between the predicted values and the actual
values in a dataset. The lower the RMSE, the better a model
fits a dataset.
• The root mean squared error is 4. This tells us that the
average deviation between the predicted points scored and
the actual points scored is 4.
• Notice that the interpretation of the root mean squared error
is much more straightforward than the mean squared error
because we’re talking about ‘points scored’ as opposed to
‘squared points scored.’