1.1.
1 Bias:
To understand it, we must know its general meaning. Cambridge dictionary states as,
The action of supporting or opposing a particular person or thing in an unfair way,
because of allowing personal opinions to influence your judgment.
→ So in the world of stats, it is defined as,
Statistical bias is a feature of a statistical technique or of its results whereby the
expected value of the results differs from the true underlying quantitative parameter
being estimated.
Enough with the ‘Bookish’ definition, let us understand it by more relatable analogy with
the real world.
→ In simple English, “The inability of machine learning techniques to capture the true
relationship is Bias”.
• Low Bias: Predicted data points are close to the target. Also, the model
suggests less assumptions about the form of the target function.
• High-Bias: Predicted data points are far from the target. Also, the model
suggests more assumptions about the form of the target function.
• Examples of low-bias machine learning algorithms: Decision Trees,
k-Nearest Neighbors and Support Vector Machines.
• Examples of high-bias machine learning algorithms: Linear
Regression, Linear Discriminant Analysis and Logistic Regression.
→ So ideally we mostly aim for low bias overall (but not always). Model with high bias
pays very little attention to the training data and oversimplifies the model. It always
leads to a high error on training and test data.
Bias
Let’s assume we have trained the model and are trying to predict values with input
‘x_train’. The predicted values are y_predicted. Bias is the error rate of y_predicted and
y_train.
In simple terms,think of bias as the error rate of the training data.
When the error rate is high, we call it High Bias and when the error rate is low, we call it
Low Bias
1.1.2 Variance:
Again to understand it, we must know its general meaning. Cambridge dictionary states
as,
The fact that two or more things are different, or the amount or number by which they
are different.
→ So in the world of stats, it is defined as,
In probability theory and statistics, variance is the expectation of the squared
deviation of a random variable from its mean. Informally, it measures how far a set of
(random) numbers are spread out from their average value.
That’s again a ‘Bookish’ definition. So let us get the idea clear by seeing it’s real-world
analogy.
→ In plain English, “Its the variability of model prediction for a given data point or a
value which tells us the spread of our data”. The spread of data here is nothing but the
square of the difference between data points and the mean (which is Squared Deviation,
σ²)
• Low Variance: Data points are close to each as a result close to function.
Also, the model Suggests small changes to the estimate of the target function
with changes to the training dataset.
• High Variance: Data points are spreaded and as a result far from the
function. Suggests large changes to the estimate of the target function with
changes to the training dataset.
• Examples of low-variance machine learning algorithms: Linear
Regression, Linear Discriminant Analysis and Logistic Regression.
• Examples of high-variance machine learning algorithms: Decision
Trees, k-Nearest Neighbors and Support Vector Machines.
→ So ideally we mostly aim for low bias overall. (but not always). Model with high
variance pays a lot of attention to training data and does not generalize on the data which
it hasn’t seen before. As a result, such models perform very well on training data but has
high error rates on test data.
Variance
Let’s assume we have trained the model and this time we are trying to predict values with input
‘x_test’. Again, the predicted values are y_predicted. Variance is the error rate of the
y_predicted and y_test
In simple terms, think of variance as the error rate of the testing data.
When the error rate is high, we call it High Variance and when the error rate is low, we call it
Low Variance
Underfitting
When the model has a high error rate in the training data, we can say the model is underfitting.
This usually occurs when the number of training samples is too low. Since our model performs
badly on the training data, it consequently performs badly on the testing data as well.
A high error rate in training data implies a High Bias, therefore
In simple terms, High Bias implies underfitting
OverFitting
When the model has a low error rate in training data but a high error rate in testing data, we
can say the model is overfitting. This usually occurs when the number of training samples is too
high or the hyperparameters have been tuned to produce a low error rate on the training data.
E.g. Think of a student who studied a certain set of questions and then gave a mock exam which
contains those exact questions they studied. They might do well on the mock exam but on the
real exam, which contains unseen questions, they might not necessarily do well. If the student
gets a 95% in the mock exam but a 50% in the real exam, we can call it overfitting.
A low error rate in training data implies Low Bias whereas a high error rate in testing data
implies a High Variance, therefore
In simple terms, Low Bias and Hight Variance implies overfittting
Overfitting, Underfitting in Regression
Source: [Link]
[Link]/stable/auto_examples/model_selection/plot_underfitting_overfitting.html
In the first image, we try to fit the data using a linear equation. The model is rigid and not at all
flexible. Due to the low flexibility of a linear equation, it is not able to predict the samples
(training data), therefore the error rate is high and it has a High Bias which in turn means it’s
underfitting. This model won’t perform well on unseen data.
In the second image, we use an equation with degree 4. The model is flexible enough to predict
most of the samples correctly but rigid enough to avoid overfitting. In this case, our model will
be able to do well on the testing data therefore this is an ideal model.
In the third image, we use an equation with degree 15 to predict the samples. Although it’s able
to predict almost all the samples, it has too much flexibility and will not be able to perform well
on unseen data. As a result, it will have a high error rate in testing data. Since it has a low error
rate in training data (Low Bias) and high error rate in training data (High Variance), it’s
overfitting.
Overfitting, Underfitting in Classification
Assume we have three models ( Model A , Model B , Model C) with the following error rates on
training and testing data.
+---------------+---------+---------+---------+
| Error Rate | Model A | Model B | Model C |
+---------------+---------+---------+---------+
| Training Data | 30% | 6% | 1% |
+---------------+---------+---------+---------+
| Testing Data | 45% | 8% | 25% |
+---------------+---------+---------+---------+
For Model A, The error rate of training data is too high as a result of which the error rate of
Testing data is too high as well. It has a High Bias and a High Variance, therefore it’s underfit.
This model won’t perform well on unseen data.
For Model B, The error rate of training data is low and the error rate ofTesting data is low as
well. It has a Low Bias and a Low Variance, therefore it’s an ideal model. This model will
perform well on unseen data.
For Model C, The error rate of training data is too low. However, the error rate of Testing data
is too high as well. It has a Low Bias and a High Variance, therefore it’s overfit. This model won’t
perform well on unseen data.
Bias-Variance Tradeoff
Source: [Link]
When the model’s complexity is too low, i.e a simple model, the model won’t be able to
perform well on the training data nor the testing data, therefore it’s underfit
At the sweet spot, the model has a low error rate on the training data as well as the testing
data, therefore, that’s the ideal model
As the complexity of the model increases, the model performs well on the training data but it
doesn’t perform well on the testing data and therefore it’s overfit
Precision and Recall in Machine Learning
While building any machine learning model, the first thing that comes to our mind is
how we can build an accurate & 'good fit' model and what the challenges are that will
come during the entire procedure. Precision and Recall are the two most important but
confusing concepts in Machine Learning.
Precision and recall are performance metrics used for pattern recognition and
classification in machine learning.
These concepts are essential to build a perfect machine learning model which gives
more precise and accurate results. Some of the models in machine learning require
more precision and some model requires more recall. So, it is important to know the
balance between Precision and recall or, simply, precision-recall trade-off.
In this article, we will understand Precision and recall, the most confusing but important
concepts in machine learning that lots of professionals face during their entire data
science & machine learning career. But before starting, first, we need to understand
the confusion matrix concept. So, let's start with the quick introduction of Confusion
Matrix in Machine Learning.
Confusion Matrix in Machine Learning
Confusion Matrix helps us to display the performance of a model or how a model
has made its prediction in Machine Learning.
Confusion Matrix helps us to visualize the point where our model gets confused in
discriminating two classes. It can be understood well through a 2×2 matrix where the
row represents the actual truth labels, and the column represents the predicted
labels.
This matrix consists of 4 main elements that show different metrics to count a number
of correct and incorrect predictions. Each element has two words either as follows:
o True or False
o Positive or Negative
If the predicted and truth labels match, then the prediction is said to be correct, but
when the predicted and truth labels are mismatched, then the prediction is said to be
incorrect. Further, positive and negative represents the predicted labels in the matrix.
There are four metrics combinations in the confusion matrix, which are as follows:
o True Positive: This combination tells us how many times a model correctly
classifies a positive sample as Positive?
o False Negative: This combination tells us how many times a model incorrectly
classifies a positive sample as Negative?
o False Positive: This combination tells us how many times a model incorrectly
classifies a negative sample as Positive?
o True Negative: This combination tells us how many times a model correctly
classifies a negative sample as Negative?
Hence, we can calculate the total of 7 predictions in binary classification problems using
a confusion matrix.
Now we can understand the concepts of Precision and Recall.
What is Precision?
Precision is defined as the ratio of correctly classified positive samples (True Positive) to a
total number of classified positive samples (either correctly or incorrectly).
1. Precision = True Positive/True Positive + False Positive
2. Precision = TP/TP+FP
o TP- True Positive
o FP- False Positive
o The precision of a machine learning model will be low when the value of;
1. TP+FP (denominator) > TP (Numerator)
o The precision of the machine learning model will be high when Value of;
1. TP (Numerator) > TP+FP (denominator)
Hence, precision helps us to visualize the reliability of the machine learning model
in classifying the model as positive.
Difference between Precision and Recall in Machine Learning
Precision Recall
It helps us to measure the ability to classify It helps us to measure how many positive
positive samples in the model. samples were correctly classified by the ML
model.
While calculating the Precision of a model, While calculating the Recall of a model, we only
we should consider both Positive as well as need all positive samples while all negative
Negative samples that are classified. samples will be neglected.
When a model classifies most of the positive When a model classifies a sample as Positive,
samples correctly as well as many false- but it can only classify a few positive samples,
positive samples, then the model is said to be then the model is said to be high accuracy, high
a high recall and low precision model. precision, and low recall model.
The precision of a machine learning model is Recall of a machine learning model is
dependent on both the negative and positive dependent on positive samples and
samples. independent of negative samples.
In Precision, we should consider all positive The recall cares about correctly classifying all
samples that are classified as positive either positive samples. It does not consider if any
correctly or incorrectly. negative sample is classified as positive.
Why use Precision and Recall in Machine Learning models?
This question is very common among all machine learning engineers and data
researchers. The use of Precision and Recall varies according to the type of problem
being solved.
o If there is a requirement of classifying all positive as well as Negative samples as
Positive, whether they are classified correctly or incorrectly, then use Precision.
o Further, on the other end, if our goal is to detect only all positive samples, then
use Recall. Here, we should not care how negative samples are correctly or
incorrectly classified the samples.
Conclusion:
In this tutorial, we have discussed various performance metrics such as confusion matrix,
Precision, and Recall for binary classification problems of a machine learning model.
Also, we have seen various examples to calculate Precision and Recall of a machine
learning model and when we should use precision, and when to use Recall.