Machine Learning
Dr. Irfan Yousuf
Department of Computer Science (New Campus)
UET, Lahore
(Week 5; March 09 - 13, 2026)
Outline
• Bias Variance Tradeoff
• Overfitting vs. Underfitting
Bias and Variance Trade-off
• In general, a machine learning (or deep learning) model
analyses the data, find patterns in it and make predictions.
• While training, the model learns these patterns in the
dataset and applies them to test data for prediction.
• While making predictions, a difference occurs between
prediction values made by the model and actual
values/expected values, and this difference is known as
bias errors or Errors due to bias.
Bias
• It can be defined as an inability of an algorithm to capture
the true relationship between the data points.
• Each algorithm begins with some amount of bias because
bias occurs from assumptions in the model, which makes
the target function simple to learn.
• Low Bias: A low bias model will make fewer assumptions
about the form of the target function.
• High Bias: A model with a high bias makes more
assumptions, and the model becomes unable to capture the
important features of our dataset. A high bias model also
cannot perform well on new data.
Bias
Variance
• The variance would specify the amount of variation in the
prediction if different training data was used.
• In simple words, variance tells that how much a random
variable is different from its expected value.
• Ideally, a model should not vary too much from one
training dataset to another, which means the algorithm
should be good in understanding the hidden mapping
between inputs and output variables.
• Variance errors are either of low variance or high variance.
Variance
• Low variance means there is a small variation in the
prediction of the target function with changes in the
training data set.
• High variance shows a large variation in the prediction of
the target function with changes in the training dataset.
Bias and Variance Trade-off
• Let us consider the problem of fitting a curve through a
given set of points.
• The points are drawn from a sinusoidal function (the true
f(x) )
Bias and Variance Trade-off
Bias and Variance Trade-off
Bias and Variance Trade-off
Bias and Variance Trade-off
Bias and Variance Trade-off
Bias and Variance Trade-off
The expected value
is the mean of the
possible values a
random variable
can take, weighted
by the probability
of those outcomes.
Expected value =
Weighted Average
Bias and Variance Trade-off
Bias and Variance Trade-off
Bias and Variance Trade-off
Bias and Variance Trade-off
Overfitting
• When a model performs very well for training data but has
poor performance with test data (new data), it is known as
overfitting.
• In this case, the learning model learns the details and noise
in the training data such that it negatively affects the
performance of the model on test data.
• Overfitting can happen due to low bias and high variance.
Reasons for Overfitting
• Data used for training is not cleaned and contains noise
(garbage values) in it.
• The model has a high variance.
• The size of the training dataset used is not enough.
• The model is too complex.
Underfitting
• When a model has not learned the patterns in the training
data well and is unable to generalize well on the new data,
it is known as underfitting.
• An underfit model has poor performance on the training
data and will result in unreliable predictions.
• Underfitting occurs due to high bias and low variance.
Reasons for Underfitting
• Data used for training is not cleaned and contains noise
(garbage values) in it.
• The model has a high bias.
• The size of the training dataset used is not enough.
• The model is too simple
Methods to detect overfitting and underfitting
• Split the data: One of the simplest and most effective
methods to detect overfitting and underfitting is to split the
data into three sets: training, validation, and test. The
training set is used to fit the model, the validation set is
used to tune the model parameters and select the best
model, and the test set is used to evaluate the model
performance on new data.
• If the model has a high accuracy on the training set but a
low accuracy on the validation or test set, it is likely
overfitting. If the model has a low accuracy on both the
training and validation or test set, it is likely underfitting.
Methods to detect overfitting and underfitting
Methods to detect overfitting and underfitting
• Use cross-validation: Another method to detect overfitting
and underfitting is to use cross-validation, which is a
technique that splits the data into multiple folds and uses
each fold as a validation set while training the model on
the rest of the folds.
• By averaging the cross-validation scores, you can assess
how well the model generalizes to new data. A low cross-
validation score indicates underfitting, while a high
variance between the cross-validation scores indicates
overfitting.
Methods to detect overfitting and underfitting
Methods to detect overfitting and underfitting
• Plot learning curves: A third method to detect overfitting
and underfitting is to plot learning curves, which are
graphs that show how the model performance changes as
a function of the training set size or the number of
iterations.
• A typical learning curve has two lines: one for the training
score and one for the validation score. If the two lines are
close and low, it means the model is underfitting. If the
two lines are far apart and high, it means the model is
overfitting. If the two lines converge and plateau, it means
the model is optimal.
Methods to detect overfitting and underfitting
• Plot learning curves:
Methods to detect overfitting and underfitting
• Plot learning curves:
Methods to detect overfitting and underfitting
• Choose the right model: A fourth method to prevent
overfitting and underfitting is to choose the right model
for the data and the problem.
• Different models have different assumptions, capabilities,
and limitations that affect how they fit and generalize to
the data.
• For example, linear models are simple and interpretable,
but they may not capture nonlinear or complex patterns.
On the other hand, neural networks are powerful and
flexible, but they may overfit or require more data and
computation.
Methods to detect overfitting and underfitting
• Apply regularization: A fifth method to prevent overfitting
and underfitting is to apply regularization, which is a
technique that adds a penalty term to the model loss
function to reduce the complexity or flexibility of the
model.
• Regularization can help avoid overfitting by preventing the
model from learning noise or irrelevant features from the
data. It can also help avoid underfitting by encouraging the
model to learn more from the data.
Regularization in Deep Learning
• Regularization is a set of methods for reducing overfitting
in machine / deep learning models. Typically,
regularization trades a marginal decrease in training
accuracy for an increase in generalizability.
• Regularization techniques help improve a neural network’s
generalization ability by reducing overfitting. They do this
by minimizing needless complexity and exposing the
network to more diverse data.
• Regularization methods typically lead to less
accurate predictions on training data but more
accurate predictions on test data.
Regularization in Deep Learning
• Early Stopping
• Dropout
• Batch Normalization
• Noice Injection
• Data Augmentation
• L1 Regularization
• L2 Regularization
Summary
• Bias Variance Tradeoff