Understanding Underfitting

& Overfitting
What is the Goal of Training a Model?
• The goal of any ML/DL model is to learn patterns from training data
so it can predict accurately on new (unseen) data.
• To check this, we look at:
Training Error: How well the model performs on the training dataset.
Testing Error: How well the model performs on unseen data
(generalization).
Bias and Variance
• Bias: is the error that happens when a NN model is too simple and doesn't
learn enough details from the data. It's like assuming all birds can only be
small and fly, so the model fails to recognize big birds like ostriches or
penguins that can't fly and get biased with predictions.
• These assumptions make the model easier to train but may prevent it from
capturing the underlying complexities of the data.
• High bias typically leads to underfitting, where the model performs poorly
on both training and testing data because it fails to learn enough from the
data.
• Example: A linear regression model applied to a dataset with a non-linear
relationship.
Bias and Variance
• Variance: Error that happens when a NN model learns too much from
the data, including random noise.
• A high-variance model learns not only the patterns but also the noise
in the training data, which leads to poor generalization on unseen
data.
• High variance typically leads to overfitting, where the model
performs well on training data but poorly on testing data.
What is Underfitting?
• Underfitting occurs when a model is too simple to capture the underlying
patterns in the data. It performs poorly on both training and testing sets.
• Analogy: A student is preparing for an exam but doesn’t study enough or
uses only very basic materials.
•The student is like a model
• Their study material is like the training data.
•Their exam performance is like testing the model on unseen data.
• For ex: Predicting house prices using only the number of rooms, ignoring
location, area, etc. The model is too simple to be accurate.
• It fails to capture the real relationship between features and price.
• Underfitting models are like students who don’t study enough. They don’t
do well in practice tests or real exams. Note: The underfitting model has
High bias and low variance.
 Reasons for Underfitting
• The model is too simple, So it may be not capable to represent the
complexities in the data.
• The input features which is used to train the model is not the
adequate representations of underlying factors influencing the target
variable.
• The size of the training dataset used is not enough.
• Excessive regularization are used to prevent the overfitting, which
constraint the model to capture the data well.
• Features are not scaled.
Techniques to Reduce Underfitting
• Increase model complexity.
• Increase the number of features, performing feature engineering.
• Remove noise from the data.
• Increase the number of epochs or increase the duration of training to
get better results.
What is Overfitting?
• Overfitting happens when a model learns not just the pattern but also
the noise in the training data. It performs well on training data but
poorly on unseen data.
• Real-World Example:
Memorizing answers for an exam rather than understanding
concepts. You ace the practice tests but fail the actual exam with new
questions.
What is Overfitting?
• For example, you build a very complex model to predict the house
prizes
• The model learns not only the general pattern (bigger house → higher
price) But, also the random noise and outliers in the data
• Training accuracy is very high
• But, when you test it on new data, it performs very poorly because it
has memorized instead of learning the general trend.
Characteristics of Overfitting:
•Model fits too closely to training data
•Learns noise and irrelevant patterns
•Performs poorly on test/new data
Reasons for Overfitting

• High variance and low bias.

• The model is too complex.
• The size of the training data.
Techniques to Reduce Overfitting
• Improving the quality of training data reduces overfitting by focusing on
meaningful patterns, mitigate the risk of fitting the noise or irrelevant
features.
• Increase the number of training data can improve the model's ability to
generalize to unseen data and reduce the likelihood of overfitting.
• Reduce model complexity.
• Early stopping during the training phase (have an eye over the loss over the
training period as soon as loss begins to increase stop training).
• Ridge Regularization and Lasso Regularization.
• Use dropout for neural networks to tackle overfitting.
Underfitting = Not learning enough
Overfitting = Learning too much and memorizing
Visual Representation of underfitting and overfitting
for house prize prediction problem

•Underfitting : Straight line trying to fit a curved dataset but cannot capture the data's patterns,
leading to poor performance on both training and test sets.

•Overfitting: A squiggly curve passing through all training points, failing to generalize performing well
on training data but poorly on test data.

•Appropriate Fitting: Curve that follows the data trend without overcomplicating to capture the true
patterns in the data.
bias-variance trade-off
• Increasing model complexity reduces bias but increases variance (risk
of overfitting).
• Simplifying the model reduces variance but increases bias (risk of
underfitting).
• The goal is to find an optimal balance where both bias and variance
are minimized, resulting in good generalization performance.
bias-variance trade-off
Why Do We Need Regularization?
• In training, a model can memorize the data instead of learning
general patterns.
• Memorization happens when weights get too large, making the
model overly sensitive.
• Example: Predicting marks from study hours.
• If , one extra hour changes the prediction by +50 marks. That’s unstable!
Regularization
Dropout
• Dropout is a regularization technique that prevents overfitting by
randomly turning off (dropping) a fraction of neurons during
training.
• Each forward pass → some neurons are randomly ignored.
• This forces the network to not depend on specific neurons and learn
more robust patterns.
• Only applied during training, not during inference/testing
Dropout
Dropout
• The term "dropout" refers to dropping out the nodes (input and hidden
layer) in a neural network.
• All the forward and backwards connections with a dropped node are
temporarily removed, thus creating a new network architecture out of the
parent network.
• The nodes are dropped by a dropout probability of p.
• For ex. x: {1, 2, 3, 4, 5} is input to the fully connected layer.
• We have a dropout layer with probability p = 0.2 (or keep probability = 0.8).
• During the forward propagation (training) from the input x, 20% of the
nodes would be dropped, i.e. the x could become {1, 0, 3, 4, 5} or {1, 2, 0,
4, 5} and so on.
 Dropout
• Similarly, if the hidden layers have 1000 neurons (nodes) and a
dropout is applied with drop probability = 0.5, then 500 neurons
would be randomly dropped in every iteration (batch).
• Generally, for the input layers, the keep probability i.e. (1- drop
probability), is closer to 1. 0.8 being the best as suggested by the
authors.
• For the hidden layers, the greater the drop probability more sparse
the model, where 0.5 is the most optimized keep probability, that
states dropping 50% of the nodes.
How does dropout solve the overfitting problem?
• In overfitting, the model learns statistical noise present in the data.
• The main motive of training is to decrease the loss function, given all the
neurons. So ,in overfitting, a unit may change in a way that fixes up the
mistakes of the other units.
• This leads to complex co-adaptations, which in turn leads to the overfitting
problem because this complex co-adaptation fails to generalize on the
unseen dataset.
• Now, if we use dropout, it prevents these units to fix up the mistake of
other units, thus preventing co-adaptation, as in every iteration the
presence of a unit is highly unreliable.
• So ,by randomly dropping a few units (nodes), it forces the layers to take
more or less responsibility for the input by taking a probabilistic approach.
 Dropout
• Mathematically ,Dropout means we multiply neuron outputs by a
random mask.
Dropout
Dropout
Dropout
Dropout
 Why scaling is Needed?
• In dropout, during training, we randomly “turn off” (set to zero) some
neurons in each forward pass.
• Example: dropout rate = 0.3 → 30% of neurons are dropped each time.
• The idea: prevent neurons from becoming too dependent on each other →
reduces overfitting.
• During training, fewer neurons are active (because of dropout).
• During testing, all neurons are active.
• If we don’t adjust the outputs, the neuron activations during testing will be
larger than during training, and the model’s behavior will be inconsistent.
 How scaling fixes mismatch?
Mismatch is fixed in two ways (frameworks usually do one of these automatically):

a) Inverted Dropout (most common)

•During training, we scale up the active neurons’ outputs by 1 / (1 - dropout_rate).
•Example: dropout rate = 0.3 → scale factor = 1 / (1 - 0.3) = 1 / 0.7 ≈ 1.43.
•This way, during testing we don’t need any scaling.

b) Classical Dropout
•During training, we don’t scale.
•During testing, we scale down the outputs by multiplying with (1 -dropout_rate).

•In short, Scaling in dropout ensures the average output magnitude stays the same
between training and testing, so the model doesn’t get confused by different activation
levels.
Early stopping
• Early stopping is a form of regularization used to prevent overfitting in
machine learning and deep learning models.
• It involves stopping the training process before the model starts to
overfit.
• The idea is to monitor the model’s performance on a validation set
during the training process and stop training when the performance
starts to degrade, which is an indication that the model is beginning
to overfit the training data.
Early stopping
How Early Stopping Works?
• Validation Set: During training, a portion of the training data is set aside as
a validation set. This set is used to evaluate the model’s performance at
each epoch (iteration over the entire dataset).
• Performance Monitoring: The model’s performance on the validation set is
monitored at each epoch. Common metrics used for this purpose include
accuracy, loss, or any other metric relevant to the problem at hand.
• Early Stopping Criterion: If the performance on the validation set starts to
degrade (e.g., the loss increases or the accuracy decreases), it’s an
indication that the model is beginning to overfit the training data. At this
point, early stopping is triggered, and the training process is halted.
• Model Selection: Since the training is stopped before overfitting occurs,
the model at the point of early stopping is typically the best model, as it
has not yet learned the noise in the training data
Batch Normalization
• Batch Normalization makes training deep neural networks faster,
more stable, and less sensitive to initialization by keeping the inputs
to each layer at a consistent scale and distribution during training.
• Why It’s Needed ?
When training deep networks,
• As the data moves through layers, the distribution of inputs to each
layer changes because the parameters of the previous layers are
updating.
• This is called Internal Covariate Shift
What is Internal Covariate Shift
What is Internal Covariate Shift
What is Internal Covariate Shift
• Why is it called “distribution shift”?
• If you plotted the histogram of activations coming into layer L at
different training steps, you’d notice:
• The mean might drift (shift left/right)
• The variance might grow or shrink
• The shape might change (become skewed, heavy-tailed, etc.)
• This is the “internal covariate shift” — the internal input distribution
to each layer is not stationary; it keeps changing because upstream
layers keep updating.
Why Internal Covariate Shift is a problem?
•Slower convergence – Every time the input distribution changes, the next layer
must relearn how to map these new inputs.
•Instability – Large shifts can cause exploding/vanishing gradients, making
training erratic.
•Optimization difficulty – It’s harder for gradient descent to settle into a good set
of weights when the “target” mapping is constantly moving.

Imagine teaching someone to hit a target in archery — but every few minutes,
the target changes size and location.
They’d have to keep adjusting their aim. That’s what a neural network layer faces
when its input distribution keeps shifting during training.
What Batch Normalization Does to solve internal
covariate problem?
For each mini-batch during training:
• Calculate the mean of the activations (for each feature).
• Calculate the variance of the activations.
• Normalize:

This makes the activations have mean 0 and variance 1 for that batch.
Batch Normalization
Scale and Shift:

γ (scale) and β (shift) are learnable parameters that let the network undo
normalization if needed.
Why Scale & Shift are Important?
• Without γ and β, every layer would be forced to have zero mean and
variance 1, which could limit representation power.
• γ and β allow the network to choose the best distribution for each
layer while still getting the benefits of normalization.
Key Benefits of Batch Normalization
• Faster convergence (can use higher learning rates).
• More stable training (reduces exploding/vanishing gradients).
• Acts like regularization (slight noise from batch stats).
• Reduces sensitivity to initialization.
 What is data augmentation?
• Data augmentation involves creating new training samples from
existing ones by applying transformations like rotations, flips, scaling,
cropping, and more.
• For example, in image recognition, you might rotate an image, flip it
horizontally, or zoom in on a specific part to generate new training
examples.
• These transformations are designed to simulate real-world variations
that the model might encounter, making it more robust and less
prone to overfitting.
Why is data augmentation used in DNNs?
• Increased training data:
• DNNs often require large amounts of data to train effectively. Data augmentation helps
overcome data scarcity by creating a larger, more diverse training set.
• Improved generalization:
• By exposing the model to various transformations, data augmentation helps it learn
features that are more generalizable to unseen data.
• Reduced overfitting:
• Overfitting occurs when a model learns the training data too well, including its noise and
specific characteristics, leading to poor performance on new data. Data augmentation helps
prevent overfitting by introducing variations that prevent the model from memorizing the
training set.
• Addressing class imbalance:
• In some datasets, certain classes may be under-represented. Data augmentation can be
used to generate more samples for these under-represented classes, helping to balance the
dataset.
Common data augmentation techniques:
• Geometric transformations: These include rotations, flips, shifts,
scaling, and cropping.
• Color space augmentations: Adjusting brightness, contrast,
saturation, and hue.
• Noise injection: Adding random noise to the data.
• Mixup: Creating new samples by linearly combining two or more
existing samples.
Benefits of data augmentation:
• Improved model accuracy and performance.
• Reduced overfitting and improved generalization.
• More robust models that can handle variations in input data.
• Cost-effective way to increase training data without acquiring new
data.