1.
Regularization in Machine Learning
When training a machine learning model, the model can be easily overfitted or under fitted.
To avoid this, we use regularization in machine learning to properly fit the model to our test
set.
Definition
Regularization is a technique used in machine learning to prevent overfitting, which
otherwise causes models to perform poorly on unseen data. By adding a penalty for
complexity, regularization encourages simpler and more generalizable models.
Prevents overfitting: Adds constraints to the model to reduce the risk of memorizing
noise in the training data.
Improves generalization: Encourages simpler models that perform better on new,
unseen data.
It works by adding a penalty term to the loss function—commonly
L1 (Lasso) - adds the absolute values of the model weights to the loss function
L2 (Ridge) - which adds the squared values of the weights.
By incorporating these penalty terms, regularization strikes a balance between fitting the
training data and keeping the model simple, ultimately leading to better performance on new
data.
Overfitting
Overfitting happens when the model learns too much from the training data, including noise
and outliers. It performs very well on training data but poorly on test data.
Overfitting happens due to:
Model too complex
Too many features
Very little data
No regularization
High variance
Underfitting
Underfitting happens when the model fails to learn important patterns. It performs poorly on
both training and testing data.
Underfitting happens due to:
Model is too simple
Very high regularization
Features are weak or missing
Not enough training
High bias
Bias and Variance
Bias is the error that occurs when a model is too simple to capture the true patterns in the
data.
High bias: The model oversimplifies, misses patterns and underfits the data.
Low bias: The model captures patterns well and is closer to the true values.
Overfitting = Low Bias + High Variance
Underfitting = High Bias + Low Variance
Bias Variance Tradeoff
The relationship between bias and variance is referred to as the bias-variance trade-off
• High bias, low variance: models - consistent but inaccurate on average
• Low bias, high variance: models - accurate on average but inconsistent across
different datasets.
• Low bias, low variance: models - accurate and consistent on training and new data,
indicating a good balance between model complexity and performance on unseen
data.
• High bias, high variance: models - inaccurate and inconsistent, performing poorly in
training and on new data.
Types of Regularization
There are mainly 3 types of regularization techniques, each applying penalties in different
ways to control model complexity and improve generalization.
1. Lasso Regression
A regression model which uses the L1 Regularization technique is called LASSO (Least
Absolute Shrinkage and Selection Operator) regression.
It adds the absolute value of magnitude of the coefficient as a penalty term to the loss
function(L).
This penalty can shrink some coefficients to zero which helps in selecting only the important
features and ignoring the less important ones.
2. Ridge Regression
A regression model that uses the L2 regularization technique is called Ridge regression.
It adds the squared magnitude of the coefficient as a penalty term to the loss function(L).
It handles multicollinearity by shrinking the coefficients of correlated features instead of
eliminating them.
3. Elastic Net Regression
Elastic Net Regression is a combination of both L1 as well as L2 regularization.
We add the absolute norm of the weights as well as the squared measure of the weights. With
the help of an extra hyperparameter that controls the ratio of the L1 and L2 regularization.
Benefits of Regularization
Prevents Overfitting
Enhances Performance
Stabilizes Models
Prevents Complexity
Handles Multicollinearity
Promotes Consistency
[Link] Regularization
• Dropout regularization helps overcome this by randomly deactivating a portion of
neurons during training hence forcing the model to learn more robust and independent
features.
Key features of dropout
• Randomly deactivates a set percentage of neurons in each training step.
• Works with dense, convolutional and recurrent layers.
• Dropout rate controls how many neurons are dropped.
• Remaining active neurons are scaled to maintain output stability.
• Reduces co-dependency among neurons and improves generalization.
• Dropout works by randomly turning off a fraction of neurons during each training
pass.
• When a neuron is dropped it sends no output and receives no weight updates for that
batch. The dropout rate P determines the probability of removing a neuron.
• A neuron output h is modified as
'
h =h⋅ m
where
• m : Dropout mask randomly sampled for each neuron.
• m=1: neuron is kept
• m=0 : neuron is dropped
• To keep activation values consistent during training inverted dropout is used where
active neurons are scaled as:
' h ⋅m
h=
1−P
where
• h : original output of a neuron before dropout.
'
• h : output after applying dropout (scaled or zeroed out).
Types of Dropout Regularization
• Standard dropout
• Spatial dropout
• Dropout2d / dropout3d
• Drop connect
• Alpha dropout
• Variational dropout
• Monte Carlo dropout