Regularization
A Complete Beginner-Friendly Guide
This guide explains regularization in machine learning from the ground up: why it exists, how it
mathematically works, its main types, and how it connects to other overfitting-prevention techniques like
early stopping and dropout. No coding knowledge is assumed.
1. The Problem: Overfitting
Before understanding regularization, you need to understand the problem it solves: overfitting.
1.1 What Is Overfitting?
Overfitting happens when a machine learning model learns the training data too closely — including its
random noise and quirks — instead of learning the true, general pattern behind the data. An overfit
model performs extremely well on the data it was trained on, but performs poorly when it sees new,
unseen data.
1.2 Underfitting vs. Overfitting vs. Good Fit
There are three general outcomes when training a model:
• Underfitting: The model is too simple to capture the pattern in the data. It performs poorly on both
training data and new data.
• Good fit: The model captures the true underlying pattern. It performs well on training data and
generalizes well to new data.
• Overfitting: The model is too complex and captures noise along with the pattern. It performs very
well on training data but poorly on new data.
1.3 The Bias-Variance Tradeoff
This concept explains why overfitting happens. 'Bias' refers to error caused by a model being too simple
to capture the real pattern (this leads to underfitting). 'Variance' refers to error caused by a model being
overly sensitive to small fluctuations in the training data (this leads to overfitting). As you make a model
more complex (adding more parameters, deeper trees, more layers), bias tends to decrease but
variance tends to increase. The goal of a well-trained model is to find a balance point where both bias
and variance are kept reasonably low.
Note: Regularization is one of the main tools used to control this tradeoff. It reduces variance
(overfitting) at the cost of slightly increasing bias, usually resulting in a model that performs better overall
on new data.
2. What Is Regularization?
2.1 Core Idea
Regularization is a technique that discourages a machine learning model from becoming too complex. It
works by adding an extra penalty term to the model's error calculation (called the 'loss function'), based
on the size of the model's parameters. 'Parameters' here refers to the internal numeric values the model
learns during training — for example, in linear regression, these are the coefficients (weights) assigned
to each input feature.
2.2 Why Penalizing Large Parameters Helps
When a model is allowed to assign very large weights to its parameters, it can bend and twist its
predictions to match every small detail (including noise) in the training data. By penalizing large
parameter values, regularization forces the model to keep its parameters smaller, which naturally
produces a smoother, simpler prediction function — one that is less likely to overreact to noise.
2.3 The General Formula
Without regularization, a model tries to minimize only the prediction error:
Note: Total Error = Prediction Error (how wrong the predictions are)
With regularization, an extra penalty term is added:
Note: Total Error = Prediction Error + (Regularization Strength × Penalty on Parameter Size)
The model is now trained to minimize this combined total, meaning it must balance two competing goals:
fitting the training data well, and keeping its parameters small.
2.4 The Regularization Strength Parameter
The 'Regularization Strength' in the formula above (often written as lambda or alpha) is a value you
choose before training. It controls how much weight is given to the penalty term relative to the prediction
error:
• If this value is set to 0, regularization has no effect — the model behaves as if there were no penalty
at all.
• If this value is set very high, the penalty dominates, forcing parameters to shrink drastically — this
can cause underfitting if set too high.
• A well-chosen value strikes a balance between preventing overfitting and still allowing the model to
fit the true pattern in the data.
This value is typically chosen using techniques like cross-validation, where different values are tested
and the one that performs best on unseen validation data is selected.
3. L1 Regularization (Lasso)
3.1 What It Is
L1 regularization, also known as Lasso (Least Absolute Shrinkage and Selection Operator), adds a
penalty equal to the sum of the absolute values of all the model's parameters.
3.2 Key Characteristic: Sparsity
The defining feature of L1 regularization is that it can shrink some parameters all the way down to
exactly zero. When a parameter becomes zero, it means that feature is effectively removed from the
model entirely — the model simply ignores it when making predictions.
3.3 Why This Matters
Because L1 regularization can eliminate features completely, it acts as a built-in feature selection
method. Instead of manually deciding which features to keep or remove, the training process itself
identifies and removes the least useful ones automatically.
3.4 When to Use L1
• When you suspect that only a small number of your features are actually useful, and the rest are
irrelevant or redundant.
• When you want a simpler, more interpretable model with fewer active features.
• When you're working with a very high number of features and want automatic feature reduction.
4. L2 Regularization (Ridge)
4.1 What It Is
L2 regularization, also known as Ridge regression, adds a penalty equal to the sum of the squared
values of all the model's parameters.
4.2 Key Characteristic: Shrinkage Without Elimination
Unlike L1, L2 regularization shrinks all parameters toward smaller values, but it rarely forces any
parameter to become exactly zero. This means all features generally remain in the model, just with
reduced influence on the final prediction.
4.3 Why Squaring Matters
Because the penalty is based on squared values, L2 regularization penalizes large parameter values
much more heavily than small ones (since squaring amplifies larger numbers more than smaller
numbers). This makes L2 particularly effective at preventing any single feature from dominating the
model's predictions.
4.4 When to Use L2
• When you believe most or all of your features contribute at least some useful information.
• When you want to reduce the impact of highly correlated features without removing any of them.
• When you want a smoother, more stable model, especially useful when features are correlated with
each other.
5. L1 vs. L2: Side-by-Side Comparison
Aspect L1 (Lasso) L2 (Ridge)
Penalty based on Absolute value of parameters Squared value of parameters
Effect on parameters Can shrink to exactly zero Shrinks toward zero, rarely
reaches it
Feature selection Yes — automatic No — keeps all features
Best when Few features are truly useful Most features are somewhat
useful
Model interpretability Higher (simpler, fewer features) Lower (keeps all features)
Handles correlated May pick one, drop the rest Distributes weight across
features correlated features
5.1 Elastic Net: Combining Both
Elastic Net is a third approach that combines both L1 and L2 penalties together, using a weighted mix of
the two. This allows it to perform automatic feature selection (like L1) while also handling correlated
features more gracefully (like L2). It is useful when you're unsure which of L1 or L2 would work better, or
when your dataset has many correlated features alongside some genuinely irrelevant ones.
6. Early Stopping
6.1 What It Is
Early stopping is a different technique for preventing overfitting, used specifically for models that are
trained iteratively — meaning they improve gradually over many repeated rounds (called 'epochs' or
'iterations'). Examples include neural networks and gradient boosting models.
6.2 How It Works
During training, the dataset is split into a training set and a separate validation set (data the model does
not learn from, but is checked against periodically). As training progresses:
• Training error typically keeps decreasing the longer training continues.
• Validation error decreases at first, but eventually starts increasing again — this is the point where
the model begins overfitting.
Early stopping monitors the validation error during training and stops the process as soon as validation
error stops improving (or starts getting worse), even though training error might still be improving. This
prevents the model from continuing to overfit past its optimal point.
6.3 Why It's Considered a Form of Regularization
Although early stopping doesn't add a penalty term to the loss function like L1 or L2, it achieves a similar
goal: it limits how much the model can adjust and complicate itself, effectively constraining its complexity
by controlling training duration rather than parameter size.
7. Dropout
7.1 What It Is
Dropout is a regularization technique used specifically in neural networks — models made up of layers of
connected computing units called 'neurons.' During training, dropout randomly and temporarily disables
(sets to zero) a fraction of neurons in a layer for each training step.
7.2 How It Works
At each training step, a different random subset of neurons is turned off. This means the network can
never fully rely on any specific neuron or specific small group of neurons being present, since they might
be missing on the next training step.
7.3 Why This Prevents Overfitting
Without dropout, certain neurons can become overly specialized, essentially memorizing very specific
patterns (including noise) from the training data. By randomly removing neurons during training, dropout
forces the network to spread out its learned knowledge across many neurons, resulting in more robust,
general-purpose patterns that transfer better to new, unseen data.
7.4 An Important Detail
Dropout is only applied during training. When the model is actually used to make predictions on new
data (after training is complete), all neurons are active and used together.
8. Choosing the Right Regularization Approach
8.1 Quick Decision Guide
• Linear or logistic regression models: Start with L2 (Ridge). Switch to L1 (Lasso) if you
specifically want automatic feature selection. Consider Elastic Net if you have many correlated
features.
• Neural networks: Use dropout, often combined with L2 regularization and early stopping together.
• Gradient boosting models (e.g., XGBoost, LightGBM): Early stopping is commonly used,
alongside built-in regularization parameters these libraries provide.
8.2 Signs You Need More Regularization
• Training accuracy is much higher than validation/test accuracy.
• The model's predictions change drastically with small changes in input data.
• The model has a very large number of parameters relative to the amount of training data available.
8.3 Signs You Have Too Much Regularization
• Both training and validation accuracy are low (this indicates underfitting).
• The model performs worse than a simpler baseline approach.
9. Summary: Quick Revision Notes
Concept Core Idea
Overfitting Model learns noise in training data; performs poorly on new data
Bias-Variance Tradeoff Simpler models = higher bias; complex models = higher variance
Regularization (general) Adds a penalty for large parameters to reduce model complexity
L1 (Lasso) Penalizes absolute value; can zero out features (feature
selection)
L2 (Ridge) Penalizes squared value; shrinks all features, rarely to zero
Elastic Net Combines L1 and L2 penalties together
Regularization Strength Controls how strong the penalty is; tuned via cross-validation
Early Stopping Stops training when validation error stops improving
Dropout Randomly disables neurons during neural network training
Bottom Line
Regularization trades a small increase in training error for a large improvement in how well a model
generalizes to new, unseen data. L1 and L2 are the two foundational penalty-based methods; early
stopping and dropout extend the same underlying idea — controlling model complexity — to iterative
training processes and neural networks respectively.