Regularization is a technique that has been used long before deep learning.

It helps models like

linear regression, logistic regression, and neural networks avoid overfitting by preventing
the model from becoming too complex.
When a model has very large parameter values (weights), it can fit the training data extremely
well but perform poorly on new data. Regularization fixes this by penalizing large weights.
Normally, a model tries to minimize a loss function 𝐽 𝜃 .
With regularization, we add an extra penalty term: 𝐽ሚ 𝜃 = 𝐽 𝜃 + 𝛼Ω 𝜃
𝐽 𝜃 :original loss (training error)
Ω 𝜃 :penalty on model parameters (usually their size)
𝛼: controls how strong the regularization is
𝛼 = 0: no regularization
larger 𝛼: stronger regularization (smaller weights)
So, the model now tries to:
fit the data well
keep the parameters small
In neural networks, we usually regularize only the weights, not the biases.
Reason:
Weights represent interactions between input features → harder to estimate accurately →
higher risk of overfitting
Biases affect only one variable → easier to estimate → less risk of overfitting
Regularizing biases can: cause unnecessary underfitting
reduce model flexibility without much benefit
So: 𝑤: weights (regularized)
𝜃: all parameters (weights + biases)
Concrete “Before vs After”
❌ Without Weight Decay ✅ With Weight Decay
A filter strongly activates on high-contrast shadows Shadow-only cues are penalized
Large weight → shadow ≈ pedestrian Filters must cooperate (edges + shape + motion)
False positives spike at dusk Fewer false positives, better generalization

Another Quick Example: Face Recognition on Phones

Without weight decay: Model overfits to glasses, beard styles, lighting
With weight decay: Learns stable facial structures (eyes, nose geometry)
Result: Consistent recognition across lighting and accessories
Why L2 Is Preferred Here
Keeps all weights small (not zeroed)
Maintains rich feature combinations
Produces smooth decision boundaries—critical for vision
In deep neural networks:
•Each layer can technically have its own regularization strength 𝛼
•But tuning many 𝛼 values is computationally expensive
Practical approach:
•Use the same regularization (weight decay) for all layers
•This reduces the number of hyperparameters to tune
•Still provides good control over overfitting
What is L2 Regularization?
L2 regularization (also called weight decay, ridge regression, or Tikhonov regularization)
works by penalizing large weights.
𝟏
It adds this term to the loss function: 𝛀 𝒘 = ∥ 𝒘 ∥𝟐𝟐
𝟐

This pulls all weights toward zero, but usually does not make them exactly zero.
How Weight Decay Changes Learning
𝜶
The regularized loss becomes: 𝑱෨ 𝒘 = 𝑱 𝒘 + 𝟐 𝒘𝑻 𝒘

During gradient descent, the weight update becomes: 𝒘 ← 𝟏 − 𝝐𝜶 𝒘 − 𝝐𝜵𝒘 𝑱 𝒘

Every training step does two things: Shrinks the weights slightly (multiplication by 1 − 𝜖𝛼)
Then updates them using the gradient of the loss
So, even if the gradient is zero, weights will slowly decay toward zero.
Weights 𝐰→ Bandwidth allocated to each data flow Loss function 𝐽 𝐰 →Network congestion cost
•Each weight 𝑤𝑖 =amount of bandwidth assigned to This represents:
flow i Packet loss
• Video call High latency
• Netflix stream Queue overflow
• Cloud backup Jitter
• Web browsing
Large weights = aggressive bandwidth usage → risk of
congestion
Regularization term Regularization strength 𝛼→ Congestion sensitivity
1 factor
Ω 𝐰 = ∥ 𝐰 ∥22
2 High 𝛼→ very conservative router
Penalty for excessive bandwidth usage Low 𝛼→ aggressive throughput
• Squaring the bandwidth means large flows are 📌 Example:
penalized heavily Peak hours → high 𝛼
• Encourages moderate, balanced usage rather than a Off-peak hours → low 𝛼
few greedy flows 𝛼
Regularized loss 𝐽ሚ 𝐰 = 𝐽 𝐰 + 𝐰 𝑇 𝐰
• 📌 Real-world meaning: 2
➡Router now optimizes two things at once:
• “Using too much bandwidth on many connections
Reduce congestion
stresses the router.”
Avoid excessive bandwidth allocation
Gradient Descent Update (Router Adjustment)
𝐰 ← 1 − 𝜖𝛼 𝐰 − 𝜖𝛻𝐰 𝐽 𝐰
Breaking Down Each Term
🔹 Learning rate 𝜖→ Speed of router reaction
•Small 𝜖: slow adaptation
•Large 𝜖: fast but unstable
Shrinkage term 1 − 𝜖𝛼 𝐰
➡Automatic bandwidth decay Even if traffic is stable:
Router slowly reduces bandwidth allocations Prevents long-term congestion buildup
📌 Real-world analogy: “If traffic is quiet, the router gradually releases reserved bandwidth.”
Gradient term −𝜖𝛻𝐰 𝐽 𝐰
➡Reactive adjustment to congestion If packet loss increases → reduce bandwidth
If queues are empty → allow more bandwidth
To understand the long-term effect, imagine the loss surface near its minimum:
•The loss function looks like a bowl (quadratic shape)
•Some directions are steep (important features)
•Some directions are flat (unimportant features)
The Hessian matrix tells us how curved the loss is in each direction
 Large curvature (large eigenvalue): Small movement → big increase in loss
 Small curvature (small eigenvalue): Movement barely changes the loss
After regularization, the optimal weights become: ෥ = ቀ𝑯 + 𝜶𝑰)−𝟏 𝑯𝒘∗
𝒘
This means:
Each component of 𝑤 ∗ is scaled down
𝝀𝒊
Scaling factor for each direction:
𝝀𝒊 +𝜶
Important directions (large 𝜆𝑖 :)
Scaling ≈ 1 → weights stay mostly unchanged
Unimportant directions (small 𝜆𝑖 :)
Scaling ≈ 0 → weights shrink toward zero
📌 L2 regularization removes weights in directions that don’t help reduce loss.
Identifying Genuine News on a News Platform
Regularized optimal weights 𝐰෥ = ቀ𝐇 + 𝛼𝐈)−1 𝐇𝐰 ∗
w∗ → Ideal but overconfident news weights
These are the weights learned without regularization
Assign very large importance to: Clickbait words, Writing style quirks, Rare author phrases
📌 Real-world meaning: “The model is overreacting to patterns that worked only on past
articles.”
H → Curvature of the loss surface (news reliability sensitivity)
𝐇 tells how strongly each feature affects classification error
Eigenvectors = directions in feature space
Eigenvalues 𝜆𝑖 =importance of those directions
📌 In news terms: Large 𝜆𝑖 :strongly correlated with genuine news
Small 𝜆𝑖 :weak or noisy signals
αI → Editorial caution / regularization policy
Adds skepticism toward all features equally
Prevents trusting weak patterns too much
📌 Example: During elections, platforms raise 𝛼 to reduce misinformation risk.
Eigenvalue Scaling — What Really Happens
𝜆𝑖
For each direction 𝑖: Scaling factor = 𝜆 +𝛼
𝑖
Important Directions (Large 𝜆𝑖 → )Trusted News Signals

Examples: Meaning:
Verified source reputation “These signals truly reduce classification
Cross-reference with fact-checking databases error, so we keep them.”
Presence of official citations ✔ Weights stay almost the same
Historical accuracy of the publisher ✔ Genuine journalism remains recognized
𝜆𝑖
Effect: ≈1
𝜆𝑖 +𝛼

Unimportant Directions (Small 𝜆𝑖 → )Noisy or Misleading 📌 Meaning:

Signals “These features don’t reliably indicate
Examples: Sensational adjectives (“shocking”, “unbelievable”) truth.”
Writing style mimicking trusted outlets ❌ Weights shrink toward zero
Rare phrase patterns ❌ Clickbait influence disappears
Short-term trending keywords
Effect:
𝜆𝑖
≈0
𝜆𝑖 + 𝛼
Geometric Intuition (Figure Explanation)
•Ellipses = contours of original loss
•Circles = contours of L2 penalty
•Final solution = balance between: minimizing loss and minimizing weight magnitude
Result:
•Weights in flat directions shrink heavily
•Weights in steep directions remain large
Flat directions often correspond to: noise and redundant or weak features
Shrinking them:
o improves generalization
o reduces overfitting
L2 Regularization in Linear Regression
For linear regression:

Without regularization: 𝑤 = ൫𝑋 𝑇 𝑋)−1 𝑋 𝑇 𝑦

With L2 regularization: 𝑤 = ൫𝑋 𝑇 𝑋 + 𝛼𝐼)−1 𝑋 𝑇 𝑦

What changed?
•Added 𝛼𝐼to 𝑋 𝑇 𝑋
•This increases the perceived variance of input features
Final Intuition for Linear Regression
Features with strong correlation to the output → kept
Features with weak correlation → weights shrink
Model becomes: more stable, less sensitive to noise, better at generalization
What is L1 Regularization?
L1 regularization penalizes the sum of absolute values of weights: Ω 𝑤 =∥ 𝑤 ∥1 = σ𝑖 ∣ 𝑤𝑖 ∣
Instead of smoothly shrinking weights (like L2), L1 actively pushes some weights to exactly
zero.
Regularized Objective Function
With L1 regularization, the loss becomes:
𝐽ሚ 𝑤 = 𝐽 𝑤 + 𝛼 ∥ 𝑤 ∥1
𝛼controls how strong the penalty is
Larger 𝛼→ more aggressive shrinking
How L1 Changes the Gradient
The gradient (technically, a subgradient) becomes:
𝛻𝑤 𝐽ሚ 𝑤 = 𝛻𝑤 𝐽 𝑤 + 𝛼 sign 𝑤
Key difference from L2:
L2: penalty grows with weight size
L1: penalty is a constant force pulling toward zero
📌 This constant pull is what creates sparsity.
Intuition Using a Quadratic Loss
Near the optimal unregularized solution 𝑤 ∗ ,the loss looks like a quadratic bowl.
Assuming features are uncorrelated (diagonal Hessian):
1
መ𝐽 𝑤 = ෎ ൤ 𝐻𝑖,𝑖 𝑤𝑖 − 𝑤𝑖∗ )2 + 𝛼 ∣ 𝑤𝑖 ∣
2
𝑖
Each weight dimension can be analyzed independently.
Soft-Thresholding Solution
The optimal solution for each weight is:
𝛼
𝑤𝑖 = sign 𝑤𝑖∗ max ∣ 𝑤𝑖∗ ∣ − 𝐻 0
𝑖,𝑖
This leads to two cases:
Two Possible Outcomes for Each Weight
Case 1: Weight becomes zero
If:
𝛼
∣ 𝑤𝑖∗ ∣≤
𝐻𝑖,𝑖
Then:
𝑤𝑖 = 0
👉 Regularization dominates → feature is discarded.
Case 2: Weight is shrunk but nonzero
If:
∗ 𝛼
∣ 𝑤𝑖 ∣>
𝐻𝑖,𝑖
Then:
∗ 𝛼
𝑤𝑖 = 𝑤𝑖 − ⋅ sign 𝑤𝑖∗
𝐻𝑖,𝑖
👉 Weight is reduced by a fixed amount.
Property L1 Regularization L2 Regularization
Shrinking behavior Constant pull Proportional to size
Produces zeros ✅ Yes ❌ No
Sparsity High Low
Feature selection Automatic No
Stability Less More
Feature Selection with L1 (LASSO)
Because L1 drives some weights to zero:
•Corresponding features can be removed
•Model becomes simpler and more interpretable
This is exactly what LASSO (Least Absolute Shrinkage and Selection Operator) does:
•Linear regression + L1 penalty and Built-in feature selection
Bayesian Interpretation
L1 regularization is equivalent to MAP estimation with a Laplace (double-exponential)
prior on weights: 𝑝 𝑤𝑖 ∝ 𝑒 −𝛼∣𝑤𝑖 ∣
Laplace prior has a sharp peak at zero and Encourages weights to be exactly zero
📌 In contrast: L2 ↔ Gaussian prior
L1 ↔ Laplace prior
Two Ways to Control Model Complexity
There are two equivalent ways to prevent model parameters from becoming too large:
[Link]-based regularization
1. Add a term like 𝛼Ω 𝜃 to the loss
2. Example: L1 or L2 weight decay
[Link]-based regularization
1. Directly restrict how large the parameters are allowed to be
2. Example: “Keep ∥ 𝜃 ∥≤ 𝑘”
These two views are mathematically connected
Regularization as a Constraint
Instead of minimizing: 𝐽ሚ 𝜃 = 𝐽 𝜃 + 𝛼Ω 𝜃
we can think of the problem as:
Minimize the loss, but only among parameters whose norm is below a limit
That is: Ω 𝜃 ≤𝑘
Role of the Lagrangian (Intuition Only)
To solve a constrained problem, we build a Lagrangian: 𝐿 𝜃 𝛼 =𝐽 𝜃 +𝛼 Ω 𝜃 −𝑘

𝛼 acts like a pressure knob

If parameters grow too large → increase pressure
If parameters are well within the limit → reduce pressure
At the optimum: Parameters are as large as they need to be
But not larger than allowed
Why the Penalized Form Works
If we fix the optimal value 𝛼 ∗ ,minimizing the Lagrangian becomes:

min 𝐽 𝜃 + 𝛼∗Ω 𝜃
𝜃
👉 This is exactly the same as standard regularization.
So:
•Penalty form = constraint form with unknown boundary
•Increasing 𝛼→ smaller allowed region
•Decreasing 𝛼→ larger allowed region
Geometry Intuition
L2 regularization → parameters lie inside an L2 ball (circle/sphere)
L1 regularization → parameters lie inside an L1 region (diamond)
We usually don’t know the exact size 𝑘of this region, but tuning 𝛼controls it indirectly.
Explicit Constraints vs Penalties
Instead of adding penalties, we can enforce constraints directly:
How? i. Take a gradient descent step on 𝐽 𝜃
ii. If Ω 𝜃 > 𝑘 ,project 𝜃back into the allowed region
This is called reprojection.
When Explicit Constraints Are Better
Penalty-based methods can sometimes cause problems, especially in neural networks:
Problem: “Dead units” Penalties constantly push weights toward zero
Some neurons end up with tiny incoming or outgoing weights
These neurons stop contributing → dead units
Explicit constraints: Do not pull weights toward zero
Only act when weights become too large and Preserve useful neurons better
Stability with High Learning Rates
With large learning rates:
•Large weights → large gradients → even larger weights
•This feedback loop can cause numerical explosion
Explicit constraints stop this by:
•Preventing weights from growing beyond a safe limit
•Making training more stable
This allows:
•Faster learning
•Safer exploration of parameter space
Column-wise Norm Constraints (Important in Practice)
Instead of constraining the entire weight matrix, we can:
Constrain each column separately
Each column corresponds to one hidden unit
Benefits:
No single neuron can dominate
Prevents extremely large weights in any one unit
Improves generalization and stability