ML Placement Prep — Chapter 5: Ridge Regression (L2) 1

CHAPTER 5
Ridge Regression (L2)
Taming Overfitting with a Mathematically Elegant Penalty

Chapter Overview
Ridge Regression is the first and most fundamental regularisation technique in machine
learning. It extends linear regression by adding a penalty on the size of the coefficients,
preventing any single feature from dominating and shrinking all coefficients toward zero to
combat overfitting. This chapter builds Ridge Regression from first principles: we derive the
cost function, prove the closed-form solution, explain why the λI term guarantees invertibility of
what would otherwise be a singular matrix, visualise the circular L2 constraint geometry,
derive the bias–variance trade-off as a function of λ, contrast Ridge with OLS, show the
connection to weight decay in deep learning, and implement the complete workflow from
scratch in NumPy and via sklearn’s Ridge and RidgeCV. Every concept includes a rigorous
derivation and a concrete numerical example.

5.1 The Overfitting Problem and Why Regularisation Helps

Before introducing Ridge Regression, we need to deeply understand the problem it solves.
Overfitting is the central challenge of supervised machine learning, and regularisation is the
primary mathematical tool for fighting it.

What Overfitting Looks Like

Overfitting occurs when a model learns the training data so thoroughly that it captures not just
the true underlying signal but also the random noise specific to those particular samples. The
model becomes excessively complex — fitting every wiggle and quirk of the training set — and
consequently fails to generalise to new, unseen data.
The classic symptom: training error is very low (the model fits training data beautifully), but test
error is dramatically higher (the model fails on new data). The gap between training and test
performance is the empirical fingerprint of overfitting.
In the context of linear regression, overfitting manifests as extremely large coefficient values.
When a model overfits, individual coefficients θj can take values of +10,000 or −15,000 — the
model has learned to make predictions by having large positive and negative coefficients that
mostly cancel each other out. This produces correct predictions on training samples (because
the cancellations are tuned to those exact points) but wildly wrong predictions on new samples
where the inputs are slightly different.

AI/ML Placement Prep Book | Part II — Regularisation

ML Placement Prep — Chapter 5: Ridge Regression (L2) 2

Why Coefficients Become Large During Overfitting

Consider a polynomial regression with degree 15 fit to 16 training points. The model has enough
parameters to pass through every training point exactly (interpolation). To do this, it must fit
sharp oscillations between training points, requiring very large coefficient values at alternating
signs. On new test points that fall between training points, these oscillations produce predictions
far from the true function.
The OLS Normal Equation θ = (XᵀX)⁻¹Xᵀy minimises MSE on the training set with no
constraints on the size of θ. It will happily produce θ values of ±100,000 if that is what minimises
training MSE. There is nothing in the OLS formulation that penalises large coefficients — it is
entirely agnostic about the magnitude of the solution.

How Regularisation Helps

Regularisation solves overfitting by adding a penalty term to the objective function that grows
with the size of the coefficients. Instead of minimising only the fit to training data (MSE), we
simultaneously minimise the fit AND a measure of coefficient size:
Total objective = Data fit term + λ · Regularisation term
The scalar λ (lambda) controls how strongly we penalise large coefficients. Large λ puts heavy
weight on keeping coefficients small (at the cost of worse training fit). Small λ puts most weight
on training fit (approaching OLS as λ→0).
By penalising large coefficients, regularisation prevents the model from fitting noise. The model
is forced to use simpler, smaller-coefficient solutions that generalise better. The bias-variance
trade-off is moved toward slightly higher bias (the model cannot fit training data as perfectly) but
much lower variance (predictions are more stable across different training sets).

The Core Insight

Regularisation is not about 'helping' the model learn better from training data. It deliberately
makes the model learn slightly less from training data, in exchange for much better
generalisation to test data. It trades a small increase in training error for a large decrease in test
error. This is the bias-variance trade-off in action, controlled by the regularisation strength λ.

5.2 The Ridge Cost Function — MSE + λΣθj²

Ridge Regression uses the L2 norm squared of the coefficient vector as its penalty term. The
Ridge cost function is:
J_Ridge(θ) = (1/m)Σi(yi − ŷi)² + λ · Σj=1 to n θj²
Or in compact matrix form:
J_Ridge(θ) = (1/m)‖Xθ − y‖² + λ ‖θ_excl‖²

Dissecting the Cost Function

The MSE term: (1/m)Σ(yi−ŷi)²

AI/ML Placement Prep Book | Part II — Regularisation

ML Placement Prep — Chapter 5: Ridge Regression (L2) 3

This is exactly the same term as in standard OLS linear regression. It measures how well the
model fits the training data. Minimising this term alone gives the OLS solution with potentially
large coefficients.

The penalty term: λ · Σj=1 to n θj²

This is the L2 penalty — the sum of squared coefficients, scaled by λ. Note two critical design
choices:
• The sum starts at j=1, NOT j=0. The intercept θ0 is intentionally excluded from the
penalty. Penalising the intercept would bias predictions in a scale-dependent way; the
intercept represents the overall mean level of y and should be free to take whatever value
best centres the predictions.
• The coefficients are squared. This creates a smooth, differentiable penalty (unlike L1
regularisation used in Lasso, where |coefficient| has a corner at zero). Squaring means
small coefficients incur very little penalty while large coefficients incur a disproportionately
large penalty, aggressively shrinking outlier-sized weights.

The regularisation strength λ ≥ 0

Lambda is the single hyperparameter of Ridge Regression:
• λ = 0: the penalty disappears entirely and J_Ridge = MSE exactly. Ridge reduces to
standard OLS. No regularisation.
• Small λ (e.g., 0.001): weak penalty. Coefficients are allowed to be moderately large.
Predictions are close to OLS. Slight bias, small variance reduction.
• Moderate λ (e.g., 1): balanced. Coefficients are meaningfully shrunk. Good generalisation
for many problems.
• Large λ (e.g., 1000): strong penalty. All coefficients are shrunk close to zero. Model
makes predictions close to the mean of y for all inputs. High bias, very low variance.
• λ → ∞: all coefficients → 0 (except intercept θ0 which is not penalised). The model
predicts a constant value (θ0 = ȳ) for all inputs.

Matrix Form of the Ridge Cost Function

In matrix notation (where θ includes the intercept θ0 and we define a modified penalty matrix):
J_Ridge(θ) = (1/m)(Xθ−y)ᵀ(Xθ−y) + λθᵀΛθ
Where Λ is a diagonal matrix that is the identity with a zero in the top-left corner (to exclude the
intercept from the penalty):
Λ = diag(0, 1, 1, ..., 1) shape: (n+1) × (n+1)
In practice, when you standardise features (which you should always do before Ridge), the
intercept's value is approximately zero and its exclusion from the penalty matters less. sklearn's
Ridge implementation always excludes the intercept from the penalty regardless.

AI/ML Placement Prep Book | Part II — Regularisation

ML Placement Prep — Chapter 5: Ridge Regression (L2) 4

5.3 Closed-Form Solution — θ = (XᵀX + λI)⁻¹Xᵀy

One of Ridge Regression's most elegant properties is that it retains the closed-form solution of
OLS. Unlike Lasso (L1 regularisation) which has no closed form, the L2 penalty preserves the
quadratic structure of the objective function, allowing exact solution by setting the gradient to
zero.

Derivation from First Principles

To find the minimum of J_Ridge(θ), take the gradient with respect to θ and set it to zero. Using
the matrix form J = (1/m)(Xθ−y)ᵀ(Xθ−y) + λθᵀθ (simplifying by treating intercept uniformly):
∇J_Ridge(θ) = ∇[(1/m)(Xθ−y)ᵀ(Xθ−y)] + ∇[λθᵀθ]

Term 1 (from Chapter 2): (2/m)Xᵀ(Xθ−y)

Term 2 (ridge penalty): 2λθ ← ∂(λθᵀθ)/∂θ = 2λθ

Full gradient: ∇J_Ridge = (2/m)Xᵀ(Xθ−y) + 2λθ

Setting ∇J_Ridge = 0:
(2/m)Xᵀ(Xθ−y) + 2λθ = 0
XᵀXθ − Xᵀy + mλθ = 0 ← multiply by m/2
(XᵀX + mλ I)θ = Xᵀy

θ_Ridge* = (XᵀX + mλ I)⁻¹ Xᵀy

This is the Ridge Normal Equation. It has exactly the same structure as the OLS Normal
Equation θ_OLS = (XᵀX)⁻¹Xᵀy, with one crucial modification: mλ is added to every diagonal
element of XᵀX before inverting.

The Standard Form with a Single λ

In most textbooks and sklearn, λ absorbs the 1/m factor (i.e., sklearn's alpha corresponds to mλ
in our derivation). The standard Ridge Normal Equation is written as:
θ* = (XᵀX + λI)⁻¹ Xᵀy
Where λ here is what sklearn calls alpha. This is the form you should quote in interviews. The
added λI term on the diagonal is what makes Ridge fundamentally different from OLS and gives
it its special properties.

Numerical Worked Example

Design matrix X (2 samples, 1 feature, with bias column):
X = [[1, 1], [1, 3]] y = [2, 6]ᵀ λ = 1.0
XᵀX = [[1,1],[1,3]]ᵀ [[1,1],[1,3]] = [[2, 4], [4, 10]]
Xᵀy = [[1,1],[1,3]]ᵀ [2,6]ᵀ = [8, 20]ᵀ

OLS: (XᵀX)⁻¹Xᵀy = inv([[2,4],[4,10]]) @ [8,20]

inv = [[10,-4],[-4,2]] / (20-16) = [[2.5,-1],[-1,0.5]]

AI/ML Placement Prep Book | Part II — Regularisation

ML Placement Prep — Chapter 5: Ridge Regression (L2) 5

theta_OLS = [2.5*8 + (-1)*20, (-1)*8 + 0.5*20] = [0, 2]

Ridge: (XᵀX + λI)⁻¹Xᵀy = inv([[3,4],[4,11]]) @ [8,20]

det = 3*11 - 4*4 = 33-16 = 17
inv = [[11,-4],[-4,3]] / 17
theta_Ridge = [(11*8 + (-4)*20)/17, ((-4)*8 + 3*20)/17]
= [(88-80)/17, (-32+60)/17]
= [0.471, 1.647]
The OLS slope is exactly 2.0 (true value since y = 0 + 2x). Ridge shrinks it to 1.647 and adjusts
the intercept. This shrinkage is the price of regularisation — Ridge accepts slightly higher
training error (bias) to gain resistance to overfitting.

5.4 Why λI Guarantees Invertibility

One of Ridge Regression's most practically important properties is often overlooked: by adding
λI to XᵀX, Ridge guarantees that the matrix being inverted is always invertible, regardless of the
data. This solves a fundamental problem with OLS.

The OLS Invertibility Problem

The OLS Normal Equation requires inverting XᵀX. This matrix is NOT invertible (singular) in two
common situations:
• Perfect multicollinearity: one feature is an exact linear combination of others (e.g., feature
3 = feature 1 + feature 2 always). Then the columns of X are linearly dependent, X has
rank less than n+1, and XᵀX is singular with determinant zero.
• More features than samples (p > n): when n > m (more features than data points), X
cannot have full column rank (it has at most m linearly independent columns), so XᵀX (an
(n+1)×(n+1) matrix) also cannot have full rank. This is common in genomics (thousands of
genes, hundreds of patients), text analysis, and other high-dimensional problems.
In both cases, XᵀX is singular: its determinant is zero, its inverse does not exist, and the OLS
Normal Equation has either no solution or infinitely many solutions. This is a fundamental failure
of OLS in these settings.

Why λI Fixes This — The Eigenvalue Argument

The key insight uses eigenvalues. A matrix A is invertible if and only if none of its eigenvalues
are zero. For a positive semi-definite matrix like XᵀX, all eigenvalues are non-negative: λ_i(XᵀX)
≥ 0. Singular XᵀX means at least one eigenvalue equals exactly zero.
Now consider (XᵀX + λI). The eigenvalues of this sum are:
λ_i(XᵀX + λI) = λ_i(XᵀX) + λ
Since λ_i(XᵀX) ≥ 0 and λ > 0:
λ_i(XᵀX + λI) = λ_i(XᵀX) + λ ≥ 0 + λ = λ > 0

AI/ML Placement Prep Book | Part II — Regularisation

ML Placement Prep — Chapter 5: Ridge Regression (L2) 6

Every eigenvalue of (XᵀX + λI) is strictly positive, regardless of whether XᵀX had zero
eigenvalues. Therefore (XᵀX + λI) is strictly positive definite and always invertible for any λ > 0.

The Geometric Interpretation

Adding λI to a matrix is called Tikhonov regularisation or ridge-shifting. Geometrically, it shifts
the ellipsoid defined by XᵀX outward in every direction uniformly by λ. An ellipsoid that was
degenerate (collapsed to a lower-dimensional manifold because some eigenvalues were zero)
becomes a full-dimensional ellipsoid after the shift. The inversion of this non-degenerate
ellipsoid is always well-defined.

The Condition Number Improvement

Even when XᵀX is technically invertible (all eigenvalues positive), it can be nearly singular: the
condition number κ = λ_max/λ_min is very large, making numerical inversion unstable (small
rounding errors in X lead to huge errors in the computed θ). Adding λI:
κ(XᵀX + λI) = (λ_max + λ) / (λ_min + λ) < λ_max / λ_min = κ(XᵀX)
The condition number always decreases, making the numerical inversion more stable. With very
large λ, the condition number approaches 1 (perfectly stable), but at the cost of ignoring the
data entirely. This reveals the tension between numerical stability and data fidelity.

Practical Takeaway
The fact that Ridge always has a unique, well-defined solution is a major practical advantage
over OLS. In real datasets with near-multicollinear features (correlations of 0.95+), OLS
coefficients are numerically unstable and can vary wildly between samples. Ridge stabilises
them. In p > n settings (more features than samples), OLS completely fails while Ridge still
works. This is why Ridge is the default regularisation method for high-dimensional linear
regression.

5.5 Geometric Interpretation — Circular L2 Constraint

The geometric picture of Ridge Regression is one of the most elegant in all of machine learning.
It reveals why L2 regularisation shrinks all coefficients but never sets any exactly to zero — a
fundamental distinction from Lasso (L1).

The Constrained Optimisation View

Ridge Regression can equivalently be formulated as a constrained optimisation problem.
Instead of adding a penalty term to the objective, we minimise MSE subject to a constraint on
the sum of squared coefficients:
Minimise: J(θ) = (1/m)Σ(yi − ŷi)²
Subject to: Σj=1 to n θj² ≤ s²
For each value of the constraint budget s, there is a corresponding value of λ that makes the
penalised form equivalent to this constrained form (via Lagrangian duality). Larger s allows
larger coefficients (weaker regularisation, smaller λ). Smaller s restricts coefficients more

AI/ML Placement Prep Book | Part II — Regularisation

ML Placement Prep — Chapter 5: Ridge Regression (L2) 7

(stronger regularisation, larger λ). The two formulations are mathematically equivalent for the
purposes of finding the optimal θ.

The Constraint Region: A Circle (Sphere in Higher Dimensions)

The constraint Σθj² ≤ s² defines a ball (circle in 2D, sphere in 3D) in the parameter space. This is
the L2 ball of radius s. All parameter vectors θ inside this ball satisfy the constraint. The Ridge
solution is the point inside this ball that achieves the lowest MSE.
In 2D parameter space (θ1, θ2), the constraint region is a circle centred at the origin: θ1² + θ2² ≤
s². The MSE cost function has elliptical contour lines (as we saw in Chapter 2). Ridge finds the
point on or inside the circle that is closest to the centre of the MSE ellipses (the unconstrained
OLS solution).

Why Ridge Never Produces Exact Zeros

The critical geometric observation: the boundary of the L2 ball (the circle) is smooth — it has no
corners, no edges, no special points. When the MSE ellipse makes contact with the circle
boundary, it touches at a generic smooth point. The probability that this contact point falls
exactly on one of the coordinate axes (θ1 = 0 or θ2 = 0) is essentially zero.
This is in sharp contrast to the L1 ball (a diamond shape for 2D Lasso) which has four corners
precisely at the axes. The MSE ellipses very frequently make first contact with the L1 diamond
at one of these corners, producing an exact zero coefficient. The circular L2 ball has no corners
— no mechanism to produce exact zeros.
Therefore: Ridge always shrinks coefficients toward zero but never reaches exactly zero (for λ <
∞). This is Ridge's fundamental limitation compared to Lasso: it cannot perform automatic
feature selection by zeroing out irrelevant features.

The Shrinkage Mechanism — Which Features Are Shrunk More?

Ridge shrinks different features by different amounts, depending on their variance and
correlation structure. The Ridge estimator in terms of the Singular Value Decomposition (SVD)
of X = UDVᵀ is:
θ_Ridge = Σk [dk² / (dk² + λ)] · (ukᵀy / dk) · vk
Where dk are the singular values of X and vk are the right singular vectors. Each component of
θ is shrunk by the factor dk²/(dk²+λ):
• Features with large singular values dk (high variance, much information about the
direction of variation in X): the factor dk²/(dk²+λ) is close to 1. These features are shrunk
very little.
• Features with small singular values dk (low variance, little information): the factor is close
to λ/(dk²+λ) ≈ 1, meaning these features are shrunk heavily toward zero.
This is conceptually elegant: Ridge automatically shrinks the least informative directions (those
with small singular values) most aggressively, while preserving the most informative directions.

AI/ML Placement Prep Book | Part II — Regularisation

ML Placement Prep — Chapter 5: Ridge Regression (L2) 8

5.6 Effect of λ — Bias↑ Variance↓ Trade-off

The regularisation parameter λ directly controls where Ridge Regression sits on the bias-
variance trade-off spectrum. Understanding this relationship precisely is essential for choosing λ
and for explaining Ridge's behaviour in interviews.

As λ Increases: What Happens to the Coefficients

Ridge coefficients as a function of λ follow a smooth regularisation path. As λ increases from 0
to ∞:
• At λ = 0: θ_Ridge = θ_OLS. Full OLS solution, no regularisation. Coefficients can be large
and variable across bootstrap samples of the training data.
• As λ increases: all non-intercept coefficients shrink monotonically toward zero. The rate at
which each coefficient shrinks depends on its correlation with other features and its
contribution to reducing MSE.
• At λ → ∞: θ_Ridge → 0 for all non-intercept coefficients. The model predicts θ0 (the
intercept) for all inputs, which converges to the mean of y. The model is maximally biased
but has zero variance in the slope coefficients.

Bias² as a Function of λ
The bias of the Ridge estimator measures how far the average Ridge prediction is from the true
prediction, averaged over all training datasets. For the true parameter vector θ*:
Bias(θ_Ridge) = E[θ_Ridge] − θ* = [−λ(XᵀX + λI)⁻¹] θ*
The bias is zero at λ = 0 (Ridge = OLS, which is unbiased for linear models). The bias increases
monotonically with λ. At large λ, the bias approaches −θ* (the model's prediction approaches
zero, ignoring the true signal). Bias² always increases with λ.

Variance as a Function of λ
The variance of the Ridge estimator measures how much θ_Ridge fluctuates across different
training datasets drawn from the same distribution:
Var(θ_Ridge) = σ² (XᵀX + λI)⁻¹ XᵀX (XᵀX + λI)⁻¹
At λ = 0: Var(θ_OLS) = σ²(XᵀX)⁻¹, which is large when XᵀX is near-singular (correlated
features). As λ increases, the variance decreases monotonically — the Ridge solution becomes
more stable across different training datasets. The total MSE of the estimator (not to be
confused with the training MSE) is:
E[(θ_Ridge − θ*)²] = Bias²(θ_Ridge) + Var(θ_Ridge)
This total estimator MSE has a U-shape as a function of λ: zero bias at λ=0 (but high variance);
decreasing variance for increasing λ (but increasing bias). The optimal λ minimises the sum.
When the OLS variance is large (near-singular XᵀX, correlated features, high-dimensional data),
the variance decrease from Ridge can far outweigh the bias increase, making Ridge
dramatically superior to OLS.

λ value Coefficients Training Test MSE Bias Variance Overfittin

MSE g risk

AI/ML Placement Prep Book | Part II — Regularisation

ML Placement Prep — Chapter 5: Ridge Regression (L2) 9

0 (OLS) Unconstrained, Lowest Often high Zero High High

potentially huge possible

Small Nearly OLS, slight Slightly Slightly better Low Slightly Moderate
(0.001) shrinkage higher reduced

Moderate Meaningfully shrunk Moderate Often best Moderate Low Low

(1) increase

Large Near zero High Possibly High Very low Underfittin

(100) worse g risk

∞ All zero (non- Worst Worst Maximum Zero Underfittin

intercept) g

5.7 Ridge vs OLS — When Coefficients Never Reach Zero

Understanding the differences between Ridge and OLS is fundamental for deciding which to
use and for answering comparison questions in interviews.

Key Structural Difference: Shrinkage Without Sparsity

Ridge shrinks all coefficients toward zero but never sets any to exactly zero. OLS either gives
exact coefficients (when XᵀX is invertible) or fails entirely (when it is not). Lasso (L1
regularisation) can set coefficients to exactly zero. This creates three distinct solution profiles:
• OLS: θ = (XᵀX)⁻¹Xᵀy — exact, potentially large, no regularisation. Fails when XᵀX is
singular.
• Ridge: θ = (XᵀX + λI)⁻¹Xᵀy — shrunk, dense (all non-zero). Always works.
• Lasso: minimises MSE + λΣ|θj| — sparse (some exactly zero). Performs feature selection.

Why Ridge Cannot Zero Out Coefficients: The Gradient Argument

At any non-zero value of θj, the gradient of the Ridge penalty with respect to θj is:
∂(λθj²)/∂θj = 2λθj
As θj approaches zero, this gradient smoothly approaches zero as well. There is no finite value
of λ for which the gradient is non-zero at θj = 0 but zero just beyond (which would be needed to
'pin' θj at zero). The gradient is zero exactly at θj = 0, meaning the penalty itself is in equilibrium
at zero, but the MSE gradient (which has no reason to be zero at θj = 0) pulls the coefficient
away from zero. The net result: all coefficients sit at some non-zero value that balances the
MSE gradient against the penalty gradient.
For Lasso: the gradient of |θj| is ±1 for θj ≠ 0 and undefined (a subgradient of any value in
[−1,1]) at θj = 0. The penalty exerts a constant 'pull' of magnitude λ toward zero even very close
to zero, which is strong enough to pin small coefficients exactly at zero. This is the fundamental
geometric reason for sparsity.

Comprehensive Ridge vs OLS Comparison

AI/ML Placement Prep Book | Part II — Regularisation

ML Placement Prep — Chapter 5: Ridge Regression (L2) 10

Property OLS vs Ridge

Solution OLS: (XᵀX)⁻¹Xᵀy | Ridge: (XᵀX+λI)⁻¹Xᵀy

Invertibility OLS fails if XᵀX singular | Ridge always invertible

Coefficients OLS: exact (can be huge) | Ridge: shrunk toward 0, always non-
zero

Feature selection OLS: none | Ridge: none (all features kept)

Works when p > n OLS: No | Ridge: Yes

Near-multicollinearity OLS: unstable, huge variance | Ridge: stable, reduced variance

Bias OLS: zero (unbiased estimator) | Ridge: non-zero (introduced by

λ)

Variance OLS: high when XᵀX near-singular | Ridge: always lower

Test MSE OLS: can be very high | Ridge: often much lower

Hyperparameters OLS: none | Ridge: 1 (λ, chosen via CV)

Interpretability Both: coefficients are weights; Ridge’s are biased

Feature scaling OLS: invariant (scale cancels) | Ridge: REQUIRED (penalty is

scale-dependent)

When to Use Ridge vs OLS

Use Ridge when: (1) features are correlated (VIF > 5) and OLS coefficients are unstable; (2)
you have more features than samples (p > n); (3) training R² is much higher than test R²
(overfitting signal); (4) OLS coefficient magnitudes are implausibly large. Use OLS when: (1)
you need an unbiased estimator for statistical inference; (2) the five OLS assumptions are well-
satisfied; (3) you have abundant data relative to features; (4) you want coefficient p-values and
confidence intervals (Ridge invalidates standard OLS inference formulas).

5.8 Weight Decay in Deep Learning as Ridge Regularisation

One of the most important connections in machine learning is that weight decay — the most
common regularisation technique in deep learning — is exactly equivalent to L2 regularisation
(Ridge). Understanding this connection links the classical statistics of Ridge Regression to the
practical engineering of neural network training.

Weight Decay in Neural Networks

When training a neural network, weight decay adds a penalty on the L2 norm of all weight
parameters to the training loss:
L_total = L_data(θ) + (λ/2) · Σ wij²

AI/ML Placement Prep Book | Part II — Regularisation

ML Placement Prep — Chapter 5: Ridge Regression (L2) 11

Where L_data is the data loss (cross-entropy, MSE, or whatever loss the task requires), wij are
all the weight parameters of the network (often millions of them), and λ is the weight decay
coefficient (typically 10⁻⁴ to 10⁻²).
The gradient of the total loss with respect to a weight wij is:
∂L_total/∂wij = ∂L_data/∂wij + λ · wij
The gradient descent update becomes:
wij := wij − α · [∂L_data/∂wij + λ · wij]
= wij(1 − αλ) − α · ∂L_data/∂wij
The factor (1−αλ) multiplies the current weight before the gradient update. This means each
weight is scaled down by (1−αλ) at every step — hence the name 'weight decay': weights decay
toward zero at every update, slightly.

The Equivalence with L2 Regularisation

For a linear model, the weight decay update is exactly equivalent to adding λΣwij² to the training
loss — Ridge Regression. For a neural network, the same principle applies: weight decay
implements L2 regularisation on all network weights simultaneously.
The effect in both cases is identical: large weights are penalised, preventing any single
connection in the network from dominating the predictions. Small weights contribute little to the
loss and are free to grow if the data supports it; large weights face a quadratic penalty that
suppresses them.

Adam + Weight Decay: The Subtle Difference

An important subtlety: when using adaptive optimisers like Adam, there is a difference between
L2 regularisation (adding λΣw² to the loss) and weight decay (multiplying weights by (1−αλ) at
each step). With SGD, the two are exactly equivalent. With Adam, they are not, because Adam
scales the gradient by the inverse of the estimated gradient variance, which also scales the
regularisation gradient. The result is that L2 regularisation with Adam is ineffective — Adam
adapts away the regularisation signal.
AdamW (Adam with decoupled weight decay) addresses this by applying the weight decay
directly to the weights rather than through the loss gradient, making it equivalent to true L2
regularisation regardless of the optimiser. AdamW is the standard choice in modern deep
learning (used in BERT, GPT, ViT, and most transformer models).

Practical Weight Decay Values

Model type Typical λ (weight Why
decay)

ResNet image 1e−4 Standard for vision models; strong enough to prevent
classification overfit, not so strong as to underfit ImageNet

BERT / Transformer 0.01 Higher weight decay for transformer models;

NLP essential for stable training

Small MLP (tabular 1e−4 to 1e−2 Depends on dataset size; tune via CV
data)

GPT-style language 0.1 Large models benefit from stronger regularisation

AI/ML Placement Prep Book | Part II — Regularisation

ML Placement Prep — Chapter 5: Ridge Regression (L2) 12

model

Ridge regression Tune via RidgeCV Cross-validate on log scale [0.001, 0.01, 0.1, 1, 10,
(linear) 100]

5.9 Choosing λ via Cross-Validation

λ is a hyperparameter — it cannot be learned from the training data itself (the model would
trivially choose λ = 0 to minimise training MSE). It must be selected using a method that
estimates test performance without using test data. Cross-validation is the gold standard.

The Standard Protocol: K-Fold Cross-Validation

1. Choose a grid of candidate λ values. Always search on a log scale: [0.001, 0.01, 0.1, 1,
10, 100, 1000]. Equal spacing on log scale explores many orders of magnitude
efficiently.
2. Split the training data into K folds (K = 5 or 10). For each candidate λ: train Ridge on K-1
folds, evaluate MSE on the held-out fold, repeat K times, average the K fold MSEs.
3. Select the λ with the lowest average cross-validated MSE.
4. Retrain the final model on the full training set using the selected λ.
5. Report final performance on the test set (which was never used in any of the above
steps).

The Regularisation Path

Plotting the cross-validated MSE as a function of λ (on a log scale) reveals the regularisation
path:
• Left region (small λ, OLS-like): CV-MSE is high due to high variance (overfitting). Training
MSE is low but test MSE is high.
• Middle region (optimal λ): CV-MSE is minimised. Bias and variance are balanced.
• Right region (large λ, shrinkage-dominant): CV-MSE rises again due to increasing bias
(underfitting). Both training and test MSE are high.
The regularisation path also shows the coefficient paths: plotting θj(λ) as a function of λ gives
curves that all start at the OLS values at λ=0 and monotonically shrink toward zero as λ
increases.

The One-Standard-Error Rule

A useful heuristic: instead of choosing the λ with the absolute minimum CV-MSE, choose the
largest λ whose CV-MSE is within one standard error of the minimum. This selects a simpler
(more regularised) model that is statistically indistinguishable from the best in terms of
generalisation performance. The rationale: given that CV-MSE estimates have uncertainty
(different random fold splits give slightly different results), a simpler model within the uncertainty
range is preferred over the nominally best but more complex model.

AI/ML Placement Prep Book | Part II — Regularisation

ML Placement Prep — Chapter 5: Ridge Regression (L2) 13

5.10 sklearn Implementation — Ridge and RidgeCV

Part 1: Ridge from Scratch (NumPy)

# ridge_from_scratch.py
import numpy as np
from [Link] import StandardScaler
from sklearn.model_selection import train_test_split
from [Link] import mean_squared_error, r2_score

# ================================================================
# RIDGE REGRESSION FROM SCRATCH
# Implements: theta = (X^T X + lambda I)^{-1} X^T y
# ================================================================

class RidgeRegressionScratch:
def __init__(self, lam=1.0):
[Link] = lam # regularisation strength lambda
[Link] = None # learned parameters including intercept

def fit(self, X, y):

"""
Fit Ridge using the closed-form solution:
theta = (X^T X + lambda * I)^{-1} X^T y

Intercept is handled by NOT penalising theta[0].

We add a bias column to X and zero out the lambda*I
contribution for the intercept dimension.
"""
m, n = [Link]

# Add bias column (column of ones) for intercept

ones = [Link]((m, 1))
X_bias = [Link]([ones, X]) # shape (m, n+1)

# Build penalty matrix: lambda * I but with 0 for intercept

# Lambda_mat = diag(0, lambda, lambda, ..., lambda)
Lambda_mat = [Link] * [Link](n + 1)
Lambda_mat[0, 0] = 0.0 # do NOT penalise intercept

# Ridge Normal Equation: (X^T X + Lambda)^{-1} X^T y

A = X_bias.T @ X_bias + Lambda_mat # shape (n+1, n+1)
b = X_bias.T @ y # shape (n+1,)
[Link] = [Link](A, b) # more stable than inv(A)@b

return self

def predict(self, X):

ones = [Link](([Link][0], 1))
X_bias = [Link]([ones, X])
return X_bias @ [Link]

@property
def intercept_(self):

AI/ML Placement Prep Book | Part II — Regularisation

ML Placement Prep — Chapter 5: Ridge Regression (L2) 14

return [Link][0]

@property
def coef_(self):
return [Link][1:]

# ── Generate data with correlated features (Ridge's strength) ─

[Link](42)
m, n = 200, 5
X_base = [Link](m, 1)
# Deliberately correlated features: x2-x5 are x1 + noise
X = [Link]([
X_base,
X_base + 0.1*[Link](m,1), # nearly identical to x1
X_base + 0.1*[Link](m,1), # nearly identical to x1
[Link](m,1), # independent
[Link](m,1), # independent
])
true_coefs = [Link]([2.0, -1.5, 0.5, 1.0, -0.8])
y = 3 + X @ true_coefs + 0.5*[Link](m)

X_train,X_test,y_train,y_test =
train_test_split(X,y,test_size=0.2,random_state=42)

# CRITICAL: Scale features before Ridge!

scaler = StandardScaler()
X_train_s = scaler.fit_transform(X_train)
X_test_s = [Link](X_test)

# ── Compare OLS vs Ridge at several lambda values ─────────────

print('='*60)
print(f'{"Lambda":>10} {"Coefs (max)": >20} {"Train R2":>10} {"Test R2":>10}')
print('='*60)

for lam in [0, 0.01, 0.1, 1, 10, 100, 1000]:

model = RidgeRegressionScratch(lam=lam).fit(X_train_s, y_train)
yp_tr = [Link](X_train_s)
yp_te = [Link](X_test_s)
max_coef = float([Link](model.coef_).max())
r2_tr = r2_score(y_train, yp_tr)
r2_te = r2_score(y_test, yp_te)
print(f'{lam:>10.3f} {max_coef:>20.4f} {r2_tr:>10.4f} {r2_te:>10.4f}')

# Expected output pattern:

# Lambda Coefs(max) Train R2 Test R2
# 0.000 very large 1.000 negative (OLS overfits with correlated
features)
# 0.010 large 0.997 high (slight regularisation helps a
lot)
# 1.000 moderate 0.985 best (optimal regularisation)
# 100.000 small 0.900 dropping (too much shrinkage)
# 1000.000 very small 0.600 poor (underfitting)

AI/ML Placement Prep Book | Part II — Regularisation

ML Placement Prep — Chapter 5: Ridge Regression (L2) 15

Part 2: sklearn Ridge and RidgeCV

# ridge_sklearn.py
import numpy as np
import [Link] as plt
from sklearn.linear_model import Ridge, RidgeCV, LinearRegression
from [Link] import StandardScaler
from sklearn.model_selection import train_test_split, cross_val_score
from [Link] import Pipeline
from [Link] import r2_score, mean_squared_error

# ── Data (same correlated-feature setup) ─────────────────────

[Link](42)
m, n = 300, 8
X = [Link](m, n)
# Add multicollinearity: feature 1 and 2 are nearly identical
X[:, 1] = X[:, 0] + 0.05 * [Link](m)
true_coefs = [Link]([3., -1.5, 0.5, 1., -0.8, 0.3, -0.2, 0.1])
y = 4 + X @ true_coefs + [Link](m)

X_tr,X_te,y_tr,y_te = train_test_split(X,y,test_size=0.2,random_state=42)

# ── Method 1: Ridge with manual lambda ───────────────────────

pipe_ridge = Pipeline([
('scaler', StandardScaler()),
('ridge', Ridge(alpha=1.0)) # alpha = lambda in sklearn
])
pipe_ridge.fit(X_tr, y_tr)
print(f'Ridge (alpha=1): Test R2 = {r2_score(y_te,
pipe_ridge.predict(X_te)):.4f}')
print(f'Coefficients: {pipe_ridge.named_steps["ridge"].coef_.round(3)}')

# ── Method 2: RidgeCV — automatic lambda selection ───────────

# RidgeCV uses efficient leave-one-out CV internally (O(n) per alpha)
# Much faster than manual KFold cross-validation
alphas = [Link](-3, 4, 50) # 50 values from 0.001 to 10000

pipe_cv = Pipeline([
('scaler', StandardScaler()),
('ridgecv', RidgeCV(
alphas=alphas,
cv=5, # 5-fold CV (default is LOO if cv=None)
scoring='r2', # optimise cross-validated R2
store_cv_values=True # keep CV scores for plotting
))
])
pipe_cv.fit(X_tr, y_tr)

best_alpha = pipe_cv.named_steps['ridgecv'].alpha_
print(f'\nRidgeCV best alpha: {best_alpha:.4f}')
print(f'Test R2 with best alpha: {r2_score(y_te, pipe_cv.predict(X_te)):.4f}')

# ── Method 3: Manual RidgeCV with coefficient path ───────────

scaler = StandardScaler()
X_tr_s = scaler.fit_transform(X_tr)

AI/ML Placement Prep Book | Part II — Regularisation

ML Placement Prep — Chapter 5: Ridge Regression (L2) 16

X_te_s = [Link](X_te)

alphas_grid = [Link](-3, 4, 100)

coef_paths = [] # track coefficients for each alpha
cv_scores = [] # track cross-validated R2

for alpha in alphas_grid:

model = Ridge(alpha=alpha)
# Cross-validated R2
cv = cross_val_score(model, X_tr_s, y_tr, cv=5, scoring='r2')
cv_scores.append([Link]())
# Coefficient values at this alpha
[Link](X_tr_s, y_tr)
coef_paths.append(model.coef_.copy())

coef_paths = [Link](coef_paths) # shape (n_alphas, n_features)

best_idx = [Link](cv_scores)
best_alpha_manual = alphas_grid[best_idx]

# ── Plot: regularisation path and CV scores ───────────────────

fig, (ax1, ax2) = [Link](1, 2, figsize=(14, 5))

# Plot 1: Coefficient regularisation path

for j in range(n):
[Link](alphas_grid, coef_paths[:, j], lw=1.5,
label=f'coef {j+1}')
[Link](best_alpha_manual, color='black', lw=2,
linestyle='--', label=f'Best alpha={best_alpha_manual:.3f}')
ax1.set_xlabel('lambda (log scale)')
ax1.set_ylabel('Coefficient value')
ax1.set_title('Ridge Regularisation Path')
[Link](fontsize=7, loc='upper right')
[Link](True, alpha=0.3)

# Plot 2: Cross-validated R2 vs lambda

[Link](alphas_grid, cv_scores, 'b-', lw=2)
[Link](best_alpha_manual, color='black', lw=2,
linestyle='--', label=f'Best alpha={best_alpha_manual:.3f}')
ax2.set_xlabel('lambda (log scale)')
ax2.set_ylabel('Cross-validated R2')
ax2.set_title('Ridge: CV Score vs Lambda')
[Link](); [Link](True, alpha=0.3)

[Link]('Ridge Regression: Regularisation Path and CV Curve', fontsize=13)

plt.tight_layout()
[Link]('ridge_cv_path.png', dpi=150, bbox_inches='tight')
[Link]()

# ── Final model comparison ────────────────────────────────────

print('\n=== Final Model Comparison ===')
for name, mdl in [('OLS', LinearRegression()), ('Ridge (best alpha)',
Ridge(alpha=best_alpha_manual))]:
[Link](X_tr_s, y_tr)
yp = [Link](X_te_s)
print(f'{name:25s}: Test R2={r2_score(y_te,yp):.4f}, Max
coef={[Link](mdl.coef_).max():.2f}')

AI/ML Placement Prep Book | Part II — Regularisation

ML Placement Prep — Chapter 5: Ridge Regression (L2) 17

5.11 Interview Q&A — Chapter 5

Q: What is Ridge Regression and what problem does it solve?
A: Ridge Regression is a regularised extension of linear regression that adds an L2 penalty
(λΣθj²) to the OLS cost function: J_Ridge = (1/m)Σ(y−ŷ)² + λΣθj². It solves two related problems:
(1) Overfitting: OLS minimises training MSE without any constraint on coefficient magnitude,
producing large coefficients that fit noise. Ridge penalises large coefficients, shrinking them
toward zero and producing a model that generalises better to test data. (2)
Multicollinearity/singular XᵀX: OLS requires inverting XᵀX, which fails when features are
perfectly correlated or when p > n. Ridge adds λI to XᵀX before inverting, guaranteeing the
matrix is always invertible for any λ > 0.

Q: Derive the Ridge closed-form solution.

A: Start with J_Ridge(θ) = (1/m)(Xθ-y)ᵀ(Xθ-y) + λθᵀθ. Take the gradient: ∇J_Ridge =
(2/m)Xᵀ(Xθ-y) + 2λθ. Set to zero: (2/m)Xᵀ(Xθ-y) + 2λθ = 0. Multiply by m/2: XᵀXθ - Xᵀy + mλθ =
0. Collect θ: (XᵀX + mλ I)θ = Xᵀy. Solve: θ* = (XᵀX + mλ I) ⁻¹ Xᵀy. In the convention where
sklearn’s alpha = mλ (absorbing the 1/m factor): θ* = (XᵀX + λI)⁻¹Xᵀy. This is the Ridge Normal
Equation — identical to OLS except λI is added to XᵀX before inverting.

Q: Why does adding λI to XᵀX guarantee invertibility?

A: XᵀX is a positive semi-definite (PSD) matrix, so all its eigenvalues are non-negative:
λ_i(XᵀX) ≥ 0. If XᵀX is singular, at least one eigenvalue equals zero. The eigenvalues of (XᵀX +
λI) are λ_i(XᵀX) + λ. Since λ > 0, every eigenvalue of (XᵀX + λI) is at least λ > 0. A matrix with all
positive eigenvalues is strictly positive definite, which implies it is invertible (its determinant, the
product of eigenvalues, is strictly positive). Therefore (XᵀX + λI) is always invertible for any λ >
0, regardless of whether XᵀX is singular.

Q: Why does Ridge never set coefficients to exactly zero while Lasso does?
A: The difference comes from the gradient of each penalty at zero. The Ridge penalty gradient
is ∂(λθj²)/∂θj = 2λθj, which approaches zero as θj → 0. There is no gradient force holding θj at
exactly zero — the MSE gradient (which is non-zero at θj=0 unless the feature is irrelevant)
pulls θj away from zero. Result: all coefficients settle at some non-zero value. The Lasso
penalty gradient is ∂(λ|θj|)/∂θj = λ·sign(θj), which has magnitude λ everywhere near zero (it is
constant, not shrinking). This constant force toward zero is strong enough to overwhelm the
MSE gradient for small coefficients, pinning them exactly at zero. Geometrically: the L2 ball is
smooth (no corners); the L1 diamond has corners at the axes where sparsity occurs.

Q: Explain the bias-variance trade-off in Ridge Regression as λ varies.

A: At λ = 0: Ridge equals OLS, which is unbiased (Bias = 0). But if features are correlated or
the model has many parameters, OLS has high variance — different training sets give very

AI/ML Placement Prep Book | Part II — Regularisation

ML Placement Prep — Chapter 5: Ridge Regression (L2) 18

different coefficients, and test MSE is high. As λ increases: Bias increases (predictions are
systematically pulled toward zero away from the true values), but Variance decreases
(coefficients are more stable across training sets). The total test error = Bias² + Variance +
noise has a U-shape in λ. The optimal λ minimises this sum. For datasets with strong
multicollinearity, the OLS variance is enormous and even a small λ causes a large variance
reduction with a small bias increase, giving dramatically better test performance. Cross-
validation finds the optimal λ empirically.

Q: What is weight decay in deep learning and how does it relate to Ridge Regression?
A: Weight decay adds a penalty (λ/2)Σwij² to the neural network’s training loss, causing the
gradient update to include an extra term: w := w(1-αλ) - α·∂L_data/∂w. The factor (1-αλ)
multiplies each weight before the gradient step, so weights decay toward zero at every update.
For linear models with SGD, weight decay and L2 regularisation are exactly equivalent:
minimising MSE + (λ/2)Σw² produces the Ridge solution. For neural networks, the principle is
the same but the optimiser matters: with vanilla SGD, L2 regularisation and weight decay are
equivalent. With Adam, they are NOT equivalent because Adam scales the gradient, also
scaling the regularisation gradient. AdamW decouples the weight decay from the gradient,
restoring the equivalence and is the modern standard.

Q: How do you choose the regularisation parameter λ for Ridge Regression?

A: Always use cross-validation on a log-spaced grid. Steps: (1) Choose a grid of candidate λ
values spanning several orders of magnitude, e.g., [Link](-3, 4, 50) gives 50 values from
0.001 to 10,000. (2) For each candidate, perform K-fold CV (K=5 or 10): train on K-1 folds,
evaluate CV-MSE on the held-out fold, average across K folds. (3) Select the λ with the lowest
average CV-MSE. Optionally, apply the one-standard-error rule: select the largest λ within one
standard error of the minimum (preferring a simpler model). (4) Retrain on the full training set
using the chosen λ. (5) Evaluate final performance on the held-out test set. In sklearn: use
RidgeCV(alphas=[Link](-3,4,50), cv=5) which automates steps 1-4 efficiently.

Q: Why must features be scaled before applying Ridge Regression?

A: The Ridge penalty is λΣθj². The magnitude of θj depends on the scale of feature xj: if xj is
measured in kilometres, θj is in units of (price per km), and if measured in metres, θj is 1000×
smaller. Penalising both coefficients with the same λθj² is unfair: the coefficient for the km-
measured feature is penalised 1,000,000× more than the coefficient for the metre-measured
feature, despite measuring the same relationship. After StandardScaling (zero mean, unit
variance), all features are on the same scale, so the penalty treats all coefficients equally.
Without scaling, the regularisation is dominated by whichever feature happens to have the
largest scale, producing biased, unpredictable results. OLS is scale-invariant (scale changes
cancel out in (XᵀX)⁻¹Xᵀy), but Ridge is not.

Chapter 5 Summary — What You Must Know

AI/ML Placement Prep Book | Part II — Regularisation

ML Placement Prep — Chapter 5: Ridge Regression (L2) 19

Ridge cost: J = (1/m)Σ(y-ŷ)² + λΣθj² (θ0 not penalised). Closed-form: θ* = (XᵀX + λI) ⁻¹Xᵀy. λI
guarantees invertibility: adds λ to every eigenvalue of XᵀX, all become > 0. Geometric view: L2
ball (circle) has no corners → solutions never on axes → no exact zeros. Bias↑ as λ↑;
Variance↓ as λ↑; optimal λ found by cross-validation on log grid. Ridge never produces exact
zeros (unlike Lasso). Weight decay in deep learning = L2 regularisation = Ridge. Feature
scaling is mandatory. Use RidgeCV for automatic λ selection in sklearn.

AI/ML Placement Prep Book | Part II — Regularisation