Introduction to Machine Learning II

Course 14 - Regularization
4th year Statistics and Data Science

Ayoub Asri

23 February 2026

Ayoub Asri Introduction to Machine Learning II 23 February 2026 1 / 82

 Section 1

Introduction

Ayoub Asri Introduction to Machine Learning II 23 February 2026 2 / 82

Learning goals

Overfitting
Motivation of regularization
First overview of techniques
Pattern of regularized ERM
formula

Ayoub Asri Introduction to Machine Learning II 23 February 2026 3 / 82

What is Regularization?

Methods that add inductive bias to model, usually some “low

complexity” priors (shrinkage and sparsity) to reduce overfitting and
get better bias-variance tradeoff

Explicit regularization: penalize explicit measure of model

complexity in ERM (e.g., L1/L2)
Implicit regularization: early stopping, data augmentation,
parameter sharing, dropout or ensembling (Mostly in Deep
Learning)
Structured regularization: structural prior knowledge over
groups of parameters or subnetworks (e.g., group lasso)

Ayoub Asri Introduction to Machine Learning II 23 February 2026 4 / 82

Recap: Overfitting

Occurs when model reflects noise or artifacts in training data

Model often does not generalize well (small train error, high test
error) – or at least works better on train than on test data

Ayoub Asri Introduction to Machine Learning II 23 February 2026 5 / 82

Example I: Overfitting

Data set: daily maximum ozone level in LA; n = 50

12 features: time (weekday, month); weather (temperature at
stations, humidity, wind speed); pressure gradient
Original data was subsetted, so it feels “high-dimensional” now
(low n in relation to p)
LM with all features (L2 loss)
MSE evaluation under 10 × 10 REP-CV

Model fits train data well, but generalizes poorly.

Ayoub Asri Introduction to Machine Learning II 23 February 2026 6 / 82

Example II: Overfitting

We train an Xgboost and a CART on the mtcars data

Both models are not regularized
And configured to make overfitting more likely

Model Train MSE Test MSE

XgbBoost 3.68 19.98
CART 0.00 10.21

(And we now switch back to the Ozone example. . . )

Ayoub Asri Introduction to Machine Learning II 23 February 2026 7 / 82

Avoiding Overfitting – Collect More Data

We explore our results for increased dataset size.

Fit slightly worsens, but test error decreases.

But: Often not feasible in practice.

Ayoub Asri Introduction to Machine Learning II 23 February 2026 8 / 82

Avoiding Overfitting – Reduce Complexity

We try the simplest model: a constant. So for L2 loss the mean of

y (i) .
We then increase complexity by adding one feature at a time.

NB: We added features in a specific (clever) order, so we cheated a bit.

Ayoub Asri Introduction to Machine Learning II 23 February 2026 9 / 82

Avoiding Overfitting – Optimize Less
Now: polynomial regression with temperature as single feature

d
X
f (x|θ) = θk · (xT )k
k=0

We set d = 15 to overfit to small data. To investigate early stopping,

we don’t analytically solve the OLS problem, but run GD stepwise.

We see: Early stopping GD can

improve results. NB: GD for
poly-regr usually needs many iters
before it starts to overfit, so we
used a very small training set.

Ayoub Asri Introduction to Machine Learning II 23 February 2026 10 / 82

Regularized Empirical Risk Minimization

We have contradictory goals:

maximizing fit (minimizing the train loss)

minimizing complexity of the model

We saw how we can include features in a binary fashion.

But we would rather control complexity on a continuum.

Ayoub Asri Introduction to Machine Learning II 23 February 2026 11 / 82

Regularized Empirical Risk Minimization

Common pattern:
n
X  
Rreg (f ) = Remp (f ) + λ · J(f ) = L y (i) , f (x(i) ) + λ · J(f )
i=1

J(f ): complexity penalty, roughness penalty or regularizer

λ ≥ 0: complexity control parameter

Ayoub Asri Introduction to Machine Learning II 23 February 2026 12 / 82

Regularized Empirical Risk Minimization

n
X  
Rreg (f ) = Remp (f ) + λ · J(f ) = L y (i) , f (x(i) ) + λ · J(f )
i=1

The higher λ, the more we penalize complexity

λ = 0: We just do simple ERM; λ → ∞: we don’t care about loss,
models become as “simple” as possible
λ is hard to set manually and is usually selected via CV

As for Remp , Rreg and J are often defined in terms of θ:

Rreg (θ) = Remp (θ) + λ · J(θ)

Ayoub Asri Introduction to Machine Learning II 23 February 2026 13 / 82

 Section 2

Ridge Regression

Ayoub Asri Introduction to Machine Learning II 23 February 2026 14 / 82

Learning goals

Regularized linear model

Ridge regression / L2 penalty
Understand parameter
shrinkage
Understand correspondence to
constrained optimization

Ayoub Asri Introduction to Machine Learning II 23 February 2026 15 / 82

Regularization in LM

Can also overfit if p large and n small(er)

OLS estimator requires full-rank design matrix
For highly correlated features, OLS becomes sensitive to random
errors in response, results in large variance in fit

We now add a complexity penalty to the loss:

n  2
y (i) − θ⊤ x(i)
X
Rreg (θ) = + λ · J(θ).
i=1

Ayoub Asri Introduction to Machine Learning II 23 February 2026 16 / 82

Ridge Regression / L2 Penalty

Intuitive measure of model complexity is deviation from 0-origin;

coefficients then have no or a weak effect. So we measure J(θ)
through a vector normalization, shrinking coefficients closer to 0.

n  2 p
y (i) − θ ⊤ x(i)
X X
θ̂ridge = arg min +λ θj2
θ
i=1 j=1

= arg min ∥y − Xθ∥22 + λ∥θ∥22

θ

Can still analytically solve this:

θ̂ridge = (X⊤ X + λI)−1 X⊤ y

We add positive entries along the diagonal “ridge” of X⊤ X

Ayoub Asri Introduction to Machine Learning II 23 February 2026 17 / 82
Ridge Regression / L2 Penalty

Let y = 3x1 − 2x2 + ϵ, ϵ ∼ N (0, 1). The true minimizer is

θ∗ = (3, −2)⊤ , with θ̂ridge = arg minθ ∥y − Xθ∥2 + λ∥θ∥2 .

With increasing regularization, θ̂ridge is pulled back to the origin

(contour lines show unregularized objective).

Ayoub Asri Introduction to Machine Learning II 23 February 2026 18 / 82

Ridge Regression / L2 Penalty

Contours of regularized objective for different λ values.

θ̂ridge = arg min ∥y − Xθ∥2 + λ∥θ∥2

θ

Green = true coefficients of the data generating process (DGP) and red
= ridge solution.

Ayoub Asri Introduction to Machine Learning II 23 February 2026 19 / 82

Ridge Regression / L2 Penalty
We understand the geometry of these 2 mixed components in our
regularized risk objective much better, if we formulate the optimization
as a constrained problem (see this as Lagrange multipliers in reverse).

n 
X 2
min y (i) − f (x(i) |θ)
θ
i=1

s.t. ∥θ∥22 ≤ t

NB: There is a bijective relationship between λ and t: λ ↑⇒ t ↓ and

vice versa.
Ayoub Asri Introduction to Machine Learning II 23 February 2026 20 / 82
Ridge Regression / L2 Penalty

Inside constraints perspective:

From origin, jump from
contour line to contour line
(better) until you become
infeasible, stop before.
We still optimize the Remp (θ),
but cannot leave a ball around
the origin.
Remp (θ) grows monotonically
if we move away from θ̂
(elliptic contours).
Solution path moves from
origin to border of feasible
region with minimal L2
distance.
Ayoub Asri Introduction to Machine Learning II 23 February 2026 21 / 82
Ridge Regression / L2 Penalty

Outside constraints
perspective: From θ̂, jump
from contour line to contour
line (worse) until you become
feasible, stop then.
So our new optimum will lie
on the boundary of that ball.
Solution path moves from
unregularized estimate to
feasible region of regularized
objective with minimal L2
distance.

Ayoub Asri Introduction to Machine Learning II 23 February 2026 22 / 82

Ridge Regression / L2 Penalty

Here we can see entire solution

path for ridge regression
Cyan contours indicate feasible
regions induced by different λs
Red contour lines indicate
different levels of the
unregularized objective
Ridge solution (red points)
gets pulled toward origin for
increasing λ

Ayoub Asri Introduction to Machine Learning II 23 February 2026 23 / 82

Example: Polynomial Ridge Regression

Consider y = f (x) + ϵ where the true (unknown) function is

f (x) = 5 + 2x + 10x2 − 2x3 (in red).
Let’s use a dth-order polynomial
d
X
f (x) = θ0 + θ1 x + · · · + θd xd = θj x j .
j=0

Using model complexity d = 10 overfits:

Ayoub Asri Introduction to Machine Learning II 23 February 2026 24 / 82

Example: Polynomial Ridge Regression

With an L2 penalty we can now select d “too large” but regularize our
model by shrinking its coefficients. Otherwise we have to optimize over
the discrete d.

Ayoub Asri Introduction to Machine Learning II 23 February 2026 25 / 82

 Section 3

Lasso Regression

Ayoub Asri Introduction to Machine Learning II 23 February 2026 26 / 82

Learning goals

Lasso regression / L1 penalty

Know that lasso selects
features
Support recovery

Ayoub Asri Introduction to Machine Learning II 23 February 2026 27 / 82

Lasso Regression

Another shrinkage method is the so-called lasso regression (least

absolute shrinkage and selection operator), which uses an L1 penalty
on θ:

n  2 p
y (i) − θ ⊤ x(i)
X X
θ̂lasso = arg min +λ |θj |
θ
i=1 j=1

= arg min (y − Xθ)⊤ (y − Xθ) + λ∥θ∥1

θ

Optimization is much harder now. Rreg (θ) is still convex, but in

general there is no analytical solution and it is non-differentiable.

Ayoub Asri Introduction to Machine Learning II 23 February 2026 28 / 82

Lasso Regression

Let y = 3x1 − 2x2 + ϵ, ϵ ∼ N (0, 1). The true minimizer is

θ∗ = (3, −2)T . LHS = L1 regularization; RHS = L2

With increasing regularization, θ̂lasso is pulled back to the origin, but

takes a different “route”. θ2 eventually becomes 0!

Ayoub Asri Introduction to Machine Learning II 23 February 2026 29 / 82

Lasso Regression

Contours of regularized objective for different λ values.

Green = true minimizer of the unregularized objective and red = lasso

solution.

Ayoub Asri Introduction to Machine Learning II 23 February 2026 30 / 82

Lasso Regression

Regularized empirical risk Rreg (θ1 , θ2 ) using squared loss for λ ↑. L1

penalty makes non-smooth kinks at coordinate axes more pronounced,
while L2 penalty warps Rreg toward a “basin” (elliptic paraboloid).

Ayoub Asri Introduction to Machine Learning II 23 February 2026 31 / 82

Lasso Regression

We can also rewrite this as a constrained optimization problem. The

penalty results in the constrained region to look like a diamond shape.

n 
X 2
min y (i) − f (x(i) |θ) subject to: ∥θ∥1 ≤ t
θ
i=1

The kinks in L1 enforce sparse solutions because “the loss contours first
hit the sharp corners of the constraint” at coordinate axes where
(some) entries are zero.

Ayoub Asri Introduction to Machine Learning II 23 February 2026 32 / 82

L1 and L2 Regularization With Orthonormal Design

For special case of orthonormal design X⊤ X = I we can derive a

closed-form solution in terms of θ̂OLS = (X⊤ X)−1 X⊤ y = X⊤ y:

θ̂lasso = sign(θ̂OLS )(|θ̂OLS | − λ)+ (sparsity)

Function S(θ, λ) := sign(θ)(|θ| − λ)+ is called soft thresholding

operator:
For |θ| ≤ λ it returns 0, whereas params |θ| > λ are shrunken
toward 0 by λ.

Ayoub Asri Introduction to Machine Learning II 23 February 2026 33 / 82

L1 and L2 Regularization With Orthonormal Design

Comparing this to θ̂Ridge under orthonormal design:

θ̂OLS
θ̂Ridge = (X⊤ X+λI)−1 X⊤ y = ((1+λ)I)−1 θ̂OLS = (no sparsity)
1+λ

Ayoub Asri Introduction to Machine Learning II 23 February 2026 34 / 82

Comparing Solution Paths for L1/L2

Ridge results in smooth solution path with non-sparse parameters

Lasso induces sparsity, but only for large enough λ

Ayoub Asri Introduction to Machine Learning II 23 February 2026 35 / 82

Support Recovery of Lasso
When can lasso select true support of θ, i.e., only the non-zero
parameters?
This an be formalized as a sign-consistency problem:
P (sign(θ̂) = sign(θ)) → 1 as n → ∞ (where sign(0) := 0)
Suppose the true DGP given a partition into subvectors
θ = (θ1 , θ2 ) is

Y = Xθ + ε = X1 θ1 + X2 θ2 + ε with ε ∼ (0, σ 2 I)

and only θ1 is non-zero. Let X1 denote the n × q matrix with the

relevant features and X2 the matrix of noise features. It can be shown
that θ̂lasso is sign consistent under an irrepresentable condition:
|(X⊤ ⊤ −1
2 X1 )(X1 X1 ) sign(θ1 )| < 1 (element-wise)

In fact, lasso can only be sign-consistent if this condition holds.

Intuitively, the irrelevant
Ayoub Asri
variables in X2 must not be23too
Introduction to Machine Learning II
correlated36 / 82
February 2026
 Section 4

Lasso vs. Ridge

Ayoub Asri Introduction to Machine Learning II 23 February 2026 37 / 82

Learning goals

Properties of ridge vs. lasso

Coefficient paths
What happens with correlated
features
Why we need feature scaling

Ayoub Asri Introduction to Machine Learning II 23 February 2026 38 / 82

Lasso vs. Ridge Geometry

n 
X 2
min y (i) − f (x(i) |θ) s.t. ∥θ∥pp ≤ t
θ
i=1

Ayoub Asri Introduction to Machine Learning II 23 February 2026 39 / 82

Lasso vs. Ridge Geometry

n 
X 2
min y (i) − f (x(i) |θ) s.t. ∥θ∥pp ≤ t
θ
i=1

In both cases (and for sufficiently large λ), the solution which
minimizes Rreg (θ) is always a point on the boundary of the
feasible region.
As expected, θ̂lasso and θ̂ridge have smaller parameter norms than
θ̂.
For Lasso, solution likely touches a vertex of constraint region.
Induces sparsity and is a form of variable selection.
For p > n: lasso selects at most n features.

Ayoub Asri Introduction to Machine Learning II 23 February 2026 40 / 82

Coefficient Paths and 0-Shrinkage

Example 1: Motor Trend Car Roads Test (mtcars)

We see how only lasso shrinks to exactly 0.

NB: No real overfitting here, as data is so low-dimensionial

Ayoub Asri Introduction to Machine Learning II 23 February 2026 41 / 82

Coefficient Paths and 0-Shrinkage

Example 2: High-dimensional correlated simulated data:

p = 50; n = 100

14
X
y = 10 · (x1 + x2 ) + 5 · (x3 + x4 ) + 1 · xj + ϵ
j=5

36/50 variables are noise; ϵ ∼ N (0, 1); x ∼ N (0, Σ); Σk,l = 0.7|k−l|

Ayoub Asri Introduction to Machine Learning II 23 February 2026 42 / 82

Regularization and Feature Scaling

Typically we omit θ0 in penalty J(θ) so that the “infinitely”

regularized model is the constant model (but can be
implementation-dependent).
Unregularized Linear Model has rescaling equivariance, if you
scale some features, can simply “anti-scale” coefficients and risk
does not change.
Not true for Regularized LM: if you down-scale features,
coefficients become larger to counteract. They are then penalized
stronger in J(θ), making them less attractive without any relevant
reason.
So: usually standardize features in regularized models,
whether linear or non-linear!

Ayoub Asri Introduction to Machine Learning II 23 February 2026 43 / 82

Regularization and Feature Scaling
Let the DGP be y = 5j=1 θj xj + ε for θ = (1, 2, 3, 4, 5)⊤ ,
P

ε ∼ N (0, 1)
Suppose x5 was measured in m but we change the unit to cm
(x̃5 = 100 · x5 ):

Method θ̂1 θ̂2 θ̂3 θ̂4 θ̂5 MSE

OLS 0.984 2.147 3.006 3.918 5.205 0.812
OLS Rescaled 0.984 2.147 3.006 3.918 0.052 0.812

Table 2: Parameter estimates and MSE comparison for OLS

Estimate θ̂5 gets scaled by 1/100 while other estimates and MSE
are invariant
Running ridge regression with λ = 10 on same data shows that
rescaling of x5 does not result in inverse rescaling of θ̂5 (everything
changes!)
Ayoub Asri Introduction to Machine Learning II 23 February 2026 44 / 82
Regularization and Feature Scaling

This is because θ̂5 now lives on small scale while L2 constraint

stays the same. Hence remaining estimates can “afford” larger
magnitudes.

Method θ̂1 θ̂2 θ̂3 θ̂4 θ̂5 MSE

Ridge 0.709 1.874 2.661 3.558 4.636 1.366
Ridge Rescaled 0.802 1.943 2.675 3.569 0.051 1.08

Table 3: Parameter estimates and MSE comparison for Ridge

For lasso, especially for very correlated features, we could

arbitrarily force a feature out of the model through a unit change.

Ayoub Asri Introduction to Machine Learning II 23 February 2026 45 / 82

Correlated Features: L1 vs. L2
Simulation with n = 100:

y = 0.2x1 + 0.2x2 + 0.2x3 + 0.2x4 + 0.2x5 + ϵ

x1 -x4 are independent, but x4 and x5 are strongly correlated.

L1 removes x5 early, L2 has similar coeffs for x4 , x5 for larger λ

Also called “grouping property”: for ridge highly correlated
features tend to have equal effects; lasso however “decides” what
to select
L1 selection is somewhat “arbitrary”
Ayoub Asri Introduction to Machine Learning II 23 February 2026 46 / 82
Correlated Features: L1 vs. L2

More detailed answer: The “random” decision is in fact a complex

deterministic interaction of data geometry (e.g., correlated structures),
the optimization method, and its hyperparamters (e.g., initialization).
The theoretical reason for this behavior relates to the convexity of the
penalties.
Considering perfectly co-linear features x4 = x5 in the last example, we
can obtain some more formal intuition for this phenomenon:

Because L2 penalty is strictly convex:

x4 = x5 ⇒ θ̂4,ridge = θ̂5,ridge (grouping property)

Ayoub Asri Introduction to Machine Learning II 23 February 2026 47 / 82

Correlated Features: L1 vs. L2

L1 penalty is not strictly convex. Hence, no unique solution exists

if x4 = x5 , and sum of coefficients can be arbitrarily allocated to
both features while remaining minimizers (no grouping property!):
For any solution θ̂4,lasso , θ̂5,lasso , equivalent minimizers are given by

θ̃4,lasso = s · (θ̂4,lasso + θ̂5,lasso )

and

θ̃5,lasso = (1 − s) · (θ̂4,lasso + θ̂5,lasso ) ∀s ∈ [0, 1]

Ayoub Asri Introduction to Machine Learning II 23 February 2026 48 / 82

Summary

Neither ridge nor lasso can be classified as better overall

Lasso can shrink some coefficients to zero, so selects features; ridge

usually leads to dense solutions, with smaller coefficients
Lasso likely better if true underlying structure is sparse
ridge works well if there are many (weakly) influential features
Lasso has difficulties handling correlated predictors; for high
correlation, ridge dominates lasso in performance
Lasso: for (highly) correlated predictors, usually an “arbitrary”
one is selected, with large coefficients, while the others are (nearly)
zeroed
Ridge: coefficients of correlated features are similar

Ayoub Asri Introduction to Machine Learning II 23 February 2026 49 / 82

 Section 5

Elastic Net and regularized GLMs

Ayoub Asri Introduction to Machine Learning II 23 February 2026 50 / 82

Learning goals

Compromise between L1 and

L2
Regularized logistic regression

Ayoub Asri Introduction to Machine Learning II 23 February 2026 51 / 82

Elastic Net as L1/L2 Combination

n
(y (i) − θ ⊤ x(i) )2 + λ1 ∥θ∥1 + λ2 ∥θ∥22
X
Relnet (θ) =
i=1

n  
(y (i) − θ ⊤ x(i) )2 + λ (1 − α)∥θ∥1 + α∥θ∥22
X
=
i=1

λ2 λ1 +λ2
where α = λ1 +λ2 , λ= 2

Ayoub Asri Introduction to Machine Learning II 23 February 2026 52 / 82

Elastic Net as L1/L2 Combination

n
(y (i) − θ ⊤ x(i) )2 + λ1 ∥θ∥1 + λ2 ∥θ∥22
X
Relnet (θ) =
i=1

n  
(y (i) − θ ⊤ x(i) )2 + λ (1 − α)∥θ∥1 + α∥θ∥22
X
=
i=1

2nd formula is simply more convenient to interpret hyperparameters; λ

controls how much we penalize, α sets the “L2-portion”

Correlated features tend to be either selected or zeroed out

together
Selection of more than n features possible for p > n

Ayoub Asri Introduction to Machine Learning II 23 February 2026 53 / 82

Simulated Example
5-fold CV with ntrain = 100 and 20 repetitions with ntest = 10000 for
setups:

y = x⊤ θ + ϵ; ϵ ∼ N (0, 0.12 ); x ∼ N (0, Σ); Σk,l = 0.8|k−l|

Lasso better for sparse features:

θ = (1, . . . , 1, 0, . . . , 0)
| {z } | {z }
5 495

Ridge better for dense features:

θ = (1, . . . , 1, 1, . . . , 1)
| {z }
500

Ayoub Asri Introduction to Machine Learning II 23 February 2026 54 / 82

Simulated Example

LHS: ridge estimates of noise features hover around 0 while lasso/e-net

produce 0s.
RHS: ridge cannot perform variable selection compared to lasso/e-net.
Lasso more frequently ignores relevant features than e-net (longer tails
in violin plot).

Ayoub Asri Introduction to Machine Learning II 23 February 2026 55 / 82

Regularized Logistic Regression

Penalties can be added very flexibly to any model based on ERM

E.g.: L1- or L2-penalized logistic regression for high-dimensional
spaces and feature selection
Now: LR with polynomial features for x1 , x2 up to degree 7 and
L2 penalty on 2D “circle data” below

λ = 0: LR without penalty seems to overfit

λ = 0.0001: We get better
λ = 1: Fit looks pretty good

Ayoub Asri Introduction to Machine Learning II 23 February 2026 56 / 82

 Section 6

Other Regularizers

Ayoub Asri Introduction to Machine Learning II 23 February 2026 57 / 82

Learning goals

L1/L2 regularization induces

bias
Lq (quasi-)norm regularization
L0 regularization
SCAD and MCP

Ayoub Asri Introduction to Machine Learning II 23 February 2026 58 / 82

Ridge and Lasso are Biased Estimators

Although ridge and lasso have many nice properties, they are biased
estimators and the bias does not (necessarily) vanish as n → ∞.
For example, in the orthonormal case (X⊤ X = I) the bias of the lasso
is

E|θ̂j − θj | = 0

 if θj = 0
E|θ̂j − θj | ≈ θj if |θj | ∈ [0, λ]

E|θ̂ − θ | ≈ λ

if |θj | > λ
j j

To reduce the bias/shrinkage of regularized estimators various

penalties were proposed, a few of which we briefly introduce now.

Ayoub Asri Introduction to Machine Learning II 23 February 2026 59 / 82

LQ Regularization
Besides L1/L2 we could use any Lq (quasi-)norm penalty λ∥θ∥qq

Top: loss contours and L1/L2 constraints. Bottom: Constraints for Lq

norms j |θj |q .
P

For q < 1 penalty becomes non-convex but for q > 1 no sparsity is

achieved
Non-convex Lq has nice properties like oracle property:
consistent (+ asymetric unbiased) parameter estimation and
variable selection
Downside: non-convexity makes optimization even harder than L1
(no unique global minimum
Ayoub Asri
but multiple local minima)
Introduction to Machine Learning II 23 February 2026 60 / 82
L0 Regularization

X
Rreg (θ) = Remp (θ) + λ∥θ∥0 := Remp (θ) + λ |θj |0 .
j

Ayoub Asri Introduction to Machine Learning II 23 February 2026 61 / 82

L0 Regularization

X
Rreg (θ) = Remp (θ) + λ∥θ∥0 := Remp (θ) + λ |θj |0 .
j

L0 “norm” simply counts the nr of non-zero params

Induces sparsity more aggressively than L1, but does not shrink
AIC and BIC are special cases of L0
L0-regularized risk is not continuous or convex
NP-hard to optimize; for smaller n and p somewhat tractable,
efficient approximations are still current research

Ayoub Asri Introduction to Machine Learning II 23 February 2026 62 / 82

SCAD
Smoothly Clipped Absolute Deviations:
non-convex, γ > 2 controlls how fast penalty “tapers off”


 λ|θ| if |θ| ≤ λ
2γλ|θ|−θ2 −λ2

SCAD(θ|λ, γ) = if λ < |θ| < γλ
 2 2(γ−1)
 λ (γ+1)

2 if |θ| ≥ γλ

Lasso, quadratic, then

constant
Smooth
Contrary to lasso/ridge, SCAD
continuously relaxes
penalization rate as |θ|
increases above λ
Ayoub Asri Introduction to Machine Learning II 23 February 2026 63 / 82
MCP

Minimax Concave Penalty:

also non-convex; similar idea as SCAD with γ > 1

θ2
(
λ|θ| − 2γ , if |θ| ≤ γλ
MCP(θ|λ, γ) = 1 2
2 γλ , if |θ| > γλ

As with SCAD, MCP starts by applying same penalization rate as

lasso, then smoothly reduces rate to zero as |θ| ↑
Different from SCAD, MCP immediately starts relaxing the
penalization rate, while for SCAD rate remains flat until |θ| > λ
Both SCAD and MCP possess oracle property: they can
consistently select true model as n → ∞ while lasso may fail

Ayoub Asri Introduction to Machine Learning II 23 February 2026 64 / 82

Example: Comparing Regularizers

Let’s compare coefficient paths for lasso, SCAD, and MCP.

We simulate n = 100 samples from the following DGP:

y = x⊤ θ + ε, θ = (4, −4, −2, 2, 0, . . . , 0)⊤ ∈ R1500 , xj , ε ∼ N (0, 1)

Vertical lines mark optimal λ from 10CV.

Conclusion: Lasso underestimates true coefficients while SCAD/MCP
achieve unbiased estimation and better variable selection
Ayoub Asri Introduction to Machine Learning II 23 February 2026 65 / 82
 Other Ideas

Ayoub Asri Introduction to Machine Learning II 23 February 2026 66 / 82

 Section 7

Geometry of L2 Regularization

Ayoub Asri Introduction to Machine Learning II 23 February 2026 67 / 82

Learning goals

Approximate transformation of
unregularized minimizer to
regularized
Principal components of
Hessian influence where
parameters are decayed

Ayoub Asri Introduction to Machine Learning II 23 February 2026 68 / 82

Geometric Analaysis of L2 Regularization

Quadratic Taylor approximation of the unregularized objective

Remp (θ) around its minimizer θ̂:

1
R̃emp (θ) = Remp (θ̂) + ∇θ Remp (θ̂) · (θ − θ̂) + (θ − θ̂)⊤ H(θ − θ̂)
2

where H is the Hessian of Remp (θ) at θ̂

We notice:

First-order term is 0, because gradient must be 0 at minimizer

H is positive semidefinite, because we are at the minimizer

1
R̃emp (θ) = Remp (θ̂) + (θ − θ̂)⊤ H(θ − θ̂)
2

Ayoub Asri Introduction to Machine Learning II 23 February 2026 69 / 82

Geometric Analaysis of L2 Regularization / 2
The minimum of R̃emp (θ) occurs where ∇θ R̃emp (θ) = H(θ − θ̂) is 0.
Now we L2-regularize R̃emp (θ), such that
λ
R̃reg (θ) = R̃emp (θ) + ∥θ∥22
2
and solve this approximation of Rreg for the minimizer θ̂ridge :

∇θ R̃reg (θ) = 0

λθ + H(θ − θ̂) = 0

(H + λI)θ = H θ̂

θ̂ridge = (H + λI)−1 H θ̂

We see: minimizer of L2-regularized version is (approximately!)

transformation of minimizer of the unpenalized version.
Ayoub Asri Introduction to Machine Learning II 23 February 2026 70 / 82
Geometric Analaysis of L2 Regularization / 3

As λ approaches 0, the regularized solution θ̂ridge approaches θ̂.

What happens as λ grows?
Because H is a real symmetric matrix, it can be decomposed as
H = QΣQ⊤ , where Σ is a diagonal matrix of eigenvalues and Q
is an orthonormal basis of eigenvectors.

Ayoub Asri Introduction to Machine Learning II 23 February 2026 71 / 82

Geometric Analaysis of L2 Regularization / 4

Rewriting the transformation formula with this:

 −1
θ̂ridge = QΣQ⊤ + λI QΣQ⊤ θ̂
h i−1
= Q(Σ + λI)Q⊤ QΣQ⊤ θ̂

= Q(Σ + λI)−1 ΣQ⊤ θ̂

So: We rescale θ̂ along axes defined by eigenvectors of H.

The component of θ̂ that is associated with the j-th eigenvector of
σj
H is rescaled by factor of σj +λ , where σj is eigenvalue.

Ayoub Asri Introduction to Machine Learning II 23 February 2026 72 / 82

Geometric Analaysis of L2 Regularization / 5

First, θ̂ is rotated by Q⊤ , which we can interpret as projection of θ̂ on

rotated coord system defined by principal directions of H:

Ayoub Asri Introduction to Machine Learning II 23 February 2026 73 / 82

Geometric Analaysis of L2 Regularization / 6

σj
j-th (new) axis is rescaled by σj +λ before we rotate back.

Ayoub Asri Introduction to Machine Learning II 23 February 2026 74 / 82

Geometric Analaysis of L2 Regularization / 7

σ
j
Decay: σj +λ
Along directions where eigenvals of H are relatively large, e.g.,
σj ≫ λ, effect of regularization is small.
Components / directions with σj ≪ λ are strongly shrunken.
So: Directions along which parameters contribute strongly to
objective are preserved relatively intact.
In other directions, small eigenvalue of Hessian means that moving
in this direction will not decrease objective much.
For such unimportant directions, corresponding components of θ
are decayed away.

Ayoub Asri Introduction to Machine Learning II 23 February 2026 75 / 82

Geometric Analaysis of L2 Regularization / 8

Ayoub Asri Introduction to Machine Learning II 23 February 2026 76 / 82

 Section 8

Geometry of L1 Regularization

Ayoub Asri Introduction to Machine Learning II 23 February 2026 77 / 82

Learning goals

Approximate transformation of
unregularized minimizer to
regularized
Soft-Thresholding

Ayoub Asri Introduction to Machine Learning II 23 February 2026 78 / 82

L1-regularization
The L1-regularized risk of a model f (x|θ) is
X
Rreg (θ) = Remp (θ) + λ|θj |
j

and the (sub-)gradient is:

∇θ Remp (θ) + λ · sign(θ)

Unlike in L2, contribution to gradient doesn’t scale with θj

elements.
Again: quadratic Taylor approximation of Remp (θ) around its
minimizer θ̂, then regularize:

1 X
R̃reg (θ) = Remp (θ̂) + (θ − θ̂)T H(θ − θ̂) + λ|θj |
2 j

Ayoub Asri Introduction to Machine Learning II 23 February 2026 79 / 82

L1-regularization / 2
To cheat and simplify, we assume the H is diagonal, with Hj,j ≥ 0
Now R̃reg (θ) decomposes into sum over params θj (separable!):
X 1  X
2
R̃reg (θ) = Remp (θ̂) + Hj,j (θj − θ̂j ) + λ|θj |
j
2 j

We can minimize analytically:

( )
λ
θ̂lasso,j = sign(θ̂j ) max |θ̂j | − ,0
Hj,j

θ̂ + Hλj,j , if θ̂j < − Hλj,j


 j


= 0, if θ̂j ∈ [− Hλj,j , Hλj,j ]

 λ λ

θ̂j − Hj,j , if θ̂j > Hj,j

Shows how lasso (approx) transforms the normal minimizer

If Hj,j = 0 exactly, θ̂lasso,j = 0
Ayoub Asri Introduction to Machine Learning II 23 February 2026 80 / 82
L1-regularization / 3

λ
If 0 < θ̂j ≤ Hj,j or 0 > θ̂j ≥ − Hλj,j , the optimal value of θj (for the
regularized risk) is 0 because the contribution of Rreg (θ̂) to
Rreg (θ) is overwhelmed by the L1 penalty, which forces it to be 0.

Ayoub Asri Introduction to Machine Learning II 23 February 2026 81 / 82

L1-regularization / 4

λ
If 0 < Hj,j < θ̂j or 0 > − Hλj,j > θ̂j , the L1 penalty shifts the optimal
λ
value of θj toward 0 by the amount Hj,j .

Yellow dotted lines are limits from soft-thresholding

Therefore, the L1 penalty induces sparsity in the parameter vector.

Ayoub Asri Introduction to Machine Learning II 23 February 2026 82 / 82