STAT452

Ridge and LASSO Regression

Hoàng Văn Hà

University of Science, VNU - HCM
hvha@[Link]

V. H. Hoang Ridge and LASSO Regression 1 / 30

Outline

1 Introduction

2 Ridge Regression

3 LASSO Regression

4 Ridge and LASSO Regression in R

V. H. Hoang Ridge and LASSO Regression 2 / 30

 Introduction

Limitations of Ordinary Least Squares (OLS)

Ordinary Least Squares (OLS) is widely used in linear regression for estimating model
parameters, but it suffers from several important limitations, especially in
high-dimensional or noisy data settings:
Multicollinearity:
When predictor variables are highly correlated, the design matrix becomes nearly
singular.
This leads to unstable and highly sensitive coefficient estimates.
Small changes in data can cause large fluctuations in parameter values.
High Variance:
OLS minimizes residual sum of squares, which may lead to overfitting in presence of
many predictors.
As the number of predictors (p) approaches or exceeds the number of observations
(n), the model’s predictions become highly variable.
Poor generalization to unseen data.
Lack of Variable Selection:
OLS includes all predictors regardless of their relevance.
Irrelevant or redundant predictors increase model complexity and reduce interpretability.
Low Interpretability:
With many predictors, it becomes difficult to explain how each variable influences the
response.
Difficult to identify the most significant predictors.

V. H. Hoang Ridge and LASSO Regression 4 / 30

 Introduction

Ridge Regression

Ridge Regression (also known as L2 regularization) is a technique designed to address

overfitting and multicollinearity in linear regression by adding a penalty term to the loss
function.
Regularization Term:
Adds λ pj=1 βj2 to the residual sum of squares.
P

λ is a tuning parameter that controls the strength of the penalty.

Shrinkage Effect:
Coefficient estimates are shrunk toward zero but never exactly zero.
Reduces the impact of less informative predictors.
Reduced Variance:
By shrinking coefficients, Ridge lowers the model variance.
Leads to better generalization and prediction accuracy on unseen data.
Limitation – No Variable Selection:
All predictors remain in the model regardless of their significance.
Less interpretable when dealing with high-dimensional data.

V. H. Hoang Ridge and LASSO Regression 5 / 30

 Introduction

LASSO Regression

LASSO (Least Absolute Shrinkage and Selection Operator) is a regularization technique

using L1 penalty, promoting sparsity in the model.
Regularization Term:
Adds λ pj=1 |βj | to the loss function.
P

Encourages some coefficients to become exactly zero.

Variable Selection:
Performs automatic feature selection by excluding irrelevant predictors.
Enhances model interpretability and simplifies the final model.
Comparison with Ridge:
Ridge shrinks coefficients continuously; LASSO can shrink some to zero.
Ridge is better when all predictors are relevant; LASSO excels when only a subset
matters.
Limitation:
May behave unstably when predictors are highly correlated.
May select only one variable from a group of correlated ones.

V. H. Hoang Ridge and LASSO Regression 6 / 30

 Introduction

OLS vs. Ridge vs. Lasso

Ordinary Least Squares minimizes:

n 
X 2
yi − β̂0 − β̂1 xi1 − · · · − β̂p xip
i=1

Ridge Regression minimizes:

n  2 p
X X
yi − β̂0 − β̂1 xi1 − · · · − β̂p xip subject to β̂j2 ≤ t
i=1 j=1

Lasso Regression minimizes:

n  2 p
X X
yi − β̂0 − β̂1 xi1 − · · · − β̂p xip subject to |β̂j | ≤ t
i=1 j=1

Note: No constraint is placed on the magnitude of the intercept β̂0 .

V. H. Hoang Ridge and LASSO Regression 7 / 30

 Introduction

Geometric Illustration of Ridge and Lasso Estimates

Ellipses are contours of the loss

function:
n
X
(yi − β̂0 − β̂1 xi1 − β̂2 xi2 )2
i=1

Centered at the OLS solution:

(β̂1,OLS , β̂2,OLS ).
Left: Ellipse intersects the circle
constraint (L2 ) → Ridge estimate.
Right: Ellipse intersects the diamond
constraint (L1 ) → LASSO estimate.

V. H. Hoang Ridge and LASSO Regression 8 / 30

 Introduction

Equivalent Forms of Ridge and Lasso

By the Lagrange multiplier method, minimizing the residual sum of squares under
constraints:
n p p
X X X
(yi − β̂0 − β̂1 xi1 − · · · − β̂p xip )2 subject to β̂j2 ≤ t or |β̂j | ≤ t,
i=1 j=1 j=1

is equivalent to the penalized (regularized) forms:

Ridge Regression, minimizing:

n p
X X
(yi − β̂0 − β̂1 xi1 − · · · − β̂p xip )2 + λ β̂j2 .
i=1 j=1

Lasso, minimizing:
n p
X X
(yi − β̂0 − β̂1 xi1 − · · · − β̂p xip )2 + λ |β̂j |.
i=1 j=1

V. H. Hoang Ridge and LASSO Regression 9 / 30

 Ridge Regression

Ridge Regression: Motivation

Ridge regression addresses limitations of OLS, especially when predictors are highly
correlated or p > n.
OLS estimates can have large variance and unstable coefficients.
Ridge introduces a penalty term that shrinks coefficients towards zero, reducing
variance and improving prediction.
It provides biased but lower-variance estimators.

V. H. Hoang Ridge and LASSO Regression 11 / 30

 Ridge Regression

Bias-Variance Trade-Off

The Mean Squared Error (MSE) is decomposed as:

MSE = Bias2 + Variance.
Increasing model complexity reduces bias but increases variance.
Ridge finds a balance: it slightly increases bias but significantly reduces variance.

V. H. Hoang Ridge and LASSO Regression 12 / 30

 Ridge Regression

Standardization and Intercept in Ridge Regression

Before estimating the Ridge coefficients, two key assumptions are made:
Intercept is not penalized: The penalty applies only to the slope coefficients
(β1 , β2 , . . . ).
Predictors are standardized:
Ensures comparability between coefficients.
Prevents the penalization from disproportionately affecting variables with larger scales.
Standardization transforms the data as follows:
n
1X xij
β0 = ȳ = yj , x̄ij = q P .
n j=1 1 n
− x̄j )2
n j=1 (xij

where ȳ , x̄ij , and x̄j are the standardized values.

V. H. Hoang Ridge and LASSO Regression 13 / 30

 Ridge Regression

Mathematical Formulation

Ridge regression solves the following penalized optimization problem:

  2 

X n p p 

X X
β̂ridge = arg min yi − (β0 + βj xij ) + λ βj2 .
β  i=1 
j=1 j=1 

The second term is the `2 penalty.

λ controls the strength of shrinkage.
Intercept β0 is typically not penalized.

V. H. Hoang Ridge and LASSO Regression 14 / 30

 Ridge Regression

Mathematical Formulation

In matrix notation:
 
1
β̂ridge = arg min ky − X βk2 + λkβk2 .
β 2
The closed-form solution is:

β̂ridge = (X T X + λI )−1 X T y .

Adding λI ensures that the matrix is invertible even under multicollinearity.

V. H. Hoang Ridge and LASSO Regression 15 / 30

 Ridge Regression

Effect of Regularization on X T X

When X T X is nearly singular due to multicollinearity, inversion is unstable.

Adding λI stabilizes the inversion process.

V. H. Hoang Ridge and LASSO Regression 16 / 30

 Ridge Regression

Significance of λ

λ is the tuning parameter controlling shrinkage.

λ = 0 gives OLS; λ → ∞ shrinks coefficients to zero.
Chosen using cross-validation to minimize prediction error.
Affects bias-variance tradeoff and effective degrees of freedom.

V. H. Hoang Ridge and LASSO Regression 17 / 30

 Ridge Regression

Effective Degrees of Freedom

In Ridge, degrees of freedom is no longer p.

Defined as:
p
X dj2
df (λ) = tr[(X T X + λI )−1 X T X ] = .
j=1
dj2+λ

dj are singular values of X .

As λ → ∞, df (λ) → 0.

V. H. Hoang Ridge and LASSO Regression 18 / 30

 Ridge Regression

Cross-Validation for Optimal λ

To select the optimal shrinkage parameter λ, we use cross-validation:
Try a range of λ values on training folds.
Compute prediction error on validation folds.
Choose λ that minimizes average validation error.

Optimal λ found: λ ≈ 0.404

V. H. Hoang Ridge and LASSO Regression 19 / 30
 Ridge Regression

Ridge Estimates Are Biased but Have Smaller Variance

Recall OLS estimate for β = (β0 , β1 , . . . , βp )T is:

β̂OLS = (X T X )−1 X T Y .

Ridge estimate is:

β̂ridge = (X T X + λIp )−1 X T Y .

Assumes X is standardized: mean 0 and variance 1 for each predictor.

Expected value of Ridge estimate:

E[β̂ridge ] = (Ip + λX T X )−1 β 6= β.

⇒ Ridge estimate is biased.

If predictors are standardized and uncorrelated:
1
β̂j,λ,Ridge = β̂j,OLS
1+λ
Variance of Ridge estimate is smaller than that of OLS, especially effective when
data has multicollinearity.

V. H. Hoang Ridge and LASSO Regression 20 / 30

 Ridge Regression

Why Ridge Shrinks But Keeps All Variables

Pp
Ridge Regression uses an L2 penalty: λ j=1 βj2
The penalty discourages large coefficients by shrinking them towards zero.
However, L2 norm does not allow coefficients to become exactly zero.

Geometric Intuition
The constraint region for Ridge is a ball (circle or sphere).
Ellipses of residuals (RSS contours) typically touch the constraint at non-zero values.
Hence, all predictors are retained in the model with reduced magnitudes.

V. H. Hoang Ridge and LASSO Regression 21 / 30

 LASSO Regression

Mathematical Formulation

LASSO minimizes the following objective:

 

  2 

X n p p 

X X
β̂lasso = arg min y − β + β x + λ |β |
 
 j  0 j ij  j
β  
 i=1
 j=1 j=1 


Interpretation:

λ is the shrinkage parameter.

P
L1 penalty: kβk1 = |βj |.
Different from Ridge (which uses L2 penalty).
LASSO can shrink some coefficients exactly to zero.

V. H. Hoang Ridge and LASSO Regression 23 / 30

 LASSO Regression

Matrix Formulation

LASSO can also be expressed as:

 
1
β̂lasso = arg min ky − X βk2 + λkβk1
β 2

Provides a unique solution when X T X is full rank.

No closed-form solution, because the constraint given by the L1 penalty is in
absolute value, which cannot be differentiated.
The solutions for the lasso problem are nonlinear in yi because the constraint has a
non-smooth nature.

V. H. Hoang Ridge and LASSO Regression 24 / 30

 LASSO Regression

Properties of Lasso Estimates

No closed-form formula for the Lasso estimates

Also biased (toward 0)
Smaller variance than OLS estimates
Does not perform as well as Ridge when data have multicollinearity problem
Greatest advantage of Lasso: Sparsity

V. H. Hoang Ridge and LASSO Regression 25 / 30

 LASSO Regression

Sparsity of Lasso Estimates

In a model with many predictors:

Y = β0 + β1 X1 + · · · + βp Xp + ε.

We may believe many of the βj ’s are actually 0.

Hence, we seek a set of sparse solutions.
Lasso estimates will set some coefficients exactly equal to 0 when λ is large (or t is
small).

So the LASSO will perform model selection for us!

V. H. Hoang Ridge and LASSO Regression 26 / 30

 LASSO Regression

How to Choose λ?

We need a disciplined way of choosing λ

Obviously want to choose λ that minimizes the mean squared error
This issue is part of the bigger problem of variable selection

V. H. Hoang Ridge and LASSO Regression 27 / 30

 LASSO Regression

Choosing λ Using Cross-Validation

A good model should predict well on new data

Split data into two parts — training data and test data
For each λ:
Use the training set to train the model
Use the model to predict values in the test set
Compute the root mean square error (RMSE):
s X
RMSE = (yi − ŷi )2 /n.
test data

Choose the λ that has the smallest RMSE

Randomly select training/test sets. Optionally, use multiple splits and average the
RMSEs.

V. H. Hoang Ridge and LASSO Regression 28 / 30

 Ridge and LASSO Regression in R

Ridge Regression in R

See R-Markdown document . . .

V. H. Hoang Ridge and LASSO Regression 30 / 30