Great — here are two fully worked numerical examples showing L2 (ridge) and L1 (lasso) regularization step-b
I use the same small dataset for both so you can compare shrinkage behavior.
DATASET (through-origin model y = b x, no intercept)
Points:
(x1,y1) = (1,1)
(x2,y2) = (2,3)
(x3,y3) = (3,5)
(x4,y4) = (4,7)
Compute the two key sums (we will reuse them):
- A = sum_{i=1}^4 x_i^2 = 1^2 + 2^2 + 3^2 + 4^2
= 1 + 4 + 9 + 16 = 30.
- C = sum_{i=1}^4 x_i y_i = 1*1 + 2*3 + 3*5 + 4*7
= 1 + 6 + 15 + 28 = 50.
The OLS (no regularization) solution through the origin is
b_OLS = C / A = 50 / 30 = 5/3 ≈ 1.6666667.
(We'll use this as baseline.)
---
1) L2 regularization (Ridge) — closed form
Objective (one coefficient): minimize sum (y_i - b x_i)^2 + λ b^2.
Differentiate and set to zero gives
b_ridge = C / (A + λ).
Example: λ = 10
- Denominator A+λ = 30 + 10 = 40.
- b_ridge = 50 / 40 = 5/4 = 1.25.
Compute predictions ŷ_i = b x_i and residuals r_i = y_i - ŷ_i:
- x1 = 1: ŷ1 = 1 * 1.25 = 1.25; r1 = 1 - 1.25 = -0.25.
r1^2 = 0.25^2 = 0.0625.
- x2 = 2: ŷ2 = 2 * 1.25 = 2.50; r2 = 3 - 2.50 = 0.50.
r2^2 = 0.5^2 = 0.25.
- x3 = 3: ŷ3 = 3 * 1.25 = 3.75; r3 = 5 - 3.75 = 1.25.
r3^2 = 1.25^2 = 1.5625.
- x4 = 4: ŷ4 = 4 * 1.25 = 5.00; r4 = 7 - 5.00 = 2.00.
r4^2 = 2^2 = 4.0000.
Sum of squared errors (SSE) = 0.0625 + 0.25 + 1.5625 + 4.0 = 5.875.
Penalty term = λ b^2 = 10 * (1.25^2) = 10 * 1.5625 = 15.625.
Total objective = SSE + penalty = 5.875 + 15.625 = 21.5.
(For comparison, with λ=0 we had b_OLS = 1.6666667 and SSE_OLS = 0.6666667.)
You can see ridge shrinks the coefficient from 1.6667 down to 1.25 when λ=10.
---
2) L1 regularization (Lasso) — soft-threshold closed form for single coefficient
Objective (one coefficient): minimize sum (y_i - b x_i)^2 + λ |b|.
For scalar b this reduces to (define A = sum x_i^2, C = sum x_i y_i):
minimize A b^2 - 2 C b + λ |b| =>
b_lasso = S( C/A, λ/(2A) )
where S(θ,γ) = sign(θ) * max(|θ| - γ, 0) (soft-thresholding).
Example: λ = 10
- C/A = 50/30 = 5/3 ≈ 1.6666667.
- Threshold γ = λ/(2A) = 10 / (2*30) = 10/60 = 1/6 ≈ 0.1666667.
Apply soft threshold:
b_lasso = sign(1.6667) * max(1.6667 - 0.1666667, 0) = 1 * 1.5 = 1.5.
Compute predictions and residuals:
- x1 = 1: ŷ1 = 1 * 1.5 = 1.5; r1 = 1 - 1.5 = -0.5; r1^2 = 0.25.
- x2 = 2: ŷ2 = 2 * 1.5 = 3.0; r2 = 3 - 3.0 = 0; r2^2 = 0.
- x3 = 3: ŷ3 = 3 * 1.5 = 4.5; r3 = 5 - 4.5 = 0.5; r3^2 = 0.25.
- x4 = 4: ŷ4 = 4 * 1.5 = 6.0; r4 = 7 - 6.0 = 1.0; r4^2 = 1.0.
SSE = 0.25 + 0 + 0.25 + 1.0 = 1.5.
Penalty term = λ |b| = 10 * 1.5 = 15.0.
Total objective = SSE + penalty = 1.5 + 15.0 = 16.5.
---
Quick summary / comparison (λ = 10)
- OLS (λ=0): b = 1.6666667, SSE = 0.6666667, objective = 0.6666667.
- Ridge (L2, λ=10): b = 1.25, SSE = 5.875, penalty = 15.625, objective = 21.5.
- Lasso (L1, λ=10): b = 1.5, SSE = 1.5, penalty = 15.0, objective = 16.5.
Observations:
- Ridge (L2) shrinks coefficients continuously toward 0 (here 1.6667 → 1.25 for λ=10).
- Lasso (L1) also shrinks but with soft-thresholding; for large enough λ it can set coefficients exactly to 0 (varia
Example: if λ >= 2C = 100 here, then b_lasso = 0.
- Numerical tradeoff: regularization increases SSE but penalizes complexity; the chosen λ balances fit vs coeffi
---
If you want, I can:
- Repeat the same demonstration for a two-coefficient example (shows coordinate-descent for lasso),
- Produce a PDF with these calculations (I can generate it right away), or
- Provide small Python code to compute ridge/lasso for any dataset and plot the coefficient path vs λ.
Which of those would you like next?