Least Squares Estimation and Regularization

A simple illustration using our class dataset

Lecture annotations for NEU BA64

1. The Linear Model

To further address the poor performance on the homework involving estimating model parameters
or model coefficients for a linear regression model, I present the following further illustration and
embed afterwards the notion of regularization. Recall, how I mentioned in class, that a model can
overfit, i.e, memorize the examples and perform poorly in a general class of the problem. As a
general rule, when you learn about a model, one should ask, how can I prevent overfitting in this
model?
Ok, let’s begin by going over how a linear model relates a numeric output ŷ to three features
from our example dataset:
ŷi = β0 + β1 hi + β2 wi + β3 gi ,
where:

• hi = height (cm) for the ith datapoint

• wi = weight (lbs) for the ith datapoint

• gi = GPA for the ith datapoint

• yi = actual class label (0 for Group 2, 1 for Group 1) for the ith datapoint

Goal. Find the coefficients β0 , β1 , β2 , β3 that make ŷi close to yi for all i.

2. The Quadratic (Least Squares) Loss Function

To measure how far our predictions are from the true labels, we use the sum of squared errors:
n n
X X 2
(yi − ŷi )2 =

E(β) = yi − (β0 + β1 hi + β2 wi + β3 gi ) .
i=1 i=1

Minimizing this quadratic function gives the least squares solution.

3. Normal Equation (Analytic Solution)

In matrix form:
y = Xβ + ϵ,

1
where      
1 h1 w1 g1 β y1
   0  
1 h2 w2 g2  β   y2 
 1
X = . . , β =  , y =  . .
   
 .. .. .. ..  β2   .. 
 . . 
    
1 hn wn gn β3 yn
To minimize E(β), set its derivative to zero:

∂E
= −2X⊤ (y − Xβ) = 0.
∂β
Solving gives the normal equation:

β = (X⊤ X)−1 X⊤ y .

4. Worked Example (3 Data Points)

We use three samples from our reference dataset:

h w g y

160 120 1.9 0

178 165 3.0 1
182 210 1.7 0

So,    
1 160 120 1.9 0
   
X=
1 178 165 3.0,
 y=
1 .

1 182 210 1.7 0

Step 1: Compute X⊤ X.
 
3 520 495 6.6
 
520 93248 87590
1198.4
X⊤ X = 
 

495 87590 75625 1135.5
 
6.6 1198.4 1135.5 14.9

Step 2: Compute X⊤ y.  
1
 
178
⊤
X y= .
 
165
 
3.0

2
Step 3: Solve for β.
β = (X⊤ X)−1 X⊤ y.
For simplicity, suppose we compute (numerically):
 
−4.5
 
0.015
β≈ .
 
0.005
 
0.4

Interpretation. - β0 = −4.5 → base level prediction. - β1 = 0.015 → as height increases,

predicted probability of Group 1 slightly increases. - β2 = 0.005 → weight contributes less strongly.
- β3 = 0.4 → GPA has the largest effect on group membership.
The model is:
ŷ = −4.5 + 0.015h + 0.005w + 0.4g.

5. Regularization in Least Squares Form

When data are noisy or correlated (height and weight often are), (X⊤ X) may be nearly singular
— leading to unstable coefficients. Regularization corrects this.

(a) L2 Regularization (Ridge Regression)

Add a penalty proportional to the square of the coefficients:
X X
EL2 = (yi − ŷi )2 + λ βj2 .
i j

This leads to a modified normal equation:

β = (X⊤ X + λI)−1 X⊤ y .

Interpretation. - λ shrinks all coefficients towards zero. - Controls overfitting by discouraging

large β values. - Ensures (X⊤ X + λI) is always invertible.

(b) L1 Regularization (Lasso Regression)

Add a penalty proportional to the absolute values:
X X
EL1 = (yi − ŷi )2 + λ |βj |.
i j

This does not have a closed-form solution but can be minimized by iterative coordinate descent.

Effect. - Some coefficients become exactly zero. - Acts like feature selection: removes less useful
variables.

3
6. Geometric View (Intuition)
- The least-squares solution chooses β at the minimum of a smooth quadratic bowl (error surface).
- The L2 penalty adds circular contours — pulling the solution toward the origin. - The L1 penalty
adds diamond-shaped contours — whose corners often touch the axes, setting some coefficients to
zero.

7. Summary Comparison

Type Penalty Effect Geometry

Ordinary LS None Fits data exactly Elliptic minimum

λ βj2
P
L2 (Ridge) Smooth shrinkage Circular contours
P
L1 (Lasso) λ |βj | Sparse (zero coefficients) Diamond contours

8. Final Thoughts
Starting from a simple model Y = βX and a quadratic loss, we can estimate coefficients using least
squares. When data are noisy or features are correlated, we use regularization: - L2 (Ridge) makes
the model more stable. - L1 (Lasso) makes the model simpler.
Both approaches balance two objectives:

Fit to data (small error) and Model simplicity (small coefficients).