Question 1: Numerical Problem: Ridge Regression

You are given the dataset:

𝒙 𝒚
𝟏 𝟐
𝟐 𝟓
𝟑 𝟏𝟎

𝒚 = 𝒘𝟏 𝒙 + 𝒘𝟐 𝒙 𝟐
Regularization parameter:

𝝀=𝟏
Find the optimal ridge regression weights.

Solution: Feature matrix:

𝟏 𝟏
𝑿 = [𝟐 𝟒]
𝟑 𝟗
Target vector:
𝟐
𝒚=[𝟓]
𝟏𝟎
the optimal ridge regression weights using:

𝒘 = (𝑿𝑻 𝑿 + 𝝀𝑰)−𝟏 𝑿𝑻 𝒚

Step 1: Compute 𝑿𝑻
𝟏 𝟐 𝟑
𝑿𝑻 = [ ]
𝟏 𝟒 𝟗

Step 2: Compute 𝑿𝑻 𝑿
𝟏 𝟏
𝑻 𝟏 𝟐 𝟑
𝑿 𝑿=[ ] [𝟐 𝟒]
𝟏 𝟒 𝟗
𝟑 𝟗
Compute elements:

(1,1):

𝟏𝟐 + 𝟐𝟐 + 𝟑𝟐 = 𝟏 + 𝟒 + 𝟗 = 𝟏𝟒
(1,2):
 𝟏(𝟏) + 𝟐(𝟒) + 𝟑(𝟗) = 𝟏 + 𝟖 + 𝟐𝟕 = 𝟑𝟔
(2,1) = 36

(2,2):

𝟏𝟐 + 𝟒𝟐 + 𝟗𝟐 = 𝟏 + 𝟏𝟔 + 𝟖𝟏 = 𝟗𝟖
So,
𝟏𝟒 𝟑𝟔
𝑿𝑻 𝑿 = [ ]
𝟑𝟔 𝟗𝟖
Step 3: Add Ridge Term (𝝀 = 𝟏)
𝟏𝟒 𝟑𝟔 𝟏 𝟎 𝟏𝟓 𝟑𝟔
𝑿𝑻 𝑿 + 𝝀𝑰 = [ ]+[ ]=[ ]
𝟑𝟔 𝟗𝟖 𝟎 𝟏 𝟑𝟔 𝟗𝟗
Step 4: Compute 𝑿𝑻 𝒚
𝟐
𝟏 𝟐 𝟑
𝑿𝑻 𝒚 = [ ][ 𝟓 ]
𝟏 𝟒 𝟗
𝟏𝟎
Compute:

First element:

𝟏(𝟐) + 𝟐(𝟓) + 𝟑(𝟏𝟎) = 𝟐 + 𝟏𝟎 + 𝟑𝟎 = 𝟒𝟐

Second element:

𝟏(𝟐) + 𝟒(𝟓) + 𝟗(𝟏𝟎) = 𝟐 + 𝟐𝟎 + 𝟗𝟎 = 𝟏𝟏𝟐

𝟒𝟐
𝑿𝑻 𝒚 = [ ]
𝟏𝟏𝟐
Step 5: Compute Inverse of
𝟏𝟓 𝟑𝟔
𝑨=[ ]
𝟑𝟔 𝟗𝟗
Determinant:

𝑫 = 𝟏𝟓(𝟗𝟗) − 𝟑𝟔(𝟑𝟔)
= 𝟏𝟒𝟖𝟓 − 𝟏𝟐𝟗𝟔 = 𝟏𝟖𝟗
Inverse:
𝟏 𝟗𝟗 −𝟑𝟔
𝑨−𝟏 = [ ]
𝟏𝟖𝟗 −𝟑𝟔 𝟏𝟓

Step 6: Compute Final Weights

𝒘 = 𝑨−𝟏 𝑿𝑻 𝒚
 𝟏 𝟗𝟗 −𝟑𝟔 𝟒𝟐
𝒘= [ ][ ]
𝟏𝟖𝟗 −𝟑𝟔 𝟏𝟓 𝟏𝟏𝟐

Multiply:

First weight:

𝟗𝟗(𝟒𝟐) − 𝟑𝟔(𝟏𝟏𝟐)
= 𝟒𝟏𝟓𝟖 − 𝟒𝟎𝟑𝟐 = 𝟏𝟐𝟔

Second weight:

−𝟑𝟔(𝟒𝟐) + 𝟏𝟓(𝟏𝟏𝟐)
= −𝟏𝟓𝟏𝟐 + 𝟏𝟔𝟖𝟎 = 𝟏𝟔𝟖
So,
𝟏 𝟏𝟐𝟔
𝒘= [ ]
𝟏𝟖𝟗 𝟏𝟔𝟖
𝟏𝟐𝟔 𝟐
𝒘𝟏 = =
𝟏𝟖𝟗 𝟑
𝟏𝟔𝟖 𝟖
𝒘𝟐 = =
𝟏𝟖𝟗 𝟗
Final Answer

𝟐
𝒘𝟏 =
𝟑
𝟖
𝒘𝟐 =
𝟗
Final Ridge Model

𝟐 𝟖
𝒚= 𝒙 + 𝒙𝟐
𝟑 𝟗
Question 2: You are given the following 1-D dataset:
𝒙 𝒚
−𝟐 𝟒. 𝟏
−𝟏 𝟏. 𝟐
𝟎 𝟎. 𝟏
𝟏 𝟏. 𝟑
𝟐 𝟑. 𝟗

1. Show that a simple linear regression model

𝒚 = 𝒘𝟎 + 𝒘𝟏 𝒙

is not appropriate for this dataset.

2. Perform feature expansion by mapping the input as:

𝝓(𝒙) = [𝟏, 𝒙, 𝒙𝟐 ]
Write the design matrix 𝑿.

3. Formulate the Ridge Regression objective function with regularization

parameter 𝝀.

4. Derive the closed-form ridge solution for this expanded feature space.

Solution:

1.

This forms a U-shaped curve, which indicates a quadratic relationship.

A linear model produces only a straight line and cannot capture curvature.

⇒ Simple l inear r egression i

2.

Feature Expansion

Given:

𝜙(𝑥) = [1, 𝑥, 𝑥 2 ]

Construct matrix 𝑋:
1 −2 4
1 −1 1
𝑋= 1 0 0
1 1 1
[1 2 4]

Target vector:
4.1
1.2
𝑦 = 0.1
1.3
[3.9]
3.

Ridge Regression Objective Function

Ridge regression minimizes:

𝐽(𝑤) =∥ 𝑦 − 𝑋𝑤 ∥2 + 𝜆 ∥ 𝑤 ∥2

Expanded form:
5

𝐽(𝑤) = ∑(𝑦𝑖 − (𝑤0 + 𝑤1 𝑥𝑖 + 𝑤2 𝑥𝑖2 ))2 + 𝜆(𝑤02 + 𝑤12 + 𝑤22 )

𝑖=1
4.

Closed-Form Ridge Solution

General Formula

𝑤 = (𝑋 𝑇 𝑋 + 𝜆𝐼)−1 𝑋 𝑇 𝑦

Step 1: Compute 𝑿𝑻 𝑿
Using summations:
5 0 10
𝑇
𝑋 𝑋=[0 10 0]
10 0 34

Step 2: Add Regularization

5+𝜆 0 10
𝑋 𝑇 𝑋 + 𝜆𝐼 = [ 0 10 + 𝜆 0 ]
10 0 34 + 𝜆

Step 3: Compute 𝑿𝑻 𝒚
10.6
𝑋 𝑇 𝑦 = [−0.3]
34.5

Step 4: Solve System

Solve for 𝒘𝟏

(10 + 𝜆)𝑤1 = −0.3

−0.3
𝑤1 =
10 + 𝜆

Solve for 𝒘𝟎 , 𝒘𝟐

(5 + 𝜆)𝑤0 + 10𝑤2 = 10.6

10𝑤0 + (34 + 𝜆)𝑤2 = 34.5

Determinant:

𝐷 = (5 + 𝜆)(34 + 𝜆) − 100
𝐷 = 𝜆2 + 39𝜆 + 70

Final Closed-Form Ridge Weights

15.4 + 10.6𝜆
𝑤0 =
𝜆2 + 39𝜆 + 70
−0.3
𝑤1 =
10 + 𝜆
 66.9 + 34.5𝜆
𝑤2 =
𝜆2 + 39𝜆 + 70

Final Ridge Model

𝑦̂ = 𝑤0 + 𝑤1 𝑥 + 𝑤2 𝑥 2