Scribd
Upload
4 views9 pages

? Tutorial Note On Gradient Descent

This tutorial explains Gradient Descent as an optimization algorithm for minimizing the Mean Squared Error (MSE) in Simple Linear Regression. It details the model setup, the loss function, and the process of updating parameters through calculated gradients. Key takeaways include the importance of the learning rate, the convex nature of the loss surface for linear regression, and the expected decrease in loss over epochs.

Uploaded by

wisdominnocent2004
Copyright
© All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
4 views9 pages

? Tutorial Note On Gradient Descent

This tutorial explains Gradient Descent as an optimization algorithm for minimizing the Mean Squared Error (MSE) in Simple Linear Regression. It details the model setup, the loss function, and the process of updating parameters through calculated gradients. Key takeaways include the importance of the learning rate, the convex nature of the loss surface for linear regression, and the expected decrease in loss over epochs.

Uploaded by

wisdominnocent2004
Copyright
© All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd

Tutorial Note on Gradient Descent (Using MSE Loss)

Introduction to Gradient Descent

Gradient Descent is an optimization algorithm used to minimize a loss (cost) function by


iteratively adjusting model parameters.

For this tutorial, we will use:

• Model: Simple Linear Regression

• Loss Function: Mean Squared Error (MSE)

Problem Setup – Linear Regression

We assume a simple linear model:

𝑦̂ = 𝑤𝑥 + 𝑏

Where:

• 𝑤= weight (slope)

• 𝑏= bias (intercept)

• 𝑦̂= predicted output

• 𝑦= actual output

Mean Squared Error (MSE) Loss Function

The MSE is defined as:


𝑛
1
𝐽(𝑤, 𝑏) = ∑( 𝑦𝑖 − 𝑦̂𝑖 )2
𝑛
𝑖=1

Substituting 𝑦̂𝑖 = 𝑤𝑥𝑖 + 𝑏:


𝑛
1
𝐽(𝑤, 𝑏) = ∑( 𝑦𝑖 − (𝑤𝑥𝑖 + 𝑏))2
𝑛
𝑖=1
Our goal is to minimize 𝐽(𝑤, 𝑏).

How Gradient Descent Works

Gradient Descent updates parameters using:


∂𝐽
𝑤: = 𝑤 − 𝛼
∂𝑤
∂𝐽
𝑏: = 𝑏 − 𝛼
∂𝑏

Where:

• 𝛼= learning rate
∂𝐽
• = partial derivative of loss w.r.t. weight
∂𝑤

∂𝐽
• = partial derivative of loss w.r.t. bias
∂𝑏

Deriving the Gradients (Using Chain Rule)

We now compute the derivatives step by step.

Step 1: Define the Loss


1
𝐽= ∑(𝑦 − (𝑤𝑥 + 𝑏))2
𝑛

Let:

𝑒 = 𝑦 − (𝑤𝑥 + 𝑏)

So:
1 2
𝐽= ∑𝑒
𝑛
Step 2: Partial Derivative w.r.t. 𝒘

Using chain rule:


∂𝐽 ∂𝐽 ∂𝑒
= ⋅
∂𝑤 ∂𝑒 ∂𝑤

First term:
∂ 2
(𝑒 ) = 2𝑒
∂𝑒

Since averaging over n:


∂𝐽 2
= 𝑒
∂𝑒 𝑛

Second term:

Recall:

𝑒 = 𝑦 − (𝑤𝑥 + 𝑏)
∂𝑒
= −𝑥
∂𝑤

Combine (Chain Rule)


∂𝐽 2
= 𝑒(−𝑥)
∂𝑤 𝑛
2
= − 𝑥(𝑦 − (𝑤𝑥 + 𝑏))
𝑛

Final form:

∂𝐽 2
= − ∑𝑥𝑖 (𝑦𝑖 − 𝑦̂𝑖 )
∂𝑤 𝑛
Step 3: Partial Derivative w.r.t. 𝒃

Using same method:


∂𝐽 ∂𝐽 ∂𝑒
= ⋅
∂𝑏 ∂𝑒 ∂𝑏

We already know:
∂𝐽 2
= 𝑒
∂𝑒 𝑛

Since:
∂𝑒
= −1
∂𝑏

Therefore:

∂𝐽 2
= − ∑(𝑦𝑖 − 𝑦̂𝑖 )
∂𝑏 𝑛

Example with 3 Epochs

Dataset

xy

13

25

37

49

5 11

This follows:

𝑦 = 2𝑥 + 1
Initialization

𝑤 = 0, 𝑏 = 0

Learning rate:

𝛼 = 0.01

Epoch 1

Predictions:

𝑦̂ = 0

Loss:

Step 1: Compute Predictions

Since:

𝑦̂ = 0 ⋅ 𝑥 + 0 = 0

All predictions are:

x y ŷ

1 3 0

2 5 0

3 7 0

4 9 0

5 11 0
Step 2: Compute Errors

Error = 𝑦 − 𝑦̂

y ŷ Error

3 0 3

5 0 5

7 0 7

9 0 9

11 0 11

Step 3: Square the Errors

(𝑦−𝑦̂)2

Error Squared Error

3 9

5 25

7 49

9 81

11 121

Step 4: Sum the Squared Errors

9 + 25 + 49 + 81 + 121
= 285
Step 5: Divide by n (Compute Mean)
285
𝑀𝑆𝐸 =
5
𝑀𝑆𝐸 = 57

Gradients:
∂𝐽
= −62
∂𝑤
∂𝐽
= −14
∂𝑏

Update:

𝑤 = 0 − 0.01(−62) = 0.62
𝑏 = 0 − 0.01(−14) = 0.14

Epoch 2

Loss ≈ 28.63

Updated parameters:

𝑤 ≈ 1.06
𝑏 ≈ 0.24

Epoch 3

Loss ≈ 14.41

Updated parameters:

𝑤 ≈ 1.39
𝑏 ≈ 0.32
Observation

• Loss decreases every epoch.

• Parameters move closer to true values (2 and 1).

• This shows convergence.

Graphs (Visualization Code)

You can run this Python code to generate graphs:


Why Gradient Descent Works

• The gradient gives the direction of steepest increase.

• We move in the opposite direction.

• Repeating this reduces the loss.

• For MSE in linear regression, the loss surface is convex → guarantees global minimum.

Key Takeaways

MSE measures squared prediction error


Chain rule is essential for computing gradients
Learning rate controls step size
Loss should decrease over epochs
For linear regression + MSE → convex optimization

You might also like