Derivatives, Partial Derivatives, Gradients and Gradient

Descent

Your Name

November 14, 2025

Your Name Derivatives, Partial Derivatives, Gradients and GradientNovember

Descent 14, 2025 1 / 36
Outline

1 Functions and Change

2 Slopes and Rates of Change

3 Derivatives

4 Partial Derivatives

5 The Gradient

6 Gradient Descent

7 Exercise: Gradient Computation

8 Backpropagation

Your Name Derivatives, Partial Derivatives, Gradients and GradientNovember

Descent 14, 2025 2 / 36
Functions

A function assigns a real output to a real input:

f : R → R.

Examples:
f (x) = x 2 , g (x) = 3x + 5, h(x) = e x .
We want to understand how the output changes when the input changes.

Your Name Derivatives, Partial Derivatives, Gradients and GradientNovember

Descent 14, 2025 3 / 36
Why Study Change

In many applications we want to know:

how sensitive a function is to its input
where it increases or decreases
where it reaches maxima or minima
For this we need a measure of local change.

Your Name Derivatives, Partial Derivatives, Gradients and GradientNovember

Descent 14, 2025 4 / 36
Graphical Intuition

Imagine the graph of a function.

The steepness at each point indicates how fast the function is changing.
We need a quantitative measure of steepness.

Your Name Derivatives, Partial Derivatives, Gradients and GradientNovember

Descent 14, 2025 5 / 36
Slope of a Line

For a straight line:

∆y
slope =.
∆x
This tells us how much y changes when x increases.

Your Name Derivatives, Partial Derivatives, Gradients and GradientNovember

Descent 14, 2025 6 / 36
Curved Functions

Most functions are not straight.

Their steepness changes from point to point.
The slope formula must be adapted.

Your Name Derivatives, Partial Derivatives, Gradients and GradientNovember

Descent 14, 2025 7 / 36
Secant Line

To approximate slope on a curve we use:

f (x + h) − f (x)
.
h
This measures average change over the interval h.

Your Name Derivatives, Partial Derivatives, Gradients and GradientNovember

Descent 14, 2025 8 / 36
Towards an Instantaneous Slope

As we shrink h, the secant line approaches the tangent line.

The tangent line at a point describes the instantaneous rate of change.
This leads to the definition of the derivative.

Your Name Derivatives, Partial Derivatives, Gradients and GradientNovember

Descent 14, 2025 9 / 36
Derivative as Instantaneous Rate of Change

The derivative is defined as:

f (x + h) − f (x)
f ′ (x) = lim .
h→0 h
Interpretation:
slope of the tangent line

Your Name Derivatives, Partial Derivatives, Gradients and GradientNovember

Descent 14, 2025 10 / 36
Basic Rules: Exponential, Logarithmic and Trigonometric

d x
(e ) = e x
dx
d 1
(ln x) =
dx x
d d
(sin x) = cos x, (cos x) = − sin x
dx dx

Your Name Derivatives, Partial Derivatives, Gradients and GradientNovember

Descent 14, 2025 11 / 36
Product and Chain Rules

Product rule:
(fg )′ = f ′ g + fg ′
Chain rule:
(f (g (x)))′ = f ′ (g (x))g ′ (x)

Your Name Derivatives, Partial Derivatives, Gradients and GradientNovember

Descent 14, 2025 12 / 36
Example of a Composite Function

Let
f (x) = 2x 2 + sin(3x).
Compute f ′ (x) using power and chain rules.

Your Name Derivatives, Partial Derivatives, Gradients and GradientNovember

Descent 14, 2025 13 / 36
Solution

d d
(2x 2 ) = 4x, (sin(3x)) = 3 cos(3x).
dx dx
Thus:
f ′ (x) = 4x + 3 cos(3x).

Your Name Derivatives, Partial Derivatives, Gradients and GradientNovember

Descent 14, 2025 14 / 36
Multivariable Functions

Many real problems involve several variables:

f (x1 , x2 , . . . , xn ).

We want to measure how the function changes with respect to each

variable.

Your Name Derivatives, Partial Derivatives, Gradients and GradientNovember

Descent 14, 2025 15 / 36
Definition of Partial Derivative

∂f f (x1 , . . . , xi + h, . . . , xn ) − f (x1 , . . . , xi , . . . , xn )
= lim .
∂xi h→0 h
Other variables remain fixed.

Your Name Derivatives, Partial Derivatives, Gradients and GradientNovember

Descent 14, 2025 16 / 36
Example

For
f (x, y ) = x 2 y + 3y ,
we have:
∂f ∂f
= 2xy , = x 2 + 3.
∂x ∂y

Your Name Derivatives, Partial Derivatives, Gradients and GradientNovember

Descent 14, 2025 17 / 36
The Gradient Vector

For f : Rn → R:  ∂f 
∂x1
 ∂f 
 ∂x 
∇f (x) =  . 2  .
 .. 
∂f
∂xn

The gradient points in the direction of steepest increase.

Your Name Derivatives, Partial Derivatives, Gradients and GradientNovember

Descent 14, 2025 18 / 36
Interpretation of the Gradient

The direction where the function increases most rapidly.

Its magnitude tells how steep that direction is.
Used to guide optimization algorithms.

Your Name Derivatives, Partial Derivatives, Gradients and GradientNovember

Descent 14, 2025 19 / 36
Optimization Problem

We want to find
min f (x).
x

Gradient descent is a basic iterative method for this task.

Your Name Derivatives, Partial Derivatives, Gradients and GradientNovember

Descent 14, 2025 20 / 36
Gradient Descent Update Rule

Given a learning rate α > 0, the update is:

xk+1 = xk − α∇f (xk ).

Move opposite to the gradient because the gradient points uphill.

Your Name Derivatives, Partial Derivatives, Gradients and GradientNovember

Descent 14, 2025 21 / 36
Choosing the Learning Rate

If α is too small, progress is slow.

If α is too large, the algorithm may diverge.
Often chosen empirically.

Your Name Derivatives, Partial Derivatives, Gradients and GradientNovember

Descent 14, 2025 22 / 36
Exercise Setup

Consider the function:

f (x1 , x2 ) = 2x12 x22 + x2 .

We want to:
compute the gradient
evaluate the function at (3, 1)
perform one step of gradient descent with α = 0.01

Your Name Derivatives, Partial Derivatives, Gradients and GradientNovember

Descent 14, 2025 23 / 36
Step 1: Compute Gradient

Compute partial derivatives:

∂f ∂f
= 4x1 x22 , = 4x12 x2 + 1.
∂x1 ∂x2
Hence:
4x1 x22
 
∇f (x1 , x2 ) = .
4x12 x2 + 1

Your Name Derivatives, Partial Derivatives, Gradients and GradientNovember

Descent 14, 2025 24 / 36
Step 2: Evaluate at (3, 1)

Gradient at:  
12
∇f (3, 1) = .
37
Function value:
f (3, 1) = 19.

Your Name Derivatives, Partial Derivatives, Gradients and GradientNovember

Descent 14, 2025 25 / 36
Step 3: Gradient Descent Step

Update with α = 0.01:

xnew = (3, 1) − 0.01(12, 37)

xnew = (2.88, 0.63).

The function value decreases to approximately:

f (2.88, 0.63) ≈ 7.21.

Your Name Derivatives, Partial Derivatives, Gradients and GradientNovember

Descent 14, 2025 26 / 36
ML

Where is Calculus Hidden in Machine Learning?

Your Name Derivatives, Partial Derivatives, Gradients and GradientNovember

Descent 14, 2025 27 / 36
From the Exercise to Machine Learning

In the exercise we used gradient descent to minimize a simple function. In

machine learning the same idea is applied to the loss function of a model.
The function to minimize is the loss:

L(θ) = L(f (x; θ), y )

The variables x1 , x2 in the exercise become the model parameters θ

The gradient tells us how changing the parameters affects the loss
A gradient descent step updates the parameters so that future
predictions improve
Thus, training a machine learning model with gradient descent applies it
to the loss function rather than on a simple mathematical function.

Your Name Derivatives, Partial Derivatives, Gradients and GradientNovember

Descent 14, 2025 28 / 36
What a Neural Network Layer Computes
A neural network processes information one layer at a time. Each layer
applies a differentiable function that depends both on the input coming
from the previous layer and on that layer’s parameters.
Thus the activation of layer k is a function of:
the previous activation ak−1
the parameters θk = (Wk , bk )
Mathematically, the layer computes:

ak = fk (ak−1 ; θk ).

This shows explicitly that the output of each layer depends on the previous
activation and on its own parameters.
The network prediction is the final activation:

ŷ = aK .

Your Name Derivatives, Partial Derivatives, Gradients and GradientNovember

Descent 14, 2025 29 / 36
Why the Loss Depends on All Layers

Because each activation depends on the previous one, a change in any

layer affects all later layers and therefore the prediction.
Thus the prediction is a nested composition of all layer functions:

ŷ = fK (fK −1 (. . . f1 (x; θ1 ) . . .)).

The loss measures how wrong the prediction is:

L = L(ŷ , y ).

Since the prediction depends on all parameters, the loss is also a function
of all parameters:
L = L(θ1 , θ2 , . . . , θK ).

Your Name Derivatives, Partial Derivatives, Gradients and GradientNovember

Descent 14, 2025 30 / 36
Why Training Requires Gradients

Training means improving the model so that the loss decreases.

To improve the model, we must adjust the parameters in the direction that
makes the loss smaller. This requires knowing how the loss changes when
each parameter changes.
In words:
if a small change in a parameter reduces the loss, we keep moving
that way
if it increases the loss, we move in the opposite direction
Mathematically, this sensitivity is captured by the gradient:
∂L
.
∂θk
Therefore, computing these gradients is essential for learning.

Your Name Derivatives, Partial Derivatives, Gradients and GradientNovember

Descent 14, 2025 31 / 36
Why We Cannot Compute the Gradient Directly
For simple functions, such as
f (x) = x 2 ,
we can compute the derivative immediately:
f ′ (x) = 2x.
But in a neural network the loss is not a simple function of one variable. It
is a deep composition of many functions:


L = L fK (fK −1 (. . . f1 (x; θ1 ) . . .); θK ), y .
This means:
each activation depends on the previous activation
each activation depends on multiple parameters
the loss depends on all layers through nested compositions
Because of this structure, we cannot differentiate the loss by inspection.
We need a systematic way to compute derivatives of composed functions.
This is exactly what the chain rule provides.
Your Name Derivatives, Partial Derivatives, Gradients and GradientNovember
Descent 14, 2025 32 / 36
How Backprop Starts: Derivative at the Output Layer
Backpropagation begins by computing the derivative of the loss with
respect to the final activation aK , which is the prediction ŷ .
The loss is a function of the prediction:

L = L(ŷ , y ), ŷ = aK .

Its derivative can be computed as:

∂L ∂L
= .
∂aK ∂ ŷ
This derivative is easy to compute because the loss is defined explicitly.
Examples:
Mean squared error (MSE)
1 ∂L
L = (ŷ − y )2 ⇒ = ŷ − y
2 ∂aK
This quantity is the starting point of the backward pass.
Your Name Derivatives, Partial Derivatives, Gradients and GradientNovember
Descent 14, 2025 33 / 36
How Do We Compute Gradients for Earlier Layers?

Once we have the gradient at the output layer

∂L
δK = ,
∂aK
we need the gradient for layer K − 1, then K − 2, and so on.
Each activation affects the loss only through the next activation:

ak = fk (ak−1 ; θk ).

Therefore, by the chain rule,

∂L ∂L ∂ak
= · .
∂ak−1 ∂ak ∂ak−1

Your Name Derivatives, Partial Derivatives, Gradients and GradientNovember

Descent 14, 2025 34 / 36
 ∂ak
What Is ∂ak−1 ?

The quantity
∂ak
∂ak−1
measures how the output of layer k changes when its input ak−1 changes.
The quantity ∂a∂ak−1
k
can always be computed by applying the standard
derivative rules (derivative of the activation function, derivative of linear
operations, product rule, etc.).
We do not need to write the full expression by hand. What matters is that:
∂ak
is a known, computable derivative of the layer.
∂ak−1
Once we can compute this local derivative for each layer, we can propagate
gradients backward through the entire network via the recursion:
∂ak
δk−1 = δk · .
∂ak−1
This generalizes the chain rule to any network architecture.
Your Name Derivatives, Partial Derivatives, Gradients and GradientNovember
Descent 14, 2025 35 / 36
Do We Always Use Backpropagation?
Backpropagation is the standard method for training neural networks
because it computes exact gradients efficiently.
However, it is not the only possible approach.
Yes — when:
we can compute layer derivatives
This includes almost all modern deep learning models.
No — alternatives are used when:
the model is not differentiable
the loss is not differentiable
Examples:
numerical finite differences
evolutionary or gradient-free optimization
reinforcement learning policy search methods
Backpropagation remains dominant because it provides fast and exact
gradients for differentiable neural networks.
Your Name Derivatives, Partial Derivatives, Gradients and GradientNovember
Descent 14, 2025 36 / 36