Mathematics of Neural Networks

Complete Derivation Notes

Topics: Forward Pass · Activation Functions · Cost Function · Backpropagation · Gradient Descent

1. Network Architecture & Notation

We use a 2 → 3 → 2 network as our worked example.

Layers are indexed from 0 (input) upwards. Neurons within a layer are indexed from 0.

Input = A(0) → ← Output = A(2)

a(1)0
a(0)0 a(2)0

a(1)1

a(0)1 a(2)1
a(1)2
Layer 0 Layer 1 Layer 2
(Input) (Hidden) (Output)

W(1): weight matrix, layer 1 W(2): weight matrix, layer 2

2. Labeling Convention

Activation of neuron j in layer L → a(L)_j (superscript = layer, subscript = neuron)

Weight from neuron k in layer L-1 to neuron j in layer L → W(L)_jk

Bias of neuron j in layer L → b(L)_j

Intermediate (pre-activation) value → Z(L)_j = weighted sum + bias

Weight matrix size

W(L) has shape: (neurons in layer L) × (neurons in layer L-1)

Note: This notation follows Andrew Ng (Coursera) and Nielsen's 'Neural Networks and Deep Learning'.

—1—
3. What Each Neuron Does

Each neuron performs TWO operations:

Step 1 — Compute weighted sum + bias (linear part):

Z (pre-activation)
Z(L)_j = W(L)_j0 · a(L-1)_0 + W(L)_j1 · a(L-1)_1 + ... + b(L)_j

Step 2 — Pass Z through non-linear activation function:

Activation output
a(L)_j = f( Z(L)_j ) where f = activation function

Why non-linear?
Without non-linearity, any deep network collapses to a single linear transformation
→ it could only solve linearly-separable problems (useless for real tasks!)

Common activation functions:

ReLU f(z) = max(0, z) f'(z) = 1 if z>0, else 0

Sigmoid f(z) = 1/(1+e^(-z)) f'(z) = f(z)·(1 - f(z))

Tanh f(z) = (e^z - e^(-z))/(e^z + e^(-z)) f'(z) = 1 - f(z)^2

Linear f(z) = z f'(z) = 1 (used in output for regression)

4. Forward Pass — Matrix Form

Instead of looping over neurons, we use matrix operations (layer-wise):

Full Forward Pass (2-layer network)

Z(1) = W(1) · A(0) + b(1)

A(1) = f1( Z(1) ) ← apply activation element-wise

Z(2) = W(2) · A(1) + b(2)

A(2) = f2( Z(2) ) ← output of network

Matrix sizes (for our 2→3→2 example):

A(0): 2×1 W(1): 3×2 b(1): 3×1 → Z(1), A(1): 3×1

A(1): 3×1 W(2): 2×3 b(2): 2×1 → Z(2), A(2): 2×1

General: W(L) is [neurons_L × neurons_{L-1}]

★ The activation function may differ between layers, but is shared among all neurons in the same layer.
—2—
5. Cost Function

The cost function measures how wrong the network's output is.

For output neuron i: o_i = a(L)_i (last-layer activation), y_i = desired output

Mean Squared Error (MSE)

C = (1/2) · Σ_i ( o_i - y_i )²

(The 1/2 factor makes derivatives cleaner — it cancels with the exponent 2)

Key properties of MSE:

C ≥ 0 always

C = 0 if and only if o_i = y_i for all i (perfect prediction!)

Any deviation from the target → positive cost → network must improve

C is a function of all weights and biases W(1), b(1), W(2), b(2), ...

6. Gradient Descent — Minimizing the Cost

We want to find W, b that minimize C. Gradient descent does this iteratively:

Gradient Descent Update Rule (for any parameter θ)

θ_new = θ_old - η · (∂C / ∂θ)

η (eta) = learning rate (hyperparameter, e.g. 0.01)

Intuition:
The gradient ∂C/∂θ points in the direction of steepest ASCENT

We subtract it → we move in the direction of steepest DESCENT

Small η: slow but stable | Large η: fast but may overshoot / diverge

We repeat until C is small enough (convergence)

Hyperparameters vs. Parameters:

Parameters (learned): W(1), b(1), W(2), b(2), ...
Hyperparameters (set by user): learning rate η, number of epochs, network architecture

Epoch: One full pass through all training examples.

Total updates = epochs × number of training examples

—3—
7. Chain Rule of Partial Differentiation

The cost C depends on weights/biases INDIRECTLY through a chain of operations:

W(2)_jk Z(2)_j a(2)_j = o_j C

By the Chain Rule:

Chain Rule applied to ∂C/∂W(2)_jk

∂C/∂W(2)_jk = (∂C/∂o_j) · (∂o_j/∂Z(2)_j) · (∂Z(2)_j/∂W(2)_jk)

= (o_j - y_j) · f2'(Z(2)_j) · a(1)_k

8. Defining the Error Term δ (delta)

To simplify notation, define the ERROR at each neuron:

Error (delta) at layer L, neuron j

δ(L)_j = (∂C/∂o_j) · f_L'(Z(L)_j)

= the 'blame' this neuron takes for the total error

For the OUTPUT layer (layer 2 in our example):

Output layer error vector δ(2)

δ(2) = ( A(2) - Y ) ■ f2'( Z(2) )

(■ = element-wise multiplication, also called Hadamard product)

Meaning: compare what the network produced vs what we wanted,

then scale by how sensitive the activation function is at that point.

★ Once we have δ for a layer, computing ∂C/∂W and ∂C/∂b for that layer is straightforward!

—4—
9. Backpropagation — Computing All Gradients

Backprop works BACKWARDS from the output layer to the first hidden layer.

Step 1: Compute output error:

δ(L) = (A(L) - Y) ■ f_L'( Z(L) )

Step 2: Compute gradients for last layer:

∂C/∂W(L) = δ(L) · A(L-1)■ ∂C/∂b(L) = δ(L)

Step 3: Propagate error backward:

δ(L-1) = ( W(L)■ · δ(L) ) ■ f_{L-1}'( Z(L-1) )

Step 4: Compute gradients for previous layer:

∂C/∂W(L-1) = δ(L-1) · A(L-2)■ ∂C/∂b(L-1) = δ(L-1)

Step 5: Repeat Steps 3–4 until first hidden layer.

10. Why ∂C/∂b(L) = δ(L) ?

Z(L)_j = Σ_k W(L)_jk · a(L-1)_k + b(L)_j

∂Z(L)_j / ∂b(L)_j = 1 → so the chain rule gives:

∂C/∂b(L)_j = δ(L)_j · 1 = δ(L)_j (bias gradient = error at that neuron)

11. Why ∂C/∂W(L) = δ(L) · A(L-1)■ ?

∂Z(L)_j / ∂W(L)_jk = a(L-1)_k

So: ∂C/∂W(L)_jk = δ(L)_j · a(L-1)_k
Stacking all j, k into matrices: this is exactly the outer product δ(L) · A(L-1)■

—5—
 Complete Summary — Training Algorithm
One training step for a single example

■ FORWARD PASS ■ BACKWARD PASS

for L = 1, 2, ..., last layer: Compute output error:

δ(last) = (A(last)-Y) ■ f'(Z(last))

Z(L) = W(L) · A(L-1) + b(L)

for L = last, ..., 1:

A(L) = f_L( Z(L) )

dW(L) = δ(L) · A(L-1)■

Output: A(last) = prediction db(L) = δ(L)

Compute cost: δ(L-1) = W(L)■·δ(L) ■ f'(Z(L-1))

C = (1/2) · ||A(last) - Y||²

Parameter Update (Gradient Descent)

W(L) ← W(L) - η · dW(L) b(L) ← b(L) - η · db(L)

(repeat for every layer L, then repeat the whole cycle for the next training example / epoch)

Quick Reference — All Key Equations

Forward Z(L) = W(L)·A(L-1) + b(L) A(L) = f_L(Z(L))

Output err δ(out) = (A(out) - Y) ■ f'(Z(out))

Back prop δ(L-1) = (W(L)■·δ(L)) ■ f'(Z(L-1))

Gradients dW(L) = δ(L) · A(L-1)■ db(L) = δ(L)

Update W ← W - η·dW b ← b - η·db

■ = element-wise (Hadamard) product · = matrix multiplication ■ = transpose f' = derivative of activation fn

—6—