Spring 2026 Department of AI, IIT Kharagpur

AI61002: Deep Learning Foundations

and Applications
Module 2: Basics of Feedforward Neural Networks

Dr. Somdyuti Paul

1
Feedforward Neural Network
• A general feedforward neural network (NN)
architecture consists of an input layer, one or more
hidden layers and an output layer, connected
sequentially.

• The connections between neurons in a

feedforward neural network does not form cycles
or loops

• The number of layers in a feedforward NN

determines its depth.

• The number of units in a given layer of the NN

determines its width.
2
Weights, Biases and Activation Functions
• Each neuron in a neural network is associated with the following:

• Weights: parameters that control the strength of the connections of a neuron to its inputs.

• Bias: parameters that shift the decision boundary of each neuron, imparting exibility to the model.

• Activation function: activation functions are applied to the weighted linear combination of the inputs
to introduce non-linearity to the network that enables it to learn complex data relationships and
intricate patterns.

• The total number of trainable parameters is total number of weights and biases of all neurons in the
network.

• The weights and bias of each neuron of a neural network are adjusted during training to minimize the
error between predicted and actual outputs.
3

fl
Common Activation Functions

Name Functional form Common usage

Linear g(z) = z Output layer

ReLU g(z) = max{0, z} Hidden layers

1 Output layer
Logistic g(z) = Hidden layer in legacy
1 + e −z architectures
Hyperbolic e z − e −z Hidden layer in legacy
Tangent g(z) = z architectures
e + e −z
zi
e
Softmax g(zi) = zj Output layers
∑j e

4
 Neural Networks as Universal Function Approximators
A feedforward network with a single hidden layer and any squashing activation function can approximate any
continuous function de ned on compact domain with any precision, provided that there are enough hidden units.

Universal approximation theorem (G. Cybenko, 1989):

n
Let In be the unit cube [0,1] and C(In) be the space of all continuous functions on In, and σ : ℝ → ℝ be a
sigmoidal function that satis es:

{0 if z → − ∞
1 if z → ∞
σ(z) →

αiσ(wiT x + bi), with σ being a sigmoidal function such that

∑
Then for every f(x) ∈ C(In) and ϵ > 0, ∃ g(x) =
i
sup{ | g(x) − f(x) | : x ∈ In} < ϵ
The universal approximation property of neural networks holds for any arbitrary non-linear activation function
(K. Hornik, 1991)
5
fi
fi
Neural Networks as Universal Function Approximators
Intuition

• Sigmoid functions could be

used to approximate a step
function.

• A network with 3 neurons

can compute a pulse.

6
Neural Networks as Universal Function Approximators

• A train of pulses can approximate a

continuous function to any arbitrary
precision.

• The concept generalizes to higher

dimensions.

7
Parametric Supervised Learning Framework for Neural Networks

To train a neural network in a parametric, supervised learning framework, the following must be de ned:

(i) (i) M
• Dataset: {(x , y )}i=1

(i) (i)
• Model: ŷ = hθ(x ).
M
ℓ(hθ(x ), y )
(i) (i)
∑
Cost function: J(θ) =
•
i=1

• Optimizer: θnew = θ − α ∇θ J(θ)

2
• Performance metrics: MSE, R score, accuracy, precision, recall, etc.

fi
Computational Graph of a Neural Network

• A computational graph is a graphical representation of the ow of data and operations.

• A neural network can be represented by a directed graph such that
• Nodes represent operations (or variables)
• Edges represent data ow (or operations)
• The graph is de ned with respect to a set of allowable operations.

9
fi
fl
fl
 Computational Graph of a Neural Network
• The computational graph facilitates two key processes in neural networks
• Forward pass: involves performing sequential computations as de ned by the computational
graph to determine the output of the network as well as the associated cost function.

• Backpropagation: involves computing the gradients of the cost with respect to the weights of
each layer by traversing the computational graph backward node by node, starting with the
output node and ending at the input node.

• The computational graph also enables the following:

• Optimization for e cient execution: shared subgraphs could be computed once and re-used.
• Parallelization: independent operations in the graph could be parallelized over multiple GPUs.

10
ffi
fi
Computational Graph of a Neural Network
Example: Computational graph of a 2 layer neural network:

(1)
w11 (1)
h =W x+b (1) (1)

(1)
w12 ⋅ (1)
h1 (1)
a1
(1)
a = ReLU(h ) (1)

+ ReLU (2)
h =W a +b (2) (1) (2)
x1 ⋅ ⋅ (2)
ŷ = σ(h )
(2)
x2 (1) w11
b1 h (2)
ŷ
(1)
w21 b (2) + σ
⋅ +
(1)
h2
ReLU
a2
(1)

(1)
w22
⋅ ⋅
(2)
w12
(1)
b2
11
Learning Neural Network Parameters Using Backpropagation
A neural network may be represented as a repeated composition of functions

x1 x2 x3 xn−1 xn ŷ
f1(x1, W1) f2(x2, W2) ⋯ fn−1(xn−1, Wn−1) fn(xn, Wn)

ŷ = fn( fn−1(⋯( f2( f1(x1, W1), W2)⋯), Wn−1), Wn)

M M
(i) (i) (i) (i)
ℓ(y , ŷ ) =
∑ ∑
J= ℓ(y , fn( fn−1(⋯( f2( f1(x1 , W1), W2)⋯), Wn−1), Wn))
i i=1

Backpropagation involves the recursive application of the chain rule of derivatives:

∂J ∂J ∂fn(xn, Wn) ∂fn−1(xn−1, Wn−1) ∂fj+1(xj+1, Wj+1) ∂fj(xj, Wj) ∂J ∂fj(xj, Wj)
= ⋯ =
∂Wj ∂y ̂ ∂xn ∂xn−1 ∂xj+1 ∂Wj ∂xj+1 ∂Wj

12
Gradient Flow
∂J ∂J ∂fj(xj, Wj) ∂J ∂fn(xn, Wn) ∂fn−1(xn−1, Wn−1) ∂fj+1(xj+1, Wj+1) ∂fj(xj, Wj)
= = ⋯
∂Wj ∂xj+1 ∂Wj ∂y ̂ ∂xn ∂xn−1 ∂xj+1 ∂Wj
Computed recursively

If the gradient
∂J
is available, the gradient of the preceding xj xj+1
∂xj+1
layer can be computed as follows:
∂fj(xj, Wj)
fj
∂xj
∂J ∂J ∂xj+1 ∂J ∂fj(xj, Wj) ∂J ∂fj(xj, Wj) ∂J
= =
∂xj ∂xj+1 ∂xj ∂xj+1 ∂xj ∂xj+1 ∂xj ∂xj+1
Upstream gradient
Downstream gradient = Upstream gradient × Local gradient Downstream gradient Local gradient

13
Gradient Flow
When a layer has more than one input, a local gradient is computed with respect to each input.

z = f(x, y, W)
x
∂J ∂J ∂z ∂J ∂f(x, y, W) ∂z
z
f
= =
∂x ∂z ∂x ∂z ∂x
∂J ∂J ∂f(x, y, W)
∂x
= ∂z
∂J ∂J ∂z ∂J ∂f(x, y, W) ∂x ∂z ∂x ∂J
∂y
= =
∂z ∂y ∂z ∂y
∂y ∂z
Downstream gradient = Upstream gradient ×
y
Upstream gradient
Local gradient
Local gradient
∂J ∂J ∂f(x, y, W)
=
∂y ∂z ∂y
Downstream gradients

14
Gradient Flow Patterns
p p
∂J
∂p
=
∂J
∂r +
r
∂J
∂J
∂p
=q⋅
∂J
∂r · r
∂J
q ∂r q ∂r
∂J ∂J
= ∂J ∂J
∂q ∂r =p⋅
∂q ∂r

Addition: gradient distributor Multiplication: gradient switcher

15
Gradient Flow Patterns
p q>p
p
∂J1
∂J r p ∂p
=0 max
∂p
∂J2
∂J ∂J1 ∂J2
q ∂r ∂p
+
∂p p
∂p
∂J ∂J
=
∂q ∂r

Max: gradient router Copy: gradient adder

16
Vector Derivatives
∂f
Partial derivative: x, y ∈ ℝ, f(x, y) ∈ ℝ , ∈ℝ
∂x
Gradient: x ∈ ℝn, f(x) ∈ ℝ, ∇x f ∈ ℝn
∂f ∂f ∂f
∇x f = [
∂xn ]
, ⋯,
∂x1 ∂x2
Note: the gradient ∇x f can be written as a row vector (using numerator layout) or as a column vector (using denominator
layout). Here, we are using the former convention.

n ∂f
m
Jacobian: x ∈ ℝ , f(x) ∈ ℝ , ∈ ℝm×n (numerator layout)
∂x
∂f1 ∂f1
∂x1
⋯ ∂xn
∂f
= ⋮ ⋱ ⋮
∂x ∂fm ∂fm
∂x1
⋯ ∂xn

17
Gradients for Linear Functions
Each layer of a feedforward neural network performs linear operations of the form:
z = Wx + b (x ∈ ℝ , W ∈ ℝ
n m×n
,b∈ℝ ) m

The gradients with respect to the cost function on backpropagation through this linear function are then computed as:

∂J ∂J ∂z ∂J ∂J ∂z ∂J ∂J ∂z
= = =
∂x ∂z ∂x ∂W ∂z ∂W ∂b ∂z ∂b
∂zi ∂J ∂z
= Wij Let = δ (gradient of shape 1 × m) Let = I (m × m identity matrix)
∂xj ∂z ∂b
∂z ∂zi ∂J ∂J
∴ = W (Jacobian of shape m × n ) = xj = I
∂x ∂Wij ∂b ∂z
∂J ∂J ∂J ∂J ∂z
= W = = δi xj
∂x ∂z ∂Wij ∂z ∂Wij
∂J T T
=δ x
∂W
18
 Gradients for Pointwise Activation Functions
Each hidden layer of feedforward neural network applies element-wise activation functions on the output of
the linear operations as follows:
m
a = σ(z) (z ∈ ℝ )
∂J ∂J ∂a ∂J
= = ⊙ [σ′(z1), σ′(z2), ⋯σ′(zm)]
∂z ∂a ∂z ∂a

Examples:

tanh activation Logistic activation ReLU activation

z
e −e −z 1 σ(z) = max(0, z)
σ(z) = z σ(z) = σ′(z) = 1z>0
e + e −z 1 + e −z
2
σ′(z) = 1 − σ(z) σ′(z) = σ(z)(1 − σ(z))

19






Backpropagation Through a Neural Network
• Consider the following 2 layer feedforward neural network:

(x ∈ ℝ , W ∈ ℝ )
(1) (1) (1) n×1 (1) m×n (1) m×1
h =W x+b ,b ∈ℝ
(h ∈ ℝ )
(1) (1) (1) m×1
a = ReLU(h )

(a ∈ ℝ , W )
(2) (2) (1) (2) m×1 (2) k×m (2) k×1
h =W a +b ∈ℝ ,b ∈ℝ
(h ∈ ℝ )
(2) (2) k×1
ŷ = σ(h )
k
ℓ(y, y)̂ = − yjlog(yĵ ) + (1 − yj)log(1 − yĵ )
∑
j=1

• The back propagation through this neural network works as follows:

Step 1: Gradient of the loss w.r.t the output

∂ℓ yĵ − yj ∂ℓ
( )
(1×1)
= ∈ℝ
∂yĵ yĵ (1 − yĵ ) ∂yĵ
20
Backpropagation Through a Neural Network
Step 2: Gradients of the loss w.r.t output layer parameters:
∂ℓ ∂ℓ ∂yĵ ∂ℓ y ̂ − yj
( )
(2) (2) j
= = σ(hj
) 1 − σ(hj
) = yĵ (1 − yĵ ) = yĵ − yj
(2)
∂hj ∂yĵ ∂hj
(2) ∂yĵ yĵ (1 − yĵ )

∂ℓ ∂ℓ
( (2) ∈ ℝ )
T (1×k)
= ( y ̂ − y)
∂h (2) ∂h
(2)
∂ℓ ∂ℓ ∂h ∂ℓ
( )
T
( (2) )
(1) T (k×m)
= = ( y ̂ − y)(a ) ∈ ℝ
∂W (2) ∂h (2) ∂W (2) ∂W
(2)
∂ℓ ∂ℓ ∂h ∂ℓ
( (2) ∈ ℝ )
T (1×k)
= = ( y ̂ − y) Ik×k
∂b (2) ∂h (2) ∂b (2) ∂b

21
Backpropagation Through a Neural Network
Step 3: Gradients of the loss w.r.t hidden layer parameters:

(2)
∂ℓ ∂ℓ ∂h ∂ℓ (2) ∂ℓ
( (1) ∈ ℝ )
T (2) (1×m)
= = W = ( y ̂ − y) W
∂a (1) ∂h (2) ∂a (1) ∂h (2) ∂a

∂ℓ ∂ℓ ∂a (1) ∂ℓ
( (1) ∈ ℝ )
T (2) ′ (1) (1×m)
= = ( y ̂ − y) W ⊙ ReLU (h )
∂h (1) ∂a (1) ∂h (1) ∂h
(1)
∂ℓ ∂ℓ ∂h ∂ℓ
( )
T
( ) ( (1) )
T (2) ′ (1) T T (m×n)
= = ( y ̂ − y) W ⊙ ReLU (h ) x ∈ ℝ
∂W (1) ∂h (1) ∂W (1) ∂W
(1)
∂ℓ ∂ℓ ∂h ∂ℓ
( (1) ∈ ℝ )
T (2) ′ (1) (1×m)
= = ( y ̂ − y) W ⊙ ReLU (h )Im×m
∂b (1) ∂h (1) ∂b (1) ∂b

22



 Backpropagation Using Computational Graph
(1)
w11 (1)
h =W x+b (1) (1)

(1)
w12 ⋅ (1)
h1 (1)
a1
(1)
a = ReLU(h ) (1)

+ ReLU (2)
h =W a +b (2) (1) (2)

x1 ⋅ (2)
⋅ ∂ ℓ
h
( 2)
(2)
ŷ = σ(h )
x2 (1) ∂ℓ w11 ∂ℓ ∂
b1 h (2)
(1)
w21
∂h (1) b (2) ∂h (2)
+ σ ŷ
⋅ ∂ℓ 2 (1)
h2 (1)
a2
(1)
+ ∂ℓ ∂ŷ ∂ℓ
∂h 2 ReLU
(1)
w22
⋅
∂ℓ
⋅ ∂ŷ ∂h (2)

∂ℓ
∂ℓ ∂y ̂

2)
ReLU′(h (1))

∂h (
2 ∂h (2) w (2)
∂ℓ (1)
∂a2 12 )
x ( 1
(1) 2 ℓ a2
∂ℓ

∂h2
2 )

∂
∂h (1

(1) )
b2 (2)
w12 ( 2
∂h
23