Deep Learning
Prof. FATIMA-EZZAHRAA BEN-BOUAZZA
Lecture 3: Convolutional networks
ESM6ISS , UM6SS
/09/2025
Prepared by:
PhD. EMSSAAD Ilyass
PhD. CHAKOUR EL MEZALI Manal
1
Plan for Today
How to make neural networks see?
• Visual perception
• Convolutions
• Pooling
• Convolutional networks
2 / 56
Visual perception
3 / 56
Visual perception
In 1959-1962, David Hubel and Torsten Wiesel identify the neural basis of
information processing in the visual system. They are awarded the Nobel Prize
of Medicine in 1981 for their discovery.
4 / 56
5 / 56
Inductive biases
Can we equip neural networks with inductive biases tailored for vision?
Locality (as in simple cells)
Invariance translation (as in complex cells)
Hierarchical compositionality (as in hypercomplex cells)
7 / 56
Invariance and equivariance to translation.
—
Credits: Simon J.D. Prince, Understanding Deep Learning, 8 / 56
2023.
Neocognitron
In 1980, Fukushima proposes a direct neural network implementation of the
hierarchy model of the visual nervous system of Hubel and Wiesel.
Built upon
convolutions and
enables the
composition of a
feature hierarchy.
Biologically-inspired
training algorithm,
which proves to be
largely inefficient.
—
Credits: Kunihiko Fukushima, Neocognitron: A Self-organizing Neural Network Model, 1980. 9 / 56
Convolutional networks
In the 1980-90s, LeCun trains a convolutional network by backpropagation. He
advocates for end-to-end feature learning in image classification.
—
Credits: LeCun et al, Handwritten Digit Recognition with a Back-Propagation Network, 1990. 10 / 56
Convolutions
11 / 56
Convolutional layers
A convolutional layer applies the same linear transformation locally everywhere
while preserving the signal structure.
—
Credits: Francois Fleuret, EE559 Deep Learning, 12 / 56
EPFL.
1d convolution
For the one-dimensional input x ∈ R W and the convolutional kernel u ∈ R w ,
the discrete convolution x ⊛ u is a vector of size W − w + 1 such that
Technically, ⊛ denotes the cross-correlation operator. However, most machine
learning libraries call it convolution.
13 / 56
Convolutions can implement differential operators:
or crude template matchers:
—
Credits: Francois Fleuret, EE559 Deep Learning, 14 / 56
EPFL.
2d convolution
For the 2d input tensor x ∈ R H×W and the 2d convolutional kernel u ∈ R h×w , the
discrete convolution x ⊛ u is a matrix of size ( H − h + 1) × (W − w + 1) such
that
15 / 56
Channels
The 2d convolution can be extended to tensors with multiple channels.
For the 3d input tensor x ∈ R C ×H×W and the 3d convolutional kernel
u ∈ R C×h×w , the discrete convolution x ⊛ u is a tensor of size
( H − h + 1) × (W − w + 1) such that
16 / 56
Convolutional layers
A convolutional layer is de ned by a set of K kernels u k of size C × h × w. It
applies the 2d convolution operation to the input tensor x of size C × H × W
to produce a set of K feature maps o k .
17 / 56
―
Credits: Francois Fleuret, EE559 Deep Learning, 18 / 56
EPFL.
Convolutions have three additional parameters:
• The padding specifies the size of a zeroed frame added arount the input.
• The stride specifies a step size when moving the kernel across the signal.
• The dilation modulates the expansion of the filter without adding weights.
Padding=1 Stride=2 Dilation=2
―
Credits: Francois Fleuret, EE559 Deep Learning, 19 / 56
EPFL.
Equivariance
Formally, a function f is equivariant to g if f (g(x)) = g(f (x)) .
Parameter sharing used in a convolutional layer causes the layer to be equivariant
to translation.
If an object moves in the input image, its representation will move the same amount in the output.
―
Credits: LeCun et al, Gradient-based learning applied to document recognition, 1998. 20 / 56
Convolutions as matrix multiplications
As a guiding example, let us consider the convolution of single-channel tensors
x ∈ R4×4 and u ∈ R3×3 :
21 / 56
The convolution operation can be equivalently re-expressed as a single matrix
multiplication:
the convolutional kernel u is rearranged as a sparse Toeplitz circulant matrix,
called the convolution matrix:
the input x is flattened row by row, from top to bottom:
Then,
which we can reshape to a 2 × 2 matrix to obtain x ⊛ u.
22 / 56
The same procedure generalizes to x ∈ R H×W and convolutional kernel
u ∈ Rh×w , such that:
- the convolutional kernel is rearranged as a sparse Toeplitz circulant matrix
U of shape ( H − h + 1)(W − w + 1) × H W where
• each row i identifies an element of the output feature map,
• each column j identifies an element of the input feature map,
• the value U i , j corresponds to the kernel value the element j is multiplied with in output i;
- the input x is f lat tened into a column vector v(x) of shape H W × 1;
- the output feature map x ⊛ u is obtained by reshaping the
( H − h + 1)(W − w + 1) × 1 column vector Uv(x) as a
( H − h + 1) × (W − w + 1) matrix.
Therefore, a convolutional layer is a special case of a fully connected layer:
23 / 56
Fully connected vs convolutional layers.
—
Credits: Simon J.D. Prince, Understanding Deep Learning, 24 / 56
2023.
Pooling
25 / 56
When the input volume is large, pooling layers can be used to reduce the input
dimension while preserving its global structure, in a way similar to a down- scaling
operation.
26 / 56
Pooling
Consider a pooling area of size h × w and a 3d input tensor x ∈ R C×(rh)×(sw) .
Max-pooling produces a tensor o ∈ R C×r×s such that
Average pooling produces a tensor o ∈ R C×r×s such that
Pooling is very similar in its formulation to convolution.
27 / 56
―
Credits: Francois Fleuret, EE559 Deep Learning, 28 / 56
EPFL.
Invariance
Formally, a function f is invariant to g if f (g(x)) = f (x).
Pooling layers provide invariance to any permutation inside one cell, which results in
(pseudo-)invariance to local translations.
―
Credits: Francois Fleuret, EE559 Deep Learning, 29 / 56
EPFL.
Convolutional networks
30 / 56
A convolutional network is generically de ned as a composition of convolutional
layers (CONV), pooling layers (POOL), linear recti ers (ReLU) and fully connected
layers (FC).
31 / 56
The most common convolutional network architecture follows the pattern:
INPUT → [[CONV → ReLU]*N → POOL?]*M → [FC → ReLU]*K → FC
where:
* indicates repetition;
POOL? indicates an optional pooling layer;
N ≥ 0 (and usually N ≤ 3), M ≥ 0, K ≥ 0 (and usually K < 3);
the last fully connected layer holds the output (e.g., the class scores).
32 / 56
Some common architectures for convolutional networks following this pattern
include:
INPUT → FC, which implements a linear classifier ( N = M = K = 0).
INPUT → [FC → ReLU]∗K → FC, which implements a K - layer MLP.
INPUT → CONV → ReLU → FC.
INPUT → [CONV → ReLU → POOL]*2 → FC → ReLU → FC.
INPUT → [[CONV → ReLU]*2 → POOL]*3 → [FC → ReLU]*2 → FC.
33 / 56
34 / 56
(Demo)
[Link]
Architectures (some)
—
Credits: Bianco et al, 2018. 36 / 56
LeNet-5 (LeCun et al, 1998)
Composition of two CONV + POOL layers, followed by a block of fully-connected
layers.
—
Credits: Dive Into Deep Learning, 2020. 37 / 56
AlexNet (Krizhevsky et al, 2012)
Composition of a 8-layer convolutional neural
network with a 3-layer MLP.
The original implementation was made of
two parts such that it could fit within two
GPUs.
LeNet vs. AlexNet
―
Credits: Dive Into Deep Learning, 2020. 39 / 56
VGG (Simonyan and
Zisserman, 2014)
Composition of 5 VGG blocks
consisting of CONV + POOL
layers, followed by a block of
fully connected layers.
The network depth increased
up to 19 layers, while the
kernel sizes reduced to 3.
AlexNet vs. VGG
―
Credits: Dive Into Deep Learning, 2020. 40 / 56
The effective receptive field is the part of the visual input that affects a given
unit indirectly through previous convolutional layers.
It grows linearly with depth when chaining convolutional layers.
It grows exponentially with depth when pooling layers (or strided convolutions)
are interleaved with convolutional layers.
41 / 56
—
Credits: Simon J.D. Prince, Understanding Deep Learning, 2023. 42 / 56
ResNet (He et al, 2015)
Composition of convolutional and pooling layers organized in
a stack of residual blocks. Extensions consider more residual
blocks, up to a total of 152 layers (ResNet-152).
Regular ResNet block vs. ResNet block with 1 × 1 convolution.
―
Credits: Dive Into Deep Learning, 2020. 43 / 56
Training networks of this depth is made possible because of the skip connections
h = x + f (x) in the residual blocks. They allow the gradients to shortcut the layers
and pass through without vanishing.
―
Credits: Dive Into Deep Learning, 2020. 44 / 56
45 / 56
The benefits of depth
46 / 56
... and width and resolution (Tan and Le, 2019)
—
Credits: Tan and Le, 2019. 47 / 56
Under the hood
48 / 56
Understanding what is happening in deep neural networks after training is
complex and the tools we have are limited.
In the case of convolutional neural networks, we can look at:
-the network's kernels as images
-internal activations on a single sample as images
-distributions of activations on a population of samples
-derivatives of the response with respect to the
-input maximum-response synthetic samples
―
Credits: Francois Fleuret, EE559 Deep Learning, EPFL. 49 / 56
Looking at filters
LeNet's first convolutional layer, all filters.
―
Credits: Francois Fleuret, EE559 Deep Learning, EPFL. 50 / 56
LeNet's second convolutional layer, first 32 filters.
―
Credits: Francois Fleuret, EE559 Deep Learning, EPFL. 51 / 56
AlexNet's first convolutional layer, first 20 filters.
―
Credits: Francois Fleuret, EE559 Deep Learning, EPFL. 52 / 56
Maximum response samples
Convolutional networks can be inspected by looking for synthetic input images x
that maximize the activation hℓ,d (x) of a chosen convolutional kernel u at layer ℓ
and index d in the layer filter bank.
These samples can be found by gradient ascent on the input space:
Here, Lℓ,d (x) represents the L2 norm of the activation hℓ,d (x) , x0 is the initial
random input, xt+1 is the updated input at iteration t + 1, and γ is the learning
rate.
53 / 56
VGG-16, convolutional layer 1-1, a few of the 64 filters
―
Credits: Francois Chollet, How convolutional neural networks see the world, 2016. 54 / 56
VGG-16, convolutional layer 2-1, a few of the 128 filters
―
Credits: Francois Chollet, How convolutional neural networks see the world, 2016. 54 / 56
VGG-16, convolutional layer 3-1, a few of the 256 filters
―
Credits: Francois Chollet, How convolutional neural networks see the world, 2016. 54 / 56
VGG-16, convolutional layer 4-1, a few of the 512 filters
―
Credits: Francois Chollet, How convolutional neural networks see the world, 2016. 54 / 56
VGG-16, convolutional layer 5-1, a few of the 512 filters
―
Credits: Francois Chollet, How convolutional neural networks see the world, 2016. 54 / 56
The network appears to learn a hierarchical composition of patterns:
-The first layers seem to encode basic features such as direction and color.
-These basic features are then combined to form more complex textures, such
as grids and spots.
-Finally, these textures are further combined to create increasingly intricate
patterns.
55 / 56
Biological plausibility
"Deep hierarchical neural networks are beginning to transform neuroscientists’
ability to produce quantitatively accurate computational models of the sensory
systems, especially in higher cortical areas where neural response properties
had previously been enigmatic."
Credits: Yamins et al, Using goal-driven deep learning models to understand sensory cortex, 2016.
56 / 56
The
end.
56 / 56