Module 2

Convolutional Neural Network

O. V. Ramana Murthy
 Contents
 Convolution operations
 Pooling layer
 Variants of the basic Convolution Function,
 Efficient Convolution algorithms,
 Random and unsupervised features,
 Neuro Scientific Basis for Convolutional Networks
Reference:
9.1, 9.3, 9.5, 9.8, 9.9, 9.10
Ian Goodfellow, Yoshua Bengio, Aaron Courville, Deep
Learning, MIT Press, 2016
2
 Recap: Shallow NN

Input Value1

Input Value2 Output Value

Input Value3

Input Layer
3
 Recap: Deep NN
.. .. ..
Input Value1
.. .. ..

Input Value2
.. .. ..

Input Value3
.. .. ..

Input Layer .. .. ..

Hidden Layer

4
 Convolution Neural Network

Source: [Link]

5
 Convolution

6
 Convolution
To apply the convolution:
 Overlay the Kernel on the Image: Start from the top-left corner of
the image and place the kernel so that its center aligns with the
current image pixel.
 Element-wise Multiplication: Multiply each element of the kernel
with the corresponding element of the image it covers.
 Summation: Sum up all the products obtained from the element-
wise multiplication. This sum forms a single pixel in the output
feature map.
 Continue the Process: Slide the kernel over to the next pixel and
repeat the process across the entire image.

7
 Convolution

8
9
 CNN Terms – Convolution Layers
Generally, the number of kernel channels are always identical
to the number of input channels.

𝑊𝑖𝑑𝑡ℎ − 𝐹𝑖𝑙𝑡𝑒𝑟𝑠𝑖𝑧𝑒 + (2 × 𝑃𝑎𝑑𝑑𝑖𝑛𝑔)

𝑜𝑢𝑡𝑝𝑢𝑡 = +1
𝑆𝑡𝑟𝑖𝑑𝑒

10
 Convolution
Some types of filters are
1. Blurring filter ones(3,3)/9
2. Sharpening filter [0 -1 0;-1 5 -1;0 -1 0]
3. Horizontal Sobel filter for edge detection
[-1 -2 -1; 0 0 0; 1 2 1]
4. Vertical Sobel filter for edge detection
[-1 0 1;-2 0 2;-1 0 1]

11
 Convolution
 The convolution layer operates in a very different way
compared to the other neural network layers.
 This layer does not employ connection weights and a
weighted sum. Instead, it contains filters that convert images.
We will call these filters convolution filters. They usually come
in 5  5 or 33 matrices to perform convolution operations,
sometimes including an optional trainable bias.
 These convolution operations involved moving the kernels
over the input in steps called strides (spaces the kernels
skipped between each convolution).

12
 convolution2dLayer

13
 convolution2dLayer
 The process of the inputting the image through the
convolution filters yields the feature map.
 The feature map that the convolution filter creates is
processed through the activation function before the layer
yields the output.
 The activation function of the convolution layer is identical to
that of the ordinary neural network.
 Although the ReLU function is used in most of the recent
applications, the sigmoid function and the tanh
function are often employed as well

14
 convolution2dLayer
Convolutions can also be applied to the output of other
convolutional layers or pooling layers.
In deep learning, the masks are determined as part of the
learning process. Each pixel in a mask has a weight and may
have a bias; these are computed from the learning data.
Convolution should be highlighting important features in the
data. Subsequent convolution layers narrow down features.
The function has two inputs: the filterSize, specifying the height
and width of the filters as either a scalar or an array of [h
w]and numFilters, the number of filters.

15
 Variants of Convolution
 Standard
 Stride
 Tiles

16
 Tiled Convolution
The Image is split into tiles (or sub-blocks).The kernel is
applied independently to each tile. Tile outputs are aggregated
to form the final feature map.
Reduces computational complexity by focusing on smaller
regions. Optimized for memory access patterns, improving
performance in hardware implementations e.g. GPU. May
introduce padding between tiles to account for boundary
effects.

17
 Example
1 2 3
1 0
𝐼 = 4 5 6 ;𝐾 =
0 −1
7 8 9
Stride Convolution Result (Stride = 2):

Stride Convolution Result (Stride = 1):

𝑂=
Tile Convolution Result (Tile size = 2):

18
 Example
1 2 3
1 0
𝐼 = 4 5 6 ;𝐾 =
0 −1
7 8 9
Stride Convolution Result (Stride = 2):
[-4]
Stride Convolution Result (Stride = 2):
−4 −4
𝑂=
−4 −4
Tile Convolution Result (Tile size = 2):

19
 Efficient Convolution Algorithms
 Fourier Transform
 Separable Convolution

20
 Fourier Transform

21
 Fourier Transform
1. Fourier Transform: Transform the input image and the
kernel (mask) to the frequency domain.
2. Pointwise Multiplication: Multiply the transformed image
and the transformed kernel element-wise in the frequency
domain.
3. Inverse Fourier Transform: Transform the result back to the
spatial domain.

22
 Fourier Transform
1 2 3
1 1
𝐼 = 4 5 6 ;𝐾 = ;
1 1
7 8 9
1. Pad the Matrices
Both the image and kernel need to be padded to the same size to avoid
boundary issues. The size will be the sum of the dimensions of I and K
minus 1:
Output Size=(3+2−1)×(3+2−1)=4×4
1 2 3 0 1 1 0 0
𝐼𝑝𝑎𝑑 = 4 5 6 0 ; 𝐾𝑝𝑎𝑑 = 1 1 0 0
7 8 9 0 0 0 0 0
0 0 0 0 0 0 0 0
23
 Fourier Transform (Appendix)
2. Compute the Fourier Transform
𝐼𝐷𝐹𝑇 = 𝑓𝑓𝑡2 𝐼𝑝𝑎𝑑 ; 𝐾𝐷𝐹𝑇 = 𝑓𝑓𝑡2 𝐾𝑝𝑎𝑑
45 −6 − 15𝑗 15 −6 + 15𝑗
4 −5 + 8𝑗 −6 − 5𝑗 5 − 4𝑗
𝐼𝑝𝑎𝑑= ;
15 −2 − 5𝑗 5 −2 + 5𝑗
−18 + 15𝑗 5 + 4𝑗 −6 + 5𝑗 −5 − 8𝑗
4 2 − 2𝑗 0 2 + 2𝑗
2 − 2𝑗 −2𝑗 0 2
𝐾𝑝𝑎𝑑 =
0 0 0 0
2 + 2𝑗 2 0 2𝑗
3. Pointwise Multiplication in the Frequency Domain
𝑀𝑓𝑓𝑡 = 𝐼𝐷𝐹𝑇 ⨀𝐾𝐷𝐹𝑇
180 −42 − 18𝑗 0 −42 + 18𝑗
−66 + 6𝑗 16 + 10𝑗 0 10 − 8𝑗
𝑀𝑓𝑓𝑡 =
0 0 0 0
24
−66 − 6𝑗 10 + 8𝑗 0 16 − 10𝑗
 Fourier Transform
4. Inverse Fourier Transform
Inverse Fourier Transform to get convolution output
12 16 9 6
𝑀 = 𝑖𝑓𝑓𝑡2 𝑀𝑓𝑓𝑡 = 24 28 15 10
21 24 9 6
7 8 3 2
The final valid region corresponds to the TOP 2×2 convolution result
12 16
24 28
Output by standard convolution
12 16
24 28
25
 Separable Convolution

26
 Separable Convolution
• Breaks a 2D kernel into two 1D kernels (e.g., M×1 and 1×M)
• Applies one kernel in one direction (e.g., horizontal) and the
other in the perpendicular direction (e.g., vertical).
• Reduces the number of multiplications significantly.
Preconditions for kernel
The kernel must be separable, meaning it can be decomposed
into the outer product of two smaller 1D kernels.
Mathematically: K(x, y)=Kx(x)⋅Ky(y)
(Symmetry or repetitive structure in the kernel can often
indicate separability.)

27
 Separable Convolution
1 2 3
1 1
𝐼 = 4 5 6 ;𝐾 = ;
1 1
7 8 9
1
Step 1: Apply row wise
1
𝑂ℎ =
Step 2: : Apply 1 1 column wise
𝑂=
The direct convolution result matches the separable convolution result

28
 Separable Convolution
1 2 3
1 1
𝐼 = 4 5 6 ;𝐾 = ;
1 1
7 8 9
1
Step 1: Apply row wise
1
5 7 9
𝑂ℎ =
11 13 15
Step 2: : Apply 1 1 column wise
12 16
𝑂=
24 28
The direct convolution result matches the separable convolution result

29
 FFT vs Normal vs Separable
Image (N×N) and the kernel (M×M)
 Standard Convolution (Direct Method):
Time complexity: O(N2⋅M2)
Situations where precise spatial relationships must be maintained
 FFT-Based Convolution:
Total: O(N2logN)
FFT becomes more efficient when M (kernel size) approaches N
(image size).
 Separable Convolution
Time complexity: O(2N2⋅M)
Used for efficiency in computationally intensive tasks. e.g., in mobile
30 or embedded devices architectures like MobileNet.
 Subsampling Layers
Two-dimensional subsampling layers are non-trainable kernels
or windows to down-sample input features.
This typically reduced the size of the features significantly .
Average Pooling and Max Pooling.
Both methods compute either the average or maximum of the
values present in each kernel to be included in the resulting
feature map.
The kernel moves over each (e.g. 2×2) input section in a non-
overlapping motion, shifting (e.g. 2) places between each
operation.

31
 maxPooling2dLayer
The pooling layer reduces the size of the image, as it combines
neighboring pixels of a certain area of the image into a single
representative value.

The
representative
value is usually
set as the mean
or maximum of
the neighboring
pixels

32
 Subsampling Layers

𝑊𝑖𝑑𝑡ℎ − 𝐹𝑖𝑙𝑡𝑒𝑟𝑠𝑖𝑧𝑒 + (2 × 𝑃𝑎𝑑𝑑𝑖𝑛𝑔)

𝑜𝑢𝑡𝑝𝑢𝑡 = +1
𝑆𝑡𝑟𝑖𝑑𝑒

33
 Advantages
 Dimensionality reduction: They help in reducing the spatial
dimensions of the feature maps. This reduces the computational
cost and also helps in avoiding overfitting by reducing the number
of parameters in the model.
 Translation invariance: Pooling layers are also useful in achieving
translation invariance in the feature maps. This means that the
position of an object in the image does not affect the classification
result, as the same features are detected regardless of the position
of the object.
 Feature selection: Pooling layers can also help in selecting the
most important features from the input, as max pooling selects the
most salient features and average pooling preserves more
information.

34
 Disadvantages
 Information loss: They discard some information from the input
feature maps, which can be important for the final classification or
regression task.
 Over-smoothing: Pooling layers can also cause over-smoothing of
the feature maps, which can result in the loss of some fine-grained
details that are important for the final classification or regression
task.
 Hyperparameter tuning: Pooling layers also introduce
hyperparameters such as the size of the pooling regions and the
stride, which need to be tuned in order to achieve optimal
performance. This can be time-consuming and requires some
expertise in model building.

35
 Fully Connected Layers
They typically are included as the last few layers of most CNNs,
appearing after several convolution and subsampling operations
were performed.
Fully connected layers are independent neural networks that
possess one or more hidden layers.
Their operations involve multiplying their inputs by trainable
weight vectors, with a trainable bias sometimes summed to
those results.
The output of these layers was traditionally sent through
activation functions, similarly to convolution layers.

36
 fullyConnectedLayer
The fully connected layer connects all of the inputs to the
outputs with weights and biases. For example:
layer = fullyConnectedLayer(10);
creates ten outputs from any number of [Link] don’t have
to specify the inputs. Effectively, this is the equation:
y = ax + b
If there are m inputs and n outputs, b is a column bias matrix of
length n and a is n by m.

37
 CNN Terms - Fully Connected Layers

38
 softmaxLayer
softmax finds a maximum of a set of values using the
logistic function. If 𝑣𝑖 is output of previous layer
𝑒 𝑣𝑖
The set of values are 𝑝𝑘 = 𝜑 𝑣𝑖 = σ𝑀 𝑣𝑖
𝑖=1 𝑒

This is just a method of smoothing the inputs.

Softmax is used for multiclass classification because it
guarantees a well-behaved (the sum of the probabilities is 1.)
probability distribution.

39
 Convolution Neural Network

Source: [Link]

40
 How CNNs Work
 Layers: Lower layers capture basic features, while deeper layers
identify more complex patterns like parts of objects or entire
objects.
 Learning Process: CNNs learn the filters during training. The
network adjusts the filters to minimize the loss between the
predicted and actual outcomes, thus optimizing the feature
extraction process.
 Pooling Layers: After the convolution operations, pooling takes
place, which reduces the spatial size of the representation, while
preserving important information.
 Activation Functions: Neural networks use activation functions,
like ReLU (Rectified Linear Unit), at the end to introduce non-
linearities. This helps the model learn more complex patterns.

41
 AlexNet (ImageNet)

[Link]

42
 AlexNet

[Link]

43
 Random Features
 Initialize the Convolution layers RANDOMLY.
 During training, update only the weights of the densely
connected layer

44
 Random Features (HW: Compare)
# Build the CNN model
model = Sequential([ Conv2D(32, kernel_size=(3,
3), activation='relu', input_shape=(28, 28, 1)),
MaxPooling2D(pool_size=(2, 2)), Flatten(),
Dense(128, activation='relu'), Dense(10,
activation='softmax') ])

# Freeze the convolutional layers

for layer in [Link][:-2]:
# Exclude the dense layers
[Link] = False
# Compile the model
[Link](optimizer='adam',
loss='categorical_crossentropy’,
45 metrics=['accuracy'])
 Unsupervised features,
1. Extract Patches from the Training Images: Divide the input
images into small patches of the same size as the
convolutional filters.
2. Apply K-Means Clustering: Use k-means to find cluster
centers for the patches, which serve as the initial weights
for the convolution filters.
3. Assign K-Means Centers as Convolution Weights: Use the
cluster centers to initialize the convolutional filters.
4. Freeze the Convolutional Layers: Set the trainable attribute
of the convolutional layers to False.

46
 Unsupervised features,
# Function to extract patches from images def
extract_patches(images, patch_size):
patches = []
for img in images:
for i in range([Link][0] - patch_size + 1):
for j in range([Link][1] - patch_size + 1):
patch = img[i:i + patch_size, j:j+ patch_size]
[Link](patch)
return [Link](patches)
# Extract 3x3 patches from the training
imagespatch_size = 3
patches = extract_patches(x_train[:, :, :, 0],
patch_size)
# Remove channel dimension for clustering
47
 Unsupervised features,
# Flatten patches for clustering
patches_flat =
[Link]([Link][0], -1)
# Apply K-Means clustering
num_filters = 32
kmeans = KMeans(n_clusters=num_filters,
random_state=42)
[Link](patches_flat)
# Get the cluster centers and reshape to
convolutional filter
shapefilters =
kmeans.cluster_centers_.reshape(num_filters,
patch_size, patch_size, 1)
48
 Unsupervised features,
# Build the CNN model
model = Sequential([ Conv2D(num_filters,
kernel_size=(3, 3), activation='relu',
input_shape=(28, 28, 1)),
MaxPooling2D(pool_size=(2, 2)), Flatten(),
Dense(128, activation='relu'), Dense(10,
activation='softmax’)])
# Initialize the convolutional layer with k-
means
[Link][0].set_weights([filters,
[Link](num_filters)])[Link][0].trainab
le = False
# Freeze the convolutional layer
49
 Neuroscientific Basis for CNNs
In Convolutional Neural Networks (CNNs), specific layers can
be loosely compared to different regions of the brain based on
their function and hierarchical processing:

50
 Neuroscientific Basis for CNNs
Early Layers (Input
and Initial
Convolutional
Layers)
1. Brain Analogy:
Visual Cortex (V1)
2. Function: Detect
low-level features
like edges, textures,
and simple shapes,
similar to how the
primary visual
cortex processes
basic visual stimuli.
51
 Neuroscientific Basis for CNNs
Intermediate Layers
1. Brain Analogy:
Extrastriate Cortex
(e.g., V2, V4)
2. Function: Detect
more complex
patterns and
combinations of
features, akin to
how the extrastriate
regions process
complex forms and
object parts.

52
 Neuroscientific Basis for CNNs
Deeper Layers (Fully
Connected Layers or
Deep Convolutions)
• Brain Analogy:
Inferotemporal
Cortex (IT)
• Function: Recognize
abstract features and
entire objects,
analogous to the IT
cortex's role in object
recognition and
categorization.
53
 Neuroscientific Basis for CNNs
Output Layer
• Brain Analogy:
Decision-Making
Areas (e.g., Prefrontal
Cortex)
• Function: Perform
classification or
decision-making based
on the learned
features, similar to
how the prefrontal
cortex integrates
information for
cognitive tasks.
54
55
 Neuroscientific Basis for CNNs

56
 Thank You All Very Much

57
 Appendix – DFT Matrix
Fourier Transform of a matrix is obtained using DFT matrix as
𝐼𝐷𝐹𝑇 = 𝑓𝑓𝑡2 𝐼𝑝𝑎𝑑 = 𝐹𝐹𝑇4 × 𝐼𝑝𝑎𝑑 × 𝐹𝐹𝑇4𝑇
1 1 1 1
Where 𝐹𝐹𝑇4 =
1 𝜔 𝜔2 𝜔3 ;
1 𝜔2 𝜔4 𝜔6
1 𝜔
2𝑗𝜋
3 𝜔6 𝜔9
𝜔 = 𝑒− = 𝑗(imaginary unit)
4
1 1 1 1
1 −𝑗 −1 𝑗
𝐹𝐹𝑇4 =
1 −1 1 −1
1 𝑗 −1 −𝑗
58
 Appendix – DFT Matrix
Inverse Fourier transform is obtained by Conjugate transpose (H)
1
𝐼𝑝𝑎𝑑 = 𝑖𝑓𝑓𝑡2 𝐼𝐷𝐹𝑇 = 𝐹𝐹𝑇4𝐻 × 𝐼𝐷𝐹𝑇
(𝑁 =)4
1 1 1 1
𝐻 1 𝑗 −1 −𝑗
Where 𝐹𝐹𝑇4 =
1 −1 1 −1
1 −𝑗 −1 𝑗

59