Machine Learning and Deep Learning

(ITDLC6031)

Module 4:
Convolutional Neural Networks

Course In-charge:
Dr. Radhika Kotecha
Professor and Head, Department of Information Technology
Contents
Total
COs Hours /
Module Subtopics Hours /
Mapped Subtopic
Module
Introduction to Convolutional Neural
Networks (CNN), Components, ReLU
CO4 02
Layer, Training a Convolutional
Network.
Case Studies of Convolutional
07
Architectures: AlexNet, GoogLeNet,
CO4 03
ResNet, The Effects of Depth, Pre-
trained Models.
Applications of CNN. CO4 02

2
Convolutional Neural Networks (CNN)
• Yann LeCun, director of Facebook’s AI Research Group, is
the pioneer of convolutional neural networks.
• He built the first convolutional neural network called LeNet
in 1988. LeNet was used for character recognition tasks like
reading zip codes and digits.
• A convolutional neural network is a feed-forward neural
network that is generally used to analyze visual images by
processing data with grid-like topology. It’s also known as a
ConvNet.
• A convolutional neural network is used to detect and
classify objects in an image.

3
Convolutional Neural Networks (CNN)
• In CNN, every image is represented in the form of an array of pixel values.

4
Analogy
• Using a Feed-forward Neural Network
• Extensive Computation

Source: Codebasics

5
Analogy
• Step-wise Detection

6
Analogy
• Step-wise Detection

7
Analogy
• Step-wise Detection

8
Analogy
• Step-wise Detection

9
Analogy
• Step-wise Detection

10
Analogy
• Step-wise Detection

11
Analogy

Source: Codebasics

12
Layers in CNN
• A convolution neural network has multiple hidden layers that help in extracting
information from an image.
• The four important layers in CNN are:
• Convolution layer
• Activation layer
• Pooling layer
• Fully Connected layer

13
Convolution Layer
• Convolution:
• Convolution is the process involving combination of two functions that
produces the other function as a result.
• In CNN’s, the input image is subjected to convolution with use of
filters that produces a Feature map.

• Filters / Kernels:
• Filters are randomly generated / customized vectors in the network
consisting of weights and biases.
• The same weights and bias are shared among various neurons in
CNN instead of unique weights and bias for each neuron.
• Many filters can be generated where every filter captures unique
feature from input.
• Filters are also referred as Kernels.

14
Convolution Layer Operation
• Convolution layer generally has:
• Input vectors (Image Feature Map)
• Filters (Feature Detector)
• Output vectors (Output Feature Map)

15
Convolution Operation

16
Convolution Operation (3 Channels)

17
Calculating Output Size

18
Calculating Output Size

19
Activation Layer: ReLU
• ReLU stands for the rectified
linear unit.
• Once the feature maps are
extracted, the next step is to move
them to an Activation: ReLU layer.
This is referred to as activation,
because only the activated
features are carried forward into
the next layer.
• ReLU performs an element-wise
operation and sets all the negative
pixels to 0.
• It introduces non-linearity to the
network, and the generated output
is a rectified feature map.

20
Pooling Layer
• Pooling is a down-sampling operation that reduces the
dimensionality of the feature map.
• The rectified feature map now goes through a pooling layer
to generate a pooled feature map.

21
Pooling Layer
• Pooling is a down-sampling operation that reduces the
dimensionality of the feature map.
• The rectified feature map now goes through a pooling layer
to generate a pooled feature map.

22
Pooling Layer
• Pooling is a down-sampling operation that reduces the
dimensionality of the feature map.
• The rectified feature map now goes through a pooling layer
to generate a pooled feature map.

23
Flattening
• The next step in the process is called
flattening.
• Flattening is used to convert all the
resultant 2-Dimensional arrays from
pooled feature maps into a single long
continuous linear vector.

24
Overall CNN Architecture

25
Hyper-parameters of CNN
• Size of Filters
• No. of Filters/Kernels
• Stride
• Padding

26
Multiple Filters
• Convolutional neural networks do not learn a single filter;
they, in fact, learn multiple features in parallel for a given
input.
• For example, it is common for a convolutional layer to learn
from 32 to 512 filters in parallel for a given input.
• The number of filters affects the depth of the output. For
example, three distinct filters would yield three different
feature maps, creating a depth of three.

27
Multiple Features through Multiple Filters
• We can use multiple filters to detect various features simultaneously.
• Let us consider the following example in which we see vertical edge &
curve in the input RGB image.
• We will have to use two different filters for this task, and the output
image will thus have two feature maps.

28
Strides
• Stride is the distance, or number of pixels, that the kernel moves over the
input matrix.
• If stride is set to 1, filter moves across 1 pixel at a time and if stride is 2,
filter moves 2 pixels at a time.
• More the value of stride, smaller will be the resulting output and vice
versa.

29
Padding
• Padding is used to make dimension of output equal to input
by adding zeros to the input frame of matrix. Padding allows more
spaces for kernel to cover image and is accurate for analysis of
images. Due to padding, information on the borders of images are
also preserved similarly as at the center of image.

30
Padding
There are three types of padding:
• Valid padding: This is also known as no padding. In this
case, the last convolution is dropped if dimensions do not
align.
• Same padding: This padding ensures that the output layer
has the same size as the input layer
• Full padding: This type of padding increases the size of the
output by adding zeros to the border of the input.

31
Various CNN Architectures
• LeNet

• AlexNet

• VGG

• GoogleNet

• ResNet

32
LeNet-5
• Lenet-5 is one of the earliest pre-trained models
proposed by Yann LeCun and others in the year
1998, in the research paper Gradient-Based Learning
Applied to Document Recognition.
• They used this architecture for recognizing the
handwritten and machine-printed characters.
• The main reason behind the popularity of this model
was its simple and straightforward architecture. It is a
multi-layer convolution neural network for image
classification.

33
LeNet-5 (Alternate Architecture)

34
AlexNet — ImageNet Classification with
Deep Convolutional Neural Networks

35
Problem with Sigmoid for Multi-class
Classification Problems

36
Softmax Activation Function
• The Softmax activation function calculates the relative
probabilities.
• That means it uses the value of Z21, Z22, Z23 to
determine the final probability value.

• Here, the Z represents the values from the neurons of

37
Softmax Activation Function

38
Softmax Activation Function

39
Issues in Deep Learning & Solutions
Issues:
• Bigger the model, more prone it is to overfitting. This is
particularly noticeable when the training data is small
• Increasing the number of parameters means you need to
increase your existing computational resources

Solution 1:
• 1×1 convolution: The inception architecture uses 1×1
convolution. These convolutions are used to decrease the
number of parameters (weights and biases) of the
architecture.

40
Benefit of 1x1 Convolution
• For Example, if we want to perform 5×5 convolution having
48 filters (without using 1×1 convolution as intermediate)
while maintaining the same height and width (add padding to
the max-pooling layer):

• Total Number of operations

Operations = (output height × output width × number of filters) × (kernel height × kernel width × input channels)

41
Benefit of 1x1 Convolution
• With 1×1 convolution:

• Total Number of operations

42
Inception Module – Naïve Version

43
Inception Module – With Dimension Reduction

44
GoogLeNet
• The ILSVRC 2014 winner was a
Convolutional Network from Szegedy et
al. from Google.
• Its main contribution was the development of
an Inception Module that dramatically reduced
the number of parameters in the network (4M,
compared to AlexNet with 60M).
• Additionally, this paper uses Average Pooling
instead of Fully Connected layers at the top of
the ConvNet, eliminating a large amount of
parameters that do not seem to matter much.
• There are also several followup versions to
the GoogLeNet, most recently Inception-v4

45
Issue with Going Deeper

• In the above plot, we can observe that a 56-layer CNN gives more error rate on
both training and testing dataset than a 20-layer CNN architecture.
• After analyzing more on error rate the authors were able to reach conclusion that
it is caused by vanishing/exploding gradient.
• ResNet, which was proposed in 2015 by researchers at Microsoft Research
introduced a new architecture called Residual Network.

46
Residual Network
• In order to solve the problem of the
vanishing/exploding gradient, this
architecture introduced the concept
called Residual Blocks that uses a
technique called skip connections.
• The skip connection connects
activations of a layer to further layers
by skipping some layers in between.
This forms a residual block.
• Resnets are made by stacking these
residual blocks together. The approach
behind this network is instead of layers
learning the underlying mapping, we
allow the network to fit the residual
mapping.
• Any layer reduces the performance of
architecture then it will be skipped by
regularization.

47
ResNet - 34 architecture

48
Example Questions [1/2]
1. Explain the issue with going deeper with CNN. (Hint: Vanishing Gradient /
Exploding Gradient problem from Module 3 slides).
2. Explain various layers of CNN.
3. Explain the generic architecture of CNN.
4. Explain hyper-parameters of CNN.
5. Apply Convolution operation on the given Input Feature Map using the
given Convolution Filter and derive the Output Feature Map. (Numerical)
6. Apply CNN for hand-written digit recognition (digits 0-9) / character
recognition (alphabets a-z) & explain its various layers.
7. An RGB image of size 6 × 6 × 3 is processed using two filters of size 3 ×
3 × 3 with stride 1 and no padding. Find the output size.
8. Consider an image of size 12 × 12 × 420 and apply 5 × 5 convolution
having 36 filters — with and without using 1 × 1 convolution as
intermediate, while maintaining the same height and width / with no
padding (Numerical)
9. Apply Convolution operation on a given Input Feature Map using the
given Convolution Filter and derive the Output Feature Map. (Numerical).

49
Example Short Questions [2/2]
1. Explain the role of Kernels in CNN.
2. Explain the need of Padding in CNN.
3. Explain types of Padding in CNN.
4. Apply Max Pooling and Average Pooling on the given Feature Map using
Pool Size 2 x 2.
5. Explain Strides in context of CNN.
6. Explain Flattening in CNN.
7. List various CNN architectures.
8. Explain the role of Inception Module.
9. Elaborate the benefit and need of inception module.
10. Explain Residual Networks.
11. Explain the problem with Sigmoid for Multi-class Classification Problems.
12. Explain SoftMax activation function.
13. Explain the role of multiple features.
14. Explain ReLu activation function.

50
References
1. S. Rose, L. Kumar, D Renuka, Deep Learning using Python, Wiley.
2. I. Goodfellow, Y. Bengio and A. Courville, Deep Learning, MIT
Press.
3. T. Mitchell, Machine Learning, McGraw Hill.
4. C. Aggarwal, Neural Networks and Deep Learning: A Textbook,
Springer.
5. J. Han, M. Kamber, Data Mining Concepts and Techniques, Morgan
Kaufmann.
6. M. Nielsen, Neural Networks and Deep Learning, Determination
Press.
7. Online Blogs and Videos

51