MODULE 2
-------------------------------------------------------------------------------------------------------------------------------------------
1. What is a Convolutional Neural Network (CNN)? Explain its working principle and major applications.
Definition
• Convolutional Neural Network also known as ConvNet or CNN is a deep learning technique that consists
of multiple numbers of layers.
• ConvNets are inspired by the biological visual cortex.
• The visual cortex has small regions of cells that are sensitive to specific regions of the visual field.
Different neurons in the brain respond to different features.
Working Principle
• For example, certain neurons fire only in the presence of lines of a certain orientation, some neurons fire
when exposed to vertical edges and some when shown horizontal or diagonal edges.
• This idea of certain neurons having a specific task is the basis behind ConvNets.
• The main concept of ConvNets is to obtain local features from input (usually an image) at higher layers
and combine them into more complex features at the lower layers.
• However, due to its multilayered architecture, it is computationally exorbitant and training such networks
on a large dataset takes several days.
• Therefore, such deep networks are usually trained on GPUs.
• Convolutional neural networks are so powerful on visual tasks that they outperform almost all the
conventional methods.
Applications
• ConvNets have shown excellent performance on several applications such as image classification, object
detection, speech recognition, natural language processing, and medical image analysis.
• Convolutional neural networks are powering core of computer vision that has many applications which
include self-driving cars, robotics, and treatments for the visually impaired.
---------------------------------------------------------------------------------------------------------------------------------------------
2. Discuss the evolution of CNN models from LeNet to ResNet. Highlight the key contributions of each
architecture.
Evolution of CNN Models from LeNet to ResNet
1. LeNet (1990s)
• LeNet was the first practical convolution-based architecture which used backpropagation for training the
network.
• It was designed to classify handwritten digits (MNIST), and it was adopted to read large numbers of
handwritten checks in the United States.
• The approach did not succeed much as it did not scale well to larger problems due to:
a) Small labeled datasets
b) Slow computers
c) Use of wrong nonlinearity (sigmoid/tanh) → vanishing gradient problem.
Key Contribution: Foundation of CNNs.
2. AlexNet (2012)
• AlexNet significantly outperformed all other methods in ILSVRC 2012, reducing classification error from
26% to 15%.
• The improvements came from:
a) Use of large labeled image database (ImageNet with 15M images)
b) Training on high-speed GTX 580 GPUs for 5–6 days
c) Use of ReLU activation function (avoids vanishing gradient, faster than sigmoid/tanh).
• Architecture: 5 convolutional layers, 3 pooling layers, 3 fully connected layers, and a 1000-way softmax
classifier.
Key Contribution: Large datasets, GPU training, and ReLU activation.
3. ZFNet (2013)
• Introduced as an improved version of CNN over AlexNet.
• Reduced first-layer filter size from 11×11 to 7×7 and stride from 4 to 2.
• This resulted in more distinctive features and fewer dead features, making it the winner of ILSVRC 2013.
Key Contribution: Better feature extraction with smaller filters.
4. VGGNet (2014)
• Introduced in 2014 and increased the depth of the network to 19 layers.
• Achieved depth by adding more 3×3 convolutional layers and 2×2 max-pooling layers with stride/padding
of 1.
• Reducing filter size and increasing depth produced more accurate results, achieving 7.32% error in
ILSVRC 2014 (runner-up).
Key Contribution: Depth with uniform small filters.
5. GoogLeNet (2015)
• Developed by Google, 22 layers deep, winner of ILSVRC 2015 with 6.7% error.
• Introduced the Inception module: parallel 3×3, 5×5 convolutions and pooling, outputs concatenated into
one vector.
• Used 1×1 convolutions to reduce computation, leading to 12× fewer parameters than AlexNet.
Key Contribution: Inception module and parameter efficiency.
6. ResNet (2015)
• Proposed by Microsoft Research Asia, 152 layers deep.
• Introduced residual connections: output of conv-relu-conv added to original input, helping gradient flow
and avoiding vanishing gradient.
• Won ILSVRC 2015 with 3.6% error (better than human error of 5–10%), while having fewer parameters
than VGGNet.
Key Contribution: Residual connections enabled very deep networks with state-of-the-art accuracy.
---------------------------------------------------------------------------------------------------------------------------------------------
3. Explain the convolution operation in CNN with mathematical expression and an example.
Definition of Convolution Operation
• Convolution is a mathematical operation performed on two functions and is written as (f * g), where f and
g are two functions.
• Convolution operation is one of the important operations used in digital signal processing and is used in
many areas which includes statistics, probability, natural language processing, computer vision, and image
processing.
• Convolution operation can be applied to higher dimensional functions as well.
• It can be applied to a two-dimensional function by sliding one function on top of another, multiplying and
adding.
Mathematical Expressions
• The output of the convolution operation for domain n is defined as
(f ∗ g) (n) = Σ f(m) g (n − m)
• For time-domain functions, n is replaced by t.
• The convolution operation is commutative in nature, so it can also be written as
(f ∗ g) (n) = Σ f (n − m) g(m)
• Let the 2D input (i.e., 2D image) be denoted by A, the 2D filter of size m × n be denoted by K, and the 2D
feature map be denoted by F.
• Here, the image A is convolved with the filter K and produces the feature map F.
• This convolution operation is denoted by A * K and is mathematically given as
F (i, j) = (A ∗ K) (i, j) = Σ Σ A (m, n) K (i − m, j − n)
• The convolution operation is commutative in nature, so we can write as
F (i, j) = (A ∗ K) (i, j) = Σ Σ A (i − m, j − n) K (m, n)
Example of Convolution Operation
• Convolution operation can be applied to images to perform various transformations; here, images are
treated as two-dimensional functions.
• An example of a two-dimensional filter, a two-dimensional input, and a two-dimensional feature map is
shown in Fig. 2.4.
• The kernel K is flipped relative to the input.
• If the kernel is not flipped, then convolution operation will be same as cross-correlation operation that is:
F (i, j) = (A ∗ K) (i, j) = Σ Σ A (i + m, j + n) K (m, n)
• Many CNN libraries use cross-correlation function as convolution function because cross-correlation is
more convenient to implement than convolution operation itself.
• According to Equation the operation computes the inner product (element wise multiplication) of the filter
at every location in the image.
-------------------------------------------------------------------------------------------------------------------------------------------
4. Explain the generic architecture of a CNN. Mention the role of each layer in detail.
Generic CNN Architecture
• Convolutional neural network (CNN) is a deep neural network architecture designed to process data that
has a known grid-like topology such as time-series data, images, speech signals, and videos.
• CNN architecture is inspired by the organization of the animal visual cortex and is made up of one or more
convolutional layers, pooling layers, activation function layers, fully connected layers, and output layers.
• CNN has three key features: local receptive field, weight sharing, and subsampling.
Layers in CNN and Their Roles
a) Convolutional Layer
o Convolution layer is the core building block of a convolutional neural network which uses convolution
operation (represented by *) in place of general matrix multiplication.
o Its parameters consist of a set of learnable filters also known as kernels.
o The main task of the convolutional layer is to detect features found within local regions of the input
image that are common throughout the dataset and mapping their appearance to a feature map.
o A feature map is obtained for each filter in the layer by repeated application of the filter across subregions
of the complete image, i.e., convolving the filter with the input image, adding a bias term, and then
applying an activation function.
o The input area on which a filter is applied is called local receptive field.
o The size of the receptive field is same as the size of the filter.
o Feature map is obtained after adding a bias term and then applying a nonlinear function to the output of
the convolution operation.
o The purpose of nonlinearity function is to introduce nonlinearity in the ConvNet model.
b) Activation Function Layer
o Activation functions are mathematical functions that transform the activation level of a neuron into an
output signal.
o The convolution layer takes an input image and extracts features from it and the activation function layer
takes that feature map and generates the activation map.
o Some of the common activation functions used in deep neural networks are Sigmoid, Tanh, ReLU,
Softmax, and Swish.
o ReLU activation function computes f(x) = max(0, x) and introduces nonlinearity to the network.
c) Pooling Layer
o Pooling layer is a down-sampling layer in ConvNet.
o It is used to reduce the spatial size of the representation to reduce the amount of parameters and
computation in the network.
o The pooling layer operates on each feature map independently and resizes it spatially.
o Max pooling is a pooling operation that selects the maximum element from the region of the feature map
covered by the filter.
d) Fully Connected Layer
o Fully connected layer in CNN is used to represent the classification stage of the network.
o Each neuron in the fully connected layer is connected to every neuron in the previous layer.
o Fully connected layer takes the high-level filtered images from the convolution and pooling layers and
output the class scores.
o The output of fully connected layers is fed into a classifier such as Softmax or SVM for classification.
e) Dropout Layer
o In order to overcome the problem of overfitting, a dropout layer can be introduced in the model in which
some neurons along with their connections are randomly dropped from the network during training.
o A reduced network is left; incoming and outgoing edges to a dropped-out node are also removed.
o Only the reduced network is trained on the data in that stage.
o The removed nodes are then reinserted into the network with their original weights.
o Dropout notably reduces overfitting and improves the generalization of the model.
-------------------------------------------------------------------------------------------------------------------------------------------
5. Describe the Convolution layer in detail. How does it help in feature extraction?
Description of Convolution Layer
• Convolution layer is the core building block of a convolutional neural network which uses convolution
operation (represented by) in place of general matrix multiplication.
• Its parameters consist of a set of learnable filters also known as kernels.
• A feature map is obtained for each filter in the layer by repeated application of the filter across subregions of the
complete image, i.e., convolving the filter with the input image, adding a bias term, and then applying an
activation function.
• The input area on which a filter is applied is called local receptive field.
• The size of the receptive field is same as the size of the filter.
• Feature map is obtained after adding a bias term and then applying a nonlinear function to the output of the
convolution operation.
• The purpose of nonlinearity function is to introduce nonlinearity in the ConvNet model, and there are a number
of nonlinearity functions available.
Filters and Kernels
• The weights in each convolutional layer specify the convolution filters and there may be multiple filters in each
convolutional layer.
• Every filter contains some feature like edge, corner, etc. and during forward pass, each filter is slid across the
width and height of the input generating feature map of that filter.
Hyperparameters of Convolution Layer
• Convolutional neural network architecture has many hyperparameters that are used to control the behavior of
the model.
• Some of these hyperparameters control the size of the output while some are used to tune the running time and
memory cost of the model.
• The four important hyperparameters in the convolution layer of the ConvNet are given below:
a. Filter Size: Filters can be of any size greater than 2 × 2 and less than the size of the input but the
conventional size varies from 11 × 11 to 3 × 3.
b. Number of Filters: There can be any reasonable number of filters. AlexNet used 96 filters of size 11 × 11 in
the first convolution layer. VGGNet used 96 filters of size 7 × 7, and another variant of VGGNet used 64
filters of size 11 × 11 in first convolution layer.
c. Stride: It is the number of pixels to move at a time to define the local receptive field for a filter. Stride of
one means to move across and down a single pixel.
d. Zero Padding: This hyperparameter describes the number of pixels to pad the input image with zeros. Zero
padding is used to control the spatial size of the output volume.
Feature Extraction Role
• The main task of the convolutional layer is to detect features found within local regions of the input image
that are common throughout the dataset and mapping their appearance to a feature map.
• Each filter in the convolution layer produces a feature map of size ([A−K +2P]/S)+1 where A is the input
volume size, K is the size of the filter, P is the number of padding applied and S is the stride.
• Therefore, the output volume will be of size corresponding to the number of filters applied, and the feature
maps preserve the extracted local features of the input image.
-------------------------------------------------------------------------------------------------------------------------------------------
6. What is the ReLU activation function? Explain its advantages over Sigmoid and Tanh functions.
Definition of ReLU
• Rectified Linear Unit (ReLU) has gained some importance in recent years and currently is the most
popular activation function for deep neural networks.
• ReLU simply computes the activation by thresholding the input at zero.
• In other words, a rectified linear unit has output 0 if the input is less than 0, and raw output otherwise.
• It is mathematically given as f(x) = max(0, x).
• Rectified linear unit activation function produces a graph which is zero when x < 0 and linear with slope 1
when x > 0.
Advantages over Sigmoid and Tanh
• Neural networks with ReLU train much faster than other activation functions like sigmoid and tanh.
-------------------------------------------------------------------------------------------------------------------------------------------
7. Explain the pooling layer in CNN. Different between max pooling and average pooling.
Pooling Layer in CNN
• In ConvNets, the sequence of convolution layer and activation function layer is followed by an optional
pooling or down-sampling layer.
• The purpose of pooling is to reduce the spatial size of the input and thus reduce the number of
parameters in the network.
• A pooling layer takes each feature map output from the convolutional layer and down-samples it, i.e.,
summarizes a region of neurons in the convolution layer.
• There are few pooling techniques available and the most common pooling technique is max-pooling.
• Max-pooling simply outputs the maximum value in the input region (usually 2×2).
• Other options for pooling layers are average pooling and L2-norm pooling.
• Pooling operation discards less significant data but preserves the detected features in a smaller
representation.
• The reasoning behind pooling: feature detection is more important than feature’s exact location.
• Limitation: this strategy works well for simple problems but does not work well for some problems.
Max Pooling vs Average Pooling
• Max Pooling: outputs the maximum value in the input region (most common pooling technique).
• Average Pooling: outputs the average value in the input region (also available, but less used).
Max pooling example:
Input output
5 7 6 5 7 6
2 3 4 9 7 6
7 0 2 1
4 2 6 0
Average Pooling example:
Input output
5 7 6 5 4.25 6.0
2 3 4 9 3.25 2.25
7 0 2 1
4 2 6 0
-------------------------------------------------------------------------------------------------------------------------------------------
8. What is a fully connected (FC) layer in CNN? Explain its role in classification tasks.
• Convolutional neural networks are composed of two stages: Feature extraction stage and classification stage.
• In ConvNets, the stack of convolution and pooling layers act as feature extraction stage while the classification
stage is composed of one or more fully connected layers followed by a softmax function layer.
• The process of convolution and pooling continues until enough features are detected.
• Next step is to make a decision based on these detected features.
• In case of classification problem, the task uses the detected features in the spatial domain to obtain probabilities
that these features represent each class, that is, obtain the class score.
• This is done by adding one or more fully connected layers at the end.
• In fully connected layer, each neuron from previous layer (convolution layer or pooling layer or fully connected
layer) is connected to every neuron in the next layer and every value contributes in predicting how strongly a
value matches a particular class.
• Like convolutional layers, fully connected layers can be stacked to learn even more sophisticated combinations
of features.
• The output of last fully connected layer is fed to a classifier which outputs the class scores.
• Softmax and Support Vector Machines (SVMs) are the two main classifiers used in ConvNets.
• Softmax classifier produces probabilities for each class with a total probability of 1, and SVM produces class
scores and the class having highest score is treated as the correct class.
-------------------------------------------------------------------------------------------------------------------------------------------
9. Compare and contrast Convolutional layer, Pooling layer, and Fully Connected layer in CNN architecture.
1. Convolutional Layer
• Convolution layer is the core building block of a convolutional neural network which uses convolution
operation (represented by *) in place of general matrix multiplication.
• Its parameters consist of a set of learnable filters also known as kernels.
• The main task of the convolutional layer is to detect features found within local regions of the input image
that are common throughout the dataset and mapping their appearance to a feature map.
• A feature map is obtained for each filter in the layer by repeated application of the filter across subregions
of the complete image, i.e., convolving the filter with the input image, adding a bias term, and then
applying an activation function.
• The input area on which a filter is applied is called local receptive field. The size of the receptive field is
same as the size of the filter.
• Each filter in the convolution layer produces a feature map of size ([A − K + 2P]/S) + 1 where A is the
input volume size, K is the size of the filter, P is the number of padding applied and S is the stride.
2. Pooling Layer
• In ConvNets, the sequence of convolution layer and activation function layer is followed by an optional
pooling or down-sampling layer to reduce the spatial size of the input and thus reducing the number of
parameters in the network.
• A pooling layer takes each feature map output from the convolutional layer and down-samples it, i.e.,
pooling layer summarizes a region of neurons in the convolution layer.
• Max-pooling simply outputs the maximum value in the input region.
• Pooling layer operation discards less significant data but preserves the detected features in a smaller
representation.
• The intuitive reasoning behind pooling operation is that feature detection is more important than feature’s
exact location.
3. Fully Connected Layer
• Convolutional neural networks are composed of two stages: Feature extraction stage and classification
stage.
• In ConvNets, the stack of convolution and pooling layers act as feature extraction stage while as the
classification stage is composed of one or more fully connected layers followed by a softmax function
layer.
• In fully connected layer, each neuron from previous layer (convolution layer or pooling layer or fully
connected layer) is connected to every neuron in the next layer and every value contributes in predicting
how strongly a value matches a particular class.
• Like convolutional layers, fully connected layers can be stacked to learn even more sophisticated
combinations of features.
• The output of last fully connected layer is fed to a classifier which outputs the class scores.
• Softmax and Support Vector Machines (SVMs) are the two main classifiers used in ConvNets.
-------------------------------------------------------------------------------------------------------------------------------------------
10. Define Dropout in deep learning. How does it prevent overfitting? Explain with an example.
Overfitting (Problem Statement)
• Deep neural networks consist of multiple hidden layers enabling it to learn more complicated features.
• It is followed by fully connected layers for decision-making.
• A fully connected layer is connected to all features, and it is prone to overfitting.
• Overfitting refers to the problem when a model is trained and it works so well on training data that it negatively impacts
the performance of the model on new data.
Definition of Dropout
• In order to overcome the problem of overfitting, a dropout layer can be introduced in the model in which some neurons
along with their connections are randomly dropped from the network during training.
• A reduced network is left; incoming and outgoing edges to a dropped-out node are also removed.
• Only the reduced network is trained on the data in that stage.
• The removed nodes are then reinserted into the network with their original weights.
How Dropout Prevents Overfitting
• Dropout notably reduces overfitting and improves the generalization of the model.
Example
• In dropout, some neurons along with their connections are randomly dropped from the network during
training, leaving a reduced network which is trained on the data in
that stage.
• The removed nodes are then reinserted into the network with their original weights.
-------------------------------------------------------------------------------------------------------------------------------------------