Deep Learning & Reinforcement Learning- BAI701

MODULE-2
Basics of Supervised Deep Learning

2.1 Introduction

-The use of supervised and unsupervised deep learning models has grown
at a fast rate due to their success with learning of complex problems.

- High-performance computing resources, availability of huge amounts of

data (labeled and unlabeled) and state-of-the-art open-source libraries are
making deep learning more and more feasible for various applications.

- Convolutional Neural Network (CNN or ConvNets) is one of the most

commonly used supervised deep learning models.

·
2.2 Convolutional Neural Network (ConvNet/CNN):
-Convolutional Neural Network also known as ConvNet or CNN is a
deep learning technique that consists of multiple numbers of layers.

-ConvNets are inspired by the biological visual cortex.

-The visual cortex has small regions of cells that are sensitive to specific
regions of the visual field.

-Different neurons in the brain respond to different features.

-For example, certain neurons fire only in the presence of lines of a

certain orientation, some neurons fire when exposed to vertical edges and
some when shown horizontal or diagonal edges.

Prof. Deepali A Dixit, Dept of AI & ML ,RRCE 1

Deep Learning & Reinforcement Learning- BAI701

-This idea of certain neurons having a specific task is the basis behind
ConvNets.

-ConvNets have shown excellent performance on several applications

such as image classification, object detection, speech recognition, natural
language processing, and medical image analysis.

-Convolutional neural networks are powering core of computer vision

that has many applications which include self-driving cars, robotics, and
treatments for the visually impaired.

-The main concept of ConvNets is to obtain local features from input

(usually an image) at higher layers and combine them into more complex
features at the lower layers.

-However, due to its multilayered architecture, it is computationally

exorbitant and training such networks on a large dataset takes several
days.

-Therefore, such deep networks are usually trained on GPUs.

Convolutional neural networks are so powerful on visual tasks that they
outperform almost all the conventional methods.

2.3 Evolution of Convolutional Neural Network

Models:
- LeNet: The first practical convolution-based architecture was LeNet

which used backpropagation for training the network. LeNet was

designed to classify handwritten digits (MNIST), and it was adopted to
read large numbers of handwritten checks in the United States.

Prof. Deepali A Dixit, Dept of AI & ML ,RRCE 2

Deep Learning & Reinforcement Learning- BAI701

- Unfortunately, the approach did not get much success as it did not scale
well to larger problems.

- The main reasons for this limitation were as follows:

a. Small labeled datasets.
b. Slow computers.
c. Use of wrong nonlinearity (activation) function.

- The use of appropriate activation function in a neural network has huge

impact on the final performance. Any deep neural network that uses a
nonlinear activation function like sigmoid or tanh and is trained using
backpropagation suffers from vanishing gradient.

- Vanishing gradient is a problem found in training the neural networks

with gradient-based training methods. Vanishing gradient makes it hard
to train and tune the parameters of the top layers in a neural network. The
problem worsens as the total number of layers in the network increases.

• AlexNet: The first breakthrough came in 2012 when the convolutional

model which was named AlexNet significantly outperformed all other

conventional methods in ImageNet Large-Scale Visual Recognition
Competition (ILSVRC) 2012 that featured the ImageNet dataset.

The AlexNet brought down classification error rate from 26 to 15%, a

significant improvement at that time. AlexNet was simple but much more
efficient than [Link] improvements to overcome the above mentioned
problems were due to the following reasons:

A) Large labeled image database (ImageNet), which contained around 15

million labeled images from a total of over 22,000 categories, was used.
B) The model was trained on high-speed GTX 580 GPUs for 5 to 6 days.

Prof. Deepali A Dixit, Dept of AI & ML ,RRCE 3

Deep Learning & Reinforcement Learning- BAI701

C) ReLU (Rectified Linear Unit) f (x) max(x, 0) activation function

was used. This activation function is several times faster than the
conventional activation functions like sigmoid and tanh. The ReLU
activation function does not experience the vanishing gradient problem.

- AlexNet consists of five convolutional layers, three pooling layers, three

fully connected layers, and a 1000-way softmax classifier.

- ZFNet: In 2013, an improved version of CNN architecture called ZFNet

was introduced.

·- ZFNet reduced the filter size in the first layer from 11 × 11 to 7 × 7 and
used a stride of 2 instead of 4 which resulted in more distinctive features
and fewer dead features. ZFNet turned out to be the winner of ImageNet
Large-Scale Visual Recognition Challenge (ILSVRC) 2013.

VGGNet: VGGNet, introduced in 2014.· VGGNet, introduced in 2014,

used increased depth of the network for improving the [Link] depth
of the network was made 19 layers by adding more convolutional layers
with 3 × 3 filters, along with 2 × 2 max-pooling layers with stride and
padding of 1 in all layers.

Reducing filter size and increasing the depth of the network resulted in
CNN architecture that produced more accurate results.

VGGNet achieved an error rate of 7.32% in ILSVRC 2014 and was the
runner-up model in ILSVRC 2014.

-GoogLeNet: Google developed a ConvNet model called GoogLeNet in

2015. -The model has 22 layers and was the winner of ILSVRC 2015 for
having the error rate of 6.7%. It uses an inception module which helps in
reducing the number of parameters in the network.

Prof. Deepali A Dixit, Dept of AI & ML ,RRCE 4

Deep Learning & Reinforcement Learning- BAI701

-The inception module is actually a concatenated layer of convolutions (3

× 3 and 5 × 5 convolutions) and pooling sub-layers at different scales
with their output filter banks concatenated into a single output vector
making the input for the succeeding stage. These sub-layers are not
stacked sequentially but connected in parallel. As in figure 2.1

-1 × 1 convolution is used that results in reduced computations before

expensive 3 × 3 and 5 × 5 convolutions are performed. GoogLeNet model
has two convolutional layers, four max-pooling layers, nine inception
layers, and a softmax layer. GoogLeNet has 12 times lesser parameters
than AlexNet. Increasing the number of layers increases the number of
features which enhances the accuracy of the network. Vanishing gradients:
Some neurons in too deep networks may die during training which can
cause loss of useful [Link] difficulty: too many
parameters can make training the network a difficult task. he inception
model was refined as Inception V3 in 2016, and as Inception-ResNet in
2017.

Prof. Deepali A Dixit, Dept of AI & ML ,RRCE 5

Deep Learning & Reinforcement Learning- BAI701

ResNet :Microsoft Research Asia proposed a CNN architecture in 2015,

which is 152 layers deep and is called ResNet.
-ResNet introduced residual connections in which the output of a conv-
relu-conv series is added to the original input and then passed through
Rectified Linear Unit (ReLU). as shown in Fig. 2.2

-The information is carried from the previous layer to the next layer and
during backpropagation, the gradient flows easily because of the addition
operations, which distributes the gradient.

-ResNet proved that a complex architecture like Inception is not required

to achieve the best results but a simple and deep architecture can be
tweaked to get better results.

-ResNet performed good in classification, detection, and localization and

won ILSVRC 2015 with an incredible error rate of 3.6% which is better
than the human error rate of 5–10%. -ResNet is currently deepest network
trained on ImageNet and has lesser parameters than VGGNet which is
eight times lesser in depth.

Prof. Deepali A Dixit, Dept of AI & ML ,RRCE 6

Deep Learning & Reinforcement Learning- BAI701

Inception-ResNet: A hybrid inception model which uses residual

connections, as in ResNet, was proposed in 2017.

-This hybrid model called Inception-ResNet dramatically improved the

training speed of inception model and slightly outperformed the pure
ResNet model by a thin margin.

SqueezeNet: A convolutional neural network architecture based on

depthwise separable convolution layers is called Xception.

-The architecture is actually inspired by inception model and that is why

it is called Xception (Extreme Inception).

-Xception architecture is a pile of depthwise separable convolution layers

with residual connections.- Xception has 36 convolutional layers
organized into 14 modules, all having linear residual connections around

Prof. Deepali A Dixit, Dept of AI & ML ,RRCE 7

Deep Learning & Reinforcement Learning- BAI701

them, except for the first and last modules. -The Xception has claimed to
perform slightly better than Inception V3 on ImageNet.

-Another version of SqueezeNet model used the following three main

strategies to reduce the parameters and computational effort significantly
while maintaining high accuracy.
(a) Replace 3 × 3 filters with 1 × 1 filters.
(b) Reduce the number of input channels to 3 × 3 filters.
(c) Delay subsampling till late in the network so that convolution layers
have large activation maps.

-SqueezeNet achieved AlexNet-level accuracy on ImageNet with 50

times fewer parameters.

ShuffleNet: Another ConvNet architecture called ShuffleNet was

introduced in 2017 for devices with limited computational power, like
mobile devices, without compromising on accuracy.

-ShuffleNet used two ideas, pointwise group convolution and channel

shuffle, to considerably decrease the computational cost while
maintaining the accuracy.

2.4 Convolution Operation:

-Convolution is a mathematical operation performed on two functions
and is written as (f * g), where f and g are two functions. -The output of
the convolution operation for domain n is defined as.

-For time-domain functions, n is replaced by t.

Prof. Deepali A Dixit, Dept of AI & ML ,RRCE 8
Deep Learning & Reinforcement Learning- BAI701

-The convolution operation is commutative in nature, so it can also be

written as.

-Convolution operation is one of the important operations used in digital

signal processing and is used in many areas which includes statistics,

probability, natural language processing, computer vision, and image
processing.

-Convolution operation can be applied to higher dimensional functions as

well.

-It can be applied to a two-dimensional function by sliding one function

on top of another, multiplying and adding.

-Convolution operation can be applied to images to perform various

transformations; here, images are treated as two-dimensional functions.

-An example of a two-dimensional filter, a two-dimensional input, and a

two-dimensional feature map is shown in Fig. 2.4.

-Let the 2D input (i.e., 2D image) be denoted by A, the 2D filter of size m

× n be denoted by K, and the 2D feature map be denoted by F.

-The image A is convolved with the filter K and produces the feature map
F. -This convolution operation is denoted by A*K and is mathematically
given as.

Prof. Deepali A Dixit, Dept of AI & ML ,RRCE 9

Deep Learning & Reinforcement Learning- BAI701

The kernel K is flipped relative to the input. If the kernel is not flipped,
then convolution operation will be same as cross-correlation operation
that is given below:

-Many CNN libraries use cross-correlation function as convolution

function because cross-correlation is more convenient to implement than
convolution operation itself. According to Eq. 2.3, the operation
computes the inner product (element wise multiplication) of the filter at
every location in the image.

Prof. Deepali A Dixit, Dept of AI & ML ,RRCE 10

Deep Learning & Reinforcement Learning- BAI701

2.5 Architecture of CNN:

-In a traditional neural network, neurons are fully connected between
different layers. Layers that sit between the input layer and output layer
are called hidden layers. Each hidden layer is made up of a number of
neurons, where each neuron is fully connected to all neurons in the
preceding layer.

-The problem with the fully connected neural network is that its densely
connected network architecture does not scale well to large images. For
large images, the most preferred approach is to use convolutional neural
network.

-Convolutional neural network is a deep neural network architecture

designed to process data that has a known, grid-like topology, for
example, 1D time-series data, 2D or 3D data such as images and speech
signal, and 4D data such as videos. ConvNets have three key features:
local receptive field, weight sharing, and subsampling (pooling).

(i) Local Receptive Field:

- In a traditional neural network, each neuron or hidden unit is connected

to every neuron in previous layer or every input unit.

- Convolutional neural networks have local receptive field architecture,

i.e., each hidden unit can only connect to a small region of the input
called local receptive field. -This is accomplished by making the
filter/weight matrix smaller than the input.

-With local receptive field, neurons can extract elementary visual features
like edges, corners, end points, etc.

Prof. Deepali A Dixit, Dept of AI & ML ,RRCE 11

Deep Learning & Reinforcement Learning- BAI701

(ii) Weight Sharing:

-Weight sharing refers to using the same filter/weights for all receptive
fields in a layer. In ConvNet, since the filters are smaller than the input,
each filter is applied at every position of the input, i.e., same filter is used
for all local receptive fields. ConvNet consists of a sequence of different
types of layers to achieve different tasks.

-A typical convolutional neural network consists of the following layers:

• Convolutional layer
• Activation function layer (ReLU)
• Pooling layer
• Fully connected layer
• Dropout layer

-These layers are stacked up to make a full ConvNet architecture.

-Convolutional and activation function layers are usually stacked together

followed by an optional pooling layer. -Fully connected layer makes up
the last layer of the network, and the output of the last fully connected
layer produces the class scores of the input image.

-ConvNet may include optional layers like batch normalization layer to

improve the training time and dropout layer to address the overfitting
issue.

(iii) Subsampling (Pooling):

Subsampling reduces the spatial size of the input, thus reducing the
parameters in the network. There are few subsampling techniques
available, and the most common subsampling technique is max-pooling.

Prof. Deepali A Dixit, Dept of AI & ML ,RRCE 12

Deep Learning & Reinforcement Learning- BAI701

2.5.1 Convolution Layer:

-Convolution layer is the core building block of a convolutional neural

network which uses convolution operation (represented by *) in place of
general matrix multiplication.

-Its parameters consist of a set of learnable filters also known as kernels.

-The main task of the convolutional layer is to detect features found

within local regions of the input image that are common throughout the
dataset and mapping their appearance to a feature map.

-A feature map is obtained for each filter in the layer by repeated

application of the filter across subregions of the complete image, i.e.,
convolving the filter with the input image, adding a bias term, and then
applying an activation function.

-The input area on which a filter is applied is called local receptive field.

The size of the receptive field is same as the size of the filter. Figure 2.5
shows how a filter (T-shaped) is convolved with the input to get the
feature map.

Feature map is obtained after adding a bias term and then applying a
nonlinear function to the output of the convolution operation. The
purpose of nonlinearity function is to introduce nonlinearity in the
ConvNet model.

Prof. Deepali A Dixit, Dept of AI & ML ,RRCE 13

Deep Learning & Reinforcement Learning- BAI701

Filters/Kernels:

The weights in each convolutional layer specify the convolution filters

and there may be multiple filters in each convolutional layer.

Every filter contains some feature like edge, corner, [Link] forward
pass, each filter is slid across the width and height of the input generating
feature map of that filter.

Hyperparameters:

-Convolutional neural network architecture has many hyperparameters

that are used to control the behavior of the model.

-Some of these hyperparameters control the size of the output while some
are used to tune the running time and memory cost of the model.

Prof. Deepali A Dixit, Dept of AI & ML ,RRCE 14

Deep Learning & Reinforcement Learning- BAI701

-The four important hyperparameters in the convolution layer of the

ConvNet are given below.

·Filter Size: Filters can be of any size greater than 2 × 2 and less than the
size of the input but the conventional size varies from 11 × 11 to 3 × 3.
The size of a filter is independent of the size of input.

·Number of Filters: There can be any reasonable number of filters.

AlexNet used 96 filters of size 11 × 11 in the first convolution layer.
VGGNet used 96 filters of size 7 × 7, and another variant of VGGNet
used 64 filters of size 11 × 11 in first convolution layer.

·Stride: It is the number of pixels to move at a time to define the local

receptive field for a filter. Stride of one means to move across and down a
single pixel. The value of stride should not be too small or too large. Too
small stride will lead to heavily overlapping receptive fields and too large
value will overlap less and the resulting output volume will have smaller
dimensions spatially.

· Zero Padding: This hyperparameter describes the number of pixels to

pad the input image with zeros. Zero padding is used to control the spatial
size of the output volume.

Here are the important points extracted without changing the sentences:

Each filter in the convolution layer produces a feature map of size

[(A−K+2P)/S]+1 where A is the input volume size, K is the size of the
filter, P is the number of padding applied and SS is the stride.

Suppose the input image has size 128×128, and 5 filters of size 5×55
\times 5 are applied, with single stride and zero padding, i.e., A=128, F=5,
P=0 and S=1.

Prof. Deepali A Dixit, Dept of AI & ML ,RRCE 15

Deep Learning & Reinforcement Learning- BAI701

The number of feature maps produced will be equal to the number of

filters applied, i.e., 5.

The size of each feature map will be [(128−5+0)/1] +1 = 124]

The output volume will be 124×124×5.

2.5.2 Activation Function (ReLU):

• The output of each convolutional layer is fed to an activation

function layer.
• The activation function layer consists of an activation function that
takes the feature map produced by the convolutional layer and
generates the activation map as its output.
• The activation function is used to transform the activation level of
a neuron into an output signal.
• It specifies the output of a neuron to a given input.
• An activation function usually has a squashing effect which takes
an input (a number), performs some mathematical operation on it
and outputs the activation level of a neuron between a given range,
e.g., 0 to 1 or −1 to 1.
• A typical activation function should be differentiable and
continuous everywhere.
• Since ConvNets are trained using gradient-based methods, an
activation function should be differential at any point.
• However, if a non-gradient-based method is used, then
differentiability is not necessary.
• There are many activation functions in use with Artificial Neural
Networks (ANNs) and some of the commonly used activation
functions are as follows.

Prof. Deepali A Dixit, Dept of AI & ML ,RRCE 16

Deep Learning & Reinforcement Learning- BAI701

Logistic/Sigmoid Activation Function: The sigmoid function is

mathematically
represented as

It is an S-shaped curve as shown in Fig. 2.6. Sigmoid function squashes the input into
the range [0, 1].

Tanh Activation Function: The hyperbolic tangent function is similar to sigmoid

function but its output lies in the range [−1, 1]. The advantage of tanh over sigmoid
is that the negative inputs will be mapped strongly negative and the zero inputs
will be mapped near zero in the tanh graph as shown in Fig. 2.7.

Prof. Deepali A Dixit, Dept of AI & ML ,RRCE 17

Deep Learning & Reinforcement Learning- BAI701

The ReLU (Rectified Linear Unit) is the most widely used activation
function in deep learning because it trains faster than sigmoid or tanh. It
outputs 0 when input < 0, and the input itself when input ≥ 0.
Mathematically:

Its graph is flat (0) for negative values and a straight line with slope 1 for
positive values.

Prof. Deepali A Dixit, Dept of AI & ML ,RRCE 18

Deep Learning & Reinforcement Learning- BAI701

SWISH Activation Function:

2.5.3 Pooling Layer

In ConvNets, the sequence of convolution layer and activation function

layer is followed by an optional pooling or down-sampling layer.

Pooling reduces the spatial size of the input and thus reduces the
number of parameters in the network.

A pooling layer takes each feature map output from the convolutional
layer and down-samples it.

Pooling summarizes a region of neurons in the convolution layer.

Prof. Deepali A Dixit, Dept of AI & ML ,RRCE 19
Deep Learning & Reinforcement Learning- BAI701

The most common pooling technique is max-pooling, which outputs

the maximum value in the input region (usually 2 × 2).Other pooling
options are average pooling and L2-norm pooling.

Pooling layer operation discards less significant data but preserves the
detected features in a smaller representation.

The intuitive reasoning behind pooling operation is that feature

detection is more important than feature’s exact location.

This strategy works well for simple and basic problems but it has its
own limitations and does not work well for some problems.

2.5.4 Fully Connected Layer:

• Convolutional neural networks have two stages: feature extraction

and classification.
• Convolution and pooling layers perform feature extraction until
enough features are detected.
• Fully connected layers use these features to make decisions for
classification.
• In fully connected layers, each neuron connects to every neuron in
the next layer.
• The final classifier outputs class scores.

Prof. Deepali A Dixit, Dept of AI & ML ,RRCE 20

Deep Learning & Reinforcement Learning- BAI701

• Softmax gives probabilities (sum = 1), while SVM outputs class

scores with the highest score as the predicted class.
• Figure 2.11 shows the connection between a convolution layer and
a fully connected layer.

2.5.5 Dropout:

• Deep neural networks have multiple hidden layers to learn complex

features.
• Fully connected layers are used for decision-making but are prone
to overfitting.
• Overfitting means the model performs well on training data but
poorly on new data.
• Dropout solves overfitting by randomly dropping neurons and their
connections during training.
• This reduces overfitting and improves generalization of the model.
• In order to overcome the problem of overfitting, a dropout layer
can be introduced in the model in which some neurons along with
their connections are randomly dropped from the network during
training (See Fig. 2.12).

Prof. Deepali A Dixit, Dept of AI & ML ,RRCE 21

Deep Learning & Reinforcement Learning- BAI701

(This forces the network to learn more robust and independent features,
since it cannot depend on specific neurons always being present.)

Prof. Deepali A Dixit, Dept of AI & ML ,RRCE 22