Deep Learning (AIC 701)

Module 4 - CNN

Devanand K. Bathe
Asst. Professor
Contents:
• 4.1 Convolution operation, Padding, Stride, Relation between input,
output and filter size,
• CNN architecture: Convolution layer, Pooling Layer, Weight Sharing in
CNN,
• Fully Connected NN vs CNN, Variants of basic Convolution function.

• 4.2 Modern Deep Learning Architectures:

• LeNET: Architecture, AlexNET: Architecture.
Introduction to CNN
• In 1995, Yann LeCunand Yoshua Bengio introduced the concept of
convolutional neural networks.
• A convolutional neural network (CNN) is the artificial neural network,
most commonly applied to analyze visual imagery.
• primary category for image classification and recognition in neural
networks.
• Some areas where convolutional neural networks are widely used are
scene labeling, object detections, face recognition, etc
How CNN works
• This neural network takes an image as input, classified, and processed
under a specific category: dog, cat, lion, tiger, etc.
• The computer looks at the image as an array of pixels and depends on the
resolution of the picture.
• Based on the image resolution, it will look at the image as –
height * width * dimension.

For example, consider an RGB image as a 5 * 5 * 2 array and the grayscale

image as a 4 * 4 * 1 array.
Each input image will go through a series of layers:
convolution, padding, strides, and pooling. Post that, we will apply the
Soft-max function to classify an object with probabilistic values 0 and 1.
• Convolution Layers
• The Convolution Layers are the initial layers to pull out features from
the image.
• It maintains the relationship between pixels by learning features using
a small input data sequence.
• It is a mathematical term that takes two inputs, an image matrix and a
kernel or filter.
• The result is calculated by:
• In the given image-
The image matrix is h x w x d
The dimensions of the filter are fh x fw x d
The output is calculated as (h- fh +1)(w- fw+1) x 1
• Now, let us take an example and solve a 5x5 image matrix whose pixel
values are 0, 1 and the filter matrix as 3x3:
 The final convolution layers output matrix of a 5x5 image multiplied with a 3x3 filter will be:

The convolution of the image with different filter values can produce a blur or sharpened image.
The size of the output image is calculated by:
(m-n+1)(m-n+1)
•
Example:
• Let’s ignore channels for now and see how this works with
two-dimensional data and hidden representations.
• The input is a two-dimensional tensor with a height of 3 and width of
3. We mark the shape of the tensor as (3 x 3) or (3,3)
• The height and width of the kernel are both 2. The shape of the
kernel window (or convolution window) is given by the height and
width of the kernel (2x2).
• In the two-dimensional cross-correlation operation, When the
convolution window slides to a certain position, the input subtensor
contained in that window and the kernel tensor are multiplied
elementwise and the resulting tensor is summed up yielding a single
scalar value.
• This result gives the value of the output tensor at the corresponding
location. Here, the output tensor has a height of 2 and width of 2 and
the four elements are derived from the two-dimensional
cross-correlation operation:
0x0 + 1x1 + 3x2 + 4x3 = 19
1x0 + 2x1 + 4x2 + 5x3 = 25
3x0 + 4x1 + 6x2 + 7x3 = 37
4x0 + 5x1 + 7x2 + 8x3 = 43
• Note that along each axis, the output size is slightly smaller than the
input size. Because the kernel has width and height greater than 1,
• we can only properly compute the cross-correlation for locations
where the kernel fits wholly within the image,
• the output size is given by the input size (𝑛ℎ×𝑛𝑤) minus the size of the
convolution kernel (𝑘ℎ×𝑘𝑤) via (𝑛ℎ-kℎ+1) x (𝑛𝑤-k𝑤+1)
• This is the case since we need enough space to “shift” the
convolution kernel across the image.
• Later we will see how to keep the size unchanged by padding the
image with zeros around its boundary so that there is enough space
to shift the kernel.
Strides:
• When the array is created, the pixels are shifted over to the input
matrix.
• The number of pixels turning to the input matrix is known as the
strides.
• When the number of strides is 1, we move the filters to 1 pixel at a
time. Similarly, when the number of strides is 2, we carry the filters to
2 pixels, and so on.
• They are essential because they control the convolution of the filter
against the input, i.e., Strides are responsible for regulating the
features that could be missed while flattening the image.
• They denote the number of steps we are moving in each convolution.
The following figure shows how the convolution would work.
• In the first matrix, the stride = 0, second image: stride=1, and the
third image: stride=2. The size of the output image is calculated by:
[{(n+2p-f+1)/s}+1][{(n+2p-f+1)/s}]
• Stride controls how the filter convolves around the input volume.
• In the example we had in part 1, the filter convolves around the input
volume by shifting one unit at a time.
• The amount by which the filter shifts is the stride.
• In that case, the stride was implicitly set at 1.
• Stride is normally set in a way so that the output volume is an integer
and not a fraction.
• Let’s look at an example. Let’s imagine a 7 x 7 input volume, a 3 x 3
filter (Disregard the 3rd dimension for simplicity), and a stride of 1.
Padding:
• The padding plays a vital role in creating CNN.
• After the convolution operation, the original size of the image is
shrunk.
• Also, in the image classification task, there are multiple convolution
layers after which our original image is shrunk after every step, which
we don’t want.
• Secondly, when the kernel moves over the original image, it passes
through the middle layer more times than the edge layers, due to
which there occurs an overlap.
• To overcome this problem, a new concept was introduced named
padding. It is an additional layer that can add to the borders of an
image while preserving the size of the original picture. For example:
So, if an n x n matrix is convolved with an ff matrix with a padding p, then the size of the output image will be:

(n+2p-f+1) x (n+2p-f+1)
• CNNs commonly use convolution kernels with odd height and width
values, such as 1, 3, 5, or 7.
• Choosing odd kernel sizes has the benefit that we can preserve the
dimensionality while padding with the same number of rows on top
and bottom, and the same number of columns on left and right.
• Moreover, this practice of using odd kernels and padding to precisely
preserve dimensionality offers a clerical benefit.
• For any two-dimensional tensor X, when the kernel’s size is odd and
the number of padding rows and columns on all sides are the same,
thereby producing an output with the same height and width as the
input,
• we know that the output Y[i, j] is calculated by cross-correlation of
the input and convolution kernel with the window centered on X[i, j].
Pooling:
• The pooling layer is another building block of a CNN and plays a vital role in
pre-processing an image.
• In the pre-process, the image size shrinks by reducing the number of
parameters if the image is too large.
• When the picture is shrunk, the pixel density is also reduced, the
downscaled image is obtained from the previous layers.
• Basically, its function is to progressively reduce the spatial size of the image
to reduce the network complexity and computational cost.
• Spatial pooling is also known as downsampling or subsampling that reduces
the dimensionality of each map but retains the essential features.
• A rectified linear activation function, or ReLU, is applied to each value in
the feature map.
• Relu is a simple and effective nonlinearity that does not change the values
in the feature map but is present because later subsequent pooling layers
are added.
• Pooling is added after the nonlinearity is applied to the feature maps. There
are three types of spatial pooling:
1. Max Pooling
• Max pooling is a rule to take the maximum of a region and help to
proceed with the most crucial features from the image.
• It is a sample-based process that transfers continuous functions into
discrete counterparts.
• Its primary objective is to downscale an input by reducing its
dimensionality and making assumptions about features contained in
the sub-region that were rejected.
2. Average Pooling
• it retains information about the lesser essential features.
• It simply downscales by dividing the input matrix into rectangular
regions and calculating the average values of each area.

3. Sum Pooling
• It is similar to Max pooling, but instead of calculating the maximum
value, we calculate the mean of each sub-region.
Example:
let’s take a 6 X 6 grayscale image (i.e. only one channel):Next, we convolve this 6 X 6 matrix with a 3 X 3 filter:

After the convolution, we will get a 4 X 4 image.

Generalized dimensions can be given as:

● Input: n X n X nc
● Filter: f X f X nc
● Padding: p
● Stride: s
● Output: [(n+2p-f)/s+1] X [(n+2p-f)/s+1] X nc’
Here, nc is the number of channels in the input and filter, while nc’ is the number of filters.

look at the summary of notations for a convolution layer:

● f[l] = filter size

● p[l] = padding
● s[l] = stride
● n[c][l] = number of filters
 Weight sharing in CNNs
A typical application of weight sharing is in convolutional neural networks. CNNs work by passing a

filter over the image input. For the trivial example of a 4x4 image and a 2x2 filter with a stride size of 2,

this would mean that the filter (which has four weights, one per pixel) is applied four times, making for

16 weights total. A typical application of weight sharing is to share the same weights across all four

filters.

In this context weight sharing has the following effects:

● It reduces the number of weights that must be learned (from 16 to 4, in this case), which reduces
model training time and cost.
● It makes feature search insensitive to feature location in the image.
So we reduce training cost at the cost of model flexibility. Weight sharing is for all intents and purposes a

form of regularization. And as with other forms of regularization, it can actually increase the performance

of the model, in certain datasets with high feature location variance, by decreasing variance more than

they increase bias.

CNN is primarily used for image classification and segmentation, and it works by finding similar

patterns throughout the input. These patterns can be found by sliding a filter with shared weights across

the input. The shared weights concept allows the network to learn the same pattern, regardless of its

position in the input. CNNs employ multiple filters to find different patterns in the input, which leads to

a feature map.
Convolutional Neural Network VS Fully
connected Neural Network
FCC-
What is fully connected layer?
A fully connected layer refers to a neural network in which each neuron applies a linear transformation to the

input vector through a weights matrix. As a result, all possible connections layer-to-layer are present, meaning

every input of the input vector influences every output of the output vector.
Neural networks are a set of dependent non-linear functions. Each individual function consists of a neuron (or a
perceptron). In fully connected layers, the neuron applies a linear transformation to the input vector through a
weights matrix. A non-linear transformation is then applied to the product through a non-linear activation
function f.
Here, we are taking the dot product between the weights matrix W and the input vector x. The bias term

(W0) can be added inside the non-linear function. I will ignore it for the rest of the article as it doesn’t

affect the output sizes or decision-making and is just another weight. If we take a layer in a fully

connected neural network with an input size of nine and an output size of four, the operation can be

visualized as follows:
The activation function “f” wraps the dot product between the input of the layer and the weights matrix of that layer.
Note that the columns in the weights matrix would all have different numbers and would be optimized as the model is

trained. The input is a 1x9 vector, the weights matrix is a 9x4 matrix. By taking the dot product and applying the
non-linear transformation with the activation function we get the output vector (1x4).

You can also visualize this layer the following way:

Why Is It Called a Fully Connected Layer?
The image above shows why we call these kinds of layers “fully connected” or sometimes “densely
connected.” All possible connections layer-to-layer are present, meaning every input of the input
vector influences every output of the output vector. However, not all weights affect all outputs. Look
at the lines between each node above. The orange lines represent the first neuron (or perceptron) of
the layer. The weights of this neuron only affect output A, and do not have an effect on outputs B, C

or D.
CNN
The basic operation of a convolutional layer is a convolution that is performed between an image and a
kernel (or filter) that is equal to a square matrix. First, we perform element-wise multiplication between the
pixels of the filter and the respective pixels of the image. Then, we sum up these multiplications into a single
output value. The whole procedure is repeated for every pixel of the input image.

In the below image, we can see how convolution works between a 6 x 6 image and a 3 x 3 filter. In the depicted
step, we compute the convolution for the pixel (2, 2) of the image generating the below image patch:
WHAT IS A CONVOLUTIONAL LAYER?
A convolutional layer applies to a neural network in which not all input nodes in a neuron are

connected to the output nodes. This gives convolutional layers more flexibility in learning. The

number of weights per layer is also a lot smaller, which helps with high-dimensional inputs, such as

image data. Once again, we can visualize this convolutional layer as follows:
Convolutions are not densely connected; not all input nodes affect all output nodes.
This gives convolutional layers more flexibility in learning. Moreover, the number of
weights per layer is a lot smaller, which helps with high-dimensional inputs such as image
data.
These advantages are what give CNNs their well-known characteristic of learning features
in the data, such as shapes and textures in image data.
In a convolutional layer, we perform convolution between the input neurons and some
learnable filters, generating an output activation map of the filter.
So, the number of weights is not dependent on the number of input neurons like in the FC
layer. In a Conv layer, the number of weights is equal to the size of the kernel.
The basic difference between the two types of layers is the density of the connections. The
FC layers are densely connected, meaning that every neuron in the output is connected to every
input neuron. On the other hand, in a Conv layer, the neurons are not densely connected but are
connected only to neighboring neurons within the width of the convolutional kernel. So, if the input
is an image and the number of neurons is large, a Conv layer is more suitable.

A second main difference between them is weight sharing. In an FC layer, every output neuron
is connected to every input neuron through a different weight . However, in a Conv layer, the
weights are shared among different neurons. This is another characteristic that enables Conv
layers to be used in the case of a large number of neurons.
How to Work With Fully Connected Layers and Convolutional Neural Networks

In FC layers, the output size of the layer can be specified very simply by choosing the number of

columns in the weights matrix. The same cannot be said for convolutional layers. Convolutions have a

lot of parameters that can be changed to adapt the output size of the operation.

In this explanation of convolutions, you’ll see all the variations of convolutions, such as convolutions

with and without padding, strides, transposed convolutions and more. It’s a useful visual

interpretation of a convolution.

To determine the output size of the convolution, the following equation can be applied:
Variants of the Basic Convolution Function
Convolution in the context of NN means an operation that consists of many
applications of convolution in parallel.

● Full Convolution.
● Unshared Convolution.
● Tiled Convolution.
● Back prop in conv layer.
● Bias in after conv.
Some 0 Paddings and 1 stride
Without 0 paddings, the width of representation shrinks by one pixel less than the kernel width at each layer. We
are forced to choose between shrinking the spatial extent of the network rapidly and using small kernel. 0 padding
allows us to control the kernel width and the size of the output independently.

Special case of 0 padding:

● Valid: no 0 padding is used. Limited number of layers.

● Same: keep the size of the output to the size of input. Unlimited number of layers. Pixels near the border
influence fewer output pixels than the input pixels near the center.
● Full: Enough zeros are added for every pixels to be visited k (kernel width) times in each direction, resulting
width m + k - 1. Difficult to learn a single kernel that performs well at all positions in the convolutional
feature map.

Usually the optimal amount of 0 padding lies somewhere between ‘Valid’ or ‘Same’
Useful when we know that each feature should be a function of a small part of space, but no reason to
think that the same feature should occur accross all the space. eg: look for mouth only in the bottom
half of the image.

It can be also useful to make versions of convolution or local connected layers in which the
connectivity is further restricted, eg: constrain each output channeel i to be a function of only a subset
of the input channel.

Adv: * reduce memory consumption * increase statistical efficiency * reduce computation for both
forward and backward prop.
Tiled Convolution
Learn a set of kernels that we rotate through as we move through space. Immediately neighboring locations will
have different filters, but the memory requirement for storing the parameters will increase by a factor of the size of
this set of kernels. Comparison on locally connected layers, tiled convolution and stardard convolution: in next slide
Bias in after conv
We generally add some bias term to each output before applying nonelinearity.

● For local conncted layers: give each unit its one bias
● For tiled conv layers: share the biases with the same tiling pattern as the kernels
● For conv layers: have one bias per channel of the output and share it accross all locations within each
convolution map. If the input is fixed size, it is also possible to learn a seperate bias at each location of the
output map.
EX. 1-

We take an input image (size = 39 X 39 X 3 in our case), convolve it with 10 filters of

size 3 X 3, and take the stride as 1 and no padding.
This will give us an output of 37 X 37 X 10. We convolve this output further and get
an output of 7 X 7 X 40 as shown above.
Finally, we take all these numbers (7 X 7 X 40 = 1960), unroll them into a large
vector, and pass them to a classifier that will make predictions. This is a microcosm of
how a convolutional network works.
General pooling layer
•Accepts a volume of size W1×H1×D1
•Requires two hyperparameters:
–their spatial extent F
–the stride S
•Produces a volume of size W2×H2×D2 where:
–W2=(W1−F)/S+1
–H2=(H1−F)/S+1
–D2=D1
•Introduces zero parameters
•Other pooling functions: Average pooling, L2-norm pooling
Backpropagation. the backward pass for a max(x, y) operation routes the gradient to
the input that had the highest value in the forward pass.
•Hence, during the forward pass of a pooling layer you may keep track of the index
of the max activation (sometimes also called the switches) so that gradient routing
is efficient during backpropagation.
Example:
Example:
images consisting of a training set of 60,000 examples and a test set of 10,000 examples. Each example is a 28x28 grayscale image, associated

with a label from 10 classes.

Our convolutional neural network has architecture as follows:

[INPUT]

→[CONV 1] → [BATCH NORM] → [ReLU] → [POOL 1]

→ [CONV 2] → [BATCH NORM] → [ReLU] → [POOL 2]

→ [FC LAYER] → [RESULT]

For both conv layers, we will use kernel of spatial size 5 x 5 with stride size 1 and padding of 2. For both pooling layers, we will use max pool

operation with kernel size 2, stride 2, and zero padding.

Modern CNN Architecture
Convolutional Neural networks are a class of Deep Neural Networks, which
achieve State of the Art results not only in Computer Vision tasks but also in
other fields such as Speech recognition, Natural Language Processing, etc.
These CNNs have evolved in a long way by not only stacking layers but also
creating custom layers. In the current world, we use these models to transfer
learning and achieve better results.
A Convolutional Neural Network (CNN, or ConvNet) are a special
kind of multi-layer neural networks, designed to recognize visual patterns
directly from pixel images with minimal preprocessing.. The ImageNet
project is a large visual database designed for use in visual object recognition
software research. The ImageNet project runs an annual software contest,
the ImageNet Large Scale Visual Recognition Challenge (ILSVRC),
where software programs compete to correctly classify and detect objects
and scenes. Here I will talk about CNN architectures of ILSVRC top
competiton
LeNet and AlexNet are used to outperform the state of image classification
results on MNIST or Image Net Data Sets.
LeNet is a classic convolutional neural network. Convolutional neural networks
are a kind of feed-forward neural network whose artificial neurons can respond
to a part of the surrounding cells in the coverage range and perform well in
large-scale image processing.
It uses convolutions, pooling and fully connected layers. It was used for the
handwritten digit recognition task with the MNIST dataset.
The architectural design served as inspiration for future networks such as
AlexNet and VGG.
AlexNet was the first convolutional network which used GPU to boost
performance.
What is LeNet?

LeNet is a classic convolutional neural network. Convolutional neural networks are a kind of feed-forward neural network

whose artificial neurons can respond to a part of the surrounding cells in the coverage range and perform well in

large-scale image processing. It uses convolutions, pooling and fully connected layers.

It was used for the handwritten digit recognition task with the MNIST dataset. The architectural design served as

inspiration for future networks such as AlexNet and VGG.

Why is LeNet important?

Another reason why LeNet is an important architecture is that before it was invented, character recognition had been done

mostly by using feature engineering by hand, followed by a machine learning model to learn to classify hand engineered

features.
LeNet :
This is also known as the Classic Neural Network that was designed by Yann LeCun, Leon
Bottou, Yosuha Bengio and Patrick Haffner for handwritten and machine-printed character
recognition in 1990’s which they called LeNet-5.
The architecture was designed to identify handwritten digits in the MNIST data-set. The
architecture is pretty straightforward and simple to understand.
The input images were gray scale with dimension of 32*32*1 followed by two pairs of
Convolution layer with stride 2 and Average pooling layer with stride 1.
Finally, fully connected layers with Softmax activation in the output layer. Traditionally, this
network had 60,000 parameters in total.
The LeNet-5 architecture consists of two sets of convolutional, activation, and pooling layers,
followed by two fully-connected layers, tanh activation, and finally a softmax classifier which
classifies the MNIST digits.
LeNet-5 CNN architecture is made up of 7 layers. The layer composition consists of 3 convolutional
layers, 2 subsampling layers and 2 fully connected layers.
What is AlexNet?

AlexNet was the first convolutional network which used GPU to boost performance. 1. AlexNet

architecture consists of 5 convolutional layers, 3 max-pooling layers, 2 normalization layers, 2 fully

connected layers, and 1 softmax layer.

What is AlexNet used for?

AlexNet is a convolutional neural network that is 8 layers deep. You can load a pretrained version of

the network trained on more than a million images from the ImageNet database [1]. The pretrained

network can classify images into 1000 object categories

AlexNet :
In 2012, AlexNet significantly outperformed all the prior competitors and won the
challenge by reducing the top-5 error from 26% to 15.3%. The second place top-5
error rate, which was not a CNN variation, was around 26.2%.
The network had a very similar architecture as LeNet by Yann LeCun et al but was
deeper, with more filters per layer, and with stacked convolutional layers.
It consisted 11x11, 5x5,3x3, convolutions, max pooling, dropout, data augmentation,
ReLU activations, SGD with momentum.
It attached ReLU activations after every convolutional and fully-connected layer.
AlexNet was trained for 6 days simultaneously on two Nvidia Geforce GTX 580
GPUs which is the reason for why their network is split into two pipelines.
AlexNet was designed by the SuperVision group, consisting of Alex Krizhevsky,
Geoffrey Hinton, and Ilya Sutskever.
This network was very similar to LeNet-5 but was deeper with 8 layers, with more filters,
stacked convolutional layers, max pooling, dropout, data augmentation, ReLU and SGD.
AlexNet was the winner of the ImageNet ILSVRC-2012 competition, designed by Alex
Krizhevsky, Ilya Sutskever and Geoffery E. Hinton.
It was trained on two Nvidia Geforce GTX 580 GPUs, therefore, the network was split into
two pipelines. AlexNet has 5 Convolution layers and 3 fully connected layers. AlexNet
consists of approximately 60 M parameters.
A major drawback of this network was that it comprises of too many hyper-parameters.
GoogLeNet:
The winner of the ILSVRC 2014 competition was GoogLeNet (a.k.a. Inception
V1) from Google. It achieved a top-5 error rate of 6.67%! This was very close to
human level performance which the organisers of the challenge were now forced
to evaluate.
As it turns out, this was actually rather hard to do and required some human
training in order to beat GoogLeNets accuracy. After a few days of training, the
human expert (Andrej Karpathy) was able to achieve a top-5 error rate of
5.1%(single model) and 3.6%(ensemble).
The network used a CNN inspired by LeNet but implemented a novel element
which is dubbed an inception module. It used batch normalization, image
distortions and RMSprop. This module is based on several very small
convolutions in order to drastically reduce the number of parameters.
Their architecture consisted of a 22 layer deep CNN but reduced the number of
parameters from 60 million (AlexNet) to 4 million.
[Link]
tional-neural-network

[Link]
onal-neural-networks
Thank you