CAI2502-Deep Learning

Module 2
• Improving Deep Neural Networks:
• Convolutional Neural Network, Transfer learning
Techniques,
• Variants of CNN: DenseNet, PixelNet, ResNet, AlexNet,
• Sequence Modelling: Recurrent Neural Network and
its variants - Long Short Term Memory (LSTM), Gated
Recurrent Unit (GRU).

11/11/2025
Mr. Pakruddin B, Assit. Prof., PSCS 1
 Convolution Neural Network

11/11/2025 2
Mr. Pakruddin B, Assit. Prof., PSCS
 Some Fundamentals of Image
Processing
Images are represented as matrix of pixel values.
Can be in gray-scale or RGB

11/11/2025 3
Mr. Pakruddin B, Assit. Prof., PSCS
11/11/2025 4
Mr. Pakruddin B, Assit. Prof., PSCS
11/11/2025 5
Mr. Pakruddin B, Assit. Prof., PSCS
11/11/2025 6
Mr. Pakruddin B, Assit. Prof., PSCS
 Why not ANN?????
• Due to the large number of parameters to be learned by the model
• For an input image of size 28 x 28, the input layer has :
• [Link] inputs : 28 x 28 x 3
• No. of weights : 28 x 28 x 3 x no. of hidden neurons in next layer
• Hence the model has to learn many parameters in subsequent layers

11/11/2025 7
Mr. Pakruddin B, Assit. Prof., PSCS
CNN
• CNN are widely used tools in DL
• Frequently used with image data sets
• Also work on other inputs such as texts, signals and other continuous responses
• It has input layer, many hidden layers and an output layer.
• The hidden layers include feature learning and classification layers
• Feature learning is done using a set of operations like convolution using a
kernel(filter), ReLu and pooling

11/11/2025 8
Mr. Pakruddin B, Assit. Prof., PSCS
 • Convolution puts the image through a set of filters(kernels). Each filter activates
certain features in the image, relevant to the learning task.

• Ex: if the model has to detect faces in the image, there may be filters for edge
detection, shape detection, etc moving through the image to capture those
significant images for face detection

• Rectified linear unit (ReLU) allows for faster and more effective training by
mapping negative values to zero and maintaining positive values.

• Pooling simplifies the output by performing nonlinear down-sampling, reducing

the number of parameters that the network needs to learn about.

• These three operations are repeated over tens or hundreds of layers, with each
layer learning to detect different features.

11/11/2025 9
Mr. Pakruddin B, Assit. Prof., PSCS
• Classification Layers: After feature detection, the architecture of a CNN shifts to
classification.
• The next-to-last layer is a fully connected layer (FC) that outputs a vector of K
dimensions where K is the number of classes that the network will be able to
predict.
• This vector contains the probabilities for each class of any image being classified.
• The final layer of the CNN architecture uses a softmax function to provide the
classification output

11/11/2025 10
Mr. Pakruddin B, Assit. Prof., PSCS
CNN

11/11/2025 11
Mr. Pakruddin B, Assit. Prof., PSCS
11/11/2025 12
Mr. Pakruddin B, Assit. Prof., PSCS
 • Convolution of an image with different filters can perform operations such
as edge detection, blur and sharpen by applying filters
• Some common filters are

11/11/2025 13
Mr. Pakruddin B, Assit. Prof., PSCS
11/11/2025 14
Mr. Pakruddin B, Assit. Prof., PSCS
11/11/2025 15
Mr. Pakruddin B, Assit. Prof., PSCS
11/11/2025 16
Mr. Pakruddin B, Assit. Prof., PSCS
Details of Convolution layer
A DNN trained to recognise characters, should be able to recognise different
renditions of a character
Following are different renditions of ‘x’ and ‘o’

• Hence we need to train a model which is location invariant.

11/11/2025 17
Mr. Pakruddin B, Assit. Prof., PSCS
Reference image of ‘x’ Input image of ‘x’

• A location invariant model will be able to identify whether the features of the
reference image are present in the input image or not, rather than where they are
present.
• Convolution operation helps to build such location invariant models.
• Small patches of the reference image(filters) are mixed with the nearby locations in
the input image to see whether there is a match.

11/11/2025 18
Mr. Pakruddin B, Assit. Prof., PSCS
• Each convolution filter represents a feature of interest and the CNN algorithm
finds which features comprise the reference image(alphabet of interest)
• Convolution has 4 steps :
• Line up the feature(filter) and the input image
• Multiply the corresponding pixels of the filter and the input image
• Find the sum of these products.
• Divide this sum by the total no. of pixels.

11/11/2025 19
Mr. Pakruddin B, Assit. Prof., PSCS
 • Place this final value at the centre of the filtered image

11/11/2025 20
Mr. Pakruddin B, Assit. Prof., PSCS
 • Move the feature(filter) to every other position in the image to see how match
the feature matches with the image
• The number of steps the filter takes to move on the image is the stride.
• With a stride = 1, the final feature map is

• The output signal strength is not dependent on where the features are located, but
simply whether the features are present. Hence, an alphabet could be sitting in
different positions and the Convolutional Neural Network algorithm would still be
able to recognize it.
• Image of size nxn and filter of size fxf will result in convolved image of size n-f+1 x
n-f+1

11/11/2025 21
Mr. Pakruddin B, Assit. Prof., PSCS
 [Link]/convolutional-neural-networks-explained-
9cc5188c4939#:~:text=A%20Convolutional%20Neural%20Network%2C%
20also,binary%20representation%20of%20visual%20data
.

11/11/2025 22
Mr. Pakruddin B, Assit. Prof., PSCS
 • Stride – no. of pixel shifts taken by a filter when it moves
on the input image

• Ex1 : Input image – 5 x 5 Filter : 3 x 3 Stride = 1 Find

the feature map

• Ex2 : Input image – 5 x 5 Filter : 3 x 3 Stride = 2 Find

the feature map
• Size of the feature map including the stride is x

11/11/2025 23
Mr. Pakruddin B, Assit. Prof., PSCS
11/11/2025 24
Mr. Pakruddin B, Assit. Prof., PSCS
We have seen that convolving an input of 6 X 6 dimension with a 3 X 3
filter results in 4 X 4 output. We can generalize it and say that if the
input is n X n and the filter size is f X f, then the output size will be (n-
f+1) X (n-f+1):
•Input: n X n
•Filter size: f X f
•Output: (n-f+1) X (n-f+1)
There are primarily two disadvantages here:
[Link] time we apply a convolutional operation, the size of the image
shrinks
[Link] present in the corner of the image are used only a few number
of times during convolution as compared to the central pixels. Hence,
we do not focus too much on the corners since that can lead to
information loss

11/11/2025 25
Mr. Pakruddin B, Assit. Prof., PSCS
 Padding
To overcome these issues, we can pad the image with an additional border,
i.e., we add one pixel all around the edges. This means that the input will be
an 8 X 8 matrix (instead of a 6 X 6 matrix). Applying convolution of 3 X 3 on it
will result in a 6 X 6 matrix which is the original shape of the image. This is
where padding comes to the fore:
•Input: n X n Padding: p Filter size: f X f
•Output: (n+2p-f+1) X (n+2p-f+1)
There are two common choices for padding:
[Link]: It means no padding. If we are using valid padding, the output will
be (n-f+1) X (n-f+1)
[Link]: Here, we apply padding so that the output size is the same as the
input size, i.e., n+2p-f+1 = n , So, p = (f-1)/2

11/11/2025 26
Mr. Pakruddin B, Assit. Prof., PSCS
 Padding

the optimal amount of zero padding (in terms of test accuracy) is somewhere between
Pad = 0 and the pad that causes the output and input to have the same width and height.
Goodfellow et al.

11/11/2025 27
Mr. Pakruddin B, Assit. Prof., PSCS
 Stride
Defines how much the filter moves in the convolution

Suppose we choose a stride of 2. So, while convoluting through the

image, we will take two steps – both in the horizontal and vertical
directions separately. The dimensions for stride s will be:
•Input: n X n
•Padding: p
•Stride: s
•Filter size: f X f
•Output: [(n+2p-f)/s+1] X [(n+2p-f)/s+1]
Stride helps to reduce the size of the image, a particularly useful feature.

11/11/2025 28
Mr. Pakruddin B, Assit. Prof., PSCS
11/11/2025 29
Mr. Pakruddin B, Assit. Prof., PSCS
11/11/2025 30
Mr. Pakruddin B, Assit. Prof., PSCS
11/11/2025 31
Mr. Pakruddin B, Assit. Prof., PSCS
11/11/2025 32
Mr. Pakruddin B, Assit. Prof., PSCS
11/11/2025 33
Mr. Pakruddin B, Assit. Prof., PSCS
11/11/2025 34
Mr. Pakruddin B, Assit. Prof., PSCS
11/11/2025 35
Mr. Pakruddin B, Assit. Prof., PSCS
11/11/2025 36
Mr. Pakruddin B, Assit. Prof., PSCS
Convolutions Over Volume
Suppose, instead of a 2-D image, we have a 3-D input image of shape 6
X 6 X 3. How will we apply convolution on this image? We will use a 3 X
3 X 3 filter instead of a 3 X 3 filter. Let’s look at an example:
•Input: 6 X 6 X 3
•Filter: 3 X 3 X 3
The dimensions above represent the height, width and channels in the
input and filter. Keep in mind that the number of channels in the
input and filter should be same. This will result in an output of 4 X
4.

Since there are three channels in the input, the filter will consequently also
have three channels. After convolution, the output shape is a 4 X 4 matrix.
So, the first element of the output is the sum of the element-wise product
of the first 27 values from the input (9 values from each channel) and the
27 values from the filter. After that we convolve over the entire image.

11/11/2025 37
Mr. Pakruddin B, Assit. Prof., PSCS
 Generalized dimensions can be given as:
•Input: n X n X nc
•Filter: f X f X nc
•Padding: p
•Stride: s
•Output: [(n+2p-f)/s+1] X [(n+2p-f)/s+1] X nc’
Here, nc is the number of channels in the input and filter, while
nc’ is the number of filters.

11/11/2025 38
Mr. Pakruddin B, Assit. Prof., PSCS
 Pooling

11/11/2025 39
Mr. Pakruddin B, Assit. Prof., PSCS
11/11/2025 40
Mr. Pakruddin B, Assit. Prof., PSCS
11/11/2025 41
Mr. Pakruddin B, Assit. Prof., PSCS
11/11/2025 42
Mr. Pakruddin B, Assit. Prof., PSCS
11/11/2025 43
Mr. Pakruddin B, Assit. Prof., PSCS
RELU
• ReLu is applied to the convolution outputs.
• The main aim is to remove all the negative values from the convolution. All
the positive values remain the same but all the negative values get changed
to zero

11/11/2025 44
Mr. Pakruddin B, Assit. Prof., PSCS
11/11/2025 45
Mr. Pakruddin B, Assit. Prof., PSCS
 RELU for all Feature Maps

11/11/2025 46
Mr. Pakruddin B, Assit. Prof., PSCS
 POOLING
• Pooling layers section would reduce the number of parameters
when the images are too large.
• Spatial pooling also called subsampling or downsampling which
reduces the dimensionality of each map but retains important
information.
• Spatial pooling can be of different types: 1. Max Pooling 2.
Average Pooling 3. Sum Pooling
• Max pooling takes the largest element from the rectified feature
map.
• Taking average of all elements in the feature map is average
pooling.
• Sum of all elements in the feature map call as sum pooling.

11/11/2025 47
Mr. Pakruddin B, Assit. Prof., PSCS
POOLING

11/11/2025 48
Mr. Pakruddin B, Assit. Prof., PSCS
Pooling for the first rectified feature map with
window size = 2; stride = 2

11/11/2025 49
Mr. Pakruddin B, Assit. Prof., PSCS
Pooling for all the rectified feature maps with
window size = 2; stride = 2

11/11/2025 50
Mr. Pakruddin B, Assit. Prof., PSCS
 Stacking the Layers

11/11/2025 51
Mr. Pakruddin B, Assit. Prof., PSCS
Adding another Layer

11/11/2025 52
Mr. Pakruddin B, Assit. Prof., PSCS
 Stacking the Layers

We can reduce the image size from 4 x 4 to something lesser by stacking the layers.
Repeat the operations convolution, ReLU and Pooling after first pass

11/11/2025 53
Mr. Pakruddin B, Assit. Prof., PSCS
 Adding another Layer

Image has reduced to 2 x 2

11/11/2025 54
Mr. Pakruddin B, Assit. Prof., PSCS
Fully Connected Layer
• The last layers in the network are fully connected, meaning that neurons of preceding
layers are connected to every neuron in subsequent layers.
• This mimics high level reasoning where all possible pathways from the input to output
are considered.
• Fully connected layer is the final layer where the classification actually happens.
• Here we take our filtered and shrinked images and put them into one single list –
Flattening

11/11/2025 55
Mr. Pakruddin B, Assit. Prof., PSCS
 Classification?
• When the input image is ‘X’, there will be some element(1st, 4th, 5th, 10th and
11th values )in the flattened vector that will be high
• For image ‘O’ there will be some element in the vector that will be high.
• The concept is similar for the other alphabets as well

11/11/2025 56
Mr. Pakruddin B, Assit. Prof., PSCS
Prediction?
• we have a 12 element vector below, obtained after passing the input image
of a random letter through all the layers of our network.
• Predict it as ‘x’ or ‘o’

11/11/2025 57
Mr. Pakruddin B, Assit. Prof., PSCS
Predict

11/11/2025 58
Mr. Pakruddin B, Assit. Prof., PSCS
 Typical CNN Architecture

11/11/2025 59
Mr. Pakruddin B, Assit. Prof., PSCS
11/11/2025 60
Mr. Pakruddin B, Assit. Prof., PSCS
11/11/2025 61
Mr. Pakruddin B, Assit. Prof., PSCS
11/11/2025 62
Mr. Pakruddin B, Assit. Prof., PSCS
11/11/2025 63
Mr. Pakruddin B, Assit. Prof., PSCS
11/11/2025 64
Mr. Pakruddin B, Assit. Prof., PSCS
11/11/2025 65
Mr. Pakruddin B, Assit. Prof., PSCS
output_shape = [Link]((input_shape - pool_size) / strides) + 1
(when input_shape >= pool_size)

11/11/2025 66
Mr. Pakruddin B, Assit. Prof., PSCS
11/11/2025 67
Mr. Pakruddin B, Assit. Prof., PSCS
11/11/2025 68
Mr. Pakruddin B, Assit. Prof., PSCS
11/11/2025 69
Mr. Pakruddin B, Assit. Prof., PSCS
11/11/2025 70
Mr. Pakruddin B, Assit. Prof., PSCS
11/11/2025 71
Mr. Pakruddin B, Assit. Prof., PSCS
11/11/2025 72
Mr. Pakruddin B, Assit. Prof., PSCS
[Link]
[Link]

11/11/2025 73
Mr. Pakruddin B, Assit. Prof., PSCS
Why ConvNet should be Deep?

Rob Fergus,
NIPS 2013

For reporting error rates, a model predicts the top 5 most likely labels. The 'top-
5' error rate is the fraction of test images for which the correct label is amongst
this top 5, and the 'top-1' error rate is the fraction of test images for which the
correct label is the one judged most likely by the model

11/11/2025 74
Mr. Pakruddin B, Assit. Prof., PSCS
 Why ConvNet should be Deep?

11/11/2025 75
Mr. Pakruddin B, Assit. Prof., PSCS
 Why ConvNet should be Deep?

11/11/2025 76
Mr. Pakruddin B, Assit. Prof., PSCS
 Why ConvNet should be Deep?

11/11/2025 77
Mr. Pakruddin B, Assit. Prof., PSCS
 Why ConvNet should be Deep?

CNN code
[Link]
=sharing

11/11/2025 78
Mr. Pakruddin B, Assit. Prof., PSCS
Understanding and Calculating the number of Parameters in Convolution Neural
Networks (CNNs)

11/11/2025 79
Mr. Pakruddin B, Assit. Prof., PSCS
1. The first input layer has no parameters. Input layer has nothing to learn, at
it’s core, what it does is just provide the input image’s shape. So no learnable parameters
here. Thus number of parameters = 0.

[Link] in the second CONV1(filter shape =5*5, stride=1)

layer is: ((shape of width of filter*shape of height filter*number
of filters in the previous layer+1)*number of filters) =
(((5*5*3)+1)*8) = 608.

11/11/2025 Mr. Pakruddin B, Assit. Prof., PSCS 80

3. The third POOL1 layer has no parameters.

4. Parameters in the fourth CONV2(filter shape =5*5,

stride=1) layer is: ((shape of width of filter * shape of
height filter * number of filters in the previous layer+1) *
number of filters) = (((5*5*8)+1)*16) = 3216.

5. The fifth POOL2 layer has no parameters.

6. Parameters in the Sixth FC3 layer is((current layer c*previous layer p)

+1*c) = 120*400+1*120= 48120.

7. Parameters in the Seventh FC4 layer is: ((current layer c*previous layer
p)+1*c) = 84*120+1* 84 = 10164.

8. The Eighth Softmax layer has ((current layer c*previous layer p)+1*c)
parameters = 10*84+1*10 = 850.

11/11/2025 Mr. Pakruddin B, Assit. Prof., PSCS 81

 RNN
• Recurrent neural networks (RNN) are the state of the art
algorithms for sequential data and are used by Apple's Siri and
Google's voice search.
• Sequential Data?
the points in the dataset are dependent on the other points in the
dataset, the data is said to be Sequential data.
Ex: time series data, stock market price data, words in a sentence,
gene sequence data, etc.
• Why ANN cannot be used for sequential data?
It doesn’t consider the dependencies within a sequence data.
Ex: Given time-series data, develop a DNN to predict the outlook of
a day as sunny/rainy/windy.

11/11/2025 Mr. Pakruddin B, Assit. Prof., PSCS 82

• The traditional NN makes the prediction for each observation independent of the other
observations.
• This violates the fact that weather on a particular day is strongly correlated with the weather of the
previous day and the following day.
• a traditional neural network assumes the data is non-sequential, and that each data point is
independent of other data points.
• Hence, the inputs are analyzed in isolation, which can cause problems in case there are
dependencies in the data.
• In traditional neural networks, all the inputs and outputs are independent of each other, but in
cases when it is required to predict the next word of a sentence, the previous words are required
and hence there is a need to remember the previous words.

11/11/2025 Mr. Pakruddin B, Assit. Prof., PSCS 83

RNN are a type of Neural Network where the output from previous step are fed as
input to the current step.

Most important feature of RNN is Hidden state, which remembers some information about a
sequence.
RNN has a “memory” which remembers all information about what has been calculated in the
previous day.
It uses the same parameters for each input as it performs the same task on all the inputs or
hidden layers to produce the output.
This reduces the complexity of parameters, unlike other neural networks

11/11/2025 Mr. Pakruddin B, Assit. Prof., PSCS 84

11/11/2025 Mr. Pakruddin B, Assit. Prof., PSCS 85
 Some Applications of RNN

11/11/2025 Mr. Pakruddin B, Assit. Prof., PSCS 86

11/11/2025 Mr. Pakruddin B, Assit. Prof., PSCS 87
Why not ANN?
1. An issue with using an ANN for language translation is, we cannot fix the no.
of neurons in a layer. It depends on the no. of words in the input sentence.

11/11/2025 Mr. Pakruddin B, Assit. Prof., PSCS 88

Why not ANN?
2. Too much computations.
Input words have to be converted to vectors(word2vec) using one-hot encoding.
Hence that many neurons and parameters have to be learnt by the model.

11/11/2025 Mr. Pakruddin B, Assit. Prof., PSCS 89

Why not ANN?

3. Doesn’t preserve the sequence relationship in the input data

• A traditional neural network assumes the data is non-sequential, and that each data
point is independent of other data points.
• Hence, the inputs are analyzed in isolation, which can cause problems in case there
are dependencies in the data.
• Since each hidden layer has its own weights, bias and activations, they behave
independently.
• When the input is a sequence data, the model should be also able to identify the
relationship between successive inputs

11/11/2025 Mr. Pakruddin B, Assit. Prof., PSCS 90

If the task is to predict the next word in a sentence using a MLP.

• This will not help. All hidden layers with different

weights and bias work independently.
• To make the hidden layers preserve the sequence
relationship in the input, all hidden layers have to be
combined.
• To combine them use same weights and activation
functions
All these hidden layers can be rolled in
together in a single recurrent layer

11/11/2025 Mr. Pakruddin B, Assit. Prof., PSCS 91

How RNN works?
• Neurons in recurrent layer are called recurrent neurons
At all the time steps weights of the recurrent neurons would be the same
So a recurrent neuron stores the state of a previous input and combines with the
current input thereby preserving some relationship of the current input with the
previous input.
• RNN converts the independent activations into dependent activations by providing
the same weights and biases to all the layers, thus reducing the complexity of
increasing parameters and memorizing each previous outputs by giving each
output as input to the next hidden layer.

11/11/2025 Mr. Pakruddin B, Assit. Prof., PSCS 92

11/11/2025 Mr. Pakruddin B, Assit. Prof., PSCS 93
Entire RNN computation involves – computations to update the cell state at that
time step and computations to predict the output at that time step.

11/11/2025 Mr. Pakruddin B, Assit. Prof., PSCS 94

11/11/2025 Mr. Pakruddin B, Assit. Prof., PSCS 95
During forward pass, we calculate the outputs at each time step, to calculate
the individual loss at each time step.
The individual losses are combined to form the total loss.
This total loss is used to train the neural network

11/11/2025 Mr. Pakruddin B, Assit. Prof., PSCS 96

11/11/2025 Mr. Pakruddin B, Assit. Prof., PSCS 97
11/11/2025 Mr. Pakruddin B, Assit. Prof., PSCS 98
Desirable Characteristics of RNN for
Sequence Modeling

• Ability to handle sequences of variable

lengths.
• Information about the next word to be
predicted in the sequence, might be
present very much earlier at the
beginning of the sequence.

• Ability to capture and model Long-Term

Dependencies
• This is possible since RNNs, keep
updating information collected from
the past by updating their
recurrent/hidden cell state at each
time step.

11/11/2025 Mr. Pakruddin B, Assit. Prof., PSCS 99

Desirable Characteristics of RNN for
Sequence Modeling…

• Ability to capture differences in sequence

order
• Two sentences with same words, but
different meaning.
• But the RNNs capture this difference,
since it uses the same weight matrices
at each time step, to update its hidden
state and remembers past
information.

11/11/2025 Mr. Pakruddin B, Assit. Prof., PSCS 100

11/11/2025 Mr. Pakruddin B, Assit. Prof., PSCS 101
11/11/2025 Mr. Pakruddin B, Assit. Prof., PSCS 102
 • In FFNN, the gradient of
the loss function is back
propagated through one
feed forward network in
one time step/input.

• But in RNN, the gradient

of the total error is
propagated to the
individual time steps and
also across the time steps
from the most recent time
step to the very beginning
of the sequence.
• Hence the name
“Backpropagation through
time”

11/11/2025 Mr. Pakruddin B, Assit. Prof., PSCS 103

Types of RNN
One to One RNN
One to Many RNN
Many to One RNN
Many to Many RNN

11/11/2025 Mr. Pakruddin B, Assit. Prof., PSCS 104

One to One RNN
One to One RNN is the most basic and traditional
type of Neural network giving a single output for a
single input, as can be seen in the above image.
It is also known as Vanilla Neural Network. It is used
to solve regular machine learning problems. Ex:
image classification

One to Many
One to Many is a kind of RNN architecture is applied
in situations that give multiple output for a single
input.

Image Captioning – Here, let’s say we have an image

for which we need a textual description. So we have a
single input – the image, and a series or sequence of
words as output. Here the image might be of a fixed
size, but the output is a description of varying lengths

11/11/2025 Mr. Pakruddin B, Assit. Prof., PSCS 105

Many to One
It takes a sequence of information as input and
outputs a fixed size of the output.
Many-to-one RNN architecture is usually seen for
sentiment analysis model as a common example. As
the name suggests, this kind of model is used when
multiple inputs are required to give a single output.
Take for example The Twitter sentiment analysis
model. In that model, a text input (words as
multiple inputs) gives its fixed sentiment (single
output).
Another example could be movie ratings model
that takes review texts as input to provide a rating
to a movie that may range from 1 to 5.
Many-to-Many
Many-to-Many RNN Architecture takes multiple
input and gives multiple output.
Ex: language translaion
Input is a sentence that has many words-> output
sentence with many words

11/11/2025 Mr. Pakruddin B, Assit. Prof., PSCS 106

11/11/2025 Mr. Pakruddin B, Assit. Prof., PSCS 107
 Problem of Long-Term Dependencies
Problem of Vanishing Gradients?
Multiply two small numbers(gradients) will result in a smaller number (gradient).
It becomes harder and harder for the neurons to propagate the error to the earlier
stages.
Hence the parameters will be biased only to capture short term dependencies.
RNNs predict the next word in a sequence, based on the relevant information in the
distant past
If the distance between the distant past and the current time step is small, RRNs
predict the next word correctly.

11/11/2025 Mr. Pakruddin B, Assit. Prof., PSCS 108

As the sequence length increases, RNNs won’t be able to remember the relevant
information in the distant past and predict the next word.

This is common in real life use cases with long sequences.

This is due to the vanishing gradient problem.

11/11/2025 Mr. Pakruddin B, Assit. Prof., PSCS 109

11/11/2025 Mr. Pakruddin B, Assit. Prof., PSCS 110
11/11/2025 Mr. Pakruddin B, Assit. Prof., PSCS 111
Trick 3 :
• Keep track of the long term dependencies by using “gates” .

• These “Gates” control which information from the “distant path” should be passed
through the network to update the current cell state.

• The most commonly used variants of RNN which are capable of remembering long term
dependencies using “gated cell” is the
LSTM (Long Short Term Memory) and GRU(gated recurrent unit).

• The “gates” perform different tensor operations to decide which information can be
removed/added to the current hidden state.

11/11/2025 Mr. Pakruddin B, Assit. Prof., PSCS 112

 Problems of RNN

Recurrent Neural Networks suffer from short-term memory.

If a sequence is long enough, they’ll have a hard time carrying
information from earlier time steps to later ones. So to process a
paragraph of text to do predictions, RNN’s may leave out important
information from the beginning.

During back propagation, recurrent neural networks suffer from the

vanishing gradient problem.
Layers that get a small gradient update stops learning.
Those are usually the earlier layers.
So because these layers don’t learn, RNN’s can forget what it seen in
longer sequences, thus having a short-term memory.

11/11/2025 Mr. Pakruddin B, Assit. Prof., PSCS 113

LSTMs and GRUs as a solution
LSTM ’s and GRU’s were created as the solution to short-term memory. They
have internal mechanisms called gates that can regulate the flow of information.

11/11/2025 Mr. Pakruddin B, Assit. Prof., PSCS 114

These gates can learn which data in a sequence is important to
keep or throw away.

By doing that, it can pass relevant information down the long

chain of sequences to make predictions.

LSTM’s and GRU’s can be found in speech recognition, speech

synthesis, and text generation. You can even use them to
generate captions for videos.

11/11/2025 Mr. Pakruddin B, Assit. Prof., PSCS 115

Recap of RNN
https://
[Link]/illustrated-guide-to-lstms-and-gru-s-a-step-by-step-explanation-44e9e
b85bf21
• How a cell in a RNN calculates the hidden state? (Short-term memory)
• It combines the current input and the previous hidden state into a vector
• This vector has information on the current input and previous inputs.
• The vector goes through the tanh activation, and the output is the new
hidden state, or the memory of the network.

11/11/2025 Mr. Pakruddin B, Assit. Prof., PSCS 116

Recap of RNN

The tanh activation is used to help regulate the values flowing through the network.
It squishes values to always be between -1 and 1.
RNNs involve less computational resources
But works well only for shorter sequences

11/11/2025 Mr. Pakruddin B, Assit. Prof., PSCS 117

LSTM

An LSTM has a similar control

flow as a recurrent neural
network.
It processes data passing on
information as it propagates
forward.
The differences are the operations
within the LSTM’s cells.
LSTMs have short-term memory
in “hidden states” and long-term
memory in “cell-states”

11/11/2025 Mr. Pakruddin B, Assit. Prof., PSCS 118

The core concept of LSTM’s are the cell state, and it’s various gates.
The cell state transfers relative information all the way down the sequence
chain. It helps to preserve the “long term memory” of the network.

The cell state, helps information from the earlier time steps to make it’s way to
later time steps, reducing the effects of short-term memory. As the cell state
goes on its journey, information get’s added or removed to the cell state via
gates.

The gates are different neural networks that decide which information is
allowed on the cell state. The gates can learn what information is relevant to
keep or forget during training.

11/11/2025 Mr. Pakruddin B, Assit. Prof., PSCS 119

 LSTM…

Gates use “sigmoid” activation function to update or forget data.

Sigmoid squishes its input values between 0 and 1.

If sigmoid squishes its input X closer to 0, then X is forgotten.

If sigmoid squishes its input X closer to 1, then X is kept.

The network can learn which data is not important therefore can be forgotten
or which data is important to keep using Sigmoid.
Three different gates regulate information flow in an LSTM cell. A forget gate,
input gate, and output gate.

11/11/2025 Mr. Pakruddin B, Assit. Prof., PSCS 120

LSTM

Forget Gate:
This gate decides what information should be thrown away or kept.
Information from the previous hidden state and information from the
current input is passed through the sigmoid function.
Values come out between 0 and 1.
The closer to 0 means to forget, and the closer to 1 means to keep.

11/11/2025 Mr. Pakruddin B, Assit. Prof., PSCS 121

11/11/2025 Mr. Pakruddin B, Assit. Prof., PSCS 122
11/11/2025 Mr. Pakruddin B, Assit. Prof., PSCS 123
Output gate:

Output gate has 2 layers:

“Tanh layer” generates a vector of new information that clould be written to the
cell state
“Sigmoid layer” decides which information should be kept from the output of
tanh function.

11/11/2025 Mr. Pakruddin B, Assit. Prof., PSCS 124

11/11/2025 Mr. Pakruddin B, Assit. Prof., PSCS 125
 Cell State :
Now we should have enough information to calculate the cell state.
First, the cell state gets pointwise multiplied by the forget vector. This has a
possibility of dropping values in the cell state if it gets multiplied by values near
0.

Then we take the output from the input gate and do a pointwise addition which
updates the cell state to new values that the neural network finds relevant.

That gives us our new cell state.

11/11/2025 Mr. Pakruddin B, Assit. Prof., PSCS 126

11/11/2025 Mr. Pakruddin B, Assit. Prof., PSCS 127
 To summarize the LSTM:
The “forget gate” with sigmoid function decides which information to be forgotten from
previous steps

The “input gate” along with a tanh and a sigmoid function decides what new inputs are
added to the network

The “output gate” updates the cell state, using the outputs from the previous two gates.

The tanh and sigmoid layers decide which part of the cell state are to be output to the
hidden state.

11/11/2025 Mr. Pakruddin B, Assit. Prof., PSCS 128

To review,
the Forget gate decides what is relevant to keep from prior
steps.
The input gate decides what information is relevant to add
from the current step.
The output gate determines what the next hidden state should
be.

11/11/2025 Mr. Pakruddin B, Assit. Prof., PSCS 129

Python Pseudo Code

11/11/2025 Mr. Pakruddin B, Assit. Prof., PSCS 130

 Example 1: Forward Pass in an LSTM Cell
Let’s assume the following values for an LSTM cell with one time step and one
unit:
Given Values:
Input at time step t: xt=0.5
Previous hidden state: ht-1 = 0.4
Previous cell state: Ct-1 = 0.3
Weights and biases:
Input gate: Wi=0.2, Ui=0.3, bi=0.11
Forget gate: Wf=0.5, Uf=0.4, bf=0.2
Cell state: Wc=0.3, Uc=0.2, bc=0.1
Output gate: Wo=0.4, Uo=0.3, bo=0.1

Step 1: Compute the Gates

LSTM has four main equations:

11/11/2025 Mr. Pakruddin B, Assit. Prof., PSCS 131

1. Forget Gate:
ft=σ(Wfxt+Ufht-1+bf)
ft=σ(0.5(0.5)+0.4(0.4)+0.2)
ft=σ(0.25+0.16+0.2)=σ(0.61)
Using the sigmoid function:
ft=1/1+e−0.61≈0.647
2. Input Gate:
it=σ(Wixi+Uiht-1+bi)
it=σ(0.2(0.5)+0.3(0.4)+0.1)
it=σ(0.1+0.12+0.1)=σ(0.32)
it=11+e−0.32≈0.579
3. Candidate Cell State:
C~t=tanh(Wc xt+Ucht-1+bc)
C~t=tanh(0.3(0.5)+0.2(0.4)+0.1)
C~t=tanh(0.15+0.08+0.1)=tanh(0.33)
Using the tanh function:
C~t≈0.318

11/11/2025 Mr. Pakruddin B, Assit. Prof., PSCS 132

[Link] Gate:
Ot=σ(Woxt+Uoht-1+bo)
Ot=σ(0.4(0.5)+0.3(0.4)+0.1)
Ot=σ(0.2+0.12+0.1)=σ(0.42)
Gate / State Computed Value
Ot=11+e−0.42≈0.603
Forget Gate ftf_t 0.647
Input Gate iti_t 0.579
Step 2: Compute New Cell State
Ct=ftCt-1+it C~t Candidate Cell C~t\
0.318
tilde{C}_t
Ct=(0.647)(0.3)+(0.579)(0.318)
Output Gate oto_t 0.603
C_t = 0.194 + 0.184 = 0.378
New Cell State CtC_t 0.378
Step 3: Compute New Hidden State
ht=ot tanh(Ct) New Hidden State
0.217
hth_t
ht=(0.603)tanh(0.378)
ht=(0.603)(0.361)=0.217

🔹 Summary of Values at tt

11/11/2025 Mr. Pakruddin B, Assit. Prof., PSCS 133

GRU

The GRU is the newer generation of Recurrent Neural networks and is pretty similar
to an LSTM.
GRU’s got rid of the cell state and used the hidden state to transfer information.
It also only has two gates, a reset gate and update gate.

11/11/2025 Mr. Pakruddin B, Assit. Prof., PSCS 134

Update Gate
The update gate acts similar to the forget and input gate of
an LSTM. It decides what information to throw away and
what new information to add.
Reset Gate
The reset gate is another gate is used to decide how much
past information to forget.
GRU’s has fewer tensor operations; therefore, they are a
little speedier to train then LSTM’s.
There isn’t a clear winner which one is better.
Researchers and engineers usually try both to determine
which one works better for their use case.

11/11/2025 Mr. Pakruddin B, Assit. Prof., PSCS 135

 Recurrent neural network with Gated Recurrent Unit

11/11/2025 Mr. Pakruddin B, Assit. Prof., PSCS 136

 Gated Recurrent Unit

11/11/2025 Mr. Pakruddin B, Assit. Prof., PSCS 137

 Step 1: Update gate

When x_t is plugged into the network unit, it is multiplied by its own weight W(z).
The same goes for h_(t-1) which holds the information for the previous t-1 units and
is multiplied by its own weight U(z). Both results are added together and a sigmoid
activation function is applied to squash the result between 0 and 1. Following the
above schema, we have:

11/11/2025 Mr. Pakruddin B, Assit. Prof., PSCS 138

 Step 1: Update gate

The update gate helps the

model to determine how
much of the past
information (from previous
time steps) needs to be
passed along to the future.

11/11/2025 Mr. Pakruddin B, Assit. Prof., PSCS 139

 Step 2: Reset gate

Essentially, this gate is used from the model to decide how much of the past
information to forget. To calculate it, we use:

This formula is the same as the one for the update gate. The difference comes in
the weights and the gate’s usage, which will see in a bit. The schema below
shows where the reset gate is:

11/11/2025 Mr. Pakruddin B, Assit. Prof., PSCS 140

 Step 2: Reset gate

As before, we plug in h_(t-1) —

blue line and x_t — purple line,
multiply them with their
corresponding weights, sum the
results and apply the sigmoid
function.

11/11/2025 Mr. Pakruddin B, Assit. Prof., PSCS 141

 Step 3: Current memory content
First, we start with the usage of the reset gate. We introduce a new memory content
which will use the reset gate to store the relevant information from the past. It is
calculated as follows:

• Multiply the input x_t with a weight W and h_(t-1) with a weight U.
• Calculate the Hadamard (element-wise) product between the reset
gate r_t and Uh_(t-1). That will determine what to remove from the previous time
steps. Let’s say we have a sentiment analysis problem for determining one’s opinion
about a book from a review he wrote.

11/11/2025 Mr. Pakruddin B, Assit. Prof., PSCS 142

 Step 3: Current memory content
The text starts with “This is a fantasy book which illustrates…” and after a couple
paragraphs ends with “I didn’t quite enjoy the book because I think it captures too
many details.”

To determine the overall level of satisfaction from the book we only need the last part
of the review. In that case as the neural network approaches to the end of the text it
will learn to assign r_t vector close to 0, washing out the past and focusing only on
the last sentences.

Sum up the results of step 1 and 2.

Apply the nonlinear activation function tanh.

11/11/2025 Mr. Pakruddin B, Assit. Prof., PSCS 143

 Step 3: Current memory content

We do an element-wise
multiplication of h_(t-1) —
blue line and r_t — orange
line and then sum the result —
pink line with the input x_t —
purple line. Finally, tanh is
used to produce h’_t — bright
green line.

11/11/2025 Mr. Pakruddin B, Assit. Prof., PSCS 144

 Step 4: Final memory at current time step

As the last step, the network needs to calculate h_t — vector which holds information
for the current unit and passes it down to the network. In order to do that the update
gate is needed. It determines what to collect from the current memory content
— h’_t and what from the previous steps — h_(t-1). That is done as follows:

[Link] element-wise multiplication to the update gate z_t and h_(t-1).

[Link] element-wise multiplication to (1-z_t) and h’_t.
[Link] the results from step 1 and 2.

Let’s bring up the example about the book review. This time, the most relevant
information is positioned in the beginning of the text. The model can learn to set
the vector z_t close to 1 and keep a majority of the previous information.
Since z_t will be close to 1 at this time step, 1-z_t will be close to 0 which will ignore
big portion of the current content (in this case the last part of the review which
explains the book plot) which is irrelevant for our prediction.

11/11/2025 Mr. Pakruddin B, Assit. Prof., PSCS 145

 Step 4: Final memory at current time step

Following through, you can

see how z_t — green
line is used to calculate 1-
z_t which, combined
with h’_t — bright green
line, produces a result in
the dark red line. z_t is
also used with h_(t-1) —
blue line in an element-
wise multiplication.

Finally, h_t — blue line is a

result of the summation of
the outputs corresponding
to the bright and dark red
lines.

11/11/2025 Mr. Pakruddin B, Assit. Prof., PSCS 146

 The End

Thank You

11/11/2025 Mr. Pakruddin B, Assit. Prof., PSCS 147