Deep Learning

Subject Code – EC37T

Course Pre-requisite:EC37P

Dr. Nayana Mahajan

9/16/2025 Dr. Nayana Mahajan 1

Module III: Recurrent Neural Networks (RNNs)
and Sequence Modeling

 Introduction to RNNs, Bidirectional RNN,

Backpropagation Through Time (BTT), Vanishing and
Exploding Gradients,

 Long Short-Term Memory (LSTM) : Selective Read,

Selective write, Selective Forget, and Gated Recurrent
Units (GRUs)

9/16/2025 Dr. Nayana Mahajan 2

 Recurrent Neural Networks (RNNs) are a type of neural
network designed to handle sequential data by
maintaining a hidden state that captures information from
previous inputs in the sequence.

 Bidirectional RNNs extend this by processing the

sequence in both forward and backward directions.

9/16/2025 Dr. Nayana Mahajan 3

 Backpropagation Through Time (BPTT) is the algorithm
used to train RNNs, unfolding the network over time to
calculate gradients.

 Vanishing and exploding gradients are problems that can

occur during BPTT, where gradients become too small or
too large, hindering effective learning.

9/16/2025 Dr. Nayana Mahajan 4

Recurrent Neural Networks (RNNs):
 RNNs are specifically designed to work with sequential
data, such as text, time series, or audio.

 Unlike traditional neural networks, RNNs have a

"memory" in the form of a hidden state that allows them
to retain information from previous inputs in the
sequence.

9/16/2025 Dr. Nayana Mahajan 5

Recurrent Neural Networks (RNNs):
 At each time step, the RNN takes an input and the
previous hidden state, processes them, and produces an
output and a new hidden state.

 This allows the network to consider the context of the

entire sequence when making predictions or processing
data.

9/16/2025 Dr. Nayana Mahajan 6

 A Recurrent Neural Network works on the principle of
saving the output of a particular layer and feeding this
back to the input in order to predict the output of the
layer.

9/16/2025 Dr. Nayana Mahajan 7

What is Recurrent Neural Networks
(RNN)?
 Recurrent Neural networks imitate the function of the human brain in the
fields of Data science, Artificial intelligence, machine learning, and deep
learning, allowing computer programs to recognize patterns and solve
common issues.

 RNNs are a type of neural network that can model sequence data.

 RNNs, which are formed from feedforward networks, are similar to human
brains in their behaviour.

 Simply said, recurrent neural networks can anticipate sequential data in a

way that other algorithms can’t.
9/16/2025 Dr. Nayana Mahajan 8
 All of the inputs and outputs in standard neural networks are
independent of one another.

 However, in some circumstances, such as when predicting

the next word of a phrase, the prior words are necessary,
and so the previous words must be remembered.

 As a result, RNN was created, which used a hidden layer to

overcome the problem.

 The most important component of RNN is the hidden state,

which remembers specific information about a sequence.


9/16/2025 Dr. Nayana Mahajan 9

9/16/2025 Dr. Nayana Mahajan 10
 Recurrent Neural Networks (RNNs) differ from
regular neural networks in how they process
information.

 While standard neural networks pass information in one

direction i.e from input to output, RNNs feed
information back into the network at each step.

9/16/2025 Dr. Nayana Mahajan 11

9/16/2025 Dr. Nayana Mahajan 12
Issues in Feed-Forward network:
 Cannot Handle Sequential Data.
 Consider only the current input.
 Cannot memorize previous inputs.

9/16/2025 Dr. Nayana Mahajan 13

The solution to these issues in Recurrent
Neural Network (RNN):
 Can Handle Sequential Data.
 Accepting the current input data and previously received
inputs.
 RNNs can memorize previous inputs due to their
internal memory.

9/16/2025 Dr. Nayana Mahajan 14

9/16/2025 Dr. Nayana Mahajan 15
9/16/2025 Dr. Nayana Mahajan 16
Core Components and Operation:

Input:
 RNNs take a sequence of inputs, one at a time, at each time
step.

Hidden State:
 The network maintains a hidden state vector that stores
information about the past inputs in the sequence.

Output:
 At each time step, the RNN produces an output based on
the current input and the hidden state.
9/16/2025 Dr. Nayana Mahajan 17
Core Components and Operation:

Parameter Sharing:
 A key feature of RNNs is that they share the same set of
parameters (weights and biases) across all time steps.

 This means the same weight matrices are applied to the

input and hidden state at each step, allowing the network
to learn patterns in sequential data.

9/16/2025 Dr. Nayana Mahajan 18

Below is how you can convert a Feed-
Forward Neural Network into a Recurrent Neural Network:

9/16/2025 Dr. Nayana Mahajan 19

 The nodes in different layers of the neural network are
compressed to form a single layer of recurrent neural
networks.
 A, B, and C are the parameters of the network.

9/16/2025 Dr. Nayana Mahajan 20

9/16/2025 Dr. Nayana Mahajan 21
 Here, “x” is the input layer, “h” is the hidden layer, and “y” is
the output layer.

 A, B, and C are the network parameters used to improve the

output of the model.

 At any given time t, the current input is a combination of

input at x(t) and x(t-1).

 The output at any given time is fetched back to the network

to improve on the output.

9/16/2025 Dr. Nayana Mahajan 22

9/16/2025 Dr. Nayana Mahajan 23
Notations:

9/16/2025 Dr. Nayana Mahajan 24

How RNNs Work:
 Recurrent Neural Networks (RNNs) process sequential
data by using a self-looping mechanism where the hidden
layer's output from one step is fed back as input for the
next step.

 This allows the network to maintain a "memory" of past

inputs, enabling it to understand and predict based on
the sequence of data.

9/16/2025 Dr. Nayana Mahajan 25

9/16/2025 Dr. Nayana Mahajan 26
How RNNs Work:
[Link] Input:
 RNNs take data in a sequence, one element at a time.

2. Hidden State:
 The hidden layer, which is the core of an RNN, acts as a
memory, storing information about past inputs.

3. Feedback Loop:
 The hidden layer's output from one step is fed back as input
to the next step, creating a loop.
9/16/2025 Dr. Nayana Mahajan 27
How RNNs Work:
4. Contextual Understanding:
 By using past inputs, the RNN can understand the
context of the current input, making it useful for tasks
like language modeling and time series analysis.

5. Parameter Sharing:
 The same parameters (weights and biases) are used
across all time steps, meaning the network learns a single
set of rules that can be applied to any point in the
sequence.
9/16/2025 Dr. Nayana Mahajan 28
9/16/2025 Dr. Nayana Mahajan 29
How does RNN work?
 At each time step RNNs process units with a fixed
activation function.

 These units have an internal hidden state that acts as

memory that retains information from previous time
steps.

 This memory allows the network to store past

knowledge and adapt based on new inputs.
9/16/2025 Dr. Nayana Mahajan 30
Updating the Hidden State in RNNs

9/16/2025 Dr. Nayana Mahajan 31

9/16/2025 Dr. Nayana Mahajan 32
9/16/2025 Dr. Nayana Mahajan 33
9/16/2025 Dr. Nayana Mahajan 34
9/16/2025 Dr. Nayana Mahajan 35
2. RNN Unfolding
 RNN unfolding or unrolling is the process of expanding
the recurrent structure over time steps.

 During unfolding each step of the sequence is

represented as a separate layer in a series illustrating
how information flows across each time step.

9/16/2025 Dr. Nayana Mahajan 36

2. RNN Unfolding
 This unrolling enables backpropagation through
time (BPTT) a learning process where errors are
propagated across time steps to adjust the network’s
weights enhancing the RNN’s ability to learn
dependencies within sequential data.

9/16/2025 Dr. Nayana Mahajan 37

9/16/2025 Dr. Nayana Mahajan 38
Recurrent Neural Network Architecture
 RNNs share similarities in input and output structures with
other deep learning architectures but differ significantly in
how information flows from input to output.

 Unlike traditional deep neural networks where each dense

layer has distinct weight matrices.

 RNNs use shared weights across time steps, allowing them

to remember information over sequences.

9/16/2025 Dr. Nayana Mahajan 39

RNN Architecture

 Input Layer: This layer receives the initial element of

the sequence data. For example, in a sentence, it might
receive the first word as a vector representation.

 Hidden Layer: The heart of the RNN, the hidden layer

contains a set of interconnected neurons. Each neuron
processes the current input along with the information
from the previous hidden layer’s state. This “state”
captures the network’s memory of past inputs, allowing it
to understand the current element in context.
9/16/2025 Dr. Nayana Mahajan 40
RNN Architecture

 Activation Function: This function introduces non-

linearity into the network, enabling it to learn complex
patterns. It transforms the combined input from the
current input layer and the previous hidden layer state
before passing it on.

 Output Layer: The output layer generates the

network’s prediction based on the processed
information. In a language model, it might predict the
next word in the sequence.
9/16/2025 Dr. Nayana Mahajan 41
RNN Architecture

 Recurrent Connection: A key distinction of RNNs is

the recurrent connection within the hidden layer.

 This connection allows the network to pass the hidden

state information (the network’s memory) to the next
time step.

 It’s like passing a baton in a relay race, carrying

information about previous inputs forward
9/16/2025 Dr. Nayana Mahajan 42
9/16/2025 Dr. Nayana Mahajan 43
9/16/2025 Dr. Nayana Mahajan 44
9/16/2025 Dr. Nayana Mahajan 45
9/16/2025 Dr. Nayana Mahajan 46
How does RNN work?

 At each time step RNNs process units with a fixed

activation function.

 These units have an internal hidden state that acts as

memory that retains information from previous time
steps.

 This memory allows the network to store past

knowledge and adapt based on new inputs

9/16/2025 Dr. Nayana Mahajan 47

9/16/2025 Dr. Nayana Mahajan 48
9/16/2025 Dr. Nayana Mahajan 49
9/16/2025 Dr. Nayana Mahajan 50
9/16/2025 Dr. Nayana Mahajan 51
Bidirectional RNNs
 A Bidirectional Recurrent Neural Network (BRNN) is an
extension of the traditional RNN that processes
sequential data in both forward and backward directions.

 This allows the network to utilize both past and future

context when making predictions providing a more
comprehensive understanding of the sequence.


9/16/2025 Dr. Nayana Mahajan 52

Bidirectional RNNs
 Like a traditional RNN, a BRNN moves forward through the
sequence, updating the hidden state based on the current
input and the prior hidden state at each time step.

 The key difference is that a BRNN also has a backward

hidden layer which processes the sequence in reverse,
updating the hidden state based on the current input and the
hidden state of the next time step.


9/16/2025 Dr. Nayana Mahajan 53

Bidirectional RNNs
 Compared to unidirectional RNNs BRNNs improve
accuracy by considering both the past and future context.

 This is because the two hidden layers i.e forward and

backward complement each other and predictions are
made using the combined outputs of both layers.


9/16/2025 Dr. Nayana Mahajan 54

9/16/2025 Dr. Nayana Mahajan 55
Bidirectional RNNs:

Purpose:
 They process sequences in both forward and backward
directions simultaneously.

Mechanism:
 They use two hidden layers, one processing the sequence
from beginning to end and the other from end to
beginning.

9/16/2025 Dr. Nayana Mahajan 56

Bidirectional RNNs:

Benefit:
 This allows the network to capture context from both past
and future elements in the sequence, leading to improved
performance on certain tasks.

 Example:
 In speech recognition, a bidirectional RNN can consider both
the preceding and following sounds to better understand a
particular phoneme.

9/16/2025 Dr. Nayana Mahajan 57

Working of Bidirectional Recurrent Neural
Networks (BRNNs)
 1. Inputting a Sequence: A sequence of data points each
represented as a vector with the same dimensionality is fed
into the BRNN. The sequence may have varying lengths.

2. Dual Processing: BRNNs process data in two directions:

Forward direction: The hidden state at each time step is
determined by the current input and the previous hidden
state.
 Backward direction: The hidden state at each time step is
influenced by the current input and the next hidden state.

9/16/2025 Dr. Nayana Mahajan 58

 3. Computing the Hidden State: A non-linear activation
function is applied to the weighted sum of the input and the
previous hidden state creating a memory mechanism that
allows the network to retain information from earlier steps.

 4. Determining the Output: A non-linear activation

function is applied to the weighted sum of the hidden state
and output weights to compute the output at each step.

This output can either be:

The final output of the network.

An input to another layer for further processing.

9/16/2025 Dr. Nayana Mahajan 59
Advantages of BRNNs
 Enhanced Context Understanding: Considers both
past and future data for improved predictions.
 Improved Accuracy: Particularly effective for NLP and
speech processing tasks.
 Better Handling of Variable-Length Sequences:
More flexible than traditional RNNs making it suitable
for varying sequence lengths.
 Increased Robustness: Forward and backward
processing help filter out noise and irrelevant
information, improving robustness.
9/16/2025 Dr. Nayana Mahajan 60
Challenges of BRNNs
 High Computational Cost: Requires twice the
processing time compared to unidirectional RNNs.
 Longer Training Time: More parameters to optimize
result in slower convergence.
 Limited Real-Time Applicability: Since predictions
depend on the entire sequence hence they are not ideal
for real-time applications like live speech recognition.
 Less Interpretability: The bidirectional nature of
BRNNs makes it more difficult to interpret predictions
compared to standard RNNs.
9/16/2025 Dr. Nayana Mahajan 61
Applications of Bidirectional Recurrent
Neural Networks (BRNNs)
 BRNNs are widely used in various natural language processing
(NLP) tasks, including:
 Sentiment Analysis: By considering both past and future context
they can better classify the sentiment of a sentence.
 Named Entity Recognition (NER): It helps to identify entities in
sentences by analyzing the context in both directions.
 Machine Translation: In encoder-decoder models, BRNNs allow
the encoder to capture the full context of the source sentence in
both directions hence improving translation accuracy.
 Speech Recognition: By considering both previous and future
speech elements it enhance the accuracy of transcribing audio.

9/16/2025 Dr. Nayana Mahajan 62

Backpropagation Through Time (BPTT) in
RNNs
 Recurrent Neural Network uses the Backpropagation
algorithm, but it is applied for every timestamp.

 It is known as Backpropagation Through Time.

9/16/2025 Dr. Nayana Mahajan 63

Backpropagation Through Time (BTT)

 The backpropagation learning algorithm is an extension

of standard backpropagation that performs gradient
descent on an unfolded network.

 The gradient descent weight updates have contributions

from each timestamp.

 The errors have to be back-propagated through time as

well as through the network.
9/16/2025 Dr. Nayana Mahajan 64
Backpropagation Through Time (BTT)

 Backpropagation Through Time (BPTT) is a specialized version of

the backpropagation algorithm used to train Recurrent Neural
Networks (RNNs).

 It addresses the challenge of training RNNs on sequential data by

effectively unfolding the network through time and applying
backpropagation to the resulting unfolded structure.

 This allows the network to learn temporal dependencies and

patterns within the sequence.

9/16/2025 Dr. Nayana Mahajan 65

Backpropagation Through Time (BPTT) in
RNNs

9/16/2025 Dr. Nayana Mahajan 66

Backpropagation Through Time (BPTT) in
RNNs

9/16/2025 Dr. Nayana Mahajan 67

9/16/2025 Dr. Nayana Mahajan 68
9/16/2025 Dr. Nayana Mahajan 69
The Need for BPTT
 Traditional feedforward neural networks process data in
one direction.

 RNNs, however, have feedback loops, meaning the output

at a given time step depends on past inputs and hidden
states.

 BPTT is specifically designed to handle this temporal

dependency in RNNs, enabling the network to learn
from sequences of data.
9/16/2025 Dr. Nayana Mahajan 70
9/16/2025 Dr. Nayana Mahajan 71
How BPTT Works:
Unfolding the RNN:
 BPTT essentially unrolls the RNN into a deep feedforward network
where each layer represents the network at a specific time step.

Shared Weights:
 The weights connecting the nodes are shared across all time steps
in the unfolded network.

Gradient Calculation:
 The backpropagation algorithm is then applied to this unfolded
network. Gradients are calculated for each time step and summed
up to update the shared weights, allowing the network to learn
from the entire sequence.
9/16/2025 Dr. Nayana Mahajan 72
Why BPTT IS Important:
Temporal Dependencies:
 BPTT is crucial for RNNs to learn long-range
dependencies in sequential data.

Sequence Processing:
 It enables RNNs to be effectively trained for tasks like
natural language processing (where word order matters),
time series analysis, and speech recognition.

9/16/2025 Dr. Nayana Mahajan 73

Key Considerations:
 Computational Cost:
 BPTT can be computationally expensive, especially for long
sequences, due to the need to unfold the network through
time.

 Truncated BPTT:
 To address the computational cost, Truncated BPTT (TBPTT)
is often [Link] limits the backpropagation to a certain
number of time steps, making it more efficient while still
capturing relevant temporal dependencies.

9/16/2025 Dr. Nayana Mahajan 74

Advantages of Backpropagation Through
Time (BPTT)
 Captures Temporal Dependencies: BPTT allows
RNNs to learn relationships across time steps, crucial for
sequential data like speech, text and time series.
 Unfolding over Time: By considering all previous
states during training, BPTT helps the model understand
how past inputs influence future outputs.

9/16/2025 Dr. Nayana Mahajan 75

Advantages of Backpropagation Through
Time (BPTT)
 Foundation for Modern RNNs: BPTT forms the basis
for training advanced architectures such as LSTMs and
GRUs, enabling effective learning of long sequences.
 Flexible for Variable Length Sequences: It can
handle input sequences of varying lengths, adapting
gradient calculations accordingly.

9/16/2025 Dr. Nayana Mahajan 76

Types Of Recurrent Neural Networks
 There are four types of RNNs based on the number of inputs and outputs
in the network:
 One-to-One RNN
 This is the simplest type of neural network architecture where there is a
single input and a single output. It is used for straightforward classification
tasks such as binary classification where no sequential data is involved.

9/16/2025 Dr. Nayana Mahajan 77

9/16/2025 Dr. Nayana Mahajan 78
One-to-Many RNN
 In a One-to-Many RNN the network
processes a single input to produce
multiple outputs over time.

 This is useful in tasks where one input

triggers a sequence of predictions
(outputs).

 For example in image captioning a

single image can be used as input to
generate a sequence of words as a
caption.


9/16/2025 Dr. Nayana Mahajan 79

9/16/2025 Dr. Nayana Mahajan 80
3. Many-to-One RNN

 The Many-to-One RNN receives a

sequence of inputs and generates a
single output.

 This type is useful when the overall

context of the input sequence is
needed to make one prediction.

 In sentiment analysis the model

receives a sequence of words (like a
sentence) and produces a single
output like positive, negative or
neutral.

9/16/2025 Dr. Nayana Mahajan 81

9/16/2025 Dr. Nayana Mahajan 82
4. Many-to-Many RNN

 The Many-to-Many RNN type

processes a sequence of inputs
and generates a sequence of
outputs.

 In language translation task a

sequence of words in one language
is given as input and a
corresponding sequence in
another language is generated as
output.


9/16/2025 Dr. Nayana Mahajan 83

9/16/2025 Dr. Nayana Mahajan 84
9/16/2025 Dr. Nayana Mahajan 85
9/16/2025 Dr. Nayana Mahajan 86
9/16/2025 Dr. Nayana Mahajan 87
Advantages of Backpropagation Through
Time (BPTT)
 Captures Temporal Dependencies: BPTT allows
RNNs to learn relationships across time steps, crucial for
sequential data like speech, text and time series.

 Unfolding over Time: By considering all previous

states during training, BPTT helps the model understand
how past inputs influence future outputs.

9/16/2025 Dr. Nayana Mahajan 88

Advantages of Backpropagation Through
Time (BPTT)
 Foundation for Modern RNNs: BPTT forms the basis
for training advanced architectures such as LSTMs and
GRUs, enabling effective learning of long sequences.

 Flexible for Variable Length Sequences: It can

handle input sequences of varying lengths, adapting
gradient calculations accordingly.

9/16/2025 Dr. Nayana Mahajan 89

Limitations of BPTT
 Vanishing Gradient Problem: When backpropagating
over many time steps, gradients tend to shrink
exponentially, making early time steps contribute very
little to weight updates. This causes the network to
“forget” long-term dependencies.

 Exploding Gradient Problem: Gradients may also

grow uncontrollably large, causing unstable updates and
making training difficult.
9/16/2025 Dr. Nayana Mahajan 90
Limitations of Backpropagation Through
Time:
 Vanishing Gradient
 Exploding Gradient

 Recurrent Neural Networks enable you to model time-

dependent and sequential data problems, such as stock
market prediction, machine translation, and text generation.

 You will find, however, that recurrent Neural Networks are

hard to train because of the gradient problem.

9/16/2025 Dr. Nayana Mahajan 91

Vanishing and Exploding Gradients:
Problem:
 These are issues that can occur during BPTT, where the
gradients (which guide weight updates) become
extremely small (vanishing) or extremely large
(exploding).

Cause:
 The repeated multiplication of gradients during
backpropagation can lead to these issues, especially in
long sequences.
9/16/2025 Dr. Nayana Mahajan 92
Vanishing and Exploding Gradients:
Impact:
 Vanishing gradients make it difficult for the network to
learn long-range dependencies, while exploding gradients
can lead to unstable training and divergence.

Mitigation:
 Techniques like gradient clipping (limiting the maximum
gradient value) and more advanced architectures like
LSTMs (Long Short-Term Memory) are used to address
these issues.
9/16/2025 Dr. Nayana Mahajan 93
9/16/2025 Dr. Nayana Mahajan 94
Vanishing Gradient Problem
 RNNs suffer from the problem of vanishing gradients.

 The gradients carry information used in the RNN, and

when the gradient becomes too small, the parameter
updates become insignificant.

 This makes the learning of long data sequences difficult.

9/16/2025 Dr. Nayana Mahajan 95

Vanishing Gradients:
 Occur when gradients become extremely small as they
are propagated backward through multiple layers.

 This makes it difficult for the network to update weights

in the earlier layers, especially in deep networks.

 Can be caused by activation functions like sigmoid and

tanh, which can saturate and result in small gradients for
large inputs.

9/16/2025 Dr. Nayana Mahajan 96

Vanishing Gradients:

 Repeated multiplication of small gradients can cause

them to vanish exponentially fast, according to a Medium
article.

 Solutions include using different activation functions (like

ReLU) and techniques like batch normalization.

9/16/2025 Dr. Nayana Mahajan 97

9/16/2025 Dr. Nayana Mahajan 98
Exploding Gradient Problem
 While training a neural network, if the slope tends to
grow exponentially instead of decaying, this is called an
Exploding Gradient.

 This problem arises when large error gradients

accumulate, resulting in very large updates to the neural
network model weights during the training process.

9/16/2025 Dr. Nayana Mahajan 99

Exploding Gradients:
 Occur when gradients become extremely large during
backpropagation.

 This can cause large weight updates, leading to unstable

training and potentially causing the network to diverge
from the optimal solution.

 Exploding gradients can be caused by the repeated

multiplication of large gradients, which can grow
exponentially fast.
9/16/2025 Dr. Nayana Mahajan 100
Exploding Gradients:
 Solutions include gradient clipping, which limits the
maximum magnitude of the gradients.

 It can also be caused by the weights themselves, not just

the activation function.

9/16/2025 Dr. Nayana Mahajan 101

 During the backpropagation algorithm, gradients are
computed and propagated backward through the layers of a
neural network.

 The vanishing gradient problem occurs when these gradients

become extremely small as they propagate backward
through multiple layers.

 Conversely, the exploding gradient problem happens when

gradients grow exponentially during backpropagation.

9/16/2025 Dr. Nayana Mahajan 102

 Before delving into the reasons behind the occurrence of the
vanishing and exploding gradient problems,

 it is essential to understand the mathematics behind

backpropagation, which calculates the gradient of the loss
function with respect to the weights.

 Backpropagation involves computing the partial derivatives

of the loss function with respect to each weight in the
network, representing their contribution to the overall loss:

9/16/2025 Dr. Nayana Mahajan 103

9/16/2025 Dr. Nayana Mahajan 104
Vanishing Gradient Problem
 As we move backward through the layers of a deep neural network,
the chain rule causes the gradient to be multiplied repeatedly.

 This multiplication can lead to a significant increase in the

magnitude of the gradient.

 However, when using the sigmoid activation function (σ), which has a
saturating nature, its derivative is always less than 0.25 (i.e., much
smaller than 1) for a wide range of input values.

 Consequently, the gradients of the weights in the lower layers of the

network become the products of many small numbers.

9/16/2025 Dr. Nayana Mahajan 105

9/16/2025 Dr. Nayana Mahajan 106
Vanishing Gradient Problem
 Due to the multiplication of these small values, the gradients
associated with the weights in the lower layers gradually
diminish in size.

 This phenomenon is known as the vanishing gradient

problem.

 The small gradients make it challenging for the lower layers

to receive meaningful updates during the training process,
which can hinder the learning process and result in slower
convergence or even stagnation in training progress.

9/16/2025 Dr. Nayana Mahajan 107

Exploding Gradient Problem
 The exploding gradient problem can occur when the
product of the gradient values becomes larger than 1 as
it propagates backward through the layers of the
network.

 This can lead to gradients with magnitudes that grow

exponentially, making training unstable and potentially
causing numerical issues.


9/16/2025 Dr. Nayana Mahajan 108

Exploding Gradient Problem
 important to note that the exploding gradient problem is
not solely attributed to a specific activation function.

 It is influenced by multiple factors, including the weights’

initial values, the choice of activation functions
throughout the network, the depth and structure of the
network, and the optimization algorithm used during
training.


9/16/2025 Dr. Nayana Mahajan 109

Addressing Vanishing and Exploding Gradients:

Activation Functions:
 ReLU and its variants (e.g., Leaky ReLU) are often preferred over sigmoid
and tanh because they tend to have larger gradients and avoid saturation,
according to a Medium article.

Batch Normalization:
 Helps stabilize the training process by normalizing the activations within
each layer, which can prevent gradients from becoming too large or too
small.

Weight Initialization:
 Carefully choosing the initial weights can help prevent gradients from
vanishing or exploding.

9/16/2025 Dr. Nayana Mahajan 110

Addressing Vanishing and Exploding Gradients:

Gradient Clipping:
 Limits the maximum magnitude of the gradients during
backpropagation, preventing exploding gradients.

Fewer Layers:
 Reducing the number of layers can sometimes mitigate
the impact of vanishing or exploding gradients.

9/16/2025 Dr. Nayana Mahajan 111

Major Issues in Gradient problems:
 Long Training Time
 Poor Performance
 Bad Accuracy

9/16/2025 Dr. Nayana Mahajan 112

Solutions

 Long Short-Term Memory (LSTM): Special RNN

cells designed to maintain information over longer
sequences and mitigate vanishing gradients.
 Gradient Clipping: Limits the magnitude of gradients
during backpropagation to prevent explosion by
normalizing them when exceeding a threshold.

9/16/2025 Dr. Nayana Mahajan 113

Numerical Example on RNN

9/16/2025 Dr. Nayana Mahajan 114

9/16/2025 Dr. Nayana Mahajan 115
9/16/2025 Dr. Nayana Mahajan 116
9/16/2025 Dr. Nayana Mahajan 117
9/16/2025 Dr. Nayana Mahajan 118
9/16/2025 Dr. Nayana Mahajan 119
9/16/2025 Dr. Nayana Mahajan 120