Module 5

1. Explain how Recurrent Neural Networks (RNNs) process data

sequences. Use a diagram to show the unfolding of the
computational graph.

Recurrent Neural Networks (RNNs) are a family of neural networks

designed for processing sequential data.
Unlike feedforward networks, RNNs are specialized to process a
sequence of values
x (1), x (2), … , x (τ )
by maintaining a state that captures information from previous time
steps.
RNNs can process variable-length sequences and share
parameters across different time steps, which allows them to
generalize across sequences of different lengths.

Recurrent Computation
A classical recurrent system is defined as:
s(t )=f (s (t−1); θ)
where s(t )is the state of the system at time t .
When the system is driven by an external input sequence x (t), the
recurrence becomes:
s(t )=f (s (t−1), x (t); θ)
In recurrent neural networks, the state is represented by the hidden
units h(t):
h(t)=f (h(t−1), x (t); θ)
Here:
 h(t)summarizes information from the entire past sequence
 The same parameters θ are used at every time step

Unfolding the Computational Graph

The recurrent definition contains cycles, which makes it difficult to
visualize and compute gradients.
To address this, the recurrent computation is unfolded over time.
For a finite sequence of length τ , the recurrence is expanded
repeatedly:
h(3)=f (f (h(1); θ); θ)
After unfolding:
 Each time step is represented as a separate node
 The graph becomes a directed acyclic computational graph
 Parameters are shared across all time steps
The unfolded computational graph shows how information flows:
 Forward in time during forward propagation
 Backward in time during gradient computation

Forward Propagation in a Typical RNN

For an RNN that maps an input sequence to an output sequence, the
forward propagation equations are:
a (t)=b +Wh(t−1)+Ux(t)h(t)=tanh ⁡(a(t))o (t)=c+Vh(t) ^y (t )=softmax(o(t))
where:
 U is the input-to-hidden weight matrix
 W is the hidden-to-hidden recurrent weight matrix
 V is the hidden-to-output weight matrix
 b , c are bias vectors
The hidden state h(t)acts as a lossy summary of all past inputs up
to time t .

 The unfolding of a recurrent neural network into a computational

graph is illustrated in Figure 10.2
 The classical dynamical system unfolding is shown in Figure 10.1

 The unfolded graph contains repeated applications of the same

function f with shared parameters
 Each node corresponds to a specific time step

Thus, RNNs process data sequences by maintaining a hidden state that

is updated at each time step using the current input and the previous
state.
By unfolding the recurrent computation into a computational graph,
RNNs enable efficient learning through parameter sharing and allow
information to flow across time steps in a structured manner.
2. Discuss the challenges of learning long-term dependencies in RNNs.
Explain the vanishing and exploding gradient problem and how they
impact training.

Challenges of Learning Long-Term Dependencies in RNNs

One of the main difficulties in training recurrent neural networks is
learning long-term dependencies, that is, relationships between
inputs that are far apart in the sequence.
The mathematical challenge of learning long-term dependencies in
recurrent networks arises during gradient-based training,
particularly when using back-propagation through time (BPTT).

Vanishing and Exploding Gradient Problem

During training, gradients are propagated backward through the
unfolded computational graph of the RNN.
This involves repeated multiplication of Jacobian matrices
corresponding to the recurrent transformations at each time step.
As the number of time steps increases, these repeated
multiplications can cause the gradients to:
 Shrink exponentially → vanishing gradients
 Grow exponentially → exploding gradients

1. Vanishing Gradients
Vanishing gradients occur when the magnitudes of gradients
decrease exponentially as they are propagated backward through
many time steps.
 Long-term interactions involve the multiplication of many Jacobian
matrices
 These multiplications assign exponentially smaller weights to
long-term dependencies compared to short-term ones
 As a result, parameters that influence distant past inputs receive
very small gradient updates
Because of this:
 The network learns to rely mainly on recent inputs
 Dependencies that span long time intervals are difficult to learn
 Learning long-term structure becomes ineffective

2. Exploding Gradients
Exploding gradients occur when gradients grow exponentially as
they are propagated backward through time.
 This happens when repeated Jacobian multiplications result in very
large values
 Exploding gradients can cause numerical instability
  Parameter updates become excessively large
As a consequence:
 Training becomes unstable
 Optimization may fail completely
 The model parameters can diverge
Although exploding gradients occur less frequently than vanishing
gradients, they can cause severe damage to the optimization
process when they occur.

Impact on Training RNNs

Due to vanishing and exploding gradients:
 Learning long-term dependencies becomes very difficult
 Short-term dependencies dominate learning
 Training deep or long unfolded RNNs is computationally expensive
 Optimization becomes unstable or ineffective
Even when the recurrent network is stable and gradients do not
explode, the relative contribution of long-term dependencies
remains very small compared to short-term ones.

Thus, the primary challenge in learning long-term dependencies in RNNs is

the vanishing and exploding gradient problem, caused by repeated
gradient propagation through many time steps.
Vanishing gradients prevent learning from distant past inputs, while
exploding gradients lead to unstable training, making optimization of
recurrent neural networks difficult.

3. Explain the working principle of the Long Short-Term Memory (LSTM)

network. Discuss the functions of the input, forget, and output gates
along with their respective mathematical equations.

Long Short-Term Memory (LSTM) is a special kind of recurrent

neural network designed to overcome the difficulty of learning
long-term dependencies in standard RNNs.
Standard RNNs suffer from vanishing and exploding gradient
problems, which make it difficult to preserve information over long
time intervals.
LSTM addresses this problem by introducing a memory cell that is
capable of maintaining information for long durations.
The memory cell contains a self-loop with a weight close to one,
which allows gradients to flow unchanged across time steps.
The flow of information into and out of the memory cell is regulated
by gating mechanisms.
 Basic Structure of an LSTM Cell
An LSTM cell consists of:
 A cell state s(t )
 A hidden state h(t)
 Three gates:
1. Input gate
2. Forget gate
3. Output gate
Each gate is implemented using a sigmoid activation function,
which produces values in the range [ 0 , 1 ].
This allows the gates to control information flow smoothly.

Working Principle of LSTM

At every time step t , the LSTM performs the following operations:
 Decides what information to forget
 Decides what new information to store
 Decides what information to output
All decisions are made using the input x (t)and the previous hidden
state h(t−1).

Mathematical Equations of LSTM

Let:
 x (t)→ input at time t
 h(t−1)→ previous hidden state
 s(t−1)→ previous cell state

1. Input Gate
The input gate controls how much new information is written into
the cell state.
i(t)=σ (W i x (t)+U i h(t−1)+b i)
A candidate value for the cell state is computed as:
~s(t )=tanh ⁡(W x (t )+U h(t−1)+ b )
s s s

 The input gate determines which parts of the candidate state

are allowed into memory
 Irrelevant information is filtered out

2. Forget Gate
The forget gate controls how much of the previous cell state
should be retained.
f (t)=σ (W f x (t)+U f h(t−1)+ bf )
 If f (t)≈ 1, information is retained
 If f (t)≈ 0, information is forgotten
This allows the LSTM to reset memory when necessary.

3. Cell State Update

The cell state is updated as:
s(t )=f (t)⊙ s (t−1)+i(t )⊙ ~s(t)
This equation shows:
 Retention of old memory via the forget gate
 Addition of new memory via the input gate
The additive update helps prevent vanishing gradients.

4. Output Gate
The output gate controls how much of the cell state is exposed as
the hidden state.
o (t)=σ (W o x (t )+U o h(t−1)+ bo )
The hidden state is computed as:
h(t)=o (t) ⊙ tanh ⁡(s(t ))
 The hidden state is passed to the next time step
 It is also used to generate outputs

Functions of LSTM Gates

Input Gate
 Controls the addition of new information
 Prevents noise from entering the memory cell
Forget Gate
 Controls removal of outdated or irrelevant information
 Enables adaptive memory updates
Output Gate
 Controls the information revealed to the network
 Regulates interaction with other layers and time steps

Gradient Flow in LSTM

The memory cell’s additive update and self-loop allow gradients
to propagate across many time steps with minimal decay.
This mechanism enables LSTM networks to:
 Preserve long-term information
 Avoid vanishing gradients
 Learn dependencies over long sequences
 

The diagram shows:

o Cell state flow
o Input, forget, and output gates
o Interaction between x (t), h(t−1), and s(t−1)

Thus, the LSTM network extends standard RNNs by introducing a gated

memory cell that can store information for long durations.
The input, forget, and output gates regulate information flow using
sigmoid activations, enabling stable training and effective learning of
long-term dependencies in sequential data.

4. Discuss Bidirectional RNNs and their advantages in capturing

context from both directions of a sequence. How do Deep
Recurrent Networks improve the capacity of sequence
models?

Bidirectional Recurrent Neural Networks (BRNNs)

In many sequence modeling tasks, the output at a particular
time step depends not only on the past inputs but also on
the future inputs.
 Standard recurrent neural networks process the sequence in a
single direction and therefore can use only past context.
Bidirectional Recurrent Neural Networks are designed to
address this limitation.
A bidirectional RNN consists of:
 One RNN that processes the sequence forward in time
 Another RNN that processes the sequence backward in time
Let:
 h(t)denote the hidden state of the forward RNN
 g(t )denote the hidden state of the backward RNN

The output at time step t is computed using both representations, allowing

the model to depend on information from both the past and the future
of the sequence.

Advantages of Bidirectional RNNs

 The output at time t can depend on input values before and after
time t
 Provides richer contextual representations compared to
unidirectional RNNs
 Eliminates the need to specify a fixed-size context window
 Particularly useful when the entire input sequence is available
before prediction

Bidirectional RNNs have been successfully applied to tasks such as:

 Handwriting recognition
 Speech recognition
 Sequence labeling problems
The idea can also be extended to two-dimensional data, such as
images, by using multiple RNNs that process data in different spatial
directions.

Deep Recurrent Neural Networks

In a basic RNN architecture, the computation at each time step consists of

three main transformations:

1. Input to hidden state

2. Previous hidden state to next hidden state
3. Hidden state to output

In simple RNNs, each of these transformations is shallow, meaning it is

represented by a single affine transformation followed by a nonlinearity.

Deep Recurrent Neural Networks improve modeling capacity by

introducing depth into these transformations.

How Deep RNNs Improve Model Capacity

Depth can be introduced by:

 Stacking multiple recurrent layers

 Decomposing the state into multiple levels In deep RNNs:
 Each layer captures representations at a different level of
abstraction
 Lower layers focus on short-term patterns
 Higher layers capture more abstract and long-term structure

Experimental evidence shows that increasing depth significantly improves

the ability of RNNs to model complex sequence transformations.
Benefits of Deep Recurrent Networks

 Increased representational power

 Better modeling of complex temporal dependencies
 Improved performance on challenging sequence processing tasks
 Ability to learn hierarchical temporal features

Bidirectional RNNs enhance sequence modeling by capturing context from

both past and future directions, leading to more informative
representations at each time step.
Deep recurrent neural networks further improve sequence modeling
capacity by introducing depth, enabling the network to learn hierarchical
and complex temporal representations that shallow RNNs cannot model
effectively.

5. . Explain Recursive Neural Networks with a neat diagram. Discuss how

they differ from standard RNNs and their applications in Natural Language
Processing.

Recursive Neural Networks are a class of neural networks that operate on

structured inputs rather than simple sequences.
Unlike standard recurrent neural networks, which process data in a chain-
like temporal order, recursive neural networks process data whose
structure can be represented as a tree.

In recursive neural networks, the computational graph is defined by a

tree structure, and the same set of parameters is applied recursively at
each node of the tree.
Working Principle of Recursive Neural Networks

In a recursive neural network:

 Each node in the tree corresponds to a vector representation

 Leaf nodes represent basic input units

 Internal nodes represent compositions of their child nodes

 The same transformation function is applied at every node

The computation proceeds in a bottom-up manner, starting from the

leaf nodes and moving towards the root node.
At each node, the representations of the child nodes are combined using a
fixed function to produce the representation of the parent node.

This allows the network to map a variable-sized structured input into a

fixed-length vector representation.

 The computational graph of a recursive neural network is shown in

Figure 10.14

Difference Between Recursive Neural Networks and Standard

RNNs

Standard recurrent neural networks process inputs in a sequence, where

the depth of the computational graph grows linearly with the length of the
sequence.

Recursive neural networks differ in the following ways:

 Standard RNNs use a linear chain structure

 Recursive neural networks use a tree-structured computation

  For an input of length τ , the depth of a standard RNN is τ

 In recursive neural networks, the depth of computation can be

reduced to O(log ⁡τ )

This reduction in depth can help in handling long-range dependencies.

Applications in Natural Language Processing

Recursive neural networks are well suited for Natural Language

Processing tasks because language has an inherent hierarchical
structure.

In NLP applications:

 Sentences can be represented using parse trees

 Words correspond to leaf nodes

 Phrases and sentences correspond to internal nodes

 The recursive network composes meanings of words into phrases

and sentences

The tree structure used by the recursive neural network may be:

 Provided by an external parser, or

 Learned automatically by the model

Recursive neural networks are used to learn sentence representations

and to model syntactic and semantic structure in language.

Thus, recursive neural networks extend recurrent neural networks by

operating on tree-structured inputs instead of sequences.
They differ from standard RNNs in their computational structure and are
particularly useful for natural language processing tasks that require
modeling hierarchical relationships.

6. Describe the Encoder-Decoder sequence-to-sequence architecture. How

are LSTMs or GRUs integrated into this framework for machine translation
tasks?

Encoder–Decoder Sequence-to-Sequence Architecture

The encoder–decoder architecture is a neural network framework

designed to map an input sequence of variable length to an output
sequence of variable length.
This architecture is commonly used for sequence-to-sequence learning
tasks such as machine translation.

The model consists of two main components:

1. Encoder

2. Decoder

Encoder

The encoder is a recurrent neural network that processes the input

sequence one element at a time.

 The input sequence is

x (1), x (2), … , x (T )

 At each time step, the encoder updates its hidden state

 After processing the final input, the encoder produces a fixed-

length context vector

This context vector is intended to summarize the entire input sequence

and is passed to the decoder.

Decoder

The decoder is another recurrent neural network that generates the

output sequence.

 The decoder receives the context vector from the encoder

 It produces the output sequence

'
y (1) , y ( 2) , … , y (T )

 Each output symbol is generated one step at a time

 The decoder conditions its predictions on:

o The context vector

o Its previous hidden state

o Previously generated outputs

 The encoder–decoder sequence-to-sequence model is illustrated in

Figure BELOW
Integration of LSTMs / GRUs in Encoder–Decoder Models

Standard RNNs suffer from vanishing and exploding gradient

problems, which make them unsuitable for learning long-term
dependencies required in sequence-to-sequence tasks.

To overcome this, LSTMs or GRUs are used as the recurrent units in both
the encoder and decoder.

LSTM / GRU in the Encoder

 The encoder uses an LSTM or GRU at each time step

 These units maintain an internal state that can store information

over long time intervals

 The final hidden state (and cell state in LSTM) represents the
context vector

This allows the encoder to capture long-range dependencies in the input

sentence.

LSTM / GRU in the Decoder

 The decoder is also implemented using LSTM or GRU units

 The initial state of the decoder is set using the encoder’s final state

 At each time step, the decoder generates one output symbol

  The gating mechanisms allow the decoder to retain relevant
information while producing long output sequences

Encoder–Decoder for Machine Translation

In machine translation:

 The encoder reads the source language sentence

 The decoder generates the target language sentence

 The model learns to represent the meaning of the source sentence

in the context vector

 The decoder then translates this representation into the target

language

Using LSTMs or GRUs enables the model to:

 Handle long sentences

 Preserve semantic information

 Reduce gradient-related training problems

Thus, the encoder–decoder sequence-to-sequence architecture maps

variable-length input sequences to variable-length output sequences
using two recurrent networks.
By integrating LSTMs or GRUs into both the encoder and decoder, the
model can effectively learn long-term dependencies, making it suitable for
machine translation and other sequence-to-sequence tasks.

7. Recurrent Neural Network (RNN) as a Directed Graphical Model

A Recurrent Neural Network (RNN) can be interpreted as a directed

graphical model that defines a joint probability distribution over a
sequence of random variables.

In this view, the RNN represents dependencies between:

 Input sequence

 Hidden states
  Output sequence

across multiple time steps using directed edges.

Directed Graphical Model Representation

Consider an input sequence:

x (1), x (2), … , x (T )

and an output sequence:

y (1), y ( 2) , … , y (T )

In an RNN:

 Each output y (t )depends on the current hidden state h(t)

 Each hidden state h(t)depends on:

o The previous hidden state h(t−1)

o The current input x (t)

This defines a directed acyclic graph when unfolded over time,

where arrows represent conditional dependencies.

Factorization of the Joint Distribution

The directed graphical model defined by an RNN factorizes the joint

distribution over outputs as:
T
P( y (1), … , y (T )∣ x (1), … , x(T ))=∏ P( y (t)∣ h(t ))
t=1

where the hidden states evolve according to:

h(t)=f (h(t−1), x (t))

Thus, the hidden state acts as a summary of all past inputs, making
the model suitable for sequential data.

Unfolding the Graph Over Time

Although the recurrent definition contains cycles, the RNN can be

unfolded over time into a feedforward directed graphical model.

After unfolding:
  Each time step has its own node

 Parameters are shared across time steps

 The graph becomes suitable for gradient-based learning

This unfolded directed graphical model explicitly shows how information

flows from past to future.

 The RNN represented as a directed graphical model and its

unfolding over time is illustrated in
Figure 10.3

Significance of the Directed Graphical Model View

 Makes the conditional dependencies in RNNs explicit

 Provides a probabilistic interpretation of sequence modeling

 Clarifies how hidden states summarize past information

 Helps in understanding learning using back-propagation through

time

Thus, a Recurrent Neural Network can be viewed as a directed graphical

model in which each output depends on a hidden state that summarizes
previous inputs.
By unfolding the recurrence, the RNN defines a directed acyclic graphical
model with shared parameters across time, enabling efficient learning and
sequence modeling.
*