21/10/2025, 02:34 Convolution operation notes

UNIT IV: Recurrent and Recursive Nets, covering all the sub‑topics you listed: Unfolding
Computational Graphs, Recurrent Neural Networks, Bidirectional RNNs, Encoder‑Decoder
Sequence‑to‑Sequence Architectures, Deep Recurrent Networks, Recursive Neural Networks, The
Challenge of Long‑Term Dependencies, Echo State Networks, Leaky Units and Other Strategies for
Multiple Time Scales, The Long Short‑Term Memory and Other Gated RNNs, Optimization for Long‑Term
Dependencies, and Explicit Memory. Many of these draw on Chapter 10 of Deep Learning (Goodfellow
et al.). Deep Learning B… +2

1. Unfolding Computational Graphs

1.1 Motivation & basics

A computational graph is a directed graph that shows how inputs, parameters and computations
combine to produce outputs/losses.
In recurrent/recursive models we have cycles (feedback loops) in the graph: e.g., hidden state at
time t depends on hidden state at t − 1.
To analyze and compute gradients in such cyclic graphs, we “unfold” the graph across time (or
across tree structure) into an acyclic graph showing the repeated application of the same function
at each time step (or node).
For example: Hidden state h(t) = f (h(t−1) , x(t) ; θ). If we unfold from t = 1 to T , we see
h(1) → h(2) → ⋯ → h(T ) . Deep Learning B… +1
The key benefit: we explicitly see the parameter sharing across time (same θ used in each step)
and clearly depict the path for forward‑propagation and back‑propagation through time (BPTT).
Deep Learning B…

1.2 Formal representation

At each time step t:

h(t) = f (h(t−1) , x(t) ; θ), y (t) = g(h(t) ; θy ) ​

The full unfolding from t = 1 to T gives a graph with T copies of the cell function f , each
connected serially by the hidden state.
Parameter sharing: All copies use the same θ .
When you compute gradients with respect to θ , you sum contributions from each time‐step.
This view helps to analyze issues like vanishing/exploding gradients (see later sections).

1.3 Design patterns

Depending on the application, one may use different “configurations” of RNN input/output across
time steps:
Many‑to‑one: e.g., sentiment classification of a sentence → one output at the end.
One‑to‑many: e.g., sequence generation from a single vector.

[Link] 1/11
21/10/2025, 02:34 Convolution operation notes

Many‑to‑many: e.g., machine translation, where an input sequence maps to an output

sequence (possibly the same length or different).
Unfolded graphs help visualise how information flows through time, how the hidden state acts as
memory, and how gradients propagate backwards.

2. Recurrent Neural Networks (RNNs)

2.1 Definition & architecture

A recurrent neural network is one where connections form directed cycles, allowing information to
persist. In practice, this often means the hidden state at time t depends on the hidden state at t −
1. Deep Learning B… +1
A simple vanilla RNN update:

h(t) = tanh(Whh h(t−1) + Wxh x(t) + bh ),

​ ​ ​
y (t) = Why h(t) + by
​ ​

or sometimes softmax for classification output.

Here Whh , Wxh , Why are learned weights, shared across timesteps.
​ ​ ​

The hidden state acts like a dynamic memory of past inputs.

2.2 Why use RNNs?

Many tasks involve sequential data: language, time‑series, video frames, etc.
Standard feedforward networks cannot naturally handle variable‐length sequences nor maintain a
memory of past inputs.
RNNs provide a parameter efficient way to handle sequences of arbitrary length (through
parameter sharing) and capture temporal dependencies. Chinmay Hegde

2.3 Training – Back‑Propagation Through Time (BPTT)

Because of the unfolding, training involves back‑propagating error through many time steps
(often truncated for long sequences).
Gradients can accumulate over long paths (many time‐steps) leading
to vanishing or exploding gradients (more in section 7).
Practical implementations often use truncated BPTT (limit back‐prop steps) or gradient clipping.

2.4 Use cases & variants

RNNs used for language modelling, sequence classification, sequence generation.
Variants: Elman network, Jordan network, etc (historical).
The basic RNN serves as foundational model; however, more advanced versions handle long‐term
dependencies better (see later).

[Link] 2/11
21/10/2025, 02:34 Convolution operation notes

3. Bidirectional RNNs

3.1 Motivation
In some sequence tasks (e.g., handwriting recognition, speech recognition, NLP tagging) the
output at time t depends not only on past inputs but also on future inputs.
A standard (unidirectional) RNN processes the sequence only in the forward time direction, and
thus cannot incorporate future context at time t. Deep Learning B… +1

3.2 Architecture
A bidirectional RNN (Bi‐RNN) consists of two RNNs:
One processes the sequence from t = 1 to T (forward direction).
The other processes from t = T to 1 (backward direction).
At each time step t, the hidden states of both forward and backward RNNs are concatenated (or
combined) and used to predict output y (t) . Deep Learning B… +1
So:

h (t) = RN Nforward (x(t) , h (t−1) )

​

h (t) = RN Nbackward (x(t) , h (t+1) )

​

h(t) = [ h (t) , h (t) ]

y (t) = g(h(t) )

3.3 Benefits & limitations

Benefits: It allows each output to use full context (past & future) which often improves
performance on tasks like sequence labeling, speech, etc.
Limitations: Cannot be used in streaming/online prediction (future unavailable), may double
computation (two RNNs), and still suffers from long‑term dependency issues.
Use case: When entire input sequence is available before processing.

4. Encoder‑Decoder Sequence‑to‑Sequence Architectures

4.1 Problem statement

Many tasks require mapping an input sequence of length nx to an output sequence of length ny
​ ​

where nx and ny may differ (e.g., machine translation, summarisation).

​ ​

A plain RNN can map sequence → sequence of same length, but handling variable lengths and
differing lengths requires special architecture.

4.2 Architecture & working

[Link] 3/11
21/10/2025, 02:34 Convolution operation notes

The encoder–decoder (or seq2seq) architecture uses two RNNs:

Encoder: processes input sequence x(1) , … , x(nx ) and returns a context vector C (often the
​

final hidden state).

Decoder: takes that context vector C as initial hidden state (or input) and generates output
sequence y (1) , … , y (ny ) . Deep Learning B… +1
​

For example:

Encoder: h(t) ​
(t−1) (t)
e = f (he , x ), ​ C = h(n
e
x)
​

(u) (u−1) (u)

Decoder: hd = f (hd ​ , y (u−1) , C),
​ y (u) = g(hd ) ​

4.3 Limitations and enhancements

A key limitation: the fixed‑size context vector C must encode all information of input sequence;
when input length is large, this bottleneck may degrade performance. NCBI +1
Solutions:
Attention mechanism: Decoder can attend to all encoder hidden states rather than just a
single vector.
Variable length contexts.
The seq2seq architecture forms the basis for many modern NLP models (translation,
summarization, dialog).

4.4 Use cases

Machine translation (input sentence → translated sentence)
Text summarisation (input passage → short summary)
Speech recognition, image captioning (image encoded → sequence output)
Question‐answering, etc.

5. Deep Recurrent Networks

5.1 Why “deep”?

“Deep” here can mean two things:
1. Deep in time: many time steps (long sequences).
2. Deep in space: multiple layers per time step (stacked RNNs) or more complex transformations
of input/hidden/output. Deep Learning B… +1
Stacking or increasing depth can allow modeling more complex temporal patterns.

5.2 Architectural choices

Stacked RNNs: Multiple RNN layers where at each time‑step output of one RNN layer becomes
input to next.

[Link] 4/11
21/10/2025, 02:34 Convolution operation notes

Deep transition: Instead of a simple hidden‑to‑hidden update, use a small MLP inside each time
step (making each time step more expressive). Master Wang’s s…
Skip connections: To ease gradient flow, skip‐connections across time or layers may be
introduced (similar to residual networks). Deep Learning B… +1

5.3 Benefits & challenges

Benefits: More expressive models, can capture hierarchical temporal structure, learn complex
patterns.
Challenges: Deeper networks may be harder to train (vanishing/exploding gradients become
worse), increased computational cost, more data needed.

6. Recursive Neural Networks

6.1 Definition & motivation

A recursive neural network (not to be confused with “recurrent”) processes structured inputs
(trees) instead of simple sequences. Deep Learning B… +1
The computational graph forms a tree (or directed acyclic graph) rather than a simple chain.
Frequently used in NLP (parsing trees, semantic composition), and structured data domains.
Example: For input tree nodes x(1) , x(2) , x(3) , x(4) built into a binary tree, the compositional
structure may compute representations bottom‑up.

6.2 Architecture
Suppose two child nodes with vectors hL and hR . Then a parent node hidden representation:
​ ​

hP = f (U hL + V hR + b)
​ ​ ​

with shared matrices U , V across nodes.

Leaves correspond to input vectors; internal nodes compute via same parameters.
After building up through the tree, one may map the root representation to output
(classification/regression).

6.3 Advantages & limitations

Advantage: If the tree depth is O(log τ ) for a sequence of length τ , then the number of nonlinear
compositions is smaller, potentially helping with long‐range dependencies. [Link]…
Limitations: One must define or infer tree structure; may be harder to apply when data is naturally
sequential rather than tree‐structured; training/implementation may be more complex.

7. The Challenge of Long‑Term Dependencies

[Link] 5/11
21/10/2025, 02:34 Convolution operation notes

7.1 What’s the problem?

Vanilla RNNs struggle to learn dependencies in sequences when the relevant signal is many time‐
steps away (i.e., long‐term).
Mathematically, when you back–propagate gradient through many repeated multiplications (time
steps), the gradient may vanish (go to zero) or explode (grow very large). [Link]… +1
Example (simplified): Let h(t) = W h(t−1) . Then

h(t) = W t h(0) = QΛt Q−1 h(0)

where W = QΛQ−1 . If ∣λi ∣ < 1, then λti → 0 (vanishing). If ∣λi ∣ > 1, it diverges (exploding).
​ ​ ​

[Link]…

7.2 Consequences
Vanishing gradients impede learning of weights that influence far‐past time‐steps because the
gradient becomes negligibly small.
Exploding gradients cause numerical instability, poor convergence.
Practically: RNNs may only learn short‐term dependencies (adjacent/few steps) and fail at long‐
range correlations (e.g., subject–verb agreement across many words, long context relations).
This challenge motivates the use of architectures like LSTM, gated RNNs, skip connections, leaky
units, etc.

8. Echo State Networks

8.1 Idea & origin

Echo State Networks (ESNs) are an instance of reservoir computing where the recurrent
“reservoir” has fixed (random) weights and only the output/read‑out weights are trained.
Wikipedia +1

The reservoir is designed to be echo‐state: the network dynamics must fade out the effect of
inputs after some time (i.e., the system is stable).
The idea: leverage the dynamics of a large, fixed recurrent network to project input sequence into
a high‐dimensional “rich” state space, then learning simpler linear (or shallow) read‐out.

8.2 Architecture & operation

Input → reservoir (large recurrent network with fixed random weights) → hidden states over time.
Only weights from reservoir to output are trained (usually by linear regression).
Because the internal weights are fixed, training is efficient.
Key hyper‑parameters: spectral radius of recurrent weight matrix (should < 1 typically), input
scaling, reservoir size, connectivity.

8.3 Strengths & limitations

[Link] 6/11
21/10/2025, 02:34 Convolution operation notes

Strengths: Simple training (only read‐out), can capture complex temporal dynamics, efficient for
certain tasks.
Limitations: Less flexible (fixed internal weights), reservoir may not be well matched to task, less
prevalent today in deep learning compared to gated RNNs and attention models.

9. Leaky Units and Other Strategies for Multiple Time Scales

9.1 Motivation
Some sequences involve dependencies at different time‐scales: some signals change quickly,
others very slowly.
Standard RNNs (or even LSTMs) may struggle to simultaneously model both fast and slow
changing patterns.
Strategies: Introduce architectural components that allow multiple time‐scales or slower memory
traces (e.g., leaky units, skip connections in time, hierarchical RNNs).

9.2 Leaky units

A leaky unit introduces a “leak” coefficient controlling how much of the previous hidden state
decays:

u(t) = α u(t−1) + (1 − α) v (t) , 0≤α<1

where v (t) is the new computed state update.

If α is large (close to 1), the hidden state changes slowly → captures long‐term dynamics; if α
small, the unit responds quickly → captures short‐term dynamics. [Link]…

9.3 Skip‑connections in time

Introduce connections from h(t−k) → h(t) for k > 1, effectively allowing the network to jump over
intermediate time steps and carry information more directly across time.
This helps reduce the shortest path length for gradients and information from distant time steps,
mitigating vanishing gradient to some degree.

9.4 Other strategies

Hierarchical/clock‐work RNNs: partition hidden units into groups operating at different time
scales, with slower groups updating less frequently and feeding faster groups.
Combining multiple RNNs with different update rates.
Using gating mechanisms (like in LSTM/GRU) to regulate information flow and timescales.

10. The Long Short‑Term Memory (LSTM) and Other Gated RNNs

10.1 Motivation
[Link] 7/11
21/10/2025, 02:34 Convolution operation notes

To address vanishing/exploding gradients and enable learning of long‐term dependencies, gated

RNNs were proposed.
The most famous is the Long Short‑Term Memory (LSTM) architecture.
Key idea: provide longer paths for gradient flow (constant error carousel) via memory cell, plus
gates to control information flow.

10.2 LSTM architecture

Components: cell state c(t) , hidden state h(t) ; gates: input i(t) , forget f (t) , output o(t) .
Equations (simplified):

f (t) = σ(Wf x(t) + Uf h(t−1) + bf )

​ ​ ​

i(t) = σ(Wi x(t) + Ui h(t−1) + bi )

​ ​ ​

c~(t) = tanh(Wc x(t) + Uc h(t−1) + bc )

​ ​ ​

c(t) = f (t) ⊙ c(t−1) + i(t) ⊙ c~(t)

o(t) = σ(Wo x(t) + Uo h(t−1) + bo )

​ ​ ​

h(t) = o(t) ⊙ tanh(c(t) )

Where ⊙ is element‐wise multiplication, σ is logistic sigmoid.
These gates regulate what to forget, what to input, what to output — enabling the network to
keep or discard information as needed. (See gating mechanism descriptions) Wikipedia

10.3 Other gated variants

Gated Recurrent Unit (GRU): A simpler gated RNN with fewer gates (update and reset gates) than
LSTM; often comparable performance with fewer parameters. Wikipedia
Variants: Peephole LSTM, Coupled LSTM, Clockwork RNN, etc.
For tree/recursive structures: gated tree‐LSTM, etc.

10.4 Why they help long‑term dependencies

The “cell state” provides a path through time with (ideally) derivative ~1 (when gates allow) → less
vanishing/ exploding.
Gates allow network to decide when to remember, when to forget → flexible memory timescales.
Empirically successful: LSTMs/GRUs dominate many sequence tasks.

11. Optimization for Long‑Term Dependencies

11.1 Gradient clipping

If gradients explode (too large), clip the norm to a fixed threshold (e.g., 5). This prevents numerical
instability and large parameter updates.
[Link] 8/11
21/10/2025, 02:34 Convolution operation notes

Often used when training RNNs/LSTMs.

11.2 Appropriate parameter initialization & regularisation

Initialising recurrent weights (e.g., orthogonal initialisation) helps to stabilise signal propagation
over many steps.
Regularisation: dropout (carefully applied in RNNs), recurrent dropout, layer normalisation,
zoneout.
Gradient‐norm regularization, weight‐decay can also help.

11.3 Truncated BPTT

For very long sequences, one might only back‐propagate through the last k time steps (truncated
BPTT) to reduce computation/memory.
This has trade‐offs: shorter context length vs efficiency.

11.4 Skip/shortcut connections

Introducing skip‐connections (residual in time or layers) reduces the “shortest path” from input to
output/gradient path, helping gradient flow and reducing depth. Master Wang’s s…

11.5 Leaky units / multiple timescales strategies (see previous section)

These architectural modifications help the network to maintain useful signals over longer spans.

11.6 Curriculum learning, better optimisation algorithms

Use of advanced optimisers (Adam, RMSProp) with appropriate learning‑rate scheduling helps.
Sometimes training shorter sequences first, then longer sequences (curriculum) can ease learning.
Monitoring gradient statistics, use of gradient noise, etc.

12. Explicit Memory

12.1 Motivation
Even with LSTMs/GRUs, modeling extremely long‐term dependencies (hundreds to thousands of
steps) remains challenging.
Explicit memory architectures augment recurrent networks with an external memory (read/write)
to store past information more directly and flexibly.
Examples: Differentiable Neural Computer (DNC), Memory Networks, Neural Turing Machines.
Wikipedia

12.2 Architecture & ideas

The network has a controller (often RNN) and an external memory matrix. The controller can:
write to the memory (decide what to store)
read from the memory (retrieve prior information)

[Link] 9/11
21/10/2025, 02:34 Convolution operation notes

The read/write operations are differentiable (hence end‐to‐end training).

It decouples memory capacity from the hidden‐state dimension of the RNN and can provide
better long‐term recall and relational reasoning.

12.3 Use‑cases & limits

Suitable for tasks requiring algorithmic reasoning, long‐range coherence, storing many entities,
graphs, etc.
However, they are more complex, harder to train, require more data/computation. In many
practical sequence tasks simpler RNNs/LSTMs suffice.

Summary Table
Topic Key Points

Unfolding Computational Visualise RNNs as deep networks through time; parameter sharing;
Graphs gradient paths.

Recurrent Neural Networks Hidden‐state memory, parameter sharing across time, suited to
sequence data.

Bidirectional RNNs Use past + future context; two RNNs (forward/backward); for tasks
where full sequence available.

Encoder‑Decoder (Seq2Seq) Map input sequence → output sequence of possibly different

lengths; uses encoder then decoder RNN.

Deep Recurrent Networks Increase depth (stacked, input‐hidden‐output deeper, skip

connections) for expressivity.

Recursive Neural Networks Tree‐structured networks; process structured data (e.g., parse trees)
rather than simple chains.

Long‑Term Dependencies Vanilla RNNs struggle due to vanishing/exploding gradients; long‐

range signals decay/ blow up.

Echo State Networks Fixed reservoir + trained read‐out; efficient but less flexible than full
learnable RNNs.

Leaky Units & Multi‐ Architectural strategies (leak, skip, hierarchical time) to model
Timescales multiple time‐scales and help long‐term memory.

[Link] 10/11
21/10/2025, 02:34 Convolution operation notes

Topic Key Points

LSTM & Gated RNNs Introduce memory cell + gates to regulate information flow; major
improvement for long‐term dependencies.

Optimization Gradient clipping, orthogonal init, skip connections, truncated

BPTT, advanced optimisers.

Explicit Memory RNNs augmented with external memory (memory networks, DNC)
for even longer‐term dependencies and relational tasks.

[Link] 11/11