Unit IV

Recurrent and Recursive Networks

Unfolding Computational Graphs

What It Means

When we train neural networks that handle sequential data (like sentences, audio, or time series),
the same network is used repeatedly over time steps.

But — to understand how gradients flow and how backpropagation works, we “unfold” or “unroll”
this network through time.

So, Unfolding means representing a Recurrent Neural Network (RNN) as a regular feed-
forward network stretched across multiple time steps.

Intuition

Imagine you have an RNN that processes one input at each time step:

• Input at time ( t ): ( x_t )

• Hidden state at time ( t ): ( h_t )

• Output at time ( t ): ( y_t )

Each time, the RNN does:

[
h_t = f(W_h h_{t-1} + W_x x_t + b)
]
[
y_t = g(W_y h_t + c)
]

Now, instead of thinking of this as a loop, we unfold it across time steps:

x1 → [RNN cell] → h1 → y1
↓
x2 → [RNN cell] → h2 → y2
↓
x3 → [RNN cell] → h3 → y3
Each [RNN cell] has the same weights (W_h, W_x, W_y), but acts on different inputs ( x_t )
and hidden states ( h_{t-1} ).

Why Unfolding is Important

 • It allows us to apply Backpropagation Through Time (BPTT) — a special form of
backpropagation for sequences.

• It shows how information flows through time.

• It helps visualize long-term dependencies (how early inputs influence later outputs).

Mathematical View

For a sequence of length ( T ):

[
L = \sum_{t=1}^{T} L_t
]
where ( L_t ) is the loss at time step ( t ).

During training, the gradient of the total loss w.r.t. weights is computed by summing over
time: [
\frac{\partial L}{\partial \theta} = \sum_{t=1}^{T} \frac{\partial L_t}{\partial \theta}
]

Since the same weights are reused at every time step, unfolding makes it possible to compute these
gradients correctly.

Visualization Summary

Unfolded RNN:

t=1 t=2 t=3

x1 → h1 → y1
↓
x2 → h2 → y2
↓
x3 → h3 → y3
Each vertical connection passes hidden state information forward in time.

In Short (for Quick Revision)

• Unfolding = expanding the loop of an RNN across time steps.

• Shows how same weights are reused over time.

• Necessary for training (Backpropagation Through Time).

• Helps analyze temporal dependencies in sequences.

 Recurrent Neural Networks (RNNs)

1. What is an RNN?
A Recurrent Neural Network (RNN) is a type of neural network that’s designed to process sequential
data — data where order matters (like text, speech, time series, or video frames).

Unlike traditional feedforward networks that treat each input independently, RNNs remember
past information using a hidden state that’s carried from one time step to the next.

2. Basic Idea (The Memory Concept)

RNNs have a kind of “short-term memory.”
At every time step ( t ):

• They take the current input ( x_t )

• Combine it with the previous hidden state ( h_{t-1} )

• Produce a new hidden state ( h_t )

• Then optionally output a value ( y_t )

3. Mathematical Representation
Let’s write the core RNN equations:

h_t = f(W_h h_{t-1} + W_x x_t + b_h) y_t =

g(W_y h_t + b_y)

Where:

• ( x_t ): Input at time step t

• ( h_t ): Hidden state (memory) at time t

• ( W_x, W_h, W_y ): Weight matrices (shared across all time steps)

• ( b_h, b_y ): Bias terms

• ( f, g ): Activation functions (usually tanh, ReLU, or softmax)

4. Structure of RNN
Here’s the flow:
 ┌──────────────┐
x₁ ─────▶│ RNN Cell │──▶ y₁
└──────┬───────┘
│ (hidden state)
▼
┌──────────────┐
x₂ ─────▶│ RNN Cell │──▶ y₂
└──────┬───────┘
│
▼
┌──────────────┐
x₃ ─────▶│ RNN Cell │──▶ y₃
└──────────────┘
Every cell receives info from the previous cell — that’s how “recurrent” behavior is achieved.

5. Types of RNN Outputs

Depending on the application, RNNs can have different input/output structures:

Type Example Input-Output Structure

Image
One-to-One One input → One output
classification
One-to-Many Image captioning One input → Sequence output
Many-to-One Sentiment analysis Sequence input → One output
Many-to- Machine Sequence input → Sequence
Many translation output

6. Activation Function
Most RNNs use tanh or ReLU for hidden states:

[
h_t = \tanh(W_h h_{t-1} + W_x x_t + b_h)
]

Reason:
tanh squashes values between -1 and 1 → prevents exploding outputs, but still allows gradients to
flow.

7. Advantages of RNNs
Handles sequential or temporal data naturally.
Shares weights → fewer parameters.
Maintains internal “memory” → context-aware predictions.
Good for NLP, speech, and time-series problems.

8. Limitations
Hard to train on long sequences (gradients vanish or explode).
Can forget early information when sequences are long.
Slow to train (depends on sequence length).
Can’t handle dependencies too far apart effectively.

9. Real-World Use Cases

Application Example
Language
Predicting next word in a sentence
Modeling
Speech
Recognition Converting audio → text
Determining if a review is positive/
Sentiment Analysis
negative
Stock Forecasting Predicting next day’s price
Video Captioning Generating text from frames

10. Quick Revision Summary

Concept Description
Core Idea Network with memory that processes sequences
Hidden
Stores info from previous time steps
State
Weights Shared across all time steps
Done using Backpropagation Through Time
Training
(BPTT)
Problem Struggles with long-term dependencies

Bidirectional Recurrent Neural Networks

(BiRNNs)

1. Introduction
A Bidirectional Recurrent Neural Network (BiRNN) is an advanced version of the standard RNN that
processes data in both directions — forward and backward.
This means it looks at past (previous time steps) and future (upcoming time steps) information when
making predictions.

This dual processing helps the network gain context from the entire sequence, which is especially
useful in applications like speech recognition, language translation, and text analysis, where the
meaning of a word often depends on both its previous and next words.

2. Motivation
A traditional (unidirectional) RNN processes the sequence in one direction only, usually from left to
right:

At each step, it only knows the past context but not the future.

However, in many real-world tasks (like understanding a sentence), the context after a word also
matters.
For example, to understand the word “bank,” knowing the next word — “river” or “account” — changes
the meaning.

So, Bidirectional RNNs solve this by processing the sequence in both directions.

3. Architecture
A BiRNN consists of two RNNs:

1. Forward RNN – processes the sequence from ( t = 1 ) to ( T )

2. Backward RNN – processes the sequence from ( t = T ) to ( 1

) At each time step ( t ), both RNNs produce hidden states:

[
\overrightarrow{h_t} = f(W_x^{(f)} x_t + W_h^{(f)} \overrightarrow{h_{t-1}} + b^{(f)})
]
[
\overleftarrow{h_t} = f(W_x^{(b)} x_t + W_h^{(b)} \overleftarrow{h_{t+1}} + b^{(b)})
]

Then the final hidden state for time ( t ) is obtained by combining both directions: [
h_t = [\overrightarrow{h_t}; \overleftarrow{h_t}]
]

(where [ ; ] means concatenation)

4. Output Calculation
Each combined hidden state ( h_t ) is passed to the output layer:

[
y_t = g(W_y h_t + b_y)
]

This means that the prediction at time ( t ) depends on both past and future inputs.

5. Diagram Representation
Forward RNN: x1 → x2 → x3 → x4
Backward RNN: x1 ← x2 ← x3 ← x4

At each time step:

h_t = [Forward hidden; Backward hidden]
So, each output node receives information from both directions.

6. Advantages
• Captures context from both past and future.

• Improves accuracy for sequence labeling and context-sensitive tasks.

• More powerful than a standard RNN for understanding the full sequence.

7. Disadvantages
• More computation and memory usage (two RNNs instead of one).

• Not suitable for real-time prediction, since it needs future data (you can’t know future
steps while streaming live input).

• Training time doubles, as two RNNs are trained simultaneously.

8. Applications
Application Example
Speech Recognition Phoneme classification where full sentence context improves accuracy
Text Classification Sentiment analysis using both previous and following words
 Encoder-decoder models where context from the whole sentence is
Machine Translation
crucial
Named Entity
Identifying names, dates, or locations in text sequences
Recognition

9. Comparison: RNN vs BiRNN

Feature RNN BiRNN
Direction One (forward) Two (forward + backward)
Context Past only Past and Future
Parameter Double (one for each
Single set
s direction)
Real-time
Use Case Context-rich analysis
prediction
Accuracy Moderate High for sequence tasks

10. Key Takeaways

• BiRNN = Two RNNs → one forward, one backward.

• Useful when entire input sequence is available.

• Improves understanding of contextual dependencies.

• Commonly used in NLP, speech, and translation tasks.

Encoder–Decoder Sequence-to-Sequence
Architectures

1. Introduction
Traditional RNNs struggle when input and output sequences have different lengths — for example:

• Translating English → French (input and output lengths differ)

• Speech-to-text

• Text summarization

To solve this, the Encoder–Decoder architecture, also called Seq2Seq (Sequence-to-Sequence),

was introduced.
It allows the model to convert a sequence of variable length inputs into a sequence of variable
length outputs.
2. Basic Concept
The Encoder–Decoder model is made up of two RNNs (or their variants like LSTM/GRU):

1. Encoder: Reads the entire input sequence and compresses it into a fixed-length
context vector (representation of the whole sequence).

2. Decoder: Takes this context vector and generates the output sequence one element at
a time.

3. Architecture Flow
Step-by-Step Process:

1. Input sequence:
( x_1, x_2, x_3, ..., x_T )

2. Encoder:
Processes each input sequentially to produce hidden states:
[
h_t = f(W_x x_t + W_h h_{t-1})
]
The final hidden state ( h_T ) acts as the context vector (C), representing the entire input
sequence.

3. Decoder:
Takes the context vector ( C ) as input and predicts output
sequence: [
s_t = f(W_y y_{t-1} + W_s s_{t-1} + W_c C)
]
[
y_t = g(W_o s_t)
]
where:

◦ ( s_t ): decoder hidden state at time t

◦ ( y_t ): predicted output at time t

◦ ( C ): context vector from encoder

4. Diagram Representation
Encoder: x1 → x2 → x3 → ... → xT
↓
Context Vector (C)
↓
Decoder: y1 → y2 → y3 → ... → yT'
• The encoder compresses the input sequence into a single vector.

• The decoder unfolds it back into an output sequence.

5. Mathematical View
Encoder Phase:
[
h_t = f(W_h h_{t-1} + W_x x_t + b_h)
]
Final context:
[
C = h_T
]

Decoder Phase:
[
s_t = f(W_s s_{t-1} + W_y y_{t-1} + W_c C)
]
[
y_t = g(W_o s_t + b_o)
]

The decoder predicts one token at a time using its previous output as the next input (known as
teacher forcing during training).

6. Key Challenge: Information Bottleneck

Because the encoder compresses the entire sequence into one fixed-size vector ( C ), important
details can be lost — especially for long sentences.
This is known as the information bottleneck problem.

To solve it, attention mechanisms were later introduced (covered in advanced architectures like
LSTM with Attention and Transformers).

7. Advantages
• Handles variable-length input and output sequences.

• Captures temporal dependencies between elements in the sequence.

• Provides a foundation for advanced architectures (like attention, Transformers, etc.).

8. Disadvantages
 • The context vector may lose information for long sequences.

• Training is slower (needs two RNNs).

• Struggles with long-term dependencies if not combined with LSTM/GRU.

9. Applications
Application Example
Machine
Translation English → French translation
Text Summarization
Converting long text into short summary
Image (encoded by CNN) → Caption (decoded by RNN)
Image Captioning

Speech Recognition
Audio (input sequence) → Text (output sequence)
Chatbots Input message → Response generation

10. Quick Summary

Component Description
Converts input sequence into a fixed-length context vector
Encoder

Decoder Generates output sequence using context vector

Architecture
Many-to-Many (input and output both sequences)
Type
Problem SolvedDifferent input-output lengths
Limitation Loses detail for long sequences (solved later by attention)

Deep Recurrent Networks (Deep RNNs)

1. Introduction
A Deep Recurrent Neural Network (Deep RNN) is an extension of a basic RNN where multiple
recurrent layers are stacked on top of each other.
This deeper architecture allows the model to learn hierarchical temporal features — lower layers
capture short-term patterns, while higher layers capture long-term dependencies or more abstract
representations.

In simple terms:
 • A shallow RNN → one layer of recurrence

• A deep RNN → multiple RNN layers stacked vertically

2. Why Go Deeper?
A standard RNN can capture only limited temporal relationships due to its single hidden layer.
However, many real-world sequences (like speech or language) contain patterns at multiple time
scales — short-term (words) and long-term (sentences or context).

Stacking multiple RNN layers increases representational power, helping the network:

• Understand complex temporal dependencies

• Build multi-level abstractions

• Improve prediction accuracy

3. Architecture Overview
In a Deep RNN, each layer’s hidden state output becomes the input for the next layer.

For a 3-layer Deep RNN:

At time step ( t ):

[
h_t^{(1)} = f(W_x^{(1)} x_t + W_h^{(1)} h_{t-1}^{(1)} + b^{(1)})
]
[
h_t^{(2)} = f(W_x^{(2)} h_t^{(1)} + W_h^{(2)} h_{t-1}^{(2)} + b^{(2)})
]
[
h_t^{(3)} = f(W_x^{(3)} h_t^{(2)} + W_h^{(3)} h_{t-1}^{(3)} + b^{(3)})
]
[
y_t = g(W_y h_t^{(3)} + b_y)
]

Each layer processes the sequence at a different level of abstraction.

4. Diagram Representation
Layer 3: h1(3) → h2(3) → h3(3)
↑ ↑ ↑
Layer 2: h1(2) → h2(2) → h3(2)
↑ ↑ ↑
Layer 1: h1(1) → h2(1) → h3(1)
↑ ↑ ↑
Input: x1 x2 x3
Here:

• Each horizontal arrow = temporal connection (across time)

• Each vertical arrow = layer connection (across depth)

5. Training Deep RNNs

Deep RNNs are trained using Backpropagation Through Time (BPTT), but now the gradient
flows through:

• Time steps (horizontally)

• Layers (vertically)

This increases model power but also leads to training challenges like:

• Vanishing/exploding gradients

• Slow convergence

To counter these issues, LSTM and GRU layers are often used instead of vanilla RNN cells.

6. Advantages
Advantage Explanation
Better temporal
Learns dependencies at multiple time scales
understanding
Higher representational
Each layer extracts more complex features
power
Works better for complex tasks like translation and speech
Improved performance
recognition
Reusable features Lower layers can capture general temporal structures

7. Disadvantages
Disadvantage Explanation
More parameters and deeper gradient
Harder to train
flow
Prone to vanishing/exploding
Especially with many layers
gradients
Higher computation cost Needs more memory and training time
Requires large datasets To avoid overfitting

8. Applications
Field Example
Multiple recurrent layers extract phonetic → word → sentence-level
Speech Recognition
patterns
Language Modeling Deeper models capture sentence-level semantics
Time-Series
Handles complex multivariate time dependencies
Forecasting
Video Processing Frame-level + motion-level temporal understanding

9. Key Takeaways
• Deep RNNs = multiple RNN layers stacked vertically.

• Each layer refines the temporal representation from the previous one.

• Improves learning of long-term, hierarchical patterns.

• Needs careful training (often with LSTM/GRU and gradient clipping).

10. Quick Summary

Concept Deep RNN
Structure Multiple RNN layers stacked vertically
Capture temporal dependencies at multiple abstraction
Purpose
levels
Training Through Backpropagation Through Time
Common
Gradient vanishing/explosion
Issues
Solution Use gated units (LSTM, GRU) and gradient clipping

Recursive Neural Networks (RecNNs)

1. Introduction
A Recursive Neural Network (RecNN) is a type of neural network that works on hierarchical
structures such as trees, rather than flat sequences.
While an RNN processes data sequentially (left to right or time step by time step), a RecNN
processes data recursively, combining information from smaller parts into larger structures.

It’s particularly useful for structured data like:

• Parse trees in Natural Language Processing (NLP)

• Scene hierarchies in computer vision

• Mathematical expression trees or syntax trees

2. Key Difference Between RNN and RecNN

Feature RNN Recursive NN
Data Type Sequential (ordered data) Hierarchical (tree-structured data)
Sentence as a sequence of
Example Sentence as a parse tree of phrases
words
Flow of
Left to right in time Bottom-up (from leaves to root)
Computation
Syntax parsing, sentence meaning, scene
Typical Use Speech, sequential data
understanding

3. Architecture Concept
A Recursive Neural Network builds a tree where each parent node is computed as a function of its
child nodes.

If two child nodes have representations ( h_1 ) and ( h_2 ), then their parent node representation
( p ) is computed as:

[
p = f(W [h_1; h_2] + b)
]

where:

• ( W ) is the weight matrix (shared across all nodes)

• ( [h_1; h_2] ) is the concatenation of the child node vectors

• ( f ) is an activation function (like tanh or ReLU)

This process continues recursively up the tree until a single vector representation of the entire
structure (the root) is formed.

4. Example: Sentence Parsing

Let’s say we have a simple sentence:
“The movie was great.”

A Recursive Neural Network might process it as follows:

[Sentence]
/ \
[NP: The movie] [VP: was great]
/ \ / \
The movie was great
Each word is converted into a word embedding (vector).
The RecNN then recursively combines these embeddings bottom-up to form phrase vectors and
finally a sentence vector.

5. Mathematical Formulation
For a node ( j ) with two child nodes ( l ) and ( r ):

[
h_j = f(W [h_l; h_r] + b)
]

where:

• ( h_j ): hidden state of node ( j )

• ( h_l, h_r ): hidden states of child nodes (left, right)

• ( W ), ( b ): shared weights and bias

• ( f ): activation function

If the tree has more than two children, the same function can be applied recursively across all
children.

6. Training
Training is done through a modified version of Backpropagation called Backpropagation
Through Structure (BPTS), where gradients are propagated through the tree structure rather
than through time (as in RNNs).

The model learns by minimizing a loss function (e.g., classification error, semantic distance) using
gradient descent.

7. Advantages
Advantage Explanation
Captures hierarchical
Works on tree-structured inputs rather than flat sequences
structure
Better sentence representation Understands nested dependencies in language
Reduces complexity, since the same function is used across the
Shared parameters
tree
Strong in semantic tasks Captures compositional meaning (e.g., sentiment of entire phrase)

8. Disadvantages
Disadvantage Explanation
Needs structured input Requires a tree structure (like parse trees) before training
Computationally Traversing and backpropagating through trees is
heavy complex
Hard to parallelize Tree structures vary in depth and branching factor
Data preprocessing Requires syntactic parsing, which can introduce errors

9. Applications
Domain Example
NLP Sentiment analysis using phrase trees
Semantic Parsing Understanding sentence meaning hierarchically
Computer Vision Scene understanding based on part hierarchies
Expression Parsing and evaluating mathematical or logical
Evaluation expressions

10. Key Takeaways

• Recursive NNs handle hierarchical or tree-like data, while RNNs handle sequential data.

• They build compositional representations (combining small parts into a whole).

• Common in natural language understanding and semantic analysis.

• Training uses Backpropagation Through Structure.

• Conceptually similar to RNNs but works over trees instead of time.

11. Quick Summary

Concept Recursive Neural Network
Input Structure Tree / Hierarchical
Information
Flow Bottom-up (child → parent)
Use Case NLP parsing, sentence meaning
Backpropagation Through
Training Method
Structure
Limitation Requires tree-form data

The Challenge of Long-Term Dependencies

1. Introduction
Recurrent Neural Networks (RNNs) are built to handle sequential data and remember past
information.
However, in practice, they often fail to learn relationships between inputs that are far apart in
time.

This problem is known as the Long-Term Dependency Problem — it’s the reason why models
like LSTM and GRU were created.

2. What Are Long-Term Dependencies?

In many sequences, early inputs affect later outputs.
For example:

• In a sentence like:
“The color of the car is red,”
understanding “is red” depends on remembering “color” that appeared much earlier.

So, a model must retain information over several time steps — that’s a long-term dependency.

However, standard RNNs can only retain short-term memory due to how gradients behave during
training.

3. How RNNs Learn Across Time

RNNs are trained using Backpropagation Through Time (BPTT) — gradients are propagated
backward across time steps to adjust weights.

During this process, gradients are multiplied repeatedly by the same weight matrix at each time
step.

If the largest eigenvalue of this matrix is:

• < 1 → gradients shrink exponentially → vanishing gradient

 • > 1 → gradients grow exponentially → exploding gradient

This means information from earlier steps either disappears or explodes as it travels backward
through time.

4. Vanishing Gradient Problem

Cause:

When the activation function (like tanh or sigmoid) squashes values between −1 and 1, the gradient
becomes smaller with each time step.

Mathematically,
[
\frac{\partial L}{\partial W} \approx \prod_{t=1}^{T} \frac{\partial h_t}{\partial h_{t-1}}
]

If each term < 1, the product rapidly goes to 0 as ( T ) increases.

Effect:

• Early layers (older time steps) stop learning.

• The RNN can only remember short-term context (few steps).

• Training becomes unstable and slow.

5. Exploding Gradient Problem

Cause:

If the gradient terms > 1, the multiplication across many steps causes values to blow up
exponentially.

Effect:

• Model weights become extremely large.

• Loss becomes NaN (not a number).

• Training diverges.

Solution:

Gradient clipping (limit the gradient magnitude) is often used to fix this.

6. Impact on Learning
Because of these issues, standard RNNs:

• Forget long-term information.

• Fail on tasks requiring long memory (like paragraph-level understanding, long time series).

• Are difficult to train beyond ~10–20 time steps effectively.

This is why models like LSTM, GRU, and attention-based networks were developed — to maintain
stable gradients and store information for longer periods.

7. Illustration
Time Steps → t1 → t2 → t3 → ... → tT
↓ ↓ ↓ ↓
(gradient decays →→→) → early inputs lose influence
In long sequences, gradients fade out before reaching earlier time steps — so the RNN can’t connect
distant relationships.

8. Solutions to Long-Term Dependency Problem

Approach Description
LSTM (Long Short-Term
Uses memory cells and gates to maintain long-term information.
Memory)
GRU (Gated Recurrent Simplified version of LSTM with reset and update gates.
Leaky Units Slow down hidden state updates to retain older information.
Echo State Networks (ESN) Use fixed random recurrent connections to maintain memory
Let the network “focus” on relevant parts of the input sequence
Attention Mechanisms
regardless of distance.
Residual or Skip Help gradient flow through longer time dependencies.

9. Key Takeaways
• The core issue is vanishing or exploding gradients during backpropagation through time.

• This prevents RNNs from learning dependencies over long sequences.

• LSTM, GRU, and attention models were specifically designed to overcome this limitation.

• Managing gradient flow and memory representation is the main challenge for
sequence models.

10. Quick Summary

Long-Term Dependency
Relationship between inputs far apart in sequence

Main Problem Gradients vanish or explode during training

Consequence RNN forgets long-past information
LSTM, GRU, Leaky Units, Echo State Networks, Attention
Solutions

Root Cause Repeated multiplication of small/large gradient values

Echo State Networks (ESNs)

1. Introduction
An Echo State Network (ESN) is a special kind of Recurrent Neural Network (RNN) that belongs to
a family called Reservoir Computing models.

The key idea is simple but powerful:

• In traditional RNNs, all weights are trained.

• In an ESN, only the output weights are trained.

The recurrent part (called the reservoir) is fixed and randomly initialized.

This approach makes ESNs faster to train and more stable while still being able to model complex
temporal dynamics.

2. Architecture Overview
An Echo State Network has three main components:

1. Input Layer:
Projects input signals into the reservoir.

2. Reservoir (Dynamic Recurrent Layer):

A large, fixed network of interconnected neurons with random weights. This
layer acts as a “memory” that stores temporal information.

3. Output Layer:
Reads from the reservoir state and produces the final output. Only
this layer’s weights are learned.

Structure Diagram (Conceptual)

Input (x_t) → [Reservoir: random recurrent layer] → Output (y_t)
↑
|
Internal dynamics (echoes)
The term “echo” comes from the fact that the current state of the reservoir contains a fading memory
of all previous inputs — like an echo of past information.

3. Mathematical Model
At each time step ( t ):
1.
Reservoir State Update:
[
h_t = f(W_{in} x_t + W h_{t-1} + b)
]

◦ ( x_t ): input vector

◦ ( h_t ): reservoir (hidden) state

◦ ( W_{in} ): input weight matrix

◦ ( W ): reservoir weight matrix (fixed, random)

◦ ( f ): activation function (usually tanh)

2.
Output Calculation:
[
y_t = W_{out} [h_t; x_t]
]

◦ ( W_{out} ): trained output weight matrix

◦ Only ( W_{out} ) is updated during training.

4. Training
• The reservoir weights ( W ) and input weights ( W_{in} ) are initialized randomly
and remain fixed.

• The only parameter learned is ( W_{out} ), which is trained using linear regression
(or ridge regression).

This makes ESNs:

• Much faster to train than standard RNNs or LSTMs.

• Less prone to gradient problems, since backpropagation through time is not used.
 5. Echo State Property
The ESN’s reservoir must satisfy a critical condition called the Echo State Property (ESP): The

influence of a previous input on the current state must fade away gradually over time.

In simpler terms, the reservoir should not explode or retain infinite memory — it should “echo” past
information with decreasing strength.

Mathematically, this is ensured if the spectral radius (largest absolute eigenvalue) of the reservoir weight
matrix ( W ) is less than 1:
[
\rho(W) < 1
]

This guarantees stability and bounded memory.

6. Advantages
Advantage Explanation
Only output weights are trained (no backpropagation through time)
Fast training

Stable dynamics Avoids vanishing/exploding gradients

Good memory capacity Maintains short and medium-term dependencies
Works well with small datasets
Less prone to overfitting

7. Disadvantages
Disadvantage Explanation
Reservoir tuning required Proper initialization (size, spectral radius) is critical
Random weights Performance depends on random initialization quality
Reservoir is fixed, cannot adjust internal dynamics during
Limited adaptability
training
Not ideal for highly nonlinear
Since only output weights adapt
dependencies

8. Applications
Domain Example
Time Series Forecasting
Predicting weather, traffic, or stock data
Phoneme recognition or sound pattern analysis
Speech Processing
Control Systems System identification and adaptive control
EEG/ECG Signal
Detecting brain or heart activity patterns
Analysis

9. Comparison: RNN vs ESN

Feature Traditional RNN Echo State Network
All weights trained via
Training Only output weights trained
BPTT
Gradient
Yes (vanishing/exploding) No
Issues
Training Time High Very Low
Memory Type Learnable Fixed, dynamic reservoir
Controlled via spectral
Stability Depends on learning
radius
Applications NLP, sequences Time series, control systems

10. Key Takeaways

• ESNs replace the fully trainable recurrent core with a fixed reservoir.

• They retain a rich memory of past inputs (echoes).

• Only the output layer is trained — drastically reducing training cost.

• Proper control of the spectral radius ensures Echo State Property and stable dynamics.

• Useful for time-dependent pattern recognition and forecasting.

11. Quick Summary

Concept Echo State Network
Fixed random recurrent reservoir + trained output
Core Idea
layer
Training Only output weights learned
Stability
Spectral radius < 1
Condition
Advantage Fast, stable, memory-efficient
Limitation Limited flexibility and tunability
Leaky Units and Other Strategies for Multiple
Time Scales
1. Introduction
Sequential data often contains patterns that exist at different time scales:

• Short-term patterns (recent events)

• Long-term patterns (older but still relevant events)

Standard RNNs update the hidden state fully at every time step, which often causes rapid
forgetting of older information.

To counter this, researchers introduced leaky units and similar mechanisms that allow the network
to retain information over longer periods by controlling how fast the hidden state updates.

2. What Are Leaky Units?

A leaky unit is a recurrent neuron that updates its state slowly, meaning it blends the previous
state and the new input-driven state.

Instead of replacing the hidden state ( h_t ) entirely at each time step, a leaky unit uses a leak rate
(also called decay or time constant):

[
h_t = (1 - \alpha) h_{t-1} + \alpha f(Wx_t + Uh_{t-1} + b)
]

Where:

• ( \alpha ): leak rate (0 < α ≤ 1)

• ( h_{t-1} ): previous hidden state

• ( f(\cdot) ): activation function

3. Intuition Behind the Equation

• If ( \alpha = 1 ) → behaves like a normal RNN (full update)

• If ( \alpha ) is small (like 0.1) → the neuron keeps memory longer, updating little each step

Effect:
The unit behaves like a slow-moving average, keeping past information alive.
4. Why We Need Leaky Units
Issue in Standard RNN Solution Provided
Quickly overwrites old info Slowly updates state
Vanishing gradients for long Maintains memory over longer
dependencies time
Poor performance on long sequences Handles multiple time scales

This makes the model better at learning signals like:

• Long-term trends in time-series data

• Prosody in speech

• Sentence-level semantics in NLP

5. Multiple Time-Scale Strategies

Leaky units are one method. Other strategies include:

a) Multiple Time-Scale RNN (MTRNN)

Assign different layers different time constants:

• Lower layers: fast-changing state (short-term memory)

• Higher layers: slow-changing state (long-term memory)

b) Clockwork RNN

Different parts of the network update at different time intervals (like clocks ticking at different
speeds).

c) Skip/Residual Connections

Skip connections help gradients propagate across long time spans and layers.

d) Gated Architectures (LSTM, GRU)

LSTMs/GRUs introduce gates to control information flow and preserve long-term memory.
They can be seen as advanced, learnable leaky units.

6. Benefits
Benefit Explanation
Better long-term memory Helps model capture slow-changing patterns
Improved gradient flow Reduces vanishing gradient issues
Smooth updates avoid overfitting to recent
More stable learning
inputs
Handles complex
Useful for speech, sensor data, text
sequences

7. Limitations
Limitation Reason
More hyperparameters Requires tuning leak rate α
Harder to optimize than gated units LSTM/GRU often perform better
Not ideal for extremely long Better combined with other
dependencies models

8. Applications
Field Example
Time-Series Forecasting Weather, finance, climate
Prosody, phoneme
Speech Processing
sequence learning
Control Systems Robotic motion with memory states
Natural Language
Paragraph-level context
Processing

9. Key Takeaways
• Leaky units update hidden states slowly → preserve memory.

• Controlled by leak rate (α).

• Helps model multiple time scales in sequences.

• Used as a bridge between simple RNNs and gated networks (LSTM, GRU).

• Often paired with other mechanisms for best performance.

10. Summary Table

Concept Meaning
Leaky Unit Slow state update to retain memory
Leak Rate
How fast memory updates
( \alpha )
Purpose Capture short + long time dependencies
Advantage Better long-term retention
Needs tuning, less powerful than LSTM/
Drawback
GRU

The Long Short-Term Memory (LSTM) and

Other Gated RNNs

1. Introduction
The Long Short-Term Memory (LSTM) network is a special type of Recurrent Neural Network
(RNN) designed to remember information for long periods and avoid the vanishing/exploding
gradient problem.

LSTM was introduced by Hochreiter and Schmidhuber (1997) to allow the model to store, use,
and forget information selectively through gates — making it capable of learning long-term
dependencies.

2. The Core Idea

In a normal RNN, every new input completely updates the hidden state, often overwriting older
information.
In an LSTM, the network has an internal memory cell and three gates that control how
information flows:

• Forget Gate → decides what to erase

• Input Gate → decides what to store

• Output Gate → decides what to output

These gates help the network retain useful information over hundreds of time steps.

3. LSTM Cell Structure

An LSTM cell contains:

1. Cell state (( C_t )): The “memory” of the network

2. Hidden state (( h_t )): The output of the cell at time ( t )

3. Three gates controlling information flow

4. Mathematical Formulation
Let:

• ( x_t ): input vector at time ( t )

• ( h_{t-1} ): previous hidden state

• ( C_{t-1} ): previous cell state

Then the LSTM updates are:

1. Forget Gate:
Controls what information to discard from the previous cell state
[
f_t = \sigma(W_f [h_{t-1}, x_t] + b_f)
]

2. Input Gate:
Decides what new information to add to the cell state
[
i_t = \sigma(W_i [h_{t-1}, x_t] + b_i)
]
[
\tilde{C}t = \tanh(W_C [h{t-1}, x_t] + b_C)
]

3. Cell State Update:

[
C_t = f_t * C_{t-1} + i_t * \tilde{C}_t
]

4. Output Gate:
Determines what to output
[
o_t = \sigma(W_o [h_{t-1}, x_t] + b_o)
]
[
h_t = o_t * \tanh(C_t)
]

Here:

• ( \sigma ): sigmoid activation (values between 0 and 1)

• ( * ): element-wise multiplication

5. How It Works (Step-by-Step)

1. Forget gate checks old memory ( C_{t-1} ) and decides which parts to keep.
 2. Input gate adds new relevant information from ( x_t ).

3. Cell state is updated — part old memory, part new input.

4. Output gate produces the next hidden state ( h_t ), which is also used for the next time step.

The cell state acts like a conveyor belt, flowing through time with minor linear interactions — this is
what preserves long-term memory.

6. Why It Solves Long-Term Dependency

Unlike normal RNNs, where gradients vanish through repeated multiplication, the cell state in LSTM
provides an almost constant gradient path, allowing learning of long-range dependencies.

This makes LSTM stable during backpropagation and effective even on long sequences.

7. Advantages of LSTM
Advantage Description
Long-term memory Maintains information over many time steps
Solves vanishing gradient problem
Stable gradient flow through gates
Learns what to remember or forget automatically
Selective memory

Versatile Works for text, speech, video, and time series

8. Disadvantages of LSTM
Disadvantage Explanation
Computationally expensive
More gates → more parameters
Training time Slower than simple RNNs or GRUs
Complex architecture Harder to interpret
Needs regularization for small datasets
Overfitting risk

9. Variants of Gated RNNs

a) Gated Recurrent Unit (GRU)

A simplified version of LSTM introduced by Cho et al. (2014).

It merges the input and forget gates into a single update gate, reducing parameters. Equations:
[
z_t = \sigma(W_z [h_{t-1}, x_t])
]
[
r_t = \sigma(W_r [h_{t-1}, x_t])
]
[
\tilde{h}t = \tanh(W [r_t * h{t-1}, x_t])
]
[
h_t = (1 - z_t) * h_{t-1} + z_t * \tilde{h}_t
]

Advantages over LSTM:

• Fewer parameters → faster training

• Similar performance

• Easier to implement

b) Peephole LSTM

Adds direct connections from the cell state to gates, allowing gates to “peek” at the memory state.

c) Coupled Forget and Input Gates

Sometimes the forget and input gates are coupled:

[
f_t = 1 - i_t
]
This reduces redundancy and simplifies training.

10. Applications
Domain Use Case
Machine translation, text generation, sentiment
NLP
analysis
Speech
Sequence modeling for audio frames
Recognition
Time Series Stock, weather, energy forecasting
Video Analysis Action recognition, frame prediction
Anomaly Detection Detecting unusual temporal patterns

11. Key Takeaways

• LSTM = RNN + memory cell + gates
 • Solves long-term dependency and gradient issues

• GRU = simpler, faster variant with near-equal performance

• Both are the backbone of modern sequence models

• Foundation for attention and Transformer-based models

12. Quick Summary

Concept Description
Uses gates to manage memory over long sequences
LSTM

Forget GateDecides what to discard

Input Gate Decides what to store
Output GateControls what to output
GRU Simplified gated RNN with fewer parameters
Main Benefit
Learns long-term patterns efficiently
Problem
Vanishing/exploding gradients
Solved

Optimization for Long-Term Dependencies

1. Introduction
Training recurrent neural networks (RNNs) over long sequences is extremely challenging because of
gradient instability — either vanishing or exploding gradients during Backpropagation
Through Time (BPTT).

Even with architectures like LSTM or GRU that mitigate these issues, optimization still requires special
care to ensure stability, convergence, and proper long-term learning.

This section focuses on techniques and strategies that improve optimization for long-term dependencies.

2. Main Challenge
The Gradient Problem

When an RNN is unrolled over many time steps, the gradient of the loss function with respect to
parameters is a product of many Jacobians:
This repeated multiplication can cause:

• Vanishing gradients: earlier layers stop learning.

• Exploding gradients: numerical instability during updates.

Both lead to poor convergence and inability to capture long-term patterns.

3. Optimization Strategies
1. Gradient Clipping

Prevents exploding gradients by capping the gradient magnitude before parameter updates.

If the gradient norm exceeds a threshold ( \tau ):

[
g \leftarrow g \cdot \frac{\tau}{||g||}
]
This keeps gradients within a stable range, ensuring controlled updates.

Common thresholds: 1.0 or 5.0 (problem-dependent)

2. Proper Weight Initialization

Initialization directly affects gradient flow.

Good practices include:

• Orthogonal Initialization: ensures stable activations in recurrent connections.

• Xavier or He Initialization: scales weights to maintain variance across layers.

• Keeping the spectral radius ≤ 1 for recurrent matrices maintains stability.

3. Use of Gated Units (LSTM, GRU)

LSTMs and GRUs incorporate gating mechanisms to:

• Maintain constant error flow through memory cells.

• Reduce the need for complex tuning.

• Prevent vanishing gradients automatically through additive memory updates.

This remains the most common solution to long-term optimization challenges.

 4. Regularization Techniques

Long-term training can lead to overfitting. Use:

• Dropout in RNNs (variational dropout): applies same dropout mask across time steps.

• Weight decay (L2 regularization): penalizes large weights.

• Zoneout: randomly preserves some hidden activations from the previous time step instead
of dropping them.

These methods improve generalization and prevent instability.

5. Truncated Backpropagation Through Time (TBPTT)

Instead of backpropagating through the entire sequence, the network is trained over shorter sub-
sequences.

For example:

• Break a long sequence of 1000 steps into chunks of 20–50 steps.

• Backpropagate only within those chunks.

This reduces computational cost and mitigates vanishing gradient effects, while still learning temporal
patterns.

6. Layer Normalization

Stabilizes hidden activations by normalizing neuron outputs across features, leading to smoother gradient
flow.

Used extensively in deep and stacked RNNs to maintain consistent scales through layers and time.

7. Residual or Skip Connections

Connecting layers or time steps directly (skipping intermediate ones) allows gradients to bypass vanishing
paths, similar to ResNet principles.

For example:
[
h_t^{(l)} = h_t^{(l)} + h_t^{(l-1)}
]
This helps deep RNNs propagate long-term dependencies effectively.

8. Adaptive Optimizers
Use modern optimization algorithms like:

• Adam (Adaptive Moment Estimation)

• RMSProp

• Nadam

These adjust learning rates dynamically based on gradient history, improving stability during long-
sequence training.

4. Empirical Tricks
Technique Purpose
Gradient clipping Prevents exploding gradients
Learning rate
Smooths convergence
scheduling
Batch normalization Stabilizes intermediate activations
Prevents overfitting and improves
Noise injection
generalization
Sequence shuffling Reduces temporal bias during training

5. Visualization of Gradient Flow

Normal RNN:
[ h1 ] → [ h2 ] → [ h3 ] → ... → [ hT ]
↓
Gradients vanish → forgets early info

With Optimization Techniques:

Stable gradient flow → retains memory longer

6. Key Observations
• Optimization issues are not just architectural but also training-level problems.

• Even LSTMs need gradient clipping, good initialization, and regularization.

• Most real-world models use a combination of these strategies.

7. Applications
• Speech recognition → stable training over thousands of audio frames.

• Language modeling → capturing long-term word dependencies.

 • Financial forecasting → long-range temporal correlation.

• Video understanding → motion continuity across frames.

8. Key Takeaways
Concept Description
Gradient Instability
Main obstacle in training long-sequence models
Core Fix Gradient clipping, gated architectures
Supported via additive memory (LSTM) or slow update (leaky units)
Long-Term Memory
Training Optimization
Requires adaptive learning rates and careful regularization

Truncated BPTT Practical approach for long sequence training

9. Summary Table
Technique Purpose Effect
Gradient Clipping Limit gradient magnitude Prevent explosion
Reduce vanishing
Proper Initialization Stabilize training
gradients
LSTM / GRU Structural fix Enable long-term memory
Practical backpropagation
TBPTT Reduces cost
Layer Normalization
Smoother activations Better convergence
Adaptive Optimizers
Auto-tune learning rates Faster, stable training

Explicit Memory

1. Introduction
Traditional RNNs (including LSTM and GRU) store information implicitly inside hidden states and cell
states.
However, their memory capacity is limited — once the sequence is long or the required dependency is
far away, the model starts to forget.

To overcome this, researchers introduced Explicit Memory Mechanisms, where the network has a
separate memory component that it can read from and write to — similar to how a computer
uses RAM.
This idea enables neural networks to:

• Store large amounts of information

• Access it when needed

• Perform reasoning over stored data

2. Implicit vs Explicit Memory

Implicit Memory (RNN/
Feature Explicit Memory (Advanced Models)
LSTM)
Where memory is
Hidden states or cell states Separate memory matrix
stored
Automatically through
How it’s accessed Learned read/write operations
recurrence
Memory size Fixed (limited by hidden Scalable (variable-sized external memory)
Flexibility Stores short-term context Stores structured, long-term knowledge
Neural Turing Machine, Memory Network,
Example models RNN, LSTM, GRU
Transformer

3. Architecture Overview
An Explicit Memory Network has the following components:

1. Controller – a neural network (like an RNN or LSTM) that decides what to write/read.

2. External Memory Matrix (M) – stores information explicitly.

3. Read/Write Heads – modules that perform differentiable read/write operations on memory.

4. Output Unit – produces predictions using controller outputs and read memory content.

Conceptual Diagram

┌──────────────────────────────┐
Input ───▶ │ Controller (LSTM) │
└──────────────────────────────┘
│
┌────────────────┴────────────────┐
▼ ▼
[ Write Head ] [ Read Head ]
│ │
▼ ▼
┌──────────────────────────────────────────┐
│ External Memory Matrix (M) │
 └──────────────────────────────────────────┘
│
▼
[ Output ]

4. Mathematical Representation
Let the memory matrix at time ( t ) be ( M_t \in \mathbb{R}^{N \times D} ):

• ( N ): number of memory slots

• ( D ): dimension of each slot

Write Operation:
[
M_t = M_{t-1} + w_t^{write} \otimes c_t
]
where:

• ( w_t^{write} ): write weight vector

• ( c_t ): content to write

• ( \otimes ): outer product

Read Operation:
[
r_t = M_t^T w_t^{read}
]
where:

• ( w_t^{read} ): read weight (learned attention over memory slots)

• ( r_t ): retrieved content (used by the controller)

Both read and write weights are generated differentiably using attention-like mechanisms, enabling
the entire system to be trained via gradient descent.

5. Key Explicit Memory Architectures

a) Neural Turing Machine (NTM) – DeepMind (2014)

• Controller (LSTM) + differentiable external memory.

• Can learn algorithms like copying, sorting, and recall.

• Mimics the read/write logic of a Turing machine.

b) Memory Networks (Facebook AI Research)

 • Focused on question-answering tasks.

• Memory is a set of facts; the model learns which ones to attend to (using attention).

• Produces interpretable reasoning chains.

c) Differentiable Neural Computer (DNC)

• Advanced version of NTM with dynamic memory allocation, temporal links, and better
memory management.

d) Transformer Models

• Though not strictly "external memory," Transformers use self-attention, which acts as an
implicit memory retrieval mechanism across the entire sequence.

6. Advantages
Advantage Description
Long-term storage Keeps information for unlimited time steps
Learnable read/write Model learns how to store and retrieve relevant info
High reasoning
Can solve algorithmic and question-answering tasks
power
We can visualize what the model stores and
Interpretability
accesses

7. Disadvantages
Disadvantage Explanation
Managing external memory increases computational
High complexity
cost
Difficult training Hard to stabilize read/write attention
Requires large data Needs diverse examples to learn storage behavior
Memory
Risk of overwriting or underusing memory slots
management

8. Applications
Domain Example
Question Answering Reading comprehension (bAbI dataset)
Reasoning Multi-hop reasoning (Memory Networks, DNC)
Algorithmic Tasks Copying, sorting, associative recall
Reinforcement
Navigation and planning with long-term memory
Learning
Sequence modeling with implicit self-attention
Transformers
memory

9. Comparison Summary
Memory
Model Description
Type
RNN Implicit Limited by hidden state
LSTM/
Semi-implicit Memory cell with gates
GRU
NTM /
Explicit External read/write memory
DNC
Implicit Self-attention acts as memory
Transformer
global lookup

10. Key Takeaways

• Explicit memory allows neural networks to handle long sequences, perform reasoning, and
store structured knowledge.

• It separates storage (memory) from computation (controller).

• Trained end-to-end using differentiable read/write operations.

• Used in modern reasoning architectures and AI models requiring long-term context.

• It bridges the gap between traditional RNNs and symbolic reasoning systems.

11. Quick Summary

Concept Explicit Memory
Neural network with a dedicated external memory
Definition
module
Components Controller, Memory Matrix, Read/Write heads
Purpose Long-term storage and reasoning
Example
Neural Turing Machine, DNC, Memory Network
Models
Training Differentiable attention mechanisms
Advantage Solves long-context and reasoning tasks