MODULE 5 — RECURRENT NEURAL NETWORKS (RNN), LSTM

AND GRU
Prepared from your uploaded lecture material.

1. Introduction to Sequence Learning

In many real-world problems:
● data comes in sequence form
● current output depends on previous inputs
This is called:
Sequence Learning

Examples of Sequential Data

. Sentences
. Speech signals
. Stock prices
. Sensor signals
. Video frames
. Time-series data

Problem with Traditional Neural Networks

Feedforward Neural Networks:
● accept fixed-size inputs
● treat each input independently
But sequential data:
● may have variable length
● depends on previous context
Thus traditional networks fail in sequence learning tasks.

Example
6
Sentence:
5
“The movie was boring and long”
To predict sentiment:
4
model must remember previous words.
3
●
2
Need for Memory
1
Human brain remembers previous information while processing current
information.
Similarly:
RNNs have memory

2. Recurrent Neural Networks (RNN)

RNN is a neural network designed for sequential data.
Main feature:
● output depends on current input and previous state.
Key Idea of RNN
Information persists through:
Hidden State

Hidden State
Hidden state stores information from previous timesteps.
Denoted by:
[
s_t
]

RNN Architecture
At every timestep:
● input enters network
● hidden state updates
● output produced

RNN Equation
s_i=\sigma(Ux_i+Ws_{i-1}+b)

Output Equation
y_i=O(Vs_i+c)
Where:
● (x_i) = input
● (s_i) = hidden state
● (y_i) = output
● (U,W,V) = weights

Parameter Sharing
Same weights used at every timestep.
Advantages:
. Fewer parameters
. Handles variable-length sequences
. Generalization improves

3. Unrolling of RNN
RNN can be visualized as multiple copies of same network over time.
This is called:
Unrolling

Important Point
Although many copies appear:
● all copies share same parameters.
4. Why RNN is Better for Sequential Data?
RNN remembers previous context using hidden state.
Thus:
current prediction depends on:
. current input
. previous memory

Example
Predicting next character:
[
d \rightarrow e \rightarrow e \rightarrow p
]
Prediction depends on previous letters.

5. Applications of RNN
. Machine Translation
. Sentiment Analysis
. Speech Recognition
. Handwriting Recognition
. Stock Prediction
. Text Generation
. Activity Recognition

6. Sequence Learning Architectures

Different sequence mappings are possible.

(1) One Input → One Output

Example:
● image classification

(2) One Input → Sequence Output

Example:
● image captioning
Image:
→ “A dog is running”

(3) Sequence Input → One Output

Example:
● sentiment analysis
Sentence:
→ positive/negative

(4) Sequence Input → Sequence Output

Example:
 ● machine translation
English:
→ French

(5) Synced Sequence Input and Output

Example:
● video frame labeling

7. Problems in Vanilla RNN

Vanilla RNN suffers from:
. Vanishing Gradient
. Exploding Gradient

8. Vanishing Gradient Problem

Gradients become extremely small during backpropagation.
Result:
● earlier layers stop learning
● long-term dependencies lost

Example
In long sentence:
network forgets beginning words.

Cause
Repeated multiplication of small values.

Effects
. Poor long-term memory
. Slow learning
. Weak gradients

9. Exploding Gradient Problem

Gradients become extremely large.
Result:
● unstable training
● huge weight updates

Cause
Repeated multiplication of large values.

Effects
. Numerical instability
. Oscillating loss
. Diverging model
Solutions
. Gradient clipping
. Proper initialization
. LSTM
. GRU

10. Networks with Memory

To solve RNN limitations:
advanced recurrent architectures were introduced.
. LSTM
. GRU

Important Difference
Vanilla RNN:
● multiplicative memory updates
LSTM and GRU:
● additive memory updates
This improves gradient flow.

11. Long Short-Term Memory (LSTM)

LSTM is a special RNN designed to remember long-term dependencies.
Introduced by:
Hochreiter and Schmidhuber (1997)

Main Idea of LSTM

Uses:
. Cell State
. Gates
to control information flow.

Cell State
Cell state acts like:
Long-term memory
Information flows through network with minimal changes.

Gates in LSTM
. Forget Gate
. Input Gate
. Output Gate

12. Forget Gate

Decides:
what information should be removed from memory.
Uses sigmoid activation.
Output range:
[
0 \text{ to } 1
]

Interpretation
0
Forget completely
1
Keep completely

Forget Gate Equation

f_t=\sigma(W_f x_t+U_f h_{t-1}+b_f)

13. Input Gate

Decides:
what new information should be stored.

Input Gate Equation

i_t=\sigma(W_i x_t+U_i h_{t-1}+b_i)

Candidate Cell State

\tilde{C_t}=\tanh(W_cx_t+U_ch_{t-1}+b_c)

14. Cell State Update

Old memory updated using:
● forget gate
● input gate

Cell State Equation

C_t=f_tC_{t-1}+i_t\tilde{C_t}

15. Output Gate

Decides:
what information should be output.

Output Gate Equation

o_t=\sigma(W_ox_t+U_oh_{t-1}+b_o)

Hidden State Output

h_t=o_t*\tanh(C_t)

Advantages of LSTM
. Solves vanishing gradient problem
. Learns long-term dependencies
 . Better memory retention
. Excellent for sequential data

Applications of LSTM
. Speech recognition
. Machine translation
. Text generation
. Chatbots
. Time-series prediction

16. Gated Recurrent Unit (GRU)

GRU is simplified version of LSTM.
Introduced by:
Cho et al. (2014)

Difference from LSTM

GRU:
● fewer gates
● fewer parameters
● faster training

Gates in GRU
. Update Gate
. Reset Gate

Update Gate
Controls:
how much previous memory to keep.

Reset Gate
Controls:
how much past information to forget.

Advantages of GRU
. Faster than LSTM
. Simpler architecture
. Less computational cost
. Good performance

Limitation
Sometimes slightly weaker than LSTM for extremely long dependencies.

17. LSTM vs GRU

 LSTM GRU
3 gates 2 gates
Separate cell state No separate cell state
More parameters Fewer parameters
Slower training Faster training
Better for complex tasks Computationally efficient
18. Bidirectional RNN
Bidirectional RNN processes sequence:
. forward direction
. backward direction
Thus model gets:
● past context
● future context

Advantages
. Better context understanding
. Improved sequence prediction

Applications
. NLP
. Speech recognition
. Machine translation

19. Sequence-to-Sequence (Seq2Seq) Model

Used for converting one sequence into another.
Example:
English:
“I love you”
French:
“Je t’aime”

Components
. Encoder RNN
. Decoder RNN

Encoder
Converts input sequence into hidden representation.

Decoder
Generates output sequence from encoded representation.

Applications of Seq2Seq
. Translation
. Summarization
 . Chatbots
. Question answering

20. Advantages of RNN

. Handles variable-length sequences
. Maintains memory
. Parameter sharing
. Suitable for temporal data

21. Limitations of RNN

. Vanishing gradients
. Exploding gradients
. Slow sequential computation
. Difficult long-term learning

EXAM READY 5 MARKER ANSWERS

Q1. Explain Recurrent Neural Network (RNN).

Answer
Recurrent Neural Network (RNN) is a neural network designed for processing
sequential data.
Unlike feedforward networks, RNN maintains memory using hidden states.
Hidden state equation:
s_i=\sigma(Ux_i+Ws_{i-1}+b)
Output equation:
y_i=O(Vs_i+c)
Features:
. Handles sequential data
. Uses parameter sharing
. Maintains temporal memory
Applications:
● speech recognition
● NLP
● sentiment analysis

Q2. Explain Vanishing and Exploding Gradient Problems.

Answer
Vanishing and exploding gradients occur during backpropagation in deep
RNNs.
Vanishing Gradient
Gradients become extremely small.
Effects:
. Earlier layers stop learning
. Long-term dependencies lost
Exploding Gradient
Gradients become extremely large.
Effects:
. Unstable learning
. Large weight updates

Solutions
. Gradient clipping
. LSTM
. GRU
. Proper initialization
These problems are major limitations of vanilla RNN.

Q3. Explain LSTM Architecture.

Answer
LSTM (Long Short-Term Memory) is an advanced RNN architecture designed to
learn long-term dependencies.
Components:
. Cell State
. Forget Gate
. Input Gate
. Output Gate
Forget gate:
f_t=\sigma(W_fx_t+U_fh_{t-1}+b_f)
Cell state update:
C_t=f_tC_{t-1}+i_t\tilde{C_t}
Advantages:
. Solves vanishing gradient problem
. Learns long-term dependencies
. Better memory handling
Applications:
● machine translation
● speech recognition

Q4. Explain GRU Architecture.

Answer
GRU (Gated Recurrent Unit) is a simplified version of LSTM.
It uses:
. Update Gate
. Reset Gate
Features:
. Fewer parameters
. Faster training
. Simpler structure
Advantages:
. Computational efficiency
. Good sequential learning performance
Applications:
● NLP
● time-series forecasting
● speech processing

Q5. Differentiate RNN, LSTM and GRU.

Answer
Feature RNN LSTM GRU
Memory Weak Strong Strong
Gates No gates 3 gates 2 gates
Long-term Poor Excellent Very good
learning
Complexity Simple Complex Moderate
Training speed Fast Slow Faster
LSTM and GRU are improvements over vanilla RNN.

Q6. Explain Sequence Learning Architectures.

Answer
Sequence learning architectures process sequential inputs and outputs.
Types:
. One-to-One
. One-to-Many
. Many-to-One
. Many-to-Many
Applications:
● image captioning
● sentiment analysis
● machine translation
● video processing
RNNs provide flexible sequence modeling.

Q7. Explain Seq2Seq Model.

Answer
Sequence-to-Sequence (Seq2Seq) model converts one sequence into another.
Components:
. Encoder
. Decoder
Encoder converts input sequence into hidden representation.
Decoder generates output sequence.
Advantages:
. Handles variable-length sequences
 . Effective for translation
Applications:
● machine translation
● summarization
● chatbots

Q8. Explain Bidirectional RNN.

Answer
Bidirectional RNN processes sequences in:
. forward direction
. backward direction
Thus model uses:
● past context
● future context
Advantages:
. Better understanding of sequence
. Improved prediction accuracy
Applications:
● NLP
● speech recognition
● sequence labeling
These notes are based on your uploaded RNN, LSTM and GRU lecture material.