RECURRENT NEURAL NETWORKS (Unit – 4)

Sequential Data :
• Sequence data (or sequential data) refers to any data where the arrangement and position of elements are
crucial to its meaning.
• Sequential data points are dependent on those that come before and after them
• Key Characteristics
➢ Temporal/Chronological Order: Elements are often collected or observed in a specific order over
time (e.g., stock prices recorded daily).
➢ Contextual Dependency: The meaning of a single element (like a word in a sentence) is determined
by its surrounding context.
➢ Variable Length: Sequences can vary significantly in size, such as sentences of different lengths or
varying durations of audio signals
Architectures for Sequence Data:
• Artificial Neural Networks (ANN) fail when dealing with sequential data, because they:
➢ Require fixed-size input
➢ Do not consider order of data
➢ Have no memory of previous inputs
• Hence, they cannot handle problems where context and sequence matter, leading to the need for Recurrent
Neural Networks (RNN).
• Specialized architectures have been developed to handle this:
➢ Recurrent Neural Networks (RNNs): These networks use recursive connections to pass information
from one step to the next, acting as a "memory".
➢ Long Short-Term Memory (LSTM): An advanced RNN variant that uses gating mechanisms (input,
output, and forget gates) to manage long-term dependencies and prevent issues like vanishing
gradients.
➢ Gated Recurrent Units (GRU): A simpler, faster alternative to LSTMs with similar performance,
often used when computational resources are limited.
➢ Transformers: The current state-of-the-art that uses attention mechanisms to process the entire
sequence at once (parallel processing) while focusing on the most relevant parts.
Recurrent Neural Network (RNN):
• A Recurrent Neural Network (RNN) is a class of artificial neural network designed to process sequential
data, such as text, speech, or time-series information.
• Unlike traditional feedforward neural networks that process inputs independently, RNNs feature recurrent
connections (loops) that allow information to persist over time.
• RNNs maintain a "hidden state" that acts as a memory, storing information about previous inputs in a
sequence to influence the current output
• The same set of weights is applied at each time step, allowing the network to handle variable-length
sequences and generalize across different positions in time.

Input sequence: 𝑥1 , 𝑥2 , . . . , 𝑥𝑛 𝑈 recurrent weight matrix

𝑥𝑡 input at time step 𝑡 , 𝑉 output weight matrix
ℎ𝑡 hidden state (memory) Hidden states carry memory: ℎ1 , ℎ2 , . . . , ℎ𝑛
𝑦𝑡 prediction Outputs at each step: 𝑦1 , 𝑦2 , . . . , 𝑦𝑛
𝑊 input weight matrix

1
 • Folded Representation (Left Side):
➢ At time step t, the network receives an input 𝑥𝑡 .
➢ The hidden state ℎ𝑡 processes the input and also receives information from the previous hidden state.
➢ ℎ𝑡 = φ(𝑊𝑥𝑡 + 𝑈ℎ𝑡−1 + 𝑏ℎ )
Where φ = activation function (usually tanh).
➢ The hidden state produces an output 𝑦𝑡 .
➢ 𝑦𝑡 = 𝑔(𝑉ℎ𝑡 + 𝑏𝑦 )
Where 𝑔= activation function (e.g., softmax for classification).
➢ The loop around ℎ𝑡 represents the recurrent connection, meaning the hidden state depends on its
previous value.
• UnFolded Representation (Right Side):
➢ In the unfolded representation of an RNN, the network is expanded across multiple time steps to
show how it processes a sequence of inputs. Each time step receives an input 𝑥𝑡 , computes a hidden
state ℎ𝑡 using the previous hidden state ℎ𝑡−1 , and produces an output 𝑦𝑡 .
• Recurrent Neural Networks (RNNs) set themselves apart from other neural networks with their unique
capabilities:
➢ Internal Memory: This is the key feature of RNNs. It allows them to remember past inputs and use that
context when processing new information.
➢ Sequential Data Processing: Because of their memory, RNNs are exceptional at handling sequential data
where the order of elements matters. This makes them ideal for speech recognition, machine
translation, natural language processing (NLP), and text generation.
➢ Contextual Understanding: RNNs can analyze the current input about what they’ve “seen” before. This
contextual understanding is crucial for tasks where meaning depends on prior information.
➢ Dynamic Processing: RNNs can continuously update their internal memory as they process new data,
allowing them to adapt to changing patterns within a sequence.
Applications of RNN:
• Language Translation: RNNs are used to translate text from one language to another by processing input
and generating output sequences.
• Speech Recognition: RNNs convert spoken audio signals into text by learning temporal patterns in
speech.
• Sentiment Analysis : RNNs analyze sequences of words to determine the sentiment (positive, negative,
neutral) of a sentence.
• Named Entity Recognition (NER) : RNNs identify entities like names, locations, and organizations by
using context from surrounding words.
• Time-Series Prediction : RNNs predict future values (e.g., stock prices or weather) by learning patterns
from past data.
Types of RNN :
1. Many-to-One RNN
• In a Many-to-One RNN, a sequence of inputs 𝑥1 , 𝑥2 , . . . , 𝑥𝑛 is processed, and the network produces one
final output 𝑦after reading the entire sequence. The hidden state accumulates information from all time
steps and the final hidden state is used for prediction. This architecture is commonly used when the whole
sequence represents one label.
• Examples
➢ Sentiment analysis of a sentence (positive/negative).
➢ Email spam classification.

2
2. One-to-Many RNN
• In a One-to-Many RNN, a single input is provided to the model, and it generates a sequence of outputs
over time. The hidden state helps generate outputs step by step. This structure is used when one input
needs to produce a sequence.
• Examples
➢ Image caption generation.
➢ Music generation from an initial note or theme.

3. Many-to-Many RNN (Same Length)

• In this architecture, the input sequence and output sequence have the same length, and the model produces
an output at every time step. Each output 𝑦𝑡 depends on the current input 𝑥𝑡 and the previous hidden state.
This is useful for tasks where each input element requires a corresponding prediction.
• Examples
➢ Part-of-speech tagging in NLP.
➢ Named entity recognition.

4. Many-to-Many RNN (Different Length)

• In this architecture, the input and output sequences can have different lengths. Typically an encoder-
decoder structure is used: the
encoder reads the input sequence
and the decoder generates the
output sequence. This model is
widely used in sequence
transformation tasks.
• Examples
➢ Machine translation (English
→ French).
➢ Text summarization.

3
Vanishing and Exploring Gradients :
• During training of RNN using Backpropagation Through Time (BPTT), gradients are multiplied
repeatedly across time steps.
∂𝐿
∝ 𝑈𝑘
∂ℎ𝑡

• Because of repeated multiplication of weights 𝑈, gradients can:

➢ Become very small → Vanishing Gradient
➢ Become very large → Exploding Gradient

Vanishing Gradient :
• Vanishing gradient is a major problem in training Recurrent Neural Networks (RNNs), especially when
dealing with long sequences. During training, RNNs use Backpropagation Through Time (BPTT), where
gradients are propagated backward across multiple time steps. In this process, the gradient is repeatedly
multiplied by the recurrent weight matrix and derivatives of activation functions such as sigmoid or tanh,
which are typically less than one.
• As a result, the gradient values shrink exponentially as we move backward in time. Eventually, the
gradients become extremely small, almost zero. This prevents the network from updating weights
effectively for earlier time steps.
• Due to vanishing gradients, the RNN fails to learn long-term dependencies. It can only remember recent
inputs but forgets information from distant past. For example, in a long sentence where a word depends
on another word far away, the model cannot capture that relationship.
• This problem limits the capability of basic RNNs in tasks like language modeling and translation. To
overcome this issue, advanced architectures such as LSTM and GRU are used, which are designed to
preserve long-term information.
Exploding Gradient in RNN
• Exploding gradient is another problem that occurs during the training of Recurrent Neural Networks
(RNNs). In Backpropagation Through Time, gradients are propagated backward through multiple time
steps. If the recurrent weights have large values, the gradients get multiplied repeatedly by numbers
greater than one.
• This causes the gradient values to grow exponentially, becoming very large or even infinite. As a result,
the weight updates become extremely large, leading to unstable training. The model may oscillate or
diverge, and the loss function may become undefined (NaN).
• Exploding gradients make it difficult for the network to converge to an optimal solution. The learning
process becomes highly sensitive and unreliable.
• A common solution to this problem is gradient clipping, where gradients are limited to a maximum
threshold during training. This prevents them from becoming too large and stabilizes the learning process.
Backpropagation Through Time (BPTT):
• Backpropagation Through Time (BPTT) is the training algorithm used for Recurrent Neural Networks
(RNNs). Since RNNs process sequential data, the same network is applied repeatedly across time steps.
• To train such networks, BPTT extends standard backpropagation by unfolding the RNN across time and
propagating errors backward through all time steps.

4
 • Forward Pass
➢ At each time step 𝑡, the hidden state and output are computed as:
ℎ𝑡 = 𝜙(𝑊𝑥𝑡 + 𝑈ℎ𝑡−1 + 𝑏ℎ )
𝑦𝑡 = 𝑔(𝑉ℎ𝑡 + 𝑏𝑦 )

➢ The total loss over the sequence is: 𝐿 = ∑𝑇𝑡=1 𝐿𝑡

• Backward Pass
➢ During backpropagation, gradients are propagated backward through time. The gradient of loss
with respect to hidden state is given by:
∂𝐿 ∂𝐿 ∂𝐿 ∂ℎ
➢
∂ℎ
= ∂ℎ𝑡 + ∂ℎ ⋅ ∂ℎ𝑡+1
𝑡 𝑡 𝑡+1 𝑡
➢ This shows that the gradient at time 𝑡depends on both:
Current time step loss
Future time step gradients
• Gradient Computation : Gradients for weights are computed by summing contributions from all time
steps:
𝑇
∂𝐿 ∂𝐿 ∂ℎ𝑡
=∑ ⋅
∂𝑊 ∂ℎ𝑡 ∂𝑊
𝑡=1
𝑇
∂𝐿 ∂𝐿 ∂ℎ𝑡
=∑ ⋅
∂𝑈 ∂ℎ𝑡 ∂𝑈
𝑡=1
𝑇
∂𝐿 ∂𝐿 ∂𝑦𝑡
=∑ ⋅
∂𝑉 ∂𝑦𝑡 ∂𝑉
𝑡=1
• Weight Updation ∶
𝜕𝐿
𝑊new = 𝑊𝑜𝑙𝑑 − 𝜂
𝜕𝑈
𝜕𝐿
𝑈new = 𝑈𝑜𝑙𝑑 − 𝜂
𝜕𝑈
𝜕𝐿
𝑉new = 𝑉𝑜𝑙𝑑 − 𝜂
𝜕𝑉
Long Short-Term Memory (LSTM):
• In the sentence “Deep learning is the subject which explains many concepts, and it is very interesting”,
the word “it” refers to “Deep learning”, which appears far earlier in the sequence.
A basic RNN may fail to capture this long-distance dependency due to vanishing gradients.
• Long Short-Term Memory (LSTM) is a type of recurrent neural network designed to handle long-term
dependencies in sequential data.
• It uses special units called gates to control the flow of information through the network. The forget, input,
and output gates decide what information to remove, update, and pass to the next step.

• LSTM maintains a cell state that acts as memory, allowing it to retain important information over long
sequences.

5
 • It is widely used in applications such as language modeling, speech recognition, and time-series
prediction.
• Forget Gate
The forget gate decides how much of the previous cell state should be retained or discarded. It uses a
sigmoid function to produce values between 0 and 1, where 0 means completely forget and 1 means fully
keep the information.
• Input Gate
The input gate determines how much new information should be added to the cell state. It works along
with a candidate memory to update the cell with relevant new information.
• Output Gate
The output gate decides what part of the cell state should be passed as the hidden state. It controls the
finaloutput of the LSTM at each time step using a sigmoid function and tanh activation.

ft=σ(xt Wf + ht−1 Uf +bf) it=σ(xt Wi + ht−1 Ui +bi)

Ot=σ(xt Wo + ht−1 Uo +bo)
Ct1=tanh(xt Wc + ht−1 Uc +bc)
•
• At each time step 𝑡, the LSTM takes the current input 𝑥𝑡 , the previous hidden state ℎ𝑡−1 , and the previous
cell state 𝐶𝑡−1as inputs.
• First, the forget gate computes 𝑓𝑡 = 𝜎(𝑊𝑓 𝑥𝑡 + 𝑈𝑓 ℎ𝑡−1 + 𝑏𝑓 )and decides how much of the previous cell
state 𝐶𝑡−1 should be retained or forgotten.
• Next, the input gate computes 𝑖𝑡 = 𝜎(𝑊𝑖 𝑥𝑡 + 𝑈𝑖 ℎ𝑡−1 + 𝑏𝑖 )to determine how much new information
should be added, and a candidate cell state 𝐶̃𝑡 = tanh (𝑊𝑐 𝑥𝑡 + 𝑈𝑐 ℎ𝑡−1 + 𝑏𝑐 )is generated.
• Then, the cell state is updated using 𝐶𝑡 = 𝑓𝑡 ⊙ 𝐶𝑡−1 + 𝑖𝑡 ⊙ 𝐶̃𝑡 , which combines the retained old
information with the new information.
• After that, the output gate computes 𝑜𝑡 = 𝜎(𝑊𝑜 𝑥𝑡 + 𝑈𝑜 ℎ𝑡−1 + 𝑏𝑜 )to decide which part of the cell state
should be output.
• Finally, the hidden state is computed as ℎ𝑡 = 𝑜𝑡 ⊙ tanh (𝐶𝑡 ), which serves as the output of the current
time step and is passed to the next time step.
Advantages of LSTM:
1. Captures long-term dependencies : LSTM can remember information over long sequences, solving the
vanishing gradient problem of traditional RNNs.
6
 2. Handles sequential data effectively: It works well for tasks like language modeling, speech
recognition, and time-series prediction.
Disadvantages of LSTM:
1. Computationally expensive: LSTM has complex structure with multiple gates, making it slower to
train compared to simple models.
2. Requires large data and tuning: It needs sufficient data and careful parameter tuning for good
performance.

Gated Recurrent Unit (GRU):

• GRU is a type of recurrent neural network designed to handle sequential data and capture dependencies over
time.
• It simplifies the LSTM architecture by using only two gates: the update gate and the reset gate.
• The update gate controls how much past information is carried forward, while the reset gate determines how
much past information to forget.
• Unlike LSTM, GRU does not have a separate cell state, making it computationally more efficient.
• GRUs are widely used in tasks such as language modeling, speech recognition, and time-series prediction.

• At each time step 𝑡, the GRU takes the current input 𝑥𝑡 and the previous hidden state ℎ𝑡−1 as inputs.
• First, the reset gate computes 𝑟𝑡 = 𝜎(𝑊𝑟 𝑥𝑡 + 𝑈𝑟 ℎ𝑡−1 + 𝑏𝑟 ), which determines how much of the previous
hidden state should be ignored while computing new information.
• Next, the update gate computes 𝑧𝑡 = 𝜎(𝑊𝑧 𝑥𝑡 + 𝑈𝑧 ℎ𝑡−1 + 𝑏𝑧 ), which controls how much of the previous
hidden state should be retained in the current step.
• Then, the candidate hidden state is computed as ℎ̃𝑡 = tanh (𝑊𝑥𝑡 + 𝑈(𝑟𝑡 ⊙ ℎ𝑡−1 ) + 𝑏)
• where the reset gate decides how much past information contributes to the new state.
• After that, the final hidden state is calculated using ℎ𝑡 = (1 − 𝑧𝑡 ) ⊙ ℎ𝑡−1 + 𝑧𝑡 ⊙ ℎ̃𝑡
which combines the previous hidden state and the candidate state based on the update gate.
• Finally, the hidden state ℎ𝑡 becomes the output of the current time step and is passed to the next time step.
Advantages of GRU:
1. Simpler and faster than LSTM: GRU has fewer gates and parameters, making it computationally
efficient and faster to train.
2. Captures dependencies effectively
It can model sequential data and capture long-term dependencies reasonably well.
Disadvantages of GRU
1. Less expressive than LSTM: Due to fewer gates, GRU may not capture very complex patterns
2. Less control over memory: GRU merges cell and hidden states, giving less fine-grained control
compared to LSTM.
7
 Aspect RNN LSTM GRU
Full Form Recurrent Neural Network Long Short-Term Memory Gated Recurrent Unit
Long-term memory Moderate memory
Memory Short-term memory
(cell state) (no separate cell state)
Gates No gates 3 gates (forget, input, output) 2 gates (reset, update)
Complexity Simple Complex Moderate
Training Speed Fast Slow Faster than LSTM
Vanishing
Major problem Solved effectively Reduced problem
Gradient
Parameters Few Many Fewer than LSTM
Performance Weak for long sequences Very good Good (close to LSTM)
Usage Basic sequence tasks Complex sequence tasks Efficient sequence tasks

Bidirectional RNN :
• Bidirectional RNN reads the sequence in both directions and combines past and future information to
make better predictions.
• Eg : In the sentence “He works at Apple”, the word “Apple” can mean a fruit or a company.
A Bidirectional RNN processes the sentence both forward (He → works → at → Apple) and backward
(Apple → at → works → He), using both past and future context to correctly identify “Apple” as an
organization.
• In a Bidirectional RNN, we use two separate RNNs.
➢ One RNN processes the sentence from left to right (forward direction).
➢ Another RNN processes the sentence from right to left (backward direction).

• The given diagram represents a Bidirectional Recurrent Neural Network (BiRNN), which processes a
sequence in both forward and backward directions.
• Input Sequence : The input sequence 𝑥1 , 𝑥2 , 𝑥3 , 𝑥4 is fed into the network. Each element of the sequence
is processed at different time steps.
• Forward RNN (Top Layer):
➢ The top row represents the forward RNN, which processes the sequence from left to right.
➢ At each time step, the hidden state is updated . This captures past context.
• Backward RNN (Bottom Layer):
➢ The bottom row represents the backward RNN, which processes the sequence from right to left.
➢ The hidden state is computed . This captures future context.
• Combination of Hidden States
➢ At each time step, the forward and backward hidden states are combined (usually concatenated):
➢ This provides complete contextual information.
• Output Generation : The combined hidden state is passed to the output layer . Thus, each output depends
on both past and future inputs.

8
Applications :
Model Application Description
Captures long-term dependencies to translate long
Language Translation
sentences effectively
Processes long audio sequences with better memory
Long Short-Term Speech Recognition
retention
Memory (LSTM)
Learns long-term patterns for accurate future
Time-Series Forecasting
predictions
Text Generation Generates coherent text by maintaining long context
Real-Time Speech Faster computation makes it suitable for real-time
Processing applications
Gated Recurrent
Chatbots Maintains conversational context with fewer parameters
Unit (GRU)
Sentiment Analysis Efficiently captures sequence information in text
Time-Series Prediction Provides faster training with good performance
Uses past and future context for accurate entity
Named Entity Recognition
detection
Part-of-Speech Tagging Improves tagging using bidirectional context
Bidirectional RNN Machine Translation
Captures full sentence context before decoding
(Encoder)
Uses both past and future audio context for better
Speech Recognition
accuracy

Challenges :

Challenge Description
Model
High Computational Cost Multiple gates increase complexity and training time
Long Short-Term More Parameters Large number of weights leads to higher memory usage
Memory (LSTM) Complex structure makes training slower compared to
Slow Training
simpler models
Fewer gates may reduce ability to capture very complex
Limited Expressiveness
patterns
Slightly less powerful than LSTM for very long
Gated Recurrent Performance Trade-off
dependencies
Unit (GRU)
Still Suffers from Gradient
Can face vanishing gradient in very long sequences
Issues
Requires future data, so cannot be used in online
Not Suitable for Real-Time
prediction
Bidirectional RNN High Computation Cost Two RNNs increase processing time
Stores forward and backward states, doubling memory
Increased Memory Usage
requirements