DEEP LEARNING

CSA3007
Module 5
 RECURRENT AND RECURSIVE NETWORKS:

Dr. Praveen Kumar Tyagi

Unit V -
Computational Graphs:

• A computational graph is a graph representation of mathematical operations,

where:
• Nodes → operations (addition, multiplication, activation)

• Edges → data (inputs/outputs).

• Purpose in Deep Learning

• Represents neural network computations

• Enables automatic differentiation

• Supports backpropagation
Computational Graphs:

Forward and Backward Pass

• Forward Pass

• Compute outputs step-by-step

• Backward Pass

• Apply chain rule:

Computational graphs allow efficient

gradient computation in deep networks.
Recurrent Neural Networks (RNN):
Introduction to Recurrent Neural Networks
• Recurrent Neural Networks (RNNs) are a class of neural networks designed to

process Sequential data where order and temporal dependency are important.

• RNNs are designed for sequential data.

where: 𝑥𝑡 : input ℎ𝑡 : hidden state (memory)

Unlike Feedforward Neural Networks, RNNs have feedback connections.
• Designed for sequential and temporal data
• Maintains memory of past inputs
• Widely used in NLP, forecasting and speech tasks
Recurrent Neural Networks (RNN):
Key Features
• Memory of previous inputs
• Shared weights across time
• Suitable for sequences
Many real-world problems are sequence-dependent:

Data Type Example

Time Series ECG, EEG, Stock prices
Text Sentences, documents
Speech Audio signals
Video Frame sequences
Recurrent Neural Networks (RNN):
 Basic Structure

An RNN consists of:

• Input layer
• Hidden layer (recurrent connection)
• Output layer

 Key Characteristics

• Hidden state carries past information

• Same weights used across all time steps
• Suitable for sequence modeling
Recurrent Neural Networks (RNN):
Traditional CNN/ANN:
❌ Cannot remember past inputs
RNN:
✔ Remembers previous information (Hidden state acts as memory)
• At time step 𝑡:

ℎ𝑡 = 𝑓 𝑊𝑥ℎ 𝑋𝑡 + 𝑊ℎℎ 𝑋𝑡−1 + 𝑏

𝑦𝑡 = 𝑔(𝑊ℎ𝑦 ℎ𝑡 )
Where:
𝑋𝑡 = input at time t 𝑦𝑡 = output Basic RNN Architecture
ℎ𝑡 ​ = hidden state (memory) 𝑓 ∙ and 𝑔 ∙ = activation
Recurrent Neural Networks (RNN):
Unfolding of RNN:
• Unfolding means expanding the RNN across time steps to understand how information
flows.

• Makes time-dependency explicit

• Helps understand training Each unfolded block shares same weights.
• Required for gradient computation
Recurrent Neural Networks (RNN):
Working of RNN:
Step-by-Step Process
1. Input at time t
• Receive input 𝑥𝑡
2. Combine with previous state
ℎ𝑡 = 𝑓 𝑊𝑥ℎ 𝑋𝑡 + 𝑊ℎℎ 𝑋𝑡−1 + 𝑏
3. Update hidden state
• Stores memory
4. Generate output
𝑦𝑡 = 𝑔(𝑊ℎ𝑦 ℎ𝑡 )
5. Repeat for sequence
Recurrent Neural Networks (RNN):
Training Process
• Uses Backpropagation Through Time (BPTT)
• Computes gradients across all time steps
Backpropagation Through Time (BPTT):
• Backpropagation Through Time is the training algorithm for RNNs.
• BPTT is an extension of backpropagation used to train RNNs on sequential data.
It: Unfolds the RNN over time
• Computes loss at each time step
• Backpropagates errors backward through time
• Updates shared weights
Recurrent Neural Networks (RNN):
Forward Pass
• The network processes each input in the sequence one at a time.
• Hidden states carry information forward through time.
Compute Loss:
• Loss is usually defined as the sum (or average) of the errors across all time steps in
the sequence.
Backward Pass
• During the backward pass, gradients are propagated from the last time step to the
first, and gradients for shared weights are accumulated across all time steps.
• Error gradients flow backward through the unrolled network by applying the chain
rule across time steps..
Recurrent Neural Networks (RNN):

Advantages of RNN
 Captures Sequential Dependency
• Uses previous information
• Suitable for time-series and text
 Parameter Sharing
• Same weights reused
• Reduces number of parameters
 Flexible Input Length
• Can process sequences of varying lengths
Recurrent Neural Networks (RNN):

 Suitable for Many Applications

• Natural Language Processing (NLP)
• Speech recognition
• Time-series forecasting
• Video analysis.

Limitations of RNN
 Vanishing Gradient Problem
• Gradients become very small
• Early layers stop learning
Recurrent Neural Networks (RNN):
 Exploding Gradient Problem
• Gradients become too large
• Training becomes unstable
 Difficulty in Long-Term Dependencies
• Cannot remember long sequences effectively
 Slow Training
• Sequential processing
• Cannot be fully parallelized
 Limited Memory Capacity
• Hidden state cannot store all past information
Recurrent Neural Networks (RNN):
 Solutions to Limitations
• LSTM (Long Short-Term Memory)
• GRU (Gated Recurrent Unit)
• Attention Mechanisms
 Applications of RNN
• Language modeling
• Machine translation
• Speech recognition
• Sentiment analysis
• Stock prediction
Bidirectional RNN:

• Recurrent Neural Networks (RNNs) are designed to process sequential data by

using past information.

• However, standard RNNs only use: Past (previous) context → not future context

• In many real-world tasks, both past and future information are important.

• To overcome this limitation, Bidirectional RNNs (BiRNNs) are used.

• BiRNN is an extension of standard RNN that processes a sequence in both

forward and backward directions to capture past and future context.

• This allows the model to have complete, sequential information about all points

before and after in the sequence, which is useful in tasks like speech recognition
Bidirectional RNN :
 Concept of Bidirectional RNN
• A Bidirectional RNN processes the input sequence in both forward and backward
directions.
• Forward RNN → processes sequence left → right
• Backward RNN → processes sequence right → left
• Final output is a combination of both
• Thus, each output depends on: Past context and Future context.
Advantages
• Better context understanding
• Improved accuracy
Bidirectional RNN :

Bidirectional RNN
Bidirectional RNN :

Mathematical Representation
• Forward RNN

• Backward RNN

• Final output:

ℎ𝑡 = ℎ𝑡 , ℎ𝑡
Bidirectional RNN :
 Key Idea
At each time step:
• Combine forward hidden state
• Combine backward hidden state
• Output depends on entire sequence context

 Working of BiRNN
Step-by-Step Process
1. Input sequence:
𝑥1, 𝑥2, 𝑥3, … , 𝑥𝑇
Bidirectional RNN :
2. Forward pass:
• Process sequence from left to right
• Compute ℎ𝑡
3. ​Backward pass:
• Process sequence from right to left
• Compute ℎ𝑡
4. Combine outputs

ℎ𝑡 = ℎ𝑡 , ℎ𝑡
5. Generate final output:
𝑦𝑡 = 𝑔(ℎ𝑡 )
Bidirectional RNN :
 How BiRNN Improves Standard RNN

Comparison Table

Feature Standard RNN BiRNN

Context used Past only Past + Future

Information flow One direction Two directions

Accuracy Moderate Higher

Context understanding Limited Complete

Bidirectional RNN :

Key Improvements
 Better Context Understanding
• Uses full sequence information
• Important for ambiguous data
 Improved Prediction Accuracy
• More information → better decisions
 Handles Ambiguity
• Example:
• Word meaning depends on both previous and next words
Bidirectional RNN :

Example 1: Natural Language Processing

• Sentence:
“He went to the bank”
• “bank” can mean:
• river bank
• financial bank
• Future words help clarify meaning
BiRNN :
• Sees future words like “to deposit money”
Bidirectional RNN :
Example 2: POS Tagging
• Sentence:
“She is running fast”
• Forward context → “is”
• context → “running fast”
• Better tagging of “running” (verb vs noun)
Example 3: Speech Recognition
• Sound interpretation depends on:
• previous phonemes
• upcoming phonemes
Bidirectional RNN :
Applications of BiRNN
• Machine translation
• Speech recognition
• Text classification
• Sentiment analysis
• Biomedical signal processing
Bidirectional RNN :

Limitations of BiRNN
 Higher Computational Cost
• Two RNNs instead of one
 Cannot Be Used in Real-Time Prediction
• Requires full sequence
• Future data must be available
 Increased Complexity
• More parameters
• More training time
Case Studies in Classification:

Example:
 Image Classification

• Input: image

• CNN extracts features

• Fully connected layer → classification

 Text Classification

• RNN / LSTM processes text

• Output → sentiment / category

Case Studies in Regression:

Example:
 House Price Prediction
• Input: features

• Output: continuous value

 Time-Series Forecasting

• Stock prediction

• Weather forecasting
Classification vs Regression:

Classification vs Regression Loss Functions:

• In deep learning, a loss function measures how well a model’s predictions match

the true values.

• For classification problems → output is discrete (class labels)

• For regression problems → output is continuous (real values)

Therefore, different loss functions are used for each.

Classification vs Regression:

 Classification Loss Functions:

• Measure how well predicted probabilities match the true class labels.

Common Loss Functions:

(a) Binary Cross-Entropy (BCE)

• Used for binary classification:

Where:
𝑦= true label (0 or 1)
𝑝 = predicted probability
Classification vs Regression:

(b) Categorical Cross-Entropy

Used for multi-class classification:

 Key Characteristics

• Output is probability (0–1)

• Uses Softmax / Sigmoid activation

• Penalizes wrong class predictions heavily

 Example: Spam detection, Image classification, and Disease classification

Classification vs Regression:

 Regression Loss Functions:

• Measure difference between predicted value and actual value.

Common Loss Functions:

(a) Mean Squared Error (MSE)

Penalizes large errors heavily.

(b) Mean Absolute Error (MAE)

More robust to outliers

Classification vs Regression:

Key Characteristics
• Output is continuous value

• Measures distance between predicted and actual values

• No probability interpretation

 Example:

• House price prediction

• Temperature prediction

• Stock forecasting
Classification vs Regression:

Feature Classification Loss Regression Loss

Output type Discrete classes Continuous values

Goal Correct class prediction Minimize error magnitude

Output range 0 to 1 (probability) Any real number

Common losses Cross-entropy MSE, MAE

Activation Sigmoid / Softmax Linear

Error measurement Probability error Numerical difference

Example Cat vs Dog Price prediction

Deep Networks in Practice:

Types of Deep Models

• CNN → images

• RNN → sequences

• Transformer → NLP

• Autoencoder → feature learning

Training Pipeline
• Data preprocessing • Evaluation

• Model selection • Deployment

• Training (optimization)