An Expert Talk on

Recurrent
Neural
Networks 1st and 2nd November, 2023

Mr. Tahseen Mulla

mullatahseen@[Link]
Takeaways
Pre-requisites
● Introduction to RNN • Machine Learning
● Working Principle • Basic Calculus
• Linear Algebra
● Back propagation through Time (BPTT) • Probability
● Vanishing and Exploding Gradients • Statistics
• Python Programming
● Long Short-Term Memory
● Gated Recurrent Units
● Bidirectional LSTM’s
● Bidirectional RNNs
● Speech Recognition and Natural Language Processing using RNN

Mr. Tahseen Mulla An Expert Talk on "Recurrent Neural Networks" 2

Introduction to RNN
Introduction to RNN
● Recurrent Neural Network is a type of neural network in which the output form of
the previous step is fed as input to the current step

● In traditional neural networks, all the inputs and outputs are independent of each
other, but this is not a good idea if we want to predict the next word in a sentence

● We need to remember the previous word in order to generate the next word in a
sentence, hence the traditional neural networks are not efficient for NLP
applications

Mr. Tahseen Mulla An Expert Talk on "Recurrent Neural Networks" 4

Introduction to RNN
● RNN also have a hidden stage which is used to capture information about a
sentence

● RNN have ‘memory’ which is used to capture information about the calculations
made so far

● RNNs can use information in arbitrary long sequences, but practically they are
limited to look back only a few steps

Mr. Tahseen Mulla An Expert Talk on "Recurrent Neural Networks" 5

Working Principle of
RNN
Working Principle of RNN

Unfold

Fig: Diagrammatic representation of Recurrent Neural Network (RNN)

Mr. Tahseen Mulla An Expert Talk on "Recurrent Neural Networks" 7

Working Principle of RNN
xt

ht f ht+1
g yt+3

Fig: RNN – Compressed Representation

Lets unfold the feedback loop in gray for k=3

xt wx xt+1 wx xt+2 wx wy
ht
wh wh wh
ht+1 ht+2 ht+3 by yt+3
bh bh bh
Recurrent Recurrent Recurrent Feedforward
Layer Layer Layer Layer

Fig: RNN – Unfolded Network

Mr. Tahseen Mulla An Expert Talk on "Recurrent Neural Networks" 8

Working Principle of RNN
● A network can be unfolded for ‘k’ time steps to get the output at time step k+1

● Using activation function ‘f’ –

ht+1 = f(xt, ht, wx, wh, bh) = f(wxxt + whht + bh)

● The output ‘y’ at time ‘t’ is computed as –

yt = f(ht, wt) = f(wy . ht + by)
dot product

Mr. Tahseen Mulla An Expert Talk on "Recurrent Neural Networks" 9

Few Activation Functions
● Activation functions used in Recurrent Neural Network –
𝟏 𝒆𝒙 − 𝒆−𝒙
Sigmoid Function Tanh Function
𝟏+𝒆−𝒙 𝒆𝒙 + 𝒆−𝒙

Relu Function max(0, x)

Mr. Tahseen Mulla An Expert Talk on "Recurrent Neural Networks" 10

RNN v/s Feed-Forward Neural Network
● Activation functions used in Recurrent Neural Network –

Fig: RNN Fig: Feed-Forward NN

Mr. Tahseen Mulla An Expert Talk on "Recurrent Neural Networks" 11

Steps to train an RNN
● Repeat till the stopping criteria is met –

○ Set all h to zero

○ Repeat for t=0 to n-k

■ Forward propagate the network over the unfolded network for ‘k’ time steps to compute all ‘h’
and ‘y’

■ Compute the error as: e = yt+k – pt+k

■ Back propagate the error across the unfolded network and update the weights

Mr. Tahseen Mulla An Expert Talk on "Recurrent Neural Networks" 12

Types of RNN
● There are different types of recurrent neural networks with varying architecture
One-to-One

Many-to-One Many-to-Many One-to-Many

Mr. Tahseen Mulla An Expert Talk on "Recurrent Neural Networks" 13

Advantages and Disadvantages of RNN
Advantages Disadvantages
● It has ability to handle sequence data ● The computation can be very slow

● It has ability to handle inputs of varying ● The network does not take into account
lengths future inputs to make decisions

● It has ability to store or ‘memorize’ ● Vanishing gradient problem, where the

historical information gradients used to compute the weight
update may get very close to zero,
preventing the network from leaving new
weights

Mr. Tahseen Mulla An Expert Talk on "Recurrent Neural Networks" 14

Back Propagation
Through Time
(BPTT)
Back Propagation Through Time (BPTT)
● The Backpropagation Through Time technique applies the Backpropagation
training method to the recurrent neural network in a deep learning model trained
on sequence data

● As RNN neural network processes sequence one step at a time, gradients flow
backwards across time steps during the backpropagation process

Mr. Tahseen Mulla An Expert Talk on "Recurrent Neural Networks" 16

Back Propagation Through Time (BPTT)
● A single input is sent into the network at a time in a normal RNN and a single
output is obtained

● Backpropagation uses both the current and prior inputs as input

● This concept is referred as “timestep”, and one timestep will consist of multiple
time series data points entering the RNN at the same time

Mr. Tahseen Mulla An Expert Talk on "Recurrent Neural Networks" 17

Back Propagation Through Time (BPTT)

Fig: Back Propagation Through Time (BPTT)

Mr. Tahseen Mulla An Expert Talk on "Recurrent Neural Networks" 18

Back Propagation in RNN
● To train an RNN, we need a loss function. Lets use cross-entropy loss paired with softmax –

L = -ln(pc)

● Now, suppose if a positive text is predicted to be 95% positive by the RNN, then the loss is –

L = -ln(0.95)
= 0.051
● After calculating the loss, train the RNN using gradient descent to minimize the loss

Mr. Tahseen Mulla An Expert Talk on "Recurrent Neural Networks" 19

Steps for Back Propagation in RNN
● Computer the cross-entropy error first using the current and actual output

● For each timestep in the network, the gradient descent is calculated with respect to the
weight of each of the parameter

● Once the weight for all time step is the same, combine together the gradients for all the
timesteps

● Then update the weights for both the recurrent neurons as well as the dense layers

Mr. Tahseen Mulla An Expert Talk on "Recurrent Neural Networks" 20

Vanishing and
Exploding Gradients
Vanishing and Exploding Gradients
● There are two problems that RNNs have had to overcome –

○ Vanishing Gradients and

○ Exploding Gradients

Mr. Tahseen Mulla An Expert Talk on "Recurrent Neural Networks" 22

Vanishing and Exploding Gradients

Mr. Tahseen Mulla An Expert Talk on "Recurrent Neural Networks" 23

Vanishing and Exploding Gradients
Vanishing Gradients Exploding Gradients

● Vanishing gradients occur when the ● Vanishing gradients occur when the
gradient values are too small, causing gradient gives the weights as absurdly
the model to stop learning or take far too high priority for no apparent reason
long

● LSTM concept helped to solve this ● Truncating or Squashing the gradients is

problem (It was a big issue in the 1990s) a simple solution to this problem

Mr. Tahseen Mulla An Expert Talk on "Recurrent Neural Networks" 24

Long Short-Term
Memory
Long Short-Term Memory
● Long Short-Term Memory is an improved version of the regular RNN which has designed to
make it easy to capture long-term dependencies in sequence data

● To understand LSTM we need to have a familiar knowledge of gates and cell state

Why do we need LSTM if we have RNN?

LSTM can be used to solve problems faced by RNN model
(Vanishing Gradient and Exploding Gradient and Long Term Dependency Problem)

To handle the Vanishing Gradient problem as well by using the architectures like LSTM and
the GRU (Gated Recurrent Units)

Mr. Tahseen Mulla An Expert Talk on "Recurrent Neural Networks" 26

Long Short-Term Memory

Fig: The network architecture of LSTM

Mr. Tahseen Mulla An Expert Talk on "Recurrent Neural Networks" 27

Actual crux behind LSTM
● The key to LSTMs is the cell state
The black line represents cell state

● The cell state is kind of like a conveyer belt. It

runs straight down the entire chain, with only
some minor linear interactions

● Its very easy for information to just flow along Fig: The crux of LSTM
with it unchanged

Mr. Tahseen Mulla An Expert Talk on "Recurrent Neural Networks" 28

Actual crux behind LSTM
● The LSTM does have the ability to remove or add
information to the cell state, carefully regulated
by structures called gates

● Gates are a way to optionally let information

through

● They are composed out of a sigmoid neural net

layer and a pointwise multiplication operation

A value of zero means “let nothing through”, while a

Fig: The LSTM Gates
value of one means “let everything through”

Mr. Tahseen Mulla An Expert Talk on "Recurrent Neural Networks" 29

Long Short-Term Memory

Mr. Tahseen Mulla An Expert Talk on "Recurrent Neural Networks" 30

Long Short-Term Memory
● For Example – The cell state may remember the gender f the subject in given input
sequence so that the proper pronoun or the verb can be used (w.r.t NLP)

● Consider an example –

○ The cat which already ate _______________________ was full

○ The cats which already ate _______________________ were full

The underline represents the presence of a long sentence

but the subject has not changed yet

Mr. Tahseen Mulla An Expert Talk on "Recurrent Neural Networks" 31

Long Short-Term Memory
● The equations for the gates in LSTM are –

it = σ ( wi [ht-1, xt] + bi )

ft = σ (wf [ht-1, xt] + bf )

ot = σ (wo [ht-1, xt] + bo )

Where, it = represents input gate

ft = represents forget gate
ot = represents output gate
σ = represents sigmoid function

Mr. Tahseen Mulla An Expert Talk on "Recurrent Neural Networks" 32

Long Short-Term Memory
● The architecture of Long Short-Term Memory has three gates, to protect and control the cell
state –

○ Forget Gate

○ Input Gate

○ Output Gate

Gates in LSTM are nothing but the sigmoid activation functions, they output a value
between 0 or 1 and either 0 or 1

Mr. Tahseen Mulla An Expert Talk on "Recurrent Neural Networks" 33

Long Short-Term Memory Cell

Fig: The LSTM Cell

Mr. Tahseen Mulla An Expert Talk on "Recurrent Neural Networks" 34

Long Short-Term Memory Cell
● Forget Gate -

○ It decides how much of the past information the LSTM should remember

○ This decision is taken by the sigmoid activation function

○ It looks at the previous state and current input and outputs a number between 0 and 1
for each number in the cell state

ft = σ (wf [ht-1, xt] + bf )

Mr. Tahseen Mulla An Expert Talk on "Recurrent Neural Networks" 35

Long Short-Term Memory Cell
● Input Gate -

○ It decides how much of this unit is added to the current state

○ The input gate is responsible for the addition of information to the cell store. This
addition of information is a three-step process –
■ Regulating what values need to be added to the cell state by involving a sigmoid
function
■ Creating a vector containing all possible values that can be added to the cells
state
■ Multiplying the value of the regulatory filter and then adding this useful
information to the cell state via addition operation

it = σ ( wi [ht-1, xt] + bi )

Mr. Tahseen Mulla An Expert Talk on "Recurrent Neural Networks" 36

Long Short-Term Memory Cell
● Output Gate –

○ It decides which part of the current cell makes it to the output

○ It can be achieved into following three steps -
■ Creating a vector applying the tanh function to the cell state, thereby scaling the
values to the range -1 to _1
■ Making the filter using the values, so that it can regulate the values that need to
be output from the vector created above
■ Multiplying the value of this regulatory filter to the vector created in first step and
sending it as output and also to the hidden state of the next cell

ot = σ (wo [ht-1, xt] + bo )

ht = ot * tanh ( Ct )

Mr. Tahseen Mulla An Expert Talk on "Recurrent Neural Networks" 37

Gated Recurrent Unit
(GRU)
Gated Recurrent Unit (GRU)
● The Gated Recurrent Unit (GRU) solves the vanishing gradient problem of simple RNNs

● GRU uses fewer gates and do not separate internal memory, the cell state

Fig: Unfolded Gated Recurrent Unit (GRU)

Mr. Tahseen Mulla An Expert Talk on "Recurrent Neural Networks" 39

Gated Recurrent Unit (GRU)
● The reset gate is responsible for the short-term memory as it decides how past information
is kept and disregarded

● The value in the vector ‘r’ are bounded between 0 and 1 by a sigmoid function

● The update gate is responsible for the long-term memory and is comparable to the LSTMs
forget gate

Mr. Tahseen Mulla An Expert Talk on "Recurrent Neural Networks" 40

Gated Recurrent Unit (GRU)
● The hidden state of the current timestep is determined based on a two-step process –

○ A candidate hidden state is determined

○ The candidate hidden state is combined with the hidden state of the previous timestep
to generate the current hidden state

Mr. Tahseen Mulla An Expert Talk on "Recurrent Neural Networks" 41

Advantages and Disadvantages of GRU
Advantages Disadvantages

● Simple in architecture compared to ● GRUs do not have a separate hidden and

LSTM cell state as they might not be able to
consider observations as far into the
● Two gates are used instead of three past as LSTM
gates and one state is used instead of
two states ● GRU might also suffer from the
disadvantages of the backpropagation in
● GRUs are more efficient and faster to time to update the weighs (exploding
train as they need less memory gradients)

Mr. Tahseen Mulla An Expert Talk on "Recurrent Neural Networks" 42

Bidirectional LSTM
Bidirectional LSTM
● A bidirectional LSTM (or biLSTM) is a sequence processing model that consists of two
LSTMs (i) one taking the input in a forward direction and (ii) the other in backward direction

● biLSTMs effectively increase the amount of information available to the network, improving
the context available to the algorithm

● In bidirectional, our input flows in two directions, making biLSTMs different from the regular
LSTM.

● biLSTMs are capable of making any neural network to have the sequence information in
both directions backwards or forward

Mr. Tahseen Mulla An Expert Talk on "Recurrent Neural Networks" 44

Bidirectional LSTM

Fig: Bidirectional LSTMs (biLSTMs)

Mr. Tahseen Mulla An Expert Talk on "Recurrent Neural Networks" 45

Bidirectional RNNs
Bidirectional RNNs
● A bidirectional RNN (biRNN) is a type of RNN that processes input data in both forward and
backward directions

● The goal here is to capture the contextual dependencies in the input data by processing it in
both directions

● The network has two separate RNNs –

○ One that processes the input sequence from left to right

○ Another one that processes the input sequence from right to left

Mr. Tahseen Mulla An Expert Talk on "Recurrent Neural Networks" 47

Bidirectional RNNs

Fig: Bidirectional RNNs (biRNNs)

Mr. Tahseen Mulla An Expert Talk on "Recurrent Neural Networks" 48

Speech Recognition and
Natural Language
Processing using RNN
Speech Recognition and
Natural Language
Processing using RNN
Let’s check the Python Example
Let’s Summarize
● Introduction to RNN
● Working Principle
● Back propagation through Time (BPTT)
● Vanishing and Exploding Gradients
● Long Short-Term Memory
● Gated Recurrent Units
● Bidirectional LSTM’s
● Bidirectional RNNs
● Speech Recognition and Natural Language Processing using RNN

Mr. Tahseen Mulla An Expert Talk on "Recurrent Neural Networks" 51