Dr.

Waqas Ahmed
Air University Islamabad

Neural Networks
Deep Learning
Reinforcement learning
Lecture 6: Recurrent Neural Networks

Dr. Waqas Ahmed

 Dr. Waqas Ahmed
Air University Islamabad

Sequence Models
• Both input and output or one of these is a sequence
 Dr. Waqas Ahmed
Air University Islamabad

Artificial Neural Networks

• Outputs are not dependent

 Dr. Waqas Ahmed
Air University Islamabad

Dependencies in Biological Neurons

• Humans don’t start their thinking from scratch

every second.
• As we read this text, we understand each word
based on the understanding of previous words.
• We don’t throw everything away and start thinking
from scratch again.
• Our thoughts have persistence.
 Dr. Waqas Ahmed
Air University Islamabad

Book Reading - ANN

• Predicting the next word?

 Dr. Waqas Ahmed
Air University Islamabad

RNNs - Applications
• Sentiment Classification
– Input: Sequence of words (varying length)
– Output: Class Label
 Dr. Waqas Ahmed
Air University Islamabad

RNNs - Applications
• Image Captioning
– Input: Image
– Output: Textual description (varying length)
 Dr. Waqas Ahmed
Air University Islamabad

RNNs - Applications
• Language Translation
– Input: Sequence of words
– Output: Sequence of words
– Lengths of input and output
may vary
 Dr. Waqas Ahmed
Air University Islamabad

Why not ANNs?

• ANNs can’t deal with sequential or temporal data
• ANNs lack memory
• ANNs have a fixed architecture
• RNNs are more “biologically realistic” because of
the recurrent connectivity found in the visual
cortex of the brain
 Dr. Waqas Ahmed
Air University Islamabad

Why not ANNs?

• Image Captioning: Output is a sequence of
words of varying length
• Sentiment Classification: Input is a sequence
of words, if fed one by one, output is only
dependent on current input, no relation with
previous words
 Dr. Waqas Ahmed
Air University Islamabad

Recurrent Neural Network

 Dr. Waqas Ahmed
Air University Islamabad

How to Build a Neural Language

Model?
• Recall the Language Modeling task:
• Input: sequence of words 𝒙𝒙(𝟏𝟏), 𝒙𝒙(𝟐𝟐), … , 𝒙𝒙(𝒕𝒕)
• Output: probability distribution of the next word 𝑷𝑷 𝒙𝒙 𝒕𝒕+𝟏𝟏 𝒙𝒙 𝒕𝒕 , … , 𝒙𝒙 𝟏𝟏
• How about a window-based neural model?
Example: NERTask LOCATION

museums in Paris are amazing

Tanmoy Chakraborty
 Dr. Waqas Ahmed
Air University Islamabad

RNNs Can Be Used for Sequence Tagging

• Example: for part-of-speechtagging, named entity recognition

DT JJ NN VBN IN DT NN

the startled cat knocked over the vase

 Dr. Waqas Ahmed
Air University Islamabad

RNNs Can Be Used for Sentence Classification

• Example: for sentimentclassification positive
How to compute
sentence encoding?
Sentence
encoding Usually better:
Take element-wise
max or mean ofall
hidden states

overall I enjoyed the movie a lot

 Dr. Waqas Ahmed
Air University Islamabad

A Fixed-window Neural
Language Model books
laptops

output distribution

a zoo
hidden layer

concatenated word embeddings

words / one-hot vectors

the students opened their

fixed window

Tanmoy Chakraborty
 Dr. Waqas Ahmed
Air University Islamabad

A Fixed-window Neural
Language Model
books Approximately:[Link], et al.
(2000/2003): A Neural Probabilistic
Improvements over n-gram LM: laptops
Language Model
• No sparsity problem
• Don’t need to store all observed n-
grams a zoo We need a neural
architecture that can
process anylength
Remainingproblems:
input
• Fixed window is too small
• Enlarging window enlarges 𝑊𝑊
• 𝑥𝑥 (1) and 𝑥𝑥 (2) are multiplied by
completely different weights in 𝑊𝑊.
No symmetry in how the inputs are opened
the students their
processed.

Tanmoy Chakraborty
 Dr. Waqas Ahmed
Air University Islamabad

Recurrent Neural
Networks (RNN)
outputs
(optional)
Core idea: Apply the same
weights 𝑊𝑊 repeatedly

hidden states

input sequence
(any length)

Tanmoy Chakraborty
 Dr. Waqas Ahmed
Air University Islamabad

A Simple RNN Language books

laptops

Model
output distribution a zoo

hidden states

is the initial hidden state

word embeddings

words / one-hot vectors

the students opened their
Note: this input sequence could be much longer now!

Tanmoy Chakraborty
 Dr. Waqas Ahmed
Air University Islamabad

RNN Language books

laptops

Models
a zoo
RNN Advantages:
• Can process any length input
• Computation for step t can (in
theory) use information from many steps
back
• Model size doesn’t increase for longer input
context
• Same weights applied on every
timestep, so there is symmetry in how inputs
are processed.

RNN Disadvantages:
• Recurrent computation is slow
• In practice, difficult to access information the students opened their
from many steps back

Tanmoy Chakraborty
 Dr. Waqas Ahmed
Air University Islamabad

Training the RNNs

 Dr. Waqas Ahmed
Air University Islamabad

= negative log prob

of “students”

Loss

Predicted probability
distributions

Corpus the students opened their exams …

 Dr. Waqas Ahmed
Air University Islamabad

= negative log prob

of “opened”
Loss

Predicted probability
distributions

Corpus the students opened their exams …

 Dr. Waqas Ahmed
Air University Islamabad

= negative log prob

of “their”

Loss

Predicted probability
distributions

Corpus the students opened their exams …

 Dr. Waqas Ahmed
Air University Islamabad

= negative log prob

of “exams”
Loss

Predicted probability
distributions

Corpus the students opened their exams …

 Dr. Waqas Ahmed
Air University Islamabad
“Teacher forcing”

Loss + + + +… =

Predicted probability
distributions

Corpus the students opened their exams …

 Dr. Waqas Ahmed
Air University Islamabad

Training a RNN Language Model

• However: Computing loss and gradients across entire corpus 𝑥𝑥(1), 𝑥𝑥(2), … , 𝑥𝑥(𝑇𝑇) at once is
too expensive (memory-wise)!

• In practice, consider 𝑥𝑥(1), 𝑥𝑥(2), … , 𝑥𝑥(𝑇𝑇) as a sentence (or a document)

• Recall: Stochastic Gradient Descent allows us to compute loss and gradients for small
chunk of data, and update.
• Compute loss 𝐽𝐽(𝜃𝜃) for a sentence (actually, a batch of sentences), compute
gradients and update weights. Repeat on a new batch of sentences.

Tanmoy Chakraborty
 Dr. Waqas Ahmed
Air University Islamabad

Multivariable Chain Rule

Source:
[Link]
derivatives/differentiating-vector-valued-functions/a/multivariable-chain-rule-
simple-version

Tanmoy Chakraborty
 Dr. Waqas Ahmed
Air University Islamabad

In practice, often
… “truncated”
… after ~20
timesteps fortraining
efficiency reasons

Apply the multivariable chain rule:

Question: How do we calculate this? =1
Answer:Backpropagate over timesteps
i = t, … ,0, summing gradients as you go.
This algorithm is called “backpropagation
through time”

[Werbos, P.G.,1988, Neural Networks 1, and others]

 Dr. Waqas Ahmed
Air University Islamabad

Problems with RNNs

 Dr. Waqas Ahmed
Air University Islamabad

Vanishing and Exploding Gradients

 Dr. Waqas Ahmed
Air University Islamabad

Vanishing Gradient Intuition

 Dr. Waqas Ahmed
Air University Islamabad

Vanishing Gradient
Intuition

Chain Rule!
 Dr. Waqas Ahmed
Air University Islamabad

Vanishing Gradient
Intuition

Chain Rule!
 Dr. Waqas Ahmed
Air University Islamabad

Vanishing Gradient
Intuition

Chain Rule!
 Dr. Waqas Ahmed
Air University Islamabad

Vanishing Gradient
Intuition
Vanishing gradient
problem: When these
are small, the gradient
signal gets smallerand
smaller as it
backpropagates further

What happens if these are small?

 Dr. Waqas Ahmed
Air University Islamabad

Why is Vanishing Gradienta

Problem?
 Dr. Waqas Ahmed
Air University Islamabad

Effect of Vanishing Gradient on RNN-LM

• LM task: When she tried to print her tickets, she found that the printer was out of toner.
She went to the stationery store to buy more toner. It was very overpriced. After installing
the toner into the printer, she finally printed her

• To learn from this training example, the RNN-LM needs to model the dependency
between “tickets” on the 7th step and the target word “tickets” at the end.

• But if the gradient is small, the model can’t learn this dependency
• So, the model is unable to predict similar long-distance dependencies at test time
 Dr. Waqas Ahmed
Air University Islamabad

How to Fix the Vanishing Gradient Problem?

• The main problem is that it’s too difficult for the RNN to learn to preserve information over
many timesteps.
• In a vanilla RNN, the hidden state is constantly being rewritten

• How about an RNN with separate memory which is added to?

• LSTMs
• And then: Creating more direct and linear pass-through connections in model
• Attention, residual connections,etc.
 Dr. Waqas Ahmed
Air University Islamabad

LSTMs &GRUs
 Dr. Waqas Ahmed
Air University Islamabad

Long Short-Term Memory (LSTM)

• A type of RNN proposed by Hochreiter and Schmidhuber in 1997 as a solution to the vanishing gradients
problem.
• On step t, there is a hidden state ℎ(𝑡𝑡) and a cell state 𝑐𝑐(𝑡𝑡)
• Both are vectors lengthn
• The cell stores long-terminformation
• The LSTM can erase, write and read information from the cell
• The selection of which information is erased/written/read is controlled by three corresponding gates (gates
are calculated things whose values are probabilities)
• The gates are also vectors length n
• On each timestep, each element of the gates can be open (1), closed (0), or
somewhere in-between.
• The gates are dynamic: their valueis computed based on the current context

Tanmoy
Chakraborty
 Dr. Waqas Ahmed
Air University Islamabad

LSTM
We have a sequence of inputs 𝑥𝑥(𝑡𝑡), and we will compute a sequence of hidden states ℎ(𝑡𝑡) and cell states
𝑐𝑐(𝑡𝑡). On timestep t:
Sigmoid function: all gate
Forget gate: controls what is kept vs values are between 0 and 1
forgotten, from previous cell state

Input gate: controls what partsof the

All these are vectors of same length n

new cell content are written tocell

Output gate: controls what partsof

cell are output to hiddenstate

New cell content: this is the new

content to be written to the cell

Cell state: erase (“forget”) some

content from last cellstate, and write
(“input”) some new cell content
⊙ ⊙

Hidden state: read (“output”) some ⊙

content from the cell
Gates are applied usingelement-wise
(or Hadamard) product: ⊙
 Dr. Waqas Ahmed
Air University Islamabad

Long Short-Term Memory (LSTM)

You can think of the LSTM equations visually like this:

Source: [Link]
 Dr. Waqas Ahmed
Air University Islamabad

How Does LSTM Solve Vanishing Gradients?

• The LSTM architecture makes it much easier for an RNN to preserveinformation over
many timesteps
• For example, if the forget gate is set to 1 for a cell dimension and the input gate set to 0, then the
information of that cell is preserved indefinitely.
• In contrast, it’s harder for a vanilla RNN to learn a recurrent weight matrix Wh that preserves info in the
hidden state
• In practice, you get about 100 timesteps rather than about 7
• LSTM doesn’t guarantee that there is no vanishing/exploding gradient, but it does provide
an easier way for the model to learn long-distance dependencies.
• There are also alternative ways of creating more direct and linear pass-through
connections in models for long distance dependencies.

Tanmoy
Chakraborty
 Is Vanishing/Exploding Gradient Just an RNN
Dr. Waqas Ahmed
Air University Islamabad

Problem?
• No! It can be a problem for all neural architectures (including feed-
forward and convolutional neural networks), especially very deep
ones.
• Due to chain rule / choice of nonlinearity function, gradient can become vanishingly small as it backpropagates
• Thus, lower layers are learned very slowly (i.e., are hard to train)

• Another solution: lots of new deep feedforward/convolutional

architectures add more direct connections
(thus allowing the gradient toflow)
For example:
• Residual connections aka“ResNet”
• Also known as skip-connections
• The identity connection preserves information by
default
• This makes deep networks much easier to train
"Deep ResidualLearning for Image Recognition", He et al, 2015. [Link]
 Dr. Waqas Ahmed

Is Vanishing/Exploding Gradient Just an RNN

Air University Islamabad

Problem?
Other Methods: • Highway connections aka“HighwayNet”
• Dense connections aka“DenseNet” • Similar to residual connections, but the identity
• Directly connect each layer to all future layers! connection vs the transformation layer is controlled by a
dynamic gate
• Inspired by LSTMs, but appliedto deep
feedforward/convolutional networks

Conclusion: Though vanishing/exploding gradients are a general problem, RNNs are particularly unstable
due to the repeated multiplication by the same weight matrix.
 Dr. Waqas Ahmed
Air University Islamabad

Gated Recurrent Units

(GRUs) How does this solve vanishing gradient?
Like LSTM, GRU makes it easier to retain info
long-term (e.g. by setting update gate to 0)
• Proposed by Cho et al. in Update gate: controls what parts of
2014 as a simpler hidden state are updated vs preserved
alternative to the LSTM.
Reset gate: controls what parts of
• On each timestep t, we
previous hidden state are used to
have input 𝑥𝑥(𝑡𝑡) and hidden
compute new content
state ℎ(𝑡𝑡) (no cell state).
New hidden state content: reset gate
selects useful parts of prev hidden
state. Use this and current input to
compute new hidden content.

Hidden state: update gate

simultaneously controls what is kept
from previous hidden state, and what is
"Learning Phrase Representations using RNN Encoder– updated to new hidden state content
Decoder for Statistical Machine Translation", Cho et al.
2014, [Link]
 Dr. Waqas Ahmed
Air University Islamabad

LSTM vs
GRU
• Researchers have proposed many gated RNN variants, but LSTM and GRUare the most
widely-used.
• The biggest difference is that GRU is quicker to compute and has fewer parameters.
• There is no conclusive evidence that one consistently performs better than theother.
• LSTM is a good default choice (especially if your data has particularly long dependencies,
or you have lots of trainingdata).
• Rule of thumb: start with LSTM, but switch to GRU if you want something more efficient.
 Dr. Waqas Ahmed
Air University Islamabad

Bidirectional
RNNs Forward RNN Generally, these
two RNNs have
On timestep t : Backward RNN separate weights
Concatenated
hidden states
 Dr. Waqas Ahmed
Air University Islamabad

Multi-layer RNNs

• RNNs are already “deep” on one dimension (they unroll over many timesteps)

• We can also make them “deep” in another dimension by applying multiple RNNs – this is
a multi-layer RNN.

• This allows the network to compute more complex representations

• The lower RNNs should compute lower-level features and the higher RNNs should compute higher-
level features.

• Multi-layer RNNs are also called stacked RNNs.

 Dr. Waqas Ahmed
Air University Islamabad
The hidden states from RNN layer i
are the inputs to RNN layer i+1

RNN layer 3

RNN layer 2

RNN layer 1

the movie was terribly exciting !

 Dr. Waqas Ahmed
Air University Islamabad

LSTMs: Real-world
Success
• In 2013–2015, LSTMs started achieving state-of-the-art results
• Successful tasks include handwriting recognition, speech recognition, machine translation, parsing,
and image captioning, as well as language models
• LSTMs became the dominant approach for most NLP tasks
• Now (2019–2024), Transformers have become dominant for all tasks
• For example, in WMT (a Machine Translation conference + competition):
• In WMT 2014, there were 0 neural machine translation systems (!)
• In WMT 2016, the summary report contains “RNN” 44 times (and these systems won)
• In WMT 2019: “RNN” 7 times, ”Transformer” 105 times

Source: "Findings of the 2016 Conference on Machine Translation (WMT16)", Bojar et al. 2016, [Link]
Source: "Findings of the 2018 Conference on Machine Translation (WMT18)", Bojar et al. 2018, [Link]
Source: "Findings of the 2019 Conference on Machine Translation (WMT19)", Barrault et al. 2019, [Link]