NLP701 Lecture 10

Recurrent Neural Networks and

Long Short-Term Memory

Lecture 10: RNNs and LSTMs

Prof. Preslav Nakov
October 20 and 22, 2025
REVIEW
 Review
• What is a neural network?

3
 Neural Networks
• A neural network is a network of small
computational units:
– A neural unit is a function that takes an input
vector x, performs a computation, and produces
an output
– Example?

4
 Neural Networks
• A neural network is a network of small
computational units:
– A neural unit is a function that takes an input
vector x, performs a computation, and produces
an output
– Example: Binary logistic regression
• Input: x is a feature vector
• Output: y is a number (0 or 1)

5
 ntiment. z = w·x+ b (5.3)
ly linearinter
ept around 0 but has a sharp slope toward the
cept, is another real number that’s added to the weighted inputs. ends, it tends to squash
card regression
0 or 1. Tosolves make this
a task by learning,
decision from a training set, a vector of th
But note that nothing 1in Eq. 5.3on forcesa test
z to instance—
be a legal after
probability, we’ve
that learned
is, to lie
dure 5.1 Thesigmoid function y = 1+ e takesareal valueand mapsit to therange [0, 1].
abias terlinear
[Link] Each weight wslope
istoward
areal number, and isassociated with one
Review: Logistic Regression
−z
ause itbetweentraining—
is nearly 0 and theclassifier
1 1. 0 In
but fact, since
has a sharp i first
weightsmultiplies
theare
ends, each
real-valued,
it tends x by
the
to squash itsweight
output might w ,
evensumsup
be
probability,
esigmoid function ywe’ = 1+lle pass z
takesareal through the
valueand mapsit sigmoid function,
to therange [0,1]. i s (z). The i
erfeatures [Link]
The
[Link] aweight w represents how important that input feature is z
−z
values toward i to
rly linear
nfication
(named features,
negative;
around and
ranges sharpadds
from slope− • the
toward • bias
.
the ends,term
it tendsb.
because it looks like an s) is also called the logistic func- The
to squash resulting single number
ward 0 or 1. decision, and can be positive (meaning the feature is associated
To createweighted
ogistic a probability,sum
regression we’ll of the
pass
its name. This evidence
z through
The issigmoid
athe
function
sigmoidforthat
has the maps
function,
the class.
sany
(z). real
following The equation,
ass)
moid or negative
function (named (meaning
because it looks likethe
an s)feature
number is to realsisvalue
alsoa called not
the inassociated
[0;1]
logistic func- with the class).
ly in Fig. 5.1:
probability, we’ ll pass z through the sigmoid function, (z). The !
n,nghtand expect
gives logisticin a sentiment
regression itsname. task the
Thesigmoid
(named because it looks like an s) is also called the logistic func- word
has awesome
thefollowing X n to
equation, have a high positive
dwn graphically
abysmal
ogistic in Fig.
to
regression have 5.1: The sigmoid has the following equation,
itsname. a very negative 1 z = The bias
weight. wi xiter + m,balso called the
lly in Fig. 5.1: y = s (z) = (5.4)
s another real number that’s
y = s (z) =1+ e− z
added
1 − z to the weighted
i= 1 inputs.
(5.4)
e a decision on a test 1+ e
1 instance— after we’ve learned the weights in
y = s (z) = (5.4)
In therest of thebook
heclassifier first multiplies each xi by itsweight sumsusing we’ ll represent such wi , sumsupthedot product
uct 1+ e −z
theweighted
nd adds linear
the algebra.
bias termThe b. dot The product
resulting of two singlevectors number a andz b, written as
expresses thea·b
um ofThis the
the products
evidence of the
for the [Link] elements of each x is vector.
the input Thus the fo
is linear regression vector
equivalent formation to Eq. 5.2:
n
!
X
Figure 5.1 Thesigmoid function y = 1+1e− z takesareal
z = wi xi + bz = valueand w·x+ bmaps it to the range [0,1]. (5.2)
Because it is nearly linear around 0 but has a sharp slope toward the ends, it tends to squash
outlier values toward 0 or 1. i= 1
But note that nothing in Eq. 5.3 forces z to be a legal probability,
fd thebook we’ ll w is a set
represent of values
such (weights) that
sumsusing are product notation from
thedot
between
To create a0 probability,
and 1. In fact,
we’ since
ll pass
optimized to fitz theweights
data theare
through real-valued,
sigmoid function, sthe
(z).output
The m
bra. sigmoid
The dotfunction
product
negative; of two
z ranges
(named from vectors
because toa• and
− •it looks .like b,
an written
s) is also as a·btheislogistic
called the sum 6
of
func-
 Visualizing a Neural Unit

y
• A neural unit is a function of a a
vector of inputs. Each unit produces
one output, called activation σ
• The output symbol y is reserved for z
the final output of the network. a is
the activation or the output of a ∑
single neuron
– In this case, where we have just one w1 w2 w3 b
neural unit, y=a
x1 x2 x3 +1

7
 Neural Networks

• Combining neural units into larger networks to

model complex relationships between inputs
and outputs.

8
Neural Networks
The lines represent the weights w in the
function, which are initialized randomly,
and then updated based on the data

9
Neural Networks

We can have multiple identical functions

connected to the same input, each with
a different set of weights.

10
Neural Networks
The outputs here can serve as inputs for
the next layer, and the same process is
repeated …

σ
σ

σ
σ

11
Neural Networks

Input Layer Hidden Layer Output Layer

σ
σ

σ
σ

12
 Review
• What is an activation function?
• What activation function would you use in the
output layer for the following cases:
– 5-layer network for binary classification
– 3-layer network for multi-class classification (e.g., 5 classes)

13
 Review: Notation & Formulae

+1

x1 xn +1

14
 Review
sigmoid or softmax
𝑦 = 𝑎[2]
𝑎[2] = 𝑔 2 (𝑧 2
)
W[2] b[2] 𝑧 [2] = 𝑊 [2] 𝑎[1] + 𝑏 [2]
ReLU, tanh, sigmpod
j +1 𝑎[1] = 𝑔 1 (𝑧 1
)

W[1] b[1] 𝑧 [1] = 𝑊 [1] 𝑎[0] + 𝑏 [1]

x1 i xn +1 𝑎[0]

15
 Exercise
• Design a feed-forward neural network for sentiment
classification
– Input: one sentence
– Output: positive or negative

16
 Exercise
• Design a feed-forward neural network for sentiment
classification
– Input: one sentence
– Output: positive or negative

• Design a feed-forward neural network for entity

classification:
– Input: sentence
– Output: One tag per word, 4 classes: person, organization,
location, NONE
17
 Exercise
• Design a feed-forward neural network for sentiment
classification
– Input: one sentence
– Output: positive or negative

• Design a feed-forward neural network for entity

classification:
– Input: sentence
– Output: One tag per word, 4 classes: person, organization,
location, NONE

What are the problems? 18

RECURRENT NEURAL NETWORKS
 Recurrent Neural Networks
• The problem with feed-forward NN is that the
input and the output lengths are fixed
– Not all problems can be converted into such with
fixed-length inputs and outputs

• Recurrent neural networks allow us to work

with variable input lengths

20
 Networ
Elmanks
Networ ks (Elman, 1990). These networks are useful in their own ri
basis for more complex approaches to be discussed later in
Chapter 22.
Recurrent Neural Networks Fig. 9.2 abstractly illustrates the recurrent structure of a
feed-forward networks, an input vector representing the c
is multiplied by a weight matrix and then passed through
compute an activation value for a layer of hidden of uni
in turn, used to calculate a corresponding output, yt . Seq
• RNNs contain cycles with the network
connections
yt
– The output of a unit is used as input to
itself
– This means the activation from one time
step will augment the input in the next ht
time step
• In a way,
What do
this is a form ofyou
memory. The
network remembers the previous inputs
think
through this is?
the recurrence
xt

Figur e 9.2 Simple recurrent neural network after Elman (Elm

includes a recurrent connection as part of its input. That is, the ac
layer depends on the current input as well as the activation value
previous timestep. 21
 Networ
Elmanks
Networ ks (Elman, 1990). These networks are useful in their own ri
basis for more complex approaches to be discussed later in
Chapter 22.
Recurrent Neural Networks Fig. 9.2 abstractly illustrates the recurrent structure of a
feed-forward networks, an input vector representing the c
is multiplied by a weight matrix and then passed through
compute an activation value for a layer of hidden of uni
in turn, used to calculate a corresponding output, yt . Seq
• RNNs contain cycles with the network
connections
yt
– The output of a unit is used as input to
itself
– This means the activation from one time
step will augment the input in the next ht
time step
• In a way, this is a form of memory. The
network remembers the previous inputs
through the recurrence
xt

Figur e 9.2 Simple recurrent neural network after Elman (Elm

includes a recurrent connection as part of its input. That is, the ac
layer depends on the current input as well as the activation value
previous timestep. 22
 Recurrent Neural Networks
• The content of the previous hidden layer encodes
information about all previous inputs
9.1 • SI M PL E RECURRENT N ETWORK S

yt

ht

U W

ht-1 xt

Figure 9.3 Simple recurrent neural network illustrated as a feed-forward network.

presenting one element at a time to the network. The key difference from a fe
23
forward network lies in the recurrent link shown in the figure with the dashed l
 Recurrent Neural Networks
• The content of the previous hidden layer encodes
information about all previous inputs
How? 9.1 • SI M PL E RECURRENT N ETWORK S

yt

ht

U W

ht-1 xt

Figure 9.3 Simple recurrent neural network illustrated as a feed-forward network.

presenting one element at a time to the network. The key difference from a fe
24
forward network lies in the recurrent link shown in the figure with the dashed l
 Recurrent Neural Networks
• The content of the previous hidden layer encodes
information about all previous inputs
– Through the recursion 9.1 • SI M PL E RECURRENT N ETWORK S
• There is no fixed-length limit of the prior
context, which (in theory) allows us yt

to consider the full input V

ht
In practice, recursion is implemented as an
additional input, which is copied from the
hidden state of the previous time step U W

ht-1 xt

Figure 9.3 Simple recurrent neural network illustrated as a feed-forward network.

presenting one element at a time to the network. The key difference from a fe
25
forward network lies in the recurrent link shown in the figure with the dashed l
Sample Feed-Forward Network

y1

h1 ℎ1 = 𝑊𝑥1 + 𝑏1

x1
t=1

26
 Sample RNN

y3

y2
This is an additional input h3
at t=2 y1
h2
x3
h1 t=3
x2

x1 t=2 ℎ2
ℎ3 = 𝑊 +𝑏
t=1
ℎ1 𝑥3
ℎ2 = 𝑊 +𝑏
𝑥2
ℎ1 = 𝑊𝑥1 + 𝑏
27
 Sample RNN

y3

y2
At t=1, we can input a
h3
dummy vector h0
y1
h2
x3
h1 t=3
x2
h0 t=2
x1
t=1

ℎ𝑡−1
ℎ𝑡 = 𝑊 +𝑏
𝑥𝑡
28
 The RNN Cell

xt W

ht
ht-1

æ xt ö
ht = tanhW ç ÷
è ht-1 ø

Nonlinearity

29
 The RNN Forward Pass
C1 C2 C3

y1 y2 y3

æ xt ö
ht = tanhW ç ÷
è ht-1 ø
h1 h2 h3
yt = F(ht )
Ct = Loss(yt ,GTt )

x1 h 0 x2 h1 x3 h2

30
 The RNN Forward Pass
C1 C2 C3

y1 y2 y3

æ xt ö
ht = tanhW ç ÷
è ht-1 ø
h1 h2 h3
yt = F(ht )
Ct = Loss(yt ,GTt )

indicates shared weights

x1 h 0 x2 h1 x3 h2

31
 Recurrent Neural Networks (RNNs)
• Note that the weights are shared over time
steps
– Essentially, copies of the RNN cell are made over
time (unrolling/unfolding), with different inputs at
different time steps

32
 Exercise
• Design a recurrent neural network for sentiment
classification
– Input: one sentence
– Output: positive or negative

33
 Sentiment Classification

RNN
h1

The

34
 Sentiment Classification

RNN RNN
h1 h2

The food

35
 Sentiment Classification

hn

RNN RNN RNN

h1 h2 hn-1

The food good

36
 Sentiment Classification

Linear
Classifier

hn

RNN RNN RNN

h1 h2 hn-1

The food good

37
 Sentiment Classification

Linear
Ignore Ignore
Classifier

h1 h2 hn

RNN RNN RNN

h1 h2 hn-1

The food good

38
 Sentiment Classification

h = Sum(…)

h1 hn
h2

RNN RNN RNN

h1 h2 hn-1

The food good

39
[Link]
 Text Classification with RNNs
• The output layer is only applied after the last word in
the sentence
• The last hidden layer encodes the entire
sequence (we hope so)

40
 Sentiment Classification

Linear
Classifier

h = Sum(…)

h1 hn
h2

RNN RNN RNN

h1 h2 hn-1

The food good

41
[Link]
 What is the motivation
for each of the two models?

42
 Exercise

• Design a recurrent neural network for entity

classification:
– Input: sentence
– Output: One tag per word, 4 classes: person, organization,
location, NONE

43
 Sequence Labeling with RNNs

• Example: Parts-of-speech tagging

44
 The RNN Forward Pass
C1 C2 C3

y1 y2 y3 æ xt ö
ht = tanhW ç ÷
è ht-1 ø
yt = F(ht )
h1 h2 h3 Ct = Loss(yt ,GTt )

“Unfold” network through time by

making copies at each time-step

x1 h 0 x2 h1 x3 h2

45
Backpropagation Refresher

y = f (x;W )
C C = Loss(y, yGT )

y
SGD Update
f(x; W) ¶C
W ¬W -h
¶W
x
¶C æ ¶C ö æ ¶y ö
=ç ç ÷
¶W è ¶y ÷ø è ¶W ø

46
Chain Rule for Gradient Computation
Given: æ ¶C ö
C çè ¶y ÷ø
æ ¶C ö æ ¶C ö
We are interested in computing: ç ,ç
y è ¶W ø è ¶x ÷ø
÷
Intrinsic to the layer are:
f(x; W)
æ ¶y ö
çè ÷ø - How does output change due to params
x ¶W
æ ¶y ö
çè ÷ø - How does output change due to inputs
¶x

æ ¶C ö æ ¶C ö æ ¶y ö æ ¶C ö æ ¶C ö æ ¶y ö
çè ÷ø = ç ç ÷ çè ÷ø = ç ç ÷
¶W è ¶y ÷ø è ¶W ø ¶x è ¶y ÷ø è ¶x ø
47
 Chain Rule for Gradient Computation
Given: æ ¶C ö
çè ¶y ÷ø
æ ¶C ö æ ¶C ö æ ¶C ö
çè ¶y ÷ø We are interested in computing: ç ,ç
è ¶W ø è ¶x ÷ø
÷
Intrinsic to the layer are:
f(x; W)
æ ¶y ö
çè ÷ø - How does output change due to params
¶W
æ ¶C ö
çè ÷
¶x ø æ ¶y ö
çè ÷ø - How does output change due to inputs
¶x

æ ¶C ö æ ¶C ö æ ¶y ö æ ¶C ö æ ¶C ö æ ¶y ö
çè ÷ø = ç ç ÷ çè ÷ø = ç ç ÷
¶W è ¶y ÷ø è ¶W ø ¶x è ¶y ÷ø è ¶x ø
Equations for common layers: [Link] 48
 Multiple Layers
y1 = f1 (x;W1 )
y2 = f2 (y1;W2 )
C C = Loss(y2 , yGT )

y2
SGD Update
¶C
f2(y1; W2) W2 ¬ W2 - h
¶W2
y1 ¶C
W1 ¬ W1 - h
¶W1
f1(x; W1)

49
Chain Rule for Gradient Computation
y1 = f1 (x;W1 )
y2 = f2 (y1;W2 )
C C = Loss(y2 , yGT )
¶C ¶C
y2 Find ,
¶W1 ¶W2
f2(y1; W2) ¶C æ ¶C ö æ ¶y2 ö
=ç
¶W2 è ¶y2 ÷ø çè ¶W2 ÷ø
y1
¶C æ ¶C ö æ ¶y1 ö
=ç
f1(x; W1) ¶W1 è ¶y1 ÷ø çè ¶W1 ÷ø
æ ¶C ö æ ¶y2 ö æ ¶y1 ö
x =ç
è ¶y2 ÷ø çè ¶y1 ÷ø çè ¶W1 ÷ø
50
Application of the Chain Rule
Extension to Computational Graphs

y1 y2

y
f1(y; W1) f2(y; W2)
f(x; W)

y y
x

f(x; W)

51
Extension to Computational Graphs
æ ¶C1 ö æ ¶C2 ö
æ ¶C ö çè ¶y ÷ø çè ¶y ÷ø
1 2
çè ¶y ÷ø
f1(y; W1) f2(y; W2)
f(x; W)
æ ¶C1 ö æ ¶C2 ö
çè ¶y ÷ø çè ¶y ÷ø
æ ¶C ö
çè ÷
¶x ø
S
f(x; W)

æ ¶C ö
çè ÷
¶x ø
52
Extension to Computational Graphs
æ ¶C1 ö æ ¶C2 ö
æ ¶C ö çè ¶y ÷ø çè ¶y ÷ø
1 2
çè ¶y ÷ø
f1(y; W1) f2(y; W2)
f(x; W)
æ ¶C1 ö æ ¶C2 ö
çè ¶y ÷ø çè ¶y ÷ø
æ ¶C ö
çè ÷
¶x ø
S Gradient Accumulation

f(x; W)

æ ¶C ö
çè ÷
¶x ø
53
Backpropagation Through Time (BPTT)
• BPTT is used to train RNNs
• The unfolded network (used during the forward
pass) is treated as one big feed-forward network
– This unfolded network accepts the whole time series as
an input
– The weight updates are computed for each copy in the
unfolded network, then summed (or averaged) and then
applied to the RNN weights

54
 The Unfolded RNN
C1 C2 C3

y1 y2 y3

• Treat the unfolded network as one

big feed-forward network!

h3 • This big network takes in entire

h2 sequence as an input
h1
• Compute the gradients through the
usual backpropagation

• Update the shared weights

h2
h1
h0

x1 x2 x3 55
 The Unfolded RNN Backward
æ xt ö
C1 C2 C3 ht = tanhW ç ÷
è ht-1 ø
y1 y2 y3
yt = F(ht )
Ct = Loss(yt ,GTt )

h1 h2 h3

x1 h 0 x2 h1 x3 h2

56
 Problems with RNNs

• In NLP, using RNNs for classification of long

sequences has many drawbacks
– Simple RNNs typically encode recent words; information
from more distant context can be lost
– The errors backpropagated to earlier time steps tend to
either get too small or too large, which complicates training

Why is this?

57
The Problem of Exploding or Vanishing Gradients

• What happens to the magnitude of the gradients as

we backpropagate through many layers?
– If the weights are small, the gradients shrink
exponentially.
– If the weights are big the gradients grow
exponentially.
• Typical feed-forward neural nets can cope with
these exponential effects because they only have a
few hidden layers.

58
The Problem of Exploding or Vanishing Gradients

• In an RNN trained on long sequences (e.g., 100

time steps) the gradients can easily explode or
vanish.
– So RNNs have difficulty dealing with long-
range dependencies.

59
 Issues with the Vanilla RNNs
• In the same way, a product of k real numbers can shrink
to zero or explode to infinity, so can a product of
matrices

• Exploding gradients are often controlled with gradient

element-wise or norm clipping

1 On the difficulty of training recurrent neural networks, Pascanu et al., 2013 60

 The Unfolded RNN Backward
æ xt ö
C1 C2 C3 ht = tanhW ç ÷
è ht-1 ø
y1 y2 y3
yt = F(ht )
Ct = Loss(yt ,GTt )

h1 h2 h3

x1 h 0 x2 h1 x3 h2

61
 The Unfolded
h and
RNN Backward
are related through
t ht-1
matrix multiplication

æ xt ö
C1 C2 C3 ht = tanhW ç ÷
è ht-1 ø
y1 y2 y3
yt = F(ht )
Ct = Loss(yt ,GTt )

h1 h2 h3

x1 h 0 x2 h1 x3 h2

62
 The Unfolded
h and
RNN Backward
are related through
t ht-1
matrix multiplication

æ xt ö
C1 C2 C3 ht = tanhW ç ÷
è ht-1 ø
y1 y2 y3
yt = F(ht )
Ct = Loss(yt ,GTt )

h1 h2 h3

These can be very small gradients

corresponding to many hidden layers

x1 h 0 x2 h1 x3 h2

63
 The Identity Relationship
• Recall æ xt ö
ht = tanhW ç ÷
è ht-1 ø
yt = F(ht )
Ct = Loss(yt ,GTt )
𝜕ℎ
What is 𝑡
• Suppose that instead of a matrix multiplication, 𝜕ℎ𝑡we
−1
had an
identity relationship between the hidden states
ht = ht-1 + F(xt )
æ ¶ht ö
Þç ÷ =1
è ¶ht -1 ø
• The gradient does not decay as the error is propagated all the
way back aka “Constant Error Flow”
64
 Managing Context in RNNs
• To address the issue of vanishing gradients, we
use special network architectures that
explicitly manage past context as memory
units
– Discard information that is no longer needed
– Remember information needed for later time
steps.

65
 Long Short-Term Memory (LSTM)
• The LSTM uses this idea of “Constant Error Flow” for
RNNs to ensure that gradients don’t decay

• The key component is a memory cell that acts like an

accumulator (contains the identity relationship) over
time
– which prevents the vanishing gradient issue

Long Short-Term Memory, Hochreiter et al., 1997 66

 Long Short-Term Memory
• LSTM cells use additional functions to manage
the information flow from hidden units to the
subsequent time steps
– These additional units are called gates. An LSTM
cell has an input gate, a forget gate, and an output
gate
• LSTMs are the most widely used architecture in
NLP neural models.

67
 The LSTM Idea

ht = tanh ct
xt W Cell

ct ht
ht-1

æ xt ö
ct = ct-1 + tanhW ç ÷
è ht-1 ø

𝜕𝑐𝑡
=1
𝜕𝑐𝑡−1
68
* Dashed line indicates time-lag
 The Original LSTM Cell
xt ht-1 æ æ xt ö ö
it = s ç Wi ç ÷ + bi ÷
è è ht-1 ø ø

Wi And i is calculated based on

Input Gate it
the previous hidden state
and the input
xt W Cell

ct ht
ht-1

æ xt ö
ct = ct-1 + it Ä tanhW ç ÷
è ht-1 ø

How much to update the cell

will depend on the value of i
69
 The Original LSTM Cell
xt ht-1 𝑥𝑡 xt ht-1
𝑜 𝑡 = 𝜎 𝑊𝑜 ℎ + 𝑏𝑜
𝑡−1

Wi Wo
Input Gate it Output Gate ot

xt W Cell

ct ht
ht-1
ht = ot Ä tanhct
æ xt ö
ct = ct-1 + it Ä tanhW ç ÷ How much to output
è ht-1 ø depends on o

70
 The Popular LSTM Cell
xt ht-1 xt ht-1

Wi Wo
Input Gate it Output Gate ot

xt W Cell

ct ht
ht-1

How much to remember

ft Forget Gate from the previous cell state
Wf
depends on f
æ æ xt ö ö æ xt ö
ft = s ç W f ç ÷ + b f ÷ ct = ft Ä ct-1 + it Ä tanhW ç ÷
è è ht-1 ø
xt ht-1
ø è ht-1 ø
71
 LSTM

Basic Neural Unit Recurrent Unit LSTM Unit

72
 LSTMs: Memory

How much memory does

an LSTM have?

73
 74
[Link]
 75
[Link]
 76
[Link]
 77
[Link]
 78
[Link]
 79
[Link]
 80
[Link]
 81
[Link]
 82
[Link]
 83
[Link]
 84
[Link]
 Gated Recurrent Unit (GRU)
• A simplified version of the LSTM
– Merges the forget and the input gate into a single ‘update’
gate
– Merges the memory cell and the hidden state

• Has fewer parameters than an LSTM and has been

shown to outperform LSTM on some tasks

Learning Phrase Representations using RNN Encoder-Decoder for 85

Statistical Machine Translation, Cho et al., 2014
 GRU

æ æ xt ö ö
rt = s ç Wr ç ÷ + b f ÷
è è ht-1 ø ø

rt Reset Gate
Wf

xt ht-1

86
 GRU

æ æ xt ö ö
rt = s ç Wr ç ÷ + b f ÷
è è ht-1 ø ø

æ xt ö
h't = tanhW ç
è rt Ä ht-1 ÷ø
xt W
h’t

ht-1

rt Reset Gate
Wf

xt ht-1

87
 GRU
xt ht-1

æ æ xt ö ö
Wz rt = s ç Wr ç ÷ + b f ÷
Update Gate zt
è è ht-1 ø ø

æ xt ö
h't = tanhW ç
è rt Ä ht-1 ÷ø
xt W
h’t

ht-1
æ æ xt ö ö
zt = s ç Wz ç ÷ + b f ÷
è è ht-1 ø ø
rt Reset Gate
Wf

xt ht-1

88
 GRU
xt ht-1

æ æ xt ö ö
Wz rt = s ç Wr ç ÷ + b f ÷
Update Gate zt
è è ht-1 ø ø

æ xt ö
h't = tanhW ç
è rt Ä ht-1 ÷ø
xt W
h’t
ht
ht-1
æ æ xt ö ö
zt = s ç Wz ç ÷ + b f ÷
è è ht-1 ø ø
rt Reset Gate
Wf
ht = (1- zt )Ä ht-1 + zt Ä h't

xt ht-1

89
 Exercise

• Design a recurrent neural network for language

modeling
– Input: ?
– Output: ?

90
Neural Language Model

91
Text Generation

92
Text Generation by an LSTM

93
Text Generation by an LSTM

94
 (For Comparison)
Text Generation with n-gram Model

95
 What is Encoded in h?
• LSTMs trained as language models generate
syntactic sentences with long-range number
agreement (among other consistencies)
– How do they achieve that?

96
 What is Encoded in h?
• LSTMs trained as language models generate
syntactic sentences with long-range number
agreement (among other consistencies)
– Information in the neural units could be stored in
local or distributed ways
• A single unit or multiple units could be responsible for
encoding a syntactic or a semantic structure

97
 What is Encoded in h?
• Example: Number agreement
– If the network uses local encoding , it is possible
to identify the specific units that encode number
information
– How?

98
 What is Encoded in h?
• Example: Number agreement
– If the network uses local encoding , it is possible
to identify the specific units that encode number
information
– This can be by ablating cells (setting them to zero)
in the network and observing their impact on
performance

99
 What is Encoded in h?
• Example: Number agreement
– Data used for probing number agreement:

From: The emergence of number and syntax units in LSTM language models, 2019
100
 What is Encoded in h?
• Example: Number agreement
– Using a neural language model with 2 hidden
LSTM layers of size 650
– Two cells were identified as responsible for storing
number information
• ablating them resulted in drastic decrease in
performance

From: The emergence of number and syntax units in LSTM language models, 2019
101
 What is Encoded in h?
• Example: Number agreement
– Visualizing gate and cell dynamics:

From: The emergence of number and syntax units in LSTM language models, 2019
102
 Multi-layer RNNs
• We can of course design RNNs with multiple hidden layers
y1 y2 y3 y4 y5 y6

x1 x2 x3 x4 x5 x6

103
 Bi-directional RNNs
• RNNs can process the input sequence forward and in the reverse
direction
y1 y2 y3 y4 y5 y6

x1 x2 x3 x4 x5 x6

104
 Contextual Embeddings
• Word embeddings obtained using word2vec
and similar models are static
– Each word has a fixed representation, regardless
of its context
– New words cannot be easily modeled without
retraining
– The embeddings encode some syntactic and
semantic features, but they are implicit and hard
to disentangle

105
 Contextual Embeddings
• ELMo (Embeddings from Language Models) is
a multi-layer bidirectional LSTM trained as a
language model

– The hidden layers of the network corresponding to

each input word are used as contextualized
embeddings

106
 ELMo Architecture
Semantics

Syntax

Fixed representation
Character CNN

Image from: [Link] 107

 Summary
• RNNs allow us to process variable-length inputs and outputs
by maintaining state information across time steps
• Backpropagation through many timesteps can be tricky due to
exploding and vanishing gradients
– LSTMs address the vanishing gradient problem by controlling the
propagation of gradients through various gates
– Exploding gradients are handled by gradient clipping
• RNNs can be stacked, or bi-directional
• Architectures such as GRU have fewer parameters than LSTM
and might perform better

108
 Other Useful Resources / References
• [Link]
• [Link]

• R. Pascanu, T. Mikolov, and Y. Bengio, On the difficulty of training recurrent neural

networks, ICML 2013
• S. Hochreiter, and J. Schmidhuber, Long short-term memory, Neural computation,
1997 9(8), pp.1735-1780
• F.A. Gers, and J. Schmidhuber, Recurrent nets that time and count, IJCNN 2000
• K. Greff , R.K. Srivastava, J. Koutník, B.R. Steunebrink, and J. Schmidhuber, LSTM: A
search space odyssey, IEEE transactions on neural networks and learning systems,
2016
• K. Cho, B. Van Merrienboer, C. Gulcehre, D. Bahdanau, F. Bougares, H. Schwenk,
and Y. Bengio, Learning phrase representations using RNN encoder-decoder for
statistical machine translation, ACL 2014
• R. Jozefowicz, W. Zaremba, and I. Sutskever, An empirical exploration of recurrent
network architectures, JMLR 2015

109
 Slide References
• The slides in this lecture are adapted from
multiple other lectures from:
– Shangsong Liang, MBZUAI
– Hanan Aldarmaki, MBZUAI
– “Speech & Language Processing” 3rd edition, draft:
• [Link]

110