CHAPTER 12

Recurrent Neural Networks

In chapter 8 we studied neural networks and how we can train the weights of a network,
based on data, so that it will adapt into a function that approximates the relationship be-
tween the (x, y) pairs in a supervised-learning training set. In section 1 of chapter 10, we
studied state-machine models and defined recurrent neural networks (RNNs) as a particular
type of state machine, with a multidimensional vector of real values as the state. In this
chapter, we’ll see how to use gradient-descent methods to train the weights of an RNN so
that it performs a transduction that matches as closely as possible a training set of input-
output sequences.

1 RNN model
Recall that the basic operation of the state machine is to start with some state s0 , then
iteratively compute for t > 1::

st = f(st−1 , xt )
yt = g(st )

as illustrated in the diagram below (remembering that there needs to be a delay on the
feedback loop):

xt yt
f g
−
st−1

So, given a sequence of inputs x1 , x2 , . . . the machine generates a sequence of outputs

g(f(s0 , x1 )), g(f(f(s0 , x1 ), x2 , )), . . . .

| {z } | {z }
y1 y2

82
MIT 6.036 Fall 2019 83

A recurrent neural network is a state machine with neural networks constituting functions
f and g:

f(s, x) = f1 (W sx x + W ss s + W0ss )
g(s) = f2 (W O s + W0O ) .

The inputs, outputs, and states are all vector-valued: We are very sorry! This
course material has
xt : ` × 1 evolved from different
sources, which used
st : m × 1 W T x in the forward
yt : v × 1 . pass for regular feed-
forward NNs and Wx
for the forward pass in
RNN s. This inconsis-
The weights in the network, then, are tency doesn’t make any
technical difference, but
W sx : m × ` is a potential source of
confusion.
W ss : m × m
W0ss : m × 1
WO : v × m
W0O : v × 1

with activation functions f1 and f2 . Finally, the operation of the RNN is described by

st = f1 (W sx xt + W ss st−1 + W0ss )


yt = f2 W O st + W0O .

Study Question: Check dimensions here to be sure it all works out. Remember that
we apply f1 and f2 elementwise.

2 Sequence-to-sequence RNN
Now, how can we train an RNN to model a transduction on sequences? This problem is
sometimes called sequence-to-sequence mapping. You can think of it as a kind of regression
problem: given an input sequence,
 learn to

generate the corresponding

 output sequence. One way to think of
A training set has the form x(1) , y(1) , . . . , x(q) , y(q) , where training a sequence
classifier is to reduce it
• x(i) and y(i) are length n(i) sequences; to a transduction prob-
lem, where yt = 1 if the
• sequences in the same pair are the same length; and sequences in different pairs may sequence x1 , . . . , xt is a
positive example of the
have different lengths. class of sequences and
−1 otherwise.
Next, we need a loss function. We start by defining a loss function on sequences. There
are many possible choices, but usually it makes sense just to sum up a per-element loss
function on each of the output values, where p is the predicted sequence and y is the actual
one:
  nX(i)
 
(i) (i)
Lossseq p , y
(i) (i)
= Losselt pt , yt .
t=1

The per-element loss function Losselt will depend on the type of yt and what informa-
tion it is encoding, in the same way as for a supervised network.. Then, letting θ = So it could be NLL,
squared loss, etc.
Last Updated: 12/18/19 11:56:05
MIT 6.036 Fall 2019 84

W sx , W ss , W O , W0ss , W0O , our overall objective is to minimize
q
X  
J(θ) = Lossseq RNN(x(i) ; θ), y(i) ,
i=1

where RNN(x; θ) is the output sequence generated, given input sequence x.

It is typical to choose f1 to be tanh but any non-linear activation function is usable. We Remember that it looks
choose f2 to align with the types of our outputs and the loss function, just as we would do like a sigmoid but
ranges from -1 to +1.
in regular supervised learning.

3 Back-propagation through time

Now the fun begins! We can find θ to minimize J using gradient descent. We will work
through the simplest method, back-propagation through time (BPTT), in detail. This is gener-
ally not the best method to use, but it’s relatively easy to understand. In section 5 we will
sketch alternative methods that are in much more common use.

Calculus reminder: total derivative Most of us are not very careful about the differ-
ence between the partial derivative and the total derivative. We are going to use a nice
example from the Wikipedia article on partial derivatives to illustrate the difference.
The volume of a circular cone depends on its height and radius:

πr2 h
V(r, h) = .
3
The partial derivatives of volume with respect to height and radius are

∂V 2πrh ∂V πr2
= and = .
∂r 3 ∂h 3
They measure the change in V assuming everything is held constant except the
single variable we are changing. Now assume that we want to preserve the cone’s
proportions in the sense that the ratio of radius to height stay constant, then we can’t
really change one without changing the other. In this case, we really have to think
about the total derivative, which sums the “paths” along which r might influence V:

dV ∂V ∂V dh
= +
dr ∂r ∂h dr
2πrh πr2 dh
= +
3 3 dr
dV ∂V ∂V dr
= +
dh ∂h ∂r dh
πr2 2πrh dr
= +
3 3 dh
Just to be completely concrete, let’s think of a right circular cone with a fixed angle
α = tan r/h, so that if we change r or h then α remains constant. So we have
r = h tan − 1α; let constant c = tan−1 α, so now r = ch. Now, we know that

dV 2πrh πr2 1
= +
dr 3 3 c
dV πr2 2πrh
= + c
dh 3 3

Last Updated: 12/18/19 11:56:05

MIT 6.036 Fall 2019 85

The BPTT process goes like this:

(1) Sample a training pair of sequences (x, y); let their length be n.
(2) “Unroll" the RNN to be length n (picture for n = 3 below), and initialize s0 :

Now, we can see our problem as one of performing what is almost an ordinary back-
propagation training procedure in a feed-forward neural network, but with the dif-
ference that the weight matrices are shared among the layers. In many ways, this is
similar to what ends up happening in a convolutional network, except in the conv-
net, the weights are re-used spatially, and here, they are re-used temporally.
(3) Do the forward pass, to compute the predicted output sequence p:
z1t = W sx xt + W ss st−1 + W0ss
st = f1 (z1t )
z2t = W O st + W0O
pt = f2 (z2t )

(4) Do backward pass to compute the gradients. For both W ss and W sx we need to find
n
X
dLseq dLu
=
dW dW
u=1
(12.1)
Letting Lu = Lelt (pu , yu ) and using the total derivative, which is a sum over all the
ways in which W affects Lu , we have
n X
X n
∂Lu ∂st
= ·
∂st ∂W
u=1 t=1
(12.2)
Re-organizing, we have
n
X n
∂st X ∂Lu
= ·
∂W ∂st
t=1 u=1
(12.3)

Last Updated: 12/18/19 11:56:05

MIT 6.036 Fall 2019 86

Because st only affects Lt , Lt+1 , . . . , Ln ,

Xn n
∂st X ∂Lu
= ·
∂W u=t ∂st
t=1
 
n
X  n
X 
∂st  ∂Lt ∂Lu 
= 
· +  (12.4)
∂W 
t=1  ∂st u=t+1 ∂st 
| {z }
δst

δst is the dependence of the loss on steps after t on the state at time t. That is, δst is how
much we can blame
We can compute this backwards, with t going from n down to 1. The trickiest part is state st for all the future
figuring out how early states contribute to later losses. We define future loss element losses.
n
X
Ft = Losselt (pu , yu ) ,
u=t+1

so
∂Ft
δst = .
∂st
At the last stage, Fn = 0 so δsn = 0.
Now, working backwards,
n
∂ X
δst−1 = Losselt (pu , yu )
∂st−1 u=t
n
∂st ∂ X
= · Losselt (pu , yu )
∂st−1 ∂st u=t
" n
#
∂st ∂ X
= · Losselt (pt , yt ) + Losselt (pu , yu )
∂st−1 ∂st
u=t+1
 
∂st ∂Losselt (pt , yt )
= · + δst
∂st−1 ∂st

Now, we can use the chain rule again to find the dependence of the element loss at
time t on the state at that same time,

∂Losselt (pt , yt ) ∂z2t ∂Losselt (pt , yt )

= · ,
∂st ∂st ∂z2t
| {z } |{z} | {z }
(m×1) (m×v) (v×1)

and the dependence of the state at time t on the state at the previous time, noting that
0
we are performing an elementwise multiplication between Wss T
and the vector of f1
values, ∂st /∂z1t : There are two ways to
∂st ∂z1t ∂st think about ∂st /∂zt :
= W ss T ∗ f1 (z1t ) .
0
= · here, we take the view
∂st−1 ∂st−1 ∂z1t | {z }
that it is an m × 1 vector
| {z } | {z } |{z} not dot!
(m×m) (m×m) (m×1) and we multiply each
column of W T by it.
Putting this all together, we end up with Another, equally good,
  view, is that it is an m ×
T ∂Lt m diagonal matrix, with
δst−1 = W ss T ∗ f1 (z1t ) · W O
0
st
+ δ
| {z } ∂z2t the values along the
∂st | {z } diagonal, and then this
∂st−1 ∂Ft−1
∂st
operation is a matrix
multiply. Our software
implementation will
Last Updated: 12/18/19 11:56:05 take the first view.
MIT 6.036 Fall 2019 87

We’re almost there! Now, we can describe the actual weight updates. Using equa-
tion 12.4 and recalling the definition of δst = ∂Ft /∂st , as we iterate backwards, we
can accumulate the terms in equation 12.4 to get the gradient for the whole loss:

dLseq ∂Ft−1 ∂z1t ∂st ∂Ft−1

+= =
dW ss ∂W ss ∂W ss ∂z1t ∂st
dLseq ∂Ft−1 ∂z1t ∂st ∂Ft−1
+= =
dW sx ∂W sx ∂W sx ∂z1t ∂st

We can handle W O separately; it’s easier because it does not effect future losses in
the way that the other weight matrices do:
n
X ∂Lt X ∂Lt ∂z2 n
∂Lseq t
= = ·
∂W O ∂W O
t=1
∂z2t ∂W O t=1

Assuming we have ∂L t
∂z2t
= (pt − yt ), (which ends up being true for squared loss,
softmax-NLL, etc.), then on each iteration
∂Lseq
+ = (pt − yt ) · sTt
∂W O
| {z } | {z } |{z}
v×1 1×m
v×m

Whew!

Study Question: Derive the updates for the offsets W0ss and W0O .

4 Training a language model

(i) (i) (i)
A language model is just trained on a set of input sequences, (c1 , c2 , . . . , cni ), and is used
to predict the next character, given a sequence of previous tokens: A “token” is generally a
character or a word.
ct = RNN(c1 , c2 , . . . , ct−1 )

We can convert this to a sequence-to-sequence training problem by constructing a data

set of (x, y) sequence pairs, where we make up new special tokens, start and end, to signal
the beginning and end of the sequence:

x = (hstarti, c1 , c2 , ,̇cn )
y = (c1 , c2 , . . . , hendi)

5 Vanishing gradients and gating mechanisms

Let’s take a careful look at the backward propagation of the gradient along the sequence:
 
∂st ∂Losselt (pt , yt )
δst−1
= · +δst
.
∂st−1 ∂st
Consider a case where only the output at the end of the sequence is incorrect, but it depends
critically, via the weights, on the input at time 1. In this case, we will multiply the loss at
step n by
∂s2 ∂s3 ∂sn
· ··· .
∂s1 ∂s2 ∂sn−1

Last Updated: 12/18/19 11:56:05

MIT 6.036 Fall 2019 88

In general, this quantity will either grow or shrink exponentially with the length of the
sequence, and make it very difficult to train.
Study Question: The last time we talked about exploding and vanishing gradients, it
was to justify per-weight adaptive step sizes. Why is that not a solution to the prob-
lem this time?
An important insight that really made recurrent networks work well on long sequences
is the idea of gating.

5.1 Simple gated recurrent networks

A computer only ever updates some parts of its memory on each computation cycle. We
can take this idea and use it to make our networks more able to retain state values over time
and to make the gradients better-behaved. We will add a new component to our network,
called a gating network. Let gt be a m × 1 vector of values and let W gx and W gs be m × l
and m × m weight matrices, respectively. We will compute gt as It can have an offset,
too, but we are omitting
gt = sigmoid(W gx xt + W gs st−1 ) it for simplicity.

and then change the computation of st to be

st = (1 − gt ) ∗ st−1 + gt ∗ f1 (W sx xt + W ss st−1 + W0ss ) ,

where ∗ is component-wise multiplication. We can see, here, that the output of the gating
network is deciding, for each dimension of the state, how much it should be updated now.
This mechanism makes it much easier for the network to learn to, for example, “store”
some information in some dimension of the state, and then not change it during future
state updates, or change it only under certain conditions on the input or other aspects of
the state.
Study Question: Why is it important that the activation function for g be a sigmoid?

5.2 Long short-term memory

The idea of gating networks can be applied to make a state-machine that is even more like
a computer memory, resulting in a type of network called an LSTM for “long short-term
memory.” We won’t go into the details here, but the basic idea is that there is a memory Yet another awesome
cell (really, our state vector) and three (!) gating networks. The input gate selects (using name for a neural net-
work!
a “soft” selection as in the gated network above) which dimensions of the state will be
updated with new values; the forget gate decides which dimensions of the state will have
its old values moved toward 0, and the output gate decides which dimensions of the state
will be used to compute the output value. These networks have been used in applications
like language translation with really amazing results. A diagram of the architecture is
shown below:

Last Updated: 12/18/19 11:56:05

MIT 6.036 Fall 2019 89

Last Updated: 12/18/19 11:56:05