Recurrent Neural Networks
27 Jan
NLP and Sequential Data
• NLP is full of sequential data
• Words in sentences
• Characters in words
• Sentences in discourse
• Can you think of other examples?
Long-distance
Dependencies in Language
• Agreement in number, gender, etc.
He does not have very much confidence in himself.
She does not have very much confidence in herself.
• Selectional preference
The reign has lasted as long as the life of the queen.
The rain has lasted as long as the life of the clouds.
Can be Complicated!
• What is the referent of “it”?
The trophy would not fit in the brown suitcase because it was too big.
Trophy
The trophy would not fit in the brown suitcase because it was too small.
Suitcase
(from Winograd Schema Challenge:
[Link]
Recurrent Neural Networks
(Elman 1990)
• Tools to “remember” information
Feed-forward NN Recurrent NN
label label
predict predict
transform transform
lookup lookup
context context
Recurrent Neural Networks
An unrolled recurrent neural
network
Standard RNN
Unrolling in Time
• What does processing a sequence look like?
label label label label
predict predict predict predict
RNN RNN RNN RNN
I hate this movie
Training RNNs
sum total loss
label 1 label 2 label 3 label 4
loss 1 loss 2 loss 3 loss 4
prediction 1 prediction 2 prediction 3 prediction 4
predict predict predict predict
RNN RNN RNN RNN
I hate this movie
RNN Training
• The unrolled graph is a well-formed (DAG)
computation graph—we can run backprop
total loss
sum
• Parameters are tied across time, derivatives are
aggregated across all time steps
• This is historically called “backpropagation through
time” (BPTT)
Parameter Tying
sum total loss
label 1 label 2 label 3 label 4
loss 1 loss 2 loss 3 loss 4
prediction 1 prediction 2 prediction 3 prediction 4
predict predict predict predict
RNN RNN RNN RNN
I hate this movie
Parameters are shared! Derivatives are accumulated.
Applications of RNNs
What Can RNNs Do?
• Represent a sentence
• Read whole sentence, make a prediction
• Represent a context within a sentence
• Read context up until that point
Representing Sentences
prediction
predict
RNN RNN RNN RNN
I hate this movie
• Sentence classification
• Conditioned generation
• Retrieval
Representing Contexts
label label label label
predict predict predict predict
RNN RNN RNN RNN
I hate this movie
• Tagging
• Language Modeling
• Calculating Representations for Parsing, etc.
e.g. Language Modeling
I hate this movie </s>
predict predict predict predict predict
RNN RNN RNN RNN RNN
<s> I hate this movie
• Language modeling is like a tagging task, where
each tag is the next word!
Bi-RNNs
• A simple extension, run the RNN in both directions
PRN VB DET NN
softmax softmax softmax softmax
concat concat concat concat
RNN RNN RNN RNN
RNN RNN RNN RNN
I hate this movie
Vanishing Gradient
• Gradients decrease as they get pushed back
• Why? “Squashed” by non-linearities or small
weights in matrices.
A Solution:
Long Short-term Memory
(Hochreiter and Schmidhuber 1997)
• Basic idea: make additive connections between
time steps
• Addition does not modify the gradient, no vanishing
• Gates to control the information flow
Reminder: Standard RNN
LSTM with four interacting layers
The cell state
Gates
sigmoid layer
Step-by-Step LSTM Walk Through
Forget gate layer
Input gate layer
The current state
Output layer
Other Alternatives
• Lots of variants of LSTMs (Gers and Schmidhuber,
2000)
• Gated recurrent units (GRUs; Cho et al., 2014)
• All follow the basic paradigm of “take input, update
state”
Stacked LSTM
(Graves et al. 2013)
Image source: [Link]
Why adding layers?
• In feed-forward NNs:
• create new representations at higher levels of
abstraction
• In RNNs:
• chunk observations over time
• represent the problem at different time scales
• Resources:
• Graves et al. 2013 paper:
[Link]
• Blog by Jason Brownlee:
[Link]
short-term-memory-networks/
Recurrent Neural Networks:
Process Sequences
Recurrent Neural Networks: Process
Sequences
e.g. Image Captioning
image -> sequence of words
Recurrent Neural Networks: Process
Sequences
e.g. Sentiment Classification
sequence of words -> sentiment
Recurrent Neural Networks: Process
Sequences
e.g. Machine Translation
seq of words -> seq of words
Recurrent Neural Networks: Process
Sequences
e.g. Tagging, e.g. Named
Entity (NE) or POS tagging
Credits
• These slides were adapted from
• the course by Graham Neubig:
[Link]
• and Christopher Olah’s blog:
[Link]
Understanding-LSTMs/
• Stanford’s CS231n course:
[Link]
017_lecture10.pdf