Introduction to Deep Learning
20. Advanced Recurrent Networks
STAT 157, Spring 2019, UC Berkeley
Alex Smola and Mu Li
[Link]/berkeley-stat-157
© 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Deep
RNNs
[Link]/berkeley-stat-157
Using RNNs
• Encode sequence
• Decode sequence
• Do both
[Link]/berkeley-stat-157 (image courtesy of [Link])
Using RNNs
Poetry Sentiment Question Named
Generation Analysis Answering Entity
Tagging
Document Machine
Classification Translation
[Link]/berkeley-stat-157 (image courtesy of [Link])
Recall - Recurrent Neural Networks
Output o
Explanation h
Observation x
• Hidden State update
ht = ϕ(Whhht−1 + Whx xt−1 + bh)
• Observation update
How to make
ot = ϕ(Whoht + bo) more nonlinear?
[Link]/berkeley-stat-157
Plan A - Nonlinearity in the units
Output o
Explanation h
Observation x
• Hidden State update
ht = ϕ(Whhht−1 + Whx xt−1 + bh)
• Observation update
Replace with
ot = ϕ(Whoht + bo) MLP?
[Link]/berkeley-stat-157
Plan A - Nonlinearity in the units
• Keeps the structure of the latent space
• More complex gradients (very costly)
• E.g. Zoph et al, 2018 learned cells with ~40 units
(slow and expensive - nobody uses them in practice)
• Hidden State update
ht = ϕ(Whhht−1 + Whx xt−1 + bh)
• Observation update
Replace with
ot = ϕ(Whoht + bo) MLP?
[Link]/berkeley-stat-157
Plan B - We go deeper
• Shallow RNN
• Input
• Hidden layer
• Output
• Deep RNN
• Input
• Hidden layer
• Hidden layer
…
• Output
[Link]/berkeley-stat-157
Plan B - We go deeper
Ht = f(Ht−1, Xt)
Ot = g(Ht)
.
H1t = f1(H1t−1, Xt)
Hjt = fj(Hjt−1, Hj−1
t )
Ot = g(HLt)
[Link]/berkeley-stat-157
Code …
[Link]/berkeley-stat-157
N N s
na l R
e c tio
ir
Bid
[Link]/berkeley-stat-157
The Future Matters
I am _____
I am _____ very hungry,
I am _____ very hungry, I could eat half a pig.
[Link]/berkeley-stat-157
The Future Matters
I am happy.
I am not very hungry,
I am very very hungry, I could eat half a pig.
[Link]/berkeley-stat-157
The Future Matters
I am happy.
I am not very hungry,
I am very very hungry, I could eat half a pig.
• Very different words to fill in, depending on past and
future context of a word.
• RNNs so far only look at the past
• In interpolation (fill in) we can use the future, too.
[Link]/berkeley-stat-157
Flashback - Graphical Models
• Hidden Markov Model
p(ht | ht−1, xt−1) and p(xt | ht, xt−1)
h
• Can model sequence jointly and solve by dynamic
programming
[Link]/berkeley-stat-157
Dynamic programming
• Joint probability
T
∏
p(x, h) = p(h1)p(x1 | h1) p(ht | ht−1)p(xt | ht)
i=2
[Link]/berkeley-stat-157
Dynamic programming
T
∑ ∏
p(x) = p(h1)p(x1 | h1) p(ht | ht−1)p(xt | ht )
h i=2
∑ ∑ ∏
= p(h1)p(x1 | h1)p(h2 | h1) p(x2 | h2) p(ht | ht−1)p(xt | ht )
h2,…hT h1 i=2
=:π2(h2)
∑ ∑ ∏
= π2(h2)p(x2 | h2)p(h3 | h2) p(x3 | h3) p(ht | ht−1)p(xt | ht )
h3,…hT h2 i=3
=:π3(h3)
[Link]/berkeley-stat-157
Dynamic programming
• Joint probability
T
∏
p(x, h) = p(h1)p(x1 | h1) p(ht | ht−1)p(xt | ht)
i=2
• Forward pass
∑
πt+1(ht+1) = πt(ht)p(xt | ht)p(ht+1 | ht)
ht
[Link]/berkeley-stat-157
Dynamic programming
T−1
∑∏
p(x) = p(ht | ht−1)p(xt | ht ) ⋅ p(hT | hT−1)p(xT | hT )
h i=1
T−1
∑ ∏ ∑
= p(ht | ht−1)p(xt | ht ) ⋅ p(hT | hT−1)p(xT | hT )
h1,…hT−1 i=1 hT
=:ρT−1(hT−1)
T−2
∑ ∏ ∑
= p(ht | ht−1)p(xt | ht ) ⋅ p(hT−1 | hT−2)p(xT−1 | hT−1)
h1,…hT−2 i=1 hT−1
=:ρT−2(hT−2)
[Link]/berkeley-stat-157
Dynamic programming
• Joint probability
T
∏
p(x, h) = p(h1)p(x1 | h1) p(ht | ht−1)p(xt | ht)
i=2
• Forward pass
∑
πt+1(ht+1) = πt(ht)p(xt | ht)p(ht+1 | h1)
ht
• Backward pass
∑
p(xj | x−j) ∝ πj(hj)ρj(hj)p(xj | hj)
hj
[Link]/berkeley-stat-157
Can we do this with RNNs, too?
[Link]/berkeley-stat-157
Bidirectional RNN
• One RNN forward
• Another one
backward
• Combine both
hidden states for
output generation
[Link]/berkeley-stat-157
This does not work for
sequence generation
[Link]/berkeley-stat-157
Why?
[Link]/berkeley-stat-157
Reasons
• Training time
• Test time
? Next
lecture
Can still use it to encode the sequence
[Link]/berkeley-stat-157
Residual
Connections
for RNNs
[Link]/berkeley-stat-157
Flashback - Does adding layers improve accuracy?
? ✔
[Link]
Flashback - Residual Networks
• Adding a layer
changes function
class
• We want to add to
the function class
• ‘Taylor expansion'
style parametrization
f(x) = x + g(x)
[Link] He et al., 2015
Flashback - Deep RNNs
• Deep RNN
• Input
• Hidden layer
• Hidden layer
…
• Output
[Link]/berkeley-stat-157
Drumroll …
[Link]/berkeley-stat-157
Residual RNNs
O1 O2 O3 … OT H̄(2i)
t = H (2i)
t + H (2i)−1
t
(L) (L) (L) (L)
• Input of every second layer
H1 H2 H3 … HT
is also added to its output
… … … … (residual connection)
(2) (2) (2) (2)
• Variants
H1 H2 H3 … H…
T
• Simple addition
(1) (1) (1) (1)
• Nonlinearity before
H1 H2 H3 … HT
addition
X1 X2 X3 XT
• Could also concatenate
[Link]/berkeley-stat-157
What about DenseNet?
[Link]/berkeley-stat-157
RNN with DenseNet Connections
O1 O2 O3 … OT
H̄(t)
t = [H(t)
t , H̄ t−1
t ]
(L) (L) (L) (L)
H1 H2 H3 … HT
… … … … • Concatenate outputs of
(2) (2) (2) (2)
previous layers as input top
H1 H2 H3 … H…
T
the next layer
(1) (1) (1) (1) • Occasionally add transition
H1 H2 H3 … HT
layers to reduce
dimensionality
X1 X2 X3 XT
[Link]/berkeley-stat-157
Regularization
In RNNs
[Link]/berkeley-stat-157
Overfitting
• RNNs overfit just like any other model
• Sequential dependence is more difficult to control
• Capacity in depth can be controlled, e.g. by dropout
• For sequential part need to decide how to deal with
variable inputs, e.g. input might be skipped)
• If we use dropout we might miss relevant aspects in the
coordinates.
[Link]/berkeley-stat-157
Flashback - Applying Dropout
• Often apply dropout on the output of hidden fully-
connected layers
h = σ(W1x + b1)
h′ = dropout(h)
o = W2h′ + b2
y = softmax(o)
• At inference time dropout is inactive, i.e. h′ = dropout(h)
[Link]/berkeley-stat-157
Variational Dropout (Gal & Ghahramani, 2015)
• Regular Dropout • Same mask across all
• Use only per time slice time windows
• Different mask per • Use also for temporal
slice aspect
[Link]/berkeley-stat-157
Zoneout (Krueger et al., 2016)
• Robustness against skipping
observations in sequence
• Robustness of state representation
relative to hidden state updates
• Skip hidden state update and keep
the same as previously during
training
ht = ht−1
[Link]/berkeley-stat-157
Many more tricks
• Parameter averaging (Merity et al., 2017)
Train RNN and average weights over run
• Stochastic Weight Averaging (Wilson et al., 2018)
Same approach but keep on changing learning rate
• Fraternal Dropout (Zolna et al., 2017)
Dropout while minimizing variation between outputs to
increase robustness to parametrization
[Link]/berkeley-stat-157