Scribd
Upload
23 views11 pages

Gated RNNs: Training and Architectures

The document discusses the challenges of training Recurrent Neural Networks (RNNs), including slow training and issues with exploding/vanishing gradients. It presents remedies such as using ReLU activation, truncated backpropagation through time (BPTT), and changing architectures to Long Short-Term Memory (LSTM) or Gated Recurrent Units (GRU). Additionally, it explains the structure and function of LSTM and GRU cells in managing long-term dependencies.

Uploaded by

rliuna
Copyright
© All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
23 views11 pages

Gated RNNs: Training and Architectures

The document discusses the challenges of training Recurrent Neural Networks (RNNs), including slow training and issues with exploding/vanishing gradients. It presents remedies such as using ReLU activation, truncated backpropagation through time (BPTT), and changing architectures to Long Short-Term Memory (LSTM) or Gated Recurrent Units (GRU). Additionally, it explains the structure and function of LSTM and GRU cells in managing long-term dependencies.

Uploaded by

rliuna
Copyright
© All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd

Gated RNNs

Geena Kim

University of Colorado Boulder


How hard is it to train an RNN?

• Slow to train (TBPTT helps)

• RNN can suffer exploding/vanishing gradient

• First or early memory or info get lost through the time step
Remedies

• ReLU activation function


• Truncated BPTT
• Clip gradient
• Use learning rate scheduling
• Add residual connection
• Change architectures- LSTM, GRU
Long-term dependencies

• Skip connections

• Leaky units
Long Short-Term Memory cell
What is LSTM cell?

A Vania RNN cell An LSTM cell

RNN Ct-1 Ct
ht-1 ht LSTM
Cell Cell
ht-1 ht

Xt
Xt
Inside the LSTM cell

Ct-1 * + Ct

LSTM Cell

ht-1 * ht

Xt
Inside the LSTM cell

C * + Ct
t-
1 * *
LSTM Cell
f i g o
h ht
t-
1
Xt
Gated Recurrent Unit (GRU)

ht-1 + ht
* *
LSTM Cell 1- *

Xt
Gated Recurrent Unit (GRU)

ht-1 g + ht
* *
LSTM Cell 1- *
r z

Xt
Gated Recurrent Unit (GRU)

ht- g + ht
1 * *
LSTM Cell 1- *
r z

Xt

You might also like