Contents
• Recurrent Neural Network
• RNN Cell
• RNN Applications
• Types of RNN
• RNN Single Cell (Revisited)
• RNN back propagation
College of Computer Science and Engineering University of
1
Jeddah
RNN Cell
• Assume that the input to RNN is x1
• Hidden layer vector from last stage h0
• Hidden layer vector to next stage h1
h1 = tanh( hhh0 + h 1)
College of Computer Science and Engineering
University of Jeddah 2
𝑥
𝑊
𝑊
𝑥
RNN Cell
• Assume that the input to RNN is x1
• Hidden layer vector from last stage h0
• Hidden layer vector to next stage h1
h1 = tanh( hhh0 + h 1)
1 = ( h h1)
College of Computer Science and Engineering
University of Jeddah 3
𝑦
𝑥
𝑦
𝑠
𝑜
𝑓
𝑡
𝑚
𝑊
𝑎
𝑥
𝑊
𝑊
𝑥
RNN Cell
• Assuming some values of Whh, Whx and Why
h1 = tanh( hhh0 + h 1)
College of Computer Science and Engineering
1 = ( h h1) University of Jeddah 4
𝑦
𝑥
𝑦
𝑠
𝑜
𝑓
𝑡
𝑚
𝑊
𝑎
𝑥
𝑊
𝑊
𝑥
RNN (unrolled)
• Assuming some values of Whh, Whx and Why
h1 = tanh( hhh0 + h 1)
College of Computer Science and Engineering
1 = ( h h1) University of Jeddah 5
𝑦
𝑥
𝑦
𝑠
𝑜
𝑓
𝑡
𝑚
𝑊
𝑎
𝑥
𝑊
𝑊
𝑥
RNN (Applications)
College of Computer Science and Engineering
University of Jeddah 6
RNN (Applications)
College of Computer Science and Engineering
University of Jeddah 7
Tagging a Text Sequence (One to One Sequence Mapping)
Training examples don’t have to be of same length
Input Length Output
My car Works 3 Possessive, noun, verb
My dog ate my assignment 5 Possessive, noun, verb, pronoun, noun
My mother saved the day 5 Possessive, noun, verb, pronoun, noun
The smart kid solved the problem 6 Pronoun, qualifier, noun, verb, pronoun, noun
College of Computer Science and Engineering
University of Jeddah 8
Tagging a Text Sequence (One to One Sequence Mapping)
Assuming a vocabulary of 1000 words and 20 output tags
Sentence Input Output
My car Works T=1000x3 T=20x3
My dog ate my assignment T=1000x5 T=20x5
My mother saved the day T=1000x5 T=20x5
The smart kid solved the problem T=1000x6 T=20x6
Question: How do we created batches if inputs and outputs have different size?
Answer 1: We don’t. process one by one.
Answer 2: Zero padding. We can make all vectors T= 1000 x 6 i.e. T = 4 x 1000 x 6
College of Computer Science and Engineering
University of Jeddah 9
RNN (Applications)
College of Computer Science and Engineering
University of Jeddah 10
Scoring sentiment of text (Many to One Sequence Mapping)
Training examples don’t have to be of same length
Input Length Output
This restaurant has good food 5 Positive
This restaurant is bad 4 Negative
This restaurant is the best 5 Positive
This restaurant is on Tahlia Street 6 Neutral
College of Computer Science and Engineering
University of Jeddah 11
RNN (Applications)
College of Computer Science and Engineering
University of Jeddah 12
Machine Translation (Enc-Dec- Many to Many Sequence Mapping)
Training
Input training samples do not have to be of the same length !
Input Output
este restaurante tiene buena comida This restaurant has good food
el mundo no es suficiente The world is not enough
College of Computer Science and Engineering
University of Jeddah 13
Machine Translation (Enc-Dec- Many to Many Sequence Mapping)
During Training
College of Computer Science and Engineering
University of Jeddah 14
RNN (Applications)
College of Computer Science and Engineering
University of Jeddah 15
Text Generation (Many to Many Sequence Mapping)
College of Computer Science and Engineering
University of Jeddah 16
Text Generation (Many to Many Sequence Mapping)
College of Computer Science and Engineering
University of Jeddah 17
Text Generation (Many to Many Sequence Mapping)
College of Computer Science and Engineering
University of Jeddah 18
Text Generation (Many to Many Sequence Mapping)
College of Computer Science and Engineering
University of Jeddah 19
Text Generation (Many to Many Sequence Mapping)
College of Computer Science and Engineering
University of Jeddah 20
Text Generation (Many to Many Sequence Mapping)
College of Computer Science and Engineering
University of Jeddah 21
Types of RNNs (Bidirectional RNNs)
College of Computer Science and Engineering
University of Jeddah 22
Types of RNNs (Stacked RNNs)
College of Computer Science and Engineering
University of Jeddah 23
RNN
College of Computer Science and Engineering
University of Jeddah 24
RNN
College of Computer Science and Engineering
University of Jeddah 25
RNN
College of Computer Science and Engineering
University of Jeddah 26
RNN
College of Computer Science and Engineering
University of Jeddah 27
RNN
College of Computer Science and Engineering
University of Jeddah 28
RNN: Computational Graph
College of Computer Science and Engineering
University of Jeddah 29
RNN: Computational Graph
College of Computer Science and Engineering
University of Jeddah 30
RNN: Computational Graph
College of Computer Science and Engineering
University of Jeddah 31
RNN: Computational Graph - Many to Many
College of Computer Science and Engineering
University of Jeddah 32
RNN: Computational Graph - Example
College of Computer Science and Engineering
University of Jeddah 33
RNN: Computational Graph - Example
College of Computer Science and Engineering
University of Jeddah 34
RNN: Computational Graph - Example
College of Computer Science and Engineering
University of Jeddah 35
RNN: Computational Graph - Example
College of Computer Science and Engineering
University of Jeddah 36
RNN: Image Captioning
College of Computer Science and Engineering
University of Jeddah 37
RNN: Image Captioning
College of Computer Science and Engineering
University of Jeddah 38