RNN: For what type of data?

• Sequence data modelling

• Time series analysis

Typically 𝒕𝒏−𝟏 depends on data before it, after it or both
• Language processing

𝒕𝟏 𝒕𝟐 𝒕𝟑 … . 𝒕𝒏−𝟏 𝒕𝒏

2
Language Modelling

3
Language Modelling

4
You use Language Models every day!

5
n-gram Language Models
• Question: How to learn a Language Model?
• Answer (pre- Deep Learning): learn an n-gram Language Model!
• Definition: A n-gram is a chunk of n consecutive words.
• unigrams: “the”, “students”, “opened”, ”their”
• bigrams: “the students”, “students opened”, “opened their”
• trigrams: “the students opened”, “students opened their”
• 4-grams: “the students opened their”
• Idea: Collect statistics about how frequent different n-grams are and
use these to predict next word.

6
Language Models: Example
Suppose we are learning a 4-gram Language Model.

7
n-gram Language Models
Markov assumption: 𝑥 (𝑡+1) depends only on the preceding n-1 words

Question: How do we get these n-gram and (n-1)-gram probabilities?

Answer: By counting them in some large corpus of text!

8
Sparsity Problems with n-gram LMs

9
Storage Problems with n-gram LMs

10
n-gram Language Models in practice
• You can build a simple trigram Language Model over a1.7 million
word corpus (Reuters) in a few seconds on your laptop

11
Generating text with a n-gram LM
You can also use a Language Model to generate text

12
Generating text with a n-gram LM
You can also use a Language Model to generate text

13
Generating text with a n-gram LM
You can also use a Language Model to generate text

14
Generating text with a n-gram LM
You can also use a Language Model to generate text

15
How to build a Neural Language Model?
• Language Modelling Task
• Input: Sequence of words 𝑥 (1) , 𝑥 (2) , … , 𝑥 (𝑡)
• Output: Prob. Dist. of next words 𝑃 𝑥 𝑡+1 𝑥 1
,𝑥 2
,…,𝑥 𝑡

16
How to build a Neural Language Model?

Word2vec,
Glove, FastText

17
 Fixed Window Neural Language Model
• Improvements over n-gram LM
• No sparsity problem
We need a neural • Don’t need to store all observed n-
architecture that grams
can process any • Remaining problems
length input • Fixed window is too small
• Enlarging window enlarges 𝑊
• Window can never be large
enough!
• 𝑥 (1) and 𝑥 (2) are multiplied by
completely different weights in 𝑊
• What about 𝑥 (2) followed by 𝑥 (1) ?

18
 Recurrent Neural Network (RNN)
𝑦ො 𝑡

𝑊 ℎ 𝑡

𝑥 𝑡

19
RNN Language Model

Recurrence

20
Some Observations
• Same set of parameters (𝑊ℎ , 𝑊𝑒 , 𝑏1 ) in all time steps
• We can model sequence of arbitrary size
• No of parameters remains to be the same
• The input word represented as one-hot vector
• One hot vector: 𝑥 ∈ {0,1}|𝑉| (only one component 1)
• The matrix E (Embedding matrix) holds dense representation of words
• Computed through methods like word2vec etc.
• Dimension of E: ℝ𝑑× 𝑉 𝑑 ≪ |𝑉|
• Fetching the embedding vector
• 𝑒 = 𝐸𝑥

21
Training an RNN Language Model
A BIG text corpus (all the Wikipedia or All the Web)
1.0
Prediction Target (next word)
𝑦ො (1) ≈? 𝑦 (1) Cross-Entropy
0.0
𝑦ො (1) 𝑦 (1)

𝑥 (1) 𝑥 (2) 𝑥 (3) 𝑥 (𝑡) 𝑥 (𝑡+1) 𝑥 (𝑇)

𝑡=1 What is the input? 𝑥 (1)

What is the target 𝑦 (1) ? 𝑥 (2)
What is 𝑦ො (1) ? A Probability Distrbution
What is the Loss? 𝑦 (1) ≈ 𝑦ො (1)
22
 Training an RNN Language Model
1.0

Cross-Entropy One-hot for 𝑥 (𝑡+1)

0.0

𝑦ො (𝑡) 𝑦 (𝑡)

𝑥 (1) = 𝑘𝑔𝑝 𝑥 (2) = 𝑖𝑠 𝑦ො (1) = [0.6 0.3 0.1] 𝑦 (1) = [0 1 0]

𝐽1 𝜃
𝑉 = {𝑘𝑔𝑝, 𝑖𝑠, 𝑔𝑟𝑒𝑎𝑡} = − 0 × log 0.6 + 1 × log 0.3 + 0 × log 0.1
= − log 0.3 23
Training an RNN Language Model
1.0

Cross-Entropy One-hot for 𝑥 (𝑡+1)

0.0

𝑦ො (𝑡) 𝑦 (𝑡)

Average to compute the training loss for entire dataset

24
Training an RNN Language Model

25
Training an RNN Language Model

26
Training an RNN Language Model

27
Training an RNN Language Model

28
Training an RNN Language Model

Teacher Forcing

29
Training an RNN Language Model

Backpropagation Through Time (BPTT)

30
Perplexity: Evaluating Language Models
Normalized by
number of words

Inverse probability of
the training corpus
Perplexity: the exponential of the cross-entropy loss

31
Other Applications of RNN Model
Sequence Tagging

Part-of-speech tagging, named entity recognition

32
RNN: Sentence Classification

33
RNN: Sentence Classification

34
RNN: Vanishing Gradient Problem

35
RNN: Vanishing Gradient Problem

36
RNN: Vanishing Gradient Problem

37
RNN: Vanishing Gradient Problem

38
RNN: Vanishing Gradient Problem

39
RNN: Vanishing Gradient Problem

40
Why is vanishing gradient a problem?

41
Self test
• Explain how an RNN differs from a feedforward neural network in
terms of information flow. Why is the “hidden state” important in
RNNs, and how does it help the model process sequential data?

42
Effect of vanishing gradient on RNN-LM
• LM task: When she tried to print her tickets, she found that the
printer was out of toner. She went to the stationery store to buy more
toner. It was very overpriced. After installing the toner into the
printer, she finally printed her ________
• To learn from this training example, the RNN-LM needs to model the
dependency between “tickets” on the 7th step and the target word
“tickets” at the end.
• But if gradient is small, the model can’t learn this dependency
• So, the model is unable to predict similar long-distance dependencies at test
time

43
Thank you!

44