Conditional
Language Modeling
Chris Dyer
Review: Unconditional LMs
A language model assigns probabilities to sequences of
words, w = (w1 , w2 , . . . , w` ).
We saw that it is helpful to decompose this probability
using the chain rule, as follows:
p(w) = p(w1 ) ⇥ p(w2 | w1 ) ⇥ p(w3 | w1 , w2 ) ⇥ · · · ⇥
p(w` | w1 , . . . , w` 1)
|w|
Y
= p(wt | w1 , . . . , wt 1)
t=1
This reduces the language modeling problem to modeling
the probability of the next word, given the history of
preceding words.
Unconditional LMs with RNNs
random variable
p(W5 |w1 ,w2 ,w3 ,w4 )
z }| {
RNN hidden state
vector, length=|vocab| softmax
h1 h2 h3 h4
h0 w1 w2 w3 w4
w1 w2 w3 w4
observed
vector
context word (word embedding)
Conditional LMs
A conditional language model assigns probabilities to
sequences of words, w = (w1 , w2 , . . . , w` ) , given some
conditioning context, x .
As with unconditional models, it is again helpful to use
the chain rule to decompose this probability:
Ỳ
p(w | x) = p(wt | x, w1 , w2 , . . . , wt 1 )
t=1
What is the probability of the next word, given the history of
previously generated words and conditioning context x?
Conditional LMs
x “input” w “text output”
An author A document written by that author
A topic label An article about that topic
{SPAM, NOT_SPAM} An email
A sentence in French Its English translation
A sentence in English Its French translation
A sentence in English Its Chinese translation
An image A text description of the image
A document Its summary
A document Its translation
Meterological measurements A weather report
Acoustic signal Transcription of speech
Conversational history + database Dialogue system response
A question + a document Its answer
A question + an image Its answer
Conditional LMs
x “input” w “text output”
An author A document written by that author
A topic label An article about that topic
{SPAM, NOT_SPAM} An email
A sentence in French Its English translation
A sentence in English Its French translation
A sentence in English Its Chinese translation
An image A text description of the image
A document Its summary
A document Its translation
Meterological measurements A weather report
Acoustic signal Transcription of speech
Conversational history + database Dialogue system response
A question + a document Its answer
A question + an image Its answer
Conditional LMs
x “input” w “text output”
An author A document written by that author
A topic label An article about that topic
{SPAM, NOT_SPAM} An email
A sentence in French Its English translation
ek
A sentence in English Its French translation
this we
A sentence in English Its Chinese translation
An image A text description of the image
A document Its summary
A document Its translation
Meterological measurements A weather report
e k
weAcoustic signal Transcription of speech
x t
ne
Conversational
k s history + database Dialogue system response
e e
o w A question + a document Its answer
tw
A question + an image Its answer
Data for training conditional LMs
To train conditional language models, we need paired
N
samples, {(x i , w i i=1.
)}
Data availability varies. It’s easy to think of tasks that
could be solved by conditional language models, but the
data just doesn’t exist.
Relatively large amounts of data for:
Translation, summarisation, caption generation,
speech recognition
Algorithmic challenges
We often want to find the most likely w given some x . This
is unfortunately generally an intractable problem.
⇤
w = arg max p(w | x)
w
We therefore approximate it using a beam search or with
(i)
Monte Carlo methods since w ⇠ p(w | x) is often
computationally easy.
Improving search/inference is an open research question.
How can we search more effectively?
Can we get guarantees that we have found the max?
Can we limit the model a bit to make search easier?
Evaluating conditional LMs
How good is our conditional language model?
These are language models, we can use cross-entropy
or perplexity. okay to implement, hard to interpret
Task-specific evaluation. Compare the model’s most likely
output to human-generated expected output using a
task-specific evaluation metric L .
w⇤ = arg max p(w | x) ⇤
L(w , wref )
w
Examples of L : BLEU, METEOR, WER, ROUGE.
easy to implement, okay to interpret
Human evaluation.
hard to implement, easy to interpret
Evaluating conditional LMs
How good is our conditional language model?
These are language models, we can use cross-entropy
or perplexity. okay to implement, hard to interpret
Task-specific evaluation. Compare the model’s most likely
output to human-generated expected output using a
task-specific evaluation metric L .
w⇤ = arg max p(w | x) ⇤
L(w , wref )
w
Examples of L : BLEU, METEOR, WER, ROUGE.
easy to implement, okay to interpret
Human evaluation.
hard to implement, easy to interpret
Lecture overview
The rest of this lecture will look at “encoder-decoder”
models that learn a function that maps x into a fixed-size
vector and then uses a language model to “decode”
that vector into a sequence of words, w.
x Kunst kann nicht gelehrt werden…
w Artistry can’t be taught…
Lecture overview
The rest of this lecture will look at “encoder-decoder”
models that learn a function that maps x into a fixed-size
vector and then uses a language model to “decode”
that vector into a sequence of words, w.
w A dog is playing on the beach.
Lecture overview
• Two questions
• How do we encode x as a fixed-size vector, c ?
- Problem (or at least modality) specific
- Think about assumptions
• How do we condition on c in the decoding
model?
- Less problem specific
- We will review solution/architectures
Kalchbrenner and Blunsom 2013
Encoder
c = embed(x)
s = Vc
Recurrent connection
Embedding of wt 1
Recurrent decoder
Source sentence
ht = g(W[ht 1 ; wt 1 ] + s + b])
0 Learnt bias
ut = Pht + b
p(Wt | x, w<t ) = softmax(ut )
Recall unconditional RNN
ht = g(W[ht 1 ; wt 1 ] + b])
K&B 2013: Encoder
How should we define c = embed(x)?
The simplest model possible:
X
c= xi
i
x1 x2 x3 x4 x5 x6
x1 x2 x3 x4 x5 x6
What do you think of this model?
K&B 2013: CSM Encoder
How should we define c = embed(x)?
Convolutional sentence model (CSM)
K&B 2013: CSM Encoder
• Good
• Convolutions learn interactions among features in a local
context
• By stacking them, longer range dependencies can be learnt
• Deep ConvNets have a branching structure similar to trees,
but no parser is required
• Bad
• Sentences have different lengths, need different depth trees;
convnets are not usually so dynamic, but see*
* Kalchbrenner et al. (2014). A convolutional neural network for modelling sentences. In Proc. ACL.
K&B 2013: RNN Decoder
Encoder
c = embed(x)
s = Vc
Recurrent connection
Embedding of wt 1
Recurrent decoder
Source sentence
ht = g(W[ht 1 ; wt 1 ] + s + b])
0 Learnt bias
ut = Pht + b
p(Wt | x, w<t ) = softmax(ut )
Recall unconditional RNN
ht = g(W[ht 1 ; wt 1 ] + b])
K&B 2013: RNN Decoder
h1
h0 x1
<s>
K&B 2013: RNN Decoder
p̂1
softmax
h1
h0 x1
<s>
K&B 2013: RNN Decoder
p(tom | s, hsi)
tom
⇠
p̂1
softmax
h1
h0 x1
<s>
K&B 2013: RNN Decoder
p(tom | s, hsi) ⇥p(likes | s, hsi, tom)
tom likes
⇠
p̂1
softmax softmax
h1 h2
h0 x1 x2
<s>
K&B 2013: RNN Decoder
p(tom | s, hsi) ⇥p(likes | s, hsi, tom)
⇥p(beer | s, hsi, tom, likes)
s
tom likes beer
⇠
⇠
p̂1
softmax softmax softmax
h1 h2 h3
h0 x1 x2 x3
<s>
K&B 2013: RNN Decoder
p(tom | s, hsi) ⇥p(likes | s, hsi, tom)
⇥p(beer | s, hsi, tom, likes)
⇥p(h\si | s, hsi, tom, likes, beer)
s
tom likes beer </s>
⇠
⇠
p̂1
softmax softmax softmax softmax
h1 h2 h3 h4
h0 x1 x2 x3 x4
<s>
Sutskever et al. (2014)
LSTM encoder
(c0 , h0 ) are parameters
(ci , hi ) = LSTM(xi , ci 1 , hi 1)
The encoding is (c` , h` ) where ` = |x| .
LSTM decoder
w0 = hsi
(ct+` , ht+` ) = LSTM(wt 1 , ct+` 1 , ht+` 1 )
ut = Pht+` + b
p(Wt | x, w<t ) = softmax(ut )
Sutskever et al. (2014)
START
Aller Anfang ist schwer STOP
Sutskever et al. (2014)
Beginnings
START
Aller Anfang ist schwer STOP
Sutskever et al. (2014)
Beginnings are
START
Aller Anfang ist schwer STOP
Sutskever et al. (2014)
Beginnings are difficult
START
Aller Anfang ist schwer STOP
Sutskever et al. (2014)
Beginnings are difficult STOP
START
Aller Anfang ist schwer STOP
Sutskever et al. (2014)
• Good
• RNNs deal naturally with sequences of various lengths
• LSTMs in principle can propagate gradients a long
distance
• Very simple architecture!
• Bad
• The hidden state has to remember a lot of information!
(We will return to this problem on Thursday.)
Sutskever et al. (2014): Tricks
Beginnings are difficult STOP
START
Aller Anfang ist schwer STOP
Sutskever et al. (2014): Tricks
Read the input sequence “backwards”: +4 BLEU
Beginnings are difficult STOP
START
Aller Anfang ist schwer STOP
Sutskever et al. (2014): Tricks
Use an ensemble of J independently trained models.
Ensemble of 2 models: +3 BLEU
Ensemble of 5 models: +4.5 BLEU
Decoder:
(j) (j) (j) (j) (j)
(ct+` , ht+` ) = LSTM (wt 1 , ct+` 1 , ht+` 1 )
(j) (j) (j)
ut = Pht +b
XJ
1 (j 0 )
ut = u
J 0
j =1
p(Wt | x, w<t ) = softmax(ut )
A word about decoding
In general, we want to find the most probable (MAP) output
given the input, i.e.
w⇤ = arg max p(w | x)
w
|w|
X
= arg max log p(wt | x, w<t )
w
t=1
A word about decoding
In general, we want to find the most probable (MAP) output
given the input, i.e.
w⇤ = arg max p(w | x)
w
|w|
X
= arg max log p(wt | x, w<t )
w
t=1
This is, for general RNNs, a hard problem. We therefore
approximate it with a greedy search:
w1⇤ = arg max p(w1 | x)
w1
w2⇤ = arg max p(w2 | x, w1⇤ )
w2
..
.
wt⇤ = arg max p(wt | x, w⇤<t )
w2
A word about decoding
In general, we want to find the most probable (MAP) output
given the input, i.e.
w⇤ = arg max p(w | x)
w
|w|
X
= arg max log p(wt | x, w<t )
w
t=1
undecidable :(
This is, for general RNNs, a hard problem. We therefore
approximate it with a greedy search:
w1⇤ = arg max p(w1 | x)
w1
w2⇤ = arg max p(w2 | x, w1⇤ )
w2
..
.
wt⇤ = arg max p(wt | x, w⇤<t )
w2
A word about decoding
A slightly better approximation is to use a beam search with
beam size b. Key idea: keep track of top b hypothesis.
E.g., for b=2:
x = Bier trinke ich
beer drink I
hsi
logprob=0
w0 w1 w2 w3
A word about decoding
A slightly better approximation is to use a beam search with
beam size b. Key idea: keep track of top b hypothesis.
E.g., for b=2:
x = Bier trinke ich
beer drink I
beer
hsi logprob=-1.82
logprob=0
I
logprob=-2.11
w0 w1 w2 w3
A word about decoding
A slightly better approximation is to use a beam search with
beam size b. Key idea: keep track of top b hypothesis.
E.g., for b=2:
x = Bier trinke ich drink
beer drink I logprob=-6.93
beer I
hsi logprob=-1.82 logprob=-5.80
logprob=0
I
logprob=-2.11
w0 w1 w2 w3
A word about decoding
A slightly better approximation is to use a beam search with
beam size b. Key idea: keep track of top b hypothesis.
E.g., for b=2:
x = Bier trinke ich drink
beer drink I logprob=-6.93
beer I
hsi logprob=-1.82 logprob=-5.80
logprob=0
I beer
logprob=-2.11 logprob=-8.66
drink
logprob=-2.87
w0 w1 w2 w3
A word about decoding
A slightly better approximation is to use a beam search with
beam size b. Key idea: keep track of top b hypothesis.
E.g., for b=2:
x = Bier trinke ich drink
beer drink I logprob=-6.93
beer I
hsi logprob=-1.82 logprob=-5.80
logprob=0
I beer
logprob=-2.11 logprob=-8.66
drink
logprob=-2.87
w0 w1 w2 w3
A word about decoding
A slightly better approximation is to use a beam search with
beam size b. Key idea: keep track of top b hypothesis.
E.g., for b=2:
x = Bier trinke ich drink drink
beer drink I logprob=-6.93 logprob=-6.28
beer I like
hsi logprob=-1.82 logprob=-5.80 logprob=-7.31
logprob=0
I beer beer
logprob=-2.11 logprob=-8.66 logprob=-3.04
drink wine
logprob=-2.87 logprob=-5.12
w0 w1 w2 w3
A word about decoding
A slightly better approximation is to use a beam search with
beam size b. Key idea: keep track of top b hypothesis.
E.g., for b=2:
x = Bier trinke ich drink drink
beer drink I logprob=-6.93 logprob=-6.28
beer I like
hsi logprob=-1.82 logprob=-5.80 logprob=-7.31
logprob=0
I beer beer
logprob=-2.11 logprob=-8.66 logprob=-3.04
drink wine
logprob=-2.87 logprob=-5.12
w0 w1 w2 w3
A word about decoding
A slightly better approximation is to use a beam search with
beam size b. Key idea: keep track of top b hypothesis.
E.g., for b=2:
x = Bier trinke ich drink drink
beer drink I logprob=-6.93 logprob=-6.28
beer I like
hsi logprob=-1.82 logprob=-5.80 logprob=-7.31
logprob=0
I beer beer
logprob=-2.11 logprob=-8.66 logprob=-3.04
drink wine
logprob=-2.87 logprob=-5.12
w0 w1 w2 w3
Sutskever et al. (2014): Tricks
Use beam search: +1 BLEU
x = Bier trinke ich drink drink
beer drink I logprob=-6.93 logprob=-6.28
beer I like
hsi logprob=-1.82 logprob=-5.80 logprob=-7.31
logprob=0
I beer beer
logprob=-2.11 logprob=-8.66 logprob=-3.04
drink wine
logprob=-2.87 logprob=-5.12
w0 w1 w2 w3
Image caption generation
• Neural networks are great for working with multiple
modalities—everything is a vector!
• Image caption generation can therefore use the same
techniques as translation modeling
• A word about data
• Relatively few captioned images are available
• Pre-train image embedding model using another
task, like image identification (e.g., ImageNet)
Kiros et al. (2013)
• Looks a lot like Kalchbrenner and Blunsom (2013)
• convolutional network on the input
• n-gram language model on the output
• Innovation: multiplicative interactions in the
decoder n-gram model
Kiros et al. (2013)
Encoder x = embed(x)
Kiros et al. (2013)
Encoder x = embed(x)
Unconditional n-gram LM: Embedding of wt 1
ht = W[wt n+1 ; wt n+2 ; . . . ; wt 1]
ut = Pht + b
p(Wt | x, wtt 1
n+1 ) = softmax(ut )
Kiros et al. (2013)
Encoder x = embed(x)
Simple conditional n-gram LM:
ht = W[wt n+1 ; wt n+2 ; . . . ; wt 1 ] +Cx
ut = Pht + b
p(Wt | x, wtt 1
n+1 ) = softmax(ut )
Kiros et al. (2013)
Encoder x = embed(x)
Simple conditional n-gram LM:
ht = W[wt n+1 ; wt n+2 ; . . . ; wt 1 ] +Cx
ut = Pht + b
p(Wt | x, wtt 1
n+1 ) = softmax(ut )
Multiplicative n-gram LM:
wi = ri,w
Kiros et al. (2013)
Encoder x = embed(x)
Simple conditional n-gram LM:
ht = W[wt n+1 ; wt n+2 ; . . . ; wt 1 ] +Cx
ut = Pht + b
p(Wt | x, wtt 1
n+1 ) = softmax(ut )
Multiplicative n-gram LM:
wi = ri,w
wi = ri,j,w xj
Kiros et al. (2013)
Encoder x = embed(x)
Simple conditional n-gram LM:
ht = W[wt n+1 ; wt n+2 ; . . . ; wt 1 ] +Cx
ut = Pht + b
p(Wt | x, wtt 1
n+1 ) = softmax(ut )
Multiplicative n-gram LM:
wi = ri,w how big is this tensor?
wi = ri,j,w xj
Kiros et al. (2013)
Encoder x = embed(x)
Simple conditional n-gram LM:
ht = W[wt n+1 ; wt n+2 ; . . . ; wt 1 ] +Cx
ut = Pht + b
p(Wt | x, wtt 1
n+1 ) = softmax(ut )
Multiplicative n-gram LM:
wi = ri,w
Kiros et al. (2013)
Encoder x = embed(x)
Simple conditional n-gram LM:
ht = W[wt n+1 ; wt n+2 ; . . . ; wt 1 ] +Cx
ut = Pht + b
p(Wt | x, wtt 1
n+1 ) = softmax(ut )
Multiplicative n-gram LM:
wi = ri,w
wi = ri,j,w xj
Kiros et al. (2013)
Encoder x = embed(x)
Simple conditional n-gram LM:
ht = W[wt n+1 ; wt n+2 ; . . . ; wt 1 ] +Cx
ut = Pht + b
p(Wt | x, wtt 1
n+1 ) = softmax(ut )
Multiplicative n-gram LM:
wi = ri,w
wi = ri,j,w xj
what’s the intuition here?
Kiros et al. (2013)
Encoder x = embed(x)
Simple conditional n-gram LM:
ht = W[wt n+1 ; wt n+2 ; . . . ; wt 1 ] +Cx
ut = Pht + b
p(Wt | x, wtt 1
n+1 ) = softmax(ut )
Multiplicative n-gram LM:
wi = ri,w how big is this tensor?
wi = ri,j,w xj
Kiros et al. (2013)
Encoder x = embed(x)
Simple conditional n-gram LM:
ht = W[wt n+1 ; wt n+2 ; . . . ; wt 1 ] +Cx
ut = Pht + b
p(Wt | x, wtt 1
n+1 ) = softmax(ut )
Multiplicative n-gram LM:
wi = ri,w
wi = ri,j,w xj
Kiros et al. (2013)
Encoder x = embed(x)
Simple conditional n-gram LM:
ht = W[wt n+1 ; wt n+2 ; . . . ; wt 1 ] +Cx
ut = Pht + b
p(Wt | x, wtt 1
n+1 ) = softmax(ut )
Multiplicative n-gram LM:
wi = ri,w
wi = ri,j,w xj
|V |⇥d d⇥k
wi = uw,i vi,j (U 2 R , V2R )
rt = W[wt n+1 ; wt n+2 ; . . . ; wt 1 ] + Cx
Kiros et al. (2013)
Encoder x = embed(x)
Simple conditional n-gram LM:
ht = W[wt n+1 ; wt n+2 ; . . . ; wt 1 ] +Cx
ut = Pht + b
p(Wt | x, wtt 1
n+1 ) = softmax(ut )
Multiplicative n-gram LM:
|V |⇥d d⇥k
wi = uw,i vi,j (U 2 R , V2R )
rt = W[wt n+1 ; wt n+2 ; . . . ; wt 1 ] + Cx
ht = (Wf r rt ) (Wf x x)
ut = Pht + b
p(Wt | x, w<t ) = softmax(ut )
Kiros et al. (2013)
• Two take-home messages:
• Feed-forward n-gram models can be used in
place of RNNs in conditional models
• Modeling interactions between input modalities
holds a lot of promise
• Although MLP-type models can approximate
higher order tensors, multiplicative models
appear to make learning interactions easier
Questions?