Machine Translation and
Encoder-Decoder Models
Dr M Janaki Meena
Introduction
• Encoder-decoder or sequence-to-sequence models are used
for a different kind of sequence modeling in which the output
sequence is a complex function of the entire input sequencer;
• we must map from a sequence of input words or tokens to a
sequence of tags that are not merely direct mappings from
individual words.
• the words of the target language don’t necessarily agree with
the words of the source language in number or order.
Introduction
• Consider translating the following made-up English sentence
into Japanese.
Introduction
• Note that the elements of the sentences are in very different
places in the different languages.
• In English, the verb is in the middle of the sentence, while in
Japanese, the verb kaita comes at the end.
• The Japanese sentence doesn’t require the pronoun he, while
English does.
• Such differences between languages can be quite complex.
Introduction
• In the following actual sentence from the United Nations,
notice the many changes between the Chinese sentence
(we’ve given in red a word-by-word gloss of the Chinese
characters) and its English equivalent.
Word Order Typology
• As we hinted it in our example above comparing English and
Japanese, languages differ in the basic word order of verbs, subjects,
and objects in simple declarative clauses.
• German, French, English, and Mandarin, for example, are all SVO
(Subject-Verb-Object) languages, meaning that the verb tends to come
between the subject and object.
• Hindi and Japanese, by contrast, are SOV languages, meaning that the
verb tends to come at the end of basic clauses, and Irish and Arabic
are VSO languages.
• Two languages that share their basic word order type often have other
similarities.
• For example, VO languages generally have prepositions, whereas OV
languages generally have postpositions.
Word Order Typology
• SVO English sentence, the verb wrote is followed by its object
a letter and the prepositional phrase to a friend, in which the
preposition to is followed by its argument a friend.
• Arabic, with a VSO order, also has the verb before the object
and prepositions.
• By contrast, in the Japanese example that follows, each of
these orderings is reversed; the verb is preceded by its
arguments, and the postposition follows its argument.
Word Order Typology
• mm
Word Order Typology
• SVO English sentence, the verb wrote is followed by its object
a letter and the prepositional phrase to a friend, in which the
preposition to is followed by its argument a friend.
• Arabic, with a VSO order, also has the verb before the object
and prepositions.
• By contrast, in the Japanese example that follows, each of
these orderings is reversed; the verb is preceded by its
arguments, and the postposition follows its argument.
The Encoder-Decoder Model
The Encoder-Decoder Model
• Encoder-decoder networks consist of three components:
• 1. An encoder that accepts an input sequence, x1n , and
generates a corresponding sequence of contextualized
representations, hn1 . LSTMs, convolutional networks, and
Transformers can all be employed as encoders.
• 2. A context vector, c, which is a function of hn1 , and conveys
the essence of the input to the decoder.
The Encoder-Decoder Model
• A decoder, which accepts c as input and generates an
arbitrary length sequence of hidden states hm1 , from which a
corresponding sequence of output states ym, can be obtained.
• Just as with encoders, decoders can be realized by any kind
of sequence architecture.
Encoder-Decoder with RNNs
• Recall the conditional RNN language model for computing
p(y), the probability of a sequence y.
• Like any language model, we can break down the probability
as follows:
Encoder-Decoder with RNNs
• At a particular time t, we pass the prefix of t − 1 tokens through
the language model, using forward inference to produce a
sequence of hidden states, ending with the hidden state
corresponding to the last word of the prefix.
• We then use the final hidden state of the prefix as our starting
point to generate the next token.
Encoder-Decoder with RNNs
• More formally, if g is an activation function like tanh or ReLU, a
function of the input at time t and the hidden state at time t − 1,
and f is a softmax over the set of possible vocabulary items,
then at time t the output yt and hidden state ht are computed
as:
Encoder-Decoder with RNNs
• We only have to make one slight change to turn this language
model with autoregressive generation into a translation model
that can translate from a source text in one language to a
target text in a second:
• add a sentence separation marker at the end of the source
text, and then simply concatenate the target text.
Encoder-Decoder with RNNs
• If we call the source text x and the target text y, we are
computing the probability p(y|x) as follows:
Encoder-Decoder with RNNs
Encoder-Decoder with RNNs
• Fig. 10.4 shows an English source text (“the green witch
arrived”), a sentence separator token (<s>, and a Spanish
target text (“llegó la bruja verde”).
• To translate a source text, we run it through the network
performing forward inference to generate hidden states until
we get to the end of the source.
• Then we begin autoregressive generation, asking for a word in
the context of the hidden layer from the end of the source
input as well as the end-of-sentence marker.
Encoder-Decoder with RNNs
• Subsequent words are conditioned on the previous hidden
state and the embedding for the last word generated.
• Let’s formalize and generalize this model a bit in Fig. 10.5. (To
help keep things straight, we’ll use the superscripts e and d
where needed to distinguish the hidden states of the encoder
and the decoder.)
Encoder-Decoder with RNNs
Encoder-Decoder with RNNs
• The elements of the network on the left process the input
sequence x and comprise the encoder.
• While our simplified figure shows only a single network layer
for the encoder, stacked architectures are the norm, where the
output states from the top layer of the stack are taken as the
final representation.
• A widely used encoder design makes use of stacked biLSTMs
where the hidden states from top layers from the forward and
backward passes are concatenated as described in Chapter 9
Encoder-Decoder with RNNs
• The entire purpose of the encoder is to generate a
contextualized representation of the input.
• This representation is embodied in the final hidden state of the
encoder, hen . This representation, also called c for context, is
then passed to the decoder.
• The decoder network on the right takes this state and uses it
to initialize the first hidden state of the decoder.
• That is, the first decoder RNN cell uses c as its prior hidden
d
Encoder-Decoder with RNNs
• The decoder autoregressively generates a sequence of
outputs, an element at a time, until an end-of-sequence
marker is generated.
• Each hidden state is conditioned on the previous hidden state
and the output generated in the previous state.
Encoder-Decoder with RNNs
Encoder-Decoder with RNNs
• One weakness of this approach as described so far is that the
influence of the context vector, c, will wane as the output
sequence is generated.
• A solution is to make the context vector c available at each
step in the decoding process by adding it as a parameter to
the computation of the current hidden state, using the
following equation (illustrated in Fig. 10.6):
Encoder-Decoder with RNNs
• Now we’re ready to see the full equations for this version of
the decoder in the basic encoder-decoder model, with context
available at each decoding timestep.
• Recall that g is a stand-in for some flavor of RNN and ŷt−1 is
the embedding for the output sampled from the softmax at the
previous step:
Encoder-Decoder with RNNs
• Finally, as shown earlier, the output y at each time step
consists of a softmax computation over the set of possible
outputs (the vocabulary, in the case of language modeling or
MT).
• We compute the most likely output at each time step by taking
the argmax over the softmax output:
Training the Encoder-Decoder Model
Attention
• The simplicity of the encoder-decoder model is its clean
separation of the encoder—which builds a representation of
the source text—from the decoder, which uses this context to
generate a target text.
• This final hidden state is thus acting as a bottleneck: it must
represent absolutely everything about the meaning of the
source text, since the only thing the decoder knows about the
source text is what’s in this context vector
Attention
• Information at the beginning of the sentence, especially for
long sentences, may not be equally well represented in the
context vector.
• The attention mechanism is a solution to the bottleneck
problem, a way of allowing the decoder to get information from
all the hidden states of the encoder, not just the last hidden
state.
• In the attention mechanism, as in the vanilla encoder-decoder
model, the context vector c is a single vector that is a function
Attention
• Because the number of hidden states varies with the size of the
input, we can’t use the entire tensor of encoder hidden state
vectors directly as the context for the decoder
• Idea of attention is instead to create the single fixed-length vector
c by taking a weighted sum of all the encoder hidden states.
• Weights focus on (‘attend to’) a particular part of the source text
that is relevant for the token the decoder is currently producing.
• Attention thus replaces the static context vector with one that is
dynamically derived from the encoder hidden states, different for
each token in decoding.
Attention
Attention
• This context vector, ci , is generated a new with each decoding
step i and takes all of the encoder hidden states into account
in its derivation.
• We then make this context available during decoding by
conditioning the computation of the current decoder hidden
state on it (along with the prior hidden state and the previous
output generated by the decoder), as we see in this equation
(and Fig. 10.9):
Attention
Attention
• The first step in computing ci is to compute how much to focus
on each encoder state, how relevant each encoder state is to
the decoder state captured in hdi−1 .
• We capture relevance by computing— at each state i during
decoding—a score(hdi−1 , hej ) for each encoder state j.
• The simplest such score, called dot-product attention,
implements relevance as similarity: measuring how similar the
decoder hidden state is to an encoder hidden state, by
Attention
• Score that results from this dot product is a scalar that reflects
the degree of similarity between the two vectors.
• Vector of these scores across all the encoder hidden states
gives us the relevance of each encoder state to the current
step of the decoder.
• To make use of these scores, we’ll normalize them with a
softmax to create a vector of weights, αij , that tells us the
proportional relevance of each encoder hidden state j to the
d
Attention
• Finally, given the distribution in α, we can compute a fixed-
length context vector for the current decoder state by taking a
weighted average over all the encoder hidden states.
Attention
• With this, we finally have a fixed-length context vector that
takes into account information from the entire encoder state
that is dynamically updated to reflect the needs of the decoder
at each step of decoding.
• Fig. 10.10 illustrates an encoder-decoder network with
attention, focusing on the computation of one context vector
ci .
Attention
Attention
• It’s also possible to create more sophisticated scoring
functions for attention models.
• Instead of simple dot product attention, we can get a more
powerful function that computes the relevance of each
encoder hidden state to the decoder hidden state by
parameterizing the score with its own set of weights, Ws .
Attention
• The weights Ws , which are then trained during normal end-to-
end training, give the network the ability to learn which aspects
of similarity between the decoder and encoder states are
important to the current application.
• This bilinear model also allows the encoder and decoder to
use different dimensional vectors, whereas the simple dot-
product attention requires that the encoder and decoder
hidden states have the same dimensionality.