Explanatory Notes for 6.

3900

Shaunticlair Ruiz

Fall 2024
 Contents

8 Transformers 3
8.0.1 CNNs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
8.0.2 The problem with locality . . . . . . . . . . . . . . . . . . . . . . . . . 4
8.0.3 RNNs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5
8.0.4 Transformers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7
8.1 Vector embeddings and tokens . . . . . . . . . . . . . . . . . . . . . . . . . . 8
8.1.1 One-hot encoding isn’t enough . . . . . . . . . . . . . . . . . . . . . . 8
8.1.2 Word Embeddings: Similarity between words . . . . . . . . . . . . . 8
8.1.3 Vector Similarity: Dot Products . . . . . . . . . . . . . . . . . . . . . . 9
8.1.4 Word2vec . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11
8.1.5 Probability . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11
8.1.6 "Adding" words together . . . . . . . . . . . . . . . . . . . . . . . . . 14
8.1.7 Tokenization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15
8.2 Attention . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16
8.2.1 The Attention Mechanism: queries, keys . . . . . . . . . . . . . . . . 17
8.2.2 The Attention Mechanism: attention weights . . . . . . . . . . . . . . 18
8.2.3 Scaling factor for softmax . . . . . . . . . . . . . . . . . . . . . . . . . 21
8.2.4 The Attention Mechanism: values, attention . . . . . . . . . . . . . . 22
8.2.5 Why we need context . . . . . . . . . . . . . . . . . . . . . . . . . . . 27
8.2.6 Why we need attentive context . . . . . . . . . . . . . . . . . . . . . . 27
8.2.7 Self-attention . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29
8.2.8 Self-attention in matrix form . . . . . . . . . . . . . . . . . . . . . . . 29
8.2.9 Positional Encoding . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32
8.2.10 Masking . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32
8.2.11 Attention Heads . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34
8.3 Transformers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36

1
MIT 6.390 Fall 2024 2

8.3.1 How to create embeddings . . . . . . . . . . . . . . . . . . . . . . . . 36

8.3.2 Attention Heads . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37
8.3.3 Residual Connections . . . . . . . . . . . . . . . . . . . . . . . . . . . 40
8.3.4 Layer Normalization . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41
8.3.5 Feed Forward . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43
8.3.6 Transformer Block . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45
8.3.7 Translation Task: training . . . . . . . . . . . . . . . . . . . . . . . . . 47
8.3.8 Encoder + Decoder Structure . . . . . . . . . . . . . . . . . . . . . . . 49
8.3.9 Predicting a token . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 52
8.3.10 Training Process . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 54
8.3.11 Variations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 54
8.4 Terms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 55

Last Updated: 12/25/24 08:36:30

 CHAPTER 8

Transformers

In this chapter, we want to focus on processing language. In particular:

Definition 1
Natural Language Processing (NLP) is a field of machine learning all about process-
ing, understanding, and using human language.

• Example: Chatbots, language translation, etc.

We’ll start by considering a few candidate models for NLP, before moving to the state-of-
the-art: transformers.

8.0.1 CNNs
In the previous chapter, we introduced the notion of a CNN:

• Convolutional Neural Networks (CNNs) view small regions of data, searching for
patterns across the image.

3
MIT 6.390 Fall 2024 4

In this example, we focus on a 3x3 segment of our data.

This kind of structure is useful for image processing: nearby pixels tend to be related to
each other. They might form a sin-
gle line, or a corner, for
example.
• By prioritizing "nearby" information, we can create models that easily find those lo-
calized patterns.

• We called this property spatial locality.

Concept 2
CNNs are designed to represent locality:

• In a CNN, nearby data is used to search for patterns.

This allows us to use smaller, simpler models:

• Rather than thinking about every possible connection between data, we only
connect "nearby" data. Thus, we need fewer parameters.

8.0.2 The problem with locality

This presents one simple weakness, that we’ve ignored so far:

• If we focus on information that is nearby, we’re missing out on information that’s far
away.

• We need a way to encode "distance" of information, that doesn’t ignore the "distant"
info.

Concept 3
If information is spread over long distances, our CNN model won’t capture it.

• If a pattern is too big for our CNN filter, we’ll have more trouble finding it.

This can become especially problematic for language processing.

Last Updated: 12/25/24 08:36:30

MIT 6.390 Fall 2024 5

Example: Consider the following sentence:

• The sweater that I found in the back of my old closet, which I hadn’t opened since
we moved into the house several years ago, still fits me perfectly.

Note that the beginning and the end of this sentence are linked as a single idea: "The
sweater still fits me perfectly".

• But there’s a huge gap between these phrases: it might be difficult to connect infor-
mation over such a wide gap, while ignoring what’s in-between.

• This also comes up in longer passages: in a paragraph, the first sentence might create
context for the last sentence.

Concept 4
In language, words can be far apart, while still providing important context for the
meaning of the text.

• Thus, language processing is difficult for models which focus too much on local-
ity.

8.0.3 RNNs
One useful observation might be that language tends to be sequential: words come in a very
particular order.

Concept 5
In image processing, we see many pixels at the same time: the whole image is pro-
cessed in parallel.

In language processing, we hear/read words one-by-one, in order: the data has a

sequential structure.

Recurrent Neural Networks (RNNs) are, thus, a sequential model, designed for processing
language.

Last Updated: 12/25/24 08:36:30

MIT 6.390 Fall 2024 6

Each xt is one word in our sentence: we process the text, one word at a time. After every
word, we update our memory ("state" st ). yt is our output at time t.

By storing information about previous words (using a state), our model can "read" each
word in order, while still remembering earlier parts of the text.

• While a CNN can only observe k consecutive pixels/words in a row, our RNN might
be able to contain some information about words that are much further back in time.

How well does this work? RNNs have seen success in the past, but it struggles with for-
getting: our RNN can only store so much information about words it’s seen before.

• As a passage gets longer, our RNN is only paying attention to words it’s seen re-
cently.

Moreover, our RNN doesn’t have any way to choose which words to prioritize: each new
word will have to replace some information about older words.

• So, our RNN naturally prioritizes the most recent words. The more recent words
haven’t been replaced
• But the most recent word isn’t always the most important one, as we saw above (in yet.
the sweater example)!

Concept 6
RNNs (Recurrent Neural Networks) tend to struggle with longer bodies of text:

• The longer we run our RNN, the less it usually remembers about the distant
past.

Moreover, it prioritizes recent words, even when more distant words may be more
important.

In the end, RNNs have, in most language applications, been replaced by transformers: a
different model for language processing. However, some trans-
former models have be-
gun using the concepts
of LSTMs, an RNN vari-
ant. We won’t cover
this topic here.

Last Updated: 12/25/24 08:36:30

MIT 6.390 Fall 2024 7

8.0.4 Transformers
One clever way to think about this problem is to recognize that our goal is to decide which
words are related to each other, whether they’re nearby or far apart.

• In other words, which words should we pay attention to, in order to understand the
text we’re reading?

This is exactly the problem that transformer models solve, using the appropriately named
attention mechanism.

Clarification 7
In this chapter, we’ll use transformers to process language, using the mechanism of
attention.

• But the same tools can be applied to many other problems: image and audio
processing, robotics, etc.

We’ll develop this model in several steps:

• First (11.1), we’ll convert words into vectors. One-hot encoding is too simple, so we’ll
use a different approach: vector embeddings.

• Next (11.2), we’ll figure out which words in a passage are relevant(or connected) to
each other, using a clever system called attention.

• Finally (11.3), we’ll put together these ideas to create a complete model, known as a
transformer.

Last Updated: 12/25/24 08:36:30

MIT 6.390 Fall 2024 8

8.1 Vector embeddings and tokens

8.1.1 One-hot encoding isn’t enough
First, we want to turn words into something computable, like a vector. It’s difficult to try to
do math on the word
The simplest approach would be one-hot encoding. "cheddar". It’s not nu-
merical.
• Example: Suppose that we want to classify furniture as table, bed, couch, or chair.

 
table
 
 bed 

couch
 (8.1)
 
chair

• For each class:

       
0 1 0 0
       
0 0  0 1
vchair =
0
 vtable =
0
 vcouch =
 1
 vbed =
0
 (8.2)
       
1 0 0 0

This approach is simple, but often, it’s too simple.

Concept 8
One-hot encoding loses a lot of information about the objects it’s representing.

• It’s hard to say which words are "similar" to each other, for example.

Example: You probably associate the word "sugar" with "sweet", and "salt" with "savory".

• But, if you use one-hot encoding, all of these words are "equally different". You could shuffle the
rows of one-hot vectors,
and represent the same
        information.
0 1 0 0
So, we can’t use the
       
0 0 0 1
vsalt =
0
 vsavory =
0
 vsugar =
1
 vsweet =
0
 (8.3) order of 1’s and 0’s to
        determine "closeness":
1 0 0 0 the order can be freely
changed.
In order to incorporate this information, we’ll need a better way to represent words as
vectors.

8.1.2 Word Embeddings: Similarity between words

Our new approach will convert each word w into a vector vw of length d. Unlike one-hot encod-
ing, we don’t require
that d equals the size of
Last Updated: 12/25/24 08:36:30 our vocabulary.
MIT 6.390 Fall 2024 9

w −→ vw v w ∈ Rd (8.4)

How do we want to convert words into vectors? Above, we mentioned that one-hot doesn’t
tell us how similar two words are.

Clarification 9
There are many ways for words to be similar: similar word length, similar choice of
letters, etc.

But in our case, we’re interested in semantics: the meanings of the words. We want to
know which words have similar meanings.

• Example: We don’t consider "sugar" and "sweet" to be similar because they both start
with "s".

– They’re similar because of meaning: sugar tastes sweet. Sweet strawberries con-
tain sugar.

Concept 10
We often want our word embeddings vw to tell us which words are semantically
similar to each other: which words have similar meanings.

va and vb are similar vectors ⇐⇒ a and b are semantically similar words

Our goal is to make this statement true. But we have a problem: these are concepts, rather
than computable numbers.

• So, we’ll have to turn each side into something computable.

8.1.3 Vector Similarity: Dot Products

First, we’ll handle the left side: how do we know if vectors are similar?

• We’ve come across this problem multiple times, and we’ll solve it the same way as
always: using the dot product.

Last Updated: 12/25/24 08:36:30

MIT 6.390 Fall 2024 10

Concept 11
Review from the Classification chapter

You can use the dot product between vectors u and v, normalized by their magni-
tudes, to measure their "cosine similarity".

u·v
SC (u, v) =
|u| · |v|

If two vectors are more similar, they have a larger normalized dot product.

• This function ranges from -1 (opposite vectors) to +1 (identical vectors). Perpen-

dicular vectors receive a 0.
We call it "cosine sim-
ilarity", because this is
equal to the cosine of
the angle α between u
and v.

We can see here what we mean by "similar" or "dissimilar".

Clarification 12
You can use SC (u, v) to measure the similarity between two vectors, ignoring magni-
tude.

But for simplicity, we’ll skip the normalizing step, and just take the dot product:

SD (u, v) = u · v = u⊤ v

We’re getting closer to a computable form:

Similar vectors
z }| {
(va · vb ) is large ⇐⇒ a and b are semantically similar words (8.5)

Last Updated: 12/25/24 08:36:30

MIT 6.390 Fall 2024 11

8.1.4 Word2vec
Next, we should get into the math of how to determine which words are likely to be simi-
lar.

• But this is a bit cumbersome, and isn’t really necessary for understanding transform-
ers. The short version: we
expect words which
frequently appear in
So, we relegate this mathematical labor to Appendix D, where we’ll get into the details of
the same contexts, to be
skipgram and word2vec. similar.

Definition 13
We can think of word2vec as a system for word embeddings where words which have
similar meanings, have similar vector embeddings.

• Most commonly, we measure "vector similarity" with the dot product.

Instead, we’ll skip a couple steps, and look at things from a high level.

8.1.5 Probability
Our goal is to be able to numerically talk about the "similarity" or "relatedness" of words.

Above, we represent this with a dot product: this gives us a real number u · v ∈ R.

• This number isn’t very meaningful, though. For example, what does a "similarity of
37" even mean? Is that high? Is that low?

Generally, our best bet for understanding a number like this is to compare it to other num-
bers. You know that someone
who is 6’5" is really tall,
So, let’s think about the relative similarity of words: if we have two words, w1 and w2 , because you know how
tall other people tend to
which one is v more related to? be.
We’ll focus on one simple tool for comparison: probability.

• One way to think about it is, "how likely is wi to be the most relevant word to v, in
any given context?" In skipgram, our prob-
ability comes from ask-
• Alternatively, "how confident are we that these words are actually closely related, ing, "how likely is wi
to show up in the same
compared to others?" context?"

The higher the probability of word w1 , the lower the probability of word w2 , and vice
versa.

Last Updated: 12/25/24 08:36:30

MIT 6.390 Fall 2024 12

We’ll represent this "relatedness of word wi to word v" as probability P wi | v .

Concept 14
One way to describe the relatedness of different words wi is with a probability
P(wi | v).

This has a few advantages:

• A probability is easier to interpret than a real number.

• We can directly compare different words.

• We can systematically convert our dot product to a probability.

This "probability" inter-
pretation is a bit better
How do we turn a real number va · vb into a probability P b | a ?


justified if you read the
skipgram section.
• For a probability, we need to compare b to every other word: this is a multi-class
problem, using the softmax function.

e zk
Softmax(zk ) = P z (8.6)
ie
i

Let’s review the concept behind "softmax":

Definition 15
Suppose that we have n possible words (n "classes"), and we want to figure out which
one is correct.

The kth class has a score, zk , used to compute probability.

• The bigger zk is, the more likely k is to be the correct class.

To keep it positive, zk is converted to ezk : each ezi competes to see which class is more
likely.

• To create a probability, we compare the score of class k to all of our other classes,
using softmax.

All classes
Class k
z}|{ z }| {
X ezk
ezk vs e zi =⇒ Softmax(zk ) = P z
ie
i
i

We repeat this process for every possible word i, to get all of our predictions.

Last Updated: 12/25/24 08:36:30

MIT 6.390 Fall 2024 13

What is our "score" zk ? We could use (va · vb ):

• The higher (va · vb ) is, the more similar/related we expect a and b to be.

• The same is true for zk : if zk is larger, then our probability goes up.

So, we can use our dot product as a "score" zk :

zb = v a · v b (8.7)

We’ll plug this into our probability equation:

Key Equation 16
The more similar (bigger dot product) a and b are, the more likely we predict to find
them together.

• We use a softmax to compute this probability for each possible word b.

eva ·vb
P b a = P v ·v
ie
a i

Or, in alternate notation:

 
exp va · vb
P b a =
P  
i exp va · vi

This kind of interpretation makes our word embeddings a bit more useful.

• Later, we’ll find that it’s the most important part of making transformers work!

Last Updated: 12/25/24 08:36:30

MIT 6.390 Fall 2024 14

8.1.6 "Adding" words together

Our word2vec system works under the hope that these vector embeddings can accurately
represent the meanings of words.

• In practice, this assumption works surprisingly well, for being so simple.

One example is the idea of "adding" words together. Normally, it’s hard to say how to "add
words" together, but we do know how to add vectors.

Consider the following example:

vking − vman + vwoman ≈ vqueen (8.8)

This sort of reasoning makes sense to most english speakers:

ruler
z }| { female ruler
z }| {
vking − vman + vwoman ≈ vqueen (8.9)

We can repeat this process for other words: Paris is the capital of
France, and Rome is the
capital of Italy.
vparis − vfrance + vitaly ≈ vrome (8.10)

Concept 17
Transforming a word into a vector allows you to use vector operations, like addition
and subtraction.

• The result can be surprisingly meaningful, for some word combinations.

This approach doesn’t always work, but the fact that it works sometimes suggests that
our vectors might capture real information about the "meanings" of words.

That said, this approach is often an over-simplification:

Concept 18
Reducing a word to a single vector can cause problems, because the same word might
change its meaning, based on context.

• Example: For example, the word "bank" has a very different meaning when you
compare "bank account" to "river bank".

This idea of "context" is what we hope to solve next.

Last Updated: 12/25/24 08:36:30

MIT 6.390 Fall 2024 15

8.1.7 Tokenization
One clarification, before we move on: so far, we’ve talked about predicting whole words,
because it’s easy to work with.

• But often, for language analysis, we break up words into parts, called tokens.

• These are the objects we study/predict, rather than whole words.

Definition 19
Rather than using/predicting entire words, we use small parts of words, called to-
kens.

• A "token" is the smallest unit in our language model.

• Example: You might break up the word "eating" into "eat" and "ing": both are mean-
ingful, by themselves.

• This process of turning words into tokens is called tokenization.

While "tokens" are used more often than "words", words often make for better examples,
so we’ll keep using them through the rest of this chapter.

Clarification 20
We’ll continue using words (instead of tokens) for examples, when it’s convenient.

Last Updated: 12/25/24 08:36:30

MIT 6.390 Fall 2024 16

8.2 Attention
Our word embedding technique has given us a basic way to talk about which words are
"related".

• We can even use this to learn some about the "meanings" of words.

But there’s some work to be done:

Concept 21
Our word embedding technique has two major problems, for representing the mean-
ings of words:

• There’s a lot of information we’re missing: similarity to other words isn’t enough.
We’ll need a vector to represent that information.

• The meaning of a word is contextual: the sentence you put a word in, will affect
its meaning.

It may not look like it, but our word embedding technique has already given us the basic
tools we need to solve these problems.

Here’s the basic idea, for how we handle each problem:

Concept 22
We’ll create a system that solves both of these problems, using 3 word embeddings: v,
k, and q.

• We’ll embed information about each word in a value vector v.

• When finding the meaning of a word, we’ll calculate context from nearby words.

– We’ll use word similarity to figure out which parts of the context are most
important.

– For this purpose, word will need two embeddings: a key vector k, and a
query vector q.

The result is a powerful model called the attention mechanism.

This description is over-simplified, which is why we’ll need to go into detail below.

Last Updated: 12/25/24 08:36:30

MIT 6.390 Fall 2024 17

8.2.1 The Attention Mechanism: queries, keys

Let’s consider an example, to get used to the idea of k, v, and q.

Suppose we want a general idea of what "mexican" food is like. We’ll need to consider lots
of foods, and take an average of those we consider to be "mexican". Admittedly, we’re turn-
ing "mexican-ness" into
a number, which can be
a bit strange.
This problem comes in three parts: let’s consider the first two, "query" and "key".
It may help to think
this way: "if someone
• Query q: we’re searching for "mexican" food. The word "mexican" is represented by is talking about mexi-
a query vector q. can food, how often are
they talking about this
food?"
– This is like our previous word2vec embedding: if two vectors are similar, then
we expect them to have similar/related meanings.

– So, we’ll compare q to each food, to see which foods are ’close’ to mexican.

Definition 23
The query vector q represents a word, that we’re comparing to several other
words ("keys").

– It answers the question, "what kinds of words are we searching for?"

Using word embeddings, we design q to be "meaningful": similar words, should

have similar vectors.

– And we expect similar words to be more relevant to our query.

• Key k: Each food (apple, burrito, sushi...) has a key vector k, representing it.

– A word2vec-style embedding, just like the query.

– Combining k and q will tell us which foods are ’more’ mexican.

Definition 24
The key vector k represents a word, that we want to compare to the query q.

– It answers the question, "what kinds of searches does this word match"?

Because it’s a word embedding, which encodes meaning, we expect that, if k and
q are similar, then our key word is more relevant to our query.

Each embedding has a role: a query is used to search for relevant words, and a key is
responding to that search, on behalf of one word. Reminder that when
we say "word", we’re
simplifying: we could
Last Updated: 12/25/24 08:36:30 talk about any kind of
token.
MIT 6.390 Fall 2024 18

Concept 25
Another way we could view keys vs. queries:

• Query vector q: asks, "how relevant are these words/tokens to me?"

• Key vector k: asks, "how relevant is my word to the query?"

Notice that we’ve made a perspective shift, in how we view word embeddings:

Concept 26
When we were developing word2vec, we wanted similar vectors to represent seman-
tically similar words.

• But, in this case, we’re less focused on "similarity", than relevance.

We look for keys that are the most relevant to our query.

These two ideas don’t necessarily conflict, but they have somewhat different goals.

8.2.2 The Attention Mechanism: attention weights

How do we compute how similar k and q are? The same way as we did for word2vec: we
use a dot product.

Key Equation 27
We can get a score for how relevant the word b is to word a, by taking the dot product
between b’s key, and a’s query.

q a · kb

We can also write this as matrix multiplication:

q · k = qT k

This gives us a "score": the higher k · q is, the more similar they are.

Last Updated: 12/25/24 08:36:30

MIT 6.390 Fall 2024 19

We convert w1 and wq into a key and query, respectively, before taking the dot product.

Notation 28
Note that k and q have to have the same length: they’re both (dk × 1) column vectors.
If their lengths don’t
mach, we can’t take the
But we’re not just considering one key word: we’re considering all of of them. dot product.

• In our "mexican food" example, we need to check every food, to see which ones best
fit the category.

We re-use our query q for every single dot product.

Notation 29
We have N distinct keys.
In the official notes, we
use n instead of N. This
How do we compare each of these keys? doesn’t affect any of our
math.
• Once again, we’ll reuse a tool from word2vec: softmax.

Last Updated: 12/25/24 08:36:30

MIT 6.390 Fall 2024 20

Key Equation 30
We can compute the relative relevance of a key kj , by:

• Comparing each key ki to q (q · ki )

• Use softmax to compute p(kj |q): given query q, how important is kj ?

eq·kj
P kj q = P q·k
ie
i

P kj q tells you, "how much attention should q pay to kj "?

• Thus, we call P kj q an attention weight.

Finally, we’ve converted each word into their "probability" of being relevant.

One notational thing: we can write this a bit more densely.

• So far, we’ve been computing qT ki for each ki term separately.

• But, one benefit of matrix multiplication, is that we can combine multiple operations
into one.

First, we’ll combine all of our key vectors into a matrix K: This matrix has shape
(N × dk ): the transpose
of what you might ex-
 ⊤ pect.

K = k1 k2 ... kN  (8.11)

 

With that, we can compute all of our dot products at the same time: This product has shape
(1 × N).

Last Updated: 12/25/24 08:36:30

MIT 6.390 Fall 2024 21

 ⊤
q · k1
 q · k2 
 
q ⊤ KT = 
 .. 
 (8.12)
 . 
q · kN

And we can combine all of these together into a softmax.

Key Equation 31
By combining all of our keys into a matrix K, we can compute all of our attention
weights at the same time.


⊤
P k1 q
 
 P k q

2
softmax q⊤ K⊤
 

P K q = 
 ..

 =

 . 


P kN q

It has shape (1 × N).

Note that here, softmax
creates a row vector.

Now, our diagram is visually simpler, though it reflects the same information. "MatMul"
means "Matrix Multiplication".

8.2.3 Scaling factor for softmax

One pragmatic detail. First, let’s quickly define:

Last Updated: 12/25/24 08:36:30

MIT 6.390 Fall 2024 22

Notation 32
Reminder that keys and queries are both vectors of length (dk × 1).

We have one problem: the larger dk is, the more terms in our dot product: our dot product
can grow unreasonably large.

• This can cause computational issues.

√
So, we normalize our dot product by a factor of dk .

Key Equation 33
When computing attention weights, we normalize our dot product qT k by a factor
√
dk .

• This compensates for the fact that longer vectors will create larger dot products.

So, when computing our attention weights a, we use the formula:

 q ⊤ K⊤ 
a(q, K) = softmax √
dk

It still has shape (1 × N).

We scale down our MatMul by the appropriate factor.

8.2.4 The Attention Mechanism: values, attention

Now, we have a collection of attention weights: each one tells us relevant each word is to
q.

Last Updated: 12/25/24 08:36:30

MIT 6.390 Fall 2024 23

• Now, we want to make them useful. Our original goal was to get an average sense of
what "mexican" food is like.

To make this concrete, we’ll introduce our third embedding: the value vector.

• Value v: Each food has a value vector, directly storing information about a word.

– Unlike the key/query vectors, this embedding isn’t based on similarity to other
words.

– Instead, it usually contains more direct information about our word: in this ex-
ample, maybe it contains the price, calories, ingredients, etc. Note that, in a real
model, value vectors
are often "learned" dur-
ing training. So, they
Definition 34
won’t always contain
The value vector v represents a word, and stores useful information that it can such simple, easily ex-
plained data.
contribute to the query.

– It answers the question, "what useful data could this word contribute to the
query?"

By adding together the value vectors from each word relevant to the query, we
can get an overall "averaged value" for q.

Each word has both a value and a key attached to it.

For our example, let’s suppose that the value vector contains price, calories, and salt.

 
pricei
vi =  cali  (8.13)
 

salti

We want to get an "average" calorie count for mexican food.

• Some foods are common for mexican food, and some are more rare.

• So, to get an average, we’ll need to emphasize more "common" mexican food.

How do we do that? Using our attention weights: the larger the attention weight, the more
"relevant" a food is to our mexican food calculation.

Last Updated: 12/25/24 08:36:30

MIT 6.390 Fall 2024 24

If we use q to represent mexican food, and ki is the key for the ith food, we get:

Weighted average
z
X }| {
calq = P(ki |q) cali (8.14)
i

Rather than repeating this process for each row of v, we can just do a weighted average of
the whole vector, at the same time:

X
vq = P(ki |q) vi (8.15)
i

Key Equation 35
Each word i has a value vector vi , which represents all of the useful information it can
provide to the query.

• We can use a weighted average to combine all of these value vectors together:
this provides the "overall context" for the query.

• Each value is weighted based on its attention weight P(ki |q): how likely it is to
be relevant.

X
vq = P(ki |q) vi
i

This is the calculation for attention.

Just like we did for the ki · q operation, we can re-write this in terms of matrix multiplica-
tion.

• We’ll change from P(ki |q) to P(K|q).


⊤
P k1 q
 
 P k q

 2 
P K q = 
 ..

 = softmax(qT KT )

 . 


P kN q

• We’ll stack all of our value functions vi into a matrix V.

 ⊤

V = v1 v2 ... vN  (8.16)

 

Last Updated: 12/25/24 08:36:30

MIT 6.390 Fall 2024 25

Notation 36
We’ll assume that we have N value vectors of length dk .

• vi has shape (dk × 1), V has shape (N × dk ).

Now, we can compute with every value vector at once: If you study the classic
"Attention is all you
need" paper, you’ll find
Key Equation 37 that their version of k
and q are transposed
We can compute attention using matrix multiplication: compared to ours.

 q ⊤ K⊤ 
Attention q, K, V = softmax √ V
dk
 ⊤ ⊤

q√ K
Where softmax dk
computes our attention weights.

• Under this definition, attention has shape (1 × dk ).

Definition 38
Attention(q, K, V) is the weighted average of all of our value vectors (transposed).

• Attention is the result of aggregating information from N different words: each

word is represented by a key ki , and a value vector vi .

We now have a completed representation of attention.

In the "Attention is all
you need" paper, this
With this, we can summarize the basic idea of attention: diagram is analogous to
Figure 2 (left).

Here, we omit the

"Mask" layer (discussed
Last Updated: 12/25/24 08:36:30 later).
MIT 6.390 Fall 2024 26

Concept 39
Attention is a mechanism that allows you to combine information from multiple to-
kens, weighting each token by how relevant it is.

This mechanism is broken into three parts:

• Value vector v: what information are we trying to combine?

• Query vector q: what kinds of words are relevant to this search?

• Key vector k: what kinds of searches is this word relevant for?

Each token has a value vector (information from that token), and a key vector (used to
compare this token to the query).

Note that this isn’t the only way to do attention:

Clarification 40
There are multiple ways we can implement attention.

• For example, we use q · k to measure similarity, but we could replace it with a

different metric.
Reminder: a "metric" is
just "a way of measur-
ing something. The dot
product is a similarity
metric for vectors.

Last Updated: 12/25/24 08:36:30

MIT 6.390 Fall 2024 27

So far, we’ve mostly focused on the mathy details of how attention works: an abstract idea
of "relevance" between words, "combining" the value ("meaning") of different words, etc.

• Here, we’ll try something different: we’ll focus more on why we use attention, and
how it applies to a real, concrete situation.

8.2.5 Why we need context

Attention is designed to integrate information from other, nearby words. But why do we
need to do this?

• Because language is heavily dependent on context.

Consider the task of language translation: we have a sentence in one language, and we
want to convert it into another language, while preserving the meaning.

Let’s translate the sentence:

I miss her warm smile.

We’ll focus on the word "warm".

• Most commonly, "warm" means "higher-than-average temperature". For example,

being under a blanket is warm.

• But most humans would say that, in this situation, the word "warm" means ’friendly’
or ’kind’.

We know this because of the context: a "warm smile" usually means a "kind smile". The
word ’smile’ has changed the meaning of the word ’warm’.

Concept 41
The meaning of a word can change based on the other words which are nearby.

• This is why we need to integrate context for language processing.

If our machine blindly translated "warm", without context, we could’ve ended up with the
wrong meaning in another language.

8.2.6 Why we need attentive context

So, we need to use context. But what makes attention special?

Last Updated: 12/25/24 08:36:30

MIT 6.390 Fall 2024 28

• It allows us to figure out which words are most important to us!

In the above sentence, the word "smile" changed the meaning of "warm".

• How do we know that "smile" is the important context word? "her" is equally far
from "warm".

• Attention handles this for us: we "pay more attention" to the word ’smile’ than the
word ’her’, when we’re trying to understand "warm".

Concept 42
Attention allows us to determine which parts of the context are most important to a
particular word.

Last Updated: 12/25/24 08:36:30

MIT 6.390 Fall 2024 29

8.2.7 Self-attention
Attention has given us a tool for comparing one word to every other word in a sentence.

"dog" is represented by a query qdog , compared to the key for every other word in the
sentence. This gives us our attention weights.
Next, we combine these weights with the value vector for each word. This gives us our
attention: the "contextual meaning" of the word dog.

This has a limitation: we’re only focusing on a single word, "dog".

• But we need to get the meaning of every word in the sentence, based on the context
from other words.

This time, we use the query qbites for the word "bites". However, the key and value vectors
are still the same for each word.
We need to repeat this attention process once for each word.

This is interesting: we’re seeing how much each word affects each other word in the sen-
tence. We’re seeing how the sentence provides context for itself.

• This is why we call this self-attention.

Definition 43
Self-attention is the process of using attention on every word in a passage.

• For the ith word, we compare it to every other word in the passage.

This allows us to interpret each word, based on the context provided by the rest of the
sentence.
Technically, we also
compare each word to
itself.
8.2.8 Self-attention in matrix form
How do we handle this, mathematically?

• When we are getting the attention for word wi , we use its query qi to compare it to
other words in the sentence.

Last Updated: 12/25/24 08:36:30

MIT 6.390 Fall 2024 30

We’ve gone from having a single query q to having many qi : one for each word in the
sentence.

 T

Q = q1 q2 ... qN  (8.17)

 

Let’s note some conventions:

Notation 44
A few useful dimensions: in an attention problem, we have...

• nk keys of length dk .

• nq queries of length dq .

• nv values of length dv .

In practice, we usually take dk = dq = dv , and simply refer to all three as dk .

• Each column vector (ki , vi , qi ) has shape (dk × 1).

In self-attention, we take nk = nq = nv , and simply refer to all three as N.

• Matrices K, V, and Q all have shape (N × dk ).

Each of these queries will create a separate set of attention weights, softmax(qTi K).

Key Equation 45
We define the self-attention weight matrix A, to represent all attention weights:

  √ 
softmax qT1 KT / dk
 softmax qT KT /√d 
  
T
 = softmax QK
 2 k  
A= .. √
  dk
 .
 
 √ 
softmax qTN KT / dk

This is an (N × N) matrix.

• Row i tells us all of the attention weights applied to query qi .

• Col j tells us the attention weights for key kj .

• Element αij (row i, col j) tells us, "how important is word j (key) as context for
word i (query)"?

Last Updated: 12/25/24 08:36:30

MIT 6.390 Fall 2024 31

Note that the elements

in row i must add up to
We can use this to get the total attention: 1: we have softmax.

This is not true for col-

Key Equation 46 umn j: they’re prob-
abilities for different
The self-attention equation is given as queries.

 QKT 
Attention(Q, K, V) = AV = softmax √ V
dk

It is a (N × dk ) matrix.

Row i gives the averaged value vector y(i) for the ith word, based on all of the surrounding
context. We could view this as
the "output" for the ith
word.
• We can write this in element-wise form:

X
N
(i)
y = αij vj
j=1

One theme we’ll run into, many times in this chapter, is that attention-based models benefit
from being able to parallelize:

Concept 47
Transformer Parallelization I

Computing self-attention can be strongly parallelized:

• Each qTj ki term is independent of the others: we can compute all of the key-
query dot products at the same time, rather than waiting for one to finish before
starting the others.

• We can compute each softmax term at the same time, as well.

This remains true if
we’re using cross-
attention, where the
keys and queries come
from different words.

Last Updated: 12/25/24 08:36:30

MIT 6.390 Fall 2024 32

8.2.9 Positional Encoding

First, a problem we need to address:

• Currently, our key ki is determined by asking the identity of the word at index i.

• This key doesn’t encode information about the position of this word in the sentence.

But clearly, the position of a word will determine its meaning.

• Example: "The cat lies on the green table" and "the green cat lies on the table" are not
the same: moving the word "green" to a different index changes its meaning.

We fix this by adding information to keep track of this position.

Definition 48
We apply positional encoding to each word embedding: each embedding includes
information about the position of a word in the text.

• This allows our attention mechanism to use this information when deciding the
relevance of different words.

8.2.10 Masking
One common use for transformer models is text prediction: learning what word should
come next, based on what it has seen so far.

Typically, we would give our model the text, and give it a chance to try to predict each
index, before it can see it.

We need to prevent our model from being able to cheat:

• We don’t want our model to be able to see the words it’s supposed to be predicting.

So, we’ll hide those words, so our model can’t see them. In this case, we want our model
to predict the next word: "dog".

This is called masking.

Last Updated: 12/25/24 08:36:30

MIT 6.390 Fall 2024 33

Definition 49
Masking is a technique where we hide some information from our model, so it can’t
use that information.

• For example, if our model is being used to predict text, we hide the text that it’s
trying to predict.

However, the word "masking" can apply to any situation where we want to hide tokens
from the model.

Last Updated: 12/25/24 08:36:30

MIT 6.390 Fall 2024 34

8.2.11 Attention Heads

We have a system for "attention": deciding which words provide the most important con-
text/information, and paying more attention to those words.

But there’s something we haven’t considered: the "importance" of different words, depends
on what you’re interested in. Let’s consider a couple examples:

• Syntax: which words are subjects, objects, verbs, adjectives?

Example: "The boy kicks the red ball": our focus is on the word "ball".

– "red" is important for color.

– "kicks" is important for knowing what’s happening to the ball.

– "boy" is important for knowing who is acting on the ball.

• Semantics: which words change the meaning of our target word?

Example: "I miss her warm smile": our focus is on the word "warm".

– The word "smile" changes the meaning of warm from ’high temperature’ to
’kind’.

• Coreference: which words are referring to the same object?

Example: "John said that he isn’t hungry": our focus is on the word "John".

– "he" refers to the same object as "John": if we apply something to the word "he",
it also applies to "John".

Concept 50
What is "important" in a sentence can change, based on what you’re trying to study.

• And generally, these ideas of "important" won’t agree with each other.

Above, we suggested several different perspectives on "what is important".

• Rather than having our attention mechanism try to handle all of these kinds of im-
portance, we could create a separate mechanism for each one of them.

We’ll do just that: each "perspective" will be represented by a different mechanism. We call
each of these, attention heads.

Last Updated: 12/25/24 08:36:30

MIT 6.390 Fall 2024 35

Definition 51
A transformer model may use multiple attention mechanisms at the same time:

• Each attention mechanism is a different "perspective" on our data: it focuses on

different aspects of the text (grammar, meaning, tone, etc.)

• To accomplish this, each one represents a word w with a different k, q, and v.

We call each mechanism one attention head.

If we have 3 different attention heads, each one may encode the word "silly" differently.
We could have three different keys for this one word: k1 , k2 , and k3 .

• Each head will require a distinct word encoding: K(h) , Q(h) , and V (h) .

Concept 52
Transformer Parallelization II

Each attention head uses calculations which are independent from the others: we can
compute each attention head at the same time!

Last Updated: 12/25/24 08:36:30

MIT 6.390 Fall 2024 36

8.3 Transformers
Now that we’ve built up attention, we’ll use it to build a transformer. We’ll assume our
transformer uses self-attention, though the math works out similarly even if it doesn’t.

Definition 53
A transformer block is a collection of attention heads running in parallel, applied to
the same text.

A transformer is composed of several transformer blocks in series: the output of one

block is the input of another.

8.3.1 How to create embeddings

Something we’ve ignored for a while is, "how do we construct our embeddings K, Q, and
V"?

• We aren’t actually given them: we’re given a sequence of tokens: each token is a
vector x representing a word. So, our whole body of text is a matrix X. Each vector is length d:
this is different from the
length of the embed-
We’ll compute each embeddings by using a linear transformation:
ding, dk .

Key Equation 54
We use projection matrices Wk , Wq , and Wv to transform each token x(i) into embed-
dings k,q, and v.

ki = Wk⊤ x(i)
qi = Wq⊤ x(i)
vi = Wv⊤ x(i)

All three projection matrices have shape (d × dk ).

Reminder that:

All of our tokens are stored in matrix X: d is the original length

of x(i)
 T dk is the length after
embedding.
X = x(1) x(2) ... x(N)  (8.18)
 
Unlike our usual X, this
is transposed: shape
(N × d).
We can get the keys, queries, and values for all of our vectors in matrix form:

Last Updated: 12/25/24 08:36:30

MIT 6.390 Fall 2024 37

Key Equation 55
We can compute K, Q, and V:

K = XWk
Q = XWq
V = XWv

Based on this linear transform, we modify our diagram:

We have to generate V, K, and Q before we can use them.

Concept 56
One benefit of computing keys, values, and queries based on weight matrices is that
we can train these matrices:

• Rather than manually designing the embeddings, we can allow our model to
learn whichever embedding is most useful.

8.3.2 Attention Heads

What if we have multiple attention heads?

Notation 57
If we have H attention heads in a transformer block we’ll indicate the hth head with:

K(h) = XWh,k
Q(h) = XWh,q
V (h) = XWh,v

Each attention head is applied in parallel:

Last Updated: 12/25/24 08:36:30

MIT 6.390 Fall 2024 38

Here’s an example with H = 3 attention heads. Each uses a distinct set of keys, values, and
queries.

To finish off our multi-headed attention unit, we do two more things:

• Transform each token back into the original dimensions: going from length-dk to
length-d.

• Combine the results from each attention head: we’ll do a weighted average.

Key Equation 58
After computing attention for each head, we take a weighted average of our heads,
combining them together:

• For each head, we use matrix Wh,c to scale the weight of each head, and convert
them back to their original shape.

– Wh,c has shape (dk , d).

• We add together the results, gathering information from each head.

X
H
Attention Q(h) , K(h) , V (h) Wh,c


u=
h=1

u, the final output of our multi-headed attention, has shape (N × d), where the jth
column represents the jth token.

Last Updated: 12/25/24 08:36:30

MIT 6.390 Fall 2024 39

We have our completed multi-headed attention unit!

In the "Attention is all
you need" paper, this
diagram is analogous to
Figure 2 (right).

Instead of directly do-

ing a weighted sum,
they concatenate each
attention head, and then
apply a linear weight
Wo.

These are equivalent.

Note that this is the same shape as our original input, X:

 ⊤

U = u(1) u(2) ... u(N)  (8.19)

 

In fact, we can compute this multi-headed attention, one u(i) at a time. Reminder that αij is an
attention weight from
A, and v(j) is a value
Key Equation 59 vector of V.
We can combine our multi-attention heads as

Heads Attention
z }| { z }| {
XH XN
(h) (h)

⊤
u(i) = Wh,c αij vj
h=1 j=1
This
P is a nested sum:
h,j (·),

not a product of two

sums,
P P
( h (·)) · ( j (·))

Last Updated: 12/25/24 08:36:30

MIT 6.390 Fall 2024 40

8.3.3 Residual Connections

Our next component will handle a problem with deep neural nets that we’ve addressed
before: vanishing/exploding gradient.

Definition 60
(Review from Neural Networks 2)

Vanishing gradient occurs when a deep neural network ends up with very small gra-
dients in the earlier layers.

This happens because a deeper neural network has a longer chain rule: if all of the
terms are less than one, they’ll multiply into a very small value, "vanishing".

This means that our gradient descent will have almost no effect on these earlier weights,
slowing down our algorithm considerably.

In short: the "further away" from our input layer, the messier our gradients get.

One simple solution is to include our original, unmodified input, deeper in the neural
network: we just add it, so that our second layer gets to see the input data, too.

Our output contains direct information about the input. This hopefully improves training.

Definition 61
In a residual block, the input x is added to the output F(x) of the block (in our case,
multi-headed attention) .

output = F(x) + x

• This is designed to reduce the risk of vanishing gradient, by directly exposing

deeper layers to the input

– The long chain rule is what causes vanishing gradient: we’ve created a
shorter chain rule.
If you ever hear some-
one refer to a "ResNet"
or "Residual Network",
this is a CNN that uses
the same technique!
Last Updated: 12/25/24 08:36:30
MIT 6.390 Fall 2024 41

8.3.4 Layer Normalization

Another topic from the NN chapter: batch normalization.

Definition 62
(Review from Neural Networks 2)

Batch Normalization is a process where we

• Standardize the pre-activation for each layer across data points in the batch us-
ing mean µi and standard deviation σi (for the ith dimension).

Zij − µi
Zij =
σi

• Choose the new mean and standard deviation for the pre-activation using (n×1)
vectors G and B

bik = Gi ∗ Zij + Bi
Z
In short: we set the
(mean, sd) to (0,1) and
We would get the same kinds of benefits from normalization in transformers as we did then scale it back up to
(Gi , Bi ).
before in NNs.
Stabilizing our training
But rather than normalizing across multiple data points (batch), we’ll normalize across the process, mostly.
features (layer) of a single token.

Key Equation 63
h iT
Suppose we have a (d × 1) data point z = z1 z2 · · · zd .

Layer normalization computes the mean µz and standard deviation σz across our fea-
tures zi

v
1X u1 X
u d
µz = zi σz = t (zi − µz )2
d d
i i=1

And then normalizes them.

z − µz
znorm =
σz

Finally, we scale them back up, to have mean β and s.d. γ.

z − µ 
z
LayerNorm(z; γ, β) = γ +β
σz
Layer normalization
can be used on a single
Now that we understand this process, we can apply this to our transformer model: data point, while batch
normalization requires
many.
Last Updated: 12/25/24 08:36:30
MIT 6.390 Fall 2024 42

• After we get u + x (creating the residual block), we use layernorm on each token sep-
arately:

Concept 64
At the end of our residual block, we apply LayerNorm to each of our tokens separately

• We take our (N × d) object u + X and normalize the features of each of our N

tokens (shape (d × 1)) separately.

u(i)
norm = LayerNorm(u
(i)
+ X(i) , γ1 , β1 )

We append a LayerNorm layer. We’ll follow the convention from the "Attention is all you
need" paper and combine these into a single unit: "Add+Norm".

With this, our Residual Connection is complete.

Last Updated: 12/25/24 08:36:30

MIT 6.390 Fall 2024 43

8.3.5 Feed Forward

In our CNNs, after convolution, we would use a fully-connected feed-forward network
to analyze the processed data.

• We’ll follow the same sort of pattern here: the main difference being that we apply
feed-forward after only one layer of multi-headed attention.

Key Equation 65
After we apply Add & Norm to our Multi-headed attention, we run the output through
a feed-forward layer, processing the data it receives.

• We use a linear layer W1 , a ReLU layer, and another linear layer W2 .

 
z = W2T ReLU W1T unorm

We can think of this as apply a hidden FC layer to our network, followed by another linear
transform.

Linear, ReLU, linear. Once again, following "Attention is all you need", we simply call this
the "Feed forward" Layer.

We’ll follow this up with another LayerNorm: Meaning, we use an-

other residual block.

Concept 66
After our feed-forward layer, we apply Add & Norm again.

z(i)
norm = LayerNorm(z
(i)
+ u(i)
norm , γ2 , β2 )

This is the final output of our transformer block.

Last Updated: 12/25/24 08:36:30

MIT 6.390 Fall 2024 44

znorm is the final result of our transformer block.

Last Updated: 12/25/24 08:36:30

MIT 6.390 Fall 2024 45

8.3.6 Transformer Block

With this, we can assemble our transformer block, top to bottom:

We have a transformer block!

Definition 67
A transformer block is made up of several functions composed together:

• Multi-headed attention

– Each head encodes the input text X as keys K(h) , a value Qh, and vectors
V (h) : one for each token.

– Based on these, we compute attention.

– Finally, we linearly combine information from across all H heads.

• Add & Norm

• Feed-forward

– We apply a fully-connected layer (linear+ ReLU), then another linear unit.

• Add & Norm

Both "Add & Norm" layers accomplish the same thing: they create a residual connec-
tion.

• We add the input to the output, and then layer normalize.

Last Updated: 12/25/24 08:36:30

MIT 6.390 Fall 2024 46

Concept 68
Each layer of our transformer block serves an important function:

• The multi-headed attention layer explores connections between tokens, and pro-
vides information about the internal structure of our data.

• The feed-forward layer processes our information nonlinearly (via ReLU).

• The add & norm layers create residual connections between the input/output of
the preceding layer, improving our gradient-training process.

From here, we can design a transformer model by combining many of these transformer
units in series.

Last Updated: 12/25/24 08:36:30

MIT 6.390 Fall 2024 47

8.3.7 Translation Task: training

We just have one more layer of complexity, before we finish. Let’s consider a training
example, for the task of translating from english to spanish.

I’m not hungry yet =⇒ Todavía no tengo hambre

Our transformer will start by predicting the first word in the sentence: presumably "to-
davía".

• But not necessarily: if our model isn’t well-trained yet, it might predict some random
word, like "espacio". It’s also possible for us
to have multiple valid
translations, but we’ll
Now, we want to predict the second word in our output. But we just brought up an impor-
ignore that for now.
tant problem:

• The best "second word" in our translation is dependent on the first word. We should
factor that into our model, when predicting the second word.

• If our first word was wrong, then we’re more likely to use an incorrect second word!

The solution? Instead of using the first word we predicted, we use the correct first word.

• Only one condition we need to remember: we need to mask the rest of the "correct"
output sentence, so our model can’t use it to cheat.

Concept 69
When training our model to complete a language task, our model predicts each word
(token) one-by-one, based on two pieces of data:

• The entire input prompt

• The desired output sequence for every token before the one we want to predict.

Example: Suppose we’re predicting the third word in our above sentence. We’ll use the
first two "correct" words as part of our model: In this case, "tengo" is
the word we want to
" # predict.
I’m not hungry yet
=⇒ tengo
Todavía no

Last Updated: 12/25/24 08:36:30

MIT 6.390 Fall 2024 48

Our model is actually trained with two inputs.

Something to take note of: we predict the ith token based on the input, and the first i − 1
desired inputs.

• That means that, when predicting token gi , we don’t care what we predicted for the
previous tokens!

• We don’t need to finish predicting token i to predict token i + 1: we can do them at

the same time!

Concept 70
Transformer Parallelization III

Predicting token i is an independent calculation from predicting a second token j.

• That means we can predict every token in our sentence at the same time!

This is a huge advantage in training transformers: it can essentially think about the entire
sentence at the same time, massively speeding up training.

Clarification 71
We can’t parallelize token generation when we’re using our model after training:

• We can parallelize during training because we’re using the desired output for
the previous i − 1 tokens.

• When using our model for unseen data, we don’t have "desired output": we have
to use our actual output for the previous tokens.

We have to wait for our model to predict the first i − 1 tokens, before it predicts the ith
token.

Last Updated: 12/25/24 08:36:30

MIT 6.390 Fall 2024 49

8.3.8 Encoder + Decoder Structure

Now, we have to structure our transformer model to be able to handle both the input and
desired output.

Concept 72
There are three tasks we want our model to complete:

• Process our input sequence,

• Process our target sequence (desired output),

• Combine the two sequences of information

• Predict the next character based on this data.

We’ll choose functions to handle each of these tasks.

• We’ll process our prompt using a complete transformer block. This unit is our en-
coder: we encode our prompt in a form that is more meaningful to our computer.

• However, for our desired output, we’ll only use attention, learning about the internal
structure of the output. Why not add the feed-
forward layer? We’ll
add it later: there’s an-
– We’ll also use this unit to mask our output (so our transformer can’t "look
other component we
ahead" at future tokens). want to add first (see
below).

Last Updated: 12/25/24 08:36:30

MIT 6.390 Fall 2024 50

We’ll add the second feed-forward unit later. First, we want to combine information from
our input, with the earlier tokens of our output.

We accomplish this with another attention unit: this time, we’ll use cross-attention.

Definition 73
In cross-attention, our queries come from one sequence of text, while our keys/values
come from a different sequence of text.

• As opposed to self-attention, where our keys/values/queries all come from the

same sequence.

Our goal is to use the earlier part of our output sentence to determine which parts of our
input we should pay attention to in our input sentence, when choosing the next token. For example: if our out-
put sentence already in-
cludes a word, it might
• Our keys/values represent the words we might want to pay attention to.
be less likely we’ll need
to use that word again.
• Our queries help us decide what to pay attention to.
We can use "attend" as
Thus, we’ll use keys/values from our encoded input, and queries from our previous out- a verb meaning "pay
attention to": this is
put tokens. common when talking
about transformers.

Concept 74 For example, in this

We integrate our input tokens and our previous output tokens using cross-attention: case, we’re deciding
"which input tokens to
we apply attention, using attend to".

• Our encoded input as our keys and values.

• Our attended output as our queries.

This is our encoder-decoder layer.

Last Updated: 12/25/24 08:36:30

MIT 6.390 Fall 2024 51

Now that we’ve integrated information from both our input and output, we finally include
our feed-forward unit: we’ll process our integrated information.

This unit on the right is called our decoder.

We’ve got a complete encoder/decoder setup:

Last Updated: 12/25/24 08:36:30

MIT 6.390 Fall 2024 52

Concept 75
We break our transformer into an "encoder" and "decoder" unit:

• The encoder transforms our input into a representation that contains more useful
information: connections between tokens in the prompt, etc.

• The decoder transforms that encoding into a output/response: this decoder takes
the information we’ve gathered, and applies it to our problem.

Consider the translation example:

• The encoder stores our English text in a form that hopefully represents the meaning.

• The decoder "decodes" that representation into a form we can read, but in a different
language: Spanish, in our example.

In this analogy, we’ve created a special "code" that we write in English, and read in Spanish.

8.3.9 Predicting a token

Only one step left: using this decoded information, we need to choose our token. This is
the multi-class classification problem: we use the same protocol as we always do.

• We linearly transform our data: each token gets a "score", based on how likely we
think it is to be the correct one.

• We apply softmax, to turn these scores into probabilities. We get a probability for
every possible token.

Last Updated: 12/25/24 08:36:30

MIT 6.390 Fall 2024 53

This is essentially a completed transformer.

This is the (now-famous) diagram from the "Attention is all you need" paper! We’ve excluded the ini-
tial embedding (turning
Only one detail still missing: words into vectors) and
positional embedding
(adding information
• Our decoder/encoder typically has several copies of the same unit in a row: for about the position of
example, we might have 3 transformer blocks in a row for our encoder. each word in the sen-
tence).
Notably, this is only one kind of transformer model: which architecture we use depends We could include those
on the problem, cost constraints, etc. for completeness, but
that would just take up
more space.

Last Updated: 12/25/24 08:36:30

MIT 6.390 Fall 2024 54

8.3.10 Training Process

We typically train transformer models in two stages: pre-training and fine-tuning.

Definition 76
In pre-training, we expose our model to a very large dataset of human language, so it
can learn patterns in that language.

• We can use unlabelled data in this stage: thus, we have an unsupervised/self-

supervised problem.

This stage of training is typically expensive.

Definition 77
In fine-tuning, we take our pre-trained model, and train it for a specific task.

• We use labelled data in this stage.

It tends to be much faster and less expensive than pre-training.

8.3.11 Variations
We could make variations on this network:

• Use more/fewer decoder/encoder units.

• Use a different style of attention (rather than the dot product, we use some other
similarity metric).

• Move LayerNorm to different parts of the network.

Last Updated: 12/25/24 08:36:30

MIT 6.390 Fall 2024 55

8.4 Terms
• Natural Language Processing (NLP)

• (Review) Convolutional Neural Networks (CNNs)

• Locality

• Recurrent Neural Networks

• (Review) Word Embedding

• Co-occurrence

• Context window

• Skipgram

• Word2vec

• Token

• Key Vector

• Query Vector

• Value Vector

• Attention Weights

• dk

• Attention

• Self-attention

• Positional Encoding

• Masking

• Attention Head

• Projection Matrix

• Multi-headed attention

• Residual Block

• Residual Connection

• Layer Normalization

• Add & Norm

• Feed-forward layer (transformers)

Last Updated: 12/25/24 08:36:30

MIT 6.390 Fall 2024 56

• Transformer Block

• Cross-attention

• Encoder (Transformers)

• Decoder (Transformers)

• Encoder-Decoder Layer

• Pre-training

• Fine-tuning

Last Updated: 12/25/24 08:36:30