Chapter 2:

Large Language Models

Ernest K. Ryu
University of California, Los Angeles

1
Natural language processing (NLP)
Natural language processing (NLP) is concerned with computationally processing natural
(human) languages. The goal is to design and/or train a system that can understand, and
process information written in documents.

A natural language or ordinary language is any language that has evolved naturally in
humans through use and repetition without conscious planning or premeditation such as
English or Korean. They are distinguished from formal and constructed languages such as
C, Python, Lojban, and Esperanto.

NLP was once a field that relied on insight into linguistics, but modern NLP is dominated by
data-driven deep-learning based approaches.

2
Task: Sentiment analysis
Given a review 𝑋 ∈ 𝒳 on a reviewing website, decide whether its label 𝑌 ∈ 𝒴 = −1,0, +1 is
negative (−1), neutral (0), or positive (+1).

Eg.
Review: I do not like this movie
Sentiment: Negative

Review: I love this movie

Sentiment: Positive

Input is variable-length. Output is fixed-size.

3
Sentiment analysis with BOW
A bag of words (BOW) model makes the prediction with a linear combination of tokenized
word. This is a simple baseline.

+ + + . + bias
= score

weights
lookup lookup lookup lookup

I love this movie

More generally “bag of words” refers to models that view a sentence as an unordered
collection (bag) of words. Completely disregarding word order is a significant drawback of
the method.

4
Sequence (seq) notation
Let 𝒰 be any set. Define 𝑘-tuples of 𝒰 as
𝒰𝑘 = 𝑢1 , … , 𝑢𝑘 𝑢1 , … , 𝑢𝑘 ∈ 𝒰
The Kleene star notation
𝒰∗ = ራ 𝒰𝑘 = 𝑢1 , … , 𝑢𝑘 𝑢1 , … , 𝑢𝑘 ∈ 𝒰, 𝑘 ≥ 0
𝑘≥0

denotes sequences of 𝒰 of arbitrary finite length.

Although unimportant in most practical setups, we define the empty sequence is a valid sequence of length 0 and write ∈ 𝒰∗ . 5
Characters
Let 𝒞 be a set of “characters”.
• 𝒞 can be the set of English characters, space, and some punctuation.
• 𝒞 can be the set of all unicode characters.
Let 𝒳 = 𝒞 ∗ be the set of finite-length sequence of characters, i.e., 𝑋 ∈ 𝒳 is raw text.

(The definition and scope of “characters” is not obvious, and different choices present
different trade-offs. The modern approach is the view each “byte” of the unicode encoding to
be the indivisible basic unit. More on this when we cover byte pair encodings.)

6
Tokenization
Neural networks perform arithmetic on vectors and numbers, so tokenizers convert text into
a sequence of vectors.

Given 𝑋 = 𝑐1 , … , 𝑐𝑇 ∈ 𝒞 ∗ , a tokenizer is a function 𝜏 ∶ 𝒞 ∗ → ℝ𝑛 ∗

such that
𝜏 𝑐1 , 𝑐2 , … , 𝑐𝑇 = 𝑢1 , 𝑢2 , … , 𝑢𝐿
where 𝑢1 , 𝑢2 , … , 𝑢𝐿 ∈ ℝ𝑛 . 𝑇 and 𝐿 are often not the same. (Usually 𝑇 ≥ 𝐿.)Sometimes 𝜏 is
fixed, and sometimes it is trainable (e.g. word2vec).

For text generation, we want the tokenizer to be invertible.

7
Character-level tokenizer v.0
Example: 𝒞 = 𝑎, 𝑏, … , 𝑧,_,.,?,! and
𝜏 𝑋 = 𝜏 𝑐1 , … , 𝑐𝐿 = 𝜏 𝑐1 , … , 𝜏 𝑐𝐿
𝜏 𝑎 = 1, 𝜏 𝑏 = 2, … 𝜏 𝑧 = 26, …
So 𝑛 = 1 and 𝐿 = 𝑇.

This doesn’t work very well.

We want distinct tokens to be vectors of distinct directions. Neural networks are better at
distinguishing directions than magnitudes.

8
Character-level tokenizer v.1
Example: 𝒞 = 𝑎, 𝑏, … , 𝑧,_,.,?,!
𝜏 𝑋 = 𝜏 𝑐1 , … , 𝑐𝐿 = 𝜏 𝑐1 , … , 𝜏 𝑐𝐿
1 0 0
0 1 0
𝜏 𝑎 = 0 , 𝜏 𝑏 = 0 , … 𝜏 ! = 0
⋮ ⋮ ⋮
0 0 1

So 𝑛 = 30 and 𝑇 = 𝐿. The output vectors are called one-hot-encodings as only one element
of the encoded vector is nonzero (hot).

9
Word-level tokenizer
Examples: 𝒞 = 𝑎, 𝑏, … , 𝑧,_ (so English letters and space) and 𝒲 = English words

𝜏 𝑋 = 𝜏 𝑐1 , … , 𝑐𝑇 = 𝜏 𝑤1 , … , 𝑤𝐿 = 𝜏 𝑤1 , … , 𝜏 𝑤𝐿
1 0 0
0 1 0
𝜏 ‘aardvark’ = 0 , 𝜏 ‘ability’ = 0 , … , 𝜏 ‘Zyzzyva’ = 0 , …
⋮ ⋮ ⋮
0 0 1
where 𝑤1 , … , 𝑤𝐿 ∈ 𝒲. So 𝑛 = 𝒲 = size of dictionary and 𝐿 ≤ 𝑇.

I.e., this is a one-hot encoding of words.

10
End-of-string (EOS) token
Given 𝑋 ∈ 𝒳 and its length 0 ≤ 𝑇 < ∞, we equivalently consider a special “end-of-string”
token <EOS> to be the final 𝑇 + 1 -th element. In other words,
𝑋 = 𝑐1 , 𝑐2 , … , 𝑐𝑇 = 𝑐1 , 𝑐2 , … , 𝑐𝑇 , <EOS>
for any 𝑋 ∈ 𝒳, where 𝑐1 , … , 𝑐𝑇 ∈ 𝒞.

We use the same notation for elements in 𝒰∗ , i.e.,

𝑢1 , 𝑢2 , … , 𝑢𝐿 = 𝑢1 , 𝑢2 , … , 𝑢𝐿 , <EOS> ∈ 𝒰∗

11
Discussion on tokenizers
Q) Advantage of word-level tokenizer over character-level tokernizer?
A) Shorter tokenized sequence. Uses dictionary. (Model need not learn words from scratch.)

Q) Advantage of character-level tokenizer over word-level tokernizer?

A) Can learn to handle misspellings (‘learning’ ≈ ‘lerning’) and inflections (‘running’ = ‘run’ +
‘ing’). Better for multi-language models. (Dictionaries of multiple languages is too large.)

Q) Are there other tokenizers?

A) Word2Vec and subword tokenization (byte-pair encoding) are trained tokenizers.
More on these later.

12
Basic BOW implementation
Let 𝜏 be a word-level tokenizer with dictionary 𝒲.

For 𝑋 = 𝑤1 , … , 𝑤𝐿 the bag-of-word (BOW) model 𝑓𝜃 is

𝐿 𝐿

𝑓𝜃 𝑋 = 𝑏 + 𝑎 ⋅ ෍ 𝜏 𝑤ℓ = 𝑏 + 𝑎 ⋅ ෍ 𝜏 𝑋 ℓ
ℓ=1 ℓ=1

where 𝜃 = 𝑎, 𝑏 ∈ ℝ𝑛+1 is the trainable parameter.

+ + + . + bias
= score

weights
lookup lookup lookup lookup

I love this movie 13

Sentiment analysis with DNN
Modern state-of-the-art NLP methods are based on deep neural networks (DNN).

Some Complicated
Neural Network
. + bias
= score

features weights
lookup lookup lookup lookup lookup

I love this movie <EOS>

14
 Task: Language model (LM)
A language model (LM) achieves one or two of the Some Complicated
probability
following goals. Neural Network

Goal 1: Assign probabilities/likelihoods to sentences.

The best things in life are free <EOS>

Goal 2: Generate likely (coherent) sentences. I have ever done

1. Traveling to different countries.

Some Complicated 2. Volunteering to help those in need.
Why is an LM useful? Prior to ChatGPT, this Neural Network 3. Graduating from college.
4. Starting my own business.
was not at all an obvious question. 5. Taking a gap year to explore the world.
… 6. Learning a new language <EOS>
The best things

(This definition excludes encoder-only transformer models such as BERT, but we will not be overly
concerned with these definitions.) 15
Applications of LM: Voice-to-text
In a voice-to-text system, two interpretations can be auditorily ambiguous but semantically
not ambiguous. An LM can determine which interpretation is more likely.

“The parcel was secured by grey tape.” (✓)

“The parcel was secured by great ape.”

“he was a lighthouse keeper” (✓)

“he was a light housekeeper”

A similar application with spelling correction.

16
Applications of LM: Autocomplete
An autocomplete system can assist writing by suggesting likely completions of a sentence.

Meeting Arrangement

professor@[Link]

Meeting Arrangement

Dear professor,

What would be the right time to contact you?

I will be looking forward to hearing from you

17
Applications of LM: SSL pre-training and
universal interface
Training a NN to be a language model is a useful pretext task for transfer learning in the
sense of self-supervised learning (SSL). Pre-trained language models serve as foundation
models that can be fine-tuned for other downstream tasks.
• More on this when we talk about ELMo, BERT, and GPT 1.

A sufficiently powerful LM can serve as a universal language-based interface to the

capabilities that the language model has learned.
• More on this when we talk about T5 and GPT3.

18
Probabilities with sequences
Assume a sequence
𝑢1 , 𝑢2 , … , 𝑢𝐿 = 𝑢1 , 𝑢2 , … , 𝑢𝐿 , <EOS> ∈ 𝒰∗
is generated randomly, i.e., we can assign a probability
ℙ 𝑢1 , 𝑢2 , … , 𝑢𝐿 , <EOS> ∈ 0,1

The sequence length 𝐿 is also a random variable. Imagine 𝑢1 , 𝑢2 , … being generated

sequentially. Given 𝑢1 , 𝑢2 , … , 𝑢ℓ , the next token may be 𝑢ℓ+1 =<EOS> and the sequence
terminates. Otherwise, 𝑢ℓ+1 ≠ <EOS> and the generation continues to 𝑢ℓ+2 .

19
Probability notation with <EOS>
Clarification) Given 0 ≤ 𝐿 < ∞ and 𝑢1 , 𝑢2 , … , 𝑢𝐿 ∈ 𝒰,

is the probability that a random sequence in 𝒰∗ has values 𝑢1 , 𝑢2 , … , 𝑢𝐿 for the first 𝐿
elements and then terminates, i.e., 𝑢𝐿+1 =<EOS>.

On the other hand, if 𝑢1 , 𝑢2 , … , 𝑢𝐿 ∈ 𝒰,

is the probability that a random sequence in 𝒰∗ has values 𝑢1 , 𝑢2 , … , 𝑢𝐿 for the first 𝐿
elements (and none of them are <EOS>) but 𝑢𝐿+1 but may or may not be <EOS>. In particular,

20
Conditional probabilities with sequences
With the chain rule (conditional probability), we have

where ℙ 𝑢ℓ 𝑢1 , … , 𝑢ℓ−1 is the probability of 𝑢ℓ conditioned on the past. (For ℓ = 1, we mean

ℙ 𝑢1 𝑢1 , … , 𝑢0 = ℙ 𝑢1 .) So, the probability of the entire sequence 𝑢1 , 𝑢2 , … , 𝑢𝐿 =
𝑢1 , 𝑢2 , … , 𝑢𝐿 , <EOS> is the product of the conditional probabilities.

To clarify, we have made no assumptions on the sequence probabilities. (We have not
assumed that anything is Markov or that anything is independent.)
21
Cond. prob. with continuous sequences
If sequence elements 𝑢𝑡 are continuous random variables, then we need density functions
instead of discrete probability mass functions. However, calculations are essentially the
same, so we do not repeat it.

In NLP, vocabulary is finite, so consider seqs with discrete elements.

Some RL problems have continuous states and rewards.

For image patches (vision transformers), seq elements are (essentially) continuous.

22
Autoregressive (AR) modelling
An autoregressive model of a sequence learns to predict 𝑢ℓ given the past ovservations
𝑢1 , … , 𝑢ℓ−1 . Goal is to learn a model 𝑓𝜃 that approximates the full conditional distribution
𝑓𝜃 𝑢ℓ ; 𝑢1 , … , 𝑢ℓ−1 ≈ ℙ 𝑢ℓ 𝑢1 , … , 𝑢ℓ−1

(Etymology is ‘auto’ ≈ ’self’ and ‘regress’ ≈ ’fit’.)

23
Sequence likelihood with AR model
Given a trained autoregressive model 𝑓𝜃 𝑢ℓ ; 𝑢1 , … , 𝑢ℓ−1 ≈ ℙ 𝑢ℓ 𝑢1 , … , 𝑢ℓ−1 , we can
(approximately) compute the likelihood of a sequence 𝑢1 , … , 𝑢𝐿 with

24
Sequence generation with AR model
Given a trained autoregressive model 𝑓𝜃 𝑢𝑡 ; 𝑢1 , … , 𝑢ℓ−1 ≈ ℙ 𝑢ℓ 𝑢1 , … , 𝑢ℓ−1 , and an un-
terminated sequence 𝑢1 , … , 𝑢ℓ−1 (if ℓ = 1, then start generation from nothing) we can
generate 𝑢1 , … , 𝑢ℓ−1 , 𝑢ℓ , … , 𝑢𝐿 ∼ ℙ 𝑢𝑡 , … , 𝑢𝐿 , 𝑢𝐿+1 =<EOS> 𝑢1 , … , 𝑢ℓ−1 by sampling

which is justified by

25
Modern NLP and sequence processing
Modern NLP solves various tasks, especially language modelling, with deep neural networks.

We need a general approach to process sequences (variable-length data) as inputs and

outputs. We start with RNNs and then move on to transformers.

Why still learn RNNs? Although transformers have been replacing RNNs and CNNs in recent
years, RNNs and CNNs are not yet obsolete. Also, much of the architecture design of
transformers are inspired by practices inherited from the RNN era. One still needs to know
RNNs to fully understand modern NLP.

26
Learning with variable-sized inputs
In image classification, the input 𝑋 ∈ ℝ3×𝑛×𝑚 is of fixed size and processed by a deep CNN.
We now want to process variable-sized input 𝑋 ∈ 𝒞 ∗ with a neural network.

Simple idea: Zero-pad up to length of longest sequence.

• + This can work as a quick and temporary solution.

• − Does not scale well for long sequences if fully-connected layer is used.
• − Maximum length must be specified.

27
Process one input per layer

Idea: Process one input per layer

• + Shorter sequences require fewer layers to evaluate.
• + Each layer is much smaller than a giant layer one would need to process the whole
sequence at once.
• − Total number of weights and biases increase with maximum sequence length.
• − Exploding/vanishing gradients.
28
Weight sharing
Idea: What if the parameters are the same (use weight sharing) for all layers?

Same 𝐴 and 𝑏

• + Can process an arbitrary number of inputs.

• − Exploding/vanishing gradients.

This is called a recurrent neural network (RNN).

29
Recurrent neural networks (RNN)
More generally, an RNN has the form

where 𝜃,෨ 𝐴, and 𝑏 are the trainable parameters.

The 𝑞𝜃෩ is called the recurrent function.

The exploding/vanishing gradient problem still remains.

RNNs work only if 𝑞𝜃෩ is chosen to mitigate this problem.
J. L. Elman, Finding structure in time, Cognitive Science, 1990. 30
RNNs are extremely deep networks
Seq. length of 100s or 1000s is common.

Multiplying many numbers is unstable:

• If most of the numbers > 1, we get ∞ (“Exploding gradients”. Can fix with gradient clipping.)
• If most of the numbers < 1, we get 0 (“Vanishing gradients”. Bigger problem.)
Reasonably-sized product if numbers are all close to 1.

For matrices, a similar reasoning holds with eigenvalues or singular values.

Y. Bengio, P. Simard, and P. Frasconi, Learning long-term dependencies with gradient descent is difficult, IEEE Transactions on Neural Networks, 1994. 31
Exploding gradients and gradient clipping
The exploding gradient problem occurs when the gradient magnitude is very large.

Exploding gradients imply the output is very sensitive to small changes of the parameters in
a certain direction. Sometimes, such gradients are unworkable and the neural network
architecture must be changed.

Sometimes, however, the direction of the gradient

is fine. If so, one can use gradient clipping and
use the clipped gradient in the optimization.

Gradient clipping with threshold value 𝑣:

32
Vanishing gradients
The vanishing gradient problem occurs when the magnitude of a gradient is very small.

Intuitively, vanishing gradients means the gradient signal does not reach the earlier layers.
In an RNN, for example, may not be small but

can be small.

This means changes in ℎℓ do not affect the output ℒ. Since , this further implies
that (small) changes in 𝑢ℓ do not affect ℒ. We can intuitively understand this as the RNN not
utilizing information of 𝑢ℓ , i.e., RNN does not remember# 𝑢ℓ at step 𝐿. Moreover, gradient
signal is lost by the time backprop reaches time ℓ, and the model can’t learn to
how to utilize 𝑢ℓ . (The model forgets 𝑢ℓ by step 𝐿, and gradient updates can’t fix this.)
#This argument is not precisely correct since large changes in 𝑢ℓ may affect ℒ. 33
Promoting better gradient flow
Consider

If the Jacobian is close to identity, i.e., then we say the

gradient flows through the layer ℎℓ+1 well.

If , then and we say the gradient does not flow well through the layer
ℎℓ+1 well since the information contained in is lost.

34
Promoting better gradient flow
So then, do we always want good gradient flow? Do we always want ?

No. We want when we want to remember information.

We want when we want to forget.

Solution) Design a “neural circuit” that explicitly controls when to remember information and
when to forget information.

“cell state”

“forget gate”
35
 LSTM cells
Long short-term memory (LSTM) cells 𝑐ℓ, 𝑓ℓҧ , 𝑖ℓҧ , 𝑔ҧℓ, 𝑜ҧℓ have same
has an intricate and somewhat dimension as ℎℓ.
arbitrary structure. (Cf. GRU cell#
which simplifies the LSTM structure.)
“cell state”

“forget gate”
Works much better than a naïve RNN!

Cell state 𝑐ℓ serves as memory.

(In retrospect, the cell state should be
called the hidden state, as it is more “hidden state”
similar to the hidden states of RNNs or
hidden Markov models. However, this
notation is now standard.)
S. Hochreiter and J. Schmidhuber, Long short-term memory, Neural Computation, 1997.
F. A. Gers, J. Schmidhuber, and F. Cummins, Learning to forget: continual prediction with LSTM, Neural Computation, 2000.
#J. Chung, C. Gulcehre, K. Cho, and Y. Bengio, Empirical evaluation of gated recurrent neural networks on sequence modeling, NeurIPS Workshop on
36
Deep Learning and Representation Learning, 2014.
 LSTM with output projection
𝑐ℓ, 𝑓ℓҧ , 𝑖ℓҧ , 𝑔ҧℓ, 𝑜ҧℓ have same dimension
Sometimes, you want the LSTM to output a ℎℓ has a different (usually larger) dimension
large hidden state while maintaining a
reasonably-sized internal computation.
“cell state”

“forget gate”
(In LMs, the output size can be the vocabulary
size or the number of possible tokens with
byte pair encoding, both are large.)

Solution) Output projection. “hidden state”

H. Sak, A. W. Senior, F. Beaufays, Long short-term memory based recurrent neural network architectures for large vocabulary speech recognition, arXiv,
2014. 37
LSTM name meaning
To clarify, “long short-term memory” does not mean long-term & short-term memory.

Rather, it means that the cell state serves as a longer short-term memory. In contrast, a
naïve RNN (that uses an MLP rather an LSTM cell as the recurrent function) would have a
much shorter short-term memory.

A true long-term memory would correspond to some external storage, which an LSTM RNN
doesn’t have.

38
Proprietary LLMs now
have long-term memory
Recently, OpenAI announced its new memory
features in ChatGPT.

Other chatbot services will likely follow-up with

similar features.

LLMs now have long-term memory, and this will

profoundly affect how we interact with LLMs.

[Link] 39
Aside: Exploding/vanishing gradient
problem
The exploding/vanishing gradient problem arises in for many deep neural networks, not just
RNNs.

The ResNet architectue, and more generally the use of residual connections is one
approach to mitigate the exploding/vanishing gradient problem.

Another technique is the use of normalization layers such as batch norm.

RNNs can use batch norm#, but it is not common.

#T. Cooijmans, N. Ballas, C. Laurent, C. Gülçehre, and A. Courville, Recurrent batch normalization, ICLR, 2017. 40
 𝑌 = ‘positive’

Sentiment analysis with LSTM ℒ

The output hidden state can be used Linear

for the single (non-sequence) output.
sentence
encoding

ℎ0
𝑞𝜃෩ 𝑞𝜃෩ 𝑞𝜃෩ 𝑞𝜃෩

𝑐0

𝜏 𝜏 𝜏 𝜏
The hamburger was good
41
 𝑌 = ‘positive’

ℒ
Sentiment analysis with LSTM
Linear
Pooling all of the hidden states often
performs better than then using only sentence
the last one for learning a single (non- encoding
sequence) output.
Pool
(Avg pool or Max Pool)

ℎ0
𝑞𝜃෩ 𝑞𝜃෩ 𝑞𝜃෩ 𝑞𝜃෩

𝑐0

𝜏 𝜏 𝜏 𝜏
The hamburger was good 42
 𝑌 = ‘positive’

ℒ
Stacked RNN
Linear

sentence
Stacked RNNs use more depth and can learn more encoding

complex representations. Avg pool

or Pool
Max Pool

(2)
ℎ0
Rule of thumb is to use 2–8 stacked LSTM layers.
𝑞𝜃෩ (3) 𝑞𝜃෩ (3) 𝑞𝜃෩ (3) 𝑞𝜃෩ (3)
(2)
𝑐0
• 2 layers is almost always better than 1.
• 3 layers is not always better than 2. (1)
ℎ0
𝑞𝜃෩ (2) 𝑞𝜃෩ (2) 𝑞𝜃෩ (2) 𝑞𝜃෩ (2)
(1)
𝑐0

Each layer of an RNN transforms a sequence to a (0)

sequence: ℎ0
𝑞𝜃෩ (1) 𝑞𝜃෩ (1) 𝑞𝜃෩ (1) 𝑞𝜃෩ (1)
(0)
𝑐0

𝜏 𝜏 𝜏
The hamburger was
𝜏
good 43
Example task: Parts of speech tagging
sum
ℒ𝑜𝑠𝑠
For some RNN tasks, the output is a sequence, and
the total loss is the sum of the losses incurred at each PRON VERB NUM NOUN
sequence term. ℓCE ℓCE ℓCE ℓCE

Linear Linear Linear Linear

(1)
ℎ0
𝑞𝜃෩ 2 𝑞𝜃෩ 2 𝑞𝜃෩ 2 𝑞𝜃෩ 2
(1)
𝑐0

(0)
ℎ0
𝑞𝜃෩ 1 𝑞𝜃෩ 1 𝑞𝜃෩ 1 𝑞𝜃෩ 1
(0)
𝑐0

𝜏 𝜏 𝜏 𝜏
he saw two birds 44
 Y = ‘positive’

Bidirectional RNN ℒ

Linear
(Unidirectional) RNNs process information forward
in time. In language, however, it is common for
later words to provide necessary context for
understanding a previous word. concat

ℎ0
𝑞𝜃෡
A bidirectional RNN combines forward and reverse 𝑞𝜃෡ 𝑞𝜃෡ 𝑞𝜃෡
𝑐0
directional RNNs to process a sequence without a
single sense of direction. ℎ0
𝑞𝜃෩ 𝑞𝜃෩ 𝑞𝜃෩ 𝑞𝜃෩
𝑐0

𝜏 𝜏 𝜏 𝜏
The hamburger was good

M. Schuster and K. K. Paliwal , Bidirectional recurrent neural networks, IEEE TSP, 1997. 45
 ℒ𝑜𝑠𝑠

ℓCE ℓCE ℓCE ℓCE

PRON VERB NUM NOUN

Stacked bidirectional RNN Linear Linear Linear Linear

concat
concat

concat
concat
Stacking and bidirectionality can be combined. ℎ0
𝑞𝜃෡ 2 𝑞𝜃෡ 2 𝑞𝜃෡ 2 𝑞𝜃෡ 2
𝑐0

ℎ0
𝑞𝜃෩ 2 𝑞𝜃෩ 2 𝑞𝜃෩ 2 𝑞𝜃෩ 2
𝑐0

concat
concat

concat
concat
ℎ0
𝑞𝜃෡ 1 𝑞𝜃෡ 1 𝑞𝜃෡ 1 𝑞𝜃෡ 1
𝑐0

ℎ0
𝑞𝜃෩ 1 𝑞𝜃෩ 1 𝑞𝜃෩ 1 𝑞𝜃෩ 1
𝑐0

𝜏 𝜏 𝜏 𝜏
46
The hamburger was good
 RNN-LM
The RNN language model (RNN-LM) is trained as an autoregressive model with the
following structure.
ℒ𝑜𝑠𝑠

𝑢2 ℓCE 𝑢3 ℓCE 𝑢4 ℓCE 𝑢𝐿+1 ℓCE

𝜏 𝜏 𝜏 𝜏
cat sat on <EOS>

LSTM
ℎ0 = 0
𝑐0 = 0 LSTM LSTM LSTM …

𝑢1 𝑢2 𝑢3 𝑢𝐿

𝜏 𝜏 𝜏 𝜏
the cat sat mat

T. Mikolov, S. Kombrink, L. Burget, J. H. Černocký, and S. Khudanpur, Extensions of recurrent neural network language model, ICASSP, 2011. 47
LM loss: Next token prediction
Let us interpret the loss

We are given a sequence of tokens 𝑢1 , … , 𝑢𝐿 , which are one-hot vectors indicating the
specific token. RNN predicts the next token
𝑢ℓ+1 ≈ ℎℓ = 𝑓𝜃 𝑢1 , … , 𝑢ℓ
for ℓ = 1, … , 𝐿. (ℎℓ are logits, so ≈ means softmax ℎℓ ≈ 𝑢ℓ+1 , i.e. assigns high probability to
the token 𝑢ℓ+1 .) The (in)accuracy of the prediction is measured by the cross entropy loss

for ℓ = 1, … , 𝐿. The sequence 𝑢1 , … , 𝑢𝐿 creates 𝐿 next-token-prediction problems. Namely,

given 𝑢1 , … , 𝑢ℓ , predict 𝑢ℓ+1 for ℓ = 1, … , 𝐿. Our loss ℒ is the sum of the losses on these 𝐿
problems.

48
Sentence prob. vs. language generation
The training and inference of language models consider two different tasks, and it is
instructive to view the distinction from an RL perspective.

Language models are trained on:

• Assigning probabilities/likelihoods to a given sentence via next-token-prediction.
• Next-token-prediction is analogous to imitation learning.
• Training done on real sentences (in distribution).

Language models (LM) are often used to:

• Generate likely (coherent) sentences.
• Generation is analogous to policies acting in an environment.
• Unless the LM is very good, partial generations will be unnatural, and the generation
conditioned on the unnatural generation (out of distribution) will be worse. This is a
consistent problem with acting on a policy trained with imitation learning.
49
Teacher forcing
Teacher forcing# (TF) is the practice of feeding ground-truth sequence values back into the
RNN after each step, even though the RNN would not generate such a sequence, exactly
like imitation learning (IL). This forces the RNN to stay in the training distribution.

However, once the model makes a mistake (generate one bad token) it does not know how
to recover, since the model was trained only on good text through TF/IL.

Resolution 1: Start training with TF and transition into something else.% What this something
else should be is not obvious. Can’t use DAgger due to data constraints. (Why?)
Resolution 2: Just make the LM large and train on lots of data, including both good text and
poorly written messy internet text. This is the solution of modern LLMs.

#R. J. Williams and D. Zipser, A learning algorithm for continually running fully recurrent neural networks, Neural Computation, 1989.
%S. Bengio, O. Vinyals, N. Jaitly, and N. Shazeer, Scheduled sampling for sequence prediction with recurrent neural networks, NeurIPS, 2015. 50
Trainable tokenizer
The tokenizer is the first contact between language and our algorithm.

Instead of using one-hot encodings with a fixed dictionary, it is better to have some trainable
component in the tokenizer.

Currently, byte-pair encoding has become the standard choice, but we shall consider the
historical context.

51
What does a word mean?
Denotational semantics: A word is the collection of the objects it describes.
This is the intuitive and straightforward view of the meaning of words, but it is not a very
actionable definition.

Distributional semantics: A word is characterized by the company it keeps.

A word’s meaning is given by the words that frequently appear close-by. This is much more
actionable in the realm of computational linguistics.

52
Word2vec
Train a model such that a word predicts a neighboring word.

“Context word” defined by a window of size 𝑚.

𝑚 is a hyperparameter that is tuned
𝑃 𝑤𝑡−2 𝑤𝑡 𝑃 𝑤𝑡−1 𝑤𝑡 𝑃 𝑤𝑡+1 𝑤𝑡 𝑃 𝑤𝑡+2 𝑤𝑡

… problems turning into banking crises as …

outside context words center word outside context words

in window of size 2 at position 𝑡 in window of size 2

We use two vectors per word:

• 𝑣𝑤 when 𝑤 is a center word Then, for a center word 𝑤𝑡 and context word 𝑤𝑡+𝑘 :
exp 𝑢𝑤𝑡+𝑘 ⋅ 𝑣𝑤𝑡
• 𝑢𝑤 when 𝑤 is a context word ℙ 𝑤𝑡+𝑘 𝑤𝑡 =
σ𝑤∈𝑉 exp 𝑢𝑤 ⋅ 𝑣𝑤𝑡

T. Mikolov, K. Chen, G. Corrado, and J. Dean, Efficient estimation of word representations in vector space, ICLR Workshop, 2013.
T. Mikolov, I. Sutskever, K. Chen, G. S. Corrado, and J. Dean, Distributed representations of words and phrases and their compositionality, NeurIPS, 2013. 53
NeurIPS 2023 Test of Time Award
Word2vec
Specifically, we minimize the loss function
𝐿

ℒ 𝜃 =෍ ෍ − log 𝑝𝜃 𝑤ℓ+𝑘 𝑤𝑘
ℓ=1 −𝑚≤𝑘≤𝑚
𝑘≠0

where 𝜃 = 𝑣1 , … , 𝑣𝑛 , 𝑢1 , … , 𝑢𝑛 is the trainable parameter.

Insight: Words 𝑤 and 𝑤

෥ are “similar” if ℙ neighboring words 𝑤 ≈ ℙ neighboring words 𝑤
෥
and this happens when 𝑣𝑤 ≈ 𝑣𝑤෥ .

54
Word2vec: Trained word-level tokenizer
Goal of word2vec is a trained word-level tokenizer:
Given input 𝑋 = 𝑤1 , … , 𝑤𝐿 chunked into words 𝑤1 , … , 𝑤𝐿 ∈ 𝒲, the trained tokenizer 𝜏𝜃
outputs the corresponding 𝑣-vectors. This is a data-driven way to map similar words to
similar vectors.

Using word2vec as inputs to RNNs significantly improves performance, compared to simple

one-hot tokenizers.

Downside: The word-level tokenizer 𝜏𝜃 𝑤ℓ does not consider the context in which the word
𝑤ℓ is used in (cf. polysemy).

T. Mikolov, K. Chen, G. Corrado, and J. Dean, Efficient estimation of word representations in vector space, ICLR Workshop, 2013.
T. Mikolov, I. Sutskever, K. Chen, G. S. Corrado, and J. Dean, Distributed representations of words and phrases and their compositionality, NeurIPS, 2013. 55
 Self-supervised learning
In self-supervised learning (SSL), models learn
useful representations from unlabeled data by
solving pretext tasks that automatically generate
supervisory signals. The pretext task is not by
itself useful, but it is related to the downstream
task of interest. The pre-trained model is
subsequently trained on downstream tasks,
usually with a smaller labeled dataset.

The success of SSL is most dramatic in NLP, and

Word2Vec was the first notable success case. In computer vision, predicting random rotations of
an image is one example of a pre-text task.

S. Gidaris, P. Singh, and N. Komodakis, Unsupervised representation learning by predicting image rotations, ICLR, 2018. 56
ELMo
Embeddings from Language Models (ELMo) is an in-context tokenizer. Produces word
representations in the context of the entire sentence.

Uses bidirectional LSTM structure. The states of RNNs are hidden states, but they can also be
considered tokernized values of the given words.

ELMo has its own tokenizer layer

with trainable parameters (like a
transformer’s embedding layer), but
we won’t pay attention to it.

M. E. Peters, W. Ammar, C. Bhagavatula, and R. Power, Semi-supervised sequence tagging with bidirectional language models, ACL, 2017.
M. E. Peters, M. Neumann, M. Iyyer, M. Gardner, C. Clark, K. Lee, and L. Zettlemoyer, Deep contextualized word representations, NAACL, 2018. 57
Bidirectional LM loss for pre-training
Pre-training uses the loss

where 𝜃LSTM and 𝜃LSTM are the

parameters of the forward and
backward LSTM cells and 𝜃other are the
shared parameters for the input
(tokenizer) and output (softmax) stages.

Training data is unlabeled text.

(Labels are the ground truth words.)

58
Non-causal language model
Causal language models learn
𝑓𝜃 𝑢ℓ ; 𝑢1 , … , 𝑢ℓ−1 ≈ ℙ 𝑢ℓ 𝑢1 , … , 𝑢ℓ−1
i.e., the causal LM learns to predict the next token left-to-right.

ELMo and BERT are non-causal language models. (Half of ELMo is a causal language
model, but that is not the point.) ELMo and BERT can understand language and solve many
NLP tasks, but it cannot generate text.

GPT is a causal language model, and it can generate text.

59
ELMo fine-tuning
Given a prior NLP method (which can be very specialized and tailored to the specific task)
that takes in 𝑥ℓ 𝐿ℓ=1 , replace the input 𝑥ℓ 𝐿ℓ=1 with 𝑥෤ℓ 𝐿ℓ=1 , where

where 𝐾 is the depth of the LSTM RNN, 𝑘 = 0 corresponds to the tokenization layer, and
𝑠𝑘task are the task-specific trainable parameters. (The sum is over the LSTM depth.)

𝐾
Then, train the entire pipeline, including the ELMo weights, 𝑠𝑘task 𝑘=0
, and the weights of
the NLP method on labeled task-specific data.

60
Results
ELMo achieved state-of-the-art performance on a wide range of NLP tasks.
• Question answering
• Textual entailment (determining whether a “hypothesis” is true, given a “premise”)
• Semantic role labeling (Answers “Who did what to whom”)
• Coreference resolution (clustering mentions in text that refer to the same underlying real-
world entities)
• Named entity extraction (finding four types of named entities (PER, LOC, ORG, MISC) in
news articles)
• Sentiment analysis (whether paragraph is positive or negative)

61
Discussion of ELMo
Although the idea of self-supervised learning through large-scale pre-training and fine-
tuning was not new (Word2Vec is a predecessor and [Dai and Le 2015] proposed an idea
like ELMo) ELMo executed it very well and advanced the state of the art substantially. ELMo
make it clear that self-supervised pre-training is an indispensable paradigm.

However, LSTM RNN is not the best architecture. The left- and right-directional RNNs only
process information unidirectionally. What if the model needs to examine the entire
sentence to make inference? Also, RNNs are fundamentally computationally inefficient.

The overall approach is still not universal; each task needs a tailored method and ELMo
only served to provide better tokenization.

A. M. Dai and Q. V. Le, Semi-supervised sequence learning, NeurIPS, 2015. 62

BERT
Bidirectional Encoder Representations from Transformers (BERT) (i) replaces the LSTMs of
ELMo with (encoder-only) transformer layers and (ii) proposed a more universal approach.
BERT set a new state-of-the-art on almost every benchmark.

J. Devlin, M. Chang, K. Lee, and K. Toutanova, BERT: Pre-training of deep bidirectional transformers for language understanding, NAACL, 2019. 63
Transformers
Transformer architectures are sequence-to-sequence models. They “transform” a sequence
to another sequence in each layer.

There are 3 types of transformers, listed in order of complexity.

• Encoder-only (BERT)
• Decoder-only (GPT)
• Encoder-decoder (Original transformer of Vaswani et al. 2017)

We first discuss the encoder-only transformer.

64
Encoder-only transformer
The transformer architecture relies on the following components
• Token embeddings
• Multi-head self-attention
• Residual connections
• Layer normalization
• Position-wise FFN
• Positional encodings

65
Token embeddings
Let 𝜏 be a tokenizer such that
𝜏 𝑋 = 𝑢1 , 𝑢2 , … , 𝑢𝐿
where we can think of 𝑢𝑖 = 𝑘 ∈ 1, … , 𝑛 or, equivalently, 𝑢𝑖 = 𝑒𝑘 ∈ ℝ𝑛 . Here, 𝑒𝑘 denotes the
𝑘-th unit (one-hot) vector, and 𝑛 denotes the number of distinct tokens. A word-level
tokenizer is one of the simplest instances of this.

Then, the token embedding layer maps 𝑢1 , 𝑢2 , … , 𝑢𝐿 ↦ 𝑣1 , 𝑣2 , … , 𝑣𝐿 , where

𝑣ℓ = 𝑎𝑢ℓ or 𝑣ℓ = 𝐴𝑢ℓ
depending on whether we view 𝑢ℓ = 𝑘 or 𝑢ℓ = 𝑒𝑘 . Here, 𝐴 = 𝑎1 𝑎2 ⋯ 𝑎𝑛 ∈ ℝ𝑑×𝑛 is a
trainable parameter (defined as an “embedding layer” in PyTorch).

The token embedding layer is applied at the beginning of the transformer.

66
Single-head self attention

𝐴2: = softmax 𝐴෪2: , 𝑦2 = 𝐴21𝑣1 + ⋯ + 𝐴2𝐿 𝑣𝐿

𝐴෪
2𝐿

𝐴෪
21

𝑞2 𝑘1 … 𝑘𝐿 𝑣1 … 𝑣𝐿

𝑊𝑄 ⊤
𝑊𝐾 ⊤

𝑊𝑉 ⊤

𝑥1 𝑥2 … 𝑥𝐿
67
Attention is a pseudo-linear operation
Functions 𝑓 ∶ ℝ𝑛 → ℝ𝑚 of the form
𝑓 𝑥 =𝐴 𝑥 𝑥
are said to be “pseudo-linear”. (It is not linear because they the matrix 𝐴 𝑥 ∈ ℝ𝑛×𝑚 .)

Attention is a pseudo-linear mapping from 𝑉 ∈ ℝ𝐿×𝑑𝑉 to 𝑌 ∈ ℝ𝐿×𝑑V .

Pseudo-linear operations are common in signal processing and kernel methods.

(I quickly point this out as it is a nice and simple observation.)

68
Multi-head self attention (MHA)
Just as one uses multiple CNN channels, we use multiple attention heads.

𝐿 𝐿
Seq-to-seq transformation 𝑥ℓ ℓ=1 ↦ 𝑧ℓ ℓ=1 . Often 𝑑𝑋 = 𝑑𝑍 required by residual connection.

69
 𝐿
𝑘+1
𝑥ℓ
Encoder-only transformer ℓ=1

One transformer layer consists of:

View one layer of TF as a sequence-to-sequence

𝐿 𝐿
transformation 𝑥ℓ𝑘 ↦ 𝑥ℓ𝑘+1 . TF stacks
ℓ=1 ℓ=1
many such layers.

The “addition” block is a residual connection,

which helps with optimization.

𝐿
𝑘
𝑥ℓ
ℓ=1
Figure due to:
R. Xiong, Y. Yang, D. He, K. Zheng, X. Zheng, C. Xing, H. Zhang, Y. Lan, L. Wang, and T.-Y. Liu, On layer normalization in the transformer architecture, 70
ICML, 2020.
Layer normalization
Layer normalization (LN) also stabilizes training by normalizing the features and thereby
avoiding exploding and vanishing gradients.

Normalization across the features. Does not normalize over sequence lengths or batch
elements. Assume 𝑋 has dimension (batch × sequence length × channel/feature)
𝐶
1
𝜇ො ∶, ∶ = ෍ 𝑋[∶, ∶, 𝑐]
𝐶
𝑐=1
𝐶
2
1 2
𝜎ො ∶, ∶ = ෍ 𝑋 ∶, ∶, 𝑐 − 𝜇ො ∶, ∶
𝐶
𝑐=1
𝑋 ∶, ∶, 𝑐 − 𝜇ො ∶, ∶
LN𝛾,𝛽 𝑋 [∶, ∶, 𝑐] = 𝛾 𝑐 +𝛽 𝑐 𝑐 = 1, … , 𝐶
2
𝜎ො ∶, ∶ + 𝜀

J. L. Ba, J. R. Kiros, and G. E. Hinton, Layer normalization, arXiv, 2016. 71

CNN LN ≠ TF LN
How LN is used in CNNs is different from how it's used in Transformers (including ViT).

For CNNs, LN normalize over channels and spatial dimensions. For transformers, LN
normalizes over channels and not over spatial dimensions.
BN LN in CNNs LN in Transformer

seq. length
H, W

H, W

C N C N C N
BatchNorm takes in a 4d tensor and normalizes across LayerNorm takes in a 3d tensor and normalizes
3 dimensions: spatial and batch dimensions. across 1 dimension: vector (channel) dimension. 72
Position-wise FFN
Position-wise FFN is a 2-layer MLP with ReLU, GELU, or SiLU activation functions:

Let 𝑑 be the token size, i.e., 𝑥 ∈ ℝ𝑑 .

Often, 𝑊1 ∈ ℝ4𝑑×𝑑 , 𝑊2 ∈ ℝ𝑑×4𝑑 (expansion factor of 4).

Applies independently on each sequence element, i.e.,

73
GELU, SiLU, Swish activations
Gaussian error linear unit (GELU), Sigmoid-weighted linear unit (SiLU), and Swish are
smooth non-monotone activation functions. The three are qualitatively similar: they
decrease near 0 and then increase nearly linearly.

D. Hendrycks and K. Gimpel, Gaussian error linear units (GELUs), arXiv, 2016.
P. Ramachandran, B. Zoph, and Q. V. Le, Searching for activation functions, arXiv, 2017. 74
S. Elfwing, E. Uchibe, and K. Doya, Sigmoid-weighted linear units for neural network function approximation in reinforcement learning, Neural Networks, 2018.
Positional encoding/embedding
Problem: Encoder-only transformer architecture is permutation equivariant and it does not
know positional information of tokens. Relative positions of tokens (word order or patch
location) obviously carries important meaning in language.

Solution: After token embedding layer 𝑋 ↦ 𝑢ℓ 𝐿ℓ=1 ↦ 𝑣ℓ 𝐿

ℓ=1 , add positional embedding
vectors 𝑝ℓ 𝐿ℓ=1 and then pass
𝑣ℓ + 𝑝ℓ 𝐿ℓ=1
as input to the transformer layers.

S. Sukhbaatar, A. Szlam, J. Weston, and R. Fergus, End-to-end memory networks, NeurIPS, 2015.
J. Gehring, M. Auli, D. Grangier, D. Yarats, and Y. N. Dauphin, Convolutional sequence to sequence learning, ICML, 2017. 75
A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin, Attention is all you need, NeurIPS, 2017.
Positional encoding/embedding
NLP transformers often use the
sinuisoidal positional encoding 𝑝1 , … , 𝑝𝐿 ∈ ℝ𝑑

(Feels like a very arbitrary design, but this work well and is hard to beat.) Since NLP
transformers must accommodate arbitrary sequence length 𝐿, using a positional encoding
with an analytical formula makes sense.

On the other hand, vision transformers let 𝑝ℓ 𝐿ℓ=1 be trainable. Possible since image
resolution and hence sequence length 𝐿 is fixed.
76
Positional encoding/embedding
Idea is often attributed to Vaswani et al. 2017,

However, Sukhbaatar et al. 2015 and Gehring et al. 2017 did publish the positional
encoding technique earlier. The sinusoidal encoding is due to Vaswani et al. 2017.

S. Sukhbaatar, A. Szlam, J. Weston, and R. Fergus, End-to-end memory networks, NeurIPS, 2015.
J. Gehring, M. Auli, D. Grangier, D. Yarats, and Y. N. Dauphin, Convolutional sequence to sequence learning, ICML, 2017. 77
A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin, Attention is all you need, NeurIPS, 2017.
Post-LN vs Pre-LN TF architectutres
There are 2 variants of the transformer architecture based on the position of LN.
The original (Vaswani et al. 2017) paper illustrates postLN in its figure. However, their
updated official codebase uses pre-LN. It is later reported that Pre-LN is more stable.

R. Xiong, Y. Yang, D. He, K. Zheng, X. Zheng, C. Xing, H. Zhang, Y. Lan, L. Wang, and T.-Y. Liu, On layer normalization in the transformer architecture, 78
ICML, 2020.
Transformer depth
Thanks to the residual connections and layer norm, transformers can often be much deeper
than stacked RNNs. (ELMo has 2 layers, while BERT has 24 layers.)

To clarify, the layer norm and the residual connection are used to mitigate the
exploding/vanishing gradient problem across the transformer depth.

The transformer does not have exploding/vanishing gradient problem along the sequence
length 𝐿 due to its use of attention mechanism.

79
Why transformers over RNNs?
Handling long sequence length:
RNNs struggle with long input sequences due to a fixed memory size and vanishing or
exploding gradients. LSTMs are designed to mitigate this problem, but transformers really
solve this problem. Transformers allows the full input sequence to be considered without
having to compress the information into a hidden state vector.

Efficient parallel computation:

RNNs are inherently sequential (inefficient) during training. (RNNs are efficient during
inference.) In contrast, transformers are completely parallelizable in training, and we can
better leverage efficient large-scale GPU computation.

80
 BERT pre-training
BERT pre-training uses two losses.

1. Masked LM (MLM)
Randomly mask out 15% of the words and let BERT
predict it. Output tokens corresponding to masked words
are fed into softmax and CE loss.

2. Next sentence prediction (NSP)

Provide two sentences A and B separated with [SEP]
token with 50% probability of B following A and 50%
probability of B unrelated to A, and make binary
prediction. Attach classification head to the output
corresponding to [CLS] token.
81
BERT fine-tuning
Many NLP tasks roughly fit the
MLM and NSP shape.

For fine-tuning, make minimal

modifications to the BERT
baseline and fine-tune the whole
model.

Fine-tuning is computationally
very cheap (<1 hour on a single
Google TPU).

82
Vision transformer
Vision transformer is an encoder-only
transformer architecture.

Given an image, each 16 × 16 patch is a

token, and the patches are placed into a
linear sequence.

Supervised pre-training on image classification had

Output corresponding to [CLS] token is better performance compared to self-supervised
used for classification. pre-training with masked patch prediction.

A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, J. Uszkoreit, and N.
Houlsby, An image is worth 16x16 words: Transformers for image recognition at scale, ICLR, 2021. 83
GPT-1
GPT (generative pre-training) uses a causal language model loss

Initially, GPT was trained to be an self-supervised pre-trained model in the vein of BERT,
and the its text generation ability was not that strong.

(However, the focus of GPT-2 shifted to text generation.)

Compared to BERT’s encoder only transformer, GPT-1’s decoder-only transformer

additionally utilizes (i) masked attention and (ii) output projection layers.

A. Radford, K. Narasimhan, T. Salimans, and I. Sutskever, Improving language understanding by generative pre-training, 2018. 84
Masked attention
In RNNs, information is naturally processed sequentially.

However, there is a problem with using an encoder-only (BERT-style) transformer for a

causal language model: The model can see the entire sequence, the past and the future.

Therefore, GPT uses a masked attention that allows the current sequence element to only
query earlier sequence elements.

85
Masked single-head self attention
Only lower-triangular
components of 𝐴ሚ are finite.
Only lower-triangular
components of 𝐴 are nonzero.

Crucially, 𝑞𝑖 is allowed to query only 𝑘1 , … , 𝑘𝑖 .

𝑦ℓ is a linear combination of 𝑣1 , … , 𝑣ℓ .

𝑦ℓ only depends on 𝑥1 , … , 𝑥ℓ .
( 𝑥ℓ 𝐿ℓ=1 ↦ 𝑦ℓ 𝐿ℓ=1 has causal dependency)

86
Masked multi-head self attention

𝐿 𝐿
Seq-to-seq transformation 𝑥ℓ ℓ=1 ↦ 𝑧ℓ ℓ=1 with causal dependence:
𝑧ℓ only depends on 𝑥1 , … , 𝑥ℓ .

Since other components of transformer all act positionwise, the transformer with causal
MHA is a seq-to-seq transformation with causal dependence.
87
 Decoder-only transformer
Output projection
layer softmax function 𝜇
set of probability mass vectors

trainable
The transformer layers produce
embedding vectors 𝑣1 , … , 𝑣𝐿 ∈ output projection 𝑤ℓ = 𝐵𝑣ℓ
ℝ𝑑 . These must be enlarged to
dimension 𝑛 to produce a
probability distribution on the 𝑛 hidden dimension 𝑑
possible tokens. (Usually 𝑑 < 𝑛.) doesn’t change × Depth TF layers with causal mask
due to residual
connection
𝑣ℓ = 𝐴𝑢ℓ + 𝑝ℓ
The output projection layer
token embedding + positional encoding
𝑤ℓ = 𝐵𝑣ℓ
does this.
Llama 3.1 405B GPT3 175B
𝜏 depth (layers) 126 depth (layers) 96
𝑑 (hidden) 16k 𝑑 (hidden) 12k
𝑋 𝑛 (tokens) 128k 𝑛 (tokens) 50k
88
Output projection layer
It is customary to do weight tying, and set
𝐵 = 𝐴⊤
where 𝐴 ∈ ℝ𝑑×𝑛 is the matrix for the token embedding layer. Empirically, this works well.#

Why might this be a good idea? In modern signal processing (dictionary learning and
compressed sensing), the transpose of a matrix is often used as an approximate inverse.
Indeed,
𝐴 ∶ token ↦ emb. vector
𝐵 ∶ emb. vector ↦ token
are, loosely speaking, inverse operations. Interestingly, at initialization, 𝐴⊤ does act like an
inverse of 𝐴. More precisely, if 𝐴 ∼ 𝒩 0, 𝜎 2 IID with 𝐴 ∈ ℝ𝑑×𝑛 , then
𝐴⊤ 𝐴 ∝ 𝐼

#O. Press and L. Wolf, Using the output embedding to improve language models, EACL, 2017. 89
Output projection layer
1
At initialization, if 𝐴 ∼ 𝒩 0, 𝜎 2 IID with 𝐴 ∈ ℝ𝑑×𝑛 , then 𝑑𝜎 2 𝐴⊤ 𝐴 ≈ 𝐼.
d, n, sigma = 100, 50, 2.0
A = [Link](scale=sigma, size=(d, n))
print("Sample of A^T A:\n", (A.T @ A /(d*sigma**2))[:5, :5]) # print 5x5 block

Sample of A^T A:
[[ 1. -0.01 -0.06 0. 0. ]
[-0.01 0.9 -0.04 0.09 0.07]
[-0.06 -0.04 0.83 -0.07 -0.08]
[ 0. 0.09 -0.07 0.95 0.04]
[ 0. 0.07 -0.08 0.04 1.19]]
(There is no reason to expect 𝐴⊤ 𝐴 ∝ 𝐼 to continue to hold once training begins.)
Main point: (i) 𝐵 is conceptually performing the inverse operation of 𝐴 (ii) 𝐴⊤ is an
approximate inverse of 𝐴 at initialization (iii) it is sensible to weight-tie 𝐵 = 𝐴⊤ .
90
Self-supervised pre-training
Let 𝑋 be the input text tokenzed as 𝜏 𝑋 = 𝑢1 , 𝑢2 , … , 𝑢𝐿 .

𝐿 𝐿
Let 𝑓𝜃 be the transformer mapping 𝑢ℓ ℓ=1 ↦ 𝑤ℓ ℓ=1 , where 𝑤ℓ ∈ ℝ𝑛 . Then,

and

where ℓCE is the cross-entropy loss with 𝑢ℓ+1 ∈ 1, … , 𝑛 above viewed as an integer.

91
Supervised fine-tuning
First, transform the relevant
text into sequence with
appropriate delimiter tokens.

At the end of the transformer,

the token corresponding to
the “extract” token position is
extracted fed into a linear
layer.

The full GPT-1 model and the

final linear layer is fine-tuned.

92
Supervised fine-tuning
For classification, given an input text 𝑋 and a tokenizer 𝜏, the transformer maps
<Start>, 𝜏 𝑋 , <Extract> = 𝑢ℓ 𝐿ℓ=1 ↦ 𝑤ℓ 𝐿ℓ=1
The final token 𝑤𝐿 corresponding to the <Extract> token, is extracted. The loss is
loss 𝐴𝑤𝐿 + 𝑏, 𝑌
where 𝐴 and 𝑏 are the parameters of the linear layer and 𝑌 is the label corresponding to 𝑋. (Only
𝑤𝐿 is used for the loss and 𝑤ℓ 𝐿−1
ℓ=1 is not used).

BERT had a <Cls> token at the start of the input, and it basically served the same role as the
<Extract> token for GPT. Different from BERT, GPT is a causal language model, so the <Extract>
token must be at the end if we want 𝑤𝐿 to encode information about the full sentence.

The full GPT-1 model (the pre-trained TF), the final linear layer, and the vector embeddings
corresponding to <Start>, <Extract>, and <Delim> are trained.

(LLM fine-tuning is no longer done this way.)

93
 Example task: Machine translation
In machine translation, training data contains translation pairs between different languages.

Classically with an RNN, the encoding stage encodes (summarizes) the entire sentence into
a latent vector, and the decode generates translation text autoregressively.

(For better performance, a stacked Encoder Decoder

bidirectional RNN encoder and a Tu as faim ?
stacked unidirectional RNN decoder
should be used.)

<GO>
(Interestingly, reversing the input Embedding
sentence often improves performance.)
Are you hungry ?

Timestep 1 2 3 4 5 6 7 8
Bahdanau attention and cross attention
The problem with the previous approach is that the hidden state passed from the encoder
RNN to the decoder RNN acts as a bottleneck, and the hidden state may not be able to
retain all the necessary information. 𝑠 𝑠 𝑠 1 𝑡−1 𝑡 𝑠𝑇𝑑

Q K V
Solution: Allow the decoder RNN Query Key Value

cells to access the hidden states of LSTM,

GRU,
LSTM,
GRU,
LSTM,
GRU,
LSTM,
GRU,

the encoder RNNs. RNN RNN RNN RNN

ℎ1 ℎ2 ℎ𝑇𝑒 −1 ℎ𝑇𝑒

Context
This attention mechanism is now
called cross attention, and it is now LSTM,
GRU,
LSTM,
GRU,
LSTM,
GRU,
LSTM,
GRU,
Decoder

commonly used to attend across RNN RNN RNN RNN

different modalities. E.g. text

decoder attends to image patches.
Encoder
D. Bahdanau, K. Cho, Y. Bengio, Neural machine translation by jointly learning to align and translate, ICLR, 2015. 95
(ICLR 2025 Test of Time award Runner Up.)
Transformer and cross attention
Vaswani et al. questioned whether the RNN mechanism
was necessary. They concluded “Attention is all you need”.

𝐿
Cross attention layer derives 𝑞ℓ ℓ=1 from previous layer’s
𝐿 ′ ′
𝑥ℓdec ℓ=1 but 𝑘ℓ 𝐿ℓ=1 and 𝑣ℓ 𝐿ℓ=1 are derived from the
encoder layer’s 𝑥ℓenc 𝐿ℓ=1 . (In cross attention, number of
queries need not match the number of keys and values.)

(Figure incorrectly depicts post-LN.)

A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin, Attention is all you need, NeurIPS, 2017. 96
Understanding TF from historical context
The transformer architecture feels somewhat arbitrary, but we can understand the
designers’ intent through the historical context.

There is no mathematical or first-principles reason that things must be the way they are,
and the standard architecture will likely change in the future.

The historical context does inform us of the intended purpose of the components, and it
gives us a rough guideline of what things will certainty not work and what new components
may work.

97
Byte-pair encoding
Byte-pair encoding (BPE)
is a sub-word tokenizer.

The GPT-1 popularized the use of BPE for LLMs.

Tokenizers are trained separately (before) from the LLM, and the
training minimizes the sequence length of the training corpus,
given a fixed vocabulary size.

For text generation, LLMs generate a sequence of tokens

(integers) and those tokens are reconstructed into text.
Tiktokenizer website: [Link]
P. Gage, A new algorithm for data compression, C Users J., 1994.
R. Sennrich, B. Haddow, and A. Birch, Neural machine translation of rare words with subword units, ACL, 2016. 98
A. Radford, K. Narasimhan, T. Salimans, and I. Sutskever, Improving language understanding by generative pre-training, 2018.
 Byte-pair encoding
The BPE tokenizer can gracefully
deal with misspellings.

Subtleties of the BPE tokenizers:

• Pre-pending space to a word
changes the tokenization.
• Numbers are chunked up
somewhat arbitrarily (making
arithmetic quite difficult for LLMs.)

Andrej Karpathy’s tutorial Let's build the GPT Tokenizer is an excellent resource for learning
about further details of the BPE tokenizer:
[Link]
99
Modern transformers: RMS Norm
Recall that LayerNorm have the form

where 𝑥 = 𝑋 𝑏, ℓ, ∶ (i.e., LN normalizes across the channel/feature dimension). The 𝛾 and 𝛽

are trainable parameters.

Root mean square layer normalization (RMSNorm) simplifies this to

where 𝛾 is the only trainable parameter.

B. Zhang and R. Sennrich, Root mean square layer normalization, NeurIPS, 2019. 100
Modern transformers: RMS Norm
RMSNorm is clearly simpler than LN, but does it matter?

RMSNorm performs (generalizes) similarly to LN.#

RMSNorm is meaningfully faster than LN (due to low-level memory movement issues)
despite having essentially the same flop count.%

#B. Zhang and R. Sennrich, Root mean square layer normalization, NeurIPS, 2019.
%A. Ivanov, N. Dryden, T. Ben-Nun, S. Li, and T. Hoefler, Data movement is all you need: A Case study on optimizing transformers, MLSys, 2021. 101
Modern transformers: No bias
In fact, most modern transformers mostly avoid bias terms.
• Token embedding and output projection layers have no bias terms.
• Attention layers are designed without bias terms.

The original transformer used bias terms in the FFN layer:

But modern implementations do not. If not gated, the have the form:

102
Modern transformers: GLU
Gated Linear Units (GLU) slightly improve the FFN layers:

where ⊙ is elementwise multiplication. The trainable parameters are 𝑊1 , 𝑊2 , and 𝑉.

• If 𝜎 is ReLU, this FFN is called ReGLU.

• If 𝜎 is GeLU, this FFN is called GeGLU.
• If 𝜎 is swish, this FFN is called SwiGLU.

As the table from# shows, GLU variants work slightly bettter

than non-gated FFNs.

Y. N. Dauphin, A. Fan, M. Auli, and D. Grangier, Language modeling with gated convolutional networks, ICML, 2017.
#N. Shazeer, GLU variants improve transformer, arXiv, 2020. 103
Modern transformers: GLU
But doesn’t this increase the compute cost?

For non-gated FFNs, the standard dimension parameters are

𝑊1 ∈ ℝ4𝑑×𝑑 and 𝑊2 ∈ ℝ𝑑×4𝑑 . (Many independent tuning attempts have lead to the
hyperparameter value of 4 )

For gated FFN, the standard dimension parameters are

𝑊1 , 𝑉 ∈ ℝ 8/3 𝑑×𝑑 and 𝑊2 ∈ ℝ𝑑× 8/3 𝑑
.

With the value 8/3 , the flop count is ∼ 16𝑑2 for both architectures.

104
Classical positional embeddings
𝐿 𝐿
Recall classical positional embeddings: After token embedding layer 𝑋 ↦ 𝑢ℓ ℓ=1 ↦ 𝑣ℓ ℓ=1 ,
add positional embedding vectors 𝑝ℓ 𝐿ℓ=1 and then pass
𝑣ℓ + 𝑝ℓ 𝐿ℓ=1
as input to the transformer layers. Then,
𝑞ℓ = 𝑊𝑄⊤ 𝑣ℓ + 𝑝ℓ
𝑘ℓ = 𝑊𝐾⊤ 𝑣ℓ + 𝑝ℓ
𝑣ℓ = 𝑊𝑉⊤ 𝑣ℓ + 𝑝ℓ

A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin, Attention is all you need, NeurIPS, 2017. 105
Modern transformers: RoPE
Rotary Position Embedding (RoPE) instead identifies
𝑊𝑄⊤ 𝑣ℓ , 𝑊𝐾⊤ 𝑣ℓ ∈ ℂ𝑑𝐾 /2 ≅ ℝ𝑑𝐾
and

So,

J. Su, Y. Lu, S. Pan, A. Murtadha, B. Wen, and Y. Liu, RoFormer: Enhanced transformer with rotary position embedding, Neurocomputing, 2024. 106
Modern transformers: RoPE
Then, RoPE feeds the rotated 𝑞 and 𝑘 vectors into the attention function. (𝑣 vectors are not
rotated.)

J. Su, Y. Lu, S. Pan, A. Murtadha, B. Wen, and Y. Liu, RoFormer: Enhanced transformer with rotary position embedding, Neurocomputing, 2024. 107
Modern transformers: RoPE
Properties of RoPE:
• The rotation operation ensures that only relative positional information influences the
attention mechanism, i.e.,

for some function 𝑓.

• Long-term decay in the sense of 𝐴ሚ𝑖𝑗 → 0 as 𝑖 − 𝑗 → ±∞ given fixed embedding vectors
𝑥𝑖 and 𝑥𝑗 . This is desirable since we expect the tokens’ interactions to weaken as they
get farther away.
• Can be implemented very efficiently by further simplifying the formulas and using

(rotating forward by ℓ𝜃𝑖 and backward by ℓ′𝜃𝑖 = rotating forward by ℓ − ℓ′ 𝜃𝑖 .) 108

RoPE works quite well.

[Link] 109
Apply RoPE on all layers
Classical positional embeddings usually applied once, at the input layer.
• Applying positional encoding to all of the layers is arguably unnecessary, since the
residual connections provide the positional encoding information to all of the layers.

In contrast, RoPE is usually applied many times, at all attention layers.

110
 NoPE: No positional encodings
Actually, positional encodings aren’t required for decoder-only attention layers. (The un-
masked encoder only transformers like BERT or ViT do need positional encodings.)

Transformers without position encodings (NoPE) may work just fine.#

Llama 4% uses interleaved attention layers without positional embeddings. (Some layers are
RoPE and some layers are NoPE.)

Decoder only transformers are not permutation-equivariant because of the causal mask.
The layers can learn to count the number of tokens they can attend, which reveals the
absolute position.
#A. Haviv, O. Ram, O. Press, P. Izsak, and O. Levy, Transformer language models without positional encodings still learn posit ional information, EMNLP, 2022.
#A. Kazemnejad, I. Padhi, K. Natesan, P. Das, and S. Reddy, The impact of positional encoding on length generalization in transformers, NeurIPS, 2023.
%[Link]
111
Modern transformers: KV caching
During inference, the LLM generates tokens 𝑦1 , … , 𝑦𝐿
sequentially.

To compute 𝑦ℓ , we need 𝑞ℓ , 𝑘1 , … , 𝑘ℓ−1 , 𝑘ℓ , 𝑣1 , … , 𝑣ℓ−1 , 𝑣ℓ .

KV caching is an optimization that stores (caches) the

previously computed 𝑘1 , … , 𝑘ℓ−1 , 𝑣1 , … , 𝑣ℓ−1 . Only 𝑘ℓ and 𝑣ℓ
are newly computed at the ℓ-th step.

KV caching uses trades off memory usage with compute.

112
Modern transformers: KV caching

Image credit: João Lages, [Link] 113

Modern transformers: Multi-query
attention (MQA)
1 𝐻
Attention layers have 𝐻 heads, so computing 𝑦ℓ , … , 𝑦ℓ requires:
ℎ ℎ ℎ
• Computing 𝑞ℓ , 𝑘ℓ , 𝑣ℓ for ℎ = 1, … , 𝐻. (Fast )
ℎ ℎ ℎ ℎ
• Loading from GPU memory 𝑘1 , … , 𝑘ℓ−1 , 𝑣1 , … , 𝑣ℓ−1 for ℎ = 1, … , 𝐻. (Slow )

GPUs are very efficient with compute but are less efficient with memory IO.

Multi-query attention (MQA) shares the keys and values across the heads. So computing
1 𝐻
𝑦ℓ , … , 𝑦ℓ requires:
ℎ
• Computing 𝑞ℓ , 𝑘ℓ , 𝑣ℓ for ℎ = 1, … , 𝐻. (Fast )
• Loading from GPU memory 𝑘1 , … , 𝑘ℓ−1 , 𝑣1 , … , 𝑣ℓ−1 . (IO is slow but volume is small )

N. Shazeer, Fast transformer decoding: One write-head is all you need, arXiv, 2019. 114
Modern transformers: Grouped-query
attention
MQA significantly reduces inference cost, albeit with a slight degradation in performance.
Grouped-query attention (GQA) offers a compromise by sharing key and value vectors
within groups. GQA nearly matches the performance of plain multi-head attention (MHA).

J. Ainslie, J. Lee-Thorp, M. de Jong, Y. Zemlyanskiy, F. Lebron, and S. Sanghai, GQA: Training generalized multi-query transformer models from multi-he
ad checkpoints, EMLNP, 2023. 115
How to decoding with causal LM
Assume a causal language model 𝑝𝜃 𝑢ℓ 𝑢1 , … , 𝑢ℓ−1 has been trained. If 𝑝𝜃 were perfect,
then naïve sampling (as defined soon) should be sufficient for text generation (decoding).
However, 𝑝𝜃 is imperfect, so effective sampling requires the following techniques:
• Naïve sampling (with temperature)
• Greedy sampling
• Beam search
• Top-k sampling
• Top-p (nucleus) sampling

116
Naïve sampling
Let 𝑓𝜃 𝑢1 , … , 𝑢ℓ ∈ ℝ𝑛 be the final ℓ-th output token of a decoder-only transformer
architecture (𝑛 is the number of distinct tokens) such that
𝑝𝜃 𝑢ℓ+1 = 𝑖 𝑢1 , … , 𝑢ℓ = softmax 𝑓𝜃 𝑢1 , … , 𝑢ℓ
𝑖
for 𝑖 = 1, … , 𝑛. Let 𝑢1 , … , 𝑢ℓ be given and 𝑢ℓ ≠ <EOS>.

Naïve sampling continues the text with

𝑢𝑠 ∼ 𝑝𝜃 ⋅ 𝑢1 , … , 𝑢𝑠−1
for 𝑠 = ℓ + 1, ℓ + 2, … until 𝑢𝑠 = <EOS> (and sets 𝐿 = 𝑠 − 1).

Problem: Low-probability words are sampled with low but non-zero probabilities, and they
tend to be bad. The imperfections of 𝑝𝜃 manifest in these low-probability words.

117
Naïve sampling with temperature
Naïve sampling with temperature adjusts the “temperature” parameter of the softmax
𝑓𝜃 𝑢1 , … , 𝑢ℓ
𝑝𝜃 𝑢ℓ+1 = 𝑖 𝑢1 , … , 𝑢ℓ = softmax
𝛽
𝑖

for 𝑖 = 1, … , 𝑛, where 𝛽 > 0. Note, 𝛽 = 1 is regular naïve sampling. With 𝛽 < 1, we suppress
the likelihood of the low-probability words. In modern LLMs, 𝛽 = 0.7 or 𝛽 = 1 are common
default choices.

In the limit of 𝛽 → 0, we recover greedy sampling.

Problem: Even with 𝛽 < 1, low-probability words are still sampled.

118
Greedy sampling
If the low probability words are problematic, then why not just sample the most likely word?

Greedy sampling continues the text with

𝑢𝑠 = argmax 𝑝𝜃 𝑢𝑠 𝑢1 , … , 𝑢𝑠−1
𝑢𝑠 =1,…,𝑛

for 𝑠 = ℓ + 1, ℓ + 2, … until 𝑢𝑠 = <EOS>. (Ties in the argmax are broken arbitrarily.)

Problem 1) Greedy sampling does not generate the most likely sequence tokens.
Problem 2) Likely text is not always good. More on this later.

119
Exact MAP decoding
Exact maximum a posteriori (MAP) decoding is the globally optimal (most likely) generation
𝐿

𝑢ℓ+1 , … , 𝑢𝐿 = <EOS> = argmax ෍ log 𝑝𝜃 𝑢𝑠 𝑢1 , … , 𝑢𝑠−1

𝑢ℓ+1 ,𝑢ℓ+2 ,…
𝑠=ℓ+1

Problem 1) Not implementable. Exponentially many candidate sequences. Computing exact

MAP decoding is intractable (NP-Hard).#
Problem 2) Likely text is not always good. More on this later.

#K. Knight, Decoding complexity in word-replacement translation models, Computational Linguistics, 1999. 120
Why greedy ≠ MAP? puppy: 0.05
A: 0.48 puppy: 0.53
is: 0.7
One: 0.51 cute: 0.45
MAP maximizes the product of the <EOS>: 0.24

probabilities (= sum of log-

probabilities). Greedy commits to
Transformer
the most likely individual word one
word at a time. Translate the French “un chiot mignon” to English: One puppy is(??)

You want to see how the

completion pans out before puppy: 0.95
puppy: 0.47
committing to a word. A: 0.48
is: 0.01
One: 0.51 cute: 0.53
<EOS>: 0.02

Sometimes, by committing to the

most likely token early on, you Transformer
eliminate better completion paths.
Translate the French “un chiot mignon” to English: A cute puppy

121
Beam search decoding
Beam search is a tractable, heuristic approximation to exact MAP decoding that produces
sequences with higher likelihood compared to greedy search.

Intuition: While choosing highest probability word on the first step may not be optimal,
choosing a very low probability word is very unlikely to lead to a good result. (We shouldn’t
be fully greedy, but we can be somewhat greedy.)

Idea: On each step of the decoder, keep track of the 𝑘 most probable partial generations.
𝑘 is also called the beam size and 𝑘 = 5 or 𝑘 = 10 are common values.

122
 Beam search decoding: Example
Beam size k = 3. Blue numbers : score 𝑦1, … , 𝑦𝑡 = σ𝑖 log 𝑃𝐿𝑀(𝑦𝑖 |𝑦1, … , 𝑦𝑖−1 , 𝑥) .
-1.2 -2.5 -3.4
little frog lived
-1.6 -1.9 -2.9
small bird built
-1.5 -2.6 -3.1
kind turtle ate
-0.5
-2.1
a -2.5 -3.2
small
-0.9
bird lived
-1.9
Once upon a time, the -2.4 -3.5
young
cat knew
-0.7 -1.3
-2.7 -3.7
every wise
dog helped
-1.7 -2.6 -3.6
kind cat lived
-1.8 -3.2 -4.1
living frog ate
-2.0 -2.0 -3.0
young dog knew 123
Beam search decoding: Pseudocode

For each sequence step ℓ:

ℓ 1 𝑘
1. For each 𝑢𝑖,𝑡 𝑡=1
that we are tracking for 𝑖 = 1, … , 𝑘, find the top 𝑘 tokens 𝑢𝑖,ℓ+1 , … , 𝑢𝑖,ℓ+1 .
2. Sort the resulting 𝑘 2 sequences (of length ℓ + 1) by their total log-probability.
3. End loop if a generation encounters <EOS> and return the completed generation.
4. Keep the top 𝑘.

When 𝑘 = 1, beam search reduces to greedy search.

124
Human text is not too predictable
High quality human text does not
follow a distribution of high
probability next words.

Beam search avoids generation

with elements of surprise and
suffers from repetitive and
uninteresting content.

Beam search is no longer used in

mainstream LLMs. As models
became better, the lower
probability tokens became better.

A. Holtzman, J. Buys, L. Du, M. Forbes, and Y. Choi, The curious case of neural text degeneration, ICLR, 2020. 125
Top-K sampling
Top-K sampling samples among the 𝐾 most likely word, with probabilities determined by softmax
applied to the top 𝐾 words with a temperature 𝛽 > 0.
1.0

(𝐾 = 6 in this example)

0.0
child train cat house man boy tree girl guy nice arrives is leaves was departs stopscrashes on rolled with

𝐾 ∼ 50 is a is a common default choice. 𝐾 = ∞ means top-K is not used.

Problem) Low-probability words can still be sampled.

A. Fan, M. Lewis, and Y. Dauphin, Hierarchical neural story generation, ACL, 2018. 126
Top-p (nucleus) sampling
Top-p sampling samples among the fewest most likely words such that their probabilities
(with a temperature 𝛽 > 0) exceeds probability 𝑝. Once the set top words 𝑉 𝑝 are defined
𝑝𝜃 𝑢ℓ+1 = 𝑖 𝑢1 , … , 𝑢ℓ
𝑝෤𝜃 𝑢ℓ+1 = 𝑖 𝑢1 , … , 𝑢ℓ =
σ𝑖∈𝑉 𝑝 𝑝𝜃 𝑢ℓ+1 = 𝑖 𝑢1 , … , 𝑢ℓ
sampling is done from 𝑝෤𝜃 . (So σ𝑖∈𝑉 𝑝 𝑝𝜃 𝑢ℓ+1 = 𝑖 𝑢1 , … , 𝑢ℓ ≥ 𝑝.)
(𝑝 = 0.9 in this example)
1.0
𝑝 ∼ 0.9 is a common default choice.

𝑝 = 1 means top-p is not used.

0.0
child train cat house man boy tree girl guy nice arrives is leaves was departs stops crashes on rolled with

A. Holtzman, J. Buys, L. Du, M. Forbes, and Y. Choi, The curious case of neural text degeneration, ICLR, 2020. 127
Combining , top-K, and top-p
Most modern LLMs combine the temperature parameter 𝛽, top-K, and top-p.

Precisely how these are combined slightly differ from model to model.

128
LLMs as a universal interface
As LLMs were scaled up, researchers started to notice a crucial capability of LLMs emerge:
LLMs can just follow textual instructions.

Prior paradigm: Labeled data defines the task. This was the case prior to (sufficiently large)
language models. Self-supervised pre-training on large unlabeled text would improve the
efficiency, but labeled data was still needed to define the task.

New paradigm: Define the task with a natural language description. Researchers gradually
crystalized this paradigm through GPT-2, GPT-3, T5, FLAN, and Flan-PaLM.

129
 GPT-2 and GPT-3
The GPT-2 and GPT-3 papers had very similar titles and messages. GPT-3 simply scales up
GPT-2 and achieves stronger results. (Architecture didn’t change much from GPT-1.)
• GPT-(1,2,3) Model size: 117M→1.5B→175B
• GPT-(1,2,3) Data size: 4GB→40GB→600GB

Main message: GPT can solve many tasks with a unified task-agnostic architecture and without
supervised fine-tuning. Task-specific training data is not used (zero-shot) or only a few is used
during inference (few-shot in-context learning).

No task-specific training data is used for training or fine-tuning. (However, having diverse task-
specific training data is helpful, as T5 and Flan-T5 shows.)

(ELMo was not at all task agnostic. BERT and GPT-1 was a little more task agnostic but had
task-specific heads.)
A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, and I. Sutskever, Language models are unsupervised multitask learners, Tech. Report, Feb. 2019.
T. B. Brown, B. Mann, N. Ryder, M. Subbiah …, A. Radford, I. Sutskever, and D. Amodei, Language models are few-shot learners, NeurIPS, 2020. (arXiv May 2020) 130
Zero-shot and few-shot learning
Few-shot learning refers to a model learning a behavior with a few labeled samples or
demonstrations. Classically, few-shot learning would have a pre-trained model fine-tuned
(gradient updates) on the few labeled data points.

Zero-shot learning would mean performing a task with no demonstrations. How is this
possible?

Zero-shot summarization with GPT-2:

“To induce summarization behavior, we add the text TL;DR: after the article and generate
100 tokens with Top-k random sampling ….”
GPT-2 can perform summarization without any explicitly paired summarization dataset.

A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, and I. Sutskever, Language models are unsupervised multitask learners, Tech. Report, Feb. 2019. 131
In-context learning (ICL)
In In-context learning (ICL), first explicitly reported with GPT-3#, the model discerns the task
implied by the context of demonstrations with or without explicit descriptions.

#T.
B. Brown, B. Mann, N. Ryder, M. Subbiah …, A. Radford, I. Sutskever, and D. Amodei, Language models are few-shot learners, NeurIPS, 2020. (arXiv
May 2020) 132
ICL example

133
In-context learning (ICL)
GPT-2# had hints of in-context learning capabilities: “Similar to translation, the context of the
language model is seeded with example question answer pairs which helps the model infer
the short answer style of the dataset.”

The term in-context learning refers to the LLM’s ability to learn the user’s intent through
examples within the context of the text and without any parameter updates. You show a
handful of demonstrations (labeled datapoints) and ask the model to “follow the examples”.
This is few-shot learning with no gradient updates.

ICL is a crucial capability, since most natural language instructions do not specificy all
details with complete precision. It is important that models understand what you mean
through examples.

#A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, and I. Sutskever, Language models are unsupervised multitask learners, Tech. Report, Feb. 2019. 134
T5 model

Text-to-text-transfer-transformer (T5) uses an encoder-decoder transformer and formats all

pre-training and fine-tuning into a text-to-text format.

Unified task-agnostic architecture. The many tasks, which are not semantically related, are
formatted into a text-to-text format. Same model, objective, training procedure and decoding
process to every task that we consider.

C. Raffel, N. Shazeer, A. Roberts, K .Lee, S. Narang, M. Matena, Y. Zhou, W. Li, and P. J. Liu, Exploring the limits of transfer learning with a unified text-
to-text transformer, JMLR, 2020. (arXiv Oct. 2019) 135
T5 pre-training
Pre-training on large unlabeled text with diverse objectives inspired by prior work.

The “inputs” are fed into the encoder block while the “target” text is generated by the
decoder one token at a time.

136
T5 fine-tuning
Simultaneously fine-tune on a wide range of tasks. Simply prompt the model differently for
each task to inform T5 of the specific task to solve.

137
T5 contribution

Advanced state-of-the-art with the pre-train-than-fine-tune approach.

Further demonstrated the idea that language models can understand and respond to
natural language instructions. We can simply tell a language model what we want and it will
follow our instructions.

Problem: The prompts were unnatural as they did not fully describe the task at hand. (What
does “stsb sentence 1” mean?) It is a half-way measure between a fully arbitrary label (like
“task 3A”) and a full natural-language description.

C. Raffel, N. Shazeer, A. Roberts, K .Lee, S. Narang, M. Matena, Y. Zhou, W. Li, and P. J. Liu, Exploring the limits of transfer learning with a unified text-
to-text transformer, JMLR, 2020. (arXiv Oct. 2019) 138
Instruction fine-tuning
Instruction fine-tuning,
presented in the FLAN# and T0*
papers, fine-tunes a pre-trained
model on a collection of
datasets described via natural-
language instructions.

Unification: All tasks are sub-

tasks of the meta task of
following the natural language
instruction.

#J. Wei, M. Bosma, V. Zhao, K. Guu, A. W. Yu, B. Lester, N. Du, A. M. Dai, Q. V. Le, Finetuned language models are zero -shot learners, ICLR, 2022.
(arXiv Sept. 2021)
*V. Sanh, A. Webson, C. Raffel, S. H. Bach, …, Alexander M. Rush, Multitask prompted training enables zero-shot task generalization, ICLR, 2022. (arXiv
139
Oct. 2021)
Instruction fine-tuning
FLAN is a 137B parameter model instruction
fine-tuned on over 60 NLP datasets with
instructions verbalized via natural language
instruction templates.

J. Wei, M. Bosma, V. Zhao, K. Guu, A. W. Yu, B. Lester, N. Du, A. M. Dai, Q. V. Le, Finetuned language models are zero-shot learners, ICLR, 2022.
(arXiv Sept. 2021) 140
Scaling instruction fine-tuning
Flan-PaLM scales instruction
fine-tuning up to a 540B
model with 1836 instruction-
finetuning tasks.

H. W. Chung, L. Hou, S. Longpre, … Jason Wei, Scaling instruction-finetuned language models, JMLR, 2024. (arXiv Oct. 2022) 141
Scaling instruction fine-tuning
Key finding: Task diversity is essential not just in having the model be multi-task, but also in
benefiting the individual task performances. Training on tasks A, B, C, … improved
performance on task A.

H. W. Chung, L. Hou, S. Longpre, … Jason Wei, Scaling instruction-finetuned language models, JMLR, 2024. (arXiv Oct. 2022) 142
Adding special tokens
During pre-training, LLMs are trained with some special tokens such as <|endoftext|>.

During instruction fine-tuning, it is important to add additional special tokens such as

<|im_start|> and <|im_end|>, which stand for “instruction message start/end”.

This is done by increasing the vocabulary size and adding new columns to the token
embedding matrix 𝐴, which also used in the output projection step when weight tying is
used. The new columns are randomly initialized.

Although the data size and the number of updates during fine-tuning is much smaller than
that of pre-training, LLMs can learn to use special tokens relatively quickly.

143
Chat template with special tokens
The special tokens <|im_start|> and <|im_end|> are used to format the conversation in a
chat format. This trick significantly improve the LLMs ability to follow instructions and
engage in turn-based conversations. (This is not optional.)

<|im_start|>system
You are a friendly chatbot who always responds in the style of a pirate<|im_end|>
<|im_start|>user
How many helicopters can a human eat in one sitting?<|im_end|>
<|im_start|>assistant
Oh just 6.<|im_end|>
<|im_start|>user
Are you sure about that?<|im_end|>
<|im_start|>assistant …

144
Before and after instruction fine-tuning
User Prompt:
Q: What is the capital of France?

Response without instruction fine-tuning: Response after instruction fine-tuning:

Q: What is the currency of France? A: The capital of France is Paris.
Q: What is the population of France?
Q: What is the official language of France?

145
Bitter Lesson II
“A counterintuitive implication of scale: trying to solve a
more general version of the problem is an easier way to
solve the original problem than directly tackling it.
Attempting a more general problem encourages you to come up with a more general and
simpler approach. This often leads to a more scalable method. By leveraging increasingly
cheaper compute, you solve the specific problem as a by-product of tackling a more general
one.
Some examples:
- Directly solving NLU tasks (e.g. question answering) vs. learning a general language
model and solving the task as a next token prediction.
- Instead of directly working on machine translation, work on a general problem of learning
all languages (mT5 vs. translation-specific models).”
— Hyung Won Chung —
October 11, 2023
[Link] 146
3-stage training of LLMs
Gradually, researchers have adopted a three-stage training process for LLMs.
(Although this paradigm is now evolving with the advent of mid-training.)

Pre-training Instruction fine-tuning RLHF alignment

1. Pre-traigning produces model with base capabilties, but the model just tries to complete text
and babble on. Model does not have the propensity to follow instructions or be helpful. (Pre-
training is the most compute-heavy.)

2. (Part of post-training) Instruction fine-tuning induces the model to follow instructions and be
“helpful.” Model can engage in chat-bot-style dialogue after instruction fine-tuning.

3. (Part of post-training) RLHF further aligns LLM with human values and expectations.
147