Lecture: Large Language

Models

AIC365: Natural Language Processing

1
Agenda
• Language Model
• T5
• In-context Learning
• GPT-3
• Codex
• Llama-2
• Mixtral of Experts
• PaLM
• Q&A Session

2
 What is Large Language Model (LLM)?¹
- Language models are computational models that
have the capability to understand and generate
human language.
- Deep learning algorithm that can perform various NLP
tasks
- Are trained on massive datasets that allow them to
recognise, translate, predict, or generate text or other
content
- Unsupervised multi-task learners

1. Chang, Y., Wang, X., Wang, J., Wu, Y., Zhu, K., Chen, H., Yang, L., Yi, X., Wang, C., Wang, Y., Ye, W., Zhang, Y., Chang, Y., Yu, P.S., Yang, Q., & Xie, X. (2023). A
Survey on Evaluation of Large Language Models. ArXiv, abs/2307.03109.

3
The Inputs and Outputs of a Trained
Transformer LLM

Figure. At a high level of abstraction, Transformer

LLMs take a text prompt and output generated
text.
Large Language Model (LLM) Comparison

Comparison Traditional Deep LLMs

ML Learning

Training Data Size Large Large Very Large

Feature Engineering Manual Automatic Automatic

Model Complexity Limited Complex Complex

Interpretability Good Poor Poorer

Performance Moderate High Highest

Hardware Low High Very High

Requirements

5
Natural Language Processing Tasks
• Natural Language Understanding
• Sentiment Analysis: This is a classification task. It analyzes and interprets text
to determine their emotional inclination. The result is usually a binary (positive
and negative) or triple (positive, neutral or negative)
• Text Classification: This is related to sentiment analysis though encompasses
more.

• Natural language Inference (NLI): determination of whether a given

“hypothesis” logically follows from a given “premise”. ChatGPT does very well on
NLI tasks.
• Semantic Understanding: Interpretation and comprehension of words,
phrases, sentences and the relationships between them.
• Reasoning
• Models must comprehend provided information and utilize reasoning and
influence to deduce answers when explicit responses are absent.
• Natural Language Generation
• Summarization: ability to create a concise abstract for a given sentence or
paragraph.
• Question answering: Creating answers to specific questions
6
T5 Framework²
• Unified (text) approach to tasks
Translate English to German: That is good
Das is gut

cola sentence: The course is jumping well.

Not acceptable

Stsb (similarity score 1-5)

T5
Sentence 1: The rhino grazed on the grass,
3.8
sentence 2: A rhino is grazing in a field

Summarize: state authorities dispatched emergency crews Six people hospitalized after a
tuesday to survey the damage after an onslaught of severe storm in attala county
weather in Mississippi

2. Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., ... & Liu, P. J. (2020). Exploring the limits of transfer learning
with a unified text-to-text transformer. The Journal of Machine Learning Research, 21(1), 5485-5551.

7
 T5: Transformer Architecture
Uses word2vec to generate numeric
representation vector for each token in
Input Input Embedding the input sequence

Positional Encoding
Decoding
Component
Decoder
Encoding
Component
Encoder Decoder

Encoder Decoder

Encoder
Linear

Softmax
Output
8
Transformer Architecture
Encoding Decoding Component
Component Decoder
Self-Attention
Encoder
Self-Attention
Encoder-Decoder Attention

Feed-Forward Feed-Forward

Decoder

Encoder

Linear

Softmax
Output
9
What is Attention??
• Imagine you’re reading this sentence:
• “The dog chased the cat because it was fast.”
• To understand what “it” refers to, your brain pays
attention to the nearby words “dog” and “cat.”
You decide — “it” most likely refers to the cat, not the
dog.
• That’s exactly what attention does in a language
model.
It helps the model focus on the most relevant words
when understanding or generating text.
10
How Attention Works?
• Let’s take a short sentence:
“I love machine learning.”
• Each word (token) gets turned into a small vector of
numbers (an embedding).
• The model then tries to understand:
• How much should “I” pay attention to “love” or
“machine”?
• Which words are most important for each other?

11
How Attention Works?
• It calculates this using three matrices for each token:

Term Meaning Analogy

Query (Q) What I’m looking for “What do I care about?”
Key (K) What I offer “What am I about?”
“What information do I
Value (V) My information
carry?”

12
How Attention Works?
• For each word:
• Compute Q, K, and V vectors.
• Compute attention score between words:
• Score = Q × Kᵀ
• (If the score is high → they relate strongly.)
• Use softmax to turn these scores into probabilities.
• Use those probabilities to mix the Value (V) vectors →
forming a new “context-aware” representation.

13
T5 Transformer Model Architecture
- Encoder/decoder blocks similar in
sizes
- Each block comprises self-
attention, optional encoder-
attention, and a feed-forward
network

14
T5 Transformer Attention³
• Encoder-decoder attention
• Queries come from the previous decoder layer
• The memory keys and values come from the output of the encoder
• Allows every position in the decoder to attend to all positions in the input
sequence
• Helps the model align words in the input sequence with the words in the output
sequence

• Encoder contains self-attention

• All of the keys, values and queries come from the same place, the output of the
previous layer in the encoder
• Each position in the encoder can attend to all positions in the previous layer of
the encoder

• Masked self-attention in the decoder

• Allows each position in the decoder to attend to all positions in the decoder up to
and including that position
• Prevents leftward information flow in the decoder via a scaled dot-product
attention
3. A. Vaswani by masking
et al., “Attention out
is All you ( setting
Need,” to in
in Advances -∞) all Information
Neural values Processing Systems
(NeurIPS), 2017.

15
T5 Architecture variants
• A major distinguishing factor is the “mask” used by
different attention mechanisms in the mode
• Blocks represent elements of sequence, lines
attention visibility
• Dark grey lines correspond to fully-visible masking
and light grey lines correspond to causal masking

16
T5 Attention Mask Patterns

17
T5 Performance of architecture variants

Architecture Objective Params Cost GLUE CNNDM SQuAD SGLUE EnDe EnFr EnRo
*Encoder-decoder Denoising 2P M 83.28 19.24 80.88 71.36 26.98 39.82 27.65
Enc-dec, shared Denoising P M 82.81 18.78 80.63 70.73 26.72 39.03 27.46
Enc-dec, 6 layers Denoising P M/2 80.88 18.97 77.59 68.42 26.38 38.40 26.95
Language model Denoising P M 74.70 17.93 61.14 55.02 25.09 35.28 25.86
Prefix LM Denoising P M 81.82 18.61 78.94 68.11 26.43 37.98 27.39
Encoder-decoder LM 2P M 79.56 18.59 76.02 64.29 26.27 39.17 26.86
Enc-dec, shared LM P M 79.60 18.13 76.35 63.50 26.62 39.17 27.05
Enc-dec, 6 layers LM P M/2 78.67 18.26 75.32 64.06 26.13 38.42 26.89
Language model LM P M 73.78 17.54 53.81 56.51 25.23 34.31 25.38
Prefix LM LM P M 79.68 17.84 76.87 64.86 26.28 37.51 26.76

18
T5 Input - Colossal Clean Crawled Corpus

- 20TB of text data extracted from web pages each

month
- Needed to be cleaned up
- Heuristics for cleaning up
- Remove boilerplate texts
- Remove duplicates
- Retained lines containing at least 5 words

19
T5: Downstream Tasks
- Measure general language learning abilities
- Sentence acceptability judgement
- Sentiment analysis
- Paraphrasing/sentence similarity
- Natural language inference
- Coreference resolution
- Sentence completion
- Word sense disambiguation
- Question answering

20
T5: Training the model
- All tasks are formulated as text-to-text tasks
- Pre-train each model for 2¹⁹ = 524,288 steps
- Use a maximum sequence of 512 and a batch size of 128
sequences
- Pack multiple sequences into each of batch 2¹⁶ or 65,536
tokens
- Pre-training batch size X number of steps 2³⁵≈ 34B tokens
- 2³⁵ tokens only covers a fraction of the entire C4 data set
- Learning rate is inverse square root schedule: 1 / √max(n,k)
where n is the current training iteration and k is the number of
warm-up steps (set to 10⁴)
- Sets the learning rate of 0.01 for the first 10⁴ steps, then
exponentially decays the learning rate until pre-training is
over.
- Learning rate of 0.001 when fine-tuning
21
T5 - Unsupervised Objectives
- Provides mechanism through which the model gains
general-purpose knowledge to apply to downstream tasks
- Ingest a sequence of token IDs corresponding to a span of
text from input unlabelled text data set
Original Text

Thank you for inviting me to your party last week.

Inputs

Thank you <X> me to your party <Y> week.

Targets
<X> for inviting <Y> last <Z>

22
T5 - Pre-training Data set
Corruption Corruption Corrupted span
High-level strategies rate length
approaches

10% 2
Language Mask
Modelling

Replace 15% 3
spans
BERT-style

25%
Drop 5

Deshuffling

50% 10

23
T5 - Performance Results

Data set Size GLUE CNNDM SQuAD SGLUE EnDe EnFr EnRo
*C4 745GB 83.28 19.24 80.88 71.36 26.98 39.82 27.65
C4, unfiltered 6.1TB 81.46 19.14 78.78 68.04 26.55 39.34 27.21
RealNews-like 35GB 83.83 19.23 80.39 72.38 26.75 39.90 27.48
WebText-like 17GB 84.03 19.31 81.42 71.40 26.80 39.74 27.59
Wikipedia 16GB 81.85 19.31 81.29 68.01 26.94 39.69 27.67
Wikipedia +TBC 20GB 83.65 19.28 82.08 73.24 26.77 39.63 27.57

24
T5 - Pre-training Loss

25
T5 Scaling
- Increasing compute power results in better
performance
- Baseline model has 220M parameters, is pre-trained
and fine tuned for 2¹⁹ and 2¹⁸ steps respectively
- Increasing training time and/or model size
- Increasing baseline model size
- Undertake longer training to improve performance
- Scale up model sizes

26
Reflections on T5
• -Text-to-text
• provides a simple way to train a single model on a wide variety of tasks
using the same loss function and decoding procedure
• Successfully applied to abstractive summarization, classification tasks
like natural language inference, and regression task like STS-B
• Comparable performance to task-specific architectures
• Architectures
• Original encoder-decoder form worked best
• Uses twice as many parameters as “encoder-only” (e.g. BERT)
• Unsupervised objectives:
• “Denoising” objectives train the model to reconstruct randomly
corrupted text performs well
• Data sets - Used the (Colossal Clean Crawled Corpus) C4 data set
• Training strategies
• Updating all of pre-trained model’s parameters
• Scaling
27
In-Context Learning
• Need for a large dataset for every task limits
applicability of language models - its not practical
• Potential to exploit spurious correlations in training data
grows with the expressiveness of the model and
narrowness of the training distribution Language
models are few-shot learners
• Humans do not require large supervised datasets to
learn most language tasks - a brief directive is sufficient
• Meta-learning or zero-shot transfer allows the model to
develop a broad set of skills and pattern recognition
abilities at training time
• Not as performant as reinforcement learning from
human feedback (RLHF)
28
 Language model meta-learning⁴
• Language model develops a broad set of skills and
pattern recognition abilities during training

29
4. Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., ... & Amodei, D. (2020). Language models are few-shot learners. Advances in neural information processing
systems, 33, 1877-1901.
Language model meta-learning
• Task: Remove random symbols from a word
• Larger models make increasingly efficient use of in-
context info
• Params: weights + biases
• Eval: GPT-3

30
GPT-3 Architecture

31
GPT-3 Training Approaches

32
GPT-3 - Training dataset
• Based on raw Common Crawl dataset of up to 1T
words
• Cleaned up original datasets by:
• Filtered Common Crawl based on similarity to a range of
high-quality reference corpora
• Performed fuzzy deduplication at the document level
• Added known high-quality reference corpora to the training
mix
• Used final cleaned up data in training

33
GPT-3 - Training dataset
- Sizes, architectures, and learning hyper-parameters (batch size in
tokens and learning rate) of the models trained
- All models were trained for a total of 300 billion tokens

Model Name nparams nlayers dmodel nheads dhead BatchSize Learning Rate
GPT-3 Small 125M 12 768 12 64 0.5M 6.0 x 10-4
GPT-3 Medium 350M 24 1024 16 64 0.5M 3.0 x 10-4
GPT-3 Large 760M 24 1536 16 196 0.5M 2.5 x 10-4
GPT-3 XL 1.3GB 24 2048 24 128 1M 2.0 x 10-4
GPT-3 2.7B 2.7GB 32 2560 32 80 1M 1.6 x 10-4
GPT-3 6.7B 6.7GB 32 4096 32 128 2M 1.2 x 10-4
GPT-3 13B 13.0B 40 5144 40 128 2M 1.0 x 10-4
GPT-3 175B or “GPT-3” 175.0B 96 12288 96 128 3.2M 0.6 x 10-4

34
GPT-3 Compute Consumption

35
GPT-3 Limitations
• Limitations in text synthesis
• Structural and algorithmic limitations
• Poor sample efficiency during pre-training
• Lack of interpretability
• Can perpetuate and amplify existing biases and
unfairness in society
• Multilingualism - majority language model researches
are done in English

36
CodeX introduction
- Recently, there has been progress in generating
programs from language models
- Surprisingly, GPT3 could generate programs, even
though it was never explicitly trained on code
- CodeX is a specialized GPT model trained on code

37
CodeX evaluation
- While the performance of generative models are
usually evaluated using match-based metrics by
comparing output to a reference solution, there may
be too many different possible output programs that
are functionally equivalent to the reference solution
for this metric to account for; indeed, it was found
that one such metric called the BLEU score is
unreliable
- An alternative method is to use functional correctness
instead, which runs the output code on test cases to
measure performance; it is preferable similar to how
humans judge code

38
CodeX evaluation
- Can evaluate functional correctness using the pass@k metric,
which calculates the fraction of problems such that at least one of
k sample codes generated passed (all test cases) for that problem;
pass@k = 1-(1-pass@1)k
- Can be interpreted as the result of evaluating the best of k samples
- Directly calculating the pass@k results in high variance; instead,
n>=k sample codes are generated, the # of passed codes c is
counted and the estimator 1-(n-cCk / nCk) is calculated for each
problem and averaged
- That estimator is unbiased, unlike directly plugging in the empirical
estimate of pass@1 into pass@k, which underestimates pass@k
- A set of 164 problems called HumanEval was created (each
containing a function signature, docstring, body, and average of
7.7 unit tests), hand-written to avoid training on potential solutions

39
CodeX training
- Fine-tuned GPT models up to 12B parameters on code
- Trained on 159GB dataset of unique Python files on
GitHub
- Used the same learning rate as the corresponding
GPT model, with a 175 step linear warmup and cosine
learning rate decay. Trained on a total of 100 billion
tokens, using the Adam optimizer with a weight decay
coefficient of 0.1.

40
CodeX results
- It was found that the cross-entropy test loss on a
held-out validation set follows a power law of
(N/5.92x107)-0.13, where N=# non-embedding params
- When only one sample can be evaluated, it was found
that compared to randomly choosing a sample to
evaluate, choosing the one with the highest mean log
probability performs better, but with the highest sum
log probability performs slightly worse.

41
CodeX comparison
Model pass@1 pass@10 pass@100
- GPT models and the largest
GPT-NEO 125M 0.75% 1.88% 2.97%
free model from Tabnine
GPT-NEO 1.3B 4.79% 7.47% 16.30%
were evaluated on
GPT-NEO 2.7B 6.41% 11.27% 21.37% HumanEval (with
GPT-J 6B 11.62% 15.74% 27.74% temperatures of 0.2, 0.4, or
TABNINE 2.58% 4.35% 7.59% 0.8)
CODEX-12M 2.00% 3.62% 8.58% - GPT-Neo and GPT-J are
CODEX-25M 3.21% 7.1% 12.89%
models similar to CodeX,
CODEX-42M 5.06% 8.8% 15.55%
trained on The Pile dataset
CODEX-85M 8.22% 12.81% 22.4%
which has 8% GitHub code,
and are the only GPT models
CODEX-300M 13.17% 20.37% 36.27%
with pass rates not close to 0
CODEX-679M 16.22% 25.7% 40.95%

CODEX-2.5B 21.36% 35.42% 59.5%

CODEX-12B 28.81% 46.81% 72.31%

42
CodeX-S
- There may be code unrelated to translating natural
language to code, which may lower performance
- A set of training problems from relevant code was
obtained from competitive programming websites and
repositories with continuous integration
- CodeX-S is a version of CodeX with this supervised
fine-tuning and it has improved performance

43
CodeX-S results
- Prefers slightly higher sampling temperatures than
CodeX, possibly due to a narrower distribution
- Outperforms CodeX by 6.5% on pass@1 and 15.1% on
pass@100

44
CodeX-D
- So far we have discussed how CodeX generates code
from docstrings, but what about the other way around?
- CodeX-D is a version of CodeX that generates
docstrings from code
- Each training problem contains the function signature,
the reference solution, and docstring
- No way to measure functional correctness for
docstrings
- graded only 10 samples for each of 1640 problems by hand
manually
- pass@1 and pass@10 are 20.3% and 46.5% respectively
(which is slightly lower than that for CodeX-S: 32.2% and
59.5% respectively)

45
CodeX Limitations
- Not sample efficient to train, as the training data
totaled hundreds of millions of lines of code
- Performance decreases exponentially in docstring
length
- Can make mistakes binding variables to operations,
especially when there are a lot of them

46
Llama-2 introduction
-Family of pretrained and fine-tuned LLMs with billions of
parameters
-Can outperform other open-source LLMs and be on par
with close-sourced LLMs

47
Llama-2 pretraining
- Most of the pretraining, architecture, and
hyperparameters were adopted from Llama-1:
- Standard transformer architecture
- pre-normalization (normalized the input of each sub-layer
instead of the output to improve training stability, like in GPT3)
- SwiGLU activation function (allows more flexibility and
expressiveness in the feed-forward layers to improve
performance of transformer models compared to using
standard activations like ReLU)
- Instead of absolute positional embeddings, used rotary
positional embeddings which use a rotation matrix which
shows relative positions of tokens and allows the model to
capture their dependencies; can improve performance
because changing positions of words in a sentence can change
its meaning.
48
Llama-2 pretraining
- 7, 13, 34, and 70 billion parameters, with context
length (i.e. amount of text that can be processed at a
time) of 4k (doubled from Llama-1)
- Learning rate of 0.0003 for smaller models, 0.00015
for larger models with grouped query attention
- Trained using AdamW optimizer with a cosine learning
rate schedule, on 2T tokens of data (40% more than
Llama-1)

49
Llama-2 pretraining evaluation
- Llama-1 and 2 base models, along with other open-
sourced models MosaicML Pretrained Transformer
(MPT) and Falcon, were evaluated on several
benchmarks for comparison
- For code, the average pass@1 score on HumanEval and
MBPP is reported
- For commonsense reasoning, world knowledge, reading
comprehension, and math, the average of scores from 8, 2,
3, and 2 different methods is reported, respectively
- Massive multitask language understanding (MMLU), Big
Bench Hard (BBH), and Artificial General Intelligence
evaluation (AGIEval) on English tasks are also reported.

50
Llama-2 pre-training evaluation

Size Code Commonsense World Reading Math MMLU BBH AGI

Reasoning Knowledge Comprehensio Eval
n
MPT 7B 20.5 57.4 41.0 57.5 4.9 26.8 31.0 23.5
30B 28.9 64.9 50.0 64.7 9.1 46.9 38.0 33.8
Falcon 7B 5.6 56.1 42.8 36.0 4.6 26.2 28.0 21.2
40B 15.2 69.2 56.7 65.7 12.6 55.4 37.1 37.0
Llama-
1 7B 14.1 60.8 46.2 58.5 6.95 35.1 30.3 23.9
13B 18.9 66.1 52.6 62.3 10.9 46.9 37.0 33.9
Llama-
1 33B 26.0 70.0 58.4 67.6 21.4 57.8 39.8 41.7
65B 30.7 70.7 60.5 68.6 30.8 63.4 43.5 47.6
Llama-
2 7B 16.8 63.9 48.9 61.3 14.6 45.3 32.6 29.3
13B 24.5 66.9 55.4 65.8 28.7 54.8 39.4 39.1
Llama-
2 34B 27.8 69.9 58.7 68.0 24.2 62.6 44.1 43.4
70B 37.5 71.9 63.6 69.4 35.2 68.9 51.2 54.2
51
Llama 2-Chat
- Llama 2-Chat is a version of Llama-2 with supervised
fine-tuning through alignment techniques
- Supervised fine tuning with publicly available
instruction fine-tuning data
- it was found that using less (thousands instead of millions)
but higher quality examples improved results
- Reinforcement learning with human feedback (RLHF)
is applied after fine-tuning so the model can
understand the user intentions to further align the
model with human preferences

52
Llama 2-Chat Human Preference Data Collection

- Data of human preferences obtained from human

feedback
- For more diversity of collected prompts, binary
comparison protocol was used to collect feedback
- Annotators write prompts for the model, then for each
prompt they choose one of 2 responses from different
variants of the model (e.x. With different temperature)
based on provided criteria (helpfulness or safety), and rank
how much better their chosen response is on a scale of 4
points
- Preference was given to helpfulness and safety
- This data was received in batches over time (so the reward
model improved over time)

53
Llama 2-Chat Reward Modeling
- Human preference data was used to train reward model
(RM) so that patterns in the preferences can be learned,
by changing internal text distribution of the base model
- The RM outputs a score based on prediction of the
quality of the model (based on human preference) given
a prompt and model response
- These scores were used as rewards for RLHF
- Initialized from pretrained model to avoid situations
where the models would end up favouring hallucinations,
with the same architecture and hyperparameters but
with a regression head for outputting rewards
- 2 RMs: for helpfulness and for safety

54
Llama 2-Chat Reward Modeling
- Used a binary ranking loss with a margin component:
Lranking = −log(σ(rθ(x, yc) − rθ(x, yr) − m(r))), where rθ is
the reward model with model weights θ that takes
(prompt, response) as input, yc is the chosen response
and yr is the rejected response
- The margin component m(r) is a function of the preference
rating (different for each reward model), which helps the RM
give more distinct scores for more different responses
- The preference data was increased by combining with
open-sourced ones
- Trained with same parameters as base model, but only
ran 1 epoch (as it may overfit otherwise) and slightly
lower learning rate
55
Llama 2-Chat Reward Modeling
- These reward models outperform SteamSHP-XL, Open
Assistant, and GPT4 on several human preference
benchmarks
- The helpfulness RM and safety RM each performed best on
their own domain (e.x. Helpfulness performed best on
Meta Helpful data, etc) as they may sometimes have
conflicts
- Optimizing one RM with both objectives would’ve confused
the model and not perform well

56
Llama 2-Chat Iterative Fine-tuning
- Successive versions of RLHF were trained as more batches
of human preference data were received; labelled -V1 to -V5
- RLHF fine-tuning explored with 2 algorithms:
- Rejection sampling fine-tuning
- At each iteration, K samples generated from model, and best one
was selected using reward model; these best samples were used
for a gradient update to tune the model for the next iteration
- Applied to the 70B model, with smaller models fine-tuned on
rejection sampled data from 70B model
- Later versions include best samples from all previous models and
not just the preceding one since, for example, V3 was trained
using only samples from V2 but its performance worsened in
certain tasks
- Sampling temperature was adjusted for each iteration, as its optimal
value significantly changes

57
Llama 2-Chat Iterative Fine-tuning
• A standard method in reinforcement learningProximal
policy optimization (PPO)
• A standard method in reinforcement learning
• Uses the reward model as an estimate of the reward function, and the
(language) model as the policy to optimize
• Policy iteratively improved by sampling prompts from dataset and
generations from policy and apply PPO

58
Llama 2-Chat Ghost Attention (GAtt)
- GAtt is a new technique from this model that helps
control dialogue flow over multiple turns
- Initially the RLHF models sometimes forgets
instructions in a dialogue after a few turns; this can
be fixed with GAtt, and was applied after RLHF-V3
- The method works by synthetically concatenating the
instruction to user messages once it’s defined
- GAtt is consistent up to 20+ turns up to the context
length

59
Llama 2-Chat RLHF model-based evaluation
- For each prompt in a test set of helpfulness and
safety, 3 annotators judge the quality on a 7 point
scale
- The figure below shows the win rate % against GPT-4
for different versions of Llama 2-Chat during SFT and
RLHF

60
Llama 2-Chat RLHF human evaluation
- Llama 2-Chat was compared with MPT-7B-chat,
Vicuna-13B-v1.1 an 33B-v1.3, Falcon-40B-instruct,
PaLM-Bison, and ChatGPT-0301 on human evaluation
with over 4000 diverse prompts, with multi-turn
prompts generated by Llama 2-Chat and/or ChatGPT
- For each prompt, 3 human annotators rate how much
better/worse one model is than the other on a 7 point
scale

61
Llama-2 Safety
- Harmful data filtered out during pretraining and fine-
tuning
- Compared to before fine-tuning, Llama 2-chat showed
great improvement in truthfulness and toxicity
- 50.18% to 64.14% on TruthfulQA and 24.60 to 0.01% ToxiGen,
respectively, for the 70B model
- Also, lowered bias, as the BOLD scores increased overall
- To test robustness against attackers, red teaming was
performed
- Over 350 diverse people, including experts in various fields,
probed the models in various unsafe situations with simulated
prompts
- These insights were used for fine-tuning and feedback training
to lower the rate of violating responses; for the 7B model, this
rate was lowered by 4x
62
Llama-2 Safety
- Overall outperforms many other models in
helpfulness and safety (Falcon, Vicuna, PaLM,
ChatGPT) based on human raters judging
- While these safety tuning approaches fixed most
safety issues, it goes too far in some instances, which
can cause the model to be overly cautious and have
false refusals, although only happens 0.05% of the
time on the helpfulness data

63
Mixtral of Experts (MoE) introduction
- Mixtral 8x7B is a sparse mixtral of experts (SMoE)
model that can outperform Llama-2 70B and GPT3.5
on most benchmarks (math, code generation,
multilingual tasks)
- Uses a subset of parameters for every token, can
change the size to make inference faster or higher
throughput

64
Mistral architecture
- Mistral 7B is an earlier version of Mixtral
- Similar architecture to Llama, but also with:
- Sliding Window Attention (SWA) limits the amount of
tokens each token can attend to by window size W, to save
computation and memory
- Rolling Buffer Cache was then used to limit cache size to
W, and the keys and values at the i-th step are stored at
the (i mod W)-th position of the cache
- Pre-fill and chunking: can pre-fill cache with prompt, or
split and pre-fill with each chunk if prompt is large; then, to
compute the attention for each chunk, only that of the
current chunk and the previous (which is in the cache) is
needed

65
Sparse Mixtral of Experts
- Mixtral has the same architecture and its parameters as
Mistral, except the context length is 32k (quadrupled from
Mistral), and feed-forward blocks are replaced by 8 MoE
layers
- Experts are individual feed-forward networks
- Generally, MoE’s output is the sum of the dot products of
the expert E(x)i and its gating network G(x)i, over each
expert i
- Mixtral uses the softmax of top k logits of linear layer:
G(x)=Softmax(TopK(x⋅Wg)), where TopK is identity for top K logits
and -inf otherwise
- Then, the total (sparse) parameter count can grow with n while
the active parameter count (used for processing a token) only
grows with k
- For mixtral, E=SwiGLU and K=2
66
 Mixtral results
- Mixtral was compared to Llama by evaluating them on
several benchmarks similar to Llama-2’s paper

Model Active MMLU Hellaswa Winogand PIQ ARC- ARC- Natural TriviaQ Human MBPP MATH GSM8K
params g er A Easy Challen Questio A Eval
ge ns

Llama-
2 7B 44.4% 77.1% 69.5% 77.9% 68.7% 43.2% 17.5% 56.6% 11.6% 26.1% 3.9% 16.0%

Llama-
2 13B 55.6% 80.7% 72.9% 80.8% 75.2% 48.8% 16.7% 64.0% 18.9% 35.4% 6.0% 34.3%

Llama-
1 33B 56.8% 83.7% 76.2% 82.2% 79.6% 54.4% 24.1% 68.5% 25.0% 40.9% 8.4% 44.1%

Llama-
2 70B 69.9% 85.4% 80.4% 82.6% 79.9% 56.5% 25.4% 73.0% 29.3% 49.8% 13.8% 69.6%

Mistral 7B 62.5% 81.0% 74.2% 82.2% 80.5% 54.9% 23.2% 62.5% 26.2% 50.2% 12.7% 50.0%

Mixtral 13B 70.6% 84.4% 77.2% 83.6% 83.1% 59.7% 30.6% 71.5% 40.2% 60.7% 28.4% 67 74.4%
Mixtral results
- Compared to Mistral, multilingual data was
significantly upsampled, allowing it to perform well on
multilingual benchmarks
- For each language, ARC-Challenge, Hellaswag, and
MMLU are reported

Model French German Spanish Italian

Llama-1 33B 39.3% 68.1% 49.9% 41.1% 63.3% 48.7% 45.7% 69.8% 52.3% 42.9% 65.4% 49.0%

Llama-2 70B 49.9% 72.5% 64.3% 47.3% 68.7% 64.2% 50.5% 74.5% 66.0% 49.4% 70.9% 65.1%

Mixtral 8x7B 58.2% 77.4% 70.9% 54.3% 73.0% 71.5% 55.4% 77.6% 72.5% 52.8% 75.1% 70.9%

68
Mixtral results
- Long range performance
- 100% retrieval accuracy of the passkey retrieval task
which measures the ability of models to retrieve a passkey
randomly inserted in a long prompt
- Perplexity decreases as context length increases
- Bias benchmarks
- Mixtral outperforms Llama-2 70B on Bias Benchmark for
QA (BBQ) (56.0% vs 51.5%)
- On Bias in Open-Ended Language Generation Dataset
(BOLD), Mixtral has higher scores overall compared to
Llama-2, which represents less social bias

69
Mixtral-Instruct
- Mixtral-Instruct is a version of Mixtral with supervised
fine-tuning on an instruction dataset followed by
Direct Performance Optimization (DPO)
- In the Large Model Systems (LMSYS) Chatbot Arena as
of December 2023, Mixtral-Instruct outperforms other
open-weight models on MT-Bench (a set of
challenging multi-turn questions) and is ranked 6th in
the arena elo (based on 200,000 human preference
votes)

70
Mixtral routing analysis
- The distribution of selected experts using The Pile
datasets was measured and reported for first, middle,
and last layers
- Only DM Mathematics had significantly different
distributions, possibly due to limited coverage of
natural language
- Its distribution at the first and last layer is very similar
to the input and output embeddings, respectively
- This suggests the router has structured syntactic
behavior

71
PaLM: Pathways Language Model⁵
- Trained as a 540-billion parameter, densely activated, Transformer Language model
- Trained on a Google 6144 Tensor Processing Units (TPU) v4 chips

- Efficient Scaling - Scaled very well across thousands or tens of thousands of accelerator chips in
a highly efficient manner

- Continued improvements from scaling - Continuous growth in improvement on device as its

TPU are scaled up

- Breakthrough capabilities - It can perform reasoning tasks via multi-step mathematics

- Discontinuous improvements - Scaling behaviour where improvements in performance not

linear e.g. scaling from 62b to 540b parameters result in dramatic jump in accuracy compared to
scaling from 8B to 62B. New capabilities emerge when the model achieves sufficient scale.

- Multilingual Understanding - More thorough work than earlier models including translation

- Bias and toxicity - Accuracy improved on gender and occupation bias. The model still has an
issue with race/religion/gender bias being very much affected by the style of prompt provided. It
still
5. Chowdhery, A.,associated Muslims
Narang, S., Devlin, with
J., Bosma, M., terrorism, extremism
Mishra, G., Roberts, and N.
A., ... & Fiedel, violence. This
(2023). Palm: is work
Scaling in progress.
language modeling with pathways. Journal
of Machine Learning Research, 24(240), 1-113.
72
PaLM: Model Architecture
- Uses a standard Transformer Model architecture in a
decoder-only setup. Each timestep can only attend to
itself and past timesteps
- SwiGLU Activation -
- Parallel Layers - each transformer block is in parallel.
The standard formulation is written as:
y = x + MLP(LayerNorm(x +
Attention(LayerNorm(x)))
whereas the parallel formulation can be written as:
y = x + MLP(LayerNorm(x)) +
Attention(LayerNorm(x))

73
PaLM: Model Scale Hyperparameters
- Compared three different model scales: 540B
parameters, 62B parameter, and 8B parameters
- Number of FLOPs per token is approx equal to number
of parameters

Model Layers # of Heads dmodel # of Parameters BatchSize

(in billions)

PaLM 8B 32 16 4096 8.63 256 -> 512

PaLM 62B 64 32 8192 62.50 512 -> 1024
PaLM 540B 118 48 18432 540.35 512 -> 1024 -> 2048

74
PaLM: Training Dataset
- Consists of high-quality corpus of 780 billion tokens
representing a wide range of natural language use cases
- Based on datasets used to train LaMDA
Total dataset size = 780 billion tokens
Data source Proportion of data
Social media conversations (multilingual) 50%
Filtered webpages (multilingual) 27%
Books (English) 13%
GitHub (code) 5%
Wikipedia (multilingual) 4%
News (English) 1%

75
PaLM: Training Infrastructure
- All models trained on TPU v4 Pods

76
PaLM: Results
● PaLM 540B across 29 NLP benchmarks

77
PaLM: BIG-bench
- BIG-bench is a collaborative benchmark aimed at producing
challenging tasks for LLM.

78
PaLM: Evaluating Reasoning
- Arithmetic reasoning - often grade-school level natural
language math problems which require multi-step logical
inference. The math itself is trivial. The difficult part is
transforming the natural language into mathematical equations.
- Input: Q: Roger has 5 tennis balls. He buys 2 more cans of tennis balls. Each can has 3 tennis balls. How many
- tennis balls does he have now?
- Answer: The answer is 11.
- Commonsense reasoning - Question answering tasks which
require strong world knowledge but are not simply factual
question answering. They require chaining multiple logical
inferences about the world
- Input: Q: Sean was in a rush to get home, but the light turned yellow and he was
- forced to do what? Answer Choices: (a) take time (b) dawdle (c) go slowly (d) ocean (e) slow down
- Answer: The answer is (e) slow down.

79
PaLM: Chain-of-thought prompting

80
PaLM: Chain-of-thought Results

81
PaLM: Code Tasks
- Text-to-code - task is to write code given a natural language
description.
- Code-to-code - task is to translate C or C++ programs to Python
- Major risks:
- Generated code may be wrong
- Presence of subtle bugs

82
PaLM: Translation
- Rewrite one human language into another one while preserving the content,
semantics and style of the input
- English-centric language pairs
- Traditional focus of past models
- English as source or target language e.g. English -> French, English->German,
etc.
- Direct language pairs
- Directly translate between any pair of languages without involving English.
For example French->German instead of French->English->German
- Extremely-low resource language pairs
- In some cases, one of the languages have little monolingual data such as
Kazakh. E.g. French and German have about 24 and 26 billion tokens in
training set while Kazakh has around 134 million tokens.

83
PaLM: Translation

0-Shot 1-Shot Few-shot Supervised

Source Target Prior PaLM Prior PaLM Prior PaLM Finetuned

SOTA 540B SOTA 540B SOTA 540B SOTA
en fr 2P 38.5 28.3 37.5 33.9 44.0 45.6
en de P 31.8 26.2 31.8 26.8 37.4 41.2
en ro P 24.2 20.6 28.2 20.5 28.7 33.4
fr en P 41.1 33.7 37.2 38.0 42.8 45.4
de en P 43.8 30.4 43.9 40.6 47.5 41.2
ro en 2P 39.9 38.6 42.1 37.3 43.8 39.1

84
PaLM: Limitations

- Contain and amplify biases in underlying data

- Gender and occupation bias
- Toxicity and bias

85
LLMs Comparison
Model Name nparams nlayers dmodel nheads dhead Learning Context Architecture
Rate length
T5 125M 12 768 12 64 1.0 x 10-3
Transformer – Encoder-
Decoder
GPT-3 175B or “GPT-3” 175.0B 96 12288 96 128 0.6 x 10-4
Transformer – Decoder

CodeX 12B 12B N/A 4k Transformer - Decoder

0.6 x 10 -4

Llama-2 70B 70B N/A 1 4k Transformer with Pre-

1.5 x 10 -4
Normalization, SwiGLU acti
vation, RoPE, GQA
Mixtral 8x7B 13B 32 14436 32 128 32k Transformer with Pre-
active, Normalization, SwiGLU acti
47B vation, RoPE, SWA, and
sparse Mixture of Experts instead
of feed-forward layer

PaLM 540B 118 18432 48 5 x10-5 Transformer – Encoder-

Decoder

86
Questions/discussions
1. Which is preferable, unsupervised learning or
reinforcement learning with human feedback?
2. What makes a large language model large?
3. Can language models be used maliciously?
4. How can we reduce societal harm due to misuse of
language models?
5. How do we remove societal implicit biases from
becoming part of foundation models?
6. What lead to emergent abilities observed in LLMs?

87
References
1. Chang, Y., Wang, X., Wang, J., Wu, Y., Zhu, K., Chen, H., Yang, L., Yi, X., Wang, C., Wang, Y., Ye, W., Zhang, Y., Chang,
Y., Yu, P.S., Yang, Q., & Xie, X. (2023). A Survey on Evaluation of Large Language Models. ArXiv, abs/2307.03109
2. Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., ... & Liu, P. J. (2020). Exploring the limits of
transfer learning with a unified text-to-text transformer. The Journal of Machine Learning Research, 21(1), 5485-5551.
3. A. Vaswani et al., “Attention is All you Need,” in Advances in Neural Information Processing Systems (NeurIPS), 2017.
4. Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., ... & Amodei, D. (2020). Language models are
few-shot learners. Advances in neural information processing systems, 33, 1877-1901.
5. Chowdhery, A., Narang, S., Devlin, J., Bosma, M., Mishra, G., Roberts, A., ... & Fiedel, N. (2023). Palm: Scaling language
modeling with pathways. Journal of Machine Learning Research, 24(240), 1-113.
6. Chen, Mark, et al. Evaluating Large Language Models Trained on Code. arXiv:2107.03374, arXiv, 14 July 2021.
[Link], [Link]
7. Touvron, Hugo, et al. Llama 2: Open Foundation and Fine-Tuned Chat Models. arXiv:2307.09288, arXiv, 19 July 2023.
[Link], [Link]
8. Jiang, Albert Q., et al. Mixtral of Experts. arXiv:2401.04088, arXiv, 8 Jan. 2024. [Link],
[Link]

88