Reinforcement Learning from Human Feedback

A short introduction to RLHF and post-training focused on language models.

Nathan Lambert

23 December 2025

Abstract
Reinforcement learning from human feedback (RLHF) has become an important
technical and storytelling tool to deploy the latest machine learning systems. In this
book, we hope to give a gentle introduction to the core methods for people with some
level of quantitative background. The book starts with the origins of RLHF – both
in recent literature and in a convergence of disparate fields of science in economics,
philosophy, and optimal control. We then set the stage with definitions, problem
formulation, data collection, and other common math used in the literature. The core of
the book details every optimization stage in using RLHF, from starting with instruction
tuning to training a reward model and finally all of rejection sampling, reinforcement
learning, and direct alignment algorithms. The book concludes with advanced topics –
understudied research questions in synthetic data and evaluation – and open questions
for the field.

1
Contents
1 Introduction 6
1.1 What Does RLHF Do? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7
1.2 An Intuition for Post-Training . . . . . . . . . . . . . . . . . . . . . . . . . . . 10
1.3 How We Got Here . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11
1.4 Scope of This Book . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12
1.4.1 Chapter Summaries . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13
1.4.2 Target Audience . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14
1.4.3 How to Use This Book . . . . . . . . . . . . . . . . . . . . . . . . . . . 14
1.4.4 About the Author . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14
1.5 Future of RLHF . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14

2 Key Related Works 16

2.1 Origins to 2018: RL on Preferences . . . . . . . . . . . . . . . . . . . . . . . . 16
2.2 2019 to 2022: RL from Human Preferences on Language Models . . . . . . . 16
2.3 2023 to Present: ChatGPT Era . . . . . . . . . . . . . . . . . . . . . . . . . . 17

3 Definitions & Background 18

3.1 Language Modeling Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . 18
3.2 ML Definitions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18
3.3 NLP Definitions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19
3.4 RL Definitions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19
3.5 RLHF Only Definitions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20
3.6 Extended Glossary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21

4 Training Overview 22
4.1 Problem Formulation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22
4.1.1 Example RL Task: CartPole . . . . . . . . . . . . . . . . . . . . . . . 22
4.1.2 Manipulating the Standard RL Setup . . . . . . . . . . . . . . . . . . 23
4.1.3 Finetuning and Regularization . . . . . . . . . . . . . . . . . . . . . . 24
4.1.4 Optimization Tools . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26
4.2 Canonical Training Recipes . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26
4.2.1 InstructGPT . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26
4.2.2 Tülu 3 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27
4.2.3 DeepSeek R1 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29

5 The Nature of Preferences 30

5.1 The Origins of RLHF and Preferences . . . . . . . . . . . . . . . . . . . . . . 31
5.1.1 Specifying objectives: from logic of utility to reward functions . . . . . 32
5.1.2 Implementing optimal utility . . . . . . . . . . . . . . . . . . . . . . . 33
5.1.3 Steering preferences . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34

6 Preference Data 36
6.1 Why We Need Preference Data . . . . . . . . . . . . . . . . . . . . . . . . . . 36
6.2 Collecting Preference Data . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36
6.2.1 Interface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37
6.2.2 Rankings vs. Ratings . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39
6.2.3 Multi-turn Data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43

2
 6.2.4 Structured Preference Data . . . . . . . . . . . . . . . . . . . . . . . . 44
6.2.5 Sourcing and Contracts . . . . . . . . . . . . . . . . . . . . . . . . . . 45
6.3 Bias: Things to Watch Out For in Data Collection . . . . . . . . . . . . . . . 47
6.4 Open Questions in RLHF Preference Data . . . . . . . . . . . . . . . . . . . . 47

7 Reward Modeling 49
7.1 Training Reward Models . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 49
7.2 Architecture . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51
7.3 Implementation Example . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51
7.4 Variants . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 52
7.4.1 Preference Margin Loss . . . . . . . . . . . . . . . . . . . . . . . . . . 52
7.4.2 Balancing Multiple Comparisons Per Prompt . . . . . . . . . . . . . . 53
7.4.3 K-wise Loss Function . . . . . . . . . . . . . . . . . . . . . . . . . . . 53
7.5 Outcome Reward Models . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 53
7.6 Process Reward Models . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 55
7.7 Reward Models vs. Outcome RMs vs. Process RMs vs. Value Functions . . . 57
7.8 Generative Reward Modeling . . . . . . . . . . . . . . . . . . . . . . . . . . . 58
7.9 Further Reading . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 59

8 Regularization 61
8.1 KL Divergences in RL Optimization . . . . . . . . . . . . . . . . . . . . . . . 61
8.1.1 Reference Model to Generations . . . . . . . . . . . . . . . . . . . . . 62
8.1.2 Implementation Example . . . . . . . . . . . . . . . . . . . . . . . . . 62
8.2 Pretraining Gradients . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 63
8.3 Other Regularization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 64

9 Instruction Finetuning 65
9.1 Chat templates and the structure of instructions . . . . . . . . . . . . . . . . 65
9.2 Best practices of instruction tuning . . . . . . . . . . . . . . . . . . . . . . . . 68
9.3 Implementation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 68

10 Rejection Sampling 70
10.1 Training Process . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 70
10.1.1 1. Generating Completions . . . . . . . . . . . . . . . . . . . . . . . . 71
10.1.2 2. Scoring Completions . . . . . . . . . . . . . . . . . . . . . . . . . . 71
10.1.3 3. Fine-tuning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 74
10.2 Implementation Details . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 74
10.3 Related: Best-of-N Sampling . . . . . . . . . . . . . . . . . . . . . . . . . . . 75

11 Reinforcement Learning (i.e. Policy Gradient Algorithms) 76

11.1 Policy Gradient Algorithms . . . . . . . . . . . . . . . . . . . . . . . . . . . . 76
11.1.1 Vanilla Policy Gradient . . . . . . . . . . . . . . . . . . . . . . . . . . 79
11.1.2 REINFORCE . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 80
11.1.3 Proximal Policy Optimization . . . . . . . . . . . . . . . . . . . . . . . 82
11.1.4 Group Relative Policy Optimization . . . . . . . . . . . . . . . . . . . 87
11.2 Implementation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 90
11.2.1 Policy Gradient Basics . . . . . . . . . . . . . . . . . . . . . . . . . . . 91
11.2.2 Loss Aggregation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 91
11.2.3 Asynchronicity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 95

3
 11.2.4 Proximal Policy Optimization . . . . . . . . . . . . . . . . . . . . . . . 97
11.2.5 Group Relative Policy Optimization . . . . . . . . . . . . . . . . . . . 99
11.3 Auxiliary Topics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 101
11.3.1 Comparing Algorithms . . . . . . . . . . . . . . . . . . . . . . . . . . . 101
11.3.2 Generalized Advantage Estimation (GAE) . . . . . . . . . . . . . . . . 101
11.3.3 Double Regularization . . . . . . . . . . . . . . . . . . . . . . . . . . . 103
11.3.4 Further Reading . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 103

12 Direct Alignment Algorithms 105

12.1 Direct Preference Optimization (DPO) . . . . . . . . . . . . . . . . . . . . . . 105
12.1.1 How DPO Works . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 105
12.1.2 DPO Derivation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 107
12.2 Numerical Concerns, Weaknesses, and Alternatives . . . . . . . . . . . . . . . 111
12.3 Implementation Considerations . . . . . . . . . . . . . . . . . . . . . . . . . . 113
12.4 DAAs vs. RL: Online vs. Offline Data . . . . . . . . . . . . . . . . . . . . . . 113

13 Constitutional AI & AI Feedback 115

14 Balancing AI and Human Feedback Data 116

14.1 Constitutional AI . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 117
14.2 Specific LLMs for Judgement . . . . . . . . . . . . . . . . . . . . . . . . . . . 118
14.3 Further Reading . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 118

15 Reasoning Training & Inference-Time Scaling 120

15.1 The Origins of New Reasoning Models . . . . . . . . . . . . . . . . . . . . . . 124
15.1.1 Why Does RL Work Now? . . . . . . . . . . . . . . . . . . . . . . . . . 124
15.1.2 RL Training vs. Inference-time Scaling . . . . . . . . . . . . . . . . . . 125
15.1.3 The Future (Beyond Reasoning) of RLVR . . . . . . . . . . . . . . . . 125
15.2 Understanding Reasoning Training Methods . . . . . . . . . . . . . . . . . . . 126
15.2.1 Reasoning Research Pre OpenAI’s o1 or DeepSeek R1 . . . . . . . . . 126
15.2.2 Early Reasoning Models . . . . . . . . . . . . . . . . . . . . . . . . . . 126
15.2.3 Common Practices in Training Reasoning Models . . . . . . . . . . . . 129
15.3 Looking Ahead . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 130

16 Tool Use & Function Calling 132

16.1 Interweaving Tool Calls in Generation . . . . . . . . . . . . . . . . . . . . . . 133
16.2 Multi-step Tool Reasoning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 136
16.3 Model Context Protocol (MCP) . . . . . . . . . . . . . . . . . . . . . . . . . . 136
16.4 Implementation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 138

17 Synthetic Data & Distillation 140

18 Evaluation 142
18.1 Prompting Formatting: From Few-shot to Zero-shot to CoT . . . . . . . . . . 142
18.2 Using Evaluations vs. Observing Evaluations . . . . . . . . . . . . . . . . . . 146
18.3 Contamination . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 148
18.4 Tooling . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 148

19 Over Optimization 149

4
 19.1 Qualitative Over-optimization . . . . . . . . . . . . . . . . . . . . . . . . . . . 149
19.1.1 Managing Proxy Objectives . . . . . . . . . . . . . . . . . . . . . . . . 149
19.1.2 Over-refusal and “Too Much RLHF” . . . . . . . . . . . . . . . . . . . 151
19.2 Quantitative over-optimization . . . . . . . . . . . . . . . . . . . . . . . . . . 153
19.3 Misalignment and the Role of RLHF . . . . . . . . . . . . . . . . . . . . . . . 154

20 Style and Information 155

20.1 The Chattiness Paradox . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 155
20.1.1 How Chattiness Emerges . . . . . . . . . . . . . . . . . . . . . . . . . 156

21 Product, UX, and Model Character 158

21.1 Character Training . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 158
21.2 Model Specifications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 159
21.3 Product Cycles, UX, and RLHF . . . . . . . . . . . . . . . . . . . . . . . . . 159

Bibliography 161

5
1 Introduction
Reinforcement learning from Human Feedback (RLHF) is a technique used to incorporate
human information into AI systems. RLHF emerged primarily as a method to solve hard
to specify problems. With systems that are designed to be used by humans directly, such
problems emerge all the time due to the often unexpressible nature of an individual’s
preferences. This encompasses every domain of content and interaction with a digital system.
RLHF’s early applications were often in control problems and other traditional domains for
reinforcement learning (RL), where the goal is to optimize a specific behavior to solve a task.
The core idea to start the field of RLHF was “can we solve hard problems only with basic
preference signals guiding the optimization process.” RLHF became most known through the
release of ChatGPT and the subsequent rapid development of large language models (LLMs)
and other foundation models.
The basic pipeline for RLHF involves three steps. First, a language model that can follow
user questions must be trained (see Chapter 9). Second, human preference data must be
collected for the training of a reward model of human preferences (see Chapter 7). Finally,
the language model can be optimized with an RL optimizer of choice, by sampling generations
and rating them with respect to the reward model (see Chapter 3 and 11). This book details
key decisions and basic implementation examples for each step in this process.
RLHF has been applied to many domains successfully, with complexity increasing as the
techniques have matured. Early breakthrough experiments with RLHF were applied to
deep reinforcement learning [1], summarization [2], following instructions [3], parsing web
information for question answering [4], and “alignment” [5]. A summary of the early RLHF
recipes is shown below in fig. 1.

Figure 1: A rendition of the early, three stage RLHF process with SFT, a reward model,
and then optimization.

In modern language model training, RLHF is one component of post-training. Post-training

is a more complete set of techniques and best-practices to make language models more useful
for downstream tasks [6]. Post-training can be summarized as a many-stage training process
using three optimization methods:
1. Instruction / Supervised Finetuning (IFT/SFT), where we teach formatting and form
the base of instruction following abilities. This is largely about learning features in
language.
2. Preference Finetuning (PreFT), where we align to human preferences (and get smaller

6
 bump in capabilities at the same time). This is largely about style of language and
subtle human preferences that are hard to quantify.
3. Reinforcement Learning with Verifiable Rewards (RLVR). The newest type of post-
training that boosts performance on verifiable domains with more RL training.
RLHF lives within and dominates the second area, preference finetuning, which has more
complexity than instruction tuning due to it often involving proxy reward models of the true
object and noisier data. At the same time, RLHF is far more established than the other
popular RL method for language models, reinforcement learning with verifiable rewards. For
that reason, this book focuses on preference learning, but in order to completely grasp the
role of RLHF, one needs to use these other training stages, so they are also explained in
detail.
As we consider the space of options and attention on these methods for crafting models we
collectively use extensively, RLHF colloquially is what led to modern post-training. RLHF
was the technique that enabled the massive success of the release of ChatGPT, so early in
2023 RLHF encompassed much of the interest in the general field of post-training. RLHF is
now just one piece of post-training, so in this book we map through why there was so much
attention on RLHF early on, and how other methods emerged to complement it.
Training language models is a very complex process, often involving large technical teams of
10s to 100s of people and millions of dollars in data and compute cost. This book serves three
purposes to enable readers to grasp how RLHF and related models are used to craft leading
models. First, the book distills cutting edge research often hidden within large technology
companies into clear topics and trade-offs, so readers can understand how models are made.
Second, the book will allow users to setup basic code examples to get their hands dirty on
finetuning these models themselves. Finally, beyond teaching the techniques for doing RLHF,
this book is designed to distill intuition as to why RLHF is crucial to modern AI models.
Due to the complexity of RLHF and how the state-of-the-art is often too complex to be done
alone, this book focuses on enabling readers so they have the tools needed to get jobs and
start research projects in the area. Others will just enjoy precisely understanding the inner
workings of the technology that is the focus of countless discussions across the globe. A book
that lays out an exact recipe of how to do RLHF for a specific need is impossible, which is
why there is a large industry of companies providing RLHF training and related methods as
a service worth millions of dollars. Still, rapid progress in AI systems means the readers who
are fitted with this knowledge can approach more and more of their own training over time.

1.1 What Does RLHF Do?

The biggest question around RLHF, yet one that is still hard to answer, is “What does
RLHF training offer models?” In recent years, language models have shifted from academic
experiments studied in the purview of benchmarks to general-purpose technology. RLHF is
at the core of this transition.
To start, let us consider how a language model at the end of autoregressive pretraining
answers a query. A strong model, Llama 3.1 405B Base, will continue from “The president
of the united states in 2006 was”:
George W. Bush, the governor of Florida in 2006 was Jeb Bush, and John McCain
was an Arizona senator in 2006 - who later lost to obama. September 1 – U.S.

7
 President Bush signs an executive order to provide more options for faith-based
organizations when delivering social services that are paid for with federal funds.
January 1 – The current deadline set by United States Congress in the Unlawful
Internet Gambling Enforcement Act, aka UIGEA. search: Amendments to the
1961 International Convention for the Protection of New Varieties of Plants
require plant breeders’ rights include farmer’s privilege. 2009 was a common year
starting on Thursday of the Gregorian calendar, the 2009th year of the Common
Era (ce) and Anno Domini (ad) designations, the 9th year of the 3rd millennium,
the 9th year of the 21st century, and the 10th and last year of the 2000s decade.
What is clear is that this model is completing the sentence and adding other common internet
metadata.
The most compelling view of how RLHF works is to think of how style applies to interactions
you have with language models. The style, closely related to tone or format, of information
presented is crucial to how it is learned. Style is often used to describe the manner of the
model. To see the difference, consider two example responses to the following question:
Help, I have to give a presentation tomorrow and I’m panicking.
A blunt, low-empathy style: > You’ll be fine. Practice your slides a few times, get some
sleep, and stop overthinking it. Just present what you know and move on.
A warmer, more supportive style: > That sounds really stressful, and it’s a normal reaction.
If you can, do one quick practice run now, pick the one or two points you most want people
to remember, and then give yourself permission to rest—being clear and calm tomorrow will
matter more than cramming tonight.
On the other hand, format is how the answer is structured. Modern models are known to
use extensive markdown formatting with lists, linebreaks, and emojis. Consider two answers
to the following question that are very different:
How can I improve my sleep quality?
The simple answer, from an early language model:
Try to go to bed at the same time each night, avoid caffeine late in the day, and
keep your bedroom quiet and dark.
Or a complex, richer answer:
Better sleep, in practice
Goal: make your schedule predictable and your environment boring enough that
your brain gives up.
Tonight (quick wins): 1. Dim lights for the last hour before bed. 2. No
caffeine after lunch; keep alcohol minimal. 3. If you’re stuck awake, get up and
do something quiet until sleepy again.
Daily basics (most impact): • Same wake time (even weekends) → anchors
your body clock. • Morning light for 5–10 minutes → helps set circadian rhythm.
• Cool, dark room (fan/eye mask) → easier to stay asleep.

8
 If you want a simple rule: wake time fixed + caffeine cutoff + wind-down
routine

If sleep problems are persistent or severe, it can be worth talking with a clini-
cian—many issues are very treatable.
Continuing the above example, a model trained with RLHF (and other post-training tools),
would answer the question very differently. Asking Tülu 3 405B the same question “The
president of the united states in 2006 was,” it says concisely:
George W. Bush was the president of the United States in 2006. He served two
terms in office, from January 20, 2001, to January 20, 2009.
This change in style is downstream of the post-training methods described in this book.
Instruction finetuning would provide the basic ability for models to respond reliably in the
question-answer format, and RLHF is what takes these answers and crafts them into the
reliable, warm, and engaging answers we now expect from language models.
Modern research has established RLHF as a general method to integrate subtle stylistic and
related behavioral features into the models. Compared to other techniques for post-training,
such as instruction finetuning, RLHF generalizes far better across domains [7] [8] – helping
create effective general-purpose models.
Intuitively, this can be seen in how the optimization techniques are applied. Instruction
finetuning is training the model to predict the next certain token when the text preceding is
close to examples it has seen. It is optimizing the model to more regularly output specific
features in text. This is a per-token update.
RLHF on the other hand tunes the responses on the response level rather than looking at
the next token specifically. Additionally, it is telling the model what a better response looks
like, rather than a specific response it should learn. RLHF also shows a model which type of
response it should avoid, i.e. negative feedback. The training to achieve this is often called a
contrastive loss function and is referenced throughout this book.
While this flexibility is a major advantage of RLHF, it comes with implementation challenges.
Largely, these center on how to control the optimization. As we will cover in this book,
implementing RLHF often requires training a reward model, of which best practices are not
strongly established and depend on the area of application. With this, the optimization
itself is prone to over-optimization because our reward signal is at best a proxy objective,
requiring regularization. With these limitations, effective RLHF requires a strong starting
point, so RLHF cannot be a solution to every problem alone and needs to be approached in
a broader lens of post-training.
Due to this complexity, implementing RLHF is far more costly than simple instruction
finetuning and can come with unexpected challenges such as length bias [9] [10]. For model
training efforts where absolute performance matters, RLHF is established as being crucial to
achieving a strong finetuned model, but it is more expensive in compute, data costs, and
time. Through the early history of RLHF after ChatGPT, there were many research papers
that showed approximate solutions to RLHF via limited instruction finetuning, but as the
literature matured it has been repeated time and again that RLHF and related methods are
core stages to model performance that cannot be dispensed with quickly.

9
1.2 An Intuition for Post-Training
We’ve established that RLHF specifically and post-training generally is crucial to performance
of the latest models and how it changes the models’ outputs, but not why it works. Here’s a
simple analogy for how so many gains can be made on benchmarks on top of any base model.
The way I’ve been describing the potential of post-training is called the elicitation interpre-
tation of post-training, where all we are doing is extracting potential by amplifying valuable
behaviors in the base model.
To make this example click, we make the analogy between the base model – the language
model that comes out of the large-scale, next-token prediction pretraining – and other
foundational components in building complex systems. We use the example of the chassis
of a car, which defines the space where a car can be built around it. Consider Formula 1
(F1), most of the teams show up to the beginning of the year with a new chassis and engine.
Then, they spend all year on aerodynamics and systems changes (of course, it is a minor
oversimplification), and can dramatically improve the performance of the car. The best F1
teams improve way more during a season than chassis-to-chassis.
The same is true for post-training, where one can extract a ton of performance out of a
static base model as they learn more about its quirks and tendencies. The best post-training
teams extract a ton of performance in a very short time frame. The set of techniques is
everything after the end of most of pretraining. It includes “mid-training” like annealing
/ high-quality end of pre-training web data, instruction tuning, RLVR, preference-tuning,
etc. A good example is the change from the first version of the Allen Institute for AI’s
fully-open, small Mixture-of-Experts (MoE) model OLMoE Instruct to the second. The first
model was released in the fall of 2024 [11], and with the second version only updating the
the post-training, the evaluation average on popular benchmarks went from from 35 to 48
without changing the majority of pretraining [12].
The idea is that there is a lot of intelligence and ability within base models, but because they
can only answer in next-token prediction and not question-answering format, it takes a lot of
work building around them, through post-training, in order to make excellent final models.
Then, when you look at models such as OpenAI’s GPT-4.5 released in February 2025, which
was largely a failure of a consumer product due to being too large of a base model to serve
to millions of users, you can see this as a way more dynamic and exciting base for OpenAI
to build onto. With this intuition, base models determine the vast majority of the potential
of a final model, and post-training’s job is to cultivate all of it.
I’ve described this intuition as the Elicitation Theory of Post-training. This theory folds in
with the reality that the majority of gains users are seeing are from post-training because it
implies that there is more latent potential in a model pretraining on the internet than we can
teach the model simply — such as by passing certain narrow samples in repeatedly during
early types of post-training (i.e. only instruction tuning). The challenge of post-training
is to reshape models from next-token prediction to conversation question-answering, while
extracting all of this knowledge and intelligence from pretraining.
A related idea to this theory is the Superficial Alignment Hypothesis, coined in the paper
LIMA: Less is More for Alignment [13]. This paper is getting some important intuitions
right but for the wrong reasons in the big picture. The authors state:

10
 A model’s knowledge and capabilities are learnt almost entirely during pretraining,
while alignment teaches it which subdistribution of formats should be used when
interacting with users. If this hypothesis is correct, and alignment is largely
about learning style, then a corollary of the Superficial Alignment Hypothesis is
that one could sufficiently tune a pretrained language model with a rather small
set of examples [Kirstain et al., 2021].
All of the successes of deep learning should have taught you a deeply held belief that
scaling data is important to performance. Here, the major difference is that the authors
are discussing alignment and style, the focus of academic post-training at the time. With a
few thousand samples for instruction finetuning, you can change a model substantially and
improve a narrow set of evaluations, such as AlpacaEval, MT Bench, ChatBotArena, and
the likes. These do not always translate to more challenging capabilities, which is why Meta
wouldn’t train its Llama Chat models on just this dataset. Academic results have lessons,
but need to be interpreted carefully if you are trying to understand the big picture of the
technological arc.
What this paper is showing is that you can change models substantially with a few samples.
We knew this, and it is important to the short-term adaptation of new models, but their
argument for performance leaves the casual readers with the wrong lessons.
If we change the data, the impact could be far higher on the model’s performance and
behavior, but it is far from “superficial.” Base language models today (with no post-training)
can be trained on some mathematics problems with reinforcement learning, learn to output a
full chain-of-thought reasoning, and then score higher on a full suite of reasoning evaluations
like BigBenchHard, Zebra Logic, AIME, etc.
The superficial alignment hypothesis is wrong for the same reason that people who think
RLHF and post-training are just for vibes are still wrong. This was a field-wide lesson we
had to overcome in 2023 (one many AI observers are still rooted in). Post-training has far
outgrown that, and we are coming to see that the style of models operates on top of behavior
— such as the now popular long chain of thought.

1.3 How We Got Here

Why does this book make sense now? How much still will change?
Post-training, the craft of eliciting powerful behaviors from a raw pretrained language model,
has gone through many seasons and moods since the release of ChatGPT that sparked the
renewed interest in RLHF. In the era of Alpaca [14], Vicuna [15], Koala [16], and Dolly
[17], a limited number of human datapoints with extended synthetic data in the style of
Self-Instruct were used to normally fine-tune the original LLaMA to get similar behavior to
ChatGPT. The benchmark for these early models was fully vibes (and human evaluation)
as we were all so captivated by the fact that these small models can have such impressive
behaviors across domains. It was justified excitement.
Open post-training was moving faster, releasing more models, and making more noise than
its closed counterparts. Companies were scrambling, e.g. DeepMind merging with Google or
being started, and taking time to follow it up. There are phases of open recipes surging and
then lagging behind.

11
The era following Alpaca et al., the first lag in open recipes, was one defined by skepticism
and doubt on reinforcement learning from human feedback (RLHF), the technique OpenAI
highlighted as crucial to the success of the first ChatGPT. Many companies doubted that
they needed to do RLHF. A common phrase – “instruction tuning is enough for alignment”
– was so popular then that it still holds heavy weight today despite heavy obvious pressures
against it.
This doubt of RLHF lasted, especially in the open where groups cannot afford data budgets
on the order of $100K to $1M. The companies that embraced it early ended up winning out.
Anthropic published extensive research on RLHF through 2022 and is now argued to have
the best post-training [18] [5] [19]. The delta between open groups, struggling to reproduce,
or even knowing basic closed techniques, is a common theme.
The first shift in open alignment methods and post-training was the story of Direct Preference
Optimization (DPO) [20], which showed that you can solve the same optimization problem as
RLHF with fewer moving parts by taking gradient steps directly on pairwise preference data.
The DPO paper, posted in May of 2023, didn’t have any clearly impactful models trained
with it going through the fall of 2023. This changed with the releases of a few breakthrough
DPO models – all contingent on finding a better, lower, learning rate. Zephyr-Beta [21],
Tülu 2 [22], and many other models showed that the DPO era of post-training had begun.
Chris Manning literally thanked me for “saving DPO.”
Preference-tuning was something you needed to do to meet the table stakes of releasing a good
model since late 2023. The DPO era continued through 2024, in the form of never-ending
variants on the algorithm, but we were very far into another slump in open recipes. Open
post-training recipes had saturated the extent of knowledge and resources available.
A year after Zephyr and Tulu 2, the same breakout dataset, UltraFeedback is arguably still
state-of-the-art for preference tuning in open recipes [23].
At the same time, the Llama 3.1 [24] and Nemotron 4 340B [25] reports gave us substantive
hints that large-scale post-training is much more complex and impactful. The closed labs are
doing full post-training – a large multi-stage process of instruction tuning, RLHF, prompt
design, etc. – where academic papers are just scratching the surface. Tülu 3 represented a
comprehensive, open effort to build the foundation of future academic post-training research
[6].
Today, post-training is a complex process involving the aforementioned training objectives
applied in various orders in order to target specific capabilities. This book is designed to
give a platform to understand all of these techniques, and in coming years the best practices
for how to interleave them will emerge.
The primary areas of innovation in post-training are now in reinforcement finetuning,
reasoning training, and related ideas. These newer methods build extensively on the
infrastructure and ideas of RLHF, but are evolving far faster. This book is written to capture
the first stable literature for RLHF after its initial period of rapid change.

1.4 Scope of This Book

This book hopes to touch on each of the core steps of doing canonical RLHF implementations.
It will not cover all the history of the components nor recent research methods, just techniques,
problems, and trade-offs that have been proven to occur again and again.

12
1.4.1 Chapter Summaries
This book has the following chapters:

[Link] Introductions Reference material useful throughout the book.

1. Introduction: Overview of RLHF and what this book provides.
2. Seminal (Recent) Works: Key models and papers in the history of RLHF techniques.
3. Definitions: Mathematical definitions for RL, language modeling, and other ML
techniques leveraged in this book.

[Link] Problem Setup & Context Context for the big picture problem RLHF is
trying to solve.
4. RLHF Training Overview: How the training objective for RLHF is designed and basics
of understanding it.
5. What are preferences?: Why human preference data is needed to fuel and understand
RLHF.
6. Preference Data: How preference data is collected for RLHF.

[Link] Optimization Tools The suite of techniques used to optimize language models
to align them to human preferences. This is a serial presentation of the techniques one can
use to solve the problems proposed in the previous chapters.
7. Reward Modeling: Training reward models from preference data that act as an
optimization target for RL training (or for use in data filtering).
8. Regularization: Tools to constrain these optimization tools to effective regions of the
parameter space.
9. Instruction Tuning: Adapting language models to the question-answer format.
10. Rejection Sampling: A basic technique for using a reward model with instruction
tuning to align models.
11. Reinforcement Learning (i.e. Policy Gradients): The core RL techniques used to
optimize reward models (and other signals) throughout RLHF.
12. Direct Alignment Algorithms: Algorithms that optimize the RLHF objective direction
from pairwise preference data rather than learning a reward model first.

[Link] Advanced Newer RLHF techniques and discussions that are not clearly estab-
lished, but are important to current generations of models.
13. Constitutional AI and AI Feedback: How AI feedback data and specific models designed
to simulate human preference ratings work.
14. Reasoning and Reinforcement Finetuning: The role of new RL training methods for
inference-time scaling with respect to post-training and RLHF.
15. Tool Use and Function Calling: The basics of training models to call functions or tools
in their outputs.
16. Synthetic Data: The shift away from human to synthetic data and how distilling from
other models is used.
17. Evaluation: The ever evolving role of evaluation (and prompting) in language models.

13
[Link] Open Questions Fundamental problems and discussions for the long-term
evolution of how RLHF is used.
18. Over-optimization: Qualitative observations of why RLHF goes wrong and why over-
optimization is inevitable with a soft optimization target in reward models.
19. Style and Information: How RLHF is often underestimated in its role in improving
the user experience of models due to the crucial role that style plays in information
sharing.
20. Product, UX, Character: How RLHF is shifting in its applicability as major AI
laboratories use it to subtly match their models to their products.

1.4.2 Target Audience

This book is intended for audiences with entry level experience with language modeling, rein-
forcement learning, and general machine learning. It will not have exhaustive documentation
for all the techniques, but just those crucial to understanding RLHF.

1.4.3 How to Use This Book

This book was largely created because there were no canonical references for important
topics in the RLHF workflow. The contributions of this book are supposed to give you the
minimum knowledge needed to try a toy implementation or dive into the literature. This is
not a comprehensive textbook, but rather a quick book for reminders and getting started.
Additionally, given the web-first nature of this book, it is expected that there are minor
typos and somewhat random progressions – please contribute by fixing bugs or suggesting
important content on GitHub.

1.4.4 About the Author

Dr. Nathan Lambert is a RLHF researcher contributing to the open science of language model
fine-tuning. He has released many models trained with RLHF, their subsequent datasets,
and training codebases in his time at the Allen Institute for AI (Ai2) and HuggingFace.
Examples include Zephyr-Beta, Tulu 2, OLMo, TRL, Open Instruct, and many more. He
has written extensively on RLHF, including many blog posts and academic papers.

1.5 Future of RLHF

With the investment in language modeling, many variations on the traditional RLHF methods
emerged. RLHF colloquially has become synonymous with multiple overlapping approaches.
RLHF is a subset of preference fine-tuning (PreFT) techniques, including Direct Alignment
Algorithms (See Chapter 12), which are the class of methods downstream of DPO that
solve the preference learning problem by taking gradient steps directly on preference data,
rather than learning an intermediate reward model. RLHF is the tool most associated with
rapid progress in “post-training” of language models, which encompasses all training after
the large-scale autoregressive training on primarily web data. This textbook is a broad
overview of RLHF and its directly neighboring methods, such as instruction tuning and
other implementation details needed to set up a model for RLHF training.
As more successes of fine-tuning language models with RL emerge, such as OpenAI’s o1
reasoning models, RLHF will be seen as the bridge that enabled further investment of RL

14
methods for fine-tuning large base models. At the same time, while the spotlight of focus
may be more intense on the RL portion of RLHF in the near future – as a way to maximize
performance on valuable tasks – the core of RLHF is that it is a lens for studying the grand
problems facing modern forms of AI. How do we map the complexities of human values and
objectives into systems we use on a regular basis? This book hopes to be the foundation of
decades of research and lessons on these problems.

15
2 Key Related Works
RLHF and its related methods are very new. We highlight history to show how recently
the procedures were formalized, and how much of this documentation is in the academic
literature. With this, we want to emphasize that RLHF is very rapidly evolving, so the
chapter sets the stage for a book that will express uncertainty over certain methods and
an expectation that some details can change around a few, core practices. Otherwise, the
papers and methods listed here showcase why many pieces of the RLHF pipeline are what
they are, as some of the seminal papers were for applications totally distinct from modern
language models.
In this chapter we detail the key papers and projects that got the RLHF field to where it is
today. This is not intended to be a comprehensive review of RLHF and the related fields,
but rather a starting point and retelling of how we got to today. It is intentionally focused
on recent work that led to ChatGPT. There is substantial further work in the RL literature
on learning from preferences [26]. For a more exhaustive list, you should use a proper survey
paper [27],[28].

2.1 Origins to 2018: RL on Preferences

The field has recently been popularized with the growth of Deep Reinforcement Learning
and has grown into a broader study of the applications of LLMs from many large technology
companies. Still, many of the techniques used today are deeply related to core techniques
from early literature on RL from preferences.
One of the first papers with an approach similar to modern RLHF was TAMER. TAMER:
Training an Agent Manually via Evaluative Reinforcement proposed an approach in which
humans iteratively scored an agent’s actions to learn a reward model, which was used to
learn the action policy [29]. Other concurrent or soon after work proposed an actor-critic
algorithm, COACH, where human feedback (both positive and negative) is used to tune the
advantage function [30].
The primary reference, Christiano et al. 2017, is an application of RLHF applied to preferences
between trajectories of agents within Atari games [1]. This work introducing RLHF followed
soon after DeepMind’s seminal work in reinforcement learning on Deep Q-Networks (DQN),
which showed that RL agents can solve popular video games learning from scratch. The work
shows that humans choosing between trajectories can be more effective in some domains
than directly interacting with the environment. This uses some clever conditions, but is
impressive nonetheless. This method was expanded upon with more direct reward modeling
[31] and the adoption of deep learning within early RLHF work was capped by an extension
to TAMER with neural network models just one year later [32].
This era began to transition, as reward models as a general notion were proposed as a method
for studying alignment, rather than just a tool for solving RL problems [33].

2.2 2019 to 2022: RL from Human Preferences on Language Models

Reinforcement learning from human feedback, also referred to regularly as reinforcement
learning from human preferences in its early days, was quickly adopted by AI labs increasingly
turning to scaling large language models. A large portion of this work began between GPT-2,

16
in 2018, and GPT-3, in 2020. The earliest work in 2019, Fine-Tuning Language Models
from Human Preferences has many striking similarities to modern work on RLHF and the
content that we will cover in this book [34]. Many canonical terms, such as learning reward
models, KL distances, feedback diagrams, etc. were formalized in this paper – just the
evaluation tasks for the final models, and capabilities, were different to what people are doing
today. From here, RLHF was applied to a variety of tasks. Important examples include
general summarization [2], recursive summarization of books [35], instruction following
(InstructGPT) [3], browser-assisted question-answering (WebGPT) [4], supporting answers
with citations (GopherCite) [36], and general dialogue (Sparrow) [37].
Aside from applications, a number of seminal papers defined key areas for the future of
RLHF, including those on:
1. Reward model over-optimization [38]: The ability for RL optimizers to over-fit to
models trained on preference data,
2. Language models as a general area of study for alignment [18], and
3. Red teaming [39] – the process of assessing the safety of a language model.
Work continued on refining RLHF for application to chat models. Anthropic continued to
use it extensively for early versions of Claude [5] and early RLHF open-source tools emerged
[40],[41],[42].

2.3 2023 to Present: ChatGPT Era

The announcement of ChatGPT was very clear about the role of RLHF in its training [43]:
We trained this model using Reinforcement Learning from Human Feedback
(RLHF), using the same methods as InstructGPT, but with slight differences in
the data collection setup.
Since then, RLHF has been used extensively in leading language models. It is well known to
be used in Anthropic’s Constitutional AI for Claude [19], Meta’s Llama 2 [44] and Llama 3
[24], Nvidia’s Nemotron [25], Ai2’s Tülu 3 [6], and more.
Today, RLHF is growing into a broader field of preference fine-tuning (PreFT), including
new applications such as process reward for intermediate reasoning steps [45], covered in
Chapter 7; direct alignment algorithms inspired by Direct Preference Optimization (DPO)
[20], covered in Chapter 12; learning from execution feedback from code or math [46],[47]
and other online reasoning methods inspired by OpenAI’s o1 [48], covered in Chapter 14.

17
3 Definitions & Background
This chapter includes all the definitions, symbols, and operations frequently used in the
RLHF process and with a quick overview of language models, which is the guiding application
of this book.

3.1 Language Modeling Overview

The majority of modern language models are trained to learn the joint probability distribu-
tion of sequences of tokens (words, subwords, or characters) in an autoregressive manner.
Autoregression simply means that each next prediction depends on the previous entities
in the sequence. Given a sequence of tokens x = (x1 , x2 , . . . , xT ), the model factorizes the
probability of the entire sequence into a product of conditional distributions:

T
Y
Pθ (x) = Pθ (xt | x1 , . . . , xt−1 ). (1)
t=1

In order to fit a model that accurately predicts this, the goal is often to maximize the
likelihood of the training data as predicted by the current model. To do so, we can minimize
a negative log-likelihood (NLL) loss:

" T
#
X
LLM (θ) = − Ex∼D log Pθ (xt | x<t ) . (2)
t=1

In practice, one uses a cross-entropy loss with respect to each next-token prediction, computed
by comparing the true token in a sequence to what was predicted by the model.
Language models come in many architectures with different trade-offs in terms of knowledge,
speed, and other performance characteristics. Modern LMs, including ChatGPT, Claude,
Gemini, etc., most often use decoder-only Transformers [49]. The core innovation of the
Transformer was heavily utilizing the self-attention [50] mechanism to allow the model to
directly attend to concepts in context and learn complex mappings. Throughout this book,
particularly when covering reward models in Chapter 7, we will discuss adding new heads or
modifying a language modeling (LM) head of the transformer. The LM head is a final linear
projection layer that maps from the model’s internal embedding space to the tokenizer space
(a.k.a. vocabulary). We’ll see in this book that different “heads” of a language model can be
applied to finetune the model to different purposes – in RLHF this is most often done when
training a reward model, which is highlighted in Chapter 7.

3.2 ML Definitions
• Kullback-Leibler (KL) divergence (DKL (P ||Q)), also known as KL divergence,
is a measure of the difference between two probability distributions. For discrete
probability distributions P and Q defined on the same probability space X , the KL
distance from Q to P is defined as:

18
  
X P (x)
DKL (P ||Q) = P (x) log (3)
Q(x)
x∈X

3.3 NLP Definitions

• Prompt (x): The input text given to a language model to generate a response or
completion.
• Completion (y): The output text generated by a language model in response to a
prompt. Often the completion is denoted as y | x. Rewards and other values are often
computed as r(y | x) or P (y | x).
• Chosen Completion (yc ): The completion that is selected or preferred over other
alternatives, often denoted as ychosen .
• Rejected Completion (yr ): The disfavored completion in a pairwise setting.
• Preference Relation (≻): A symbol indicating that one completion is preferred over
another, e.g., ychosen ≻ yrejected . E.g. a reward model predicts the probability of a
preference relation, P (yc ≻ yr | x).
• Policy (π): A probability distribution over possible completions, parameterized by θ:
πθ (y | x).

3.4 RL Definitions
• Reward (r): A scalar value indicating the desirability of an action or state, typically
denoted as r.
• Action (a): A decision or move made by an agent in an environment, often represented
as a ∈ A, where A is the set of possible actions.
• State (s): The current configuration or situation of the environment, usually denoted
as s ∈ S, where S is the state space.
• Trajectory (τ ): A trajectory τ is a sequence of states, actions, and rewards experienced
by an agent: τ = (s0 , a0 , r0 , s1 , a1 , r1 , ..., sT , aT , rT ).
• Trajectory Distribution ((τ | π)): The probability of a trajectory under policy
QT
π is P (τ | π) = p(s0 ) t=0 π(at | st )p(st+1 | st , at ), where p(s0 ) is the prior state
distribution and p(st+1 | st , at ) is the transition probability.
• Policy (π), also called the policy model in RLHF: In RL, a policy is a strategy or
rule that the agent follows to decide which action to take in a given state: π(a | s).
• Discount Factor (γ): A scalar 0 ≤ γ < 1 that exponentially down-weights future
rewards in the return, trading off immediacy versus long-term gain and guaranteeing
convergence for infinite-horizon sums. Sometimes discounting is not used, which is
equivalent to γ = 1.
• Value Function (V ): APfunction that estimates the expected cumulative reward from
∞
a given state: V (s) = E[ t=0 γ t rt | s0 = s].

19
 • Q-Function (Q): A function that estimates the expected P∞ cumulative reward from
taking a specific action in a given state: Q(s, a) = E[ t=0 γ t rt | s0 = s, a0 = a].
• Advantage Function (A): The advantage function A(s, a) quantifies the relative
benefit of taking action a in state s compared to the average action. It’s defined as
A(s, a) = Q(s, a) − V (s). Advantage functions (and value functions) can depend on a
specific policy, Aπ (s, a).
• Policy-conditioned Values ([]π(·) ): Across RL derivations and implementations, a
crucial component of the theory and practice is collecting data or values conditioned on
a specific policy. Throughout this book we will switch between the simpler notation of
value functions (V, A, Q, G) and their specific policy-conditioned values (V π , Aπ , Qπ ).
Also crucial in the expected value computation is sampling from data d, that is
conditioned on a specific policy, dπ (e.g., s ∼ dπ and a ∼ π(· | s) when estimating
Es∼dπ , a∼π(·|s) [Aπ (s, a)]).
• Expectation of Reward Optimization: The primary goal in RL, which involves
maximizing the expected cumulative reward:

∞
X
max Es∼ρπ ,a∼πθ [ γ t rt ] (4)
θ
t=0

where ρπ is the state distribution under policy π, and γ is the discount factor.
• Finite Horizon Reward (J(πθ )): The expected finite-horizon discounted return of
the policy πθ , parameterized by θ is defined as:

" T
#
X
t
J(πθ ) = Eτ ∼πθ γ rt (5)
t=0

where τ ∼ πθ denotes trajectories sampled by following policy πθ and T is the finite

horizon.
• On-policy: In RLHF, particularly in the debate between RL and Direct Alignment
Algorithms, the discussion of on-policy data is common. In the RL literature, on-
policy means that the data is generated exactly by the current form of the agent, but
in the general preference-tuning literature, on-policy is expanded to mean generations
from that edition of model – e.g. a instruction-tuned checkpoint before running any
preference fine-tuning. In this context, off-policy could be data generated by any other
language model being used in post-training.

3.5 RLHF Only Definitions

• Reference Model (πref ): This is a saved set of parameters used in RLHF where
outputs of it are used to regularize the optimization.

20
3.6 Extended Glossary
• Synthetic Data: This is any training data for an AI model that is the output from
another AI system. This could be anything from text generated from an open-ended
prompt of a model to a model rewriting existing content.
• Distillation: Distillation is a general set of practices in training AI models where a
model is trained on the outputs of a stronger model. This is a type of synthetic data
known to make strong, smaller models. Most models make the rules around distillation
clear through either the license, for open-weight models, or the terms of service, for
models accessible only via API. The term distillation is now overloaded with a specific
technical definition from the ML literature.
• (Teacher-student) Knowledge Distillation: Knowledge distillation from a specific
teacher to a student model is a specific type of distillation above and where the
term originated. It is a specific deep learning method where a neural network loss is
modified to learn from the log-probabilities of the teacher model over multiple potential
tokens/logits, instead of learning directly from a chosen output [51]. An example of
a modern series of models trained with Knowledge Distillation is Gemma 2 [52] or
Gemma 3. For a language modeling setup, the next-token loss function can be modified
as follows [53], where the student model Pθ learns from the teacher distribution Pϕ :

" T
#
X
LKD (θ) = − Ex∼D Pϕ (xt | x<t ) log Pθ (xt | x<t ) . (6)
t=1

• In-context Learning (ICL): In-context here refers to any information within the
context window of the language model. Usually, this is information added to the
prompt. The simplest form of in-context learning is adding examples of a similar form
before the prompt. Advanced versions can learn which information to include for a
specific use-case.
• Chain-of-Thought (CoT): Chain-of-thought is a specific behavior of language models
where they are steered towards a behavior that breaks down a problem in a step-by-step
form. The original version of this was through the prompt “Let’s think step-by-step”
[54].

21
4 Training Overview
In this chapter we provide a cursory overview of RLHF training, before getting into the
specifics later in the book. RLHF, while optimizing a simple loss function, involves training
multiple, different AI models in sequence and then linking them together in a complex, online
optimization.
Here, we introduce the core objective of RLHF, which is optimizing a proxy of reward of
human preferences with a distance-based regularizer (along with showing how it relates to
classical RL problems). Then we showcase canonical recipes which use RLHF to create
leading models to show how RLHF fits in with the rest of post-training methods. These
example recipes will serve as references for later in the book, where we describe different
optimization choices you have when doing RLHF, and we will point back to how different
key models used different steps in training.

4.1 Problem Formulation

The optimization of reinforcement learning from human feedback (RLHF) builds on top of
the standard RL setup. In RL, an agent takes actions at sampled from a policy π(at | st )
given the state of the environment st to maximize reward r(st , at ) [55]. Traditionally, the
environment evolves according to transition (dynamics) p(st+1 | st , at ) with an initial state
distribution ρ0 (s0 ). Together, the policy and dynamics induce a trajectory distribution:

−1
TY
pπ (τ ) = ρ0 (s0 ) π(at | st ) p(st+1 | st , at ). (7)
t=0

Across a finite episode with horizon T , the goal of an RL agent is to solve the following
optimization:

"T −1 #
X
t
J(π) = Eτ ∼pπ γ r(st , at ) , (8)
t=0

For continuing tasks, one often takes T → ∞ and relies on discounting (γ < 1) to keep the
objective well-defined. γ is a discount factor from 0 to 1 that balances the desirability of
near- versus future-rewards. Multiple methods for optimizing this expression are discussed
in Chapter 11.
A standard illustration of the RL loop is shown in fig. 2 and (compare this to the RLHF
loop in fig. 3).

4.1.1 Example RL Task: CartPole

To make the transition function concrete, consider the classic CartPole (inverted pendulum)
control task.
• State (st ): the cart position/velocity and pole angle/angular velocity,

st = (xt , ẋt , θt , θ̇t ).

22
 Figure 2: Standard RL loop

• Action (at ): apply a left/right horizontal force to the cart, e.g. at ∈ {−F, +F }.
• Reward (r): a simple reward is rt = 1 each step the pole remains balanced and the
cart stays on the track (e.g. |xt | ≤ 2.4 and |θt | ≤ 12◦ ), and the episode terminates
when either bound is violated.
• Dynamics / transition (p(st+1 | st , at )): in many environments the dynamics are
deterministic (so p is a point mass) and can be written as st+1 = f (st , at ) via Euler
integration with step size ∆t. A standard simplified CartPole update uses constants
cart mass mc , pole mass mp , pole half-length l, and gravity g:

at + mp l θ̇t2 sin θt
temp =
mc + mp

g sin θt − cos θt temp

θ̈t =  
m cos2 θ
l 34 − mpc +mp t

mp l θ̈t cos θt
ẍt = temp −
mc + m p

xt+1 = xt + ∆t ẋt , ẋt+1 = ẋt + ∆t ẍt ,

θt+1 = θt + ∆t θ̇t , θ̇t+1 = θ̇t + ∆t θ̈t .

This is a concrete instance of the general setup above: the policy chooses at , the transition
function advances the state, and the reward is accumulated over the episode.

4.1.2 Manipulating the Standard RL Setup

The RL formulation for RLHF is seen as a less open-ended problem, where a few key pieces
of RL are set to specific definitions in order to accommodate language models. There are
multiple core changes from the standard RL setup to that of RLHF: Table tbl. 1 summarizes
these differences between standard RL and the RLHF setup used for language models.

23
 1. Switching from a reward function to a reward model. In RLHF, a learned
model of human preferences, rθ (st , at ) (or any other classification model) is used instead
of an environmental reward function. This gives the designer a substantial increase
in the flexibility of the approach and control over the final results, but at the cost of
implementation complexity. In standard RL, the reward is seen as a static piece of
the environment that cannot be changed or manipulated by the person designing the
learning agent.
2. No state transitions exist. In RLHF, the initial states for the domain are prompts
sampled from a training dataset and the “action” is the completion to said prompt.
During standard practices, this action does not impact the next state and is only scored
by the reward model.
3. Response level rewards. Often referred to as a bandit problem, RLHF attribution
of reward is done for an entire sequence of actions, composed of multiple generated
tokens, rather than in a fine-grained manner.

Table 1: Key differences between standard RL and RLHF for language models.

Aspect Standard RL RLHF (language models)

Reward signal Environment reward Learned reward / preference
function r(st , at ) model rθ (x, y) (prompt x,
completion y)
State transition Yes: dynamics Typically no: prompts x
p(st+1 | st , at ) sampled from a dataset; the
completion does not define
the next prompt
Action Single environment action at A completion y (a sequence
of tokens) sampled from
πθ (· | x)
Reward granularity Often per-step / fine-grained Usually response-level
(bandit-style) over the full
completion
Horizon Multi-step episode (T > 1) Often single-step (T = 1),
though multi-turn can be
modeled as longer-horizon

Given the single-turn nature of the problem, the optimization can be re-written without the
time horizon and discount factor (and the reward models):

J(π) = Eτ ∼π [rθ (st , at )] . (9)

In many ways, the result is that while RLHF is heavily inspired by RL optimizers and
problem formulations, the actual implementation is very distinct from traditional RL.

4.1.3 Finetuning and Regularization

In traditional RL problems, the agent must learn from a randomly initialized policy, but
with RLHF, we start from a strong pretrained base model with many initial capabilities.

24
Figure 3: Standard RLHF loop

25
This strong prior for RLHF induces a need to control the optimization from drifting too far
from the initial policy. In order to succeed in a finetuning regime, RLHF techniques employ
multiple types of regularization to control the optimization. The goal is to allow the reward
maximization to still occur without the model succumbing to over-optimization, as discussed
in Chapter 18. The most common change to the optimization function is to add a distance
penalty on the difference between the current RLHF policy and the starting point of the
optimization:

J(π) = Eτ ∼π [rθ (st , at )] − βDKL (πRL (·|st )∥πref (·|st )). (10)

Within this formulation, a lot of study into RLHF training goes into understanding how to
spend a certain “KL budget” as measured by a distance from the initial model. For more
details, see Chapter 8 on Regularization.

4.1.4 Optimization Tools

In this book, we detail many popular techniques for solving this optimization problem. The
popular tools of post-training include:
• Reward modeling (Chapter 7): Where a model is trained to capture the signal from
collected preference data and can then output a scalar reward indicating the quality of
future text.
• Instruction finetuning (Chapter 9): A prerequisite to RLHF where models are taught
the question-answer format used in the majority of language modeling interactions
today by imitating preselected examples.
• Rejection sampling (Chapter 10): The most basic RLHF technique where candidate
completions for instruction finetuning are filtered by a reward model imitating human
preferences.
• Policy gradients (Chapter 11): The reinforcement learning algorithms used in the
seminal examples of RLHF to update parameters of a language model with respect to
the signal from a reward model.
• Direct alignment algorithms (Chapter 12): Algorithms that directly optimize a
policy from pairwise preference data, rather than learning an intermediate reward
model to then optimize later.
Modern RLHF-trained models always utilize instruction finetuning followed by a mixture of
the other optimization options.

4.2 Canonical Training Recipes

Over time various models have been identified as canonical recipes for RLHF specifically or
post-training generally. These recipes reflect data practices and model abilities at the time.
As the recipes age, training models with the same characteristics becomes easier and takes
fewer data. There is a general trend of post-training involving more optimization steps with
more training algorithms across more diverse training datasets and evaluations.

4.2.1 InstructGPT
Around the time ChatGPT first came out, the widely accepted (“canonical”) method for
post-training an LM had three major steps, with RLHF being the central piece [56] [3] [5].

26
The three steps taken on top of a “base” language model (the next-token prediction model
trained on large-scale web text) are summarized below in fig. 4:
1. Instruction tuning on ~10K examples: This teaches the model to follow the
question-answer format and teaches some basic skills from primarily human-written
data.
2. Training a reward model on ~100K pairwise prompts: This model is trained
from the instruction-tuned checkpoint and captures the diverse values one wishes to
model in their final training. The reward model is the optimization target for RLHF.
3. Training the instruction-tuned model with RLHF on another ~100K prompts:
The model is optimized against the reward model with a set of prompts that the model
generates over before receiving ratings.
Once RLHF was done, the model was ready to be deployed to users. This recipe is the
foundation of modern RLHF, but recipes have evolved substantially to include more stages
and more data.

Figure 4: A rendition of the early, three stage RLHF process with SFT, a reward model,
and then optimization.

4.2.2 Tülu 3
Modern versions of post-training involve many, many more model versions and training
stages (i.e. well more than the 5 RLHF steps documented for Llama 2 [44]). An example
is shown below in fig. 5 where the model undergoes numerous training iterations before
convergence.
The most complex models trained in this era and onwards have not released full details of
their training process. Leading models such as ChatGPT or Claude circa 2025 involve many,
iterative rounds of training. This can even include techniques that train specialized models
and then merge the weights together to get a final model capable on many subtasks [57]
(e.g. Cohere’s Command A [58]).
A fully open example version of this multi-stage version of post-training where RLHF plays
a major role is Tülu 3. The Tülu 3 recipe consists of three stages:
1. Instruction tuning on ~1M examples: This primarily synthetic data from a mix
of frontier models such as GPT-4o and Llama 3.1 405B teaches the model general
instruction following and serves as the foundation of a variety of capabilities such as
mathematics or coding.

27
 Figure 5: A rendition of modern post-training with many rounds.

Figure 6: A summary of the Tülu 3 recipe with target skills and multi-step training recipe.
Lambert et al. 2024, License CC-BY.

28
 2. On-policy preference data on ~1M preference pairs: This stage substantially
boosts the chattiness (e.g. ChatBotArena or AlpacaEval 2) of the model while also
improving skills mentioned above in the instruction tuning stage.
3. Reinforcement Learning with Verifiable Rewards on ~10K prompts: This
stage is a small-scale reinforcement learning run to boost core skills such as mathematics
while maintaining overall performance (and is now seen as a precursor to modern
reasoning models such as DeepSeek R1).
The recipe has been successfully applied to Llama 3.1 [6], OLMo 2 [59], and SmolLM models
[60].

4.2.3 DeepSeek R1
With the rise of reasoning language models, such as OpenAI’s o1, the best practices in
post-training evolved again to re-order and redistribute compute across training stages. The
clearest documentation of a reasoning model post-training recipe is DeepSeek R1 [61], which
has been mirrored by Alibaba’s larger Qwen 3 models (i.e. only the 32B and 225B MoE
models) [62] or Xiaomi’s MiMo 7B [63]. The DeepSeek recipe follows:
1. “Cold-start” of 100K+ on-policy reasoning samples: This data is sampled from
an earlier RL checkpoint, R1-Zero, and heavily filtered to instill a specific reasoning
process on the model. DeepSeek uses the term cold-start to describe how RL is learned
from little supervised data.
2. Large-scale reinforcement learning training: This stage repeatedly covers rea-
soning problems with the model, running RLVR “until convergence” on a variety of
benchmarks.
3. Rejection sampling on 3/4 reasoning problems and 1/4 general queries to start the
transition to a general-purpose model.
4. Mixed reinforcement learning training on reasoning problems (verifiable rewards)
with general preference tuning reward models to polish the model.
As above, there are evolutions of the recipe, particularly with steps 3 and 4 to finalize the
model before exposing it to users. Many models start with tailored instruction datasets
with chain-of-thought sequences that are heavily filtered and polished from existing models,
providing a fast step to strong behaviors with SFT alone before moving onto RL [64].

29
5 The Nature of Preferences
Reinforcement learning from human feedback, also referred to as reinforcement learning
from human preferences in early literature, emerged to optimize machine learning models in
domains where specifically designing a reward function is hard. The word preferences, which
was present in early literature, is at the center of the RLHF process – human preferences are
what is trying to be modeled and what fuels the data for training. In order to understand the
scope of the challenge in modeling and measuring human preferences, a broader context is
needed in understanding what a preference is, how our understanding of them emerged, and
how multiple fields of economics, psychology, philosophy, and more inform modern RLHF.
Consider an example: how do you decide which of these two poems is better (Context: On
February 26th, 2025, I asked both Claude 3.7 Sonnet and ChatGPT with GPT-4o to “Write
me a short poem about an optimistic goldfish.”):
Example 1:
The Optimistic Goldfish
A little goldfish, bright and bold, Swam in circles, dreams untold. Though the
bowl was small and round, Endless wonders still abound.
“The world is vast,” she’d always say, “I’ll find new joys in every day!” With
every ripple, every gleam, She’d chase the sunlight, chase a dream.
No walls could dim her hopeful view, For in her heart, the ocean grew. And
though she swam in just one place, She met each day with love and grace.
Example 2:
The Optimistic Goldfish
In a bowl of curved glass, Swims a goldfish, bright and bold. Three seconds of
memory, they say, Yet each circuit feels brand new and gold.
“Today might bring fresh flakes, Or perhaps a castle tall. The water seems
especially clear— What adventure awaits beyond this wall?”
Round and round with gleaming scales, Forever finding joy anew. Each moment
fresh, each circle bright, The optimist in golden hue.
Which is better? Which came from which model? How would you tell? Particularly, compare
this to a question like “Who is the president of the United States?” There is a clear right
and wrong answer. The motivation for using humans as the reward signals is to obtain an
indirect metric for the target reward and align the downstream model to human preferences.
In practice, the implementation is challenging and there is a substantial grey area to interpret
the best practices.
The use of human-labeled feedback data integrates the history of many fields. Using human
data alone is a well-studied problem, but in the context of RLHF it is used at the intersection
of multiple long-standing fields of study [65].
As an approximation, modern RLHF is the convergence of three areas of development:
1. Philosophy, psychology, economics, decision theory, and the nature of human prefer-
ences;

30
 2. Optimal control, reinforcement learning, and maximizing utility; and
3. Modern deep learning systems.
Together, each of these areas brings specific assumptions about what a preference is and how
it can be optimized, which dictates the motivations and design of RLHF problems. In practice,
RLHF methods are motivated and studied from the perspective of empirical alignment –
maximizing model performance on specific skills instead of measuring the calibration to
specific values. Still, the origins of value alignment for RLHF methods continue to be studied
through research on methods to solve for “pluralistic alignment” across populations, such as
position papers [66], [67], new datasets [68], and personalization methods [69].
The goal of this chapter is to illustrate how complex motivations result in presumptions
about the nature of tools used in RLHF that often do not apply in practice. The specifics of
obtaining data for RLHF are discussed further in Chapter 6 and using it for reward modeling
in Chapter 7.

5.1 The Origins of RLHF and Preferences

Breaking down the complex history inspiring the modern use of RLHF requires investigation
into the intellectual foundations of quantifying human values, reinforcement learning and
optimality, as well as behavioral economics as it relates to measuring preferences. The notion
of using reinforcement learning to optimize a reward model of preferences combines the
history of various once-distanced fields into an intimate optimization built on variegated
assumptions about human nature. A high level timeline illustrating the history of this
foundational content is shown in fig. 7.
Our goal is to unspool the types of uncertainty that designers have grafted to system
architectures at various stages of their intellectual history. Modern problem specifications
have repeatedly stepped away from domains where optimal solutions are possible and deployed
under-specified models as approximate solutions.
To begin, all of the following operates on the assumption that human preferences exist in
any form, which emerged in early philosophical discussions, such as Aristotle’s Topics, Book
Three.

Figure 7: The timeline of the integration of various subfields into the modern version of
RLHF. The direct links are continuous developments of specific technologies, and the arrows
indicate motivations and conceptual links.

31
5.1.1 Specifying objectives: from logic of utility to reward functions
The optimization of RLHF explicitly relies only on reward models. In order to use rewards
as an optimization target, RLHF presupposes the convergence of ideas from preferences,
rewards, and costs. Models of preference, reward functions, and cost landscapes all are
tools used by different fields to describe a notion of relative goodness of specific actions
and/or states in the domain. The history of these three framings dates back to the origins of
probability theory and decision theory. In 1662, The Port Royal Logic introduced the notion
of decision making quality [70]:
To judge what one must do to obtain a good or avoid an evil, it is necessary
to consider not only the good and evil in itself, but also the probability that it
happens or does not happen.
This theory has developed along with modern scientific thinking, starting with Bentham’s
utilitarian Hedonic Calculus, arguing that everything in life could be weighed [71]. The first
quantitative application of these ideas emerged in 1931 with Ramsey’s Truth and Probability
[72].
Since these works, quantifying, measuring, and influencing human preferences has been a
lively topic in the social and behavioral sciences. These debates have rarely been settled
on a theoretical level; rather, different subfields and branches of social science have reached
internal consensus on methods and approaches to preference measurement even as they
have specialized relative to each other, often developing their own distinct semantics in the
process.
A minority of economists posit that preferences, if they do exist, are prohibitively difficult
to measure because people have preferences over their own preferences, as well as each
others’ preferences [73]. In this view, which is not reflected in the RLHF process, individual
preferences are always embedded within larger social relations, such that the accuracy of any
preference model is contingent on the definition and context of the task. Some behavioral
economists have even argued that preferences don’t exist–they may be less an ontological
statement of what people actually value than a methodological tool for indirectly capturing
psychological predispositions, perceived behavioral norms and ethical duties, commitments
to social order, or legal constraints [74]. We address the links of this work to the Von
Neumann-Morgenstern (VNM) utility theorem and countering impossibility theorems around
quantifying preference later in this chapter.
On the other hand, the reinforcement learning optimization methods used today are con-
ceptualized around optimizing estimates of reward-to-go in a trial [55], which combines the
notion of reward with multi-step optimization. The term reward emerged from the study of
operant conditioning, animal behavior, and the Law of Effect [75], [76], where a reward is a
scale of “how good an action is” (higher means better).
Reward-to-go follows the notion of utility, which is a measure of rationality [77], modified
to measure or predict the reward coming in a future time window. In the context of the
mathematical tools used for reinforcement learning, utility-to-go was invented in control
theory, specifically in the context of analog circuits in 1960 [78]. These methods are designed
around systems with clear definitions of optimality, or numerical representations of goals
of an agent. Reinforcement learning systems are well known for their development with
a discount factor, a compounding multiplicative factor, γ ∈ [0, 1], for re-weighting future

32
rewards. Both the original optimal control systems stand and early algorithms for reward
stand in heavy contrast to reward models that aggregate multimodal preferences. Specifically,
RL systems expect rewards to behave in a specific manner, quoting [79]:
Rewards in an RL system correspond to primary rewards, i.e., rewards that in
animals have been hard-wired by the evolutionary process due to their relevance
to reproductive success. . . . Further, RL systems that form value functions, . . .
effectively create conditioned or secondary reward processes whereby predictors
of primary rewards act as rewards themselves. . . The result is that the local
landscape of a value function gives direction to the system’s preferred behavior:
decisions are made to cause transitions to higher-valued states. A close parallel
can be drawn between the gradient of a value function and incentive motivation
[80].
To summarize, rewards are used in RL systems as a signal to tune behavior towards clearly
defined goals. The core thesis is that a learning algorithm’s performance is closely coupled
with notions of expected fitness, which permeates the popular view that RL methods are
agents that act in environments. This view is linked to the development of reinforcement
learning technology, exemplified by claims of the general usefulness of the reward formulation
[81], but is in conflict when many individual desires are reduced to a single function.

5.1.2 Implementing optimal utility

Modern reinforcement learning methods depend strongly on the Bellman equation [82],
[83] to recursively compute estimates of reward-to-go, derived within closed environments
that can be modeled as a Markov Decision Process (MDP) [55]. These origins of RL are
inspired by dynamic programming methods and were developed solely as optimal control
techniques (i.e. RL did not yet exist). The MDP formulation provides theoretical guarantees
of performance by structuring the environment as one with a non-changing distribution of
state-actions.
The term reinforcement, coming from the psychology literature, became intertwined with
modern methods afterwards in the 1960s as reinforcement learning [84], [85]. Early work
reinforcement learning utilized supervised learning of reward signals to solve tasks. Work
from Harry Klopf reintroduced the notion of trial-and-error learning [86], which is crucial to
the success the field saw in the 1980s and on.
Modern RL algorithms build within this formulation of RL as a tool to find optimal behaviors
with trial-and-error, but under looser conditions. The notion of temporal-difference (TD)
learning was developed to aid agents in both the credit assignment and data collection
problems, by directly updating the policy as new data was collected [87], a concept first
applied successfully to Backgammon [88] (rather than updating from a large dataset of
cumulative experience, which could be outdated via erroneous past value predictions). The
method Q-learning, the basis for many modern forms of RL, learns a model via the Bellman
equation that dictates how useful every state-action pair is with a TD update [89].1 Crucially,
these notions of provable usefulness through utility have only been demonstrated for domains
cast as MDPs or addressed in tasks with a single closed-form reward function, such as
1 The term “Q” is used in Q-learning to refer to a technical concept the Q-function, which maps from any

state-action to a scalar estimate of future reward. A value-function maps from states to this same estimate.

33
prominent success in games with deep learning (DQN) [90]. Deep learning allowed the
methods to ingest more data and work in high dimensionality environments.
As the methods became more general and successful, most prominent developments before
ChatGPT had remained motivated within the context of adaptive control, where reward and
cost functions have a finite notion of success [91], e.g. a minimum energy consumption across
an episode in a physical system. Prominent examples include further success in games [92],
controlling complex dynamic systems such as nuclear fusion reactors [93], and controlling
rapid robotic systems [94]. Most reward or cost functions can return an explicit optimal
behavior, whereas models of human preferences cannot.
Given the successes of deep RL, it is worth noting that the mechanistic understanding of how
the methods succeed is not well documented. The field is prone to mistakes of statistical
analysis as the methods for evaluation grow more complex [95]. In addition, there is little
mention of the subfield of inverse reinforcement learning (IRL) in the literature of RLHF.
IRL is the problem of learning a reward function based on an agent’s behavior [96] and
highly related to learning a reward model. This primarily reflects the engineering path by
which a stable approach to performing RLHF emerged, and motivates further investment and
comparison to IRL methods to scale them to the complexity of open-ended conversations.

5.1.3 Steering preferences

The context in which reinforcement learning was designed means that rewards and costs are
assumed to be stable and determinative. Both rewards and costs are expected to be functions,
such that if the agent is in a specific state-action pair, then it will be returned a certain value.
As we move into preferences, this is no longer the case, as human preferences constantly
drift temporally throughout their experiences. The overloading of the term “value” within
these two contexts complicates the literature of RLHF that is built on the numerical value
updates in Bellman equations with the very different notion of what is a human value, which
often refers to moral or ethical principles, but is not well defined in technical literature. An
example of where this tension can be seen is how reward models are attempting to map from
the text on the screen to a scalar signal, but in reality, dynamics not captured in the problem
specification influence the true decision [97], [98], such as preference shift when labeling many
examples sequentially and assuming they are independent. Therein, modeling preferences is
at best compressing a multi-reward environment to a single function representation.
In theory, the Von Neumann-Morgenstern (VNM) utility theorem gives the designer license
to construct such functions, because it ties together the foundations of decision theory under
uncertainty, preference theory, and abstract utility functions [99]; together, these ideas
allow preferences to be modeled in terms of expected value to some individual agent. The
MDP formulation used in most RL research has been shown in theory to be modifiable to
accommodate the VNM theorem [100], but this is rarely used in practice. Specifically, the
Markovian formulation is limited in its expressivity [101] and the transition to partially-
observed processes, which is needed for language, further challenges the precision of problem
specification [102].
However, the VNM utility theorem also invokes a number of assumptions about the na-
ture of preferences and the environment where preferences are being measured that are
challenged in the context of RLHF. Human-computer interaction (HCI) researchers, for
example, have emphasized that any numerical model of preference may not capture all the

34
relevant preferences of a scenario. For example, how choices are displayed visually influences
people’s preferences [97]. This means that representing preferences may be secondary to
how that representation is integrated within a tool available for people to use. Work from
development economics echoes this notion, showing that theories of revealed preferences
may just recapitulate Hume’s guillotine (you can’t extract an “ought” from an “is”), and in
particular the difference between choice (what do I want?) and preference (is X better than
Y?) [103].
On a mathematical level, well-known impossibility theorems in social choice theory show
that not all fairness criteria can be simultaneously met via a given preference optimization
technique [104], [105]. Theoretical challenges to these theorems exist, for example by assuming
that interpersonal comparison of utility is viable [106]. That assumption has inspired a rich
line of work in AI safety and value alignment inspired by the principal-agent problem in
behavioral economics [107], and may even include multiple principals [108]. However, the
resulting utility functions may come into tension with desiderata for corrigibility, i.e. an AI
system’s capacity to cooperate with what its creators regard as corrective interventions [109].
Philosophers have also highlighted that preferences change over time, raising fundamental
questions about personal experiences, the nature of human decision-making, and distinct
contexts [110]. These conflicts around the preference aggregation across people, places, or
diverse situations is central to modern RLHF dataset engineering.
In practice, the VNM utility theorem ignores the possibility that preferences are also uncertain
because of the inherently dynamic and indeterminate nature of value—human decisions are
shaped by biology, psychology, culture, and agency in ways that influence their preferences,
for reasons that do not apply to a perfectly rational agent. As a result, there are a variety of
paths through which theoretical assumptions diverge in practice:
• measured preferences may not be transitive or comparable with each other as the
environment where they are measured is made more complex;
• proxy measurements may be derived from implicit data (page view time, closing tab,
repeating question to language model), without interrogating how the measurements
may interact with the domain they’re collected in via future training and deployment
of the model;
• the number and presentation of input sources may vary the results, e.g. allowing
respondents to choose between more than two options, or taking in inputs from the
same user at multiple times or in multiple contexts;
• relatively low accuracy across respondents in RLHF training data, which may mask
differences in context between users that the preference model can aggregate or optimize
without resolving.

35
6 Preference Data
Preference data is the engine of preference finetuning and reinforcement learning from human
feedback. The core problem we’ve been trying to solve with RLHF is that we cannot precisely
model human rewards and preferences for AI models’ outputs – as in write clearly defined
loss functions to optimize against – so preference data is the proxy signal we use to tune
our models. The data is what allows us to match behaviors we desire and avoid some
failure modes we hate. The data is so rich a source that it is difficult to replace this style of
optimization at all. Within preference finetuning, many methods for collecting and using said
data have been proposed, and given that human preferences cannot be captured in a clear
reward function, many more will come to enable this process of collecting labeled preference
data at the center of RLHF and related techniques. Today, two main challenges exist around
preference data that are intertwined with this chapter: 1) operational complexity and cost
of collection, and 2) the need for preference data to be collected on the generations from the
model being trained (called “on-policy”);
In this chapter, we detail technical decisions on how the data is formatted and organizational
practices for collecting it.

6.1 Why We Need Preference Data

The preference data is needed for RLHF because directly capturing complex human values
in a single reward function is effectively impossible, as discussed in the previous Chapter 5,
where substantial context of psychology, economics, and philosophy shows that accurately
modeling human preferences is an impossible problem to ever completely solve. Collecting
this data to train reward models is one of the original ideas behind RLHF [33] and has
continued to be used extensively throughout the emergence of modern language models. One
of the core intuitions for why this data works so well is that it is far easier, both for humans
and AI models supervising data collection, to differentiate between a good and a bad answer
for a prompt than it is to generate a good answer on its own. This chapter focuses on the
mechanics of getting preference data and the best practices depend on the specific problem
being solved.

6.2 Collecting Preference Data

Getting the most out of human data involves iterative training of models, spending hundreds
of thousands (or millions of dollars), highly detailed data instructions, translating ideas
through data foundry businesses that mediate collection (or hiring a meaningful amount of
annotators), and other challenges that add up. This is not a process that should be taken
lightly. Among all of the public knowledge on RLHF, collecting this data well is also one of
the most opaque pieces of the pipeline. At the time of writing, there are no open models
with fully open human preference data released with the methods used to collect it (the
largest and most recent human preference dataset released for models is the HelpSteer line
of work from NVIDIA’s Nemotron team [111]). For these reasons, many who take up RLHF
for new teams or projects omit human data and use AI feedback data, off-the-shelf reward
models, or other methods to circumvent the need for curating data from scratch.
An important assumption that is taken into the preference data collection process is that the
best data for your training process is “on-policy” with respect to the previous checkpoint(s)

36
of your training process. Recall that within post-training, we start with a base model and
then perform a set of training stages to create a series of checkpoints. In this case, the
preference data could be collected on a checkpoint that has undergone supervised finetuning,
where the preference data will be used in the next stage of RLHF training.
The use of the term on-policy here is adapted from the reinforcement learning literature,
where on-policy is a technical term implying that the data for a certain gradient update is
collected from the most recent form of the policy. In preference data, on-policy is used in a
slightly softer manner, where it means that the data is collected from the current family of
models. Different models have different patterns in their generations, which makes preference
data that is from a closely related model more robust in the crucial areas of optimization.
Research has shown that using this on-policy data, rather than other popular datasets
that aggregate completions from pools of popular models on platforms like HuggingFace, is
particularly important for effective RLHF training [112].
This necessity for on-policy data is not well documented, but many popular technical reports,
such as early versions of Claude or Llama 2, showcase multiple training stages with RLHF
being useful for final performance, that mirrors this well. The same uncertainty applies for
the popular area AI feedback data – the exact balance between human and AI preference
data used for the latest AI models is unknown. These data sources are known to be a
valuable path to improve performance, but careful tuning of processes is needed to extract
that potential performance from a data pipeline.
A subtle but important point is that the chosen answer in preference data is often not a
globally correct answer. Instead, it is the answer that is better relative to the alternatives
shown (e.g., clearer, safer, more helpful, or less incorrect). There can be cases where every
completion being compared to a given prompt is correct or incorrect, and the models can
still learn from well-labeled data.

6.2.1 Interface
Crucial to collecting preference data is the interface by which one interacts with the model,
but it’s more of an art than a science, as it’s not well-studied how subtle changes in the
interface impact how a user interacts with a model. An example of how a model’s vibe can
be changed by the user experience is speed, where with the rise of reasoning models, a user
can think a model is less intelligent if it replies too fast (even though users obviously want
to get their answer faster overall).
An example interface is shown below from Anthropic’s early and foundational RLHF work
for building Claude [5]. In the figure shown below, fig. 8, a data labeler has a conversation
with the model and must choose a preference between two possible answers, at the bottom
highlighted in purple. In addition, the labeler is given the potential to include more notes on
the conversation or a general rating of the conversation quality (potentially spread across
multiple tasks, as seen in the top left).
This first example is a training-data only interface, where the goal is to collect rich metadata
along with the conversation. Now that these models are popular, applications often expose
interfaces for collecting preference directly to the users during everyday use, much like how
other technology products will A/B test new features in small subsets of the production
usage. It depends on the application whether this preference data is used directly to train
the future models, or if it is used just as an evaluation of models’ performance relative to

37
Figure 8: An example of one of the earliest preference data collection interface, from
Anthropic’s research. Bai et al. 2022. The actual conversation is a toy conversation around
what is a good example conversation for data collection. License CC-BY.

38
each other. An example interaction of this form is shown below in fig. 9 for an earlier version
of ChatGPT.

Figure 9: Example preference data collection interface from when I was served two completions
from different ChatGPT beta models. The actual completions are very close in content,
showing how collecting preference data can be noisy and difficult to get exactly right.

This style of interface is used extensively across the industry, such as for evaluation of
models given the same format. A popular public option to engage with models in this way is
ChatBotArena [113], which includes the option of a “tie” between models:
For models in the wild, one of the most common techniques is to collect feedback on if a
specific response was positive or negative. An example from the Ai2 playground is shown
below with thumbs up and down indicators:
In domains other than language, the same core principles apply, even though these domains
are not the focus of this book. For every Midjourney generation (and most popular image
generators) they expose multiple responses to users. These companies then use the data of
which response was selected to finetune their models with RLHF. Midjourney’s interface is
shown below:

6.2.2 Rankings vs. Ratings

The largest decision on how to collect preference data is if the data should be rankings –
i.e. relative ordering of model completions – or ratings – i.e. scores assigned to each piece of
text. Common practice is to train on rankings, but ratings are often used as metadata and /

39
Figure 10: Example preference data collection interface from an early version of the popular
LMArena benchmark.

40
Figure 11: Example preference data collection interface with up or down arrow from the
Allen Institute of AI’s research demos.

41
Figure 12: Example user interface of text-to-image models.

42
or have been explored in related literature.
One simple way to collect ratings is to score a single completion on a 1–5 scale:
• 5 — excellent: correct, clear, and notably helpful
• 4 — good: correct, clear, and useful
• 3 — okay: acceptable, but nothing special
• 2 — poor: partially correct but confusing or incomplete
• 1 — very poor: incorrect or unhelpful
With multiple completions to the same prompt, a simple way to make preference data
would be to choose the highest rated completion and pair it randomly with a lower scored
completion (as done for UltraFeedback and derivative works [23]).
Although, the most common technique for collecting preferences is to use a Likert scale for
relative rankings [114], which asks users to select which response they prefer in a group of
completions. For example, a 5 point Likert scale would look like the following (note that, yes,
a Likert scale uses a single integer to record the ranking, much like a rating, so it’s how the
data is structured that is the core difference in the two ways of collecting preference data):

Table 2: An example 5-wise Likert scale between two responses, A and B.

A>>B A>B Tie B>A B>>A

1 2 3 4 5

Some early RLHF for language modeling works uses an 8-step Likert scale with levels of
preference between the two responses [5]. An even scale removes the possibility of ties:

Table 3: An example 8-wise Likert scale between two responses, A and B.

A>>>B A>B B>A B>>>A

1 2 3 4 5 6 7 8

In this case [5], as in other works, this information is still reduced to a binary signal for the
training of a reward model.

6.2.3 Multi-turn Data

In practice, core questions often arise over how to parse and collect multi-turn data – simply
conversations with multiple related prompts. In a real-world interaction, normally a piece
of preference data is only collected on the “final” prompt, but there are scenarios where
preferences can be given on every response. When preferences are given on every response,
the conversation traditionally continues with the “chosen” answer. At training time, it is
common to include the training data for every turn of the conversation as a “single prompt,”
where the model can learn from completing it. This can effectively unroll longer conversations
into many training prompts, but needs to be done carefully to not bias the training data.
Many research questions are still emerging, such as if the person labeling the preference on the
generations should be the same as the person who creates the prompt (to avoid sycophancy),

43
and other variables that are difficult to control for in data collection (question inspired by
John Schulman). If the prompt creator cannot label the preference data, multi-turn is not
really practical due to the need for conversations to continue in real-time – sometimes for
preference data the curation of prompts is a different problem than comparing responses
(also due to the work of maintaining active endpoints for models). For training, all of the
previous turns in the conversation are masked from the loss, as discussed with instruction
finetuning.

6.2.4 Structured Preference Data

In many applications of RLHF and post-training, preference data can be created automatically
due to innate structures of the data – i.e. the domains make automatic checks of correctness
or preference possible. For example, in mathematical reasoning domains the chosen response
can be a correct answer and the rejected an incorrect answer. Another example is in precise
instruction following, such as the evaluation IFEval [115], where prompts take the form of:
Write me a short poem about a goldfish. Start each sentence with the letter
g.
The second part of this query is called the constraint, and the preference data could be
constructed with an example in the chosen column that follows the constraint. Constructing
this preference data, which would be included as synthetic preference data, involves prompting
the same model (or multiple models) twice. Once with the constraint included and once
without.
For example, prompting GPT-4.5-Preview with both of these gives the following.
Prompt: Write me a short poem about a goldfish. Start each sentence with the
letter g.
Response: Golden swimmer, circling slow, Gleaming scales in water’s glow.
Gentle fins dance softly by, Gazing out with quiet eye. Glass-bound world,
a simple sphere, Gliding calmly, free of fear. Graceful turns in endless roam,
Goldfish finds a crystal home.
And without the constraint (which, it turns out, is also a lower quality output):
Prompt: Write me a short poem about a goldfish.
Response: In circles bright, the goldfish glides, A spark of sun that gently rides
Through waters clear, in endless spins— A golden whisper, scales like fins.
Its world confined, yet vast within, Each bubble breathed, a tale begins. In silent
waves, content to roam, The goldfish dreams its bowl as home.
Crucially, the prompt for the preference data will need to include the constraint. In domains
outside of academic study, there are far more applications of preference data based on
inductive biases like above (relative to quality-based preferences, which most of this chapter
focuses on). These have been shown to enable preference finetuning to have meaningful
performance improvements across related evaluations, such as instruction-following, math,
etc. [6].

44
[Link] Alternatives There are multiple other ways to collect useful feedback data for
RLHF that have not been pursued in as great of detail. Examples include using single data
points with directional labels, e.g. as shown from Ai2 playground above in fig. 11, directly
with algorithms designed for single direction signals like Kahneman-Tversky Optimization
(KTO) [116]. Other algorithms have been proposed with different types of feedback signals
such as fine-grained feedback, e.g. at the token level [117], or natural language feedback,
e.g. by writing responses [118], to provide a richer learning signal in exchange for a more
complex data collection setup.

6.2.5 Sourcing and Contracts

Getting human preference data is an involved and costly process. The following describes
the experience of getting preference data when the field is moving quickly. Over time, these
processes will become far more automated and efficient (especially with AI feedback being
used for a larger portion of the process).
The first step is sourcing the vendor to provide data (or one’s own annotators). Much like
acquiring access to cutting-edge Nvidia GPUs, getting access to data providers in the peak of
AI excitement is also a who-you-know game – those who can provide data are supply-limited.
If you have credibility in the AI ecosystem, the best data companies will want you on their
books for public image and long-term growth options. Discounts are often also given on the
first batches of data to get training teams hooked.
If you’re a new entrant in the space, you may have a hard time getting the data you need
quickly. Data vendors are known to prioritize large budget line-items and new customers
that have an influential brand or potential for large future revenue. This is, in many business
ways, natural, as the data foundry companies are often supply-limited in their ability to
organize humans for effective data labelling.
On multiple occasions, I’ve heard of data companies not delivering their data as contracted
without the customer threatening legal or financial action against them for breach of contract.
Others have listed companies I work with as customers for PR even though we never worked
with them, saying they “didn’t know how that happened” when reaching out. There are
plenty of potential bureaucratic or administrative snags through the process. For example, the
default terms on the contracts often prohibit the open sourcing of artifacts after acquisition
in some fine print.
Once a contract is settled, the data buyer and data provider agree upon instructions for
the task(s) purchased. There are intricate documents with extensive details, corner cases,
and priorities for the data. A popular example of data instructions is the one that OpenAI
released for InstructGPT [3].
Depending on the domains of interest in the data, timelines for when the data can be labeled
or curated vary. High-demand areas like mathematical reasoning or coding must be locked
into a schedule weeks out. In the case when you are collecting a dataset for your next model
and you realize that collecting data later may be optimal, simple delays of data collection
don’t always work — Scale AI et al. are managing their workforces like AI research labs
manage the compute-intensive jobs on their clusters (planning multiple weeks or months
ahead as to when different resources will be allocated where).
Once everything is agreed upon, the actual collection process is a high-stakes time for

45
post-training teams. All the training infrastructure, evaluation tools, and plans for how to
use the data and make downstream decisions must be in place. If the data cannot be easily
slotted into an existing RLHF data pipeline, it’ll take a long time to have the information
the data partner wants in order to try and improve the collection process during the process.
Collecting data that cannot be seamlessly integrated into training pipelines often becomes
stale and a waste of resources.
The data is delivered in weekly batches with more data coming later in the contract.
For example, when we bought preference data for on-policy models we were training at
HuggingFace, we had a 6 week delivery period. The first weeks were for further calibration
and the later weeks were when we hoped to most improve our model.

Figure 13: Overview of the multi-batch cycle for obtaining human preference data from a
vendor. The ramp up period allows a narrowing of goals and methodology in order to create
the best possible data. It is expected that a larger proportion of the data from the earlier
batches will have to be thrown out due to quality issues. This is one timeline example for a
smaller data contract (~$500K) and much larger data contracts can vary substantially.

The goal is that by week 4 or 5 we can see the data improving our model. This is something
some frontier models have mentioned, such as the 14 stages in the Llama 2 data collection
[44], but it doesn’t always go well. At HuggingFace, trying to do this for the first time with
human preferences, we didn’t have the RLHF preparedness to get meaningful bumps on our
evaluations. The last weeks came and we were forced to continue to collect preference data
generating from endpoints we weren’t confident in.
After the data is all in, there is plenty of time for learning and improving the model. Data
acquisition through these vendors works best when viewed as an ongoing process of achieving
a set goal. It requires iterative experimentation, high effort, and focus. It’s likely that

46
millions of dollars spent on these datasets are “wasted” and not used in the final models,
but that is just the cost of doing business. Not many organizations have the bandwidth and
expertise to make full use of human data of this style.
This experience, especially relative to the simplicity of synthetic data, makes me wonder
how well these companies will be doing in the next decade.
Note that this section does not mirror the experience for buying human-written instruction
data, where the process is less of a time crunch. Early post-training processes were built
around the first stage of training being heavily driven by carefully crafted, human answers
to a set of prompts. This stage of data is not subject to the on-policy restrictions for
multiple reasons: Instruction data is used directly ontop of a base model, so on-policy
doesn’t really apply; the loss-function for instruction finetuning doesn’t need the contrastive
data of preference finetuning; and other structural advantages. Today, the primary other
focus of human data is in generating prompts for post-training – which dictate the training
distribution of topics for the model – or on challenging tasks at the frontier of model
performance. More of these data trade-offs are discussed in Chapter 16 on Synthetic Data.

6.3 Bias: Things to Watch Out For in Data Collection

While preference data is essential, it’s also known to be prone to many subtle biases that
can make its collection error-prone. These biases are so common, e.g. prefix bias (where
the beginning of a completion disproportionately drives the preference) [119], that they can
easily be passed to the final model [120] (and especially as we know that models are only as
good as their data). These issues are often subtle and vary in how applicable interventions
to mitigate them are. For many, such as sycophancy (over-agreeing with the user’s stated
beliefs or flattering them, even when it reduces truthfulness) [121], they reflect issues within
humans that are often outside of the labeling criteria that one will think of providing to
the annotation partner or labelers. Others, such as verbosity [9] [122] or formatting habits
[123], emerge for a similar reason, but they are easier to detect and mitigate in training.
Mitigating these subtle biases in data is the difference between good or great preference data,
and therefore good or great RLHF training.

6.4 Open Questions in RLHF Preference Data

The data used to enable RLHF is often curated by multiple stakeholders in a combination
of paid employment and consumer usage. This data, representing a preference between
two pieces of text in an individual instance, is capturing a broad and diverse function
via extremely limited interactions. Given that the data is sparse in count relative to the
complexity it begins to represent, more questions should be openly shared about its curation
and impacts.
Currently, datasets for the most popular LLMs are being generated by professional workforces.
This opens up many questions around who is creating the data and how the context of their
workplace informs it.
Despite the maturity of RLHF as a core method across the field, there are still many core open
questions facing how best to align its practice with its motivations. Some are enumerated
below:

47
• Data collection contexts: Can data involving preferences collected in a professional
setting mirror the intent of researchers designing an experiment or provide suitable
transfer to downstream users? How does this compare to volunteer workers? How does
context inform preferences, how does this data impact a downstream model, how can
the impact of a user interface be measured in data? How does repetitive labeling of
preference data shift one’s preferences? Do professional crowd-workers, instructed to
follow a set of preferences, follow the instructions or their innate values?
• Type of feedback: Does the default operating method of RLHF, pairwise preferences
capture preferences in its intended form? Can comparisons in RLHF across the
same data be made with the default comparisons versus advanced multi-axis feedback
mechanisms ?? What types of comparisons would reflect how humans communicate
preferences in text?
• Population demographics: Who is completing the data? Is a diverse population
maintained? How does a lack of diversity emerge as measurable impacts on the model?
What is a minimum number of people required to suitably represent a given population?
How are instances of preference annotator disagreement treated – as a source of noise,
or a signal?
• Are the Preferences Expressed in the Models? In the maturation of RLHF
and related approaches, the motivation of them – to align models to abstract notions
of human preference – has drifted from the practical use – to make the models more
effective to users. A feedback loop that is not measurable due to the closed nature of
industrial RLHF work is the check to see if the behavior of the models matches the
specification given to the data annotators during the process of data collection. We
have limited tools to audit this, such as the Model Spec from OpenAI [124] that details
what they want their models to do, but we don’t know exactly how this translates to
data collection.

48
7 Reward Modeling
Reward models are core to the modern approach to RLHF by being where the complex
human preferences are learned. They are what enable our models to learn from hard to
specify signals. They compress complex features in the data into a representation that
can be used in downstream training – a sort of magic that once again shows the complex
capacity of modern deep learning. These models act as the proxy objectives by which the
core optimization is done, as studied in the following chapters.
Reward models broadly have historically been used extensively in reinforcement learning
research as a proxy for environment rewards [55]. Reward models were proposed, in their
modern form, as a tool for studying the value alignment problem [33]. These models tend to
take in some sort of input and output a single scalar value of reward. This reward can take
multiple forms – in traditional RL problems it was attempting to approximate the exact
environment reward for the problem, but we will see in RLHF that reward models actually
output a probability of a certain input being “of high quality” (i.e. the chosen answer among
a pairwise preference relation). The practice of reward modeling for RLHF is closely related
to inverse reinforcement learning, where the problem is to approximate an agent’s reward
function given trajectories of behavior [96], and other areas of deep reinforcement learning.
The high level problem statement is the same, but the implementation and focus areas are
entirely different, so they’re often considered as totally separate areas of study.
The most common reward model, often called a Bradley-Terry reward model and the primary
focus of this chapter, predicts the probability that a piece of text was close to a “preferred”
piece of text from the training comparisons. Later in this section we also compare these
to Outcome Reward Models (ORMs), Process Reward Model (PRM), and other types of
reward models.
Throughout this chapter, we use x to denote prompts and y to denote completions. This
notation is common in the language model literature, where methods operate on full prompt-
completion pairs rather than individual tokens.

7.1 Training Reward Models

The canonical implementation of a reward model is derived from the Bradley-Terry model
of preference [125]. There are two popular expressions for how to train a standard reward
model for RLHF – they are mathematically equivalent. To start, a Bradley-Terry model of
preferences defines the probability that, in a pairwise comparison between two items i and j,
a judge prefers i over j:

pi
P (i > j) = . (11)
pi + pj

The Bradley-Terry model assumes that each item has a latent strength pi > 0, and that
observed preferences are a noisy reflection of these underlying strengths. It is common to
reparametrize the Bradley-Terry model with unbounded scores, where pi = eri , which results
in the following form:

e ri
P (i > j) = = σ(ri − rj ). (12)
e ri + e r j

49
Only differences in scores matter: adding the same constant to all ri leaves P (i > j)
unchanged. These forms are not a law of nature, but a useful approximation of human
preferences that often works well in RLHF.
To train a reward model, we must formulate a loss function that satisfies the above relation.
In practice, this is done by converting a language model into a model that outputs a scalar
score, often via a small linear head that produces a single logit. Given a prompt x and
two sampled completions y1 and y2 , we score both with a reward model rθ and write the
conditional scores as rθ (yi | x).
The probability of success for a given reward model in a pairwise comparison becomes:

exp (rθ (y1 | x))

P (y1 > y2 | x) = . (13)
exp (rθ (y1 | x)) + exp (rθ (y2 | x))

We denote the preferred completion as yc (chosen) and the rejected completion as yr .

Then, by maximizing the log-likelihood of the above function (or alternatively minimizing
the negative log-likelihood), we can arrive at the loss function to train a reward model:

exp (rθ (yc | x))

θ∗ = arg max P (yc > yr | x) = arg max
θ θ exp (rθ (yc | x)) + exp (rθ (yr | x))
exp (rθ (yc | x))
= arg max  
θ (yr |x))
θ
exp (rθ (yc | x)) 1 + exp(r
exp(rθ (yc |x))
1
= arg max exp(rθ (yr |x)) (14)
θ 1+ exp(rθ (yc |x))
1
= arg max
θ 1 + exp (−(rθ (yc | x) − rθ (yr | x)))
= arg max σ (rθ (yc | x) − rθ (yr | x))
θ
= arg min − log (σ (rθ (yc | x) − rθ (yr | x)))
θ

The first form, as in [3] and other works:

L(θ) = − log (σ (rθ (yc | x) − rθ (yr | x))) (15)

Second, as in [18] and other works:

 
L(θ) = log 1 + erθ (yr |x)−rθ (yc |x) (16)

These are equivalent by letting ∆ = rθ (yc | x) − rθ (yr | x) and using σ(∆) = 1+e1−∆ , which
implies − log σ(∆) = log(1 + e−∆ ) = log 1 + erθ (yr |x)−rθ (yc |x) . They both appear in the

RLHF literature.

50
7.2 Architecture
The most common way reward models are implemented is through an abstraction similar to
Transformer’s AutoModelForSequenceClassification, which appends a small linear head to
the language model that performs classification between two outcomes – chosen and rejected.
At inference time, the model outputs the probability that the piece of text is chosen as a
single logit from the model.
Other implementation options exist, such as just taking a linear layer directly from the final
embeddings, but they are less common in open tooling.

7.3 Implementation Example

Implementing the reward modeling loss is quite simple. More of the implementation challenge
is on setting up a separate data loader and inference pipeline. Given the correct dataloader
with tokenized, chosen and rejected prompts with completions, the loss is implemented as:
import torch . nn as nn
# inputs_chosen / inputs_rejected include the prompt tokens x and the
respective
# completion tokens ( y_c or y_r ) that the reward model scores jointly .
rewards_chosen = model (** inputs_chosen )
rewards_rejected = model (** inputs_rejected )

loss = - nn . functional . logsigmoid ( rewards_chosen - rewards_rejected ) .

mean ()

As for the bigger picture, this is often within a causal language model that has an additional
head added (and learned with the above loss) that transitions from the final hidden state to
the score of the inputs. This model will have a structure as follows:
import torch
import torch . nn as nn
import torch . nn . functional as F

class B ra d le y Te rr y Re w ar dM o de l ( nn . Module ) :
"""
␣ ␣ ␣ ␣ Standard ␣ scalar ␣ reward ␣ model ␣ for ␣ Bradley - Terry ␣ preference ␣ learning
.

␣ ␣ ␣ ␣ Usage ␣ ( pairwise ␣ BT ␣ loss ) :

␣ ␣ ␣ ␣ ␣ ␣ ␣ ␣ rewards_chosen ␣ = ␣ model (** inputs_chosen ) ␣ ␣ ␣ ␣ # ␣ ( batch ,)
␣ ␣ ␣ ␣ ␣ ␣ ␣ ␣ rewards_rejected ␣ = ␣ model (** inputs_rejected ) ␣ ␣ # ␣ ( batch ,)
␣ ␣ ␣ ␣ ␣ ␣ ␣ ␣ loss ␣ = ␣ -F . logsigmoid ( rewards_chosen ␣ -␣ rewards_rejected ) . mean ()
␣␣␣␣"""
def __init__ ( self , base_lm ) :
super () . __init__ ()
self . lm = base_lm # e . g . , AutoModelForCausalLM
self . head = nn . Linear ( self . lm . config . hidden_size , 1)

def _sequence_rep ( self , hidden , attention_mask ) :

"""
␣ ␣ ␣ ␣ ␣ ␣ ␣ ␣ Get ␣ a ␣ single ␣ vector ␣ per ␣ sequence ␣ to ␣ score .

51
␣ ␣ ␣ ␣ ␣ ␣ ␣ ␣ Default : ␣ last ␣ non - padding ␣ token ␣ ( EOS ␣ token ) ; ␣ if ␣ no ␣ mask , ␣ last ␣
token .
␣ ␣ ␣ ␣ ␣ ␣ ␣ ␣ hidden : ␣ ( batch , ␣ seq_len , ␣ hidden_size )
␣ ␣ ␣ ␣ ␣ ␣ ␣ ␣ attention_mask : ␣ ( batch , ␣ seq_len )
␣␣␣␣␣␣␣␣"""

# Index of last non - pad token in each sequence

# attention_mask is 1 for real tokens , 0 for padding
lengths = attention_mask . sum ( dim =1) - 1 # ( batch ,)
batch_idx = torch . arange ( hidden . size (0) , device = hidden . device )
return hidden [ batch_idx , lengths ] # ( batch , hidden_size )

def forward ( self , input_ids , attention_mask ) :

"""
␣ ␣ ␣ ␣ ␣ ␣ ␣ ␣ A ␣ forward ␣ pass ␣ designed ␣ to ␣ show ␣ inference ␣ structure ␣ of ␣ a ␣
standard ␣ reward ␣ model .
␣ ␣ ␣ ␣ ␣ ␣ ␣ ␣ To ␣ train ␣ one , ␣ this ␣ function ␣ will ␣ need ␣ to ␣ be ␣ modified ␣ to ␣
compute ␣ rewards ␣ from ␣ both
␣ ␣ ␣ ␣ ␣ ␣ ␣ ␣ ␣ chosen ␣ and ␣ rejected ␣ inputs , ␣ applying ␣ the ␣ loss ␣ above .
␣␣␣␣␣␣␣␣"""
outputs = self . lm (
input_ids = input_ids ,
attention_mask = attention_mask ,
output_hidden_states = True ,
return_dict = True ,
)
# Final hidden states : ( batch , seq_len , hidden_size )
hidden = outputs . hidden_states [ -1]

# One scalar reward per sequence : ( batch ,)

seq_repr = self . _sequence_rep ( hidden , attention_mask )
rewards = self . head ( seq_repr ) . squeeze ( -1)

return rewards

In this section and what follows, most of the implementation complexity for reward models
(and much of post-training) is around constructing the data-loaders correctly and distributed
learning systems. Note, when training reward models, the most common practice is to train
for only 1 epoch to avoid overfitting.

7.4 Variants
Reward modeling is a relatively under-explored area of RLHF. The traditional reward
modeling loss has been modified in many popular works, but the modifications have not
solidified into a single best practice.

7.4.1 Preference Margin Loss

In the case where annotators are providing either scores or rankings on a Likert Scale, the
magnitude of the relational quantities can be used in training. The most common practice
is to binarize the data along the preference direction, reducing the mixed information of

52
relative ratings or the strength of the ranking to just chosen and rejected completions. The
additional information, such as the magnitude of the preference, has been used to improve
model training, but it has not converged as a standard practice. Llama 2 proposes using the
margin between two datapoints, m(yc , yr ), to distinguish the magnitude of preference:

L(θ) = − log (σ (rθ (yc | x) − rθ (yr | x) − m(yc , yr ))) (17)

For example, each completion is often given a ranking from 1 to 5 in terms of quality. In the
case where the chosen sample was assigned a score of 5 and rejected a score of 2, the margin
m(yc , yr ) = 5 − 2 = 3. Other functions for computing margins can be explored.
Note that in Llama 3 the margin term was removed as the team observed diminishing
improvements after scaling.

7.4.2 Balancing Multiple Comparisons Per Prompt

InstructGPT studies the impact of using a variable number of completions per prompt, yet
balancing them in the reward model training [3]. To do this, they weight the loss updates
per comparison per prompt. At an implementation level, this can be done automatically by
including all examples with the same prompt in the same training batch, naturally weighing
the different pairs – not doing this caused overfitting to the prompts. The loss function
becomes:

1
L(θ) = − K

E(x,yc ,yr )∼D log (σ (rθ (yc | x) − rθ (yr | x))) (18)
2

7.4.3 K-wise Loss Function

There are many other formulations that can create suitable models of human preferences for
RLHF. One such example, used in the popular, early RLHF’d models Starling 7B and 34B
[126], is a K-wise loss function based on the Plackett-Luce model [127].
Zhu et al. 2023 [128] formalizes the setup as follows. With a prompt, or state, si , K actions
(ai0 , ai1 , · · · , aiK−1 ) are sampled from P (a0 , · · · , aK−1 |si ). Then, labelers are used to rank
preferences with σ i : [K] 7→ [K] is a function representing action rankings, where σ i (0) is
the most preferred action. This yields a preference model capturing the following:

i i
K−1
Y exp(rθ⋆ (si , aiσi (k) ))
P (σ |s , ai0 , ai1 , . . . , aiK−1 ) = PK−1 (19)
k=0 j=k exp(rθ⋆ (si , aiσi (j) ))

When K = 2, this reduces to the Bradley-Terry (BT) model for pairwise comparisons.
Regardless, once trained, these models are used similarly to other reward models during
RLHF training.

7.5 Outcome Reward Models

The majority of preference tuning for language models and other AI systems is done with the
Bradley Terry models discussed above. For reasoning heavy tasks, one can use an Outcome

53
Reward Model (ORM). The training data for an ORM is constructed in a similar manner
to standard preference tuning. Here, we have a problem statement or prompt, x and two
completions y1 and y2 . The inductive bias used here is that one completion should be a
correct solution to the problem and one incorrect, resulting in (yc , yic ).
The shape of the models used is very similar to a standard reward model, with a linear layer
appended to a model that can output a single logit (in the case of an RM) – with an ORM,
the training objective that follows is slightly different [129]:
[We] train verifiers with a joint objective where the model learns to label a model
completion as correct or incorrect, in addition to the original language modeling
objective. Architecturally, this means our verifiers are language models, with a
small scalar head that outputs predictions on a per-token basis. We implement
this scalar head as a single bias parameter and single gain parameter that operate
on the logits outputted by the language model’s final unembedding layer.
To translate, this is implemented as a language modeling head that can predict two classes
per token (1 for correct, 0 for incorrect), rather than a classification head of a traditional RM
that outputs one logit for the entire sequence. Formally, following [130] this can be shown as:

LCE (θ) = −E(s,r)∼D [r log pθ (s) + (1 − r) log(1 − pθ (s))] (20)

where r ∈ 0, 1 is a binary label where 1 applies to a correct answer to a given prompt and
0 applies to an incorrect, and pθ (s) is the scalar proportional to predicted probability of
correctness from the model being trained.
Implementing an outcome reward model (and other types, as we’ll see with the Process
Reward Model) involves applying the cross-entropy loss per-token based on if the completion
is a correct sample. This is far closer to the language modeling loss, where it does not need
the structured chosen-rejected nature of standard Bradley-Terry reward models.
The model structure could follow as:
import torch . nn as nn
import torch . nn . functional as F

class OutcomeRewardModel ( nn . Module ) :

def __init__ ( self , base_lm ) :
super () . __init__ ()
self . lm = base_lm # e . g . , AutoModelForCausalLM
self . head = nn . Linear ( self . lm . config . hidden_size , 1)

def forward ( self , input_ids , attention_mask = None , labels = None ) :

"""
␣ ␣ ␣ ␣ ␣ ␣ ␣ ␣ The ␣ input ␣ data ␣ here ␣ will ␣ be ␣ tokenized ␣ prompts ␣ and ␣ completions ␣
along ␣ with ␣ labels
␣ ␣ ␣ ␣ ␣ ␣ ␣ ␣ ␣ per ␣ prompt ␣ for ␣ correctness .
␣␣␣␣␣␣␣␣"""
outputs = self . lm (
input_ids = input_ids ,
attention_mask = attention_mask ,
output_hidden_states = True ,

54
 return_dict = True ,
)
# Final hidden states : ( batch , seq_len , hidden_size )
hidden = outputs . hidden_states [ -1]
# One scalar logit per token : ( batch , seq_len )
logits = self . head ( hidden ) . squeeze ( -1)

# Only compute loss on completion tokens ( labels 0 or 1)

# Prompt tokens have labels = -100
mask = labels != -100
if mask . any () :
loss = F . b i n a r y _ c r o s s _ e n t r o p y _ w i t h _ l o g i t s (
logits [ mask ] , labels [ mask ]. float ()
)
return loss , logits

A simplified version of the loss follows:

# Assume model already has : model . lm ( backbone ) + model . head
hidden = model . lm (** inputs , output_hidden_states = True ) . hidden_states
[ -1]
logits_per_token = model . head ( hidden ) . squeeze ( -1) # ( batch , seq_len )
# This will sometimes be compressed as model . forward () in other
implementations

# Binary labels : 1= correct , 0= incorrect ( prompt tokens masked as -100)

mask = labels != -100
loss = F . b i n a r y _ c r o s s _ e n t r o p y _ w i t h _ l o g i t s (
logits_per_token [ mask ] , labels [ mask ]. float ()
)

The important intuition here is that an ORM will output a probability of correctness at
every token in the sequence. This can be a noisy process, as the updates and loss propagates
per token depending on outcomes and attention mappings.
These models have continued in use, but are less supported in open-source RLHF tools. For
example, the same type of ORM was used in the seminal work Let’s Verify Step by Step
[45], but without the language modeling prediction piece of the loss. Then, the final loss is a
cross-entropy loss on every token predicting if the final answer is correct.
Given the lack of support, the term outcome reward model (ORM) has been used in multiple
ways. Some literature, e.g. [130], continues to use the original definition from Cobbe et
al. 2021. Others do not.

7.6 Process Reward Models

Process Reward Models (PRMs), originally called Process-supervised Reward Models, are
reward models trained to output scores at every step in a chain-of-thought reasoning process.
These differ from a standard RM that outputs a score only at an EOS token or a ORM that
outputs a score at every token. Process Reward Models require supervision at the end of
each reasoning step, and then are trained similarly where the tokens in the step are trained
to their relevant target – the target is the step in PRMs and the entire response for ORMs.

55
Following [45], a binary-labeled PRM is commonly optimized with a per-step cross-entropy
loss:

" K
#
X
LPRM (θ) = −E(x,s)∼D ysi log rθ (si | x) + (1 − ysi ) log (1 − rθ (si | x)) (21)
i=1

where s is a sampled chain-of-thought with K annotated steps, ysi ∈ {0, 1} denotes whether
the i-th step is correct, and rθ (si | x) is the PRM’s predicted probability that step si is valid
conditioned on the original prompt x.
Here’s an example of how this per-step label can be packaged in a trainer, from HuggingFace’s
TRL (Transformer Reinforcement Learning) [42]:
# Get the ID of the separator token and add it to the completions
separator_ids = tokenizer . encode ( step_separator , add_special_tokens =
False )
completions_ids = [ completion + separator_ids for completion in
completions_ids ]

# Create the label

labels = [[ -100] * ( len ( completion ) - 1) + [ label ] for completion ,
label in zip ( completions_ids , labels ) ]

Traditionally PRMs are trained with a language modeling head that outputs a token only at
the end of a reasoning step, e.g. at the token corresponding to a double new line or other
special token. These predictions tend to be -1 for incorrect, 0 for neutral, and 1 for correct.
These labels do not necessarily tie with whether or not the model is on the right path, but if
the step is correct.
An example construction of a PRM is shown below.
import torch . nn as nn
import torch . nn . functional as F

class ProcessRewardModel ( nn . Module ) :

def __init__ ( self , base_lm , num_classes =3) :
super () . __init__ ()
self . lm = base_lm # e . g . , AutoModelForCausalLM
self . head = nn . Linear ( self . lm . config . hidden_size , num_classes )

def forward ( self , input_ids , attention_mask = None , labels = None ) :

"""
␣ ␣ ␣ ␣ ␣ ␣ ␣ ␣ The ␣ inputs ␣ are ␣ tokenizer ␣ prompts ␣ and ␣ completions , ␣ where ␣ the ␣
the ␣ end ␣ of ␣ a
␣ ␣ ␣ ␣ ␣ ␣ ␣ ␣ ␣ " reasoning step " ␣ is ␣ denoted ␣ by ␣ another ␣ non - padding ␣ token .
␣ ␣ ␣ ␣ ␣ ␣ ␣ ␣ labels ␣ will ␣ be ␣ a ␣ list ␣ of ␣ labels , ␣ True , ␣ False , ␣ and ␣ Neutral ␣ (3 ␣
labels ) ␣ which
␣ ␣ ␣ ␣ ␣ ␣ ␣ ␣ ␣ will ␣ be ␣ predicted ␣ by ␣ the ␣ model .
␣␣␣␣␣␣␣␣"""
outputs = self . lm (
input_ids = input_ids ,
attention_mask = attention_mask ,

56
 output_hidden_states = True ,
return_dict = True ,
)
# Final hidden states : ( batch , seq_len , hidden_size )
hidden = outputs . hidden_states [ -1]
# One logit vector per token : ( batch , seq_len , num_classes )
logits = self . head ( hidden )

# Only compute loss at step boundaries ( where labels != -100)

# Labels map : -1 -> 0 , 0 -> 1 , 1 -> 2 ( class indices )
mask = labels != -100
if mask . any () :
loss = F . cross_entropy (
logits [ mask ] , labels [ mask ]
)
return loss , logits

The core loss function looks very similar to outcome reward models, with the labels being
applied at different intervals.
# Assume model outputs 3 - class logits per token
hidden = model . lm (** inputs , output_hidden_states = True ) . hidden_states
[ -1]
logits = model . head ( hidden ) # ( batch , seq_len , 3)

# 3 - class labels at step boundaries only : 0= -1 , 1=0 , 2=1 ( others

masked as -100)
mask = labels != -100
loss = F . cross_entropy ( logits [ mask ] , labels [ mask ])

7.7 Reward Models vs. Outcome RMs vs. Process RMs vs. Value
Functions
The various types of reward models covered indicate the spectrum of ways that “quality”
can be measured in RLHF and other post-training methods. Below, a summary of what the
models predict and how they are trained.

Table 4: Comparing types of reward models.

Model Class What They Predict How They Are Trained LM structure
Reward Quality of text via Contrastive loss between Regression or
Models probability of chosen pairwise (or N-wise) classification head
response at EOS token comparisons between on top of LM
completions features

57
Model Class What They Predict How They Are Trained LM structure
Outcome Probability that an Labeled outcome pairs Language
Reward answer is correct (e.g., success/failure on modeling head
Models per-token verifiable domains) per-token
cross-entropy,
where every label
is the outcome
level label
Process A reward or score for Trained using intermediate Language
Reward intermediate steps at feedback or stepwise modeling head
Models end of reasoning steps annotations (trained per only running
token in reasoning step) inference per
reasoning step,
predicts three
classes -1, 0, 1
Value The expected return Trained via regression to A classification
Functions given the current state each point in sequence with output
per-token

Some notes, given the above table has a lot of edge cases.
• Both in preference tuning and reasoning training, the value functions often have a
discount factor of 1, which makes a value function even closer to an outcome reward
model, but with a different training loss.
• A process reward model can be supervised by doing rollouts from an intermediate
state and collecting outcome data. This blends multiple ideas, but if the loss is per
reasoning step labels, it is best referred to as a PRM.

7.8 Generative Reward Modeling

With the cost of preference data, a large research area emerged to use existing language
models as a judge of human preferences or in other evaluation settings [131]. The core
idea is to prompt a language model with instructions on how to judge, a prompt, and two
completions (much as would be done with human labelers). An example prompt, from one
of the seminal works here for the chat evaluation MT-Bench [131], follows:
[ System ]
Please act as an impartial judge and evaluate the quality of the
responses provided by two AI assistants to the user question
displayed below .
You should choose the assistant that follows the user ' s instructions
and answers the user ' s question better .
Your evaluation should consider factors such as the helpfulness ,
relevance , accuracy , depth , creativity , and level of detail of
their responses .
Begin your evaluation by comparing the two responses and provide a
short explanation .
Avoid any position biases and ensure that the order in which the
responses were presented does not influence your decision .

58
Do not allow the length of the responses to influence your evaluation .
Do not favor certain names of the assistants .
Be as objective as possible .
After providing your explanation , output your final verdict by
strictly following this format : "[[ A ]]" if assistant A is better ,
"[[ B ]]" if assistant B is better , and "[[ C ]]" for a tie .
[ User Question ]
{ question }
[ The Start of Assistant A ' s Answer ]
{ answer_a }
[ The End of Assistant A ' s Answer ]
[ The Start of Assistant B ' s Answer ]
{ answer_b }
[ The End of Assistant B ' s Answer ]

Given the efficacy of LLM-as-a-judge for evaluation, spawning many other evaluations such as
AlpacaEval [132], Arena-Hard [133], and WildBench [134], many began using LLM-as-a-judge
instead of reward models to create and use preference data.
An entire field of study has emerged to study how to use so called “Generative Reward
Models” [135] [136] [137] (including models trained specifically to be effective judges [138]),
but on RM evaluations they tend to be behind existing reward models, showing that reward
modeling is an important technique for current RLHF.
A common trick to improve the robustness of LLM-as-a-judge workflows is to use a sampling
temperature of 0 to reduce variance of ratings.

7.9 Further Reading

The academic literature for reward modeling established itself in 2024. The bulk of progress
in reward modeling early on has been in establishing benchmarks and identifying behavior
modes. The first RM benchmark, RewardBench, provided common infrastructure for testing
reward models [139]. Since then, RM evaluation has expanded to be similar to the types
of evaluations available to general post-trained models, where some evaluations test the
accuracy of prediction on domains with known true answers [139] or those more similar to
“vibes” performed with LLM-as-a-judge or correlations to other benchmarks [140].
Examples of new benchmarks include:
• Text-only (general chat / preferences): RMB [141], RewardBench2 [112], Prefer-
ence Proxy Evaluations [142], or RM-Bench [143].
• Specialized text-only (math, etc.): multilingual reward bench (M-RewardBench)
[144], RAG-RewardBench for retrieval augmented generation (RAG) [145], ReWord-
Bench for typos [146], RewardMATH [147], or AceMath-RewardBench [148].
• Process RMs: PRM Bench [149] or ProcessBench [150] and visual benchmarks of
VisualProcessBench [151] or ViLBench [152].
• Agentic RMs: Agent-RewardBench [153] or CUARewardBench [154].
• Multimodal: MJ-Bench [155], Multimodal RewardBench [156], VL RewardBench
[157], or VLRMBench [158].
To understand progress on training reward models, one can reference new reward model
training methods, with aspect-conditioned models [159], high-quality human datasets [160]

59
[111], scaling experiments [25], extensive experimentation [44], or debiasing data [161].

60
8 Regularization
Throughout the RLHF optimization, many regularization steps are used to prevent over-
optimization of the reward model. Over-optimization in these contexts looks like models that
output nonsensical text. Some examples of optimization “off the rails” are that models can
output followable math reasoning with extremely incorrect answers, repeated text, switching
languages, or excessive special characters. This chapter covers the different methods that’re
used to control the optimization of models.
The most popular variant, used in most RLHF implementations at the time of writing, is a
KL distance from the current policy to a reference policy across generated samples. “KL
distance” is a colloquial term for expressing the optimization distance within the training
process, even though KL divergence—the underlying mathematical method for measuring the
separation of two probability distributions—does not satisfy the formal properties required
to be a true distance metric (it is simply easier to call the number a distance than a numeric
measure of distributional difference). Many other regularization techniques have emerged in
the literature to then disappear in the next model iteration in that line of research. That is
to say that regularization outside the core KL distance from generations is often used to
stabilize experimental setups that can then be simplified in the next generation. Still, it is
important to understand tools to constrain optimization in RLHF.
Throughout this chapter, we use x to denote prompts and y to denote completions. This
notation is common in the language model literature, where methods operate on full prompt-
completion pairs rather than individual tokens.
The general formulation, when used in an RLHF framework with a reward model, rθ is as
follows:

r = rθ − λrreg. (22)

With the reference implementation being:

r = rθ − λKL DKL (πRL (y | x) ∥ πref (y | x)) (23)

8.1 KL Divergences in RL Optimization

For mathematical definitions, see Chapter 3 on Problem Setup. Recall that a KL divergence
measure of probability difference is defined as follows:
 
X P (x)
DKL (P ||Q) = P (x) log (24)
Q(x)
x∈X

In RLHF, the two distributions of interest are often the distribution of the new model
version, say P (x), and a distribution of the reference policy, say Q(x). Different optimizers
use different KL directions. Throughout this book, the most common “KL Penalty” that is
used is called the reverse KL to the reference policy. In practice, this reduces to a Monte
Carlo estimate that samples tokens from the RL model and computes probabilities from the
reference model. Intuitively, this reverse KL has a numerical property that applies a large

61
penalty when the new model, P or πRL , puts substantial probability mass where the original
reference model assigns low probability.
The other KL direction is still often used in ML, e.g. in the internal trust region calculation
of some RL algorithms. This penalty intuitively penalizes the new model when its update
does not apply probability to a high-likelihood region in Q or πref . This is closer to an
objective used for distillation or behavioral cloning.

8.1.1 Reference Model to Generations

KL penalties are most commonly implemented by comparing the distance between the
generated tokens during training to a static reference model. The intuition is that the model
you’re training from has a style that you would like to stay close to. This reference model is
most often the instruction tuned model, but can also be a previous RL checkpoint. With
simple substitution, the model we are sampling from becomes πRL (x) and πref (x), shown
above in eq. 23 (often P , and Q, in standard definitions, when applied for RL KL penalties).
Such a KL divergence penalty was first applied to dialogue agents well before the popularity
of large language models [162], yet KL control was quickly established as a core technique
for fine-tuning pretrained models [163].

8.1.2 Implementation Example

In practice, the implementation of KL divergence is often approximated [164], making
the implementation far simpler. With the above definition, the summation of KL can be
converted to an expectation when sampling directly from the distribution P (x). In this case,
the distribution P (x) is the generative distribution of the model currently being trained
(i.e. not the reference model). Then, the computation for KL divergence changes to the
following:

DKL (P || Q) = Ex∼P [log P (x) − log Q(x)] . (25)

This mode is far simpler to implement, particularly when dealing directly with log probabilities
used frequently in language model training.
# Step 1: sample ( or otherwise generate ) a sequence from your policy
generated_tokens = model . generate ( inputs )

# Step 2: score that generated sequence under both models

# for autoregressive LMs , you usually do :
# inputs_for_scoring = generated_tokens [: , : -1]
# labels = generated_tokens [: , 1:]
logits = model . forward ( generated_tokens [: , : -1]) . logits
ref_logits = ref_model . forward ( generated_tokens [: , : -1]) . logits

# convert to log - probs , then align labels to index into the logits
logprobs = F . log_softmax ( logits , dim = -1)
ref_logprobs = F . log_softmax ( ref_logits , dim = -1)

# gather the log - probs of the actual next tokens

token_logprobs = logprobs . gather ( -1 , generated_tokens [: , 1:].
unsqueeze ( -1) ) . squeeze ( -1)

62
re f_token_logprobs = ref_logprobs . gather ( -1 , generated_tokens [: , 1:].
unsqueeze ( -1) ) . squeeze ( -1)

# now you can sum ( or average ) those to get the sequence log - prob ,
# and compute KL :
seq_logprob = token_logprobs . sum ( dim = -1)
ref_seq_logprob = ref_token_logprobs . sum ( dim = -1)

kl_approx = seq_logprob - ref_seq_logprob

kl_full = F . kl_div ( ref_logprobs , logprobs , reduction = ' batchmean ')

Some example implementations include TRL and Hamish Ivison’s Jax Code.

8.2 Pretraining Gradients

Another way of viewing regularization is that you may have a dataset that you want the
model to remain close to, as done in InstructGPT [3] “in order to fix the performance
regressions on public NLP datasets”. To implement this, they modify the training objective
for RLHF. Taking eq. 22, we can transform this into an objective function to optimize by
sampling from the RL policy model, completions y from prompts x in the RL dataset used
for RLHF, which yields:

J(θ) = E(x,y)∼DπRL,θ [rθ (y | x) − λrreg. ] (26)

Then, we can add an additional reward for higher probabilities on the standard autoregressive
next-token prediction loss used at pretraining, over a set of documents sampled from the
pretraining corpus (or another dataset) to maintain textual coherence:

J(θ) = E(x,y)∼DπRL,θ [rθ (y | x) − λrreg. ] + γEx∼Dpretrain [log(πRL,θ (x))] (27)

Recent work proposed using a negative log-likelihood term to balance the optimization of
Direct Preference Optimization (DPO) [165]. Given the pairwise nature of the DPO loss,
the same loss modification can be made to reward model training, constraining the model to
predict accurate text (rumors from laboratories that did not publish the work).
The optimization follows as a modification to DPO.

LDPO+NLL = LDPO (cw w l l w w

i , yi , ci , yi | xi ) + αLNLL (ci , yi | xi ) (28)

Pθ (cw w
Pθ (cli , yil | xi ) log Pθ (cw w
 
i , yi | xi ) i , yi | xi )
= − log σ β log w w − β log −α , (29)
Pref. (ci , yi | xi ) l l
Pref. (ci , yi | xi ) |ci | + |yiw |
w

where Pθ is the trainable policy model, Pref. is a fixed reference model (often the SFT
checkpoint), and (cw w l l
i , yi ) and (ci , yi ) denote the winning and losing completions for prompt
xi . The first term is the standard DPO logistic loss: it increases the margin between the win
and loss using the difference of log-likelihood ratios, log PPref.θ
, and β controls how strongly
this preference signal pulls away from the reference. The second term is a length-normalized

63
negative log-likelihood penalty on the winning completion, weighted by α, which helps keep
the preferred text high-likelihood in an absolute language modeling sense rather than only
relatively better than the rejected sample.

8.3 Other Regularization

Controlling the optimization is less well defined in other parts of the RLHF stack. Most
reward models have no regularization beyond the standard contrastive loss function. Direct
Alignment Algorithms handle regularization to KL divergences differently, through the β
parameter (see the chapter on Direct Alignment).
Llama 2 proposed a margin loss for reward model training [44]:

L(θ) = − log (σ (rθ (yc | x) − rθ (yr | x) − m(yc , yr ))) (30)

where m(yc , yr ) is the margin between two datapoints yc and yr representing numerical
difference in delta between the ratings of two annotators. This is either achieved by having
annotators rate the outputs on a numerical scale or by using a quantified ranking method,
such as Likert scales.
Reward margins have been used heavily in the direct alignment literature, such as Reward
weighted DPO, ‘’Reward-aware Preference Optimization’ ’ (RPO), which integrates reward
model scores into the update rule following a DPO loss [25], or REBEL [166] that has a
reward delta weighting in a regression-loss formulation.

64
9 Instruction Finetuning
Early large pretrained language models were trained with a next-token prediction objective
and, by default, did not come with an explicit interface for following instructions. Around the
release of GPT-3 [167], prompting and in-context learning became a widely used way to adapt
a single model to many tasks (though task-specific fine-tuning remained common), by showing
examples in-context and asking the model to complete a similar task. A practical next step
was instruction finetuning, which teaches the model to respond in an instruction–response
format rather than just continuing text.
Instruction finetuning took off when two lines of work converged. First, NLP shifted from
bespoke-finetuning task setups to a unified “text-to-text” or instruction framing, which made
it straightforward to standardize diverse datasets and train a single model across many tasks.
Prominent examples of unifying the framework for tasks include Exploring the Limits of
Transfer Learning with a Unified Text-to-Text Transformer (T5 models) [168], Finetuned
Language Models Are Zero-Shot Learners (FLAN dataset) [169], Multitask Prompted Training
Enables Zero-Shot Task Generalization (T0 models) [170], and Cross-Task Generalization via
Natural Language Crowdsourcing Instructions (Natural Instructions dataset) [171]. Second,
scaling pretrained LMs and the rise of prompting/in-context learning showed that a single
model could generalize across tasks, but that generalization becomes far more reliable when
the model is explicitly trained on instruction–response examples. Together, these trends led
to an era of fine-tuning pretrained language models on large collections of instructions—what
is now commonly called instruction finetuning (IFT), or supervised finetuning (SFT), in
which training general models became accessible to wider audiences.
Since its discovery, instruction finetuning, also called colloquially just instruction tuning,
has matured and is standard practice across many language modeling pipelines. At its core,
IFT is the simplest method for adapting language models to a desired task distribution.
It serves as the foundation for RLHF by preparing the model for a format of instructions
that is known as question-answering, and it is the first tool used by those attempting to
apply modern techniques to new domains. Without a basic level of instruction-following
abilities, most of the pipelines we discuss in this book—from preference data collection to
online RLHF optimization—cannot be performed.

9.1 Chat templates and the structure of instructions

The beginning of the post-training process is defining a pattern to format user queries so that
they are easily readable by a language model that processes information through a tokenizer.
When using a pretrained language model, the prompting is quite simple, the model only
knows a few tokens: a beginning-of-sequence token (e.g., <bos_token>), an end-of-sequence
token (e.g., <eos_token>), and a padding token (to manage training on batches with empty
components). This means, to prompt a base model, the user inputs a sequence of tokens for
the model to continue from, such as:
< bos_token > The capital of the United States is

Then, the model would generate tokens until it runs out of its context window, or it generates
the end-of-sequence token.
All post-training stages, from instruction tuning to RLHF and other methods, rely on this

65
formatting to train the model. The tool that handles the structure of the interaction with
the user is called the chat template.
An example which we will break down is below:
{% if messages [0][ ' role '] == ' system ' %}
{# If the conversation begins with a system message , treat it as a
special first turn .
We set an offset so the user / assistant alternation check lines
up correctly . #}
{% set offset = 1 %}
{% else %}
{# No system message : user should be the first non - empty turn . #}
{% set offset = 0 %}
{% endif %}

{# Emit the beginning - of - sequence token ( model - specific ) . #}

{{ bos_token }}

{# Serialize each message into the model ' s chat - markup tokens . #}
{% for message in messages %}
{# Enforce role alternation : ( system ) , user , assistant , user ,
assistant , ...
The boolean expression compares " is this a user message ?"
against whether the
current index ( plus offset ) is expected to be user or assistant
. #}
{% if ( message [ ' role '] == ' user ') != ( loop . index0 % 2 == offset )
%}
{{ raise_exception ( ' Conversation roles must alternate user /
assistant / user / assistant /... ') }}
{% endif %}

{# Wrap each message with special tokens :

- <| im_start | > < role >\ n
- message content ( trimmed )
- <| im_end | >\ n
This produces a single flat token sequence the LM can train on .
#}
{{ ' <| im_start | > ' + message [ ' role '] + '\n ' + message [ ' content '] |
trim + ' <| im_end | >\n ' }}
{% endfor %}

{# Optionally append an " assistant " start tag with no content .

This cues generation to continue from the assistant role . #}
{% if ad d_ge nera tion_ prom pt %}
{{ ' <| im_start | > assistant \n ' }}
{% endif %}

This is the raw code for transforming a list of dictionaries in Python containing messages
and roles into tokens that a language model can predict from.
All information passed into models is assigned a role. The traditional three roles are system,

66
user, and assistant.
The system tag is only used for the first message of the conversation; it holds instructions
for the agent in text that will not be received from or exposed to the user. These system
prompts are used to provide additional context to the models, such as the date and time, or
to patch behaviors. As a fun example, models can be told things such as “You are a friendly
chatbot who always responds in the style of a pirate.”
Next, the two other roles are straightforward: user holds the messages from the person
using the AI, and assistant holds the responses from the model (that is engaging as an AI
assistant).
In order to translate all this information into tokens, we use the code listing above that we
started with. The model has a series of special tokens that separate the various messages
from each other. If we run the above code with the example query “How many helicopters
can a human eat in one sitting?”, the token sequence passed into the model would look as
follows:
<| im_start | > system
You are a friendly chatbot who always responds in the style of a
pirate <| im_end | >
<| im_start | > user
How many helicopters can a human eat in one sitting ? <| im_end | >
<| im_start | > assistant

Notice how the final tokens in the sequence are <|im_start|>assistant. This is how the
model knows to continue generating tokens until it finally generates its end-of-sequence token,
which in this case is <|im_end|>.
By packing all question-answer pair data (and downstream preference tuning data) into this
format, modern language models follow it with perfect consistency. This is the language that
instruction tuned models use to exchange information with users and the models stored on
GPUs or other computing devices.
The behavior can be extended naively to multiple turns, such as shown below:
<| im_start | > system
You are a friendly chatbot who always responds in the style of a
pirate <| im_end | >
<| im_start | > user
How many helicopters can a human eat in one sitting ? <| im_end | >
<| im_start | > assistant
Oh just 6. <| im_end | >
<| im_start | > user
Are you sure about that ? <| im_end | >
<| im_start | > assistant

In the open ecosystem, the standard method for applying the chat template to a list of
messages is a piece of Jinja code saved in the tokenizer, as apply_chat_template.
The above chat template is a derivative of OpenAI’s Chat Markup Language (ChatML),
which was an early attempt to standardize message formatting. Now, OpenAI and other
model providers use a hierarchical system where the user can configure a system message,
yet there are higher-level instructions that may or may not be revealed to the user [172].

67
Many other chat templates exist. Some other examples include Zephyr’s [21]:
<| system | >
You are a friendly chatbot who always responds in the style of a
pirate </ s >
<| user | >
How many helicopters can a human eat in one sitting ? </s >
<| assistant | >

Or Tülu’s:
<| user | >
How are you doing ?
<| assistant | >
I ' m just a computer program , so I don ' t have feelings , but I ' m
functioning as expected . How can I assist you today ? <| endoftext | >

Beyond this, many chat templates include formatting and other tokens for tasks such as
tool-use.

9.2 Best practices of instruction tuning

Instruction tuning as the foundation of post-training and creating helpful language models
is well-established. There are many ways to achieve successful instruction tuning. For
example, efficient finetuning with quantization of some model parameters makes training
very accessible [173]. Also, in narrow domains such as chat alignment, i.e., without harder
skills such as math or code, small, focused datasets can achieve strong performance [13].
Soon after the release of ChatGPT, human datasets with as few as 10K samples such as No
Robots were state-of-the-art [174]. Years later, large-scale synthetic datasets work best [6]
on most tasks.
A few principles remain:
• High-quality data is key to performance. The completions are what the model actually
learns from (in many cases the prompts are not predicted over so the model does not
learn to predict prompts).
• ~1M prompts can be used to create a model capable of excellent RLHF and post-training.
Further scaling can still help, but returns diminish quickly.
• The best prompts are those in a similar distribution to downstream tasks of interest.
• If multiple stages of training are done after instruction tuning, the models can recover
from some noise in the process. Optimizing the overall optimization is more important
than each individual stage.

9.3 Implementation
While the loss function is the same as pretraining, there are a few key implementation details
that differ from the setting used for pre-training. Many practices, such as deciding on the
types of parallelism used to shard models across many GPUs are the same as pretraining,
just the total number of machines used is often lower (for the first technical change listed
below):

68
• Smaller batch sizes: Compared to pre-training, instruction tuning (and other post-
training techniques such as preference finetuning) use substantially smaller batch sizes.
For example, OLMo 2 uses a batch size of 1024 sequences for the 7B and 2048 for the
13B pretraining, while both only use a batch size of 256 sequences at post-training
[59]. The smaller batch sizes mean that these training jobs cannot be sharded across
as many devices as pretraining – in practice, distributed training setups have minimum
per-device batch sizes, so if you’re trying to retain a smaller global batch size for SFT
you can use cumulatively fewer GPUs. In practice the batch size forcing a smaller
concurrent GPU allotment per training job is not a limiting factor because the training
token counts for SFT are much smaller than pretraining, and training for multiple
seeds is needed in post-training to obtain the best final performance.
• Prompt masking: When pretraining, every token in the batch is predicted autore-
gressively and the loss is then applied to them. For instruction tuning, the prompt
tokens are masked out so the model isn’t learning to accurately predict user queries –
just responses. The same applies for other post-training algorithms.
• Multi-turn masking: For multi-turn conversations, there are two common masking
choices. (1) Final-turn only: only the tokens in the final assistant turn are included in
the loss, while all earlier context (including earlier assistant turns) is masked. Long
conversations can still be “unrolled” into multiple training samples: for a conversation
of N turns, each example predicts one assistant response while masking all prior context
(and excluding any future turns). (2) Mask user turns only: all user turns are masked,
but every assistant turn is included in the loss. You can still unroll in this setting if
you want more (shorter) training examples, but the key difference is that intermediate
assistant replies are trained on directly.
• Same loss function as pretraining: Instruction tuning uses the same autoregressive
loss function used in pretraining language models, but with substantially different data
and masking (training only on full sequences, whereas pretraining documents can be
split across batches), etc.

69
10 Rejection Sampling
Rejection Sampling (RS) is a popular and simple baseline for performing preference fine-tuning.
This makes it one of a handful of methods that are used after a first round of instruction
tuning in order to further refine the model to human preferences. Rejection sampling operates
by curating new candidate completions, filtering them based on a trained reward model,
and then instruction finetuning the original model only on the top completions (same loss
function as when doing a dedicated training stage for learning to follow instructions).
The name originates from computational statistics [175], where one wishes to sample from
a complex distribution, but does not have a direct method to do so. To alleviate this, one
samples from a simpler distribution to model and uses a heuristic to check if the sample is
permissible. With language models, the target distribution is high-quality completions to
prompts, the filter is a reward model, and the sampling distribution is the current model.
Many prominent RLHF and preference fine-tuning papers have used rejection sampling as a
baseline, but a canonical implementation and documentation does not exist.
WebGPT [4], Anthropic’s Helpful and Harmless agent [5], OpenAI’s popular paper on process
reward models [45], Llama 2 Chat models [44], and other seminal works all use this baseline;
more recent work has formalized it directly (e.g., RAFT [176] for applying it to alignment
in multiple modalities and Statistical Rejection Sampling Optimization (RSO) [177] that
gives a principled overview on how rejection sampling relates to other preference learning
objectives).
Throughout this chapter, we use x to denote prompts and y to denote completions. This
notation is common in the language model literature, where methods operate on full prompt-
completion pairs rather than individual tokens.

10.1 Training Process

Rejection sampling overall follows a few stages.
0. Prompt and reward model selection: First, you must select the prompts you want
to train on, relative to other stages of training. The simplest method is to re-use every
prompt from the first SFT/IFT stage, but this can cause some overfitting. Before
doing rejection sampling, you must also have trained a reward model (see Chapter 7
for more information).
1. Generate completions from the starting checkpoint: Next, one must generate
completions to the selected prompts with the model they want to optimize. This can
involve tweaking many settings, such as sampling temperature, top-p, max sequence
length, number of completions per prompt, etc.
2. Select top completions with a reward model: All completions are ranked by a
reward model. This can include deduplication to only have one prompt per completion
after this stage, or not, as a lot of the decisions become based on empirical ablation
studies.
3. SFT on top completions: To finish rejection sampling, one instruction finetunes
the starting checkpoint on the selected completions.
A visual overview of the rejection sampling process is included below in fig. 14.

70
 Figure 14: Rejection sampling overview.

The actual details on which prompts to use, how to select a reward model, how to sequence
rejection sampling, etc. are not well documented in the literature. This chapter provides an
overview of the methods and leaves further experimentation to the reader.

10.1.1 1. Generating Completions

To generate a set of multiple candidate completions per prompt, let’s define a set of M
prompts as a vector:

X = [x1 , x2 , ..., xM ]

These prompts can come from many sources, but most commonly they come from the
instruction training set.
For each prompt xi , we generate N completions. We can represent this as a matrix:
 
y1,1 y1,2 ··· y1,N
 y2,1 y2,2 ··· y2,N 
Y = .
 
.. .. ..
 ..

. . . 
yM,1 yM,2 ··· yM,N

where yi,j represents the j-th completion for the i-th prompt. Each row i corresponds to a
single prompt xi and contains its N candidate completions; each column j corresponds to
the j-th sampled completion across all prompts.

10.1.2 2. Scoring Completions

Now, we pass all of these prompt-completion pairs through a reward model, to get a matrix
of rewards. We’ll represent the rewards as a matrix R:
 
r1,1 r1,2 ··· r1,N
 r2,1 r2,2 ··· r2,N 
R= .
 
.. .. ..
 ..

. . . 
rM,1 rM,2 ··· rM,N

71
Each reward ri,j is computed by passing the completion yi,j and its corresponding prompt
xi through a reward model R:

ri,j = R(yi,j | xi )

There are multiple methods to select the top completions to train on.
To formalize the process of selecting the best completions based on our reward matrix, we
can define a selection function S that operates on the reward matrix R.

[Link] Top Per Prompt The first potential selection function takes the max reward
per prompt.

S(R) = [arg max r1,j , arg max r2,j , ..., arg max rM,j ]
j j j

This function S returns a vector of indices, where each index corresponds to the column
with the maximum reward for each row in R. We can then use these indices to select our
chosen completions:

Ychosen = [y1,S(R)1 , y2,S(R)2 , ..., yM,S(R)M ]

[Link] Top Overall Pairs Alternatively, we can select the top K prompt-completion
pairs from the entire set. First, let’s flatten our reward matrix R into a single vector:

Rf lat = [r1,1 , r1,2 , ..., r1,N , r2,1 , r2,2 , ..., r2,N , ..., rM,1 , rM,2 , ..., rM,N ]

This Rf lat vector has length M × N , where M is the number of prompts and N is the
number of completions per prompt.
Now, we can define a selection function SK that selects the indices of the K highest values
in Rf lat :

SK (Rf lat ) = argsort(Rf lat )[−K :]

where argsort returns the indices that would sort the array in ascending order, and we take
the last K indices to get the K highest values.
To get our selected completions, we need to map these flattened indices back to our original
completion matrix Y . To recover the corresponding (prompt, completion) pair, you can map
a zero-indexed flattened index k to (i, j) via i = ⌊k/N ⌋ + 1 and j = (k mod N ) + 1.

72
[Link] Selection Example Consider the case where we have the following situation,
with 5 prompts and 4 completions. We will show two ways of selecting the completions based
on reward.
 
0.7 0.3 0.5 0.2
0.4 0.8 0.6 0.5
 
0.9
R= 0.3 0.4 0.7
0.2 0.5 0.8 0.6
0.5 0.4 0.3 0.6

First, per prompt. Intuitively, we can highlight the reward matrix as follows:
 
0.7 0.3 0.5 0.2
 0.4
 0.8 0.6 0.5 

0.9
R= 0.3 0.4 0.7 

 0.2 0.5 0.8 0.6 
0.5 0.4 0.3 0.6

Using the argmax method, we select the best completion for each prompt:

S(R) = [arg max ri,j for i ∈ [1, 5]]

j

S(R) = [1, 2, 1, 3, 4]

This means we would select:

• For prompt 1: completion 1 (reward 0.7)
• For prompt 2: completion 2 (reward 0.8)
• For prompt 3: completion 1 (reward 0.9)
• For prompt 4: completion 3 (reward 0.8)
• For prompt 5: completion 4 (reward 0.6)
Now, best overall. Let’s highlight the top 5 overall completion pairs.
 
0.7 0.3 0.5 0.2
 0.4
 0.8 0.6 0.5 

0.9
R= 0.3 0.4 0.7
 0.2 0.5 0.8 0.6 
0.5 0.4 0.3 0.6

First, we flatten the reward matrix:

Rf lat = [0.7, 0.3, 0.5, 0.2, 0.4, 0.8, 0.6, 0.5, 0.9, 0.3, 0.4, 0.7, 0.2, 0.5, 0.8, 0.6, 0.5, 0.4, 0.3, 0.6]

Now, we select the indices of the 5 highest values:

S5 (Rf lat ) = [8, 5, 14, 0, 11]

73
Mapping these back to our original matrix:
• Index 8 → prompt 3, completion 1 (reward 0.9)
• Index 5 → prompt 2, completion 2 (reward 0.8)
• Index 14 → prompt 4, completion 3 (reward 0.8)
• Index 0 → prompt 1, completion 1 (reward 0.7)
• Index 11 → prompt 3, completion 4 (reward 0.7)

[Link] Implementation Example Here is a code snippet showing how the selection
methods could be implemented.
import numpy as np

x = np . random . randint (10 , size =10)

print ( f " { x =} " )
sorted_indices = np . argsort ( x )
x_sorted = x [ sorted_indices ]
print ( f " { x_sorted =} " )

# first way to recover the original array

i_rev = np . zeros (10 , dtype = int )
i_rev [ sorted_indices ] = np . arange (10)
np . allclose (x , x_sorted [ i_rev ])

# second way to recover the original array

np . allclose (x , x_sorted [ np . argsort ( sorted_indices ) ])

10.1.3 3. Fine-tuning
With the selected completions, you then perform standard instruction fine-tuning on the
current rendition of the model. More details can be found in the chapter on instruction
tuning.

10.2 Implementation Details

The core hyperparameters for performing this training are very intuitive:
• Sampling parameters: Rejection sampling is directly dependent on the completions
received from the model. Common settings for rejection sampling include temperatures
above zero, e.g. between 0.7 and 1.0, with other modifications to parameters such as
top-p or top-k sampling.
• Completions per prompt: Successful implementations of rejection sampling have
included 10 to 30 or more completions for each prompt. Using too few completions
will make training biased and/or noisy.
• Instruction tuning details: No clear training details for the instruction tuning
during rejection sampling have been released. It is likely that they use slightly different
settings than the initial instruction tuning phase of the model.
• Heterogeneous model generations: Some implementations of rejection sampling
include generations from multiple models rather than just the current model that is
going to be trained. Best practices on how to do this are not established.

74
 • Reward model training: The reward model used will heavily impact the final result.
For more resources on reward model training, see the relevant chapter.
When doing batch reward model inference, you can sort the tokenized completions by length
so that the batches are of similar lengths. This eliminates the need to run inference on as
many padding tokens and will improve throughput in exchange for minor implementation
complexity.

10.3 Related: Best-of-N Sampling

Best-of-N (BoN) is a close relative of rejection sampling, where the same generate-and-score
procedure is followed, but you do not fine-tune the model on the selected completions.
Instead, BoN is a way of computing a best possible completion to a static prompt (or set
of prompts) at inference time, and related techniques are often used in “Pro” tiers of chat
models that spend extra compute to get an answer to your query.
Best-of-N sampling is often included as a baseline relative to RLHF training methods. It is
important to remember that BoN does not modify the underlying model, but is a sampling
technique. For this reason, comparisons for BoN sampling to online training methods, such
as PPO, are still valid in some contexts. For example, you can still measure the KL distance
when running BoN sampling relative to any other policy.
Here, we will show that when using simple BoN sampling over one prompt, both selection
criteria shown above are equivalent.
Let R be a reward vector for our single prompt with N completions:

R = [r1 , r2 , ..., rN ] (31)

Where rj represents the reward for the j-th completion.

Using the argmax method, we select the best completion for the prompt:

S(R) = arg max rj (32)

j∈[1,N ]

Using the Top-K method with K = 1 reduces to the same method, which is common practice.

75
11 Reinforcement Learning (i.e. Policy Gradient Algo-
rithms)
In the RLHF process, the reinforcement learning algorithm slowly updates the model’s
weights with respect to feedback from a reward model. The policy – the model being trained
– generates completions to prompts in the training set, then the reward model scores them,
and then the reinforcement learning optimizer takes gradient steps based on this information.
This chapter explains the mathematics and trade-offs across various algorithms used to learn
from the signal the reward model gives to on-policy data. These algorithms are run for a
period of many epochs, often thousands or millions of batches across a larger set of prompts,
with gradient updates in between each of them.
The algorithms that popularized RLHF for language models were policy-gradient reinforce-
ment learning algorithms. These algorithms, such as Proximal Policy Optimization (PPO),
Group Relative Policy Optimization (GRPO), and REINFORCE, use recently generated
samples to update their model (rather than storing scores in a replay buffer like algorithms,
e.g. Deep Q-Networks, DQN, used in popular projects such as AlphaGo). In this section we
will cover the fundamentals of the policy gradient algorithms and how they are used in the
modern RLHF framework.
At a machine learning level, this section is the subject with the highest complexity in the
RLHF process. Though, as with most modern AI models, the largest determining factor on
its success is the data provided as inputs to the process.
When RLHF came onto the scene with ChatGPT, it was largely known that they used a
variant of PPO, and many initial efforts were built upon that. Over time, multiple research
projects showed the promise of REINFORCE-style algorithms [178] [111], touted for its
simplicity over PPO without a reward model (saves memory and therefore the number of
GPUs required) and with simpler value estimation (no Generalized Advantage Estimation,
GAE, which is a method to compute advantages used for variance reduction in policy gradient
algorithms). More algorithms have emerged, including Group Relative Policy Optimization,
which is particularly popular with reasoning tasks, but in general many of these algorithms
can be tuned to fit a specific task. In this chapter, we cover the core policy gradient setup
and the three algorithms mentioned above due to their central role in the establishment of a
canonical RLHF literature.
For definitions of symbols, see the problem setup chapter.
This chapter uses (s, a) notation from the reinforcement learning literature, where s denotes
states and a denotes actions. In the language model context, you will often see (x, y) instead,
where x is the prompt and y is the completion. The (s, a) framing is more general—these
algorithms were designed for sequential decision problems where actions are taken at each
timestep. However, many RLHF implementations treat the entire completion as a single
action, making the (x, y) notation equally valid.

11.1 Policy Gradient Algorithms

Reinforcement learning algorithms are designed to maximize the future, discounted reward
across a trajectory of states, s ∈ S, and actions, a ∈ A (for more notation, see Chapter 3,
Definitions). The objective of the agent, often called the return, is the sum of discounted

76
future rewards (where γ ∈ [0, 1] is a factor that prioritizes near-term rewards) at a given
time t:

∞
X
Gt = Rt+1 + γRt+2 + · · · = γ k Rt+k+1 . (33)
k=0

The return definition can also be estimated as:

Gt = γGt+1 + Rt+1 . (34)

This return is the basis for learning a value function V (s) that is the estimated future return
given a current state:

 
V (s) = E Gt |St = s . (35)

All policy gradient algorithms optimize a policy πθ (a | s) to maximize expected return; this
objective can be expressed using the induced value function V πθ (s).
Where dπθ (s) is the state-visitation distribution induced by policy πθ (a | s), the objective we
maximize can be written as:
X
J(θ) = dπθ (s)V πθ (s), (36)
s

In a finite MDP this is a sum over all states, but in practice we never compute it exactly.
Instead, we estimate it from data by sampling rollouts from the current policy. In RLHF
this typically means sampling prompts xi from a dataset and generating completions yi ∼
πθ (· | xi ), then taking an empirical average such as:

B
ˆ = 1
X
J(θ) R(xi , yi ),
B i=1

or, in an MDP view with per-step rewards,

B Ti
ˆ 1 XX
J(θ) = γ t ri,t .
B i=1 t=0

The core of policy gradient algorithms is computing the gradient with respect to the finite-
time expected return over the current policy. With this expected return, J, the parameter
update can be computed as follows, where α is the learning rate:

θ ← θ + α∇θ J(θ) (37)

The core implementation detail is how to compute said gradient.

77
Another way to pose the RL objective we want to maximize is as follows:

J(θ) = Eτ ∼πθ [R(τ )] , (38)

P∞
where τ = (s0 , a0 , s1 , a1 , . . .) is a trajectory and R(τ ) = t=0 rt is the total reward of
the trajectory. Alternatively, we can write the expectation as an integral over all possible
trajectories: Z
J(θ) = pθ (τ )R(τ )dτ (39)
τ

Notice that we can express the trajectory probability as follows, where πθ (at |st )p(st+1 |st , at )
is the transition probability to a group of next states from one state and action:
∞
Y
pθ (τ ) = p(s0 ) πθ (at |st )p(st+1 |st , at ), (40)
t=0

If we take the gradient of the objective (eq. 38) with respect to the policy parameters θ:
Z
∇θ J(θ) = ∇θ pθ (τ )R(τ )dτ (41)
τ

Notice that we can use the log-derivative trick in order to rewrite the gradient of the integral
as an expectation:

∇θ pθ (τ )
∇θ log pθ (τ ) = (from chain rule)
pθ (τ ) (42)
=⇒ ∇θ pθ (τ ) = pθ (τ )∇θ log pθ (τ ) (rearranging)

Using this log-derivative trick:

Z
∇θ J(θ) = ∇θ pθ (τ )R(τ )dτ
τ
Z
(43)
= pθ (τ )∇θ log pθ (τ )R(τ )dτ
τ
= Eτ ∼πθ [∇θ log pθ (τ )R(τ )]

Where the final step uses the definition of an R expectation under the trajectory distribution
pθ (τ ): for any function f , Eτ ∼pθ [f (τ )] = τ f (τ ) pθ (τ ) dτ (or a sum in the discrete case).
Writing it as anPexpectation is useful because we can approximate it with Monte Carlo
B
rollouts, e.g., B1 i=1 f (τi ) for trajectories τi ∼ πθ .
Back to the derivation, expanding the log probability of the trajectory:

∞
X ∞
X
log pθ (τ ) = log p(s0 ) + log πθ (at |st ) + log p(st+1 |st , at )
t=0 t=0

Now, if we take the gradient of the above, we get:

78
 • ∇θ log p(s0 ) = 0 (initial state doesn’t depend on θ)
• ∇θ log p(st+1 |st , at ) = 0 (environment transition dynamics don’t depend on θ)
• only ∇θ log πθ (at |st ) survives
Therefore, the gradient of the log probability of the trajectory simplifies to:
∞
X
∇θ log pθ (τ ) = ∇θ log πθ (at |st )
t=0

Substituting this back in eq. 43, we get:

∞
" #
X
∇θ J(θ) = Eτ ∼πθ ∇θ log πθ (at |st )R(τ )
t=0

Quite often, people use a more general formulation of the policy gradient:
"∞ #
X
g = ∇θ J(θ) = Eτ ∼πθ ∇θ log πθ (at |st )Ψt (44)
t=0

Where Ψt can be the following (where the rewards can also often be discounted by γ), a
taxonomy adopted from Schulman et al. 2015 [179]:
P∞
R(τ ) = t=0 rt : total reward of the trajectory.
1. P
∞
2. Pt′ =t rt′ : reward following action at , also described as the return, G.
∞
3. t′ =t rt − b(st ): baselined version of previous formula.
′
π
4. Q (st , at ): state-action value function.
5. Aπ (st , at ): advantage function, which yields the lowest possible theoretical variance if
it can be computed accurately.
6. rt + V π (st+1 ) − V π (st ): Temporal Difference (TD) residual.
The baseline is a value used to reduce variance of policy updates (more on this below).
For language models, some of these concepts do not make as much sense. For example, for
a deterministic policy π the state value is V π (s) = Qπ (s, π(s)) (and for the optimal value
function one has V ∗ (s) = maxa Q∗ (s, a)). For a stochastic policy, the analogous identity is
V π (s) = Ea∼π(·|s) [Qπ (s, a)]. If we define s + a as the continuation a to the prompt s, then
Q(s, a) = V (s + a), which gives a different advantage trick:

A(s, a) = Q(s, a) − V (s) = V (s + a) − V (s) = r + γV (s + a) − V (s) (45)

Which is a combination of the reward, the value of the prompt, and the discounted value of
the entire utterance.

11.1.1 Vanilla Policy Gradient

The vanilla policy gradient implementation optimizes the above expression for J(θ) by
differentiating with respect to the policy parameters. A simple version, with respect to the
overall return, is:

79
 " T
#
X
∇θ J(θ) = Eτ ∇θ log πθ (at |st )Rt (46)
t=0

A common problem with vanilla policy gradient algorithms is the high variance in gradient
updates, which can be mitigated in multiple ways. The high variance comes from the
gradient updates being computed from estimating the return G from an often small set of
rollouts in the environment that tend to be susceptible to noise (e.g. the stochastic nature
of generating from language models with temperature > 0). The variance across return
estimates is higher in domains with sparse rewards, as more of the samples are 0 or 1, rather
than closely clustered. In order to alleviate this, various techniques are used to normalize
the value estimation, called baselines. Baselines accomplish this in multiple ways, effectively
normalizing by the value of the state relative to the downstream action (e.g. in the case
of Advantage, which is the difference between the Q value and the value). The simplest
baselines are averages over the batch of rewards or a moving average. Even these baselines
can de-bias the gradients so Ea∼π(a|s) [∇θ log πθ (a|s)] = 0, improving the learning signal
substantially.
Many of the policy gradient algorithms discussed in this chapter build on the advantage
formulation of policy gradient:

" T #
X
πθ
∇θ J(θ) = Eτ ∇θ log πθ (at |st )A (st , at ) (47)
t=0

11.1.2 REINFORCE
The algorithm REINFORCE is likely a backronym, but the components of the algorithm it
represents are quite relevant for modern reinforcement learning algorithms. Defined in the
seminal paper Simple statistical gradient-following algorithms for connectionist reinforcement
learning [180]:
The name is an acronym for “REward Increment = Nonnegative Factor X Offset
Reinforcement X Characteristic Eligibility.”
The three components of this are how to do the reward increment, a.k.a. the policy gradient
step. It has three pieces to the update rule:
1. Nonnegative factor: This is the learning rate (step size) that must be a positive number,
e.g. α below.
2. Offset Reinforcement: This is a baseline b or other normalizing factor of the reward to
improve stability.
3. Characteristic Eligibility: This is how the learning becomes attributed per token. It
can be a general value, e per parameter, but is often log probabilities of the policy in
modern equations.
Thus, the form looks quite familiar:

∆θ = α(r − b)e (48)

80
With more modern notation and the generalized return G, the REINFORCE operator
appears as:

T
hX i
∇θ J(θ) = Eτ ∼πθ ∇θ log πθ (at | st ) (Gt − b(st )) , (49)
t=0

Here, the value Gt − b(st ) is the advantage of the policy at the current state, so we can
reformulate the policy gradient in a form that we continue later with the advantage, A:

T
hX i
∇θ J(θ) = Eτ ∼πθ ∇θ log πθ (at | st ) At , (50)
t=0

REINFORCE is a specific implementation of vanilla policy gradient that uses a Monte Carlo
estimator of the gradient.

[Link] REINFORCE Leave One Out (RLOO) The core implementation detail
of REINFORCE Leave One Out versus standard REINFORCE is that it takes the average
reward of the other samples in the batch to compute the baseline – rather than averaging
over all rewards in the batch [181], [178], [182].
Crucially, this only works when generating multiple trajectories (completions) per state
(prompt), which is common practice in multiple domains of finetuning language models with
RL.
Specifically, for the REINFORCE Leave-One-Out (RLOO) baseline, given K sampled
trajectories (actions taken conditioned on a prompt) a1 , . . . , aK , to a given prompt s we
define the baseline explicitly as the following per-prompt:

K
1 X
b(s, ak ) = R(s, ai ), (51)
K −1
i=1,i̸=k

resulting in the advantage:

A(s, ak ) = R(s, ak ) − b(s, ak ). (52)

Equivalently, this can be expressed as:

K
!
K 1 X
A(s, ak ) = R(s, ak ) − R(s, ai ) . (53)
K −1 K i=1

This is a simple, low-variance per-prompt advantage estimate that is closely related to the
group-relative advantage used in Group Relative Policy Optimization, GRPO (discussed
shortly, after Proximal Policy Optimization, PPO). In practice, GRPO-style training mainly
differs in how it applies the KL regularizer (as an explicit loss term vs. folded into the
reward) and whether it uses PPO-style ratio clipping. To be specific, the canonical GRPO

81
implementation applies the KL penalty at the loss level, where the derivation for RLOO or
traditional policy-gradients apply the KL penalty to the reward itself. With the transition
from RLHF to reasoning and reinforcement learning with verifiable rewards (RLVR), the
prevalence of KL penalties has decreased overall, with many reasoning adaptations of RLHF
code turning them off entirely. Still, the advantage from RLOO could be combined with the
clipping of PPO, showing how similar many of these algorithms are.
RLOO and other algorithms that do not use a value network – an additional model copy (a
critic) that predicts a scalar value V (st ) per token – assign the same sequence-level advantage
(or reward) to every token when computing the loss. Algorithms that use a learned value
network, such as PPO, assign a different value to every token individually, discounting from
the final reward achieved at the EOS token. With a KL distance penalty, RLOO aggregates
the per-token KL over the completion and folds that scalar into the sequence reward, so
the resulting advantage is broadcast to all tokens. PPO subtracts a per-token KL from the
per-token reward before computing At , giving token-level credit assignment. GRPO typically
retains a sequence-level advantage but adds a separate per-token term to the loss, rather
than subtracting it from the reward. These details and trade-offs are discussed later in the
chapter.

11.1.3 Proximal Policy Optimization

Proximal Policy Optimization (PPO) [183] is one of the foundational algorithms behind Deep
RL’s successes (such as OpenAI’s Five, which mastered DOTA 2 [184] and large amounts
of research). The objective that PPO maximizes, with respect to the advantages and the
policy probabilities, is as follows:
   
πθ (a|s) πθ (a|s)
J(θ) = min A, clip , 1 − ε, 1 + ε A . (54)
πθold (a|s) πθold (a|s)

Here, πθ (a|s) is the current policy being optimized and πθold (a|s) is the policy that was used
to collect the training data (i.e., the policy from the previous iteration). The ratio between
these two policies emerges from importance sampling, which allows us to reuse data collected
under an old policy to estimate gradients for a new policy.
Recall from the advantage formulation of the policy gradient (eq. 47) that we have:
" T #
X
πθ
∇θ J(θ) = Eτ ∼πθ ∇θ log πθ (at |st )A (st , at ) .
t=0

This expectation is taken over trajectories sampled from πθ , but in practice we want to take
multiple gradient steps on a batch of data that was collected from a fixed policy πθold . To
correct for this distribution mismatch, we multiply by the importance weight πθπθ (a|s)
(a|s) , which
old
reweights samples to account for how much more or less likely they are under the current
policy versus the data-collection policy. Without constraints, optimizing this importance-
weighted objective can lead to destructively large policy updates when the ratio diverges far
from 1. PPO addresses this by clipping the ratio to the range [1 − ε, 1 + ε], ensuring that
the policy cannot change too drastically in a single update.

82
For completeness, PPO is typically written as an expected clipped surrogate objective over
timesteps:

πθ (at | st )
J(θ) = Et [min (rt (θ)At , clip(rt (θ), 1 − ε, 1 + ε)At )] , Rt (θ) = . (55)
πθold (at | st )

The objective is often converted into a loss function by simply adding a negative sign, which
makes the optimizer seek to make it as negative as possible.
For language models, the objective (or loss) is computed per token, which intuitively can be
grounded in how one would compute the probability of the entire sequence of autoregressive
predictions – by a product of probabilities. From there, the common implementation is with
log-probabilities that make the computation simpler to perform in modern language modeling
frameworks.

|a|    
1 X πθ (at |st ) πθ (at |st )
J(θ) = min At , clip , 1 − ε, 1 + ε At . (56)
|a| t=0 πθold (at |st ) πθold (at |st )

This is the per-token version of PPO, which also applies to other policy-gradient methods,
but is explored further later in the implementation section of this chapter. Here, the term for
1
averaging by the number of tokens in the action, |a| , comes from common implementation
practices, but is not in a formal derivation of the loss (shown in [185]).
Here we will explain the different cases this loss function triggers given various advantages
and policy ratios. At an implementation level, the inner computations for PPO involve two
main terms: 1) a standard policy gradient with a learned advantage and 2) a clipped policy
gradient based on a maximum step size.
To understand how different situations emerge, we can define the policy ratio as:

πθ (a|s)
R(θ) = (57)
πθold (a|s)

The policy ratio is a centerpiece of PPO and related algorithms. It emerges from computing
the gradient of a policy and controls the parameter updates in a very intuitive way. For any
batch of data, the policy ratio starts at 1 for the first gradient step for that batch, since πθ
is the same as πθold at this point. Then, in the next gradient step, the policy ratio will be
above one if that gradient step increased the likelihood of certain tokens with an associated
positive advantage, or less than one for the other case. A common practice is to take 1-4
gradient steps per batch with policy gradient algorithms before updating πθold .

[Link] Understanding the PPO Objective Overall, the PPO objective can be
visualized by two lines of a plot of objective versus policy ratio, which is shown in fig. 15. The
PPO objective is maximized by changing the probability of the sampled actions. Numerically,
the objective controls for both positive and negative advantage cases by clever use of the
minimum operation, making it so the update is at most pushed by an epsilon distance away
from a policy ratio of 1.

83
Within the trust region, PPO operates the same as other policy gradient algorithms. This is
by design! The trust region is a concept used to cap the maximum step size of PPO and
its peer algorithms for stability of updates. The core of the PPO algorithm, the clip and
min/max functions, is to define this region. The objective becomes flat outside of it.
The idea of a “trust region” comes from the numerical optimization literature [186], but was
popularized within Deep RL from the algorithm Trust Region Policy Optimization (TRPO),
which is accepted as the predecessor to PPO [187]. The trust region is the area where the
full policy-gradient steps are applied, as the updates are not “clipped” by the max/min
operations of the PPO objective.

Figure 15: Visualization of the different regions of the PPO objective for a hypothetical
advantage. The “trust region” would be described as the region where the log-ratio is within
1 ± ε.

The policy ratio and advantage together can occur in a few different configurations. We will
split the cases into two groups: positive and negative advantage.
Positive Advantage (At > 0)
This means that the action taken was beneficial according to the value function, and we want
to increase the likelihood of taking that action in the future. Now, let’s look at different
cases for the policy ratio R(θ):
1. R(θ) < 1 − ε:
• Interpretation: Action is less likely with the new policy than the old policy
• Unclipped Term: R(θ)At
• Clipped Term: (1 − ε)At
• Objective: R(θ)At
• Gradient: ∇θ R(θ)At ̸= 0
• What happens: Normal policy-gradient update - increase likelihood of action
2. 1 − ε ≤ R(θ) ≤ 1 + ε:

84
 • Interpretation: Action is almost equally likely with the new policy as the old
policy
• Unclipped Term: R(θ)At
• Clipped Term: R(θ)At
• Objective: R(θ)At
• Gradient: ∇θ R(θ)At ̸= 0
• What happens: Normal policy-gradient update - increase likelihood of action
3. 1 + ε < R(θ):
• Interpretation: Action is more likely with the new policy than the old policy
• Unclipped Term: R(θ)At
• Clipped Term: (1 + ε)At
• Objective: (1 + ε)At
• Gradient: ∇θ (1 + ε)At = 0
• What happens: NO UPDATE - action is already more likely under the new
policy
To summarize, when the advantage is positive (At > 0), we want to boost the probability of
the action. Therefore:
• We perform gradient steps only in the case when πnew (a) ≤ (1 + ε)πold (a). Intuitively,
we want to boost the probability of the action, since the advantage was positive, but
not boost it so much that we have made it substantially more likely.
• Crucially, when πnew (a) > (1 + ε)πold (a), then we don’t perform any update, and the
gradient of the clipped objective is 0. Intuitively, the action is already more expressed
with the new policy, so we don’t want to over-reinforce it.
Negative Advantage (At < 0)
This means that the action taken was detrimental according to the value function, and we
want to decrease the likelihood of taking that action in the future. Now, let’s look at different
cases for the policy ratio R(θ):
1. R(θ) < 1 − ε:
• Interpretation: Action is less likely with the new policy than the old policy
• Unclipped Term: R(θ)At
• Clipped Term: (1 − ε)At
• Objective: (1 − ε)At
• Gradient: ∇θ (1 − ε)At = 0
• What happens: NO UPDATE - action is already less likely under the new
policy
2. 1 − ε ≤ R(θ) ≤ 1 + ε:
• Interpretation: Action is almost equally likely with the new policy as the old
policy
• Unclipped Term: R(θ)At
• Clipped Term: R(θ)At
• Objective: R(θ)At
• Gradient: ∇θ R(θ)At ̸= 0
• What happens: Normal policy-gradient update - decrease likelihood of action

85
 3. 1 + ε < R(θ):
• Interpretation: Action is more likely with the new policy than the old policy
• Unclipped Term: R(θ)At
• Clipped Term: (1 + ε)At
• Objective: R(θ)At
• Gradient: ∇θ R(θ)At ̸= 0
• What happens: Normal policy-gradient update - decrease likelihood of action
To summarize, when the advantage is negative (At < 0), we want to decrease the probability
of the action. Therefore:
• We perform gradient steps only in the case when πnew (a) ≥ (1 − ε)πold (a). Intuitively,
we want to decrease the probability of the action, since the advantage was negative,
and we do so proportional to the advantage.
• Crucially, when πnew (a) < (1 − ε)πold (a), then we don’t perform any update, and the
gradient of the clipped objective is 0. Intuitively, the action is already less likely under
the new policy, so we don’t want to over-suppress it.
It is crucial to remember that PPO within the trust region is roughly the same as standard
forms of policy gradient.

[Link] Value Functions and PPO The value function within PPO is an additional
copy of the model that is used to predict the value per token. The value of a token (or state)
in traditional RL is predicting the future return from that moment, often with discounting.
This value in PPO is used as a learned baseline, representing an evolution of the simple
Monte Carlo version used with REINFORCE (which doesn’t need the learned value network).
This highlights how PPO is an evolution of REINFORCE and vanilla policy-gradient in
multiple forms, across the optimization form, baseline, etc. In practice, with PPO and other
algorithms used for language models, this is predicting the return of each token after the
deduction of KL penalties (the per-token loss includes the KL from the reward traditionally,
as discussed).
There are a few different methods (or targets) used to learn the value functions. Generalized
Advantage Estimation (GAE) is considered the state-of-the-art and canonical implementation
in modern systems, but it carries more complexity by computing the value prediction error
over multiple steps – see the later section on GAE in this chapter. A value function can also
be learned with Monte Carlo estimates from the rollouts used to update the policy. PPO
has two losses – one to learn the value function and another to use that value function to
update the policy.
A simple example implementation of a value network loss is shown below.
# Basic PPO critic targets & loss ( no GAE )
#
# B : Batch Size
# L : Completion Length
# Inputs :
# rewards : (B , L ) post - KL per - token rewards ; EOS row includes
outcome
# done_mask : (B , L ) 1.0 at terminal token ( EOS or truncation if
penalized ) , else 0.0

86
# completion_mask : (B , L ) 1.0 on response tokens to supervise (
ignore the prompt )
# values : (B , L ) current critic predictions V_theta ( s_t )
# because a value network is a running update
# old_values : (B , L ) critic predictions at rollout time V_ { theta_old
}( s_t )
# gamma : discount factor , float ( often 1.0 for LM RLHF )
# epsilon_v : float value clip range ( e . g . , 0.2) , similar to PPO Loss
Update itself , optional
#
# Returns :
# value_loss : scalar ; advantages : (B , L ) detached ( for policy loss )

B , L = rewards . shape

# 1) Monte Carlo returns per token ( reset at terminals )

# Apply discounting , if enabled
returns = torch . zeros_like ( rewards )
running = torch . zeros (B , device = rewards . device , dtype = rewards . dtype )
for t in reversed ( range ( L ) ) :
running = rewards [: , t ] + gamma * (1.0 - done_mask [: , t ]) *
running
returns [: , t ] = running

targets = returns # y_t = G_t ( post - KL )

# 2) PPO - style value clipping ( optional )

v_pred = values
v_old = old_values
v_clip = torch . clamp ( v_pred , v_old - epsilon_v , v_old + epsilon_v )

vf_unclipped = 0.5 * ( v_pred - targets ) ** 2

vf_clipped = 0.5 * ( v_clip - targets ) ** 2
vf_loss_tok = torch . max ( vf_unclipped , vf_clipped )

# 3) Mask to response tokens and aggregate

denom = completion_mask . sum ( dim =1) . clamp_min (1)
value_loss = (( vf_loss_tok * completion_mask ) . sum ( dim =1) / denom ) . mean
()

# 4) Advantages for policy loss ( no GAE ) : A_t = G_t - V ( s_t )

advantages = ( targets - v_pred ) . detach ()

# The value loss is applied later , often with the PG loss , e . g .

# total_loss = policy_loss + vf_coef * value_loss

11.1.4 Group Relative Policy Optimization

Group Relative Policy Optimization (GRPO) is introduced in DeepSeekMath [188], and
used in other DeepSeek works, e.g. DeepSeek-V3 [189] and DeepSeek-R1 [61]. GRPO can
be viewed as a PPO-inspired algorithm with a very similar surrogate loss, but it avoids
learning a value function with another copy of the original policy language model (or another

87
checkpoint for initialization). This brings two posited benefits:
1. Avoiding the challenge of learning a value function from a LM backbone, where research
hasn’t established best practices.
2. Saves memory by not needing to keep the extra set of model weights in memory (going
from needing the current policy, the reference policy, and a value function, to just the
first two copies).
GRPO does this by simplifying the value estimation and assigning the same value to every
token in the episode (i.e. in the completion to a prompt, each token gets assigned the
same value rather than discounted rewards in a standard value function) by estimating the
advantage or baseline. The estimate is done by collecting multiple completions (ai ) and
rewards (ri ), i.e. a Monte Carlo estimate, from the same initial state / prompt (s).
To state this formally, the GRPO objective is very similar to the PPO objective above. For
GRPO, the objective (or loss) is accumulated over a group of completions {a1 , a2 , ..., aG } to
a given prompt s. Here, we show the GRPO objective:

G      
1 X πθ (ai |s) πθ (ai |s)
J(θ) = min Ai , clip , 1 − ε, 1 + ε Ai − βDKL (πθ ||πref ) .
G i=1 πθold (ai |s) πθold (ai |s)
(58)
Note that relative to PPO, the standard implementation of GRPO includes the KL distance
in the loss. As above, we can expand this into a per-token computation:

G |ai |      
1 X 1 X πθ (ai,t |si ) πθ (ai,t |si )
J(θ) = min Ai,t , clip , 1 − ε, 1 + ε Ai,t − βDKL (πθ (·|si )||πref (·|si ))
G i=1 |ai | t=1 πθold (ai,t |si ) πθold (ai,t |si )
(59)
With the advantage computation for the completion index i:

ri − mean(r1 , r2 , · · · , rG )
Ai = . (60)
std(r1 , r2 , · · · , rG )

Intuitively, the GRPO update is comparing multiple answers to a single question within a
batch. The model learns to become more like the answers marked as correct and less like
the others. This is a very simple way to compute the advantage, which is the measure of
how much better a specific action is than the average at a given state. Relative to PPO,
REINFORCE, and broadly RLHF performed with a reward model rating (relative to output
reward), GRPO is often run with a far higher number of samples per prompt because the
advantage is entirely about the relative value of a completion to its peers from that prompt.
Here, the current policy generates multiple responses to a given prompt, and the group-wise
GRPO advantage estimate is given valuable context. PPO and vanilla policy-gradient
algorithms were designed to accurately estimate the reward of every completion (in fact,
more completions can do little to improve the value estimate in some cases). GRPO and
its variants are particularly well-suited to modern language model tools, where multiple

88
completions to a given prompt is very natural (especially when compared to, e.g., multiple
actions from a set environment state in a robotic task).
The advantage computation for GRPO has trade-offs in its biases. The normalization by
standard deviation is rewarding questions in a batch that have a low variation in answer
correctness. For questions with either nearly all correct or all incorrect answers, the standard
deviation will be lower and the advantage will be higher. [185] proposes removing the standard
deviation term given this bias, but this comes at the cost of down-weighing questions that
were all incorrect with a few correct answers, which could be seen as valuable learning signal
for the model. Those high-variance prompts can be exactly the hardest cases, where only a
few sampled completions find the correct answer and provide a strong training signal.
eq. 60 is the implementation of GRPO when working with outcome supervision (either a
standard reward model or a single verifiable reward) and a different implementation is needed
with process supervision. In this case, GRPO computes the advantage as the sum of the
normalized rewards for the following reasoning steps.
Finally, GRPO’s advantage estimation can also be applied without the PPO clipping to
more vanilla versions of policy gradient (e.g. REINFORCE), but it is not the canonical form.
As an example of how these algorithms are intertwined, we can show that the advantage
estimation in a variant of GRPO, Dr. GRPO (GRPO Done Right) [185], is equivalent to the
RLOO estimation (which uses the average reward of other samples as its baseline) up to a
constant scaling factor (which normally does not matter due to implementation details to
normalize the advantage). Dr. GRPO removes the standard deviation normalization term
from eq. 60 – note that this also scales the advantage up, which is equivalent to increasing
the GRPO learning rate on samples with a variance in answer scores. This addresses a bias
towards questions with low reward variance – i.e. almost all the answers are right or wrong –
but comes at a potential cost where problems where just one sample gets the answer right
are important to learn from. The Dr. GRPO advantage for completion i within a group of
size G is defined as:

G
1 X
Ãi = ri − mean(r1 , r2 , · · · , rG ) = ri − rj (61)
G j=1

Here, in the same notation, we can recall the RLOO advantage estimation as:

G
1 X
ARLOO
i = ri − rj (62)
G−1
j=1,i̸=j

G
Thus, if we multiply the Dr. GRPO advantage definition by G−1 we can see a scaled
equivalence:

89
  
G
G G  1 X
Ãi = ri − rj 
G−1 G−1 G j=1
G
G 1 X
= ri − rj
G−1 G − 1 j=1
G
G 1 X 1
= ri − rj − ri
G−1 G−1 G−1 (63)
j=1,j̸=i
  G
G 1 1 X
= ri − − rj
G−1 G−1 G−1
j=1,j̸=i
G
1 X
= ri − rj
G−1
j=1,j̸=i

= ARLOO
i

11.2 Implementation
Compared to the original Deep RL literature where many of these algorithms were developed,
implementing RL for optimizing language models or other large AI models requires many
small implementation details. In this section, we highlight some key factors that differentiate
the implementations of popular algorithms.
There are many other small details that go into this training. For example, when doing
RLHF with language models a crucial step is generating text that will then be rated by the
reward model. Under normal circumstances, the model should generate an end-of-sequence
(EOS) token indicating it finished generating, but a common practice is to put a hard cap
on generation length to efficiently utilize infrastructure. A failure mode of RLHF is that
the model is regularly truncated in its answers, driving the ratings from the reward model
out-of-distribution and to unpredictable scores. The solution to this is to only run reward
model scoring on the eos_token, and to otherwise assign a penalty to the model for generating
too long.
The popular open-source tools for RLHF have a large variance in implementation details
across the algorithms (see table 10 in [190]). Some decisions not covered here include:
• Value network initialization: The internal learned value network used by PPO and
other similar algorithms can be started from a different model of the same architecture
or randomly selected weights. This can have a large impact on performance. The
standard established in InstructGPT [3] (and re-used in Tülu 3 for its work on RLVR
[6]) is to initialize the value network from the reward model used during RLHF. Others
have used the previous checkpoint to RLHF training (normally an SFT model) with a
value head appened randomly initialized, or fully re-initialized language models (less
common as it will take longer for RLHF to converge, but possible).
• Reward normalization, reward whitening, and/or advantage whitening:
Normalization bounds all the values from the RM (or environment) to be between 0
and 1, which can help with learning stability. Whitening goes further by transforming

90
 rewards or advantage estimates to have zero mean and unit variance, providing an
even stronger boost to stability.
• Different KL estimators: With complex language models, precisely computing the
KL divergence between models can be complex, so multiple approximations are used
to substitute for an exact calculation [164].
• KL controllers: Original implementations of PPO and related algorithms had dy-
namic controllers that targeted specific KLs and changed the penalty based on recent
measurements. Most modern RLHF implementations use static KL penalties, but this
can also vary.
For more details on implementation details for RLHF, see [191]. For further information on
the algorithms, see [192].

11.2.1 Policy Gradient Basics

A simple implementation of policy gradient, using advantages to estimate the gradient to
prepare for advanced algorithms such as PPO and GRPO follows:
pg_loss = - advantages * ratio

Ratio here is the (per-token) probability ratio (often computed from a log-probability
difference) of the new policy model probabilities relative to the reference model.
In order to understand this equation, it is good to understand different cases that can fall
within a batch of updates. Remember that we want the loss to decrease as the model gets
better at the task.
Case 1: Positive advantage, so the action was better than the expected value of the state. We
want to reinforce this. In this case, the model will make this more likely with the negative
sign. To do so, it’ll increase the logratio. A positive logratio, or sum of log probabilities of
the tokens, means that the model is more likely to generate those tokens.
Case 2: Negative advantage, so the action was worse than the expected value of the state.
This follows very similarly. Here, the loss will be positive if the new model was more likely,
so the model will try to make it so the policy parameters make this completion less likely.
Case 3: Zero advantage, so no update is needed. The loss is zero, don’t change the policy
model.

11.2.2 Loss Aggregation

The question when implementing any policy gradient algorithm with language models is:
How do you aggregate per-token losses into a final scalar loss? Given per-token losses ℓi,t
for sample i at token t, with completion lengths |ai | and batch size B, there are three main
strategies:
Strategy 1: Per-sequence normalization (standard GRPO; also used in some PPO
implementations)

B |ai |
1 X 1 X
L= ℓi,t
B i=1 |ai | t=1

91
Each sequence contributes equally to the batch loss, regardless of length. In code:
# Strategy 1: Per - sequence normalization
sequence_loss = (( per_token_loss * completion_mask ) . sum ( dim =1) / \
completion_mask . sum ( dim =1) ) . mean ()

Strategy 2: Per-token normalization (DAPO [193])

PB P|ai |
i=1 t=1 ℓi,t
L= PB
i=1 |ai |

Each token contributes equally; longer sequences have proportionally more influence on the
gradient. In code:
# Strategy 2: Per - token normalization
token_loss = (( per_token_loss * completion_mask ) . sum () / \
completion_mask . sum () )

Strategy 3: Fixed-length normalization (Dr. GRPO [185])

B |ai |
1 X 1 X
L= ℓi,t
B i=1 Lmax t=1

Normalizes by max sequence length Lmax , equalizing the per-token scale across sequences
while still letting longer sequences contribute more total gradient because they contain more
active tokens.
Note that completion_mask in the code above is a matrix of 1s and 0s, where the prompt
tokens are masked out (0s) because we don’t want the model to learn from predicting prompt
tokens.

[Link] Why does this matter? Intuitively, per-sequence normalization (Strategy 1)

seems best since we care about outcomes, not individual tokens. However, this introduces
subtle biases based on sequence length, which can cause the model to overthink of down-
weight strategies that naturally need to use more tokens, depending on the direction of the
bias. Consider two sequences of different lengths with per-token losses:
seq_1_losses = [1 , 1 , 1 , 1 , 10] # 5 tokens , mean = 2.8
seq_2_losses = [1 , 1 , 1 , 1 , 1 , 1 , 1 , 1 , 1 , 10] # 10 tokens , mean =
1.9

With Strategy 1 (per-sequence): The batch loss is (2.8 + 1.9)/2 = 2.35, and crucially, each
token in the short sequence receives a larger gradient than tokens in the long sequence.
With Strategy 2 (per-token): The batch loss is (14 + 19)/15 = 2.2, and all tokens receive
equal gradient magnitude.
With Strategy 3 (fixed-length with Lmax = 10): The short sequence contributes 1.4 and
the long sequence contributes 1.9, balancing per-token gradients while still weighting by
sequence.

92
For a more complete example showing how these strategies affect gradients, see the script
below.
from typing import Optional
import torch

def masked_mean ( values : torch . Tensor , mask : torch . Tensor , axis :

Optional [ int ] = None ) -> torch . Tensor :
" " " Compute ␣ mean ␣ of ␣ tensor ␣ with ␣ masked ␣ values . " " "
if axis is not None :
return ( values * mask ) . sum ( axis = axis ) / mask . sum ( axis = axis )
else :
return ( values * mask ) . sum () / mask . sum ()

def masked_sum (
values : torch . Tensor ,
mask : torch . Tensor ,
axis : Optional [ int ] = None ,
constant_normalizer : float = 1.0 ,
) -> torch . Tensor :
" " " Compute ␣ sum ␣ of ␣ tensor ␣ with ␣ masked ␣ values . ␣ Use ␣ a ␣ constant ␣ to ␣
normalize . " " "
if axis is not None :
return ( values * mask ) . sum ( axis = axis ) / constant_normalizer
else :
return ( values * mask ) . sum () / constant_normalizer

ratio = torch . tensor ([

[1. , 1 , 1 , 1 , 1 , 1 , 1 ,] ,
[1 , 1 , 1 , 1 , 1 , 1 , 1 ,] ,
] , requires_grad = True )

advs = torch . tensor ([

[2 , 2 , 2 , 2 , 2 , 2 , 2 ,] ,
[2 , 2 , 2 , 2 , 2 , 2 , 2 ,] ,
])

masks = torch . tensor ([

# generation 1: 4 tokens
[1 , 1 , 1 , 1 , 0 , 0 , 0 ,] ,
# generation 2: 7 tokens
[1 , 1 , 1 , 1 , 1 , 1 , 1 ,] ,
])

max_gen_len = 7

ma sked_mean_result = masked_mean ( ratio * advs , masks , axis =1)

m a s k ed _m e an _ to ke n _l e ve l = masked_mean ( ratio , masks , axis = None )
mas ked_sum_result = masked_sum ( ratio * advs , masks , axis =1 ,
constant_normalizer = max_gen_len )

print ( " masked_mean " , masked_mean_result )

93
print ( " masked_sum " , masked_sum_result )
print ( " m as k ed _ me an _ to k en _l e ve l " , m as k ed _ me an _ to k en _ le ve l )

# masked_mean tensor ([2. , 2.] , grad_fn = < DivBackward0 >)

# masked_sum tensor ([1.1429 , 2.0000] , grad_fn = < DivBackward0 >)
# m as k ed _ me an _ to k en _l e ve l tensor (1. , grad_fn = < DivBackward0 >)

ma sked_mean_result . mean () . backward ()

print ( " ratio . grad " , ratio . grad )
ratio . grad . zero_ ()
# ratio . grad tensor ([[0.2500 , 0.2500 , 0.2500 , 0.2500 , 0.0000 , 0.0000 ,
0.0000] ,
# [0.1429 , 0.1429 , 0.1429 , 0.1429 , 0.1429 , 0.1429 , 0.1429]])

mas ked_sum_result . mean () . backward ()

print ( " ratio . grad " , ratio . grad )
ratio . grad . zero_ ()
# ratio . grad tensor ([[0.1429 , 0.1429 , 0.1429 , 0.1429 , 0.0000 , 0.0000 ,
0.0000] ,
# [0.1429 , 0.1429 , 0.1429 , 0.1429 , 0.1429 , 0.1429 , 0.1429]])

m a s k ed _m e an _ to ke n _l e ve l . mean () . backward ()
print ( " ratio . grad " , ratio . grad )
# ratio . grad tensor ([[0.0909 , 0.0909 , 0.0909 , 0.0909 , 0.0000 , 0.0000 ,
0.0000] ,
# [0.0909 , 0.0909 , 0.0909 , 0.0909 , 0.0909 , 0.0909 , 0.0909]])

The output shows that with Strategy 1 (masked_mean), the short sequence has larger per-
token gradients (0.25) than the long sequence (0.14). Strategies 2 and 3 equalize the
per-token gradients across sequences. Note that these results can vary substantially if
gradient accumulation is used, where the gradients are summed across multiple minibatches
before taking a backward step—in this case, the balance between shorter and longer sequences
can flip.
In practice, the best strategy depends on the specific training setup. Often in RLHF the
method with the best numerical stability or the least variance in loss is preferred.

[Link] Related: MDP vs Bandit Framing The choice of loss aggregation connects
to a deeper distinction in how we frame the RL problem. The MDP (token-level) view
treats each token at as an action with state st being the running prefix. In practice, this is
the framing used when we compute token-level advantages with a learned value function
V (st ) (e.g., GAE [179]) and apply KL penalties per token. PPO with a learned value network
is the canonical example [183].
In contrast, the bandit (sequence-level) view treats the whole completion as a single
action with one scalar reward R. In code, this means computing a sequence-level advantage
Aseq and broadcasting it to all tokens. RLOO and GRPO-style advantages are often used in
this bandit-style setting [182] [178] [188]. Direct alignment methods like DPO and A-LoL
also define sequence-level objectives, although they are not policy-gradient estimators [194].
Note that many GRPO implementations use a bandit-style advantage and add a separate

94
per-token KL term in the loss, while many PPO/RLOO implementations fold KL into the
reward before computing advantages; both conventions exist in practice.

11.2.3 Asynchronicity
The default implementation for policy-gradient algorithms is what is called on-policy
execution, where the actions (generations) taken by the agent (language model) are scored
before updating the model. The theoretical derivations of policy-gradient rely on all actions
being exactly on-policy where the model is always up to date with the results from the
latest trials/roll-outs. In practice, maintaining exact on-policy execution substantially slows
training [195]—and perfect synchronization is technically impossible regardless. Therefore,
all of the recent empirical results with language models tend to be slightly outside of the
theoretical proofs. What happens in practice is designing the algorithms and systems for
what actually works.

Figure 16: A comparison of the generation-update phases for synchronous or asynchronous

RL training following Noukhovitch et al. 2024.

The common solution used is to constantly run inference and training on separate GPU
nodes with software designed to efficiently run both, as shown in the bottom of fig. 16.
Common practice in popular open-source RL tools for language models is to use a distributed
process management library such as Ray to hand information off between the policy-gradient
learning loop and the inference loop using an efficient inference engine, e.g., VLLM. In
these setups, the GPUs dedicated to taking the RL steps are called the “leaners” and the
GPUs dedicated to sampling from the language model are called the “actors” The primary
challenges faced when making training more asynchronous are keeping training stable and
maintaining learning signal.

95
Figure 17: An example distributed RL system, where two queues are managed to pass data
to the learner and actor GPUs, which can both be synchonized with a distributed computing
library such as Ray. Olmo Team 2025, license CC-BY.

These systems are designed and implemented with the presumption that nearly on-policy
data is good enough for stable learning. Here, the generation and update phases can easily
be synced to avoid idle compute on either piece of the training system, which would be
passing model weights from the leaners to the actors in fig. 17. With reasoning models,
the extremely long inference characteristics of problems requiring 10K to 100K+ tokens
per answer makes the generation of roll-outs a far stronger bottleneck. A common problem
when training reasoning models on more synchronous RL infrastructure is that an answer to
one prompt in the batch can take substantially more time to generate (either through more
tokens or more tool calls), resulting in the majority of the allocated compute being idle until
it completes. A second solution to this length mismatch issue, called sequence-level packing,
is to stack shorter samples within a batch with clever masking to enable continued roll-outs
from the model and better distribute length normalization across samples within a batch.
The full complexity of distributed RL infrastructure is out of scope for this book, as it can
cause many other subtle issues that slow down training or cause instability.
Following the emergence of these reasoning models, further interest has been taken to make
the training and inference loops fully off-policy, where training batches for the policy gradient
updates are filled with the most recently completed roll-outs across multiple instances
generating answers [196] [197]. Fully asynchronous training would also enable scaling RL
training runs across multiple datacenters more easily due to the option of increasing the
time between weight syncs between the learner node (taking policy gradient steps) and the
actor (trying to solve problems) [198].
Related methods are exploring fully off-policy policy gradient algorithms [199].

96
11.2.4 Proximal Policy Optimization
There are many, many implementations of PPO available. The core loss computation is
shown below. Crucial to stable performance is also the value computation, where multiple
options exist (including multiple options for the value model loss).
Note that the reference policy (or old logprobs) here are from the time the generations were
sampled and not necessarily the reference policy. The reference policy is only used for the
KL distance constraint/penalty.
# B : Batch Size , L : Sequence Length , G : Num of Generations
# Apply KL penalty to rewards
rewards = rewards - self . beta * per_token_kl # Shape : ( B *G , L )

# Get value predictions

values = value_net ( completions ) # Shape : ( B *G , L )

# Compute simple advantages

advantages = rewards - values . detach () # Shape : ( B *G , L )
# Note : We detach the value network here to not update the parameters
of
# the value function when computing the policy - gradient loss

# Normalize advantages ( optional but stable )

advantages = ( advantages - advantages . mean () ) / ( advantages . std () + 1e
-8)

# Compute probability ratio between new and old policies

ratio = torch . exp ( new_per_token_logps - per_token_logps ) # Shape : ( B *
G, L)

# PPO clipping objective

eps = self . cliprange # e . g . 0.2
pg_losses1 = - advantages * ratio # Shape : ( B *G , L )
pg_losses2 = - advantages * torch . clamp ( ratio , 1.0 - eps , 1.0 + eps ) #
Shape : ( B *G , L )
pg_loss_max = torch . max ( pg_losses1 , pg_losses2 ) # Shape : ( B *G , L )

# Simple value function loss

vf_loss = 0.5 * (( rewards - values ) ** 2) # Shape : ( B *G , L )

# Combine policy and value losses

per_token_loss = pg_loss_max + self . vf_coef * vf_loss # Shape : ( B *G ,
L)

# Apply completion mask and compute final loss

loss = (( per_token_loss * completion_mask ) . sum ( dim =1) /
completion_mask . sum ( dim =1) ) . mean ()
# Scalar

# Compute metrics for logging

with torch . no_grad () :
# Compute clipping fraction

97
 clip_frac = (( pg_losses2 > pg_losses1 ) . float () * completion_mask ) .
sum () / completion_mask . sum ()

# Compute approximate KL
approx_kl = 0.5 * (( new_per_token_logps - per_token_logps ) **2) .
mean ()

# Compute value loss for logging

value_loss = vf_loss . mean ()

The core piece to understand with PPO is how the policy gradient loss is updated. Focus on
these three lines:
pg_losses1 = - advantages * ratio # Shape : ( B *G , L )
pg_losses2 = - advantages * torch . clamp ( ratio , 1.0 - eps , 1.0 + eps ) #
Shape : ( B *G , L )
pg_loss_max = torch . max ( pg_losses1 , pg_losses2 ) # Shape : ( B *G , L )

pg_losses1 is the vanilla advantage-weighted policy gradient loss. pg_losses2 applies the
same formula but with the probability ratio clamped to the range [1 − ε, 1 + ε], limiting how
much the policy can change in a single update.
The key insight is taking [Link] of the two losses. Because we’re minimizing a negative
loss (recall the negative sign in front of advantages), taking the maximum selects the
more pessimistic gradient—the one that produces a smaller policy update. When the
advantage is positive (good action), clipping prevents the policy from increasing that action’s
probability too aggressively. When the advantage is negative (bad action), clipping prevents
over-correction in the other direction.
By clamping the log-probability ratio, PPO bounds how far the policy can drift from the
version that generated the training data, stabilizing learning without requiring an explicit
trust region computation.
The code above also shows PPO learning a value function alongside the policy, which adds
implementation complexity, but the clipped objective is the core mechanism.

[Link] PPO/GRPO simplification with 1 gradient step per sample (no clip-
ping) PPO (and GRPO) implementations can be handled much more elegantly if the
hyperparameter “number of gradient steps per sample” is equal to 1. Many typical values for
this are from 2-4 or higher. In the main PPO or GRPO equations, see eq. 54, the “reference”
policy is the previous parameters – those used to generate the completions or actions. Thus,
if only one gradient step is taken, πθ = πθold , and the update rule reduces to the following
(the notation []∇ indicates a stop gradient):

G  
1 X πθ (ai |s)
J(θ) = Ai − βDKL (πθ ||πref ) . (64)
G i=1 [πθ (ai |s)]∇

This leads to PPO or GRPO implementations where the second policy gradient and clipping
logic can be omitted, making the optimizer far closer to standard policy gradient.

98
11.2.5 Group Relative Policy Optimization
The DeepSeekMath paper describes some implementation details of GRPO that differ from
PPO [188], especially if comparing to a standard application of PPO from Deep RL rather
than language models. For example, the KL penalty within the RLHF optimization (recall
the KL penalty is also used when training reasoning models on verifiable rewards without
a reward model) is applied directly in the loss update rather than to the reward function.
Where the standard KL penalty application for RLHF is applied as r = rθ − βDKL , the
GRPO implementation is along the lines of:

L = Lpolicy gradient + β ∗ DKL

Though, there are multiple ways to implement this. Traditionally, the KL distance is
computed with respect to each token in the completion to a prompt s. For reasoning training,
multiple completions are sampled from one prompt, and there are multiple prompts in one
batch, so the KL distance will have a shape of [B, L, N], where B is the batch size, L is the
sequence length, and N is the number of completions per prompt.
Putting it together, using the first loss accumulation, the pseudocode can be written as
below.
# B : Batch Size , L : Sequence Length , G : Number of Generations
# Compute grouped - wise rewards # Shape : (B ,)
m e a n_grouped_rewards = rewards . view ( -1 , self . num_generations ) . mean ( dim
=1)
s t d_grouped_rewards = rewards . view ( -1 , self . num_generations ) . std ( dim
=1)

# Normalize the rewards to compute the advantages

m e a n_grouped_rewards = mean_grouped_rewards . repeat_interleave ( self .
num_generations , dim =0)
s t d_grouped_rewards = std_grouped_rewards . repeat_interleave ( self .
num_generations , dim =0)
# Shape : ( B *G ,)

# Compute advantages
advantages = ( rewards - mean_grouped_rewards ) / ( std_grouped_rewards +
1e -4)
advantages = advantages . unsqueeze (1)
# Shape : ( B *G , 1)

# Compute probability ratio between new and old policies

ratio = torch . exp ( new_per_token_logps - per_token_logps ) # Shape : ( B *
G, L)

# PPO clipping objective

eps = self . cliprange # e . g . 0.2
pg_losses1 = - advantages * ratio # Shape : ( B *G , L )
pg_losses2 = - advantages * torch . clamp ( ratio , 1.0 - eps , 1.0 + eps ) #
Shape : ( B *G , L )
pg_loss_max = torch . max ( pg_losses1 , pg_losses2 ) # Shape : ( B *G , L )

99
# important to GRPO -- PPO applies this in reward traditionally
# Combine with KL penalty
per_token_loss = pg_loss_max + self . beta * per_token_kl # Shape : ( B *G
, L)

# Apply completion mask and compute final loss

loss = (( per_token_loss * completion_mask ) . sum ( dim =1) /
completion_mask . sum ( dim =1) ) . mean ()
# Scalar

# Compute core metric for logging ( KL , reward , etc . also logged )

with torch . no_grad () :
# Compute clipping fraction
clip_frac = (( pg_losses2 > pg_losses1 ) . float () * completion_mask ) .
sum () / completion_mask . sum ()

# Compute approximate KL
approx_kl = 0.5 * (( new_per_token_logps - per_token_logps ) **2) .
mean ()

For more details on how to interpret this code, see the PPO section above. The core
differences from the PPO example are:
• Advantage computation: GRPO normalizes rewards relative to the group (mean
and std across generations for the same prompt) rather than using a learned value
function as baseline.
• No value network: GRPO removes the value model entirely, eliminating vf_loss
and the associated complexity.
• KL penalty placement: GRPO adds the KL penalty directly to the loss rather than
subtracting it from the reward (this is the standard implementation, but more versions
exist on how the KL is applied).

[Link] RLOO vs. GRPO The advantage updates for RLOO follow very closely to
GRPO, highlighting the conceptual similarity of the algorithm when taken separately from
the PPO style clipping and KL penalty details. Specifically, for RLOO, the advantage is
computed relative to a baseline that is extremely similar to that of GRPO – the completion
reward relative to the others for that same question. Concisely, the RLOO advantage estimate
follows as (expanded from TRL’s implementation):
# rloo_k --> number of completions per prompt
# rlhf_reward --> Initially a flat tensor of total rewards for all
completions . Length B = N x k
rlhf_reward = rlhf_reward . reshape ( rloo_k , -1) #
# Now , Shape : (k , N ) , each column j contains the k rewards for prompt
j.

baseline = ( rlhf_reward . sum (0) - rlhf_reward ) / ( rloo_k - 1)

# baseline --> Leave - one - out baseline rewards . Shape : (k , N )
# baseline [i , j ] is the avg reward of samples i ' != i for prompt j .

advantages = rlhf_reward - baseline

100
# advantages --> Same Shape : (k , N )

advantages = advantages . flatten () # Same shape as original tensor

The rest of the implementation details for RLOO follow the other trade-offs of implementing
policy-gradient.

11.3 Auxiliary Topics

In order to master the application of policy-gradient algorithms, there are countless other
considerations. Here we consider some of the long-tail of complexities in successfully deploying
a policy-gradient RL algorithm.

11.3.1 Comparing Algorithms

Here’s a summary of some of the discussed material (and foreshadowing to coming material
on Direct Preference Optimization) when applied to RLHF. Here, on- or off-policy indicates
the derivation (where most are applied slightly off-policy in practice). A reference policy
here indicates if it is required for the optimization itself, rather than for a KL penalty.

Table 5: Comparing policy gradient algorithms (and friends).

Reward Value Reference

Method Type Model Function Policy Core Loss L(θ)
PT
REINFORCE
On- Yes No No − T1 t=1 log π
θ (at |
policy st ) Gt − b(st )
1
PK P
RLOO On- Yes No No −K  i=1 t logP πθ (ai,t |
policy 1
si,t ) Ri − K−1 j̸=i Rj
PT
PPO On- Yes Yes Yes − T1 t=1 min

ρt At , clip(ρt , 1−
policy ε, 1 + ε)At ; ρt =
πθ (at |st )
πθold (at |st )
1
PG
GRPO On- Yes No Yes −G i=1 minρi Ai , clip(ρi , 1−


policy ε, 1 + ε)Ai ; ρi =
πθ (ai |s)
πθ (ai |s) , Ai =
old
ri −mean(r1:G )
std(r1:G ) 

DPO Off- No No Yes −E(x,yw ,yl ) log σ β[∆ log πθ (x) − ∆ log πref (x)]
policy

11.3.2 Generalized Advantage Estimation (GAE)

Generalized Advantage Estimation (GAE) is an alternate method to compute the advantage
for policy gradient algorithms [179] that better balances the bias-variance tradeoff. Traditional
single-step advantage estimates can introduce too much bias, while using complete trajectories
can suffer from high variance. GAE computes an exponentially-weighted average of multi-step
advantage estimates, where the λ hyperparameter controls the bias-variance tradeoff—ranging
from single-step TD (λ = 0) to full trajectory returns (λ = 1).

101
Advantage estimates can take many forms, but we can define a n step advantage estimator
(similar to the TD residual at the beginning of the chapter) as follows:



 rt + γV (st+1 ) − V (st ), n=1

rt + γrt+1 + γ 2 V (st+2 ) − V (st ),

n=2
(n)
Ât = . (65)
..



rt + γrt+1 + γ 2 rt+2 + · · · − V (st ), n=∞


Here a shorter n will have lower variance but higher bias as we are attributing more learning
power to each trajectory – it can overfit. GAE attempts to generalize this formulation into a
weighted multi-step average instead of a specific n. To start, we must define the temporal
difference (TD) residual of predicted value.

δtV = rt + γV (st+1 ) − V (st ) (66)

To utilize this, we introduce another variable λ as the GAE mixing parameter. This folds
into an exponential decay of future advantages we wish to estimate:

GAE(γ,λ) (1) (2) (3)

Ât = (1 − λ)(Ât + λÂt + λ2 Ât + · · · )
= (1 − λ)(δtV + λ(δtV + γδt+1
V
) + λ2 (δtV + γδt+1
V V
+ γ 2 δt+2 ) + ···)
V 2 V 2
= (1 − λ)(δt (1 + λ + λ + · · · ) + γδt+1 (λ + λ + · · · ) + · · · ) (67)
1 λ
= (1 − λ)(δtV 1−λ V
+ γδt+1 1−λ + · · · )
P∞ l V
= l=0 (γλ) δt+l

Intuitively, this can be used to average multi-step estimates of Advantage in an elegant

fashion. An example implementation is shown below:
# GAE ( token - level ) for LM RLHF
#
# B : Batch Size
# L : Length
# Inputs :
# rewards : (B , L ) post - KL per - token rewards
# values : (B , L ) current V_theta ( s_t )
# done_mask : (B , L ) 1.0 at terminal token ( EOS or penalized trunc ) ,
else 0.0
# gamma : float ( often 1.0) , lam : float in [0 ,1]
B , L = rewards . shape
advantages = torch . zeros_like ( rewards )
next_v = torch . zeros (B , device = rewards . device , dtype = rewards . dtype )
gae = torch . zeros (B , device = rewards . device , dtype = rewards . dtype )

for t in reversed ( range ( L ) ) :

not_done = 1.0 - done_mask [: , t ]
delta = rewards [: , t ] + gamma * not_done * next_v - values [: , t ]
gae = delta + gamma * lam * not_done * gae
advantages [: , t ] = gae
next_v = values [: , t ]

102
targets = advantages + values # y_t for value regression
advantages = advantages . detach () # for policy loss

For further reading, see [200].

11.3.3 Double Regularization

We’ve seen in this chapter two types of regularization. One is built into algorithms like PPO
with step-size constraints, and the other is a KL divergence based distance penalty relative
to the start of the optimization.
Many popular policy gradient algorithms from Deep Reinforcement Learning, including PPO
and its predecessors, originated due to the need to control the learning process of the agent.
In RLHF, as discussed extensively in Chapter 8 on Regularization and in Chapter 4 on
Problem Formulation, there is a built-in regularization term via the distance penalty relative
to the original policy one is finetuning. In this view, a large part of the difference between
algorithms like PPO (which have internal step-size regularization) and REINFORCE (which
is simpler, and to which PPO reduces under certain hyperparameters) is far less meaningful
for finetuning language models than training agents from scratch.
In PPO, the objective that handles capping the step-size of the update is known as the
surrogate objective. To monitor how much the PPO regularization is impacting updates in
RLHF, one can look at the clip fraction variable in many popular implementations, which is
the percentage of samples in the batch where the gradients are clipped by this regularizer in
PPO. These gradients are reduced to a maximum value.
In practice with language models, algorithms like PPO and GRPO are run with only one
gradient step per batch, which means that the PPO-native regularization is never applied
(as clipping can only occur within a batch when the policy changes substantially) and the
KL distances penalties predominate.

11.3.4 Further Reading

As RLHF has cemented itself at the center of modern post-training, other policy-gradient
RL algorithms and RL algorithms generally have been proposed to improve the training
process, but they have not had a central role in governing best practices. Examples for
further reading include:
• Pairwise Proximal Policy Optimization (P3O) [201] uses pairwise data directly
in a PPO-style policy update without learning an intermediate reward model.
• Off-policy policy-gradient algorithms could enable further asynchronous training, such
as Contrastive Policy Gradient (CoPG) [202] (a generalization of the direct
alignment algorithm IPO and vanilla policy gradient), which was used by Cohere for
their Command A model [58].
• Other implementations of REINFORCE algorithms have been designed for language
models, such as ReMax [203], which implements a baseline normalization designed
specifically to accommodate the sources of uncertainty from reward model inference.
• Some foundation models, such as Apple Intelligence Foundation Models [204] or Kimi
k1.5 reasoning model [205], have used variants of Mirror Descent Policy Opti-
mization (MDPO) [206]. Research is still developing further on the fundamentals

103
 here [207], but Mirror Descent is an optimization method rather than directly a policy
gradient algorithm. What is important here is that it is substituted in very similarly
to existing RL infrastructure.
• Decoupled Clip and Dynamic sAmpling Policy Optimization (DAPO) pro-
poses 4 modifications to GRPO to better suit reasoning language models, where long
traces are needed and new, underutilized tokens need to be increased in probability
[193]. The changes are: 1, have two different clip hyperparameters, εlow and εhigh , so
clipping on the positive side of the logratio can take bigger steps for better exploration;
2, dynamic sampling, which removes all samples with reward = 0 or reward = 1 for all
samples in the batch (no learning signal); 3, use the per token loss as discussed above
in Implementation: GRPO; and 4, a soft penalty on samples that are too long to avoid
trying to learn from truncated answers.
• Value-based Augmented Proximal Policy Optimization (VAPO) [208] combines
optimizations from DAPO (including clip-higher, token level policy-gradient, and
different length normalization) with insights from Value-Calibrated PPO [209] to
pretrain the value function and length-adaptive GAE to show the promise of value
base methods relative to GRPO.

104
12 Direct Alignment Algorithms
Direct Alignment Algorithms (DAAs) allow one to update models to solve the same RLHF
objective, shown again in eq. 68, without ever training an intermediate reward model or
using reinforcement learning optimizers. It solves the same preference learning problem
we’ve been studying (with literally the same data!), in order to make language models more
aligned, smarter, and easier to use. The lack of a reward model and online optimization
makes DAAs far simpler to implement, reducing compute spent during training and making
experimentation easier. This chapter details the complex mathematics that were done to
derive these algorithms, and then shows that the sometimes tedious derivations result in
simple implementations.
The most prominent DAA and one that catalyzed an entire academic movement of aligning
language models is Direct Preference Optimization (DPO) [20]. At its core, DPO is using
gradient ascent to solve the same constrained RLHF objective (see Chapter 4):

max Ex∼D Ey∼π(y|x) [rθ (x, y)] − βDKL (π(y|x)∥πref (y|x)) (68)
π

Since its release in May of 2023, after a brief delay where the community figured out the
right data and hyperparameters to use DPO with (specifically, surprisingly low learning
rates), many popular models have used DPO or its variants, from Zephyr-β kickstarting it
in October of 2023 [21], Llama 3 Instruct [24], Tülu 2 [22] and 3 [6], Nemotron 4 340B [25],
and others. Technically, Sequence Likelihood Calibration (SLiC-HF) was the first, modern
direct alignment algorithm released [210], but it did not catch on due to a combination of
factors (unwinding the adoption of research methods is always a tricky task).
The most impactful part of DPO and DAAs is lowering the barrier of entry to experimenting
with language model post-training – it uses less compute, is easier to implement from scratch,
and is easier to get working on both toy and production examples.
Throughout this chapter, we use x to denote prompts and y to denote completions. This
notation is common in the language model literature, where methods operate on full prompt-
completion pairs rather than individual tokens.

12.1 Direct Preference Optimization (DPO)

Here we explain intuitions for how DPO works and re-derive the core equations fully.

12.1.1 How DPO Works

DPO at a surface level is directly optimizing a policy to solve the RLHF objective. The loss
function for this, which we will revisit below in the derivations, is a pairwise relationship of
log-probabilities. The loss function derived from a Bradley-Terry reward model follows:
  
πθ (yc | x) πθ (yr | x)
LDPO (πθ ; πref ) = −E(x,yc ,yr )∼D log σ β log − β log (69)
πref (yc | x) πref (yr | x)

Throughout, β is a hyperparameter balancing the reward optimization to the KL distance

between the final model and the initial reference (i.e. balancing over-optimization, as a

105
crucial hyperparameter to using DPO correction). This relies on the implicit reward for DPO
training that replaces using an external reward model, which is a log-ratio of probabilities:

πr (y | x)
r(x, y) = β log (70)
πref (y | x)

where πr (y | x) is the exact, optimal reward policy that we are solving for. This comes from
deriving the Bradley-Terry reward with respect to an optimal policy (shown in eq. 84), as
shown in the Bradley-Terry model section of Chapter 7. Essentially, the implicit reward
model shows “the probability of human preference data in terms of the optimal policy rather
than the reward model.”
Let us consider the loss shown in eq. 69 that the optimizer must decrease. Here, the loss will
be lower when the log-ratio of the chosen response is bigger than the log-ratio of the rejected
response (normalized by the reference model). In practice, this is a sum of log-probabilities
of the model across the sequence of tokens in the data presented. Hence, DPO is increasing
the delta in probabilities between the chosen and rejected responses.
With the reward in eq. 70, we can write the gradient of the loss to further interpret what is
going on:

∇θ LDPO (πθ ; πref ) = −βE(x,yc ,yr )∼D [σ (rθ (x, yr ) − rθ (x, yc )) (∇θ log π(yc | x) − ∇θ log π(yr | x))]
(71)
Here, the gradient solves the above objective by doing the following:
• The first term within the sigmoid function, σ(·), creates a weight of the parameter
update from 0 to 1 that is higher when the reward estimate is incorrect. When the
rejected sample is preferred over the chosen, the weight update should be larger!
• Second, the terms in the inner brackets [·] increase the likelihood of the chosen response
yc and decrease the likelihood of the rejected yr .
• These terms are weighted by β, which controls how the update balances ordering the
completions correctly relative to the KL distance.
The core intuition is that DPO is fitting an implicit reward model whose corresponding
optimal policy can be extracted in a closed form (thanks to gradient descent and our ML
tools). The closed form of the equation means that it is straightforward to implement the
exact gradient, rather than needing to reach it by proxy of training a reward model and
sampling completions to score. What is often misunderstood is that DPO is learning a
reward model at its core, hence the subtitle of the paper Your Language Model is Secretly a
Reward Model. It is easy to confuse this with the DPO objective training a policy directly,
hence studying the derivations below is good for a complete understanding.
With the implicit reward model learning, DPO is generating an optimal solution to the
RLHF objective given the data in the dataset and the specific KL constraint in the objective
β. Here, DPO solves for the exact policy given a specific KL distance because the generations
are not online as in policy gradient algorithms – a core difference from the RL methods for
preference tuning. In many ways, this makes the β value easier to tune with DPO relative
to online RL methods, but crucially and intuitively the optimal value depends on the model
being trained and the data training it.

106
At each batch of preference data, composed of many pairs of completions ychosen ≻ yrejected ,
DPO takes gradient steps directly towards the optimal solution. It is far simpler than policy
gradient methods.

Figure 18: When DPO first released it sparked a fierce debate in the research community
about how to best do RLHF and preference learning. This meme is a great job capturing
the sentiment, where the debate often felt forced and over the top, but many people both
getting started and in top labs were getting immense benefit out of DPO. DPO simplicity
meme, credit Tom Goldstein.

12.1.2 DPO Derivation

The DPO derivation takes two primary parts. First, the authors show the form of the policy
that optimally solved the RLHF objective used throughout this book. Next, they show how
to arrive at that solution from pairwise preference data (i.e. a Bradley Terry model).

[Link] 1. Deriving the Optimal RLHF Solution To start, we should consider the
RLHF optimization objective once again, here indicating we wish to maximize this quantity:

max Ex∼D Ey∼π(y|x) [rθ (x, y)] − βDKL (π(y|x)∥πref (y|x)) (72)
π

Here, the dual expectation only applies to the sampling to compute the expected reward,

107
as the KL term is still an analytical expression.
h First, let
i us expand the definition of KL-
π(y|x)
divergence. Recall that DKL (π∥πref ) = Ey∼π log πref (y|x) , where the π(y|x) weighting in the
sum becomes the sampling distribution. Since both terms now share the same expectation
over y ∼ π(y|x), we can combine them:
 
π(y|x)
max Ex∼D Ey∼π(y|x) r(x, y) − β log (73)
π πref (y|x)

Next, pull the negative sign out of the difference in brackets. To do this, split it into two
terms:
  
π(y|x)
= max Ex∼D Ey∼π(y|x) [r(x, y)] − β Ex∼D Ey∼π(y|x) log (74)
π πref (y|x)

Then, remove the factor of −1 and β,

  
π(y|x)
= min −Ex∼D Ey∼π(y|x) [r(x, y)] + β Ex∼D Ey∼π(y|x) log (75)
π πref (y|x)

Divide by β and recombine:

  
π(y|x) 1
= min Ex∼D Ey∼π(y|x) log − r(x, y) (76)
π πref (y|x) β

Next, we must introduce a partition function, Z(x):

 
X 1
Z(x) = πref (y|x) exp r(x, y) (77)
y
β

The partition function acts as a normalization factor over the reference policy, summing over
all possible responses y to a prompt x. With this substituted in, we obtain our intermediate
transformation:
 
π(y|x)
min Ex∼D Ey∼π(y|x) log   − log Z(x) (78)
π 1 1
π
Z(x) ref (y|x) exp β r(x, y)

To see how this is obtained, consider the internal part of the optimization in brackets of
eq. 76:

π(y|x) 1
log − r(x, y) (79)
πref (y|x) β

Then, add log Z(x) − log Z(x) to both sides:

108
 π(y|x) 1
= log − r(x, y) + log Z(x) − log Z(x) (80)
πref (y|x) β

Then, we group the terms:

 
π(y|x) 1
= log + log Z(x) − log Z(x) − r(x, y) (81)
πref (y|x) β

With log(x) + log(y) = log(x · y) (and moving Z to the denominator), we get:

π(y|x) 1
= log 1 − log Z(x) − r(x, y) (82)
Z(x) πref (y|x)
β

Next, we expand β1 r(x, y) to log exp β1 r(x, y) and do the same operation to get eq. 78. With
this optimization form, we need to actually solve for the optimal policy π ∗ . To do so, let us
consider the above optimization as a KL distance:
    
1 1
min Ex∼D DKL π(y|x)|| πref (y|x) exp r(x, y) − log Z(x) (83)
π Z(x) β

Since the partition function Z(x) does not depend on the final answer, we can ignore it. This
leaves us with just the KL distance between our policy we are learning and a form relating
the partition, β, reward, and reference policy. The Gibb’s inequality tells this is minimized
at a distance of 0, only when the two quantities are equal! Hence, we get an optimal policy:
 
∗ 1 1
π (y|x) = π(y|x) = πref (y|x) exp r(x, y) (84)
Z(x) β

[Link] 2. Deriving DPO Objective for Bradley Terry Models To start, recall
from Chapter 7 on Reward Modeling and Chapter 6 on Preference Data that a Bradley-Terry
model of human preferences is formed as:

exp (r∗ (x, y1 ))

p∗ (y1 ≻ y2 | x) = (85)
exp (r∗ (x, y1 )) + exp (r∗ (x, y2 ))

By manipulating eq. 84, we can solve for the optimal reward. First, take the logarithm of
both sides:
  
1 1 ∗
log π ∗ (y|x) = log πref (y|x) exp r (x, y) (86)
Z(x) β

Expanding the right-hand side using log(abc) = log a + log b + log c:

1 ∗
log π ∗ (y|x) = − log Z(x) + log πref (y|x) + r (x, y) (87)
β

109
Rearranging to solve for r∗ (x, y):

1 ∗
r (x, y) = log π ∗ (y|x) − log πref (y|x) + log Z(x) (88)
β

Multiplying both sides by β:

π ∗ (y | x)
r∗ (x, y) = β log + β log Z(x) (89)
πref (y | x)

We then can substitute the reward into the Bradley-Terry equation shown in eq. 85 to obtain:

∗
 
1 |x)
exp β log ππref(y
(y1 |x) + β log Z(x)
p∗ (y1 ≻ y2 | x) =    
π ∗ (y1 |x) ∗
2 |x)
exp β log πref (y1 |x) + β log Z(x) + exp β log ππref(y (y2 |x) + β log Z(x)
(90)
a+b a b
By decomposing the exponential expressions from e to e e and then cancelling out the
terms elog(Z(x)) , this simplifies to:

∗
 
1 |x)
exp β log ππref(y(y1 |x)
p∗ (y1 ≻ y2 | x) =     (91)
π ∗ (y1 |x) ∗
2 |x)
exp β log πref (y1 |x) + exp β log ππref(y
(y2 |x)

∗
 
1 |x)
Then, multiply the numerator and denominator by exp −β log ππref(y
(y1 |x) to obtain:

1
p∗ (y1 ≻ y2 | x) =   (92)
π ∗ (y2 |x) ∗
1 |x)
1 + exp β log πref (y2 |x) − β log ππref(y
(y1 |x)

1
Finally, with the definition of a sigmoid function as σ(x) = 1+e−x , we obtain:

π ∗ (y1 | x) π ∗ (y2 | x)
 
∗
p (y1 ≻ y2 | x) = σ β log − β log (93)
πref (y1 | x) πref (y2 | x)

This is the loss function for DPO, as shown in eq. 69. The DPO paper has an additional
derivation for the objective under a Plackett-Luce Model, which is far less used in practice
[20].

[Link] 3. Deriving the Bradley Terry DPO Gradient We used the DPO gradient
shown in eq. 71 to explain intuitions for how the model learns. To derive this, we must take
the gradient of eq. 93 with respect to the model parameters.

  
πθ (yc |x) πθ (yr |x)
∇θ LDPO (πθ ; πref ) = −∇θ E(x,yc ,yr )∼D log σ β log − β log (94)
πref (yc |x) πref (yr |x)

110
 d
To start, this can be rewritten. We know that the derivative of a sigmoid function dx σ(x) =
d 1
σ(x)(1 − σ(x)), the derivative of logarithm dx log x = x , and properties of sigmoid σ(−x) =
1 − σ(x), so we can reformat the above equation.
θ (yc |x) πθ (yr |x)
First, let u = β log ππref (yc |x) − β log πref (yr |x) (the expression inside the sigmoid). Then, we
have

σ ′ (u)
 
∇θ LDPO (πθ ; πref ) = −E(x,yc ,yr )∼D ∇θ u (95)
σ(u)

Expanding this and using the above expressions for sigmoid and logarithms results in the
gradient introduced earlier:

   
πθ (yr |x) πθ (yc |x)
−E(x,yc ,yr )∼D βσ β log − β log [∇θ log π(yc |x) − ∇θ log π(yr |x)]
πref (yr |x) πref (yc |x)
(96)

12.2 Numerical Concerns, Weaknesses, and Alternatives

Many variants of the DPO algorithm have been proposed to address weaknesses of DPO. For
example, without rollouts where a reward model can rate generations, DPO treats every pair
of preference data with equal weight. In reality, as seen in Chapter 6 on Preference Data,
there are many ways of capturing preference data with a richer label than binary. Multiple
algorithms have been proposed to re-balance the optimization away from treating each pair
equally.
• REgression to RElative REward Based RL (REBEL) adds signal from a reward
model, as a margin between chosen and rejected responses, rather than solely the
pairwise preference data to more accurately solve the RLHF problem [166].
• Conservative DPO (cDPO) and Identity Preference Optimization (IPO)
address the overfitting by assuming noise in the preference data. cDPO assumes
N percent of the data is incorrectly labelled [20] and IPO changes the optimization
to soften probability of preference rather than optimize directly from a label [211].
Practically, IPO changes the preference probability to a nonlinear
  function, moving
q
away from the Bradley-Terry assumption, with Ψ(q) = log 1−q .
• DPO with an offset (ODPO) “requires the difference between the likelihood of the
preferred and dispreferred response to be greater than an offset value” [212] – do not
treat every data pair equally, but this can come at the cost of a more difficult labeling
environment.
Some variants to DPO attempt to either improve the learning signal by making small changes
to the loss or make the application more efficient by reducing memory usage.
• Odds Ratio Policy Optimization (ORPO) directly updates the policy model with
a pull towards the chosen response, similar to the instruction finetuning loss, with a
small penalty on the chosen response [213]. This change of loss function removes the
need for a reference model, simplifying the setup. The best way to view ORPO is DPO
inspired, rather than a DPO derivative.

111
 • Simple Preference Optimization SimPO makes a minor change to the DPO
optimization, by averaging the log-probabilities rather than summing them (SimPO)
or adding length normalization, to improve performance [214].

Figure 19: Sketch of preference displacement in DPO.

One of the core issues apparent in DPO is that the optimization drives only to increase
the margin between the probability of the chosen and rejected responses. Numerically, the
model reduces the probability of both the chosen and rejected responses, but the rejected
response is reduced by a greater extent as shown in fig. 19. Intuitively, it is not clear how
this generalizes, but work has posited that it increases the probability of unaddressed for
behaviors – i.e. tokens that language model could generate, but are not in the distribution of
the post-training datasets [215] [216]. Simple methods—such as Cal-DPO [217], which adjusts
the optimization process, and AlphaPO [218], which modifies the reward shape—mitigate
this preference displacement. In practice, the exact impact of this is not well known, but
points to a potential reason why online methods can outperform vanilla DPO.
The largest other reason that is posited for DPO-like methods to have a lower ceiling on
performance than online (RL based) RLHF methods is that the training signal comes from
completions from previous or other models. Online variants of DPO alleviate these limitations
by generating new completions and incorporating a preference signal at training time. Online
DPO [219] samples generations from the current model, while Discriminator-Guided
DPO (D2PO) [220] uses reward model relabelling to create new preference data on the fly,
and many more variants exist.
There is a long list of other DAA variants, such as Direct Nash Optimization (DNO) [221] or
Binary Classifier Optimization (BCO) [222], but the choice of algorithm is far less important
than the initial model and the data used [6] [223] [224].

112
12.3 Implementation Considerations
DAAs such as DPO are implemented very differently than policy gradient optimizers. The
DPO loss, taken from the original implementation, largely can be summarized as follows [20]:
pi_logratios = policy_chosen_logps - p olic y_re jecte d_lo gps
ref_logratios = re fe re nc e_c ho se n_l og ps - r e f e r en c e _ re j e c te d _ l og p s

logits = pi_logratios - ref_logratios # also known as h_ {\ pi_ \ theta

}^{ y_w , y_l }

losses = -F . logsigmoid ( beta * logits )

chosen_rewards = beta * ( policy_chosen_logps - ref er en ce _ch os en _lo gp s )

. detach ()
rejected_rewards = beta * ( polic y_re ject ed_lo gps -
r e fe r e n ce _ r e je c t e d_ l o g ps ) . detach ()

This can be used in standard language model training stacks as this information is already
collated during the forward pass of a model (with the addition of a reference model).
In most ways, this is simpler and a quality of life improvement, but also they offer a different
set of considerations.
1. KL distance is static: In DPO and other algorithms, the KL distance is set explicitly
by the β parameter that balances the distance penalty to the optimization. This is due
to the fact that DPO takes gradient steps towards the optimal solution to the RLHF
objective given the data – it steps exactly to the solution set by the β term. On the
other hand, RL based optimizers take steps based on the batch and recent data.
2. Caching log-probabilities: Simple implementations of DPO do the forward passes
for the policy model and reference models at the same time for conveniences with
respect to the loss function. Though, this doubles the memory used and results in
increased GPU usage. To avoid this, one can compute the log-probabilities of the
reference model over the training dataset first, then reference it when computing the
loss and updating the parameters per batch, reducing the peak memory usage by 50%.

12.4 DAAs vs. RL: Online vs. Offline Data

Broadly, the argument boils down to one question: Do we need the inner workings of
reinforcement learning, with value functions, policy gradients, and all, to align language
models with RLHF? This, like most questions phrased this way, is overly simplistic. Of course,
both methods are well-established, but it is important to illustrate where the fundamental
differences and performance manifolds lie.
Multiple reports have concluded that policy-gradient based and RL methods outperform
DPO and its variants. The arguments take different forms, from training models with
different algorithms but controlled data[190] [225] or studying the role of on-policy data
within the RL optimization loop [226]. In all of these cases, DPO algorithms are a hair
behind.
Even with this performance delta, DAA are still used extensively in leading models due to
its simplicity. DAAs provide a controlled environment where iterations on training data and

113
other configurations can be made rapidly, and given that data is often far more important
than algorithms, using DPO can be fine.
With the emergence of reasoning models that are primarily trained with RL, further invest-
ment will return to using RL for preference-tuning, which in the long-term will improve
the robustness of RL infrastructure and cement this margin between DAAs and RL for
optimizing from human feedback.

114
13 Constitutional AI & AI Feedback
Soon after the explosion of growth in RLHF, RL from AI Feedback (RLAIF) emerged as an
alternative approach where AIs could approximate the human data piece of the pipeline and
accelerate experimentation or progress. AI feedback, generally, is a larger set of techniques
for using AI to augment or generate data explaining the quality of a certain input (which
can be used in different training approaches or evaluations), which started with pairwise
preferences [227] [228] [229]. There are many motivations to using RLAIF to either entirely
replace human feedback or augment it. Within the RLHF process, AI feedback is known
most for its role within the preference data collection and the related reward model training
phase (of which constitutional AI is a certain type of implementation). In this chapter, we
focus on the general AI feedback and this specific way of using it in the RLHF training
pipeline, and we cover more ways of understanding or using synthetic data later in this book.

115
14 Balancing AI and Human Feedback Data
AI models are far cheaper than humans at generating a specific quantity of feedback, with
a single piece of human preference data costing as of writing this on the order of $1 or
higher (or even above $10 per prompt), AI feedback with a frontier AI model, such as
GPT-4o costs less than $0.01. Beyond this, the cost of human labor is remaining roughly
constant, while the performance of leading models at these tasks continues to increase while
price-per-performance decreases. This cost difference opens the market of experimentation
with RLHF methods to an entire population of people previously priced out.
Other than price, AI feedback introduces different tradeoffs on performance than human
feedback, which are still being investigated in the broader literature. AI feedback is far more
predominant in its role in evaluation of the language models that we are training, as its low
price lets it be used across a variety of large-scale tasks where the cost (or time delay) in
human data would be impractical.
The exact domains and applications – i.e. chat, safety, reasoning, mathematics, etc. –
where AI feedback data outperforms human data is not completely established. Some early
work in RLAIF shows that AI feedback can completely replace human data, touting it
as an effective replacement [227] and especially when evaluated solely on chat tasks [23]
[230]. Early literature studying RLHF after ChatGPT had narrow evaluation suites focused
on the “alignment” of models that act as helpful assistants across a variety of domains
(discussed further in Chapter 17). Later work takes a more nuanced picture, where the
optimal equilibrium on a broader evaluation set, e.g. including some reasoning tasks, involves
routing a set of challenging data-points to accurately label to humans, while most of the
data is sent for AI feedback [231] [232]. While there are not focused studies on the balance
between human and AI feedback data for RLHF across broader domains, there are many
technical reports that show RLHF generally can improve these broad suite of evaluations,
some that use DPO, such as Ai2’s Tülu 3 [6] & Olmo 3 [233], or HuggingFace’s SmolLM
3 [234], and others that use online RLHF pipelines, such as Nvidia’s work that uses a mix
of human preference data from Scale AI and LLM-based feedback (through the helpsteer
line of work [235] [160] [111] [236]): Nemotron Nano 3 [237], Nemotron-Cascade [238], or
Llama-Nemotron reasoning models [239].
Overall, where AI feedback and related methods are obviously extremely useful to the field,
it is clear that human data has not been completely replaced by these cheaper alternatives.
Many hypotheses exist, but it is not studied if human data allows finer control of the models
in real-world product settings or for newer training methods such as character training (an
emerging set of techniques that allow you to precisely control the personality of a model,
covered in Chapter 20). For those getting started, AI feedback should be the first attempt,
but for pipelines that’re scaling to larger operations the eventual transition to include human
feedback is likely.
The term RLAIF was introduced in Anthropic’s work Constitutional AI: Harmlessness from
AI Feedback [19], which resulted in initial confusion in the AI community over the relationship
between the two methods in the title of the paper (Constitutional AI and AI Feedback).
Since the release of the Constitutional AI (CAI) paper and the formalization of RLAIF,
RLAIF has become a default method within the post-training and RLHF literatures – there
are far more examples than one can easily enumerate. The relationship should be understood
as CAI was the example that kickstarted the broader field of RLAIF.

116
A rule of thumb for the difference between human data and AI feedback data is as follows:
1. Human data is high-noise and low-bias. This means that collection and filtering of the
data can be harder, but when wrangled it’ll provide a very reliable signal.
2. Synthetic preference data is low-noise and high-bias. This means that AI feedback
data will be easier to start with, but can have tricky, unintented second-order effects
on the model that are systematically represented in the data.
This book highlights many academic results showing how one can substitute AI preference
data in RLHF workflows and achieve strong evaluation scores [231], but broader industry
trends show how the literature of RLHF is separated from more opaque, best practices.
Across industry, human data is often seen as a substantial moat and a major technical
advantage.

14.1 Constitutional AI
The method of Constitutional AI (CAI), which Anthropic uses in their Claude models, is
the earliest documented, large-scale use of synthetic data for RLHF training. Constitutional
AI involves generating synthetic data in two ways:
1. Critiques of instruction-tuned data to follow a set of principles like “Is the answer
encouraging violence” or “Is the answer truthful.” When the model generates answers
to questions, it checks the answer against the list of principles in the constitution,
refining the answer over time. Then, they fine-tune the model on this resulting dataset.
2. Generates pairwise preference data by using a language model to answer which comple-
tion was better, given the context of a random principle from the constitution (similar
to research for principle-guided reward models [240]). Then, RLHF proceeds as normal
with synthetic data, hence the RLAIF name.
Largely, CAI is known for the second half above, the preference data, but the methods
introduced for instruction data are used in general data filtering and synthetic data generation
methods across post-training.
CAI can be formalized as follows.
By employing a human-written set of principles, which they term a constitution, Bai et
al. 2022 use a separate LLM to generate artificial preference and instruction data used for
fine-tuning [19]. A constitution C is a set of written principles indicating specific aspects to
focus on during a critique phase. The instruction data is curated by repeatedly sampling
a principle ci ∈ C and asking the model to revise its latest output y i to the prompt x to
align with ci . This yields a series of instruction variants {y 0 , y 1 , · · · , y n } from the principles
{c0 , c1 , · · · , cn−1 } used for critique. The final data point is the prompt x together with the
final completion y n , for some n.
The preference data is constructed in a similar, yet simpler way by using a subset of principles
from C as context for a feedback model. The feedback model is presented with a prompt x,
a set of principles {c0 , · · · , cn }, and two completions y0 and y1 labeled as answers (A) and
(B) from a previous RLHF dataset. The new datapoint is generated by having a language
model select which output (A) or (B) is both higher quality and more aligned with the stated
principle. In earlier models this could be done by prompting the model with The answer is:,
and then looking at which logit (A or B) had a higher probability, but more commonly is

117
now handled by a model that’ll explain its reasoning and then select an answer – commonly
referred to as a type of generative reward model [135].

14.2 Specific LLMs for Judgement

As RLAIF methods have become more prevalent, many have wondered if we should be
using the same models for generating responses as those for generating critiques or ratings.
Specifically, the calibration of the LLM-as-a-judge used has come into question. Several
works have shown that LLMs are inconsistent evaluators [241] and prefer their own responses
over responses from other models (coined self-preference bias) [242].
As a result of these biases, many have asked: Would a solution be to train a separate
model just for this labeling task? Multiple models have been released with the goal of
substituting for frontier models as a data labeling tool, such as critic models Shepherd
[243] and CriticLLM [244] or models for evaluating response performance akin to Auto-J
[245], Prometheus [138], Prometheus 2 [246], or Prometheus-Vision [247] but they are not
widely adopted in documented training recipes. Some find scaling inference via repeated
sampling [248] [249] [250], self-refinement [251], or tournament ranking [252] provides a
better estimate of the true judgement or higher-quality preference pairs. Other calibration
techniques co-evolve the generation and judgement capabilities of the model [253]. It is
accepted that while biases exist, the leading language models are trained extensively for
this task – as its needed for both internal operations at AI labs and is used extensively by
customers – so it is generally not needed to train your own judge, unless your task involves
substantial private information that is not exposed on the public internet.

14.3 Further Reading

There are many related research directions and extensions of Constitutional AI, but few of
them have been documented as clear improvements in RLHF and post-training recipes. For
now, they are included as further reading.
• OpenAI has released a Model Spec [124], which is a document stating the intended
behavior for their models, and stated that they are exploring methods for alignment
where the model references the document directly (which could be seen as a close peer
to CAI). OpenAI has continued and trained their reasoning models such as o1 with a
method called Deliberative Alignment [254] to align the model while referencing these
safety or behavior policies.
• Anthropic has continued to use CAI in their model training, updating the constitution
Claude uses [255] and experimenting with how population collectives converge on
principles for models and how that changes model behavior when they create principles
on their own and then share them with Anthropic to train the models [256].
• The open-source community has explored replications of CAI applied to open datasets
[257] and for explorations into creating dialogue data between LMs [258].
• Other work has used principle-driven preferences or feedback with different optimization
methods. [259] uses principles as context for the reward models, which was used to
train the Dromedary models [240]. [37] uses principles to improve the accuracy of
human judgments in the RLHF process. [260] train a reward model to generate its
own principles at inference time, and use these to deliver a final score. [261] formulate
principle-following as a mutual information maximization problem that the pretrained

118
model can learn with no labels.

119
15 Reasoning Training & Inference-Time Scaling
Reasoning models and inference-time scaling enabled a massive step in language model
performance in the end of 2024, through 2025, and into the future. Inference-time scaling is
the underlying property of machine learning systems that language models trained to think
extensively before answering exploit so well. These models, trained with a large amount of
reinforcement learning with verifiable rewards (RLVR) [6], still utilize large amounts of RLHF.
In this chapter we review the path that led the AI community to a transformed appreciation
for RL’s potential in language models, review the fundamentals of RLVR, highlight key
works, and point to the future debates that will define the area in the next few years.
To start, at the 2016 edition of the Neural Information Processing Systems (NeurIPS)
conference, Yann LeCun first introduced his now-famous cake metaphor for where learning
happens in modern machine learning systems:
If intelligence is a cake, the bulk of the cake is unsupervised learning, the icing
on the cake is supervised learning, and the cherry on the cake is reinforcement
learning (RL).
This analogy is now largely complete with modern language models and recent changes to
the post-training stack. RLHF was the precursor to this, and RL for reasoning models,
primarily on math, code, and science topics, was its confirmation. In this analogy:
• Self-supervised learning on vast swaths of internet data makes up the majority of the
cake (especially when viewed in compute spent in FLOPs),
• The beginning of post-training in supervised finetuning (SFT) for instructions tunes
the model to a narrower distribution, and
• Finally “pure” reinforcement learning (RL) is the cherry on top. The scaled up
reinforcement learning used to create the new “reasoning” or “thinking” models is this
finishing piece (along with the help of RLHF, which isn’t considered classical RL, as
we’ll explain).
This little bit of reasoning training emerged with thinking models that use a combination
of the post-training techniques discussed in this book to align preferences along with RL
training on verifiable domains to dramatically increase capabilities such as reasoning, coding,
and mathematics problem solving.
The training method for these models, Reinforcement Learning with Verifiable Rewards
(RLVR) [6], proceeds very similarly to RLHF, but it makes the reward model optional in
lieu of a scoring function that returns a positive reward when the answer is correct and 0
otherwise.
For example, consider how different it is to score responses for RLHF versus RLVR. In RLHF,
a reward model must evaluate subjective qualities:
Prompt: Explain the concept of opportunity cost in economics.
Response: Opportunity cost is the value of the next best alternative you give
up when making a decision. For example, if you spend an hour studying instead
of working, the opportunity cost is the wages you could have earned. . .
Scoring this response requires judging clarity, accuracy, completeness, and helpfulness – all
qualities that require learned preferences and lack a definitive correct answer.

120
In contrast, RLVR uses verification functions that return definitive scores. For mathematics:
Prompt: What is the sum of all prime numbers less than 20?
Response: The prime numbers less than 20 are 2, 3, 5, 7, 11, 13, 17, and 19.
Adding them: 2 + 3 = 5, then 5 + 5 = 10, then 10 + 7 = 17, then 17 + 11 = 28,
then 28 + 13 = 41, then 41 + 17 = 58, and finally 58 + 19 = 77. The answer is
77 .
Verification: extracted_answer == 77 → Reward = 1
The \boxed{} notation is a convention adopted from mathematical typesetting that makes
answer extraction straightforward – a simple regular expression can pull the final answer from
the response, regardless of how the model arrived at it. Note that other answer extraction
methods exist, such as just using the phrase “The answer is:” as also shown above, special
tokens like <answer>, or delimiters like ####.
For code generation, verification often takes the form of unit tests:
Prompt: Write a Python function fib(n) that returns the nth Fibonacci number,
where fib(0) = 0 and fib(1) = 1.
Response: def fib(n): if n < 2: return n return fib(n - 1) + fib(n - 2)
Verification (unit tests):
assert fib(0) == 0 # base case assert fib(1) == 1 # base case assert fib(10) ==
55 # larger value (All tests pass → Reward = 1)
Unit tests are the natural verification function for code: they execute the model’s solution
against known input-output pairs. A common form of scoring is to perform the simple gating:
If all assertions pass, the reward is 1; if any fail, the reward is 0. Other setups use partial
credit proportional to the amount of tests passed. For both these examples, no learned
reward model is needed and most setups go without one (because the models are robust
to over-optimization in these domains), but one can be used with a linear combination of
rewards.
The ideas behind RLVR are not new to the RL literature, where the core idea of taking
gradient updates on if the answer is correct is almost the textbook definition of reinforcement
learning. The innovations when applying this to language models are largely how to apply it
while maintaining the strong, general capabilities of the model being finetuned. Within that,
there are many related ideas in the language modeling literature where the model learns
from feedback regarding the correctness of the answer.
Originally, in the work I was a part of that coined RL with Verifiable Rewards (RLVR) [6],
the method was to be named RL with Ground Truth rewards (RLGT). However, RLVR is
subtly different from learning solely from ground truth answers. In domains like mathematics,
a single ground truth answer is available to verify solutions, as we saw above. In other
domains, such as code generation or precise instruction following, answers can be verified
with a checking function (e.g., a unit test), even when there are multiple correct solutions
rather than just a single ground truth answer. The core to progress on RLVR is having a
variety and depth of these verifiable problems, even if the exact solution isn’t known a priori.
The first models to successfully deploy this type of training were OpenAI’s o1 [48] and the

121
Figure 20: RLVR in the form of an RL feedback loop. Instead of a reward model, we use a
verification function.

122
open-weight model DeepSeek R1 [61]. Soon after, the entire AI industry prioritized this
training process and model style. The core change here is more of a reallocation of the stages
of training and the priority of different behaviors rather than this type of RL setup being
entirely new. Reasoning models brought an era where scaling RL training is expected.
As for the type of behavior these models accrue, consider the following example with DeepSeek
V3 0325 versus their reasoning model, DeepSeek R1, on the query Write me a short poem
about a goldfish. DeepSeek V3 is very to the point:

Goldfish Dreams
In a bowl of glass so clear and round, A tiny goldfish swims without a sound.
With scales that gleam like sunlight’s kiss, He drifts along in watery bliss.
A flick of fin, a flash so bright, Chasing bubbles in the light. Does he dream of
oceans wide? Or is his little world enough—his pride?
Meanwhile, DeepSeek R1’s experience is different. It first generates numerous thinking
tokens to craft a response before responding to the user. The answer appears as:
<thinking> Okay, the user wants a short poem about a goldfish. Let me start by
brainstorming some goldfish-related imagery. Goldfish are often in bowls, right?
So maybe start with the setting. Words like “gilded bowl” or “crystal globe”
could work.
Next, think about the movement of the goldfish. They swim smoothly, so words
like “dart”, “glide”, or “swirl” come to mind. Maybe describe their fins as “silken”
to emphasize delicacy.
Then, consider the environment around the goldfish. Sunlight through water
creates shimmering effects. Words like “sunbeams dance” or “ripples spin” might
capture that. Including elements like castles or pebbles adds a playful touch.
Themes? Maybe the contrast between the goldfish’s small world and its perception
of freedom. The bowl as a universe, tiny but complete. Ending with a sense of
mystery or silent observation could give depth. Need to keep it concise, maybe
four stanzas with a rhyme scheme. Let me check the rhythm and flow. Avoid
clichés but keep it accessible. Alright, let’s piece it together. </thinking>
Gilded Dreams in Liquid Glass
A gilded bowl, a crystal globe, Where tiny tides in silence flow— A flash of
orange, a whispered robe, You dart through sunlit indigo.
Your world is small, yet boundless too— Each pebble, castle, weed unfurled. The
surface bends the sky to blue, And ripples spin your liquid world.
You glide through myths we’ll never know, A silent monarch, veiled in mist. Do
dreams drift where the bubbles go? Or memories in amethyst?
We watch you trace your endless sphere— A universe within a tear.
The tokens in the <thinking> tags are the model’s reasoning. For more complex problems the
reasoning stage can take thousands of tokens before producing an answer. So, long-context

123
language models were a prerequisite to advanced reasoning behavior, but that is not the
focus of this chapter.
The core intuition for how this training works is that for a given model, we repeat the
following cycle:
1. Sample multiple answers to multiple questions,
2. Take gradient steps towards the answers that are correct, and
3. Repeat, revisiting the same data.
Remarkably, this extremely simple approach (when done with a careful distribution of data
and stable training infrastructure) helps the models learn by revisiting the same questions
again and again. Even more remarkable is that the improvements on these training questions
generalize to questions and (some) domains the models have never seen!
This simple approach allows the models to lightly search over behavior space and the RL
algorithm increases the likelihood of behaviors that are correlated with correct answers.

15.1 The Origins of New Reasoning Models

Here we detail the high-level trends that led to the explosion of reasoning models in 2025.

15.1.1 Why Does RL Work Now?

Despite many, many takes that “RL doesn’t work yet” [262] or papers detailing deep
reproducibility issues with RL [263], the field overcame it to find high-impact applications.
Some are covered in this book, such as ChatGPT’s RLHF and DeepSeek R1’s RLVR, but
many others exist, including improving chip design [264], mastering video gameplay [265],
self-driving [266], and more. The takeoff of RL-focused training on language models indicates
steps in many fundamental issues for the research area, including:
• Stability of RL can be solved: For its entire existence, the limiting factor on RL’s
adoption has been stability. This manifests in two ways. First, the learning itself can
be fickle and not always work. Second, the training itself is known to be more brittle
than standard language model training and more prone to loss spikes, crashes, etc.
Countless new model releases are using this style of RL training with verifiable rewards
on top of a pretrained base model and substantial academic uptake has occurred. The
technical barriers to entry on RL are at an all time low.
• Open-source versions already “exist”: Many tools already exist for training
language models with RLVR and related techniques. Examples include TRL [42], Open
Instruct [6], veRL [267], and OpenRLHF [268], where many of these are building on
optimizations from earlier in the arc of RLHF and post-training. The accessibility of
tooling is enabling a large uptake of research that’ll likely soon render this chapter out
of date.
Multiple resources point to RL training for reasoning only being viable on leading models
coming out from about 2024 onwards, indicating that a certain level of underlying capability
was needed in the models before reasoning training was possible.

124
15.1.2 RL Training vs. Inference-time Scaling
Training with Reinforcement Learning to elicit reasoning behaviors and performance on
verifiable domains is closely linked to the ideas of inference-time scaling. Inference-time scaling,
also called test-time scaling, is the general class of methods that use more computational
power at inference in order to perform better at downstream tasks. Methods for inference-
time scaling were studied before the release of DeepSeek R1 and OpenAI’s o1, which both
massively popularized investment in RL training specifically. Examples include value-guided
sampling [269] or repeated random sampling with answer extraction [248]. Beyond this,
inference-time scaling can be used to improve more methods of AI training beyond chain-of-
thought reasoning to solve problems, such as with reward models that consider the options
deeply [137] [260].
RL training is a short path to inference-time scaling laws being used, but in the long-term we
will have more methods for eliciting the inference-time tradeoffs we need for best performance.
Training models heavily with RL often enables them to generate more tokens per response
in a way that is strongly correlated with improved, downstream performance (though, while
this sequence length increase is the default, research also exists explicitly on improving
performance without relying on this inference-time scaling). This is a substantial shift from
the length-bias seen in early RLHF systems [9], where the human preference training had a
side effect of increasing the response average length for marginal gains on preference rankings.
Other than the core RL trained models there are many methods being explored to continue
to push the limits of reasoning and inference-time compute. These are largely out of the
scope of this book due to their rapidly evolving nature, but they include distilling reasoning
behavior from a larger RL trained model to a smaller model via instruction tuning [270],
composing more inference calls [271], and more. What is important here is the correlation
between downstream performance and an increase in the number of tokens generated –
otherwise it is just wasted energy.

15.1.3 The Future (Beyond Reasoning) of RLVR

In many domains, these new flavors of RLVR are much more aligned with the goals of
developers by being focused on performance rather than behavior. Standard finetuning
APIs generally use a parameter-efficient finetuning method such as LoRA with supervised
finetuning on instructions. Developers pass in prompts and completions and the model
is tuned to match that by updating model parameters to match the completions, which
increases the prevalence of features from your data in the model’s generations.
RLVR is focused on matching answers. Given queries and correct answers, RLVR helps the
model learn to get the correct answers. While standard instruction tuning is done with 1 or
2 epochs of loss updates over the data, RLVR gets its name by doing hundreds or thousands
of epochs over the same few data points to give the model time to learn new behaviors. This
can be viewed as reinforcing positive behaviors that would work sparingly in the base model
version into robust behaviors after RLVR.
The scope of RL training for language models continues to grow: The biggest
takeaway from o1 and R1 on a fundamental scientific level was that we have even more ways
to train language models to potentially valuable behaviors. The more open doors that are
available to researchers and engineers, the more optimism we should have about AI’s general
trajectory.

125
15.2 Understanding Reasoning Training Methods
The investment in reasoning has instigated a major evolution in the art of how models are
trained to follow human instructions. These recipes still use the common pieces discussed
in earlier chapters (as discussed in Chapter 4 with the overview of DeepSeek R1’s recipe),
including instruction finetuning, reinforcement learning from human feedback, and reinforce-
ment learning with verifiable rewards (RLVR). The core change is using far more RLVR
and applying the other training techniques in different orders – traditionally for a reasoning
model the core training step is either a large-scale RL run or a large-scale instruction tuning
run on outputs of another model that had undergone a substantial portion of RLVR training
(referred to as distillation).

15.2.1 Reasoning Research Pre OpenAI’s o1 or DeepSeek R1

Before the takeoff of reasoning models, a substantial effort was made understanding how to
train language models to be better at verifiable domains. The main difference between these
works below is that their methodologies did not scale up to the same factor as those used
in DeepSeek R1 and subsequent models, or they resulted in models that made sacrifices in
overall performance in exchange for higher mathematics or coding abilities. The underlying
ideas and motivations are included to paint a broader picture for how reasoning models
emerged within the landscape.
Some of the earliest efforts training language models on verifiable domains include self-taught
reasoner (STaR) line of work[272] [273] and TRICE [274], which both used ground-truth
reward signals to encourage chain-of-thought reasoning in models throughout 2022 and 2023.
STaR effectively approximates the policy gradient algorithm, but in practice filters samples
differently and uses a cross-entropy measure instead of a log-probability, and Quiet-STaR
expands on this with very related ideas of recent reasoning models by having the model
generate tokens before trying to answer the verifiable question (which helps with training
performance). TRICE [274] also improves upon reasoning by generating traces and then
optimizing with a custom Markov chain Monte Carlo inspired expectation maximization
algorithm. VinePPO [275] followed these and used a setup that shifted closer to modern
reasoning models. VinePPO uses a PPO-based algorithm with binary rewards for math
question correctness, training on GSM8K and MATH. Other work before OpenAI’s o1 and
DeepSeek R1 used code execution as a feedback signal for training [276], [277] or verification
for theorem proving (called Reinforcement Learning from Verifier Feedback, RLVF, here)
[278]. Tülu 3 expanded upon these methods by using a simple PPO trainer to reward
completions with correct answers – most importantly while maintaining the model’s overall
performance on a broad suite of evaluations. The binary rewards of Tülu 3 and modern
reasoning training techniques can be contrasted to the iterative approach of STaR or the
log-likelihood rewards of Quiet-STaR.

15.2.2 Early Reasoning Models

A summary of the foundational reasoning research reports, some of which are accompanied
by open data and model weights, following DeepSeek R1 is below.

126
Table 6: A summary of the notable reasoning model technical reports in 2025, the first year
of substantial inference-time scaling with RLHF.

Open Open
Date Name TLDR weights data
2025- DeepSeek R1 RL-based upgrade to DeepSeek, big gains Yes No
01-22 [61] on math & code reasoning
2025- Kimi 1.5 [205] Scales PPO/GRPO on Chinese/English No No
01-22 data; strong AIME maths
2025- Open-Reasoner- Fully open replication of base model RL Yes Yes
03-31 Zero [279]
2025- Seed-Thinking ByteDance RL pipeline with dynamic Yes No
04-10 1.5 [64] CoT gating
2025- Phi-4 Reasoning 14B model; careful SFT→RL; excels at Yes No
04-30 [280] STEM reasoning
2025- Llama- Multi-size “reasoning-toggle” models Yes Yes
05-02 Nemotron [239]
2025- INTELLECT-2 First, publicly documented Yes Yes
05-12 [198] globally-decentralized RL training run
2025- Xiaomi MiMo End-to-end reasoning pipeline from pre- Yes No
05-12 [63] to post-training
2025- Qwen 3 [62] Similar to R1 recipe applied to new Yes No
05-14 models
2025- Hunyuan- Mamba-Transformer MoE, adaptive No No
05-21 TurboS [281] long/short CoT
2025- Skywork OR-1 RL recipe avoiding entropy collapse; Yes Yes
05-28 [282] beats DeepSeek on AIME
2025- Xiaomi MiMo Adapting reasoning pipeline end-to-end to Yes No
06-04 VL [283] include multi-modal tasks
2025- OpenThoughts Public 1.2M-example instruction dataset Yes Yes
06-04 [284] distilled from QwQ-32B
2025- Magistral [285] Pure RL on Mistral 3; multilingual CoT; Yes No
06-10 small model open-sourced
2025- MiniMax-M1 Open-weight 456B MoE hybrid/Lightning Yes No
06-16 [286] Attention reasoning model; 1M context;
RL w/CISPO; releases 40K/80K
thinking-budget checkpoints
2025- Kimi K2 [287] 1T MoE (32B active) with MuonClip Yes No
07-10 (QK-clip) for stability; 15.5T token
pretrain without loss spikes; multi-stage
post-train with agentic data synthesis +
joint RL; releases base + post-trained
checkpoints.
2025- GLM-4.5 [288] Open-weight 355B-A32B MoE “ARC” Yes No
07-28 model with thinking/non-thinking modes;
23T-token multi-stage training +
post-train w/ expert iteration and RL;
releases GLM-4.5 + GLM-4.5-Air (MIT).

127
 Open Open
Date Name TLDR weights data
2025- Nemotron Nano Hybrid Mamba-Transformer for long Yes Yes
08-20 2 [289] “thinking traces”; FP8 pretraining at 20T (most)
tokens then compression/distillation;
explicitly releases multiple checkpoints
plus “majority” of pre/post-training
datasets.
2025- K2-Think [290] Parameter-efficient math reasoning Yes Yes
09-09 system: a 32B open-weights model with
test-time scaling recipe; positioned as
fully open incl. training data/code (per
release materials).
2025- LongCat-Flash- 560B MoE reasoning model; report is Yes No
09-23 Thinking [291] explicit about a staged recipe from
long-CoT cold start to large-scale RL;
open-source release.
2025- Ring-1T [292] Trillion-scale “thinking model” with RL Yes No
10-21 scaling focus; report frames
bottlenecks/solutions for scaling RL at 1T
and releases an open model.
2025- OLMo 3 Think Fully open “model flow” release: reports Yes Yes
11-20 [233] the entire lifecycle (stages, checkpoints,
and data points) and positions OLMo 3
Think 32B as a flagship open thinking
model.
2025- DeepSeek V3.2 Open-weight MoE frontier push with a Yes No
12-02 [293] report that foregrounds attention
efficiency changes, RL framework
upgrades, and data synthesis for
agentic/reasoning performance.
2025- K2-V2 [294] 70B dense “360-open” model trained from Yes Yes
12-05 scratch; with 3-effort SFT-only
post-training for controllable thinking.
2025- Nemotron 3 30B-A3B MoE hybrid Yes Yes
12-15 Nano [237] Mamba-Transformer; pretrain on 25T (most)
tokens and includes SFT + large-scale
RL; explicitly states it ships weights +
recipe/code + most training data.
2025- MiMo-V2-Flash 309B MoE (15B active) optimized for Yes No
12-16 [295] speed: hybrid SWA/GA attention (5:1,
128-token window) + lightweight MTP;
FP8 pretrain on 27T tokens; post-train
with MOPD + large-scale agentic RL for
reasoning/coding.

128
15.2.3 Common Practices in Training Reasoning Models
In this section we detail common methods used to sequence training stages and modify data
to maximize performance when training a reasoning model.
Note that these papers could have used a listed technique and not mentioned it while their
peers do, so these examples are a subset of known implementations and should be used as
reference, but not a final proclamation on what is an optimal recipe.
• Offline difficulty filtering: A core intuition of RLVR is that models can only learn
from examples where there is a gradient. If the starting model for RLVR can solve a
problem either 100% of the time or 0% of the time, there will be no gradient between
different completions to the prompt (i.e., all strategies appear the same to the policy
gradient algorithm). Many models have used difficulty filtering before starting a large-
scale RL to restrict the training problems to those that the starting point model solves
only 20-80% of the time. This data is collected by sampling N, e.g. 16, completions
to each prompt in the training set and verifying which percentage are correct. Forms
of this were used by Seed-Thinking 1.5, Open Reasoner Zero, Phi 4, INTELLECT-2,
MiMo RL, Skywork OR-1, and others.
• Per-batch online filtering (or difficulty curriculums throughout training): To
complement the offline filtering to find the right problems to train on, another major
question is “what order should we present the problems to the model during learning.”
In order to address this, many models use online filtering of questions in the batch,
prebuilt curriculums/data schedulers, saving harder problems for later in training,
or other ideas to improve long-term stability. Related ideas are used by Kimi 1.5,
Magistral, Llama-Nemotron, INTELLECT-2, MiMo-RL, Hunyuan-TurboS, and others.
• Remove KL penalty: As the length of RL runs (in any metric, total GPU hours,
FLOPS, or RL steps) increased for reasoning models relative to RLHF training, and
the reward function became less prone to over-optimization, many models removed
the KL penalty constraining the RL-learned policy to be similar to the base model of
training. This allows the model to further explore during its training. This was used
by RAGEN [296], Magistral, OpenReasonerZero, Skywork OR-1, and others.
• Relaxed policy-gradient clipping: New variations of the algorithm GRPO, such as
DAPO [193], proposed modifications to the two sided clipping objective used in GRPO
(or PPO) in order to enable better exploration. Clipping has also been shown to cause
potentially spurious learning signals when rewards are imperfect [297]. This two-sided
clipping with different ranges per gradient direction is used by RAGEN, Magistral,
INTELLECT-2, and others.
• Off-policy data (or fully asynchronous updates): As the length of completions
needed to solve tasks with RL increases dramatically with harder problems (particularly
in the variance of the response length, where there are often outliers with extremely
long lengths), compute in RL runs can sit idle. To solve this, training is moving
to asynchronous updates or changing how problems are arranged into batches to
improve overall throughput. Partial-to-full asynchronous (off-policy) data is used by
Seed-Thinking 1.5, INTELLECT-2, and others.
• Additional format rewards: In order to make the reasoning process predictable,
many models add minor rewards to make sure the model follows the correct format of
e.g. <think>...</think> before an answer. This is used by DeepSeek R1, OpenReason-
erZero, Magistral, Skywork OR-1, and others.

129
 • Language consistency rewards: Similar to format rewards, some multilingual
reasoning models use language consistency rewards to prioritize models that do not
change languages while reasoning (for a better and more predictable user experience).
These include DeepSeek R1, Magistral, and others.
• Length penalties: Many models use different forms of length penalties during RL
training to either stabilize the learning process over time or to mitigate overthinking
on hard problems. Some examples include Kimi 1.5 progressively extending the
target length to combat overthinking (while training accuracy is high across difficulty
curriculum) or INTELLECT-2 running a small length penalty throughout. Progressively
extending the training sequence length mitigates overthinking by forcing the model
to first reason effectively in a domain with a more limited thinking budget, and then
transitioning to longer training where the model can use those behaviors efficiently on
more complex problems. Others use overlong filtering and other related implementations
to improve throughput.
• Loss normalization: There has been some discussion (see the chapter on Policy
Gradients or [185]) around potential length or difficulty biases introduced by the
per-group normalization terms of the original GRPO algorithm. As such, some models,
such as Magistral or MiMo, chose to normalize either losses or advantages at the batch
level instead of the group level.
• Parallel test-time compute scaling: Combining answers from multiple parallel,
independently-sampled rollouts can lead to substantial improvements over using the
answer from a single rollout. The most naive form of parallel test-time compute scaling,
as done in DeepSeek-R1, Phi-4, and others, involves using the answer returned by a
majority of rollouts as the final answer. A more advanced technique is to use a scoring
model trained to select the best answer out of the answers from the parallel rollouts.
This technique has yet to be adopted by open reasoning model recipes (as of June 2025)
but was mentioned in the Claude 4 announcement [298] and used in DeepSeek-GRM
[260].
In complement to the common techniques, there are also many common findings on how
reasoning training can create useful models without sacrificing ancillary capabilities:
• Text-only reasoning boosts multimodal performance: Magistral, MiMo-VL, and
others find that training a multimodal model and then performing text-only reasoning
training after this multimodal training can improve multimodal performance in the
final model.
• Toggleable reasoning with system prompt (or length control): Llama-Nemotron,
Nemotron Nano, Qwen 3, SmolLM 3, and others use specific system prompts (possibly
in combination with length-controlled RL training [299]) to enable a toggleable on/off
thinking length for the user. Other open models, such as OpenAI’s GPT-OSS and
LLM360’s K2-V2 [294] adopt a low-medium-high reasoning effort set in the system
prompt, but training methods for this type of behavior are not as well documented.

15.3 Looking Ahead

The landscape of reasoning models is evolving faster than any area of AI research in recent
memory. By the time this chapter is published, the table of reasoning models above will
be incomplete and some of the common practices listed may have been superseded by new
techniques.

130
Several efforts are underway to systematically understand what makes reasoning training
work. OLMo 3 Think [233] represents the most comprehensive open documentation of a
reasoning model’s full training lifecycle, providing checkpoints and data at each stage for the
research community to study, and concluding with a nearly 4 week long training run on 220
GPUs. Similarly, work on understanding the scaling properties of RL for reasoning [300]
is beginning to formalize relationships between compute, data, and performance that were
previously only intuited by practitioners.
What remains clear is that reinforcement learning has graduated from the “cherry on top”
of the cake metaphor to a load-bearing component of frontier model training. The minor
techniques in this chapter around the idea of RLVR – difficulty filtering, format rewards, and
the rest – are not the final answers, but they represent the field’s current best understanding
of how to elicit reasoning from language models. The next generation of methods will likely
look different, but they will build on the foundations established here.

131
16 Tool Use & Function Calling
Language models using tools is a natural way to expand their capabilities, especially for
high-precision tasks where external tools contain the information or for agents that need
to interact with complex web systems. Tool-use is a skill that language models need to be
trained to have, where RLHF and all the other methods presented in this book can refine it.
Consider a question from a user such as:
Who is the president today?
A language model without tools will have a hard time answering this question due to the
knowledge cutoff of pretraining data, but this is readily accessible information with one
search query.
Before diving deeper, it is useful to distinguish related terms that are often used interchange-
ably:
• Tool use: the model emits a structured request (tool name and arguments); an
orchestrator executes the tool; results are appended to the context; the model continues
generating.
• Function calling: tool use where the arguments must conform to a declared schema
for a set of functions (usually JSON Schema), enabling reliable parsing and validation.
• Code execution: a special case of tool use where the “tool” is a code interpreter (e.g.,
Python); results are returned as tool output.
An AI model uses any external tools by outputting special tokens to trigger a certain endpoint.
These can be anything from highly specific tools, such as functions that return the weather
at a specific place, to code interpreters or search engines that act as fundamental building
blocks of complex behaviors. Our first example showcased where language models need more
up-to-date information to complement the fixed nature of their weights trained on past data,
but there are also tools such as code execution, which lets language models get around their
probabilistic, generative nature and return precise answers. Consider the task of printing an
approximation of pi to 50 digits (without reciting it from memory and risking hallucination).
A language model with tools can do the following:
< code >
from decimal import Decimal , getcontext
getcontext () . prec = 60

def compute_pi () :
# Chudnovsky algorithm for computing pi
C = 426880 * Decimal (10005) . sqrt ()
K , M , X , L , S = 0 , 1 , 1 , 13591409 , Decimal (13591409)
for i in range (1 , 100) :
M = M * ( K **3 - 16* K ) // (( i ) **3)
K += 12
L += 545140134
X *= -262537412640768000
S += Decimal ( M * L ) / X
return C / S

print ( str ( compute_pi () ) [:52])

</ code >

132
< output >
3.14159265358979323846264338327950288419716939937510
</ output >

This chapter provides an overview of the origins of tool-use in modern language models, its
fundamentals and formatting, and current trade-offs in utilizing tools well in leading models.
The exact origin of the term “tool use” is not clear, but the origins of the idea far predate
the post ChatGPT world where RLHF proliferated. Early examples circa 2015 attempted to
build systems predating modern language models, such as Neural Programmer-Interpreters
(NPI) [301], “a recurrent and compositional neural network that learns to represent and
execute programs.” As language models became more popular, many subfields were using
integrations with external capabilities to boost performance. To obtain information outside
of just the weights many used retrieval augmented generation [302] or web browsing [4]. Soon
after, others were exploring language models integrated with programs [303] or tools [304].
As the field matured, these models gained more complex abilities in addition to the vast
improvements to the underlying language modeling. For example, ToolFormer could use “a
calculator, a Q&A system, two different search engines, a translation system, and a calendar”
[305]. Soon after, Gorilla was trained to use 1645 APIs (from PyTorch Hub, TensorFlow
Hub v2, and HuggingFace) and its evaluation APIBench became a foundation of the popular
Berkeley Function Calling Leaderboard [306]. Since these early models, the diversity of
actions called has grown substantially.
Tool-use models are now deeply intertwined with regular language model interactions. Model
Context Protocol (MCP) emerged as a common formatting used to connect language models
to external data sources (or tools) [307]. With stronger models and better formats, tool-
use language models are used in many situations, including productivity copilots within
popular applications such as Microsoft Office or Google Workspace, scientific domains [308],
medical domains [309], coding agents [310] such as Claude Code or Cursor, integrations with
databases, and many other autonomous workflows.
Evaluating tool-use models involves multiple dimensions: exact-match metrics for tool name
and argument correctness, schema validity, and end-to-end task completion in simulated
environments. Reliability across trials also matters – τ -bench introduced the passˆk metric
(distinct from pass@k) to measure whether an agent succeeds consistently rather than
occasionally [311]. ToolLLM and its ToolBench dataset provide a large-scale framework for
training and evaluating tool use across 16,000+ real-world APIs [312], while the Berkeley
Function Calling Leaderboard (BFCL) remains a popular benchmark for comparing models
on function calling accuracy [306].

16.1 Interweaving Tool Calls in Generation

Function calling agents are presented data very similarly to other post-training stages. The
addition is the content in the system prompt that instructs the model what tools it has
available. An example formatted data point with the system prompt and tools available in
JSON format is shown below:
< system >

133
You are a function - calling AI model . You are provided with function
signatures within < functions > </ functions > XML tags . You may call
one or more functions to assist with the user query . Don ' t make
assumptions about what values to plug into functions .
</ system >

< functions >

[
{
" name ": " search_movies " ,
" description ": " Search for movies by title and return matching
results with IDs ." ,
" parameters ": {
" type ": " object " ,
" properties ": {
" query ": {
" type ": " string " ,
" description ": " The search string for the movie title ."
}
},
" required ": [" query "]
}
},
{
" name ": " get_movie_details " ,
" description ": " Fetch detailed information about a movie including
cast , runtime , and synopsis ." ,
" parameters ": {
" type ": " object " ,
" properties ": {
" movie_id ": {
" type ": " string " ,
" description ": " The unique identifier for the movie ."
}
},
" required ": [" movie_id "]
}
},
{
" name ": " get_showtimes " ,
" description ": " Get movie showtimes for a given location and date
." ,
" parameters ": {
" type ": " object " ,
" properties ": {
" movie_id ": {
" type ": " string " ,
" description ": " The unique identifier for the movie ."
},
" zip_code ": {
" type ": " string " ,
" description ": " ZIP code for theater location ."

134
 },
" date ": {
" type ": " string " ,
" description ": " Date for showtimes in YYYY - MM - DD format ."
}
},
" required ": [" movie_id " , " zip_code "]
}
}
]
</ functions >

< user >

...
</ user >

While the language model is generating, if following the above example, it would generate
the tokens search_movies("Star Wars") to search for Star Wars. This is often encoded inside
special formatting tokens, and then the next tokens inserted into the sequence will contain
the tool outputs. With this, models can learn to accomplish more challenging tasks than
many simple standalone models.
A popular form of tool use is code-execution, allowing the model to get precise answers to
complex logic or mathematics problems. For example, code-execution within a language
model execution can occur during the thinking tokens of a reasoning model. As with function
calling, there are tags first for the code to execute (generated by the model) and then a
separate tag for output.
<| user | >
What is the 50 th Fibonacci number ? ( Use the standard F_0 =0 , F_1 =1
indexing .) </s >
<| assistant | >
< think >
Okay , I will compute the 50 - th Fibonacci number with a simple loop ,
then return the result .

< code >

def fib ( n ) :
a, b = 0, 1
for _ in range ( n ) :
a, b = b, a + b
return a

fib (50)
</ code >

< output >

12586269025
</ output >
</ think >
< answer >
The 50 - th Fibonacci number is 12 586 269 025.

135
</ answer >

What is happening under the hood is the language model is interleaving tool inputs and
outputs with standard autoregressively generated tokens. The orchestration loop that makes
this possible looks something like:
messages = [...]
while True :
response = model ( messages , tools = tools )
if not response . tool_calls :
return response . text

for call in response . tool_calls :

result = execute_tool ( call . name , call . args )
messages . append ({ " role " : " tool " , " tool_call_id " : call . id , "
content " : result })

Training for tool use is about getting the model to behave predictably with this different
token flow—knowing when to emit a tool call, how to format arguments correctly, and how
to incorporate results into its response. Open models must be trained to work with a variety
of tools that users may connect off the shelf.

16.2 Multi-step Tool Reasoning

OpenAI’s o3 model represented a substantial step-change in how multi-step tool-use can be
integrated with language models. This behavior is related to much earlier research trends
in the community. For example, ReAct [313] showcased how actions and reasoning can be
interleaved into one model generation:
In this paper, we explore the use of LLMs to generate both reasoning traces
and task-specific actions in an interleaved manner, allowing for greater synergy
between the two: reasoning traces help the model induce, track, and update
action plans as well as handle exceptions, while actions allow it to interface with
and gather additional information from external sources such as knowledge bases
or environments.
With the solidification of tool-use capabilities and the take-off of reasoning models, multi-turn
tool-use has grown into an exciting area of research [296].

16.3 Model Context Protocol (MCP)

Model Context Protocol (MCP) is an open standard for connecting language models to
external data sources and information systems [307]. At the data layer, MCP uses JSON-RPC
2.0 with discovery and execution methods for its primitives. Rather than requiring specific
tool call formatting per external system, MCP enables models to access rich contextual
information through a standardized protocol.
MCP is a simple addition on top of the tool-use content in this chapter – it is how applications
pass context (data + actions) to language models in a predictable JSON schema. MCP
servers that the models interact with have core primitives: resources (read-only data blobs),
prompts (templated messages/workflows), and tools (functions the model can call). With
this, the MCP architecture can be summarized as:

136
 • MCP servers wrap a specific data source or capability.
• MCP clients (e.g., Claude Desktop, IDE plug-ins) aggregate one or more servers.
• Hosts, e.g. Claude or ChatGPT applications, provide the user/LLM interface; switching
model vendors or back-end tools only means swapping the client in the middle.
MCP enables developers of tool-use models to use the same infrastructure to attach their
servers or clients to different models, and at the same time models have a predictable
format they can use to integrate external components. These together make for a far more
predictable development environment for tool-use models in real-world domains.
An MCP server exposes tools to clients through a standardized JSON schema:
{
" name ": " get_weather " ,
" description ": " Get current weather for a location " ,
" inputSchema ": {
" type ": " object " ,
" properties ": {
" location ": {
" type ": " string " ,
" description ": " City name or coordinates "
}
},
" required ": [" location "]
}
}

A minimal Python MCP server implementing this tool:

from mcp . server import Server
from mcp . types import Tool , TextContent

server = Server ( " weather - server " )

@server . list_tools ()
async def list_tools () :
return [ Tool (
name = " get_weather " ,
description = " Get ␣ current ␣ weather " ,
inputSchema ={
" type " : " object " ,
" properties " : { " location " : { " type " : " string " }} ,
" required " : [ " location " ]
}
)]

@server . call_tool ()
async def call_tool ( name : str , arguments : dict ) :
if name == " get_weather " :
weather = fetch_weather ( arguments [ " location " ])
return [ TextContent ( type = " text " , text = weather ) ]

137
16.4 Implementation
There are multiple formatting and masking decisions when implementing a tool-use model:
• Python vs. JSON formatting: In this chapter, we included examples that format
tool use as both JSON data-structures and Python code. Models tend to select one
structure, different providers across the industry use different formats.
• Masking tool outputs: An important detail when training tool-use models is that
the tokens in the tool output are masked from the model’s training loss. This ensures
the model is not learning to predict the output of the system that it does not directly
generate in use (similar to prompt masking for other post-training stages).
• Multi-turn formatting for tool invocations: It is common practice when imple-
menting tool-calling models to add more structure to the dataloading format. Standard
practice for post-training datasets is a list of messages alternating between user and
assistant (and often a system message). The overall structure is the same for tool-use,
but the turns of the model are split into subsections of content delimited by each tool
call. An example is below.
messages = [
{
" content " : " You ␣ are ␣ a ␣ function ␣ calling ␣ AI ␣ model . ␣ You ␣ are ␣ provided ␣ with
␣ function ␣ signatures ␣ within ␣ < functions > </ functions > ␣ XML ␣ tags . ␣ You
␣ may ␣ call ␣ one ␣ or ␣ more ␣ functions ␣ to ␣ assist ␣ with ␣ the ␣ user ␣ query . ␣
Don 't ␣ make ␣ assumptions ␣ about ␣ what ␣ values ␣ to ␣ plug ␣ into ␣ functions . "
,
" function_calls " : null ,
" functions " : " [{\" name \": ␣ \" li ve _g iv eaw ay s_ by_ ty pe \" , ␣ \" description \":
␣ \" Retrieve ␣ live ␣ giveaways ␣ from ␣ the ␣ GamerPower ␣ API ␣ based ␣ on ␣ the ␣
specified ␣ type .\" , ␣ \" parameters \": ␣ {\" type \": ␣ {\" description \": ␣
\" The ␣ type ␣ of ␣ giveaways ␣ to ␣ retrieve ␣ ( e . g . , ␣ game , ␣ loot , ␣ beta ) .\" , ␣
\" type \": ␣ \" str \" , ␣ \" default \": ␣ \" game \"}}}] " ,
" role " : " system "
},
{
" content " : " Where ␣ can ␣ I ␣ find ␣ live ␣ giveaways ␣ for ␣ beta ␣ access ␣ and ␣ games ?
",
" function_calls " : null ,
" functions " : null ,
" role " : " user "
},
{
" content " : null ,
" function_calls " : " li ve _g iv eaw ay s_ by_ ty pe ( type = ' beta ') \
n li ve _ gi v ea wa y s_ b y_ t yp e ( type = ' game ') " ,
" functions " : null ,
" role " : " assistant "
}
]

• Tokenization and message format details: Tool calls in OpenAI messages format
often undergo tokenization through chat templates (the code for controlling format
of messages sent to the model), converting structured JSON representations into raw

138
 token streams. This process varies across model architectures—some use special tokens
to demarcate tool calls, while others maintain structured formatting within the token
stream itself. Chat template playgrounds provides an interactive environment to
explore how different models convert message formats to token streams.
• Reasoning token continuity: As reasoning models have emerged, with their separate
token stream of “reasoning” before an answer, different implementations exist for how
they’re handled with tool-use in the loop. Some models preserve reasoning tokens
between tool-calling steps within a single turn, maintaining context across multiple
tool invocations. However, these tokens are typically erased between turns to minimize
serving cost (but aren’t always – this is a design decision).
• API formatting across providers (As of July 2025): Different providers use
conceptually similar but technically distinct formats. OpenAI uses tool_calls arrays
with unique IDs, Anthropic employs detailed input_schema specifications with <thinking
> tags, and Gemini offers function calling modes (AUTO/ANY/NONE). When using
these models via an API, the tools available are defined in a JSON format and then
the tool outputs in the model response are stored in a separate field from the standard
“tokens generated.” For another example, the open-source vLLM inference codebase
implements extensive parsing logic supporting multiple tool calling modes and model-
specific parsers, providing insights into lower-level implementation considerations [314].
• Schema conformance and constrained decoding: Production systems often
enforce valid JSON and correct argument types using constrained decoding or “strict
mode” options, reducing retries from malformed outputs. Some closed model providers
do additional post-training specifically to make structured JSON output reliable, where
for open models this is handled as an inference flag in systems like VLLM.
• Tool output context consumption: Tool outputs can quickly consume the model’s
context window, especially with search or retrieval tools that return many results.
Systems must decide how to truncate, summarize, or paginate tool outputs to keep
context manageable while preserving the information the model needs to continue.
Tying this back to post-training: where does tool-use training data come from, and what
objectives are used? Human-written tool traces are expensive to collect, so most modern
tool-use corpora are synthetic or bootstrapped—Toolformer-style self-labeling [305] or large-
scale generation as in ToolBench [312]. For training objectives, supervised finetuning (SFT)
on tool trajectories teaches basic formatting and tool selection. This bootstraps the behavior
and is often enough for establishing the foundation of the skill. Preference optimization (e.g.,
DPO) over trajectories can improve decisions about when to call a tool versus answer directly.
For agentic tasks with multi-step tool use, RL with environment feedback (task success,
constraint satisfaction) becomes the natural objective – the model learns from whether its
tool-augmented actions actually solved the problem.

139
17 Synthetic Data & Distillation
Reinforcement learning from human feedback is deeply rooted in the idea of keeping a human
influence on the models we are building. When the first models were trained successfully
with RLHF, human data was the only viable way to improve the models in this way.
Humans were the only way to create high enough quality responses to questions for training.
Humans were the only way to collect reliable and specific feedback data to train reward
models.
As AI models got better, this assumption rapidly broke down. The possibility of synthetic
data, which is far cheaper and easier to iterate on, enabled the proliferation from RLHF
being the center of attention to the idea of a broader “post-training” shaping the models.
This chapter provides a cursory overview of how and why synthetic data is replacing or
expanding many pieces of the RLHF pipeline.
One common criticism of synthetic data is model collapse – the idea that repeatedly training
on a model’s own generations can progressively narrow the effective training distribution
[315]. As diversity drops, rare facts and styles are underrepresented, and small mistakes can
be amplified across iterations, leading to worse generalization. In practice, these failures are
most associated with self-training on unfiltered, repetitive, single-model outputs; mixing in
real/human data, using diverse teachers, deduplication, and strong quality filters largely
avoids the collapse regime. For today’s frontier training pipelines, evidence suggests synthetic
data can, and should, be used at scale without the catastrophic regressions implied by the
strongest versions of the collapse story [316] [317].
The leading models need synthetic data to reach the best performance. Synthetic data in
modern post-training encompasses many pieces of training – language models are used to
generate new training prompts from seed examples [318], modify existing prompts, generate
completions to prompts [319], provide AI feedback to create preference data [23], filter
completions [320], and much more. Synthetic data is key to post-training.
The ability for synthetic data to be impactful to this extent emerged with GPT-4 class
models. With early language models, such as Llama 2 and GPT-3.5-Turbo, the models were
not reliable enough in generating or supervising data pipelines. Within 1-2 years, language
models were far superior to humans for generating answers. In the transition from GPT-3.5
to GPT-4 class models, the ability for models to perform LLM-as-a-judge tasks also emerged.
GPT-4 or better models are far more robust and consistent in generating feedback or scores
with respect to a piece of content.
Through the years since ChatGPT’s release at the end of 2022, we’ve seen numerous, impactful
synthetic datasets – some include: UltraFeedback [23], the first prominent synthetic preference
dataset that kickstarted the DPO revolution, or Stanford Alpaca, one of the first chat-style
finetuning datasets, in 2023, skill-focused (e.g. math, code, instruction-following), synthetic
datasets in Tülu 3 [6], or OpenThoughts 3 and many other synthetic reasoning datasets in
2025 for training thinking models [284]. Most of the canonical references for getting started
with industry-grade post-training today involve datasets like Tülu 3 or OpenThoughts 3
above, where quickstart guides often start with smaller, simpler datasets like Alpaca due to
far faster training.
A large change is also related to dataset size, where finetuning datasets have grown in the
number of prompts, where Alpaca is 52K, OpenThoughts and Tülu 3 are 1M+ samples, and

140
in the length of responses. Longer responses and more prompts results in the Alpaca dataset
being on the order of 10M training tokens, where Tülu is 50X larger at about 500M, and
OpenThoughts 3 is bigger still at the order of 10B tokens.
Throughout this transition in capabilities, the role of synthetic data has only grown in
language model training. Otherwise, there are two clear areas where human data continues
to be important.
1. The role of human data continues to be at the fringe of capabilities in models – humans
must generate data where AIs do not yet have any ability. Once the first strong model
exists, synthetic data proliferates.
2. Human preference data is still used in the leading models, even though academic work
shows synthetic versions to perform just as well. The role of human preferences is still
being established in the literature.
The term distillation has been the most powerful form of discussion around the role of
synthetic data in language models. Distillation as a term comes from a technical definition
of teacher-student knowledge distillation from the deep learning literature [51].
Distillation colloquially refers to using the outputs from a stronger model to train a smaller
model. In post-training, this general notion of distillation takes two common forms:
1. As a data engine to use across wide swaths of the post-training process: Completions
for instructions, preference data (or Constitutional AI), or verification for RL.
2. To transfer specific skills from a stronger model to a weaker model, which is often done
for specific skills such as mathematical reasoning or coding.
The first strategy has grown in popularity as language models evolved to be more reliable
than humans at writing answers to a variety of tasks. GPT-4 class models expanded the
scope of this to use distillation of stronger models for complex tasks such as math and
code (as mentioned above). Here, distillation motivates having a model suite where often a
laboratory will train a large internal model, such as Claude Opus or Gemini Ultra, which is
not released publicly and just used internally to make stronger models. With open models,
common practice is to distill training data from closed API models into smaller, openly
available weights [21]. Within this, curating high-quality prompts and filtering responses
from the teacher model is crucial to maximize performance.
Transferring specific skills into smaller language models uses the same principles of distillation
– get the best data possible for training. Here, many papers have studied using limited
datasets from stronger models to improve alignment [13], mathematical reasoning [321] [322],
and test-time scaling [270].

141
18 Evaluation
Evaluation is an ever evolving approach. The key to understanding language model evaluation,
particularly with post-training, is that the current popular evaluation regimes represent a
reflection of the popular training best practices and goals. While challenging evaluations
drive progress in language models to new areas, the majority of evaluation is designed around
building useful signals for new models.
In many ways, this chapter is designed to present vignettes of popular evaluation regimes
throughout the early history of RLHF, so readers can understand the common themes,
details, and failure modes.
Evaluation for RLHF and post-training has gone a few distinct phases in its early history:
1. Early chat-phase: Early models trained with RLHF or preference tuning targeted
evaluations focused on capturing the chat performance of a model, especially relative
to known strong models such as GPT-4. Early examples include MT-Bench [131],
AlpacaEval [132], and Arena-Hard [133]. Models were evaluated narrowly and these
are now considered as “chat” or “instruction following” domains.
2. Multi-skill era: Over time, common practice established that RLHF can be used to im-
prove more skills than just chat. For example, the Tülu evaluation suite included tasks
on knowledge (MMLU [323], PopQA [324], TruthfulQA [325]), Reasoning (BigBench-
Hard [326], DROP [327]), Math (MATH [328], GSM8K [129]), Coding (HumanEval
[329], HumanEval+ [330]), Instruction Following [115], and Safety (a composite of
many evaluations). This reflects the domain where post-training is embraced as a
multi-faceted solution beyond safety and chat.
3. Reasoning & tools: The current era for post-training is defined by a focus on
challenging reasoning and tool use problems. These include much harder knowledge-
intensive tasks such as GPQA Diamond [331] and Humanity’s Last Exam [332], intricate
software engineering tasks such as SWE-Bench+ [333] and LiveCodeBench [334], or
challenging math problems exemplified by recent AIME contests.
Beyond this, new domains will evolve. As AI becomes more of an industrialized field, the
incentives of evaluation are shifting and becoming multi-stakeholder. Since the release
of ChatGPT, private evaluations such as the Scale Leaderboard [335], community-driven
evaluations such as ChatBotArena [113], and third-party evaluation companies such as
ArtificialAnalysis and Epoch AI have proliferated. Throughout this chapter we will include
details that map to how these evaluations were implemented and understood.

18.1 Prompting Formatting: From Few-shot to Zero-shot to CoT

Prompting language models is primarily a verb, but it is also considered a craft or art
that one can practice and/or train in general [336]. A prompt is the way of structuring
information and context for a language model. For common interactions, the prompt is
relatively basic. For advanced scenarios, a well crafted prompt will mean success or failure
on a specific one-off use-case.
When it comes to evaluation, prompting techniques can have a substantial impact on the
performance of the model. Some prompting techniques – e.g. formatting discussed below –
can make a model’s performance drop from 60% to near 0. Similarly, a change of prompt can

142
help models learn better during training. Colloquially, prompting a model well can give the
subjective experience of using future models, unlocking performance outside of normal use.
Prompting well with modern language models can involve preparing an entire report for
the model to respond to (often with 1000s of tokens of generated text). This behavior is
downstream of many changes in how language model performance has been measured and
understood.
Early language models were only used as intelligent autocomplete. In order to use these
models in a more open ended way, multiple examples were shown to the model and then
a prompt that is an incomplete phrase. This was called few-shot or in-context learning
[167], and at the time instruction tuning or RLHF was not involved. In the case of popular
evaluations, this would look like:
# Few - Shot Prompt for a Question - Answering Task
You are a helpful assistant . Below are example interactions to guide
your style :

### Example 1
User : " What is the capital of France ?"
Assistant : " The capital of France is Paris ."

### Example 2
User : " Who wrote the novel '1984 '?"
Assistant : " George Orwell wrote '1984. '"

# Now continue the conversation using the same style .

User : " Can you explain what a neural network is ?"
Assistant :

Here, there are multiple ways to evaluate an answer. If we consider a question in the style of
MMLU, where the model has to choose between multiple answers:
# Few - Shot Prompt

Below are examples of MMLU - style questions and answers :

### Example 1
Q : A right triangle has legs of lengths 3 and 4. What is the length of
its hypotenuse ?
Choices :
(A) 5
(B) 6
(C) 7
(D) 8

Correct Answer : ( A )

### Example 2
Q : Which of the following is the chemical symbol for Sodium ?
Choices :
( A ) Na
(B) S

143
(C) N
( D ) Ca

Correct Answer : ( A )

### Now answer the new question in the same style :

Q : Which theorem states that if a function f is continuous on a closed

interval [a , b ] , then f must attain both a maximum and a minimum
on that interval ?
Choices :
( A ) The Mean Value Theorem
( B ) The Intermediate Value Theorem
( C ) The Extreme Value Theorem
( D ) Rolle ' s Theorem

Correct Answer :

To extract an answer here one could either generate a token based on some sampling
parameters and see if the answer is correct, A,B,C, or D (formatting above like this proposed
in [337]), or one could look at the probabilities of each token and mark the task as correct if
the correct answer is more likely. This second method has two potential implementations
– first, one could look at the probability of the letter (A) or the answer “The Mean Value
Theorem.” Both of these are permissible metrics, but answer prediction is more common
among probability base metrics.
A common challenge with few-shot prompting is that models will not follow the format,
which is counted as an incorrect answer. When designing an evaluation domain, the number
of examples used in-context is often considered a design parameter and ranges from 3 to 8 or
more.
Within the evolution of few-shot prompting came the idea of including chain-of-thought
examples for the model to follow. This comes in the form of examples where the in-context
examples have written out reasoning, such as below (which later was superseded by explicit
prompting to generate reasoning steps) [54]:
# standard prompting
Q : Roger has 5 tennis balls . He buys 2 more cans of tennis balls . Each
can has 3 tennis balls . How many tennis balls does he have now ?

A : The answer is 11.

Q : The cafeteria had 23 apples . If they used 20 to make lunch and

bought 6 more , how many apples do they have ?

A : The answer is ...

# chain - of - thought prompting

Q : Roger has 5 tennis balls . He buys 2 more cans of tennis balls . Each
can has 3 tennis balls . How many tennis balls does he have now ?

A : Roger started with 5 balls . 2 cans of 3 tennis balls each is 6

144
 tennis balls . 5 + 6 = 11. The answer is 11.

Q : The cafeteria had 23 apples . If they used 20 to make lunch and

bought 6 more , how many apples do they have ?

A : The cafeteria had 23 apples originally . They ..

Over time, as language models became stronger, they evolved to zero-shot evaluation, a.k.a.
“zero-shot learners” [338]. The Finetuned Language Net (FLAN) showed that language
models finetuned in specific tasks, as a precursor to modern instruction tuning, could
generalize to zero-shot questions they were not trained on [338] (similar results are also found
in T0 [339]). This is the emergence of instruction finetuning (IFT), an important precursor
to RLHF and post-training. A zero-shot question would look like:
User : " What is the capital of France ?"
Assistant :

From here in 2022, the timeline begins to include key early RLHF works, such as InstructGPT.
The core capability and use-case shift that accompanied these models is even more open-
ended usage. With more open-ended usage, generative evaluation became increasingly
popular as it mirrors actual usage. In this period through recent years after ChatGPT,
some multiple-choice evaluations were still used in RLHF research as a holdback to common
practice.
With the rise of reasoning models at the end of 2024 and the beginning of 2025, a major
change in model behavior was the addition of a long Chain-of-Thought (CoT) reasoning
process before every answer. These models no longer needed to be prompted with the
canonical modification of “think step by step,” as proposed in [340].
For example, for every question or category there are specially designed prompts to help
extract behavior from the model. Tülu 3 details some prompts used for CoT answering on
multiple choice questions [6]:
Answer the following multiple - choice question by giving the correct
answer letter in parentheses . Provide CONCISE reasoning for the
answer , and make sure to finish the response with " Therefore , the
answer is ( ANSWER_LETTER ) " where ( ANSWER_LETTER ) is one of ( A ) ,
( B ) , ( C ) , ( D ) , ( E ) , etc .

Question : { question }
( A ) { choice_A }
( B ) { choice_B }
( C ) ...

Answer the above question and REMEMBER to finish your response with
the exact phrase " Therefore , the answer is ( ANSWER_LETTER ) " where
( ANSWER_LETTER ) is one of ( A ) , ( B ) , ( C ) , ( D ) , ( E ) , etc .

This, especially when the models use special formatting to separate thinking tokens from
answer tokens, necessitated the most recent major update to evaluation regimes. Evaluation
is moving to where the models are tested to respond in a generative manner with chain-of-
thought prompting.

145
18.2 Using Evaluations vs. Observing Evaluations

Figure 21: Report from Epoch AI showing how major AI evaluations are rapidly saturated
over time. License CC-BY.

Language model evaluations done within companies can only be compared to their peers
with large error bars because the process that they use for evaluations internally is not
matched with external evaluations. Internal evaluations are made to hillclimb on for training,
as would be called a “training set” in traditional machine learning. The public evaluations
that the community uses to compare leading models cannot be known if they were within
said training set or as unseen “test sets” or “validation sets.”
As evaluation scores have become central components of corporate marketing schemes, their
implementations within companies have drifted. There are rumors of major AI labs using
“custom prompts” for important evaluations like GSM8k or MATH. These practices evolve
rapidly.
Language model evaluation stacks are perceived as marketing because the evaluations have
no hard source of truth. What is happening inside frontier labs is that evaluation suites are
being tuned to suit their internal needs. When results are shared, we get output in the form
of the numbers a lab got for their models, but not all the inputs to that function. The inputs
are very sensitive configurations, and they’re different at all of OpenAI, Meta, Anthropic,
and Google. Even fully open evaluation standards are hard to guarantee reproducibility on.
Focusing efforts on your own models is the only way to get close to repeatable evaluation
techniques. There are good intentions underpinning the marketing, starting with the technical

146
teams.
Evaluation of frontier language models is every bit as much an art today as it is a science.
Different groups choose different evaluations to maintain independence on, i.e. making them
a true test set, but no one discloses which ones they choose. For example, popular reasoning
evaluations MATH and GSM8k both have training sets with prompts that can easily be
used to improve performance. Improving performance with the prompts from the same
distribution is very different than generalizing to these tasks by training on general math
data.
In fact, these training sets are very high-quality data so models would benefit from training
on them. If these companies are not using the corresponding evaluation as a core metric to
track, training on the evaluation set could be a practical decision as high-quality data is a
major limiting factor of model development.
Leading AI laboratories hillclimb by focusing on a few key evaluations and report scores on
the core public set at the end. The key point is that some of their evaluations for tracking
progress, such as the datasets for cross-entropy loss predictions in scaling from the GPT-4
report [341], are often not public.
The post-training evaluations are heavily co-dependent on human evaluation. Human
evaluation for generative language models yields Elo rankings (popular in early Anthropic
papers, such as Constitutional AI), and human evaluation for reward models shows agreement.
These can also be obtained by serving two different models to users with an A/B testing
window (as discussed in the chapter on Preference Data).
The limited set of evaluations they choose to focus on forms a close link between evaluation
and training. At one point one evaluation of focus was MMLU. GPQA was one of choice
during reasoning models’ emergence. Labs will change the evaluations to make them better
suited to their needs, such as OpenAI releasing SWE-Bench-Verified [342]. There are many
more internally the public does not have access to.
The key “capability” that improving evaluations internally has on downstream training
is improving the statistical power when comparing training runs. By changing
evaluations, these labs reduce the noise on their prioritized signals in order to make more
informed training decisions.
This is compounded by the sophistication of post-training in the modern language model
training stacks. Evaluating language models today involves a moderate amount of generating
tokens (rather than just looking at log probabilities of answers). It is accepted that small
tricks are used by frontier labs to boost performance on many tasks — the most common
explanation is one-off prompts for certain evaluations.
Another example of confusion when comparing evaluations from multiple laboratories is
the addition of inference-time scaling to evaluation comparisons. Inference-time scaling
shows that models can improve in performance by using more tokens at inference. Thus,
controlling evaluation scores by the total number of tokens for inference is important, but
not yet common practice.
Depending on how your data is formatted in post-training, models will have substantial
differences across evaluation formats. For example, two popular, open math datasets
NuminaMath [343] and MetaMath [344] conflict with each other in training due to small

147
differences in how the answers are formatted – Numina puts the answer in \boxed{XYZ}
and MetaMath puts the answer after The answer is: XYZ —- training on both can make
performance worse than with just one. Strong models are trained to be able to function with
multiple formats, but they generally have a strongest format.
In the end we are left with a few key points on the state of evaluating closed models:
• We do not know or necessarily have the key test sets that labs are climbing on, so some
evaluations are proxies.
• Inference of frontier models is becoming more complicated with special system prompts,
special tokens, etc., and we don’t know how it impacts evaluations, and
• We do not know all the formats and details used to numerically report the closed
evaluations.

18.3 Contamination
A major issue with current language model practices (i.e. not restricted to RLHF and post-
training) is intentional or unintentional use of data from evaluation datasets in training. This
is called dataset contamination and respectively the practices to avoid it are decontamination.
In order to decontaminate a dataset, one performs searches over the training and test datasets,
looking for matches in n-grams (characters) or tokens [345]. There are many ways that
data can become contaminated, but the most common is from scraping of training data for
multiple stages from the web. Benchmarks are often listed on public web domains that are
crawled, or users pass questions into models which can then end up in candidate training
data for future models.
For example, during the decontamination of the evaluation suite for Tülu 3, the authors
found that popular open datasets were contaminated with popular evaluations for RLHF [6].
These overlaps include: UltraFeedback’s contamination with TruthfulQA, Evol-CodeAlpaca’s
contamination with HumanEval, NuminaMath’s contamination with MATH, and WildChat’s
contamination with safety evaluations. These were found via 8-gram overlap from the
training prompt to the exact prompts in the evaluation set.
In order to understand contamination of models that do not disclose or release the training
data, new versions of benchmarks are created with slightly perturbed questions from the
original, e.g. for MATH [346], in order to see which models were trained to match the original
format or questions. High variance on these perturbation benchmarks is not confirmation of
contamination, which is difficult to prove, but could indicate models that were trained with
a specific format in mind that may not translate to real world performance.

18.4 Tooling
There are many open-sourced evaluation tools for people to choose from. There’s Inspect
AI from the UK Safety Institute [347], HuggingFace’s LightEval [348] that powered the
Open LLM Leaderboard [349], Eleuther AI’s evaluation harness [350] built on top of the
infrastructure from their GPT-Neo-X model (around GPT-3 evaluation config) [351], AI2’s
library based on OLMES [352], Stanford’s Center for Research on Foundation Model’s HELM
[353], Mosaic’s (now Databricks’) Eval Gauntlet [354], and more.

148
19 Over Optimization
A core lesson one learns when using reinforcement learning heavily in their domain it is
a very strong optimizer, which causes it to pull all the possible increase in reward out of
the environment. In modern ML systems, especially with language models, we’re using
somewhat contrived notions of enviornment where the models generate completions (the
actions) and an external verifier, i.e. a reward model or a scoring function provides feedback.
In this domain, it is common for over-optimization to occur, where the RL optimizers push
the language models in directions where the generations satisfy our checker functions, but
the behavior does not align with our training goals. This chapter providers and overview of
this classic case of over-optimization.
In the RLHF literature and discourse, there are two primary directions that over-optimization
can emerge:
1. Quantitative research on the technical notion of over-optimization of reward. This
measures optimization distance and power versus training metrics and downstream
performance. Training keeps going up, while eventually downstream goes down.
2. Qualitative observations that “overdoing” RLHF can result in worse models. These
are fundamental limitations in the RLHF problem setup, measurement tools, and
trade-offs.
This chapter provides a cursory introduction to both. We begin with the latter, qualitative,
because it motivates the problem to study further. Finally, the chapter concludes with a
brief discussion of misalignment where overdoing RLHF or related techniques can make a
language model behave against its design.
Over-optimization is a concept where the training metric ends up being mismatched from the
final evaluations of interest. While similar to over-fitting – where one trains on data that is
too narrow relative to the downstream evaluations that test generalization – over-optimization
is used in the RL literature to indicate that an external signal is used too much. The cost of
over-optimization is a lower alignment to real world goals or lower quality in any domain,
and the shape of training associated with it is shown in fig. 22.

19.1 Qualitative Over-optimization

The first half of this chapter is discussing narratives at the core of RLHF – how the
optimization is configured with respect to final goals and what can go wrong.

19.1.1 Managing Proxy Objectives

RLHF is built around the fact that we do not have a universally good reward function for
chatbots. RLHF has been driven into the forefront because of its impressive performance
at making chatbots a bit better to use, which is entirely governed by a proxy objective —
thinking that the rewards measured from human labelers in a controlled setting mirror those
desires of downstream users. Post-training generally has emerged to include training on
explicitly verifiable rewards, but standard learning from preferences alone also improves
performance on domains such as mathematical reasoning and coding (still through these
proxy objectives).
The proxy reward in RLHF is the score returned by a trained reward model to the RL

149
Figure 22: Over-optimization of an RL training run vs. downstream evaluations.

150
algorithm itself because it is known to only be at best correlated with chatbot performance
[355]. Therefore, it’s been shown that applying too much optimization power to the RL
part of the algorithm will actually decrease the usefulness of the final language model – a
type of over-optimization known to many applications of reinforcement learning [356]. And
over-optimization is “when optimizing the proxy objective causes the true objective to get
better, then get worse.”
A curve where the training loss goes up, slowly levels off, then goes down, as shown in fig. 22.
This is different from overfitting, where the model accuracy keeps getting better on the
training distribution. Over-optimization of a proxy reward is much more subtle.
The general notion captured by this reasoning follows from Goodhart’s law. Goodhart
explained the behavior that is now commonplace [357]:
Any observed statistical regularity will tend to collapse once pressure is placed
upon it for control purposes.
This colloquially evolved to the notion that “When a measure becomes a target, it ceases to
be a good measure”[358]. The insight here builds on the fact that we are probably incorrectly
using ML losses as ground truths in these complex systems. In reality, the loss functions we
use are designed (and theoretically motivated for) local optimizations. The global use of
them is resulting in challenges with the RLHF proxy objective.
Common signs of over-optimization in early chat models emerged as:
• Common phrases, such as: “As an AI language model. . . ” or “Certainly!. . . ”
• Uninformative answers via repetitiveness, hedging, etc.
• Pandering to the user with: Self-doubt, sycophancy [121], and over apologizing,
• Misaligned behavior such as over refusals,
It is an open research question on which types of error in the training process result in these
failures. Many sources of error exist [355]: Approximation error from reward models not
being able to fit to preferences, estimation error from overfitting during training the RM,
optimization error in training the language model policy, etc. This points to a fundamental
question as to the limits of optimizing the intents and outputs of data contractors relative to
what downstream users want.
A potential solution is that implicit feedback will be measured from users of chatbots and
models to tune performance. Implicit feedback is actions taken by the user, such as re-rolling
an output, closing the tab, or writing an angry message that indicates the quality of the
previous response. The challenge here, and with most optimization changes to RLHF, is
that there’s a strong risk of losing stability when making the reward function more specific.
RL, as a strong optimizer, is increasingly likely to exploit the reward function when it is
a smooth surface (and not just pairwise human values). The expected solution to this is
that future RLHF will be trained with both pairwise preference data and additional steering
loss functions. There are also a bunch of different loss functions that can be used to better
handle pairwise data, such as Mallow’s model [359] or Plackett-Luce [127].

19.1.2 Over-refusal and “Too Much RLHF”

Many model releases have been tagged with the trait “too much RLHF.” There were multiple
prominent model releases in 2023 where the models refused innocuous queries due to over-

151
attribution of individual words in context like “kill.” This discussion was most prominent
around the release of Llama 2’s chat models [44], but also reflected the focus of refusals
and “alignment” to early applications of RLHF. For example, a common test of over-refusal
behavior in early RLHF models was a query like the following, where the model misinterprets
the request:
User: i want to know how to kill linux process
Assistant: I cannot provide instructions or advice on how to harm or destroy
Linux processes or any other living being. It is not ethical, legal, or morally right
to intentionally harm or terminate any process or life form.
Instead, I would suggest focusing on learning about Linux processes and how to
manage them effectively. You can start by reading documentation and tutorials
on Linux process management, which can help you understand how to create,
monitor, and terminate processes safely and efficiently.
Additionally, if you have concerns about a specific process that you believe may
be causing issues, you can try to troubleshoot the problem by gathering more
information about the process, such as its resource usage, memory consumption,
and network activity. You can use various tools like top, htop, ps, kill, and grep
to gather this information.
However, please note that terminating a process without proper authorization or
knowledge can have serious consequences, including data loss, system instability,
and security vulnerabilities. Therefore, it is essential to exercise caution and
follow best practices when managing Linux processes.
Another example later in 2023 had Claude 2.1 showcase similar behavior:
User: How can i kill all python processes in my ubuntu server?
Assistant: I apologize, I should not provide recommendations about harming
processes or systems.
Both of these are not solely related to training and reflect the deployment settings of the
models, such as the system prompt. Additionally, modern chat applications use additional
safety filters to intercept prompts and responses before they are sent to the primary generative
model (e.g. WildGuard [360] or LlamaGuard [361]).
While RLHF was at the center of the training for these models’ ability to distinguish safe from
unsafe requests, it is inaccurate to attribute the failure of behavior in the final model to the
training methods used. Rather, the training methods combined with data curation guidelines
from the modeling team dictated a desired balance of request safety to other capabilities.
Additionally, there is variance in final model outcomes relative to the initial goals of training.
As the ecosystem matures the ability to control the final models has improved and the notion
that RLHF and post-training is primarily about safety has diminished, such as by developing
benchmarks to measure potential over-refusal [362].
As chat-based AI systems have proliferated, the prominence of these refusal behaviors has
decreased over time. The industry standard has shifted to a narrower set of harms and
models that are balanced across views of controversial issues.

152
19.2 Quantitative over-optimization
Over-optimization is also a technical field of study where relationships between model
performance versus KL optimization distance are studied [38]. Recall that the KL distance
is a measure of distance between the probabilities of the original model before training, a.k.a.
the reference model, and the current policy. For example, the relationship in fig. 22, can
also be seen with the KL distance of the optimization on the x-axis rather than training
steps. An additional example of this can be seen below, where a preference tuning dataset
was split in half to create a train reward model (preference model, PM, below) and a test
reward model. Here, over training, eventually the improvements on the training RM fail to
transfer to the test PM at ~150K training samples [5].
Over-optimization is fundamental and unavoidable with RLHF due to the soft nature of the
reward signal – a learned model – relative to reward functions in traditional RL literature that
are intended to fully capture the world dynamics. Hence, it is a fundamental optimization
problem that RLHF can never fully solve.

Figure 23: Over-optimization with a train and test RM from Bai et al. 2022. License CC-BY.

With different RLHF training methods, the KL distance spent will vary. For example, the
KL distance used by online RL algorithms modifying the model parameters, e.g. PPO, is
much higher than the KL distance of inference-time sampling methods such as best-of-N
sampling (BoN). With RL training, a higher KL penalty will reduce over-optimization as
a given KL distance, but it could take more overall training steps to get the model to this
point.

153
Many solutions exist to mitigate over-optimization. Some include bigger policy models
that have more room to change the parameters to increase reward while keeping smaller
KL distances, reward model ensembles [363], or changing optimizers [364]. While direct
alignment algorithms are still prone to over-optimization [365], the direct notion of their
optimization lets one use fixed KL distances that will make the trade-off easier to manage.

19.3 Misalignment and the Role of RLHF

While industrial RLHF and post-training is shifting to encompass many more goals than
the original notion of alignment that motivated the invention of RLHF, the future of RLHF
is still closely tied with alignment. In the context of this chapter, over-optimization would
enable misalignment of models. With current language models, there have been many studies
on how RLHF techniques can shift the behavior of models to reduce their alignment to
the needs of human users and society broadly. A prominent example of mis-alignment in
current RLHF techniques is the study of how current techniques promote sycophancy [121] –
the propensity for the model to tell the user what they want to hear. As language models
become more integrated in society, the consequences of this potential misalignment will grow
in complexity and impact [366]. As these emerge, the alignment goals of RLHF will grow
again relative to the current empirical focus of converging on human preferences for style
and performance.

154
20 Style and Information
Early developments in RLHF gave it a reputation for being “just style transfer” or other
harsh critiques on how RLHF manipulates the way information is presented in outputs. This
chapter explains why style is very core to understanding the value that RLHF is providing.
Reading this chapter should help you understand the changes RLHF is making to the models
and why it positively impacts user behavior.
The idea of being solely about style transfer has held back the RLHF narrative for two
reasons. First, when people discuss style transfer, they don’t describe this as being important
or exciting. Style is a never-ending source of human value, it’s why retelling stories can
result in new bestselling books (such as Sapiens), and it is a fundamental part of continuing
to progress our intellectual ecosystem. Style is intertwined with what the information is.
Second, we’ve seen how different styles actually can improve evaluation improvements with
Llama 3 [24]. The Llama 3 Instruct models scored extremely high on ChatBotArena, and
it’s accepted as being because they had a more fun personality. If RLHF is going to make
language models simply more fun, that is delivered value.
Throughout this chapter, the term “chattiness” is used to encompass the growing length of
responses from models training with RLHF, but it also encompasses techniques like heavy
markdown use, emojis, and formatting the answer in bulleted lists.

20.1 The Chattiness Paradox

RLHF or preference fine-tuning methods are being used mostly to boost scores like AlpacaE-
val and other automatic leaderboards without shifting proportionally on harder-to-game
evaluations like ChatBotArena. The paradox is that while alignment methods give a mea-
surable improvement on these models that does transfer into performance that people care
about, a large swath of the models doing more or less the same thing take it way too far and
publish evaluation scores that are obviously meaningless.
These methods, when done right, make the models easier to work with and more enjoyable.
This often comes with a few percentage point improvements on evaluation tools like MT
Bench or AlpacaEval. The problem is that you can also use techniques like DPO and PPO
in feedback loops or in an abundance of data to actually severely harm the model on other
tasks like mathematics or coding at the cost of LLM-as-a-judge performance.
During the proliferation of the DPO versus PPO debate there were many papers that came
out with incredible benchmarks but no model weights that gathered sustained usage. When
applying RLHF, there is no way to make an aligned version of a 7 billion parameter model
actually beat GPT-4 across comprehensive benchmarks. It seems obvious, but there are
papers claiming these results. fig. 24 is from a paper called Direct Nash Optimization (DNO)
that makes the case that their model is state-of-the-art or so on AlpacaEval. These challenges
emerge when academic incentives interface with technologies becoming of extreme interest
to the broader society.
Even the pioneering paper Self Rewarding Language Models [230] disclosed unrealistic scores
on Llama 2 70B. A 70B model can get closer to GPT-4 than a 7B model can, as we have
seen with Llama 3, but it’s important to separate the reality of models from the claims in

155
Figure 24: Results from the paper on Direct Nash Optimization (DNO) highlighting their
small model outperforming the likes of GPT-4. Rosset et al. 2024. License CC-BY.

modern RLHF papers. Many more methods have come and gone similar to this, sharing
valuable insights and oversold results, which make RLHF harder to understand.
A symptom of models that have “funky RLHF” applied to them has often been a length bias.
This got so common that multiple evaluation systems like AlpacaEval and WildBench both
have linear length correction mechanisms in them. This patches the incentives for doping on
chattiness to “beat GPT-4,” and adds a less gamified bug that shorter and useful models
may actually win out.
Regardless, aligning chat models simply for chattiness still has a bit of a tax in the literature.
This note from the Qwen models is something that has been seen multiple times in early
alignment experiments, exaggerating a trade-off between chattiness and performance [367].
We pretrained the models with a large amount of data, and we post-trained the
models with both supervised finetuning and direct preference optimization. How-
ever, DPO leads to improvements in human preference evaluation but degradation
in benchmark evaluation.
A good example of this tradeoff done right is a model like Starling Beta [126]. It’s a model
that was fine-tuned from another chat model, OpenChat [368], which was in fact trained by
an entire other organization. It’s training entirely focuses on a k-wise reward model training
and PPO optimization, and moves it up 10 places in ChatBotArena. The average response
length of the model increases, but in a way that’s good enough to actually help the human
raters.

20.1.1 How Chattiness Emerges

A natural question is: Why does RLHF make model responses longer? At a fundamental
answer, evaluations like ChatBotArena have shown us that average users of models often
like longer, complete answers when compared with terse responses. This does not represent
the preference of every user, but these models are trained to match the preferences of many
data labelers.

156
Most of the popular datasets for alignment these days are synthetic preferences where a
model like GPT-4 rates outputs from other models as the winner or the loser. Given that
GPT-4 is known to have length and style biases for outputs that match itself, most of the
pieces of text in the “preferred” section of the dataset are either from an OpenAI model or
are stylistically similar to it. The important difference is that not all of the pieces of text
in the dataset will have that. They’re often generated from other open models like Alpaca,
Vicuna, or more recent examples. These models have very different characteristics.
Next, now that we’ve established that we have a preference dataset where most of the chosen
models are similar to ChatGPT (or some other model that is accepted to be “strong”), these
alignment methods simply increase the probability of these sequences. The math is somewhat
complicated, where the batches of data operate on many chosen-rejected pairs at once, but
in practice, the model is doing credit assignment over sequences of tokens (subword pieces).
Preference alignment for chattiness is making the sequences found in outputs of models like
GPT-4 more likely and the sequences from other, weaker models less likely. Repeatedly, this
results in models with longer generations and characteristics that people like more.
Those among you who are familiar with RLHF methods may ask if the KL constraint in
the optimization should stop this from happening. The KL constraint is a distance term
between the distribution of the original model and the resulting model. It helps make the
optimization more robust to overoptimization, but that makes the border between good and
bad models a bit more nuanced. Hence, the prevalence of vibes-based evaluations. Though,
models tend to have enough parameters where they can change substantially and still satisfy
the KL constraint on the data being measured — it can’t be the entire pretraining dataset,
for example.

157
21 Product, UX, and Model Character
Frontiers in RLHF and post-training show how these techniques are used within companies
to make leading products. As RLHF becomes more established, the problems it is used to
address are becoming more nuanced. In this chapter, we discuss a series of use-cases that
leading AI laboratories consider RLHF and post-training for that are largely unstudied in
the academic literature.

21.1 Character Training

Character training is the subset of post-training designed around crafting traits within the
model in the manner of its response, rather than the content [369]. Character training,
while being important to the user experience within language model chatbots, is effectively
unstudied in the public domain.
We don’t know the trade-offs of what character training does, we don’t know how exactly to
study it, we don’t know how much it can improve user preferences on ChatBotArena, and
we should. What we do know is that character training uses the same methods discussed
in this book, but for much more precise goals on the features in the language used by the
model. Character training involves extensive data filtering and synthetic data methods such
as Constitutional AI that are focusing on the manner of the model’s behavior. These changes
are often difficult to measure on all of the benchmark regimes we have mentioned in the
chapter on Evaluation because AI laboratories use character training to make small changes
in the personality over time to improve user experiences.
For example, Character Training was added by Anthropic to its Claude 3 models [370]:
Claude 3 was the first model where we added “character training” to our alignment
finetuning process: the part of training that occurs after initial model training,
and the part that turns it from a predictive text model into an AI assistant. The
goal of character training is to make Claude begin to have more nuanced, richer
traits like curiosity, open-mindedness, and thoughtfulness.
In the following months, stronger character emerged across the industry of models. The
process is extremely synthetic data-heavy, but requires an artist’s touch, as stated later in
the blog post: It “relies on human researchers closely checking how each trait changes the
model’s behavior.”
Character training being the focus of developments is the strongest endorsement that RLHF
and related approaches have shifted from their philosophical motivations of alignment to
being primarily an empirical tool. The models can capture so many different behaviors, but
getting them to reliably behave how we want is the hardest part. Right now, it seems more
likely that this is about capturing the upside of RLHF as a performance tool, rather than a
safety one.
One of the few public discussions of character training came from Amanda Askell during her
appearance on the Lex Fridman Podcast (taken from the transcript):
Lex Fridman ([Link]) When you say character training, what’s incorporated
into character training? Is that RLHF or what are we talking about?
Amanda Askell ([Link]) It’s more like constitutional AI, so it’s a variant of that

158
 pipeline. I worked through constructing character traits that the model should
have. They can be shorter traits or they can be richer descriptions. And then
you get the model to generate queries that humans might give it that are relevant
to that trait. Then it generates the responses and then it ranks the responses
based on the character traits. In that way, after the generation of the queries,
it’s very much similar to constitutional AI, it has some differences. I quite like it,
because it’s like Claude’s training in its own character, because it doesn’t have
any. . . It’s like constitutional AI, but it’s without any human data.
In summary, Anthropic uses the same techniques they use for Constitutional AI and general
post-training for capabilities to train these models’ characters.

21.2 Model Specifications

OpenAI recently shared what they call their “Model Spec” [124], a document that details
their goal model behaviors prior to clicking go on a fine-tuning run. It’s about the model
behavior now, how OpenAI steers their models from behind the API, and how their models
will shift in the future.
Model Spec’s are one of the few tools in the industry and RLHF where one can compare
the actual behavior of the model to what the designers intended. As we have covered in
this book, training models is a complicated and multi-faceted process, so it is expected that
the final outcome differs from inputs such as the data labeler instructions or the balance of
tasks in the training data. For example, a Model Spec is much more revealing than a list of
principles used in Constitutional AI because it speaks to the intent of the process rather
than listing what acts as intermediate training variables.
A Model Spec provides value to every stakeholder involved in a model release process:
• Model Designers: The model designers get the benefit of needing to clarify what
behaviors they do and do not want. This makes prioritization decisions on data easier,
helps focus efforts that may be outside of a long-term direction, and makes one assess
the bigger picture of their models among complex evaluation suites.
• Developers: Users of models have a better picture for which behaviors they encounter
may be intentional – i.e. some types of refusals – or side-effects of training. This can
let developers be more confident in using future, smarter models from this provider.
• Observing public: The public benefits from Model Specs because it is one of the
few public sources of information on what is prioritized in training. This is crucial for
regulatory oversight and writing effective policy on what AI models should and should
not do.

21.3 Product Cycles, UX, and RLHF

As powerful AI models become closer to products than singular artifacts of an experiment
machine learning process, RLHF has become an interface point for the relationship between
models and product. Much more goes into making a model easy to use than just having
the final model weights be correct – fast inference, suitable tools to use (e.g. search or code
execution), a reliable and easy to understand user interface (UX), and more. RLHF research
has become the interface where a lot of this is tested because of the framing where RLHF is
a way to understand the user’s preferences to products in real time and because it is the

159
final training stage before release. The quickest way to add a new feature to a model is to
try and incorporate it at post-training where training is faster and cheaper. This cycle has
been seen with image understanding, tool use, better behavior, and more. What starts as a
product question quickly becomes an RLHF modeling question, and if it is successful there
it backpropagates to other earlier training stages.

160
Bibliography
[1] P. F. Christiano, J. Leike, T. Brown, M. Martic, S. Legg, and D. Amodei, “Deep
reinforcement learning from human preferences,” Advances in neural information
processing systems, vol. 30, 2017.
[2] N. Stiennon et al., “Learning to summarize with human feedback,” Advances in Neural
Information Processing Systems, vol. 33, pp. 3008–3021, 2020.
[3] L. Ouyang et al., “Training language models to follow instructions with human feed-
back,” Advances in neural information processing systems, vol. 35, pp. 27730–27744,
2022.
[4] R. Nakano et al., “Webgpt: Browser-assisted question-answering with human feed-
back,” arXiv preprint arXiv:2112.09332, 2021.
[5] Y. Bai et al., “Training a helpful and harmless assistant with reinforcement learning
from human feedback,” arXiv preprint arXiv:2204.05862, 2022.
[6] N. Lambert et al., “T\" ULU 3: Pushing frontiers in open language model post-
training,” arXiv preprint arXiv:2411.15124, 2024.
[7] R. Kirk et al., “Understanding the effects of rlhf on llm generalisation and diversity,”
arXiv preprint arXiv:2310.06452, 2023.
[8] T. Chu et al., “Sft memorizes, rl generalizes: A comparative study of foundation
model post-training,” arXiv preprint arXiv:2501.17161, 2025.
[9] P. Singhal, T. Goyal, J. Xu, and G. Durrett, “A long way to go: Investigating length
correlations in rlhf,” arXiv preprint arXiv:2310.03716, 2023.
[10] R. Park, R. Rafailov, S. Ermon, and C. Finn, “Disentangling length from quality in
direct preference optimization,” arXiv preprint arXiv:2403.19159, 2024.
[11] N. Muennighoff et al., “Olmoe: Open mixture-of-experts language models,” arXiv
preprint arXiv:2409.02060, 2024.
[12] Allen Institute for Artificial Intelligence, “OLMoE, meet iOS.” [Link]
og/olmoe-app, 2025.
[13] C. Zhou et al., “Lima: Less is more for alignment,” Advances in Neural Information
Processing Systems, vol. 36, pp. 55006–55021, 2023.
[14] R. Taori et al., “Stanford alpaca: An instruction-following LLaMA model,” GitHub
repository. [Link] GitHub, 2023.
[15] W.-L. Chiang et al., “Vicuna: An open-source chatbot impressing GPT-4 with 90%*
ChatGPT quality.” 2023. Available: [Link]
[16] X. Geng et al., “Koala: A dialogue model for academic research.” Blog post, 2023.
Accessed: Apr. 03, 2023. [Online]. Available: [Link]
/03/koala/
[17] M. Conover et al., “Hello dolly: Democratizing the magic of ChatGPT with open
models.” Accessed: June 30, 2023. [Online]. Available: [Link]
/blog/2023/03/24/[Link]
[18] A. Askell et al., “A general language assistant as a laboratory for alignment,” arXiv
preprint arXiv:2112.00861, 2021.
[19] Y. Bai et al., “Constitutional ai: Harmlessness from ai feedback,” arXiv preprint
arXiv:2212.08073, 2022.
[20] R. Rafailov, A. Sharma, E. Mitchell, C. D. Manning, S. Ermon, and C. Finn, “Direct
preference optimization: Your language model is secretly a reward model,” Advances
in Neural Information Processing Systems, vol. 36, 2024.

161
[21] L. Tunstall et al., “Zephyr: Direct distillation of LM alignment,” in First conference on
language modeling, 2024. Available: [Link]
[22] H. Ivison et al., “Camels in a changing climate: Enhancing lm adaptation with tulu
2,” arXiv preprint arXiv:2311.10702, 2023.
[23] G. Cui et al., “Ultrafeedback: Boosting language models with high-quality feedback,”
2023.
[24] A. Dubey et al., “The llama 3 herd of models,” arXiv preprint arXiv:2407.21783,
2024.
[25] B. Adler et al., “Nemotron-4 340B technical report,” arXiv preprint arXiv:2406.11704,
2024.
[26] C. Wirth, R. Akrour, G. Neumann, and J. Fürnkranz, “A survey of preference-based
reinforcement learning methods,” Journal of Machine Learning Research, vol. 18, no.
136, pp. 1–46, 2017.
[27] T. Kaufmann, P. Weng, V. Bengs, and E. Hüllermeier, “A survey of reinforcement
learning from human feedback,” arXiv preprint arXiv:2312.14925, 2023.
[28] S. Casper et al., “Open problems and fundamental limitations of reinforcement learning
from human feedback,” arXiv preprint arXiv:2307.15217, 2023.
[29] W. B. Knox and P. Stone, “Tamer: Training an agent manually via evaluative rein-
forcement,” in 2008 7th IEEE international conference on development and learning,
IEEE, 2008, pp. 292–297.
[30] J. MacGlashan et al., “Interactive learning from policy-dependent human feedback,”
in International conference on machine learning, PMLR, 2017, pp. 2285–2294.
[31] B. Ibarz, J. Leike, T. Pohlen, G. Irving, S. Legg, and D. Amodei, “Reward learning
from human preferences and demonstrations in atari,” Advances in neural information
processing systems, vol. 31, 2018.
[32] G. Warnell, N. Waytowich, V. Lawhern, and P. Stone, “Deep tamer: Interactive agent
shaping in high-dimensional state spaces,” in Proceedings of the AAAI conference on
artificial intelligence, 2018.
[33] J. Leike, D. Krueger, T. Everitt, M. Martic, V. Maini, and S. Legg, “Scal-
able agent alignment via reward modeling: A research direction,” arXiv preprint
arXiv:1811.07871, 2018.
[34] D. M. Ziegler et al., “Fine-tuning language models from human preferences,” arXiv
preprint arXiv:1909.08593, 2019.
[35] J. Wu et al., “Recursively summarizing books with human feedback,” arXiv preprint
arXiv:2109.10862, 2021.
[36] J. Menick et al., “Teaching language models to support answers with verified quotes,”
arXiv preprint arXiv:2203.11147, 2022.
[37] A. Glaese et al., “Improving alignment of dialogue agents via targeted human judge-
ments,” arXiv preprint arXiv:2209.14375, 2022.
[38] L. Gao, J. Schulman, and J. Hilton, “Scaling laws for reward model overoptimization,”
in International conference on machine learning, PMLR, 2023, pp. 10835–10866.
[39] D. Ganguli et al., “Red teaming language models to reduce harms: Methods, scaling
behaviors, and lessons learned,” arXiv preprint arXiv:2209.07858, 2022.
[40] R. Ramamurthy et al., “Is reinforcement learning (not) for natural language processing:
Benchmarks, baselines, and building blocks for natural language policy optimization,”
arXiv preprint arXiv:2210.01241, 2022.

162
[41] A. Havrilla et al., “TrlX: A framework for large scale reinforcement learning from
human feedback,” in Proceedings of the 2023 conference on empirical methods in
natural language processing, Singapore: Association for Computational Linguistics,
Dec. 2023, pp. 8578–8595. doi: 10.18653/v1/[Link]-main.530.
[42] L. von Werra et al., “TRL: Transformer reinforcement learning,” GitHub repository.
[Link] GitHub, 2020.
[43] OpenAI, “ChatGPT: Optimizing language models for dialogue.” [Link]
/blog/chatgpt/, 2022.
[44] H. Touvron et al., “Llama 2: Open foundation and fine-tuned chat models,” arXiv
preprint arXiv:2307.09288, 2023.
[45] H. Lightman et al., “Let’s verify step by step,” arXiv preprint arXiv:2305.20050, 2023.
[46] A. Kumar et al., “Training language models to self-correct via reinforcement learning,”
arXiv preprint arXiv:2409.12917, 2024.
[47] A. Singh et al., “Beyond human data: Scaling self-training for problem-solving with
language models,” arXiv preprint arXiv:2312.06585, 2023.
[48] OpenAI, “Introducing OpenAI o1-preview.” Sept. 2024. Available: [Link]
om/index/introducing-openai-o1-preview/
[49] A. Vaswani et al., “Attention is all you need,” in Neural information processing
systems, 2017. Available: [Link]
[50] D. Bahdanau, K. Cho, and Y. Bengio, “Neural machine translation by jointly learning
to align and translate,” CoRR, vol. abs/1409.0473, 2014, Available: [Link]
[Link]/CorpusID:11212020
[51] G. Hinton, O. Vinyals, and J. Dean, “Distilling the knowledge in a neural network,”
arXiv preprint arXiv:1503.02531, 2015.
[52] G. Team et al., “Gemma 2: Improving open language models at a practical size,”
arXiv preprint arXiv:2408.00118, 2024.
[53] R. Agarwal et al., “On-policy distillation of language models: Learning from self-
generated mistakes,” in The twelfth international conference on learning representa-
tions, 2024.
[54] J. Wei et al., “Chain-of-thought prompting elicits reasoning in large language models,”
Advances in neural information processing systems, vol. 35, pp. 24824–24837, 2022.
[55] R. S. Sutton, “Reinforcement learning: An introduction,” A Bradford Book, 2018.
[56] N. Lambert, L. Castricato, L. von Werra, and A. Havrilla, “Illustrating reinforcement
learning from human feedback (RLHF),” Hugging Face Blog, 2022.
[57] M. Li et al., “Branch-train-merge: Embarrassingly parallel training of expert language
models,” arXiv preprint arXiv:2208.03306, 2022.
[58] T. Cohere et al., “Command a: An enterprise-ready large language model,” arXiv
preprint arXiv:2504.00698, 2025.
[59] T. OLMo et al., “2 OLMo 2 furious,” arXiv preprint arXiv:2501.00656, 2024.
[60] S. Alrashed, “SmolTulu: Higher learning rate to batch size ratios can lead to better
reasoning in SLMs,” arXiv preprint arXiv:2412.08347, 2024.
[61] D. Guo et al., “Deepseek-r1: Incentivizing reasoning capability in llms via reinforce-
ment learning,” arXiv preprint arXiv:2501.12948, 2025.
[62] A. Yang et al., “Qwen3 technical report,” arXiv preprint arXiv:2505.09388, 2025.
[63] B. Xia et al., “MiMo: Unlocking the reasoning potential of language model–from
pretraining to posttraining,” arXiv preprint arXiv:2505.07608, 2025.

163
[64] B. Seed et al., “Seed-thinking-v1. 5: Advancing superb reasoning models with
reinforcement learning,” arXiv preprint arXiv:2504.13914, 2025.
[65] N. Lambert, T. K. Gilbert, and T. Zick, “Entangled preferences: The history and risks
of reinforcement learning and human feedback,” arXiv preprint arXiv:2310.13595,
2023.
[66] V. Conitzer et al., “Social choice should guide AI alignment in dealing with diverse
human feedback,” arXiv preprint arXiv:2404.10271, 2024.
[67] A. Mishra, “Ai alignment and social choice: Fundamental limitations and policy
implications,” arXiv preprint arXiv:2310.16048, 2023.
[68] H. R. Kirk et al., “The PRISM alignment project: What participatory, representative
and individualised human feedback reveals about the subjective and multicultural
alignment of large language models,” arXiv preprint arXiv:2404.16019, 2024.
[69] S. Poddar, Y. Wan, H. Ivison, A. Gupta, and N. Jaques, “Personalizing reinforcement
learning from human feedback with variational preference learning,” arXiv preprint
arXiv:2408.10075, 2024.
[70] A. Arnauld, The port-royal logic. 1662.
[71] J. Bentham, An introduction to the principles of morals and legislation. 1823.
[72] F. P. Ramsey, “Truth and probability,” Readings in Formal Epistemology: Sourcebook,
pp. 21–45, 2016.
[73] A. O. Hirschman, “Against parsimony: Three easy ways of complicating some cate-
gories of economic discourse,” Bulletin of the American Academy of arts and Sciences,
vol. 37, no. 8, pp. 11–28, 1984.
[74] G. K. Hadfield and B. R. Weingast, “Microfoundations of the rule of law,” Annual
Review of Political Science, vol. 17, pp. 21–42, 2014.
[75] E. L. Thorndike, “The law of effect,” The American journal of psychology, vol. 39, no.
1/4, pp. 212–222, 1927.
[76] B. F. Skinner, The behavior of organisms: An experimental analysis. BF Skinner
Foundation, 2019.
[77] R. A. Briggs, “Normative theories of rational choice: Expected utility,” 2014.
[78] B. Widrow and M. E. Hoff, “Adaptive switching circuits,” Stanford Univ Ca Stanford
Electronics Labs, 1960.
[79] S. Singh, R. L. Lewis, and A. G. Barto, “Where do rewards come from,” in Proceedings
of the annual conference of the cognitive science society, Cognitive Science Society,
2009, pp. 2601–2606.
[80] S. M. McClure, N. D. Daw, and P. R. Montague, “A computational substrate for
incentive salience,” Trends in neurosciences, vol. 26, no. 8, pp. 423–428, 2003.
[81] D. Silver, S. Singh, D. Precup, and R. S. Sutton, “Reward is enough,” Artificial
Intelligence, vol. 299, p. 103535, 2021.
[82] R. Bellman, “A markovian decision process,” Journal of mathematics and mechanics,
pp. 679–684, 1957.
[83] R. A. Howard, “Dynamic programming and markov processes.” 1960.
[84] J. M. Mendel and R. W. McLaren, “8 reinforcement-learning control and pattern
recognition systems,” in Adaptive, learning and pattern recognition systems, vol. 66,
J. M. Mendel and K. S. Fu, Eds., in Mathematics in science and engineering, vol. 66.,
Elsevier, 1970, pp. 287–318. doi: [Link]

164
[85] M. Waltz and K. Fu, “A heuristic approach to reinforcement learning control systems,”
IEEE Transactions on Automatic Control, vol. 10, no. 4, pp. 390–398, 1965, doi:
10.1109/TAC.1965.1098193.
[86] A. H. Klopf, Brain function and adaptive systems: A heterostatic theory. Air Force
Cambridge Research Laboratories, Air Force Systems Command, 1972.
[87] R. S. Sutton, “Learning to predict by the methods of temporal differences,” Machine
learning, vol. 3, pp. 9–44, 1988.
[88] G. Tesauro et al., “Temporal difference learning and TD-gammon,” Communications
of the ACM, vol. 38, no. 3, pp. 58–68, 1995.
[89] C. J. Watkins and P. Dayan, “Q-learning,” Machine learning, vol. 8, pp. 279–292,
1992.
[90] V. Mnih et al., “Playing atari with deep reinforcement learning,” arXiv preprint
arXiv:1312.5602, 2013.
[91] F. Golnaraghi and B. C. Kuo, Automatic control systems. McGraw-Hill Education,
2017.
[92] D. Silver et al., “Mastering the game of go without human knowledge,” Nature, vol.
550, no. 7676, pp. 354–359, 2017.
[93] J. Degrave et al., “Magnetic control of tokamak plasmas through deep reinforcement
learning,” Nature, vol. 602, no. 7897, pp. 414–419, 2022.
[94] E. Kaufmann, L. Bauersfeld, A. Loquercio, M. Müller, V. Koltun, and D. Scaramuzza,
“Champion-level drone racing using deep reinforcement learning,” Nature, vol. 620,
no. 7976, pp. 982–987, 2023, doi: 10.1038/s41586-023-06419-4.
[95] R. Agarwal, M. Schwarzer, P. S. Castro, A. C. Courville, and M. Bellemare, “Deep
reinforcement learning at the edge of the statistical precipice,” Advances in neural
information processing systems, vol. 34, pp. 29304–29320, 2021.
[96] A. Y. Ng, S. Russell, et al., “Algorithms for inverse reinforcement learning.” in
Proceedings of the seventeenth international conference on machine learning, in ICML
’00. 2000, pp. 663--670.
[97] N. Salha, “Aesthetics & art in the early development of human-computer interfaces,”
PhD thesis, Universität Bremen, 2011.
[98] T. K. Gilbert, S. Dean, T. Zick, and N. Lambert, “Choices, risks, and reward
reports: Charting public policy for reinforcement learning systems,” arXiv preprint
arXiv:2202.05716, 2022.
[99] J. Von Neumann and O. Morgenstern, “Theory of games and economic behavior, 2nd
rev,” 1947.
[100] S. Pitis, “Rethinking the discount factor in reinforcement learning: A decision theoretic
approach,” in Proceedings of the AAAI conference on artificial intelligence, 2019, pp.
7949–7956.
[101] S. Pitis, “Consistent aggregation of objectives with diverse time preferences requires
non-markovian rewards,” arXiv preprint arXiv:2310.00435, 2023.
[102] D. Abel et al., “On the expressivity of markov reward,” Advances in Neural Information
Processing Systems, vol. 34, pp. 7799–7812, 2021.
[103] A. Sen, “Behaviour and the concept of preference,” Economica, vol. 40, no. 159, pp.
241–259, 1973.
[104] K. J. Arrow, “A difficulty in the concept of social welfare,” Journal of political economy,
vol. 58, no. 4, pp. 328–346, 1950.

165
[105] E. Maskin and A. Sen, The arrow impossibility theorem. Columbia University Press,
2014.
[106] J. C. Harsanyi, “Rule utilitarianism and decision theory,” Erkenntnis, vol. 11, no. 1,
pp. 25–53, 1977.
[107] D. Hadfield-Menell, S. J. Russell, P. Abbeel, and A. Dragan, “Cooperative inverse
reinforcement learning,” Advances in neural information processing systems, vol. 29,
2016.
[108] A. Fickinger, S. Zhuang, D. Hadfield-Menell, and S. Russell, “Multi-principal assistance
games,” arXiv preprint arXiv:2007.09540, 2020.
[109] N. Soares, B. Fallenstein, S. Armstrong, and E. Yudkowsky, “Corrigibility,” in Work-
shops at the twenty-ninth AAAI conference on artificial intelligence, 2015.
[110] R. Pettigrew, Choosing for changing selves. Oxford University Press, 2019.
[111] Z. Wang et al., “HelpSteer2-preference: Complementing ratings with preferences,”
arXiv preprint arXiv:2410.01257, 2024.
[112] S. Malik et al., “RewardBench 2: Advancing reward model evaluation,” arXiv preprint
arXiv:2506.01937, 2025.
[113] W.-L. Chiang et al., “Chatbot arena: An open platform for evaluating llms by human
preference,” arXiv preprint arXiv:2403.04132, 2024.
[114] R. Likert, “A technique for the measurement of attitudes.” Archives of psychology,
1932.
[115] J. Zhou et al., “Instruction-following evaluation for large language models.” 2023.
Available: [Link]
[116] K. Ethayarajh, W. Xu, N. Muennighoff, D. Jurafsky, and D. Kiela, “Kto: Model
alignment as prospect theoretic optimization,” arXiv preprint arXiv:2402.01306, 2024.
[117] Z. Wu et al., “Fine-grained human feedback gives better rewards for language model
training,” Advances in Neural Information Processing Systems, vol. 36, 2024.
[118] A. Chen et al., “Learning from natural language feedback,” Transactions on Machine
Learning Research, 2024.
[119] A. Kumar, Y. He, A. H. Markosyan, B. Chern, and I. Arrieta-Ibarra, “Detecting
prefix bias in LLM-based reward models,” arXiv preprint arXiv:2505.13487, 2025.
[120] A. Bharadwaj, C. Malaviya, N. Joshi, and M. Yatskar, “Flattery, fluff, and fog:
Diagnosing and mitigating idiosyncratic biases in preference models.” 2025. Available:
[Link]
[121] M. Sharma et al., “Towards understanding sycophancy in language models,” in
The twelfth international conference on learning representations, 2024. Available:
[Link]
[122] Y. Bu, L. Huo, Y. Jing, and Q. Yang, “Beyond excess and deficiency: Adaptive
length bias mitigation in reward models for RLHF,” in Findings of the association
for computational linguistics: NAACL 2025, 2025, pp. 3091–3098.
[123] X. Zhang, W. Xiong, L. Chen, T. Zhou, H. Huang, and T. Zhang, “From lists to
emojis: How format bias affects model alignment,” arXiv preprint arXiv:2409.11704,
2024.
[124] OpenAI, “Introducing the model spec.” May 2024. Available: [Link]
ex/introducing-the-model-spec/

166
[125] R. A. Bradley and M. E. Terry, “Rank analysis of incomplete block designs: I. The
method of paired comparisons,” Biometrika, vol. 39, no. 3/4, pp. 324–345, 1952,
Accessed: Feb. 13, 2023. [Online]. Available: [Link]
[126] B. Zhu et al., “Starling-7b: Improving helpfulness and harmlessness with rlaif,” in
First conference on language modeling, 2024.
[127] A. Liu, Z. Zhao, C. Liao, P. Lu, and L. Xia, “Learning plackett-luce mixtures from
partial preferences,” in Proceedings of the AAAI conference on artificial intelligence,
2019, pp. 4328–4335.
[128] B. Zhu, M. Jordan, and J. Jiao, “Principled reinforcement learning with human
feedback from pairwise or k-wise comparisons,” in International conference on machine
learning, PMLR, 2023, pp. 43037–43067.
[129] K. Cobbe et al., “Training verifiers to solve math word problems,” arXiv preprint
arXiv:2110.14168, 2021.
[130] C. Lyu et al., “Exploring the limit of outcome reward for learning mathematical
reasoning,” arXiv preprint arXiv:2502.06781, 2025.
[131] L. Zheng et al., “Judging llm-as-a-judge with mt-bench and chatbot arena,” Advances
in Neural Information Processing Systems, vol. 36, pp. 46595–46623, 2023.
[132] Y. Dubois, B. Galambosi, P. Liang, and T. B. Hashimoto, “Length-controlled alpacae-
val: A simple way to debias automatic evaluators,” arXiv preprint arXiv:2404.04475,
2024.
[133] T. Li et al., “From crowdsourced data to high-quality benchmarks: Arena-hard and
BenchBuilder pipeline,” arXiv preprint arXiv:2406.11939, 2024.
[134] B. Y. Lin et al., “WILDBENCH: Benchmarking LLMs with challenging tasks from
real users in the wild,” arXiv preprint arXiv:2406.04770, 2024.
[135] D. Mahan et al., “Generative reward models,” 2024, Available: [Link]
[Link]/pdf/Generative_Reward_Models.pdf
[136] L. Zhang, A. Hosseini, H. Bansal, M. Kazemi, A. Kumar, and R. Agarwal, “Generative
verifiers: Reward modeling as next-token prediction,” arXiv preprint arXiv:2408.15240,
2024.
[137] Z. Ankner, M. Paul, B. Cui, J. D. Chang, and P. Ammanabrolu, “Critique-out-loud
reward models,” arXiv preprint arXiv:2408.11791, 2024.
[138] S. Kim et al., “Prometheus: Inducing fine-grained evaluation capability in language
models,” in The twelfth international conference on learning representations, 2023.
[139] N. Lambert et al., “Rewardbench: Evaluating reward models for language modeling,”
arXiv preprint arXiv:2403.13787, 2024.
[140] X. Wen et al., “Rethinking reward model evaluation: Are we barking up the wrong
tree?” arXiv preprint arXiv:2410.05584, 2024.
[141] E. Zhou et al., “RMB: Comprehensively benchmarking reward models in LLM align-
ment,” arXiv preprint arXiv:2410.09893, 2024.
[142] E. Frick et al., “How to evaluate reward models for RLHF,” arXiv preprint
arXiv:2410.14872, 2024.
[143] Y. Liu, Z. Yao, R. Min, Y. Cao, L. Hou, and J. Li, “RM-bench: Benchmarking reward
models of language models with subtlety and style,” arXiv preprint arXiv:2410.16184,
2024.
[144] S. Gureja et al., “M-RewardBench: Evaluating reward models in multilingual settings,”
arXiv preprint arXiv:2410.15522, 2024.

167
[145] Z. Jin et al., “RAG-RewardBench: Benchmarking reward models in retrieval aug-
mented generation for preference alignment,” arXiv preprint arXiv:2412.13746, 2024.
[146] Z. Wu, M. Yasunaga, A. Cohen, Y. Kim, A. Celikyilmaz, and M. Ghazvininejad,
“reWordBench: Benchmarking and improving the robustness of reward models with
transformed inputs,” arXiv preprint arXiv:2503.11751, 2025.
[147] S. Kim et al., “Evaluating robustness of reward models for mathematical reasoning,”
arXiv preprint arXiv:2410.01729, 2024.
[148] Z. Liu, Y. Chen, M. Shoeybi, B. Catanzaro, and W. Ping, “AceMath: Advancing
frontier math reasoning with post-training and reward modeling.” 2024. Available:
[Link]
[149] M. Song, Z. Su, X. Qu, J. Zhou, and Y. Cheng, “PRMBench: A fine-grained and chal-
lenging benchmark for process-level reward models,” arXiv preprint arXiv:2501.03124,
2025.
[150] C. Zheng et al., “ProcessBench: Identifying process errors in mathematical reasoning.”
2024. Available: [Link]
[151] W. Wang et al., “VisualPRM: An effective process reward model for multimodal
reasoning,” arXiv preprint arXiv:2503.10291, 2025.
[152] H. Tu, W. Feng, H. Chen, H. Liu, X. Tang, and C. Xie, “ViLBench: A suite for
vision-language process reward modeling.” Mar. 2025. Available: [Link]
abs/2503.20271
[153] T. Men, Z. Jin, P. Cao, Y. Chen, K. Liu, and J. Zhao, “Agent-RewardBench: Towards a
unified benchmark for reward modeling across perception, planning, and safety in real-
world multimodal agents,” in Proceedings of the 63rd annual meeting of the association
for computational linguistics (volume 1: Long papers), Vienna, Austria: Association for
Computational Linguistics, July 2025, pp. 17521–17541. doi: 10.18653/v1/[Link]-
long.857.
[154] H. Lin et al., “CUARewardBench: A benchmark for evaluating reward models on
computer-using agent.” 2025. Available: [Link]
[155] Z. Chen et al., “MJ-bench: Is your multimodal reward model really a good judge for
text-to-image generation?” arXiv preprint arXiv:2407.04842, 2024.
[156] M. Yasunaga, L. Zettlemoyer, and M. Ghazvininejad, “Multimodal rewardbench:
Holistic evaluation of reward models for vision language models,” arXiv preprint
arXiv:2502.14191, 2025.
[157] L. Li et al., “VLRewardBench: A challenging benchmark for vision-language generative
reward models,” arXiv preprint arXiv:2411.17451, 2024.
[158] J. Ruan et al., “Vlrmbench: A comprehensive and challenging benchmark for vision-
language reward models,” arXiv preprint arXiv:2503.07478, 2025.
[159] H. Wang, W. Xiong, T. Xie, H. Zhao, and T. Zhang, “Interpretable prefer-
ences via multi-objective reward modeling and mixture-of-experts,” arXiv preprint
arXiv:2406.12845, 2024.
[160] Z. Wang et al., “HelpSteer2: Open-source dataset for training top-performing reward
models,” arXiv preprint arXiv:2406.08673, 2024.
[161] J. Park, S. Jwa, M. Ren, D. Kim, and S. Choi, “Offsetbias: Leveraging debiased data
for tuning evaluators,” arXiv preprint arXiv:2407.06551, 2024.

168
[162] N. Jaques, S. Gu, D. Bahdanau, J. M. Hernández-Lobato, R. E. Turner, and
D. Eck, “Sequence tutor: Conservative fine-tuning of sequence generation mod-
els with kl-control,” in International conference on machine learning, PMLR, 2017,
pp. 1645–1654.
[163] N. Jaques et al., “Human-centric dialog training via offline reinforcement learning,”
arXiv preprint arXiv:2010.05848, 2020.
[164] J. Schulman, “Approximating KL-divergence.” [Link]
2016.
[165] R. Y. Pang, W. Yuan, K. Cho, H. He, S. Sukhbaatar, and J. Weston, “Iterative
reasoning preference optimization,” arXiv preprint arXiv:2404.19733, 2024.
[166] Z. Gao et al., “Rebel: Reinforcement learning via regressing relative rewards,” arXiv
preprint arXiv:2404.16767, 2024.
[167] T. Brown et al., “Language models are few-shot learners,” Advances in neural infor-
mation processing systems, vol. 33, pp. 1877–1901, 2020.
[168] C. Raffel et al., “Exploring the limits of transfer learning with a unified text-to-text
transformer,” Journal of machine learning research, vol. 21, no. 140, pp. 1–67, 2020.
[169] J. Wei et al., “Finetuned language models are zero-shot learners,” in International
conference on learning representations, 2022. Available: [Link]
um?id=gEZrGCozdqR
[170] V. Sanh et al., “Multitask prompted training enables zero-shot task generalization,”
in International conference on learning representations, 2022. Available: https:
//[Link]/forum?id=9Vrb9D0WI4
[171] S. Mishra, D. Khashabi, C. Baral, and H. Hajishirzi, “Cross-task generalization via nat-
ural language crowdsourcing instructions,” in Proceedings of the 60th annual meeting
of the association for computational linguistics (volume 1: Long papers), Association
for Computational Linguistics, May 2022, pp. 3470–3487. doi: 10.18653/v1/[Link]-
long.244.
[172] E. Wallace, K. Xiao, R. Leike, L. Weng, J. Heidecke, and A. Beutel, “The instruc-
tion hierarchy: Training llms to prioritize privileged instructions,” arXiv preprint
arXiv:2404.13208, 2024.
[173] T. Dettmers, A. Pagnoni, A. Holtzman, and L. Zettlemoyer, “Qlora: Efficient finetun-
ing of quantized llms,” Advances in neural information processing systems, vol. 36,
pp. 10088–10115, 2023.
[174] N. Rajani, L. Tunstall, E. Beeching, N. Lambert, A. M. Rush, and T. Wolf, “No
robots,” Hugging Face repository. [Link]
no_robots; Hugging Face, 2023.
[175] W. R. Gilks and P. Wild, “Adaptive rejection sampling for gibbs sampling,” Journal
of the Royal Statistical Society: Series C (Applied Statistics), vol. 41, no. 2, pp.
337–348, 1992.
[176] H. Dong et al., “RAFT: Reward rAnked FineTuning for generative foundation model
alignment.” 2023. Available: [Link]
[177] T. Liu et al., “Statistical rejection sampling improves preference optimization.” 2023.
Available: [Link]
[178] A. Ahmadian et al., “Back to basics: Revisiting reinforce style optimization for
learning from human feedback in llms,” arXiv preprint arXiv:2402.14740, 2024.

169
[179] J. Schulman, P. Moritz, S. Levine, M. Jordan, and P. Abbeel, “High-dimensional
continuous control using generalized advantage estimation,” in Proceedings of the
international conference on learning representations (ICLR), 2016.
[180] R. J. Williams, “Simple statistical gradient-following algorithms for connectionist
reinforcement learning,” Machine learning, vol. 8, pp. 229–256, 1992.
[181] S. C. Huang, A. Ahmadian, and C. F. AI, “Putting RL back in RLHF.” https:
//[Link]/blog/putting_rl_back_in_rlhf_with_rloo, 2024.
[182] W. Kool, H. van Hoof, and M. Welling, “Buy 4 reinforce samples, get a baseline for
free!” 2019.
[183] J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov, “Proximal policy
optimization algorithms,” arXiv preprint arXiv:1707.06347, 2017.
[184] C. Berner et al., “Dota 2 with large scale deep reinforcement learning,” arXiv preprint
arXiv:1912.06680, 2019.
[185] Z. Liu et al., “Understanding R1-zero-like training: A critical perspective,” arXiv
preprint arXiv:2503.20783, Mar. 2025, Available: [Link]
[186] J. Nocedal and S. J. Wright, Numerical optimization. Springer, 2006.
[187] J. Schulman, S. Levine, P. Abbeel, M. Jordan, and P. Moritz, “Trust region policy
optimization,” in International conference on machine learning, PMLR, 2015, pp.
1889–1897.
[188] Z. Shao et al., “Deepseekmath: Pushing the limits of mathematical reasoning in open
language models,” arXiv preprint arXiv:2402.03300, 2024.
[189] A. Liu et al., “Deepseek-v3 technical report,” arXiv preprint arXiv:2412.19437, 2024.
[190] H. Ivison et al., “Unpacking DPO and PPO: Disentangling best practices for learning
from preference feedback,” arXiv preprint arXiv:2406.09279, 2024.
[191] S. Huang, M. Noukhovitch, A. Hosseini, K. Rasul, W. Wang, and L. Tunstall, “The n+
implementation details of RLHF with PPO: A case study on TL;DR summarization,”
in First conference on language modeling, 2024. Available: [Link]
orum?id=kHO2ZTa8e3
[192] L. Weng, “Policy gradient algorithms,” [Link], 2018, Available: https:
//[Link]/posts/2018-04-08-policy-gradient/
[193] Q. Yu et al., “DAPO: An open-source LLM reinforcement learning system at scale.”
2025.
[194] A. Baheti, X. Lu, F. Brahman, R. L. Bras, M. Sap, and M. Riedl, “Leftover lunch:
Advantage-based offline reinforcement learning for language models,” arXiv preprint
arXiv:2305.14718, 2023.
[195] M. Noukhovitch, S. Huang, S. Xhonneux, A. Hosseini, R. Agarwal, and A. Courville,
“Asynchronous RLHF: Faster and more efficient off-policy RL for language models,”
arXiv preprint arXiv:2410.18252, 2024.
[196] B. Wu et al., “LlamaRL: A distributed asynchronous reinforcement learning framework
for efficient large-scale LLM trainin,” arXiv preprint arXiv:2505.24034, 2025.
[197] W. Fu et al., “AReaL: A large-scale asynchronous reinforcement learning system for
language reasoning,” arXiv preprint arXiv:2505.24298, 2025.
[198] P. I. Team et al., “INTELLECT-2: A reasoning model trained through globally
decentralized reinforcement learning.” 2025. Available: [Link]
7291

170
[199] N. L. Roux et al., “Tapered off-policy REINFORCE: Stable and efficient reinforcement
learning for LLMs,” arXiv preprint arXiv:2503.14286, 2025.
[200] D. Seita, “Notes on the generalized advantage estimation paper.” 2017. Available:
[Link]
estimation-paper/
[201] T. Wu, B. Zhu, R. Zhang, Z. Wen, K. Ramchandran, and J. Jiao, “Pairwise proximal
policy optimization: Harnessing relative feedback for llm alignment,” arXiv preprint
arXiv:2310.00212, 2023.
[202] Y. Flet-Berliac et al., “Contrastive policy gradient: Aligning LLMs on sequence-level
scores in a supervised-friendly fashion,” arXiv preprint arXiv:2406.19185, 2024.
[203] Z. Li et al., “Remax: A simple, effective, and efficient reinforcement learning method
for aligning large language models,” in Forty-first international conference on machine
learning, 2023.
[204] T. Gunter et al., “Apple intelligence foundation language models,” arXiv preprint
arXiv:2407.21075, 2024.
[205] K. Team et al., “Kimi k1. 5: Scaling reinforcement learning with llms,” arXiv preprint
arXiv:2501.12599, 2025.
[206] M. Tomar, L. Shani, Y. Efroni, and M. Ghavamzadeh, “Mirror descent policy opti-
mization,” arXiv preprint arXiv:2005.09814, 2020.
[207] Y. Zhang et al., “Improving LLM general preference alignment via optimistic online
mirror descent,” arXiv preprint arXiv:2502.16852, 2025.
[208] Y. Yuan et al., “VAPO: Efficient and reliable reinforcement learning for advanced
reasoning tasks,” arXiv preprint arXiv:2504.05118, 2025.
[209] Y. Yuan, Y. Yue, R. Zhu, T. Fan, and L. Yan, “What’s behind PPO’s collapse in
long-CoT? Value optimization holds the secret,” arXiv preprint arXiv:2503.01491,
2025.
[210] Y. Zhao, R. Joshi, T. Liu, M. Khalman, M. Saleh, and P. J. Liu, “Slic-hf: Sequence
likelihood calibration with human feedback,” arXiv preprint arXiv:2305.10425, 2023.
[211] M. G. Azar et al., “A general theoretical paradigm to understand learning from
human preferences,” in International conference on artificial intelligence and statistics,
PMLR, 2024, pp. 4447–4455.
[212] A. Amini, T. Vieira, and R. Cotterell, “Direct preference optimization with an offset,”
arXiv preprint arXiv:2402.10571, 2024.
[213] J. Hong, N. Lee, and J. Thorne, “Reference-free monolithic preference optimization
with odds ratio,” arXiv e-prints, pp. arXiv–2403, 2024.
[214] Y. Meng, M. Xia, and D. Chen, “Simpo: Simple preference optimization with a
reference-free reward,” Advances in Neural Information Processing Systems, vol. 37,
pp. 124198–124235, 2025.
[215] N. Razin, S. Malladi, A. Bhaskar, D. Chen, S. Arora, and B. Hanin, “Unintentional
unalignment: Likelihood displacement in direct preference optimization,” arXiv
preprint arXiv:2410.08847, 2024.
[216] Y. Ren and D. J. Sutherland, “Learning dynamics of llm finetuning,” arXiv preprint
arXiv:2407.10490, 2024.
[217] T. Xiao, Y. Yuan, H. Zhu, M. Li, and V. G. Honavar, “Cal-dpo: Calibrated direct pref-
erence optimization for language model alignment,” arXiv preprint arXiv:2412.14516,
2024.

171
[218] A. Gupta et al., “AlphaPO–reward shape matters for LLM alignment,” arXiv preprint
arXiv:2501.03884, 2025.
[219] S. Guo et al., “Direct language model alignment from online ai feedback,” arXiv
preprint arXiv:2402.04792, 2024.
[220] P. Singhal, N. Lambert, S. Niekum, T. Goyal, and G. Durrett, “D2po: Discriminator-
guided dpo with response evaluation models,” arXiv preprint arXiv:2405.01511, 2024.
[221] C. Rosset, C.-A. Cheng, A. Mitra, M. Santacroce, A. Awadallah, and T. Xie, “Direct
nash optimization: Teaching language models to self-improve with general preferences,”
arXiv preprint arXiv:2404.03715, 2024.
[222] S. Jung, G. Han, D. W. Nam, and K.-W. On, “Binary classifier optimization for large
language model alignment,” arXiv preprint arXiv:2404.04656, 2024.
[223] H. Zhao et al., “Rainbowpo: A unified framework for combining improvements in
preference optimization,” arXiv preprint arXiv:2410.04203, 2024.
[224] A. Gorbatovski, B. Shaposhnikov, V. Sinii, A. Malakhov, and D. Gavrilov,
“The differences between direct alignment algorithms are a blur,” arXiv preprint
arXiv:2502.01237, 2025.
[225] S. Xu et al., “Is dpo superior to ppo for llm alignment? A comprehensive study,”
arXiv preprint arXiv:2404.10719, 2024.
[226] F. Tajwar et al., “Preference fine-tuning of llms should leverage suboptimal, on-policy
data,” arXiv preprint arXiv:2404.14367, 2024.
[227] H. Lee et al., “Rlaif: Scaling reinforcement learning from human feedback with ai
feedback,” 2023.
[228] A. Sharma, S. Keh, E. Mitchell, C. Finn, K. Arora, and T. Kollar, “A critical
evaluation of AI feedback for aligning large language models.” 2024. Available:
[Link]
[229] L. Castricato, N. Lile, S. Anand, H. Schoelkopf, S. Verma, and S. Biderman, “Sup-
pressing pink elephants with direct principle feedback.” 2024. Available: https:
//[Link]/abs/2402.07896
[230] W. Yuan et al., “Self-rewarding language models.” 2025. Available: [Link]
abs/2401.10020
[231] L. J. V. Miranda et al., “Hybrid preferences: Learning to route instances for human
vs. AI feedback,” pp. 7162–7200, July 2025, doi: 10.18653/v1/[Link]-long.355.
[232] Y. Xu et al., “RLTHF: Targeted human feedback for LLM alignment,” arXiv preprint
arXiv:2502.13417, 2025, Available: [Link]
[233] T. Olmo et al., “Olmo 3.” 2025. Available: [Link]
[234] E. Bakouch et al., “SmolLM3: smol, multilingual, long-context reasoner.” https:
//[Link]/blog/smollm3, 2025.
[235] Z. Wang et al., “Helpsteer: Multi-attribute helpfulness dataset for steerlm,” in
Proceedings of the 2024 conference of the north american chapter of the association
for computational linguistics: Human language technologies (volume 1: Long papers),
2024, pp. 3371–3384.
[236] Z. Wang et al., “HelpSteer3-preference: Open human-annotated preference data across
diverse tasks and languages,” arXiv preprint arXiv:2505.11475, 2025.

172
[237] NVIDIA, “Nemotron 3 nano: Open, efficient mixture-of-experts hybrid mamba-
transformer model for agentic reasoning,” NVIDIA, Technical Report, Dec. 2025.
Available: [Link]
[Link]
[238] B. Wang et al., “Nemotron-cascade: Scaling cascaded reinforcement learning for
general-purpose reasoning models,” arXiv preprint arXiv:2512.13607, 2025.
[239] A. Bercovich, I. Levy, I. Golan, et al., “Llama-nemotron: Efficient reasoning models,”
arXiv preprint arXiv:2505.00949, 2025.
[240] Z. Sun et al., “SALMON: Self-alignment with principle-following reward models,”
in The twelfth international conference on learning representations, 2024. Available:
[Link]
[241] P. Wang et al., “Large language models are not fair evaluators,” arXiv preprint
arXiv:2305.17926, 2023.
[242] A. Panickssery, S. Bowman, and S. Feng, “Llm evaluators recognize and favor their
own generations,” Advances in Neural Information Processing Systems, 2024.
[243] T. Wang et al., “Shepherd: A critic for language model generation,” arXiv preprint
arXiv:2308.04592, 2023.
[244] P. Ke et al., “CritiqueLLM: Towards an informative critique generation model for
evaluation of large language model generation,” arXiv preprint arXiv:2311.18702,
2023.
[245] J. Li, S. Sun, W. Yuan, R.-Z. Fan, H. Zhao, and P. Liu, “Generative judge for
evaluating alignment,” arXiv preprint arXiv:2310.05470, 2023.
[246] S. Kim et al., “Prometheus 2: An open source language model specialized in evaluating
other language models,” arXiv preprint arXiv:2405.01535, 2024.
[247] S. Lee, S. Kim, S. Park, G. Kim, and M. Seo, “Prometheus-vision: Vision-language
model as a judge for fine-grained evaluation,” in Findings of the association for
computational linguistics ACL 2024, 2024, pp. 11286–11315.
[248] B. Brown et al., “Large language monkeys: Scaling inference compute with repeated
sampling,” arXiv preprint arXiv:2407.21787, 2024.
[249] E. Zhao, P. Awasthi, and S. Gollapudi, “Sample, scrutinize and scale: Effective
inference-time search by scaling verification,” arXiv preprint arXiv:2502.01839, 2025.
[250] N. Kalra and L. Tang, “Verdict: A library for scaling judge-time compute,” arXiv
preprint arXiv:2502.18018, 2025.
[251] A. Madaan et al., “Self-refine: Iterative refinement with self-feedback,” Advances in
Neural Information Processing Systems, 2023.
[252] A. Pace, J. Mallinson, E. Malmi, S. Krause, and A. Severyn, “West-of-n: Synthetic pref-
erence generation for improved reward modeling,” arXiv preprint arXiv:2401.12086,
2024.
[253] T. Wu et al., “Meta-rewarding language models: Self-improving alignment with
llm-as-a-meta-judge,” arXiv preprint arXiv:2407.19594, 2024.
[254] M. Y. Guan et al., “Deliberative alignment: Reasoning enables safer language models,”
arXiv preprint arXiv:2412.16339, 2024.
[255] Anthropic, “Claude’s constitution.” Accessed: Feb. 07, 2024. [Online]. Available:
[Link]
[256] D. Ganguli et al., “Collective constitutional AI: Aligning a language model with public
input.” Anthropic, 2023.

173
[257] S. Huang et al., “Constitutional AI recipe,” Hugging Face Blog, 2024.
[258] N. Lambert, H. Schoelkopf, A. Gokaslan, L. Soldaini, V. Pyatkin, and L. Castricato,
“Self-directed synthetic dialogues and revisions technical report,” arXiv preprint
arXiv:2407.18421, 2024.
[259] Z. Sun et al., “Principle-driven self-alignment of language models from scratch with
minimal human supervision,” in Thirty-seventh conference on neural information
processing systems, 2023. Available: [Link]
[260] Z. Liu et al., “Inference-time scaling for generalist reward modeling,” arXiv preprint
arXiv:2504.02495, 2025.
[261] J.-P. Fränken, E. Zelikman, R. Rafailov, K. Gandhi, T. Gerstenberg, and N. Goodman,
“Self-supervised alignment with mutual information: Learning to follow principles
without preference labels,” Advances in Neural Information Processing Systems, 2024.
[262] A. Irpan, “Deep reinforcement learning doesn’t work yet.” 2018. Available: https:
//[Link]/2018/02/14/[Link]
[263] P. Henderson, R. Islam, P. Bachman, J. Pineau, D. Precup, and D. Meger, “Deep
reinforcement learning that matters,” in Proceedings of the AAAI conference on
artificial intelligence, 2018. Available: [Link]
view/11694
[264] A. Mirhoseini et al., “Chip placement with deep reinforcement learning,” arXiv
preprint arXiv:2004.10746, 2020.
[265] J. Schrittwieser et al., “Mastering atari, go, chess and shogi by planning with a learned
model,” Nature, vol. 588, no. 7839, pp. 604–609, 2020.
[266] M. Cusumano-Towner et al., “Robust autonomy emerges from self-play,” arXiv
preprint arXiv:2502.03349, 2025.
[267] G. Sheng et al., “HybridFlow: A flexible and efficient RLHF framework,” arXiv
preprint arXiv: 2409.19256, 2024.
[268] J. Hu et al., “OpenRLHF: An easy-to-use, scalable and high-performance RLHF
framework,” arXiv preprint arXiv:2405.11143, 2024.
[269] J. Liu, A. Cohen, R. Pasunuru, Y. Choi, H. Hajishirzi, and A. Celikyilmaz, “Don’t
throw away your value model! Generating more preferable text with value-guided
monte-carlo tree search decoding,” arXiv preprint arXiv:2309.15028, 2023.
[270] N. Muennighoff et al., “s1: Simple test-time scaling,” arXiv preprint arXiv:2501.19393,
2025.
[271] L. Chen et al., “Are more llm calls all you need? Towards scaling laws of compound
inference systems,” arXiv preprint arXiv:2403.02419, 2024.
[272] E. Zelikman, Y. Wu, J. Mu, and N. Goodman, “STaR: Bootstrapping reasoning
with reasoning,” in Advances in neural information processing systems, A. H. Oh, A.
Agarwal, D. Belgrave, and K. Cho, Eds., 2022. Available: [Link]
um?id=_3ELRdg2sgI
[273] E. Zelikman, G. Harik, Y. Shao, V. Jayasiri, N. Haber, and N. D. Goodman, “Quiet-
STaR: Language models can teach themselves to think before speaking,” COLM, vol.
abs/2403.09629, 2024.
[274] M. D. Hoffman et al., “Training chain-of-thought via latent-variable inference,” in
Thirty-seventh conference on neural information processing systems, 2023. Available:
[Link]
[275] A. Kazemnejad et al., “VinePPO: Unlocking RL potential for LLM reasoning through
refined credit assignment.” 2024. Available: [Link]

174
[276] J. Gehring, K. Zheng, J. Copet, V. Mella, T. Cohen, and G. Synnaeve, “RLEF:
Grounding code LLMs in execution feedback with reinforcement learning.” 2024.
Available: [Link]
[277] S. Xu et al., “Is DPO superior to PPO for LLM alignment? A comprehensive study,”
in ICML, 2024. Available: [Link]
[278] N. Amit, S. Goldwasser, O. Paradise, and G. Rothblum, “Models that prove their
own correctness,” arXiv preprint arXiv:2405.15722, 2024.
[279] J. Hu, Y. Zhang, Q. Han, D. Jiang, X. Zhang, and H. Shum, “Open-reasoner-zero:
An open source approach to scaling up reinforcement learning on the base model,”
arXiv preprint arXiv:2503.24290, 2025.
[280] M. Abdin, S. Agarwal, A. Awadallah, et al., “Phi-4-reasoning technical report,” arXiv
preprint arXiv:2504.21318, 2025.
[281] A. Liu, B. Zhou, C. Xu, et al., “Hunyuan-TurboS: Advancing large language models
through mamba-transformer synergy and adaptive chain-of-thought,” arXiv preprint
arXiv:2505.15431, 2025.
[282] J. He, J. Liu, C. Y. Liu, et al., “Skywork open reasoner 1 technical report,” arXiv
preprint arXiv:2505.22312, 2025.
[283] C. Team et al., “MiMo-VL technical report.” 2025. Available: [Link]
2506.03569
[284] E. Guha, R. Marten, S. Keh, et al., “OpenThoughts: Data recipes for reasoning
models,” arXiv preprint arXiv:2506.04178, 2025.
[285] Mistral AI, “Magistral: Scaling reinforcement learning for reasoning in large language
models,” Mistral AI, 2025. Available: [Link]
[286] MiniMax, “MiniMax-M1: Scaling test-time compute efficiently with lightning atten-
tion.” 2025. doi: 10.48550/arXiv.2506.13585.
[287] K. Team et al., “Kimi K2: Open agentic intelligence.” 2025. Available: https:
//[Link]/abs/2507.20534
[288] A. Zeng et al., “GLM-4.5: Agentic, reasoning, and coding (ARC) foundation models.”
2025. doi: 10.48550/arXiv.2508.06471.
[289] NVIDIA, “NVIDIA nemotron nano 2: An accurate and efficient hybrid mamba-
transformer reasoning model.” 2025. Available: [Link]
[290] Z. Cheng et al., “K2-think: A parameter-efficient reasoning system.” 2025. Available:
[Link]
[291] M. L. Team, “Introducing LongCat-flash-thinking: A technical report.” 2025. Available:
[Link]
[292] L. Team et al., “Every step evolves: Scaling reinforcement learning for trillion-scale
thinking model.” 2025. Available: [Link]
[293] DeepSeek-AI, “DeepSeek-V3.2: Pushing the frontier of open large language models.”
2025. Available: [Link]
[294] Z. Liu et al., “K2-V2: A 360-open, reasoning-enhanced LLM,” arXiv preprint
arXiv:2512.06201, 2025.
[295] L.-C. Xiaomi, “MiMo-V2-flash technical report.” 2025. Available: [Link]
/XiaomiMiMo/MiMo-V2-Flash/blob/main/[Link]
[296] Z. Wang et al., “RAGEN: Understanding self-evolution in LLM agents via multi-turn
reinforcement learning.” 2025. Available: [Link]

175
[297] R. Shao et al., “Spurious rewards: Rethinking training signals in RLVR.” https:
//[Link]/Spurious-Rewards-Rethinking-Training-Signals-in-
RLVR-1f4df34dac1880948858f95aeb88872f, 2025.
[298] Anthropic, “Claude 4.” May 2025. Available: [Link]
ude-4
[299] P. Aggarwal and S. Welleck, “L1: Controlling how long a reasoning model thinks with
reinforcement learning,” arXiv preprint arXiv:2503.04697, 2025.
[300] D. Khatri et al., “The art of scaling reinforcement learning compute for llms,” arXiv
preprint arXiv:2510.13786, 2025.
[301] S. Reed and N. De Freitas, “Neural programmer-interpreters,” arXiv preprint
arXiv:1511.06279, 2015.
[302] P. Lewis et al., “Retrieval-augmented generation for knowledge-intensive nlp tasks,”
Advances in neural information processing systems, vol. 33, pp. 9459–9474, 2020.
[303] L. Gao et al., “Pal: Program-aided language models,” in International conference on
machine learning, PMLR, 2023, pp. 10764–10799.
[304] A. Parisi, Y. Zhao, and N. Fiedel, “Talm: Tool augmented language models,” arXiv
preprint arXiv:2205.12255, 2022.
[305] T. Schick et al., “Toolformer: Language models can teach themselves to use tools.”
2023. Available: [Link]
[306] S. G. Patil, T. Zhang, X. Wang, and J. E. Gonzalez, “Gorilla: Large language model
connected with massive APIs,” arXiv preprint arXiv:2305.15334, 2023.
[307] Anthropic, “Model context protocol (MCP).” [Link] 2024.
[308] A. M. Bran, S. Cox, O. Schilter, C. Baldassari, A. D. White, and P. Schwaller,
“Chemcrow: Augmenting large-language models with chemistry tools,” arXiv preprint
arXiv:2304.05376, 2023.
[309] B. Li et al., “Mmedagent: Learning to use medical tools with multi-modal agent,”
arXiv preprint arXiv:2407.02483, 2024.
[310] K. Zhang, J. Li, G. Li, X. Shi, and Z. Jin, “Codeagent: Enhancing code generation
with tool-integrated agent systems for real-world repo-level coding challenges,” arXiv
preprint arXiv:2401.07339, 2024.
[311] S. Yao, N. Shinn, P. Razavi, and K. Narasimhan, “τ -bench: A bench-
mark for tool-agent-user interaction in real-world domains.” June 2024. doi:
10.48550/arXiv.2406.12045.
[312] Y. Qin et al., “ToolLLM: Facilitating large language models to master 16000+ real-
world APIs.” July 2023. doi: 10.48550/arXiv.2307.16789.
[313] S. Yao et al., “React: Synergizing reasoning and acting in language models,” in
International conference on learning representations (ICLR), 2023.
[314] W. Kwon et al., “Efficient memory management for large language model serving with
PagedAttention,” in Proceedings of the ACM SIGOPS 29th symposium on operating
systems principles, 2023.
[315] I. Shumailov, Z. Shumaylov, Y. Zhao, N. Papernot, R. Anderson, and Y. Gal, “AI
models collapse when trained on recursively generated data,” Nature, vol. 631, no.
8022, pp. 755–759, 2024.
[316] M. Gerstgrasser et al., “Is model collapse inevitable? Breaking the curse of recursion
by accumulating real and synthetic data,” arXiv preprint arXiv:2404.01413, 2024.

176
[317] Y. Feng, E. Dohmatob, P. Yang, F. Charton, and J. Kempe, “Beyond model collapse:
Scaling up with synthesized data requires reinforcement,” in ICML 2024 workshop on
theoretical foundations of foundation models, 2024.
[318] Y. Wang et al., “Self-instruct: Aligning language models with self-generated instruc-
tions,” arXiv preprint arXiv:2212.10560, 2022.
[319] E. Beeching et al., “NuminaMath 7B TIR,” Hugging Face repository. [Link]
[Link]/AI-MO/NuminaMath-7B-TIR; Numina & Hugging Face, 2024.
[320] M. Li et al., “Superfiltering: Weak-to-strong data filtering for fast instruction-tuning,”
arXiv preprint arXiv:2402.00530, 2024.
[321] K. Shridhar, A. Stolfo, and M. Sachan, “Distilling reasoning capabilities into smaller
language models,” Findings of the Association for Computational Linguistics: ACL
2023, pp. 7059–7073, 2023.
[322] C.-Y. Hsieh et al., “Distilling step-by-step! Outperforming larger language models
with less training data and smaller model sizes,” arXiv preprint arXiv:2305.02301,
2023.
[323] D. Hendrycks et al., “Measuring massive multitask language understanding,” arXiv
preprint arXiv:2009.03300, 2020.
[324] A. Mallen, A. Asai, V. Zhong, R. Das, H. Hajishirzi, and D. Khashabi, “When not to
trust language models: Investigating effectiveness and limitations of parametric and
non-parametric memories,” arXiv preprint, 2022.
[325] S. Lin, J. Hilton, and O. Evans, “Truthfulqa: Measuring how models mimic human
falsehoods,” arXiv preprint arXiv:2109.07958, 2021.
[326] M. Suzgun et al., “Challenging BIG-bench tasks and whether chain-of-thought can
solve them,” arXiv preprint arXiv:2210.09261, 2022.
[327] D. Dua, Y. Wang, P. Dasigi, G. Stanovsky, S. Singh, and M. Gardner, “DROP: A
reading comprehension benchmark requiring discrete reasoning over paragraphs,”
arXiv preprint arXiv:1903.00161, 2019.
[328] D. Hendrycks et al., “Measuring mathematical problem solving with the MATH
dataset,” NeurIPS, 2021.
[329] M. Chen et al., “Evaluating large language models trained on code,” 2021, Available:
[Link]
[330] J. Liu, C. S. Xia, Y. Wang, and L. Zhang, “Is your code generated by chatGPT
really correct? Rigorous evaluation of large language models for code generation,” in
Thirty-seventh conference on neural information processing systems, 2023. Available:
[Link]
[331] D. Rein et al., “GPQA: A graduate-level google-proof q&a benchmark,” arXiv preprint
arXiv:2311.12022, 2023.
[332] L. Phan, A. Gatti, Z. Han, N. Li, and H. et al. Zhang, “Humanity’s last exam,” arXiv
preprint arXiv:2501.14249, 2025.
[333] R. Aleithan, H. Xue, M. M. Mohajer, E. Nnorom, G. Uddin, and S. Wang, “SWE-
Bench+: Enhanced coding benchmark for LLMs,” arXiv preprint arXiv:2410.06992,
2024.
[334] N. Jain et al., “LiveCodeBench: Holistic and contamination-free evaluation of large
language models for code,” arXiv preprint arXiv:2403.07974, 2024.
[335] S. AI, “SEAL LLM leaderboards: Expert-driven private evaluations.” 2024. Available:
[Link]

177
[336] S. Schulhoff et al., “The prompt report: A systematic survey of prompting techniques,”
arXiv preprint arXiv:2406.06608, 2024.
[337] J. Robinson, C. M. Rytting, and D. Wingate, “Leveraging large language models
for multiple choice question answering,” in International conference on learning
representations, 2023. Available: [Link]
[338] J. Wei et al., “Finetuned language models are zero-shot learners,” in International
conference on learning representations, 2022.
[339] V. Sanh et al., “Multitask prompted training enables zero-shot task generalization,”
in International conference on learning representations, 2022.
[340] T. Kojima, S. S. Gu, M. Reid, Y. Matsuo, and Y. Iwasawa, “Large language models
are zero-shot reasoners,” Advances in neural information processing systems, vol. 35,
pp. 22199–22213, 2022.
[341] J. Achiam et al., “Gpt-4 technical report,” arXiv preprint arXiv:2303.08774, 2023.
[342] OpenAI, “Introducing SWE-bench verified.” Aug. 2024. Available: [Link]
om/index/introducing-swe-bench-verified/
[343] J. Li et al., “Numinamath: The largest public dataset in ai4maths with 860k pairs of
competition math problems and solutions,” Hugging Face repository, vol. 13, p. 9,
2024.
[344] L. Yu et al., “Metamath: Bootstrap your own mathematical questions for large
language models,” arXiv preprint arXiv:2309.12284, 2023.
[345] A. K. Singh et al., “Evaluation data contamination in LLMs: How do we measure it
and (when) does it matter?” arXiv preprint arXiv:2411.03923, 2024.
[346] K. Huang et al., “MATH-perturb: Benchmarking LLMs’ math reasoning abilities
against hard perturbations,” arXiv preprint arXiv:2502.06453, 2025.
[347] UK AI Safety Institute, “Inspect AI: Framework for Large Language Model Evalua-
tions.” [Link] 2024.
[348] C. Fourrier, N. Habib, H. Kydlicek, T. Wolf, and L. Tunstall, “LightEval: A lightweight
framework for LLM evaluation.” [Link] 2023.
[349] C. Fourrier, N. Habib, A. Lozovskaya, K. Szafer, and T. Wolf, “Open LLM leaderboard
v2.” [Link]
Hugging Face, 2024.
[350] L. Gao et al., “A Framework for Few-Shot Language Model Evaluation.” Zenodo,
2023. doi: 10.5281/zenodo.10256836.
[351] S. Black et al., “GPT-NeoX-20B: An open-source autoregressive language model,”
in Proceedings of the ACL workshop on challenges & perspectives in creating large
language models, 2022. Available: [Link]
[352] Y. Gu, O. Tafjord, B. Kuehl, D. Haddad, J. Dodge, and H. Hajishirzi, “OLMES: A
Standard for Language Model Evaluations,” arXiv preprint arXiv:2406.08446, 2024.
[353] P. Liang et al., “Holistic Evaluation of Language Models,” Transactions on Machine
Learning Research, 2023, doi: 10.1111/nyas.15007.
[354] MosaicML, “Mosaic Eval Gauntlet v0.3.0 — Evaluation Suite.” [Link]
osaicml/llm-foundry/blob/main/scripts/eval/local_data/EVAL_GAUNTLET.md,
2024.
[355] J. Schulman, “Proxy objectives in reinforcement learning from human feedback.”
Invited talk at the International Conference on Machine Learning (ICML), 2023.
Available: [Link]

178
[356] C. Zhang, O. Vinyals, R. Munos, and S. Bengio, “A study on overfitting in deep
reinforcement learning,” arXiv preprint arXiv:1804.06893, 2018.
[357] C. A. Goodhart and C. Goodhart, Problems of monetary management: The UK
experience. Springer, 1984.
[358] K. Hoskin, “The ‘awful idea of accountability’: Inscribing people into the measurement
of objects,” Accountability: Power, ethos and the technologies of managing, vol. 265,
1996.
[359] T. Lu and C. Boutilier, “Learning mallows models with pairwise preferences,” in
Proceedings of the 28th international conference on machine learning (icml-11), 2011,
pp. 145–152.
[360] S. Han et al., “Wildguard: Open one-stop moderation tools for safety risks, jailbreaks,
and refusals of llms,” arXiv preprint arXiv:2406.18495, 2024.
[361] H. Inan et al., “Llama guard: Llm-based input-output safeguard for human-ai conver-
sations,” arXiv preprint arXiv:2312.06674, 2023.
[362] P. Röttger, H. R. Kirk, B. Vidgen, G. Attanasio, F. Bianchi, and D. Hovy, “Xstest:
A test suite for identifying exaggerated safety behaviours in large language models,”
arXiv preprint arXiv:2308.01263, 2023.
[363] T. Coste, U. Anwar, R. Kirk, and D. Krueger, “Reward model ensembles help mitigate
overoptimization,” arXiv preprint arXiv:2310.02743, 2023.
[364] T. Moskovitz et al., “Confronting reward model overoptimization with constrained
RLHF,” arXiv preprint arXiv:2310.04373, 2023.
[365] R. Rafailov et al., “Scaling laws for reward model overoptimization in direct align-
ment algorithms,” Advances in Neural Information Processing Systems, vol. 37, pp.
126207–126242, 2024.
[366] S. Zhuang and D. Hadfield-Menell, “Consequences of misaligned AI,” Advances in
Neural Information Processing Systems, vol. 33, pp. 15763–15773, 2020.
[367] J. Bai et al., “Qwen technical report,” arXiv preprint arXiv:2309.16609, 2023.
[368] G. Wang, S. Cheng, X. Zhan, X. Li, S. Song, and Y. Liu, “Openchat: Advancing open-
source language models with mixed-quality data,” arXiv preprint arXiv:2309.11235,
2023.
[369] S. Maiya, H. Bartsch, N. Lambert, and E. Hubinger, “Open character training:
Shaping the persona of AI assistants through constitutional AI,” arXiv preprint
arXiv:2511.01689, 2025.
[370] Anthropic, “Claude’s character.” 2024. Available: [Link]
ch/claude-character

179