Recap
● Why is DeepSeek V3 good and cheap?
○ Mixture of Experts Architecture
○ Multihead Latent Attention
○ Multi-Token Prediction
○ Mixed Precision Training
○ Blockwise Quantization
○ Tensor and CUDA core optimizations
○ Etc
● Modern LLM Training
○ Pre-Training
○ Post-Training
■ SFT
■ RLHF
● PPO
● DPO
● GRPO
● RL Basics
DeepSeek’s Evolution
● We have covered DeepSeek
V3
● What about the best one yet,
DeepSeek R1?
DeepSeek Papers
● DeepSeek-R1: Incentivizing Reasoning
Capability in LLMs via Reinforcement
Learning
● DeepSeekMath: Pushing the Limits of
Mathematical Reasoning in Open
Language Models
● DeepSeek-V3 Technical Report
Modern LLM Post-Training
● Places where we need human interaction: (1) SFT (2) Reward Model Training
● Can we get rid of those, since dependency on human is
○ Bottleneck for scaling
○ Could have bias
DeepSeek R1-Zero - Reasoning process within <think></think>
- Answer within <answer></answer>
The answer is YES!
Verifiable task (math, coding etc) prompt.
User Prompt
For example LeetCode problems
Specs
● Pre-Training base model is DeepSeek-V3-Base
● No Supervised Fine Tuning DeepSeek-V3-Base
● No reward model training, instead do rule based
rewards:
○ Accuracy rewards: whether answer is Accuracy reward check
correct - For Math problems with deterministic results,
○ Format rewards: give additional reward to is final answer correct and in specified format?
- For LeetCode problems, does it compile?
<think> Does it pass all test cases?
RL Training RL: Iteratively
● Question 1: How to know whether LLM is giving Complete Update Model
right answer?
○ Answer: focus on tasks that’s verifiable Format reward check
(math, coding etc) - Is <think> used for reasoning?
● Question 2: Why not a neural model for reward - Is <answer> used for answer
model?
○ Answer: Neural reward model suffer from
reward hacking in large scale RL training,
DeepSeek-R1-Zero
and it needs additional training resources
(computation, memory), complex whole
training pipeline.
DeepSeek R1-Zero
Can we do better here too?
GRPO vs PPO Simplified
● PPO requires a trained value model (V(s)) to estimate the “average reward” given a state, later used
to compute advantage: Advantage = Reward - V(s)
● GRPO used group sampling to estimate the “average” reward. In practice the sampling pool size is
~16
GRPO vs PPO Deep Dive
● Both PPO and GRPO are Policy Optimization, meaning we are optimizing the policy directly. Policy is
the operation strategy of the LLM, or the LLM itself.
● Goal: maximize expected total reward
● PPO (can work with or without KL regularization, this one is without)
● GRPO
GRPO vs PPO Deep Dive
● PPO
● Policy updates via
● TLDR: for each RL training iteration, we want the set of parameters that can maximize J, capped by a
ceiling ε (new model can’t be too different than previous). Advantage here is calculated with the
trained value/critic model: Advantage = Reward - V(s)
GRPO vs PPO Deep Dive
● GRPO is very similar with PPO, with the following differences:
○ Eliminates the need for a value network for less memory/compute
○ Uses group sampling for more stable advantage estimation
● Advantage calculation is using Z-Score of a group, similar to Batch/Layer Norm in Transformer!
● Again similar to PPO, Policy updates via
○ Still capped by ceiling ε
DeepSeek R1-Zero
● Results show that pure RL trained R1-Zero shows structure
reasoning abilities
● SFT and RLHF have been the dominant training methods for
LLM. The underlying assumption is human-labeled data is
necessary for structured reasoning, RL without human
preference modeling can’t achieve the same
● DeepSeek R1-Zero proves this assumption wrong!
● Drawbacks of R1-Zero:
○ Bad readability: long and unstructured responses
○ Language mixing: responses contained multiple
languages in a single answer
○ Bad formatting: no markdown, no clear separation
between thoughts and conclusions
● DeepSeek R1 created with motivation to perfect R1-Zero
DeepSeek R1 Stage #1: Cold Start
● Goal: Push model towards better reasoning and more
DeepSeek-V3-Base
readable output, avoiding the instability of pure RL from
scratch.
● Method: Minimal supervised fine tuning on
DeepSeek-V3-Base
A small dataset (thousands of
SFT examples):
● A small dataset (thousands of examples) of high-quality, - High-quality,
long Chain-of-Thought (CoT) data is curated. This data is - Long Chain-of-Thought
- Human refined
generated through a mix of few-shot prompting, asking
models to generate detailed answers with reflection, and
refining outputs (by human annotators) from
DeepSeek-R1-Zero.
DeepSeek-R1-One
● Advantages (compared with R1-Zero):
○ Better Readability
○ Better Performance
DeepSeek R1 Stage #2: RL with GRPO
- Reasoning process within <think></think>
- Answer within <answer></answer>
● Goal: Verifiable task (math, coding etc) prompt.
○ Enhance model’s reasoning User For example LeetCode problems
Prompt
capabilities
○ Improve language consistency DeepSeek-R1-One
● Method: RL with GRPO on
Accuracy reward
DeepSeek-R1-One - For Math problems with deterministic results,
is final answer correct and in specified format?
- For LeetCode problems, does it compile?
● Similar with training DeepSeek-R1-Zero. Does it pass all test cases?
RL Training RL: Iteratively
Still ran into language mixing especially in Complete Update Model
CoT, introduced language consistency
Format reward
reward in RL. - Is <think> used for reasoning?
- Is <answer> used for answer
Language consistency:
- Is language in CoT consistent?
DeepSeek-R1-Two
DeepSeek R1 Stage #3: Rejection Sampling
● Rejection sampling is a method used to generate samples from a complex probability distribution by
drawing samples from a simpler, "proposal" distribution and then accepting or rejecting them based
on a probability related to the ratio of the target and proposal densities.
● Steps (assume target distribution is f(x))
1. Choose a proposal distribution (g(x))
2. Sample from the proposal distribution (g(x))
3. Calculate a acceptance probability
4. Accept or reject the sample
5. Repeat until samples are enough
● Steps in LLM context:
1. Generate Candidate Outputs: Use the LLM to generate a batch of K candidate responses for a
given prompt.
2. Evaluate with a Reward Model: Feed each candidate response to reward model. The reward
model outputs a scalar value representing the quality of the response.
3. Select the Best: Select the candidate with the highest reward score.
4. Fine-tune the LLM: Use the selected candidate to fine-tune the LLM.
5. Repeat
DeepSeek R1 Stage #3: Rejection Sampling
● Goal: Create a high-quality SFT dataset beyond verifiable reasoning, including writing,
role-playing, and other general-purpose tasks
● Method: Rejection sampling on DeepSeek-R1-Two
● Reasoning data:
○ Expand beyond data that can be verified by rules
○ Expanded data are rated by generative reward model, by feeding ground truth and
predictions into DeepSeek-V3
○ Filter out CoTs with bad readability
● Non Reasoning data:
○ SFT data for DeepSeek-V3 is reused
○ DeepSeek-V3 used to generated non reasoning examples
DeepSeek R1 Stage #3: Rejection Sampling
DeepSeek-R1-Two
DeepSeek-V3
Accuracy reward Generative Reward Model (DeepSeek-V3)
- For Math problems with deterministic results, - Ground truth compare with model prediction
is final answer correct and in specified format?
- For LeetCode problems, does it compile?
Does it pass all test cases?
SFT Generate
Dataset CoT
Format reward
- Is <think> used for reasoning?
- Is <answer> used for answer
200K Non
600K
Reasoning
Reasoning
Samples
Samples
DeepSeek-V3-Base
800K Total
Samples
SFT
DeepSeek-R1-Three
DeepSeek R1 Stage #4: RL for All Scenarios
● Goal: Align model with human preferences DeepSeek-R1-Three
(helpfulness and harmlessness) and further
refining reasoning.
Accuracy reward
- For Math problems with deterministic results,
● Method: RL with GRPO on is final answer correct and in specified format?
DeepSeek-R1-Three - For LeetCode problems, does it compile?
RL Training Does it pass all test cases? RL: Iteratively
Complete Update Model
● Reward model:
○ Rule-based rewards for reasoning tasks, Format reward
similar with R1-Zero and stage#2 - Is <think> used for reasoning?
○ Learned reward models to capture - Is <answer> used for answer
human preferences in complex scenarios.
For helpfulness and harmlessness, Reward Model to capture
focusing on the final summary for - Human preference
- Harmlessness
helpfulness and on the entire response - Helpfulness
for harmlessness
DeepSeek-R1
DeepSeek R1 Distillation
● Goal: Allow more efficient smaller models with same/similar level of reasoning capabilities like
DeepSeek-R1
● How Distillation Works
○ (From previous steps) Reuse 800K high-quality SFT samples
○ These samples are then used to fine-tune smaller models like Llama to replicate the larger
model’s reasoning.
○ No RL is applied to the distilled models, even though incorporating RL could further enhance
performance. This enables smaller models to retain advanced reasoning skills without
the high cost of training from scratch.
● Result: This straightforward distillation method significantly enhances the reasoning abilities of
smaller models