Large Language Diffusion Models

Shen Nie 1 * † Fengqi Zhu 1 * † Zebin You 1 † Xiaolu Zhang 2 ‡ Jingyang Ou 1 Jun Hu 2 ‡ Jun Zhou 2
Yankai Lin 1 ‡ Ji-Rong Wen 1 Chongxuan Li 1 ‡ ¶

Abstract

*H
$5&&

FV
Autoregressive models (ARMs) are widely re-

DWL

QH
*60.
arXiv:2502.09992v1 [[Link]] 14 Feb 2025

HP

UDO
garded as the cornerstone of large language mod- 

7D
DWK
  7UXWKIXO4$

VN
0
els (LLMs). We challenge this notion by intro-  

V
 

ducing LLaDA, a diffusion model trained from 0DWK 

scratch under the pre-training and supervised fine-  
tuning (SFT) paradigm. LLaDA models distri-    00/8
butions through a forward data masking process 
  //D'$%%DVH
and a reverse process, parameterized by a vanilla  //D0$%%DVH
+XPDQ(YDO 
Transformer to predict masked tokens. By op-   //D0$%%DVH
 
timizing a likelihood bound, it provides a prin-   &(YDO
cipled generative approach for probabilistic in- 
&R

0%33
ference. Across extensive benchmarks, LLaDA
GH

&00/8
demonstrates strong scalability, outperforming V H
LQH
our self-constructed ARM baselines. Remark- &K
ably, LLaDA 8B is competitive with strong LLMs
Figure 1. Zero/Few-Shot Benchmarks. We scale LLaDA to an
like LLaMA3 8B in in-context learning and, af- unprecedented size of 8B parameters from scratch, achieving com-
ter SFT, exhibits impressive instruction-following petitive performance with strong LLMs (Dubey et al., 2024).
abilities in case studies such as multi-turn dia-
logue. Moreover, LLaDA addresses the rever-
sal curse, surpassing GPT-4o in a reversal poem distribution pdata (·) by optimizing a model distribution pθ (·)
completion task. Our findings establish diffu- through maximum likelihood estimation, or equivalently KL
sion models as a viable and promising alternative divergence minimization between the two distributions:
to ARMs, challenging the assumption that key
max Epdata (x) log pθ (x) ⇔ min KL(pdata (x)||pθ (x)) . (1)
LLM capabilities discussed above are inherently θ θ
tied to ARMs. Project page and codes: https:
| {z }
Generative modeling principles
//[Link]/LLaDA-demo/.
The predominant approach relies on the autoregressive mod-
eling (ARM)—commonly referred to as the next-token pre-
1. Introduction diction paradigm—to define the model distribution:
What is now proved was once only imagined. L
placeholder,placeholder,placeho—William Blake pθ (x) = pθ (x1 )
Y
pθ (xi | x1 , . . . , xi−1 ), (2)
i=2
Large language models (LLMs) (Zhao et al., 2023) fall en-
| {z }
Autoregressive formulation
tirely within the framework of generative modeling. Specifi-
cally, LLMs aim to capture the true but unknown language where x is a sequence of length L, and xi is the i-th token.
*
Equal contribution † Work done during an internship at Ant This paradigm has proven remarkably effective (Radford,
Group ‡ Project leaders 1 Gaoling School of Artificial Intelli- 2018; Radford et al., 2019; Brown, 2020; OpenAI, 2022)
gence, Renmin University of China; Beijing Key Laboratory and has become the foundation of current LLMs. Despite
of Big Data Management and Analysis Methods 2 Ant Group.
¶
Correspondence to: Chongxuan Li <chongxuanli@[Link]>. its widespread adoption, a fundamental question remains
unanswered: Is the autoregressive paradigm the only viable
Preprint. path to achieving the intelligence exhibited by LLMs?

1
 Large Language Diffusion Models

We argue that the answer is not a simple “yes”. The key Scalability. LLaDA scales effectively up to a computational
insight overlooked previously is: it is the generative mod- budget of 1023 FLOPs, achieving comparable results to self-
eling principles (i.e., Eq. (1)), rather than the autoregres- constructed ARM baselines trained on the same data across
sive formulation (i.e., Eq. (2)) itself, that fundamentally six tasks, e.g., MMLU and GSM8K.
underpin the essential properties of LLMs, as detailed be-
In-Context Learning. Remarkably, LLaDA 8B surpasses
low. However, certain inherent limitations of LLMs can be
LLaMA2 7B (Touvron et al., 2023) on nearly all 15 standard
directly traced to their autoregressive nature.
zero/few-shot learning tasks while performing on par with
In particular, we argue that scalability is primarily a con- LLaMA3 8B (Dubey et al., 2024).
sequence of the interplay between Transformers (Vaswani,
Instruction-Following. LLaDA significantly enhances the
2017), model and data size, and Fisher consistency1 (Fisher,
ability to follow instructions after SFT, as demonstrated in
1922) induced by the generative principles in Eq. (1), rather
case studies such as multi-turn dialogue.
than a unique result of ARM. The success of diffusion trans-
formers (Bao et al., 2023; Peebles & Xie, 2023) on visual Reversal Reasoning. LLaDA effectively breaks the re-
data (Brooks et al., 2024) supports this claim. versal curse (Berglund et al., 2023) with consistent per-
formance across forward and reversal tasks. Notably, it
Furthermore, the capabilities of instruction-following and
outperforms GPT-4o in a reversal poem completion task.
in-context learning (Brown, 2020) appear to be intrinsic
properties of all proper conditional generative models on
structurally consistent linguistic tasks, rather than exclusive 2. Approach
advantages of ARMs. In addition, while ARMs can be
In this section, we introduce the probabilistic formulation2 ,
interpreted as a lossless data compressor (Deletang et al.;
along with the pre-training, supervised fine-tuning, and in-
Huang et al., 2024b), any sufficiently expressive probabilis-
ference procedures for LLaDA, as illustrated in Fig. 2.
tic model can achieve similar capabilities (Shannon, 1948).
Nevertheless, the autoregressive nature of LLMs presents 2.1. Probabilistic Formulation
notable challenges. For example, sequential token-by-token
generation incurs high computational costs, and the left- Unlike ARMs in Eq. (2), LLaDA defines a model distribu-
to-right modeling limits effectiveness in reversal reasoning tion pθ (x0 ) through a forward process and a reverse pro-
tasks (Berglund et al., 2023). These inherent limitations cess (Austin et al., 2021a; Ou et al., 2024). The forward
constrain LLMs in handling longer and more complex tasks. process gradually masks tokens independently in x0 until
the sequence is fully masked at t = 1. For t ∈ (0, 1), the
Motivated by these insights, we introduce LLaDA, a Large sequence xt is partially masked, with each being masked
Language Diffusion with mAsking, to investigate whether with probability t or remaining unmasked with probability
the capabilities exhibited by LLMs can emerge from gener- 1 − t. The reverse process recovers the data distribution by
ative modeling principles beyond ARMs in Eq. (2), thereby iteratively predicting masked tokens as t moves from 1 to 0.
addressing the fundamental question posed earlier. In con-
trast to traditional ARMs, LLaDA leverages a masked diffu- The core of LLaDA is a mask predictor, a parametric model
sion model (MDM) (Austin et al., 2021a; Ou et al., 2024), pθ (·|xt ) that takes xt as input and predicts all masked tokens
which incorporates a discrete random masking process and (denoted M) simultaneously. It is trained using a cross-
trains a mask predictor to approximate its reverse process. entropy loss computed only on the masked tokens:
This design enables LLaDA to construct a model distribu- " L #
1X
tion with bidirectional dependencies and optimize a lower L(θ) ≜ −Et,x0 ,xt 1[xit = M] log pθ (xi0 |xt ) , (3)
bound of its log-likelihood, offering an unexplored and prin- t i=1
cipled alternative to existing LLMs.
where x0 is sampled from the training data, t is sampled
We adopt the standard pipeline of data preparation, pre- uniformly from [0, 1], and xt is sampled from the forward
training, supervised fine-tuning (SFT), and evaluation, scal- process. The indicator function 1[·] ensures that the loss is
ing LLaDA to an unprecedented language diffusion of size computed only for masked tokens.
8B. In particular, LLaDA 8B was pre-trained from scratch
Once trained, we can simulate a reverse process (see Sec. 2.4
on 2.3 trillion tokens using 0.13 million H800 GPU hours,
for details) parameterized by the mask predictor and define
followed by SFT on 4.5 million pairs. Across diverse tasks,
the model distribution pθ (x0 ) as the marginal distribution
including language understanding, math, code, and Chinese,
induced at t = 0. Notably, the loss function in Eq. (3)
LLaDA demonstrates the following contributions:
has been proven to be an upper bound on the negative log-
1
It suggests the ability to recover the true data distribution with 2
Here, we focus on the approach of LLaDA. A rigorous formu-
infinite data, a sufficiently large network and optimal training.
lation of MDM is provided in Appendix A for interested readers.

2
 Large Language Diffusion Models

(a) Mask all tokens independently (b) Prompt Response (c) Prompt Response
𝑡=1
Mask ratio 𝑡 ∼ 𝑈(0,1) ...

An intermediate step
Mask predictor Mask predictor Mask predictor

Remask
Mask token Remask
...
Non-mask token Random mask
𝑡=0

Figure 2. A Conceptual Overview of LLaDA. (a) Pre-training. LLaDA is trained on text with random masks applied independently to all
tokens at the same ratio t ∼ U [0, 1]. (b) SFT. Only response tokens are possibly masked. (c) Sampling. LLaDA simulates a diffusion
process from t = 1 (fully masked) to t = 0 (unmasked), predicting all masks simultaneously at each step with flexible remask strategies.

likelihood of the model distribution (Ou et al., 2024): of any special techniques. The data are derived from online
corpora, with low-quality content filtered through manually
−Epdata (x0 ) [log pθ (x0 )] ≤ L(θ), (4) designed rules and LLM-based approaches. Beyond general
making it a principled objective for generative modeling. text, the dataset encompasses high-quality code, math, and
multilingual data. The mixing of data sources and domains
Notably, LLaDA employs a masking ratio that varies ran- is guided by scaled-down ARMs. The pre-training process
domly between 0 and 1 while masked language models (De- utilizes a fixed sequence length of 4096 tokens, incurring a
vlin, 2018) use a fixed ratio. The subtly differences have total computational cost of 0.13 million H800 GPU hours,
significant implications, especially at scale: as shown in similar to ARMs of the same scale and dataset size.
Eq. (4), LLaDA is a principled generative model with the
potential to perform in-context learning naturally, akin to For a training sequence x0 , we randomly sample t ∈ [0, 1],
LLMs. Moreover, its generative perspective ensures Fisher mask each token independently with the same probability
consistency (Fisher, 1922) in extreme cases, suggesting t to obtain xt (see Fig. 2 (a)) and estimate Eq. (3) via the
strong scalability with large data and models. Monte Carlo method for stochastic gradient descent training.
In addition, following Nie et al. (2024), to enhance the
ability of LLaDA to handle variable-length data, we set 1%
2.2. Pre-training
of the pre-training data to a random length that is uniformly
LLaDA employs a Transformer (Vaswani, 2017) as the mask sampled from the range [1, 4096].
predictor, whose architecture is similar to existing LLMs.
We adopted the Warmup-Stable-Decay (Hu et al., 2024)
However, LLaDA does not use a causal mask, as its formu-
learning rate scheduler to monitor the training progress
lation allows it to see the entire input for predictions.
without interrupting continuous training. Specifically, we
We trained two variants of LLaDA with different sizes: 1 linearly increased the learning rate from 0 to 4 × 10−4 over
billion (B) and 8B. We summarize the model architecture the first 2000 iterations and maintained it at 4 × 10−4 . After
of LLaDA 8B and LLaMA3 8B (Dubey et al., 2024) here processing 1.2T tokens, we decayed the learning rate to
and details are provided in Appendix B.2. We have ensured 1 × 10−4 and held it constant for the next 0.8T tokens to en-
consistency in most hyperparameters while making several sure stable training. Finally, we linearly reduced the learning
necessary modifications. We use vanilla multi-head atten- rate from 1 × 10−4 to 1 × 10−5 for the last 0.3T tokens. Fur-
tion instead of grouped query attention (Ainslie et al., 2023) thermore, we utilized the AdamW optimizer (Loshchilov,
for simplicity, as LLaDA is incompatible with KV caching, 2017) with a weight decay of 0.1, a batch size of 1280, and
resulting in a different number of key and value heads. Con- a local batch size of 4 per GPU. The 8B experiment was
sequently, the attention layer has more parameters, and we executed once, without any hyperparameter tuning.
reduce the FFN dimension to maintain a comparable model
size. Additionally, the vocabulary size differs slightly due 2.3. Supervised Fine-Tuning
to a tokenizer (Brown, 2020) adapted on our data.
We enhance the capability of LLaDA to follow instructions
The LLaDA model is pre-trained on a dataset comprising by supervised fine-tuning (SFT) with paired data (p0 , r0 ),
2.3 trillion (T) tokens, adhering to a data protocol that aligns where p0 is the prompt and r0 denotes the response. This
closely with existing large language models (LLMs) (Tou- is the simplest and most basic post-training method for
vron et al., 2023; Chu et al., 2024), without the incorporation

3
 Large Language Diffusion Models

LLMs. Technically, this requires to model the conditional default. In addition, the generation length is also treated as
distribution pθ (r0 |p0 ) instead of pθ (x0 ) in pre-training. a hyperparameter, specifying the length of the fully masked
sentence at the beginning of the sampling process. As de-
The implementation is similar to pre-training. As shown in
tailed in Appendix B.4, since both pre-training and SFT
Fig. 2 (b), we leave the prompt unchanged and mask the
are conducted using datasets with variable lengths, the final
tokens in the response independently, as done for x0 . Then,
results are insensitive to this length hyperparameter.
we feed both the prompt and the masked response rt to the
pre-trained mask predictor to compute the loss for SFT: At an intermediate step from time t ∈ (0, 1] to s ∈ [0, t), we
  feed both p0 and rt into the mask predictor and predict all
L′
1 X masked tokens simultaneously. Subsequently, we remask st
−Et,p0 ,r0 ,rt  1[ri = M] log pθ (r0i |p0 , rt ) , (5)
t i=1 t of the predicted tokens in expectation to obtain rs , ensuring
that the transition of the reverse process aligns with the
where L′ denotes a dynamic length specified later, and all forward process for accurate sampling (Austin et al., 2021a).
other notations remain the same as before. In principle, the remasking strategy should be purely ran-
Note that this approach is fully compatible with pre-training. dom. However, inspired by the annealing tricks of sampling
Essentially, the concatenation of p0 and r0 can be treated in LLMs (Holtzman et al., 2019; Brown, 2020), we explore
as clean pre-training data x0 , while the concatenation of two deterministic yet effective remasking strategies. Specif-
p0 and rt serves as the masked version xt . The process is ically, similarly to Chang et al. (2022), we remask the st
identical to pre-training, with the only difference being that of predicted tokens with the lowest confidence based on
all masked tokens happen to appear in the r0 portion. the predictions, called low-confidence remasking. Addition-
ally, for LLaDA after SFT, we can divide the sequence into
The LLaDA 8B model undergoes SFT on a dataset com- several blocks and generate them from left to right, called
prising 4.5 million pairs. Consistent with the pre-training semi-autoregressive remasking. Within each block, we ap-
process, both data preparation and training follow the SFT ply the reverse process to perform sampling. We provide
protocols utilized in existing LLMs (Chu et al., 2024; Yang more details and ablation studies in Appendix. B.3.
et al., 2024), without introducing any additional techniques
to optimize LLaDA’s performance. The dataset spans mul- For conditional likelihood evaluation, we can naturally uti-
tiple domains, including code, mathematics, instruction- lize the upper bound in Eq. (5). However, we find that the
following, and structured data understanding. We append following equivalent form (Ou et al., 2024) exhibits lower
|EOS| tokens to the end of short pairs in each mini-batch variance and is more stable for evaluation:
to ensure equal lengths across all data. We treat |EOS| as a " L
#
LX i
normal token during training and remove it during sampling, −El,r0 ,rl 1[r = M] log pθ (r0i |p0 , rl ) , (6)
enabling LLaDA to control the response length automati- l i=1 l
cally. Please refer to Appendix B.1 for more details.
where l is uniformly sampled from {1, 2, . . . , L}, and rl is
We train for 3 epochs on the SFT data using a similar sched- obtained by uniformly sampling l tokens from r0 without
ule to the pre-training phase. The learning rate is linearly replacement for masking. In addition, we employ the unsu-
increased from 0 to 2.5 × 10−5 over the first 50 iterations pervised classifier-free guidance (Nie et al., 2024). We refer
and then kept constant. During the final 10% of iterations, the readers to more details in Appendix A.2.
it is linearly reduced to 2.5 × 10−6 . Additionally, we set
We present the training, sampling, and likelihood evaluation
the weight decay to 0.1, the global batch size to 256, and
algorithms, along with theoretical details, in Appendix A.
the local batch size to 2 per GPU. The SFT experiment was
executed once, without any hyperparameter tuning.
3. Experiments
2.4. Inference We evaluate the scalability, instruction-following, and in-
As a generative model, LLaDA is capable of both sampling context learning capabilities of LLaDA on standard bench-
new text and evaluating the likelihood of candidate text. marks, followed by analyses and case studies on more con-
trolled datasets to provide a comprehensive assessment.
We begin with the sampling. As illustrated in Fig. 2 (c),
given a prompt p0 , we discretize the reverse process to sam-
3.1. Scalability of LLaDA on Language Tasks
ple from the model distribution pθ (r0 |p0 ), starting from a
fully masked response. The total number of sampling steps We first investigate the scalability of LLaDA on downstream
is a hyperparameter, which naturally provides LLaDA with a tasks in comparison with the ARM baselines we constructed.
trade-off between efficiency and sample quality, as analyzed Specifically, at the 1B scale, we ensured that LLaDA and
in Sec. 3.3. We employ uniformly distributed timesteps by ARM shared the same architecture, data, and all other config-

4
 Large Language Diffusion Models

60 Autoregressive Baseline
60 60 Autoregressive Baseline
Autoregressive Baseline

CMMLU (5-shot)
LLaDA LLaDA
MMLU (5-shot)

ARC-C (0-shot)
LLaDA
50 50 50
40 40 40
30 30 30
20 20 20 20 20 20
10 1021 1022 1023 10 1021 1022 1023 10 1021 1022 1023
FLOPs FLOPs FLOPs
80 Autoregressive Baseline
60 24
Autoregressive Baseline

HumanEval (0-shot)
Autoregressive Baseline
LLaDA LLaDA

GSM8K (4-shot)
LLaDA
PIQA (0-shot)

70 40 16

60 20 8

50 20 0 0
10 1021 1022 1023 1020 1021 1022 1023 1020 1021 1022 1023
FLOPs FLOPs FLOPs
Figure 3. Scalability of LLaDA. We evaluate the performance of LLaDA and our ARM baselines trained on the same data across
increasing computational FLOPs. LLaDA exhibits strong scalability, matching the overall performance of ARMs on six tasks.

urations. At larger scales, we also report results for LLaDA The selection of tasks and evaluation protocols was aligned
and ARM models of slightly different sizes trained on the with existing studies, encompassing 15 popular benchmarks
same data due to resource limit, as detailed in Appendix B.2. in general tasks, mathematics, code, and Chinese. Further
We use the computational cost as a unified scaling metric. details are provided in Appendix B.5. For a more direct
For evaluation, we focused on six standard and diverse tasks. comparison, we re-evaluated representative LLMs (Touvron
et al., 2023; Dubey et al., 2024) in our implementation.
As shown in Fig. 3, LLaDA demonstrates impressive scala-
bility, with its overall trend being highly competitive with As shown in Tab. 1, after pretraining on 2.3T tokens,
that of ARM. Notably, in tasks such as MMLU and GSM8K, LLaDA 8B demonstrates remarkable performance, surpass-
LLaDA exhibits even stronger scalability. Even on tasks like ing LLaMA2 7B on nearly all tasks, and is overall competi-
PIQA, where performance lags, LLaDA narrows the gap tive with LLaMA3 8B. LLaDA shows advantages in math
with ARMs at larger scales. To account for the significant and Chinese tasks. We conjecture that the strengths stem
influence of outliers, we opted not to fit quantitative scaling from the same factors as its relatively weaker performance
curves, avoiding potential misinterpretation. Nevertheless, in some tasks—differences in data quality and distribution,
the results clearly demonstrate the scalability of LLaDA. largely due to the closed-source situation of LLM datasets.
Nie et al. (2024) suggests that MDM requires 16 times Notably, we have carefully ruled out the possibility of data
more computation than ARM to achieve the same likeli- leakage by taking GSM8K as an example. First, as shown
hood. However, there are key differences that make the in Fig. 3, LLaDA outperformed ARM baselines regarding
conclusions of this study more broadly applicable. In partic- GSM8K. Moreover, the conclusion remains on a fully un-
ular, likelihood is a relatively indirect metric for downstream seen GSM8K-like task (Ye et al., 2024) in Appendix B.7.
task performance, and diffusion optimizes a bound of the
Further, Tab. 2 compares the performance of LLaDA 8B In-
likelihood, making it not directly comparable to ARM. Ad-
struct with existing LLMs. We observed that SFT improved
ditionally, we extended the scaling range from 1018 ∼ 1020
LLaDA’s performance on most downstream tasks. A few
in Nie et al. (2024) to 1020 ∼ 1023 in this work.
metrics, such as MMLU, showed declines, and we conjec-
ture may be due to the suboptimal quality of the SFT data.
3.2. Benchmark Results Overall, since we did not perform alignment with reinforce-
To comprehensively evaluate the in-context learning and ment learning (RL), our results are slightly behind LLaMA3
instruction-following capabilities of LLaDA 8B, we con- 8B Instruct, though the gaps in many metrics remain small.
ducted detailed comparisons with existing LLMs (Touvron Notably, even with only SFT, LLaDA demonstrates impres-
et al., 2023; Dubey et al., 2024; Chu et al., 2024; Yang et al., sive instruction-following abilities, as detailed in Sec. 3.4.
2024; Bi et al., 2024; Jiang et al., 2023) of similar scale. We leave RL-based alignment for future work.

5
 Large Language Diffusion Models

Table 1. Benchmark Results of Pre-trained LLMs. ∗ indicates that LLaDA 8B Base, LLaMA2 7B Base, and LLaMA3 8B Base are
evaluated under the same protocol, detailed in Appendix B.5. Results indicated by † and ¶ are sourced from Chu et al. (2024); Yang et al.
(2024) and Bi et al. (2024) respectively. The numbers in parentheses represent the number of shots used for evaluation. “-” indicates
unknown data.
LLaDA 8B∗ LLaMA3 8B∗ LLaMA2 7B∗ Qwen2 7B† Qwen2.5 7B† Mistral 7B† Deepseek 7B¶
Model Diffusion AR AR AR AR AR AR
Training tokens 2.3T 15T 2T 7T 18T - 2T
General Tasks
MMLU 65.9 (5) 65.4 (5) 45.9 (5) 70.3 (5) 74.2 (5) 64.2 (5) 48.2 (5)
BBH 49.8 (3) 57.6 (3) 37.3 (3) 62.3 (3) 70.4 (3) 56.1 (3) 39.5 (3)
ARC-C 47.9 (0) 53.1 (0) 46.3 (0) 60.6 (25) 63.7 (25) 60.0 (25) 48.1 (0)
Hellaswag 72.5 (0) 79.1 (0) 76.0 (0) 80.7 (10) 80.2 (10) 83.3 (10) 75.4 (0)
TruthfulQA 46.4 (0) 44.0 (0) 39.0 (0) 54.2 (0) 56.4 (0) 42.2 (0) -
WinoGrande 74.8 (5) 77.3 (5) 72.5 (5) 77.0 (5) 75.9 (5) 78.4 (5) 70.5 (0)
PIQA 74.4 (0) 80.6 (0) 79.1 (0) - - - 79.2 (0)
Mathematics & Science
GSM8K 70.7 (4) 53.1 (4) 14.3 (4) 80.2 (4) 85.4 (4) 36.2 (4) 17.4 (8)
Math 27.3 (4) 15.1 (4) 3.2 (4) 43.5 (4) 49.8 (4) 10.2 (4) 6.0 (4)
GPQA 26.1 (5) 25.9 (5) 25.7 (5) 30.8 (5) 36.4 (5) 24.7 (5) -
Code
HumanEval 33.5 (0) 34.2 (0) 12.8 (0) 51.2 (0) 57.9 (0) 29.3 (0) 26.2 (0)
HumanEval-FIM 73.8 (2) 73.3 (2) 26.9 (2) - - - -
MBPP 38.2 (4) 47.4 (4) 18.4 (4) 64.2 (0) 74.9 (0) 51.1 (0) 39.0 (3)
Chinese
CMMLU 69.9 (5) 50.7 (5) 32.5 (5) 83.9 (5) - - 47.2 (5)
C-Eval 70.5 (5) 51.7 (5) 34.0 (5) 83.2 (5) - - 45.0 (5)

Table 2. Benchmark Results of Post-trained LLMs. LLaDA only employs an SFT procedure while other models have extra reinforcement
learning (RL) alignment. ∗ indicates that LLaDA 8B Instruct, LLaMA2 7B Instruct, and LLaMA3 8B Instruct are evaluated under the
same protocol, detailed in Appendix B.5. Results indicated by † and ¶ are sourced from Yang et al. (2024) and Bi et al. (2024) respectively.
The numbers in parentheses represent the number of shots used for in-context learning. “-” indicates unknown data.

LLaDA 8B∗ LLaMA3 8B∗ LLaMA2 7B∗ Qwen2 7B† Qwen2.5 7B† Gemma2 9B† Deepseek 7B¶
Model Diffusion AR AR AR AR AR AR
Training tokens 2.3T 15T 2T 7T 18T 8T 2T
Post-training SFT SFT+RL SFT+RL SFT+RL SFT+RL SFT+RL SFT+RL
Alignment pairs 4.5M - - 0.5M + - 1M + 0.15M - 1.5M + -
General Tasks
MMLU 65.5 (5) 68.4 (5) 44.1 (5) - - - 49.4 (0)
MMLU-pro 37.0 (0) 41.9 (0) 4.6 (0) 44.1 (5) 56.3 (5) 52.1 (5) -
Hellaswag 74.6 (0) 75.5 (0) 51.5 (0) - - - 68.5 (-)
ARC-C 88.5 (0) 82.4 (0) 57.3 (0) - - - 49.4 (-)
Mathematics & Science
GSM8K 78.6 (4) 78.3 (4) 29.0 (4) 85.7 (0) 91.6 (0) 76.7 (0) 63.0 (0)
Math 26.6 (0) 29.6 (0) 3.8 (0) 52.9 (0) 75.5 (0) 44.3 (0) 15.8 (0)
GPQA 31.8 (5) 31.9 (5) 28.4 (5) 34.3 (0) 36.4 (0) 32.8 (0) -
Code
HumanEval 47.6 (0) 59.8 (0) 16.5 (0) 79.9 (0) 84.8 (0) 68.9 (0) 48.2 (-)
MBPP 34.2 (4) 57.6 (4) 20.6 (4) 67.2 (0) 79.2 (0) 74.9 (0) 35.2 (-)

6
 Large Language Diffusion Models

first time, to the best of our knowledge. See more case

Table 3. Comparison in the Poem Completion Task.
studies on remasking and reasoning tasks in Appendix B.9.
Forward Reversal
GPT-4o (2024-08-06) 82.7 34.3 4. Related Work
Qwen2.5 7B Instruct 75.9 38.0 Diffusion models (Sohl-Dickstein et al., 2015; Ho et al.,
LLaDA 8B Instruct 48.8 42.4 2020; Song et al., 2020) have excelled in visual domains but
remain unverified for LLMs despite extensive efforts.
Overall, despite the lack of data transparency, we have made A simple approach is to continuousize text data and ap-
every effort to adopt standardized procedures and introduce ply diffusion models directly (Li et al., 2022; Gong et al.,
diverse tasks, we believe they sufficiently demonstrate the 2022; Han et al., 2022; Strudel et al., 2022; Chen et al.,
extraordinary capabilities of LLaDA, which is the only com- 2022; Dieleman et al., 2022; Richemond et al., 2022; Wu
petitive non-autoregressive model to our knowledge. et al., 2023; Mahabadi et al., 2024). Alternatively, some
methods model continuous parameters of discrete distri-
3.3. Reversal Reasoning and Analyses butions instead (Lou & Ermon, 2023; Graves et al., 2023;
Lin et al., 2023; Xue et al., 2024). However, scalability
To quantify the reversal reasoning ability of mod- remains a challenge, as a 1B parameter model requires 64
els (Berglund et al., 2023), we follow the protocol estab- times the compute of an ARM to achieve comparable per-
lished in Allen-Zhu & Li (2023). Specifically, we construct formance (Gulrajani & Hashimoto, 2024).
a dataset of 496 famous Chinese poem sentence pairs. Given
a sentence from a poem, models are tasked with generating Another approach replaces continuous diffusion with dis-
the subsequent line (forward) or the preceding line (rever- crete processes featuring new forward and reverse dy-
sal) without additional fine-tuning. Examples can be found namics (Austin et al., 2021a), leading to numerous vari-
in Appendix B.8. This setting provides a straightforward ants (Hoogeboom et al., 2021b;a; He et al., 2022; Campbell
and more realistic evaluation compared to previous stud- et al., 2022; Meng et al., 2022; Reid et al., 2022; Sun et al.,
ies (Nie et al., 2024; Kitouni et al., 2024). 2022; Zheng et al., 2023; Chen et al., 2023; Gat et al., 2024;
Zheng et al., 2024; Sahoo et al., 2024; Shi et al., 2024).
As shown in Tab. 3, LLaDA effectively addresses the rever- Notably, Lou et al. (2023) showed that masked diffusion,
sal curse (Berglund et al., 2023), demonstrating consistent as a special case of discrete diffusion, achieves perplexity
zero-shot performance across both forward and reversal comparable to or surpassing ARMs at GPT-2 scale. Ou et al.
tasks. In contrast, both Qwen 2.5 and GPT-4o exhibit a (2024) established fundamental theoretical results, which
significant gap between the two. The results on forward motivated our model design, training, and inference (see
generation confirm that both ARMs are strong, benefiting Appendix A). Nie et al. (2024) explored how MDM can be
from significantly larger datasets and greater computational leveraged for language tasks such as question answering
resources than LLaDA. However, LLaDA outperforms both at GPT-2 scale. Gong et al. (2024) fine-tune ARMs in the
by a large margin in the reversal task. MDM formulation. However, improvements are confined
We emphasize that we did not design anything special for to certain metrics, and it remains unclear whether this ap-
reversal tasks. Intuitively, LLaDA treats tokens uniformly proach can yield a foundation model comparable to strong
without inductive bias, leading to balanced performance. LLMs under a comprehensive evaluation.
See more details in Appendix A.2. In comparison, this study scales MDM to an unprecedented
We also analyze the effect of remasking strategies and sam- size of 8B parameters from scratch, achieving performance
pling steps, as detailed in Appendix B.3 and Appendix B.6. comparable to leading LLMs such as LLaMA 3.
Notably, a parallel line of work on image generation (Chang
3.4. Case Studies et al., 2022; 2023) aligns with MDMs applied to text data.
We present samples generated by LLaDA 8B Instruct in
Tab. 4, showcasing its instruction-following capabilities. 5. Conclusion and Discussion
First, the table illustrates LLaDA’s ability to generate co-
In the middle of difficulty lies opportunity.
herent, fluent, and extended text in a non-autoregressive
placeholder,placeholder,placeh—Albert Einstein
manner. Second, it highlights the model’s multi-turn dia-
logue capability, effectively retaining conversation history
and producing contextually appropriate responses across We introduce LLaDA, a principled and previously unex-
multiple languages. Such chat capabilities of LLaDA are plored approach to large language modeling based on dif-
impressive, as it departs from conventional ARMs for the fusion models. LLaDA demonstrates strong capabilities in

7
 Large Language Diffusion Models

Table 4. Visualization of the Sampling Process and a Generated Multi-round Dialogue. In the response of LLaDA, darker colors
indicate tokens predicted in the later stages of sampling, while lighter colors correspond to earlier predictions.

Sampling Process
User Lily can run 12 kilometers per hour for 4 hours. After that, she runs 6 kilometers per hour.
How many kilometers can she run in 8 hours?
LLaDA

Multi-round Dialogue Case

User Could you please share with me what the initial two lines of the renowned poem ‘The
Road Not Taken’?
LLaDA Certainly! The initial two lines of the renowned poem “The Road Not Taken” by Robert
Frost are: “Two roads diverged in a yellow wood, And sorry I could not travel both.” These
lines set the stage for the poem, introducing the idea of a speaker facing a choice between
two paths.
User Please help me translate into Chinese.
LLaDA Sure! The first two lines of “The Road Not Taken” by Robert Frost can be translated into
Chinese as: “两条路分岔在黄色的树林中，遗憾我不能同时走”
User Please translate into German.
LLaDA Sure! The first two lines of “The Road Not Taken” by Robert Frost can be translated into
German as: “Zwei Wege trennten sich im gelben Wald, und leider konnte ich nicht beide
tun.”
User Please also write a poem about life choices. I require 5 sentences, and the first word of
each sentence starts with C as the first letter.
LLaDA Certainly! Here’s a poem about life choices, with each sentence starting with the letter “C”:
Choices cascade through life’s journey, Creating connections with each decision, Careful
consideration leads to better paths, Courageous choices shape our destiny, Choosing wisely
guides us through the years.

scalability, in-context learning, and instruction-following, or position embeddings were designed for LLaDA, nor
achieving performance comparable to strong LLMs. In addi- were any system-level architectural optimizations applied.
tion, LLaDA offers unique advantages such as bidirectional On the inference side, our exploration of guidance mecha-
modeling and enhanced robustness, effectively addressing nisms (Dhariwal & Nichol, 2021; Ho & Salimans, 2022) re-
several inherent limitations of existing LLMs. Our findings mains preliminary, and LLaDA currently exhibits sensitivity
not only establish diffusion models as a viable and promis- to inference hyperparameters. Furthermore, LLaDA has yet
ing alternative but also challenge the prevailing assumption to undergo alignment with reinforcement learning (Ouyang
that these essential capabilities are inherently tied to ARMs. et al., 2022; Rafailov et al., 2024), which is crucial for im-
proving its performance and alignment with human intent.
While promising, the full potential of diffusion models re-
mains to be fully explored. Several limitations of this work Looking forward, the scale of LLaDA is still smaller than
present significant opportunities for future research. leading counterparts (Achiam et al., 2023; Dubey et al.,
2024; Google, 2024; Anthropic, 2024; Yang et al., 2024;
Due to computational constraints, direct comparisons be-
Liu et al., 2024), highlighting the need for further scaling
tween LLaDA and ARMs—such as training on identical
to fully assess its capabilities. In addition, LLaDA’s abil-
datasets—were restricted to a computational budget of less
ity to handle multi-modal data remains unexplored. The
than 1023 FLOPs. To allocate resources for training the
impact of LLaDA on prompt tuning techniques (Wei et al.,
largest possible LLaDA model and showcasing its poten-
2022) and its integration into agent-based systems (Park
tial, we were unable to scale the ARM baseline to the same
et al., 2023; Wang et al., 2024) is yet to be fully under-
extent. Moreover, no specialized attention mechanisms

8
 Large Language Diffusion Models

stood. Finally, a systematic investigation into post-training International Conference on Learning Representations,
for LLaDA could facilitate the development of O1-like sys- 2025. URL [Link]
tems (OpenAI, 2024; Guo et al., 2025). id=tyEyYT267x.

Anthropic. Claude 3.5 sonnet, 2024. URL [Link]

Impact Statement [Link]/news/claude-3-5-sonnet.
This work challenges the prevailing dominance of autore-
Austin, J., Johnson, D. D., Ho, J., Tarlow, D., and Van
gressive modeling in LLMs by proposing a diffusion-based
Den Berg, R. Structured denoising diffusion models in
alternative. Our findings open new avenues for exploring
discrete state-spaces. Advances in Neural Information
alternative probabilistic paradigms in natural language pro-
Processing Systems, 34:17981–17993, 2021a.
cessing, with potential applications in conversational AI,
code generation, and complex reasoning tasks. Austin, J., Odena, A., Nye, M., Bosma, M., Michalewski,
However, diffusion-based models, like traditional LLMs, H., Dohan, D., Jiang, E., Cai, C., Terry, M., Le, Q., et al.
raise similar societal concerns. These include the environ- Program synthesis with large language models. arXiv
mental impact of large-scale training, the potential misuse preprint arXiv:2108.07732, 2021b.
for generating harmful or misleading content, and the ampli-
Bao, F., Nie, S., Xue, K., Cao, Y., Li, C., Su, H., and Zhu,
fication of biases present in training data. Addressing these
J. All are worth words: A vit backbone for diffusion
challenges is critical to ensuring the responsible develop-
models. In Proceedings of the IEEE/CVF Conference
ment and deployment of diffusion-based language models.
on Computer Vision and Pattern Recognition, pp. 22669–
22679, 2023.
Acknowledgements
Bavarian, M., Jun, H., Tezak, N., Schulman, J., McLeavey,
This work was supported by the National Natural Science C., Tworek, J., and Chen, M. Efficient training of
Foundation of China (No. 92470118); Beijing Natural Sci- language models to fill in the middle. arXiv preprint
ence Foundation (No. L247030); Beijing Nova Program arXiv:2207.14255, 2022.
(No. 20220484044); Major Innovation & Planning Interdis-
ciplinary Platform for the “Double-First Class” Initiative, Berglund, L., Tong, M., Kaufmann, M., Balesni, M., Stick-
Renmin University of China; the Fundamental Research land, A. C., Korbak, T., and Evans, O. The reversal curse:
Funds for the Central Universities, the Research Funds of Llms trained on” a is b” fail to learn” b is a”. arXiv
Renmin University of China (22XNKJ13); Ant Group Re- preprint arXiv:2309.12288, 2023.
search Fund. The work was partially done at the Engineering
Research Center of Next-Generation Intelligent Search and Bi, X., Chen, D., Chen, G., Chen, S., Dai, D., Deng, C.,
Recommendation, Ministry of Education. Ding, H., Dong, K., Du, Q., Fu, Z., et al. Deepseek llm:
Scaling open-source language models with longtermism.
arXiv preprint arXiv:2401.02954, 2024.
References
Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Bisk, Y., Zellers, R., Gao, J., Choi, Y., et al. Piqa: Reasoning
Aleman, F. L., Almeida, D., Altenschmidt, J., Altman, S., about physical commonsense in natural language. In Pro-
Anadkat, S., et al. Gpt-4 technical report. arXiv preprint ceedings of the AAAI conference on artificial intelligence,
arXiv:2303.08774, 2023. 2020.

Ainslie, J., Lee-Thorp, J., de Jong, M., Zemlyanskiy, Y., Brooks, T., Peebles, B., Holmes, C., DePue, W.,
Lebron, F., and Sanghai, S. Gqa: Training generalized Guo, Y., Jing, L., Schnurr, D., Taylor, J., Luhman,
multi-query transformer models from multi-head check- T., Luhman, E., Ng, C., Wang, R., and Ramesh,
points. In Proceedings of the 2023 Conference on Em- A. Video generation models as world simulators.
pirical Methods in Natural Language Processing, pp. 2024. URL [Link]
4895–4901, 2023. video-generation-models-as-world-simulators.

Allen-Zhu, Z. and Li, Y. Physics of Language Mod- Brown, T. B. Language models are few-shot learners. arXiv
els: Part 3.2, Knowledge Manipulation. ArXiv e-prints, preprint arXiv:2005.14165, 2020.
abs/2309.14402, September 2023. Full version available
at [Link] Campbell, A., Benton, J., De Bortoli, V., Rainforth, T., Deli-
giannidis, G., and Doucet, A. A continuous time frame-
Anonymous. Interpolating autoregressive and discrete de- work for discrete denoising models. Advances in Neural
noising diffusion language models. In The Thirteenth Information Processing Systems, 35:28266–28279, 2022.

9
 Large Language Diffusion Models

Chang, H., Zhang, H., Jiang, L., Liu, C., and Freeman, Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle,
W. T. Maskgit: Masked generative image transformer. In A., Letman, A., Mathur, A., Schelten, A., Yang, A., Fan,
Proceedings of the IEEE/CVF Conference on Computer A., et al. The llama 3 herd of models. arXiv preprint
Vision and Pattern Recognition, pp. 11315–11325, 2022. arXiv:2407.21783, 2024.

Chang, H., Zhang, H., Barber, J., Maschinot, A., Lezama, J., Fisher, R. A. On the mathematical foundations of theoretical
Jiang, L., Yang, M.-H., Murphy, K., Freeman, W. T., statistics. Philosophical transactions of the Royal Society
Rubinstein, M., et al. Muse: Text-to-image genera- of London. Series A, containing papers of a mathematical
tion via masked generative transformers. arXiv preprint or physical character, 222(594-604):309–368, 1922.
arXiv:2301.00704, 2023.
Gao, L., Tow, J., Abbasi, B., Biderman, S., Black, S., DiPofi,
Chen, M., Tworek, J., Jun, H., Yuan, Q., Pinto, H. P. D. O., A., Foster, C., Golding, L., Hsu, J., Le Noac’h, A., Li,
Kaplan, J., Edwards, H., Burda, Y., Joseph, N., Brockman, H., McDonell, K., Muennighoff, N., Ociepa, C., Phang,
G., et al. Evaluating large language models trained on J., Reynolds, L., Schoelkopf, H., Skowron, A., Sutawika,
code. arXiv preprint arXiv:2107.03374, 2021. L., Tang, E., Thite, A., Wang, B., Wang, K., and Zou,
A. A framework for few-shot language model evaluation,
Chen, T., Zhang, R., and Hinton, G. Analog bits: Gen- 07 2024. URL [Link]
erating discrete data using diffusion models with self- 12608602.
conditioning. arXiv preprint arXiv:2208.04202, 2022.
Gat, I., Remez, T., Shaul, N., Kreuk, F., Chen, R. T., Syn-
Chen, Z., Yuan, H., Li, Y., Kou, Y., Zhang, J., and Gu, Q. naeve, G., Adi, Y., and Lipman, Y. Discrete flow match-
Fast sampling via de-randomization for discrete diffusion ing. arXiv preprint arXiv:2407.15595, 2024.
models. arXiv preprint arXiv:2312.09193, 2023. Gong, S., Li, M., Feng, J., Wu, Z., and Kong, L. Diffuseq:
Sequence to sequence text generation with diffusion mod-
Chu, Y., Xu, J., Yang, Q., Wei, H., Wei, X., Guo, Z., Leng,
els. arXiv preprint arXiv:2210.08933, 2022.
Y., Lv, Y., He, J., Lin, J., et al. Qwen2-audio technical
report. arXiv preprint arXiv:2407.10759, 2024. Gong, S., Agarwal, S., Zhang, Y., Ye, J., Zheng, L., Li, M.,
An, C., Zhao, P., Bi, W., Han, J., et al. Scaling diffu-
Clark, P., Cowhey, I., Etzioni, O., Khot, T., Sabharwal, A., sion language models via adaptation from autoregressive
Schoenick, C., and Tafjord, O. Think you have solved models. arXiv preprint arXiv:2410.17891, 2024.
question answering? try arc, the ai2 reasoning challenge.
arXiv preprint arXiv:1803.05457, 2018. Google. Our next-generation model: Gemini 1.5, 2024.
URL [Link]
Cobbe, K., Kosaraju, V., Bavarian, M., Chen, M., Jun, H., google-gemini-next-generation-model-february-2024
Kaiser, L., Plappert, M., Tworek, J., Hilton, J., Nakano,
R., et al. Training verifiers to solve math word problems. Graves, A., Srivastava, R. K., Atkinson, T., and
arXiv preprint arXiv:2110.14168, 2021. Gomez, F. Bayesian flow networks. arXiv preprint
arXiv:2308.07037, 2023.
Deletang, G., Ruoss, A., Duquenne, P.-A., Catt, E., Ge-
Gulrajani, I. and Hashimoto, T. B. Likelihood-based diffu-
newein, T., Mattern, C., Grau-Moya, J., Wenliang, L. K.,
sion language models. Advances in Neural Information
Aitchison, M., Orseau, L., et al. Language modeling is
Processing Systems, 36, 2024.
compression. In The Twelfth International Conference on
Learning Representations. Guo, D., Yang, D., Zhang, H., Song, J., Zhang, R., Xu, R.,
Zhu, Q., Ma, S., Wang, P., Bi, X., et al. Deepseek-r1: In-
Devlin, J. Bert: Pre-training of deep bidirectional trans- centivizing reasoning capability in llms via reinforcement
formers for language understanding. arXiv preprint learning. arXiv preprint arXiv:2501.12948, 2025.
arXiv:1810.04805, 2018.
Han, X., Kumar, S., and Tsvetkov, Y. Ssd-lm: Semi-
Dhariwal, P. and Nichol, A. Diffusion models beat gans autoregressive simplex-based diffusion language model
on image synthesis. Advances in neural information for text generation and modular control. arXiv preprint
processing systems, 34:8780–8794, 2021. arXiv:2210.17432, 2022.

Dieleman, S., Sartran, L., Roshannai, A., Savinov, N., He, Z., Sun, T., Wang, K., Huang, X., and Qiu, X. Diffu-
Ganin, Y., Richemond, P. H., Doucet, A., Strudel, R., sionbert: Improving generative masked language models
Dyer, C., Durkan, C., et al. Continuous diffusion for with diffusion models. arXiv preprint arXiv:2211.15029,
categorical data. arXiv preprint arXiv:2211.15089, 2022. 2022.

10
 Large Language Diffusion Models

Hendrycks, D., Burns, C., Basart, S., Zou, A., Mazeika, Amodei, D. Scaling laws for neural language models.
M., Song, D., and Steinhardt, J. Measuring mas- arXiv preprint arXiv:2001.08361, 2020.
sive multitask language understanding. arXiv preprint
arXiv:2009.03300, 2020. Kitouni, O., Nolte, N., Bouchacourt, D., Williams, A., Rab-
bat, M., and Ibrahim, M. The factorization curse: Which
Hendrycks, D., Burns, C., Kadavath, S., Arora, A., Basart, tokens you predict underlie the reversal curse and more.
S., Tang, E., Song, D., and Steinhardt, J. Measuring math- arXiv preprint arXiv:2406.05183, 2024.
ematical problem solving with the math dataset. arXiv
preprint arXiv:2103.03874, 2021. Li, H., Zhang, Y., Koto, F., Yang, Y., Zhao, H., Gong, Y.,
Duan, N., and Baldwin, T. Cmmlu: Measuring mas-
Ho, J. and Salimans, T. Classifier-free diffusion guidance. sive multitask language understanding in chinese. arXiv
arXiv preprint arXiv:2207.12598, 2022. preprint arXiv:2306.09212, 2023.
Ho, J., Jain, A., and Abbeel, P. Denoising diffusion proba- Li, X., Thickstun, J., Gulrajani, I., Liang, P. S., and
bilistic models. Advances in neural information process- Hashimoto, T. B. Diffusion-lm improves controllable
ing systems, 33:6840–6851, 2020. text generation. Advances in Neural Information Process-
Hoffmann, J., Borgeaud, S., Mensch, A., Buchatskaya, E., ing Systems, 35:4328–4343, 2022.
Cai, T., Rutherford, E., Casas, D. d. L., Hendricks, L. A., Lin, S., Hilton, J., and Evans, O. Truthfulqa: Measuring
Welbl, J., Clark, A., et al. Training compute-optimal how models mimic human falsehoods. arXiv preprint
large language models. arXiv preprint arXiv:2203.15556, arXiv:2109.07958, 2021.
2022.
Lin, Z., Gong, Y., Shen, Y., Wu, T., Fan, Z., Lin, C., Duan,
Holtzman, A., Buys, J., Du, L., Forbes, M., and Choi, Y. The
N., and Chen, W. Text generation with diffusion language
curious case of neural text degeneration. arXiv preprint
models: A pre-training approach with continuous para-
arXiv:1904.09751, 2019.
graph denoise. In International Conference on Machine
Hoogeboom, E., Gritsenko, A. A., Bastings, J., Poole, B., Learning, pp. 21051–21064. PMLR, 2023.
Berg, R. v. d., and Salimans, T. Autoregressive diffusion
models. arXiv preprint arXiv:2110.02037, 2021a. Liu, A., Feng, B., Xue, B., Wang, B., Wu, B., Lu, C., Zhao,
C., Deng, C., Zhang, C., Ruan, C., et al. Deepseek-v3
Hoogeboom, E., Nielsen, D., Jaini, P., Forré, P., and Welling, technical report. arXiv preprint arXiv:2412.19437, 2024.
M. Argmax flows and multinomial diffusion: Learning
categorical distributions. Advances in Neural Information Loshchilov, I. Decoupled weight decay regularization. arXiv
Processing Systems, 34:12454–12465, 2021b. preprint arXiv:1711.05101, 2017.

Hu, S., Tu, Y., Han, X., He, C., Cui, G., Long, X., Zheng, Z., Lou, A. and Ermon, S. Reflected diffusion models, 2023.
Fang, Y., Huang, Y., Zhao, W., et al. Minicpm: Unveiling
the potential of small language models with scalable train- Lou, A., Meng, C., and Ermon, S. Discrete diffusion lan-
ing strategies. arXiv preprint arXiv:2404.06395, 2024. guage modeling by estimating the ratios of the data distri-
bution. arXiv preprint arXiv:2310.16834, 2023.
Huang, Y., Bai, Y., Zhu, Z., Zhang, J., Zhang, J., Su, T., Liu,
J., Lv, C., Zhang, Y., Fu, Y., et al. C-eval: A multi-level Mahabadi, R. K., Ivison, H., Tae, J., Henderson, J., Beltagy,
multi-discipline chinese evaluation suite for foundation I., Peters, M. E., and Cohan, A. Tess: Text-to-text self-
models. Advances in Neural Information Processing conditioned simplex diffusion, 2024.
Systems, 36, 2024a.
Meng, C., Choi, K., Song, J., and Ermon, S. Concrete score
Huang, Y., Zhang, J., Shan, Z., and He, J. Compres- matching: Generalized score matching for discrete data.
sion represents intelligence linearly. arXiv preprint Advances in Neural Information Processing Systems, 35:
arXiv:2404.09937, 2024b. 34532–34545, 2022.

Jiang, A. Q., Sablayrolles, A., Mensch, A., Bamford, C., Nie, S., Zhu, F., Du, C., Pang, T., Liu, Q., Zeng, G., Lin, M.,
Chaplot, D. S., Casas, D. d. l., Bressand, F., Lengyel, G., and Li, C. Scaling up masked diffusion models on text.
Lample, G., Saulnier, L., et al. Mistral 7b. arXiv preprint arXiv preprint arXiv:2410.18514, 2024.
arXiv:2310.06825, 2023.
OpenAI. ChatGPT: Optimizing Language Models for Di-
Kaplan, J., McCandlish, S., Henighan, T., Brown, T. B., alogue. OpenAI blog, November 2022. URL https:
Chess, B., Child, R., Gray, S., Radford, A., Wu, J., and //[Link]/blog/chatgpt/.

11
 Large Language Diffusion Models

OpenAI. Learning to reason with llms, 2024. Shannon, C. E. A mathematical theory of communication.
URL [Link] The Bell system technical journal, 27(3):379–423, 1948.
learning-to-reason-with-llms/.
Shazeer, N. Glu variants improve transformer. arXiv
Ou, J., Nie, S., Xue, K., Zhu, F., Sun, J., Li, Z., and Li, preprint arXiv:2002.05202, 2020.
C. Your absorbing discrete diffusion secretly models the
conditional distributions of clean data. arXiv preprint Shi, J., Han, K., Wang, Z., Doucet, A., and Titsias, M. K.
arXiv:2406.03736, 2024. Simplified and generalized masked diffusion for discrete
data. arXiv preprint arXiv:2406.04329, 2024.
Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C.,
Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., Shih, A., Sadigh, D., and Ermon, S. Training and infer-
et al. Training language models to follow instructions ence on any-order autoregressive models the right way.
with human feedback. Advances in neural information In Proceedings of the 31th International Conference on
processing systems, 35:27730–27744, 2022. Machine Learning, 2022.
Park, J. S., O’Brien, J., Cai, C. J., Morris, M. R., Liang,
P., and Bernstein, M. S. Generative agents: Interactive Sohl-Dickstein, J., Weiss, E., Maheswaranathan, N., and
simulacra of human behavior. In Proceedings of the 36th Ganguli, S. Deep unsupervised learning using nonequi-
annual acm symposium on user interface software and librium thermodynamics. In International conference on
technology, pp. 1–22, 2023. machine learning, pp. 2256–2265. PMLR, 2015.

Peebles, W. and Xie, S. Scalable diffusion models with Song, Y., Sohl-Dickstein, J., Kingma, D. P., Kumar, A., Er-
transformers. In Proceedings of the IEEE/CVF Interna- mon, S., and Poole, B. Score-based generative modeling
tional Conference on Computer Vision, pp. 4195–4205, through stochastic differential equations. arXiv preprint
2023. arXiv:2011.13456, 2020.

Radford, A. Improving language understanding by genera- Strudel, R., Tallec, C., Altché, F., Du, Y., Ganin, Y., Men-
tive pre-training, 2018. sch, A., Grathwohl, W., Savinov, N., Dieleman, S., Sifre,
L., et al. Self-conditioned embedding diffusion for text
Radford, A., Wu, J., Child, R., Luan, D., Amodei, D.,
generation. arXiv preprint arXiv:2211.04236, 2022.
Sutskever, I., et al. Language models are unsupervised
multitask learners. OpenAI blog, 1(8):9, 2019. Su, J., Ahmed, M., Lu, Y., Pan, S., Bo, W., and Liu, Y.
Rafailov, R., Sharma, A., Mitchell, E., Manning, C. D., Er- Roformer: Enhanced transformer with rotary position
mon, S., and Finn, C. Direct preference optimization: embedding. Neurocomputing, 568:127063, 2024.
Your language model is secretly a reward model. Ad-
vances in Neural Information Processing Systems, 36, Sun, H., Yu, L., Dai, B., Schuurmans, D., and Dai, H. Score-
2024. based continuous-time discrete diffusion models. arXiv
preprint arXiv:2211.16750, 2022.
Reid, M., Hellendoorn, V. J., and Neubig, G. Diffuser:
Discrete diffusion via edit-based reconstruction, 2022. Suzgun, M., Scales, N., Schärli, N., Gehrmann, S., Tay,
Y., Chung, H. W., Chowdhery, A., Le, Q. V., Chi,
Rein, D., Hou, B. L., Stickland, A. C., Petty, J., Pang, E. H., Zhou, D., et al. Challenging big-bench tasks and
R. Y., Dirani, J., Michael, J., and Bowman, S. R. Gpqa: whether chain-of-thought can solve them. arXiv preprint
A graduate-level google-proof q&a benchmark. arXiv arXiv:2210.09261, 2022.
preprint arXiv:2311.12022, 2023.
Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi,
Richemond, P. H., Dieleman, S., and Doucet, A. Categorical
A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P.,
sdes with simplex diffusion, 2022.
Bhosale, S., et al. Llama 2: Open foundation and fine-
Sahoo, S. S., Arriola, M., Schiff, Y., Gokaslan, A., Marro- tuned chat models. arXiv preprint arXiv:2307.09288,
quin, E., Chiu, J. T., Rush, A., and Kuleshov, V. Simple 2023.
and effective masked diffusion language models. arXiv
preprint arXiv:2406.07524, 2024. Uria, B., Murray, I., and Larochelle, H. A deep and tractable
density estimator. In Proceedings of the 31th Interna-
Sakaguchi, K., Bras, R. L., Bhagavatula, C., and Choi, Y. tional Conference on Machine Learning, 2014.
Winogrande: An adversarial winograd schema challenge
at scale. Communications of the ACM, 64(9):99–106, Vaswani, A. Attention is all you need. arXiv preprint
2021. arXiv:1706.03762, 2017.

12
 Large Language Diffusion Models

Wang, L., Ma, C., Feng, X., Zhang, Z., Yang, H., Zhang, J.,
Chen, Z., Tang, J., Chen, X., Lin, Y., et al. A survey on
large language model based autonomous agents. Frontiers
of Computer Science, 18(6):186345, 2024.
Wei, J., Wang, X., Schuurmans, D., Bosma, M., Xia, F., Chi,
E., Le, Q. V., Zhou, D., et al. Chain-of-thought prompting
elicits reasoning in large language models. Advances in
neural information processing systems, 35:24824–24837,
2022.
Wu, T., Fan, Z., Liu, X., Gong, Y., Shen, Y., Jiao, J., Zheng,
H.-T., Li, J., Wei, Z., Guo, J., Duan, N., and Chen, W.
Ar-diffusion: Auto-regressive diffusion model for text
generation, 2023.
Xue, K., Zhou, Y., Nie, S., Min, X., Zhang, X., Zhou, J.,
and Li, C. Unifying bayesian flow networks and diffusion
models through stochastic differential equations. arXiv
preprint arXiv:2404.15766, 2024.
Yang, A., Yang, B., Zhang, B., Hui, B., Zheng, B., Yu, B.,
Li, C., Liu, D., Huang, F., Wei, H., Lin, H., Yang, J., Tu,
J., Zhang, J., Yang, J., Yang, J., Zhou, J., Lin, J., Dang,
K., Lu, K., Bao, K., Yang, K., Yu, L., Li, M., Xue, M.,
Zhang, P., Zhu, Q., Men, R., Lin, R., Li, T., Xia, T., Ren,
X., Ren, X., Fan, Y., Su, Y., Zhang, Y., Wan, Y., Liu, Y.,
Cui, Z., Zhang, Z., and Qiu, Z. Qwen2.5 technical report.
arXiv preprint arXiv:2412.15115, 2024.
Ye, T., Xu, Z., Li, Y., and Allen-Zhu, Z. Physics of Lan-
guage Models: Part 2.1, Grade-School Math and the Hid-
den Reasoning Process. ArXiv e-prints, abs/2407.20311,
July 2024. Full version available at [Link]
org/abs/2407.20311.
Zellers, R., Holtzman, A., Bisk, Y., Farhadi, A., and Choi,
Y. Hellaswag: Can a machine really finish your sentence?
arXiv preprint arXiv:1905.07830, 2019.
Zhang, B. and Sennrich, R. Root mean square layer nor-
malization. Advances in Neural Information Processing
Systems, 32, 2019.
Zhao, W. X., Zhou, K., Li, J., Tang, T., Wang, X., Hou, Y.,
Min, Y., Zhang, B., Zhang, J., Dong, Z., et al. A survey of
large language models. arXiv preprint arXiv:2303.18223,
2023.
Zheng, K., Chen, Y., Mao, H., Liu, M.-Y., Zhu, J., and
Zhang, Q. Masked diffusion models are secretly time-
agnostic masked models and exploit inaccurate categor-
ical sampling, 2024. URL [Link]
abs/2409.02908.
Zheng, L., Yuan, J., Yu, L., and Kong, L. A reparameter-
ized discrete diffusion model for text generation. ArXiv,
abs/2302.05737, 2023.

13
 Large Language Diffusion Models

Algorithm 1 Pre-training of LLaDA

Require: mask predictor pθ , data distribution pdata
1: repeat
2: x0 ∼ pdata , t ∼ U(0, 1] # with a probability of 1%, the sequence length of x0 follows U[1, 4096]
3: xt ∼ qt|0 (xt |x0 ) # qt|0 is defined in Eq. (7)
1
PL i i
4: Calculate L = − t∗L i=1 1[xt = M] log pθ (x0 |xt ) # L is the sequence length of x0
5: Calculate ∇θ L and run optimizer.
6: until Converged
7: Return pθ

Algorithm 2 Supervised Fine-Tuning of LLaDA

Require: mask predictor pθ , pair data distribution pdata
1: repeat
2: p0 , r0 ∼ pdata , t ∼ U(0, 1] # please refer to Appendix B.1 for details on the SFT data processing.
3: rt ∼ qt|0 (rt |r0 ) # qt|0 is defined in Eq. (7)
1
PL′ i i ′
4: Calculate L = − t∗L ′ i=1 1[rt = M] log p (r |p
θ 0 0 t , r ) # L is the sequence length of r0
5: Calculate ∇θ L and run optimizer.
6: until Converged
7: Return pθ

Algorithm 3 Conditional Log-likelihood Evaluation of LLaDA

Require: mask predictor pθ , prompt p0 , response r0 , the number of Monte Carlo estimations nmc
1: log likelihood = 0
2: for i ← 1 to nmc do
3: l ∼ {1, 2, . . . , L} # L is the sequence length of r0
4: Obtain rl by uniformly sampling l tokens
PL from r0 without replacement for masking
5: log likelihood = log likelihood + Ll i=1 1[rli = M] log pθ (r0i |p0 , rl )
6: end for
7: log likelihood = log likelihood/nmc
8: Return log likelihood

A. Formulation of Masked Diffusion Models

A.1. Training
Masked diffusion models (MDMs) (Austin et al., 2021a; Lou et al., 2023; Ou et al., 2024) define the model distribution
pθ (x0 ) in a manner distinct from autoregressive models.
These models introduce a forward process {xt } indexed by a time t ∈ [0, 1]. This process gradually and independently
masks all tokens in the sequence x0 . At time t = 0, the data point x0 is fully observed with no masks, while for t ∈ (0, 1],
xt represents latent variables with varying mask ratios in expectation.
Formally, the conditional distribution of xt given x0 is defined by a fully factorized form:
L
Y
qt|0 (xt |x0 ) = qt|0 (xit |xi0 ), (7)
i=1

where the conditional distribution for each token is given by:

(
i i 1 − t, xit = xi0 ,
qt|0 (xt |x0 ) = (8)
t, xit = M.

Here, M denotes the mask token. Intuitively, each token either remains unchanged or is masked, with the probability of

14
 Large Language Diffusion Models

Algorithm 4 Reverse Process of LLaDA

Require: mask predictor pθ , prompt p0 , answer length L, sampling steps N
1: Set r1 is a fully masked sequence of length L.
1
2: for t ← 1 down to N step N1 do
1
3: s=t− N
4: r0 = arg maxr0 pθ (r0 |p0 , rt ) # we employ greedy sampling when predicting masked tokens
5: for i ← 1 to L do
6: if rt ̸= M then
7: r0i = rti
8: else
9: With probability st , r0i is set to M
10: end if
11: end for
12: rs = r0
13: end for
14: Return r0

being masked increasing linearly as t progresses from 0 to 1. At t = 1, all tokens are guaranteed to be masked, meaning that
x1 follows a Dirac distribution concentrated on a sequence of fully masked tokens. Notably, the linear masking probability
is analogous to but distinct from, the noise schedule in continuous diffusion models (Sohl-Dickstein et al., 2015; Ho et al.,
2020; Song et al., 2020). This linearity is motivated by the assumption that the information in the text is proportional to the
number of tokens on average, making it reasonable to lose information linearly during the forward process.
The forward process is not only reversible but also corresponds to a reverse process that is fully factorized across all
tokens (Austin et al., 2021a). The reverse process, from time t = 1 to 0, generates new data from sequences of fully masked
tokens. The conditional distribution for the reverse process, for 0 ≤ s < t ≤ 1, is factorized as:
L
Y
qs|t (xs |xt ) = qs|t (xis |xt ), (9)
i=1

where the conditional distribution for each token is:



 1, xit ̸= M, xis = xit ,
s,

xit = M, xis = M,
qs|t (xis |xt ) = t−s
t (10)
 q0|t (xs |xt ), xit = M, xis ̸= M,
i
 t


0, otherwise.

Thus, the key function to estimate is the conditional distribution q0|t (xis |xt ), which predicts the original token if it is masked
in the input xt . This is analogous to the data prediction form in continuous diffusion models.
As proven in (Ou et al., 2024), an equivalent yet time-free parameterization can be derived as:

q0|t (xis |xt ) = pdata (xi0 |xUM

t ), ∀i such that xit = M, (11)

where xUM
t denotes the collection of unmasked tokens in xt , which is identical to the corresponding tokens in the original
data x0 since unmasked tokens are solely determined by x0 and are independent of time t. Intuitively, this implies that
estimating the data prediction function is equivalent to estimating the conditional distributions on clean data, which is
time-invariant. Consequently, the time t need not be provided as input to the parametric model.
Although the development of masked diffusion is nontrivial, the implementation is straightforward. We first introduce the
mask predictor, a parametric model pθ (·|xt ) (e.g., a Transformer without a causal mask), which takes xt for any t as input
and predict all masked tokens simultaneously. Then, we define the model distribution pθ (x0 ) as follows: starting with x1 as
a sequence of fully masked tokens, we simulate an approximate reverse process parameterized by pθ (·|xt ) from t = 1 to 0.
The marginal distribution induced at t = 0 then represents the model distribution pθ (x0 ).

15
 Large Language Diffusion Models

Algorithm 5 Low-confidence Remasking Strategy of LLaDA

Require: mask predictor pθ , prompt p0 , answer length L, sampling steps N
1: Set r1 is a fully masked sequence of length L.
1
2: for t ← 1 down to N step N1 do
3: s = t − N1
4: for i ← 1 to L do
5: if rti ̸= M then
6: r0i = rti , ci = 1
7: else
8: r0i = arg maxr0i pθ (r0i |p0 , rt )
9: ci = pθ (r0i |p0 , rt )r0i
10: end if
11: end for
12: nun = ⌊L(1 − s)⌋ # the number of unmasked tokens is nun in timestep s
13: for i ← 1 to L do

14: if ci ∈ Lowest − nun {ci }L 1 then
15: r0i = M # the nun positions with the least confidence are selected for remasking.
16: end if
17: end for
18: rs = r0
19: end for
20: Return r0

Formally, the mask predictor is trained using a cross-entropy loss with masking:
" L #
1X i i
L(θ) ≜ −Et,x0 ,xt 1[xt = M] log pθ (x0 |xt ) , (12)
t i=1

where x0 is sampled from the training data, t is sampled uniformly from [0, 1], and xt is sampled from qt|0 (xt |x0 ). The
indicator function 1[·] ensures that the cross-entropy loss is computed only for masked tokens. In Ou et al. (2024), it has
been proven that the loss function L(θ) is an upper bound on the negative log-likelihood of the model distribution:
−Ex0 ∼pdata(x0 ) [log pθ (x0 )] ≤ L(θ). (13)
In summary, this principled approach trains a generative model by progressively masking tokens during a forward process
and learning to recover the data distribution during a reverse process, all under the (approximate) maximum likelihood
estimation framework.

A.2. Inference
The cross-entropy loss in Eq. (12) has several equivalent forms (Ou et al., 2024). The first one is given by
" L
#
LX
−El∼{1,2,...,L},x0 ,xl 1[xil = M] log pθ (xi0 |xl ) , (14)
l i=1

where l is uniformly sampled from {1, 2, . . . , L}, and xl is obtained by uniformly sampling l tokens from x0 without
replacement for masking. Despite masking exactly l tokens is different from masking each token independently with
probability t, these two masking methods lead to equivalent results in expectation (Ou et al., 2024).
While Eq. (12) and Eq. (14) share the same expectation, their variances differ. Intuitively, in Eq. (12), we expect xt to
have a fraction of t tokens masked. However, the randomness of the forward process (i.e., Eq. (7)) often causes deviations,
especially when xt contains few tokens. In contrast, in Eq. (14), the fraction of masked tokens in xl is deterministically Ll .
While a theoretical analysis depends on the data distribution, empirical results show that Eq. (12) requires over 1000 Monte
Carlo estimates for stable results, whereas Eq. (14) achieves stability with only 128 estimates. In addition, we can simply
modify Eq. (14) to its conditional version (i.e., Eq. (6)) based on Eq. (5).

16
 Large Language Diffusion Models

Prompt Block 0 Block 1 Block 2

Mask token 𝑡=1
Non-mask token

𝑡=0

Figure 4. A Conceptual Overview of the Semi-autoregressive Sampling.

Any-order autoregressive models (AO-ARM) (Uria et al., 2014; Hoogeboom et al., 2021a; Shih et al., 2022) characterize the
joint distribution autoregressively for all possible orders π of the L variables. To learn such a distribution, an AO-ARM
utilizes a weight-sharing neural network to model all univariate conditionals and employs mask tokens to represent absent
variables. During training, the expected negative log-likelihood over the uniform distribution of all orders Uπ is minimized:
" L #
π(i) π(<i)
X
−Ex0 ,π∼Uπ log pθ (x0 |x0 ; π) . (15)
i=1

π(<i)
Intuitively, x0 can be understood as a masked token xt with index in π(≥ i) being masked. It can be further proved that
Eq. (15) is equivalent to Eq. (12). This connection explains the bidirectional reasoning capabilities of LLaDA, even though
it was never used explicitly in the inference procedure.
Nie et al. (2024) introduce unsupervised classifier-free guidance, a plug-and-play technique that balances alignment with
prompts and text diversity. Specifically, unsupervised classifier-free guidance employs the following modified mask predictor
for inference:

pθ (r0 |p0 , rt )1+w

p̃θ (r0 |p0 , rt ) ∝ , (16)
pθ (r0 |m, rt )w

where m is a mask sequence of the same length as p0 and w is a hyperparameter that controls the strength of p0 . We employ
unsupervised classifier-free guidance in the downstream tasks, as detailed in Appendix B.5.

A.3. Algorithms
In this section, we present the training and inference algorithms. Specifically, we introduce the pre-training and supervised
fine-tuning algorithms in Algorithm 1 and Algorithm 2, respectively. In addition, the likelihood evaluation algorithm is
provided in Algorithm 3. Finally, we present the sampling algorithms in Algorithm 4 and Algorithm 5, which correspond to
the original reverse process (i.e., random remasking) and the low-confidence remasking strategy (Chang et al., 2022) of
LLaDA, respectively.
Furthermore, we apply a semi-autoregressive remasking strategy for the instruct model. Specifically, the sequence is
divided into several blocks and generated from left to right. Within each block, we apply the random remasking or the
low-confidence remasking strategy for sampling. For clarity and better visualization, we illustrate the semi-autoregressive
sampling process in Fig. 4.
Our semi-autoregressive remasking strategy is inspired by the prior work of Anonymous (2025). However, it is important to
note that Anonymous (2025) and LLaDA have entirely different training objectives and network architectures. Specifically,
the pre-training objective of Anonymous (2025) differs from Eq. (3). Moreover, their approach incorporates a causal mask
in the Transformer attention block, whereas LLaDA employs a bidirectional Transformer.

17
 Large Language Diffusion Models

Table 5. Model Architecture. We provide the configurations of our 1B and 7B ARM baselines, as well as the 1B and 8B LLaDA and the
8B LLaMA3 models.

Our ARM Baseline 1B LLaDA 1B Our ARM Baseline 7B LLaDA 8B LLaMA3 8B

Layers 22 22 28 32 32
Model dimension 2048 2048 4096 4096 4096
Attention heads 32 32 32 32 32
Vocabulary size 126,464 126,464 126,464 126,464 128,000
FFN dimension 5634 5634 13,440 12,288 14,336
Key/Value heads 4 4 8 32 8
Total parameters 1.49 B 1.49 B 6.83 B 8.02 B 8.03 B
Non-embedding parameters 0.97 B 0.97 B 5.80 B 6.98 B 6.98 B

B. Experiments
B.1. Details of SFT Data
A dynamic sequence length is employed during supervised SFT, where |EOS| tokens are appended to the end of short pairs
to ensure equal sequence lengths across all data in each mini-batch. Notably, the padding |EOS| tokens are treated as part of
the response, i.e., masked and included in the training objective. The |EOS| tokens are removed from the generated outputs
during sampling. This strategy is crucial for LLaDA and ensures that the model learns to control the length of its responses
by generating |EOS|, enabling the response length to align effectively with the given prompt.
In addition, for n-turn dialogues (p00 , r00 , p10 , r01 , . . . , pn−1
0 , r0n−1 ), we treat it as n single-turn dialogue pairs, i.e.,
0 0 0 0 1 1 0 0 1 1 n−1 n−1
(p0 , r0 ), (p0 r0 p0 , r0 ), . . . , (p0 r0 p0 r0 . . . p0 , r0 ) and randomly sample one. This data partitioning strategy not only
equips LLaDA with multi-turn dialogue capabilities but also aligns with the padding strategy.

B.2. Architectures and Scaling Experiments

This section provides detailed information about the models we trained and the scalability experiments described in Sec. 3.1.
Firstly, for efficiency, we trained an ARM and a MDM, both with 1.5B parameters and identical architectures. Additionally,
we scaled the MDM to 8B parameters. Due to computational resource constraints, we did not train an 8B autoregressive
model with the same architecture. Instead, we utilized our previously trained 7B autoregressive model for comparison.
These four models are utilized in the scalability analysis in Sec. 3.1.
We adopted a Transformer architecture similar to LLaMA (Touvron et al., 2023; Dubey et al., 2024) for the ARMs and MDMs
we trained. Specifically, we employ RMSNorm (Zhang & Sennrich, 2019) to stabilize training, used SwiGLU (Shazeer,
2020) as the activation function to enhance non-linearity, and integrate RoPE (Su et al., 2024) for more expressive positional
encoding. Tab. 5 provides an overview of the model architectures.
Additionally, we employ the widely used 6N D formulation (Kaplan et al., 2020; Hoffmann et al., 2022) to calculate the
training FLOPs in Fig. 3, where N represents the number of non-embedding parameters, and D denotes the total number of
training tokens.

B.3. Details and Ablation on Remasking

This section first explains the inference methods in detail, followed by ablation studies on various remasking strategies.
For instruct model, since our training includes a large number of |EOS| tokens, directly applying the lowest confidence
remasking strategy would result in an excessively high and unnatural occurrence of |EOS| tokens in the generated sentences.
Therefore, we divide the entire generation length into several blocks. Each block is generated from left to right, and within
each block, we apply the reverse process to perform sampling. However, for the base model, since |EOS| tokens are not
padded during training, using the semi-autoregressive remasking strategy makes little difference in the results.

18
 Large Language Diffusion Models

Table 6. Ablation on Remasking. We use GSM8K accuracy as the evaluation metric. The lowest-confidence & semi-autoregressive
remasking strategy refers to generating each block from left to right, while applying the lowest-confidence remasking strategy within each
block.

LLaDA 8B Base LLaDA 8B Instruct

Randomly remasking 52.3 72.0
Lowest confidence remasking 64.7 12.9
Lowest confidence & semi-autoregressive remasking 64.4 73.8

Table 7. Ablation on Generated Length. We use the GSM8K accuracy as the evaluation metric and set the number of sampling steps to
half of the generated length.

Length LLaDA 8B Base LLaDA 8B Instruct

256 62.5 75.3
512 64.7 73.8
1024 65.9 75.3

For efficiency, we set the answer length to 512 and the number of sampling steps to 256 in our ablation studies. For the
semi-autoregressive remasking strategy, we set the block length to 32. To simplify the evaluation, we use GSM8K accuracy
as the performance metric. As shown in Table 6, for the base model, applying the lowest confidence remasking strategy
significantly improves performance, whereas the semi-autoregressive remasking strategy does not yield any noticeable
benefits. In contrast, for the instruct model, directly applying the lowest confidence remasking strategy leads to a substantial
performance drop. However, combining the lowest confidence remasking strategy with the semi-autoregressive remasking
strategy results in the best performance for the instruct model.
We provide the specific remasking strategies and hyperparameters used for each benchmark in Appendix B.5.

B.4. Ablation on Generated Length

In this section, we conduct ablation studies on the generated length. For simplicity, we use GSM8K as the evaluation metric.
To ensure efficiency and fairness, for each setting, we set the number of sampling steps to half of the generated length,
ensuring that in each sampling step, two tokens are transferred from the mask to the text.
For the LLaDA 8B Base model, we employ the lowest-confidence remasking strategy (i.e., Algorithm 5). For the LLaDA
8B Instruct model, we adopt semi-autoregressive sampling with a block length of 32.
As reported in Tab. 7, both the Base model and the Instruct model exhibit minimal sensitivity to the length hyperparameter.

B.5. Standard Benchmarks and Evaluation

In this section, we briefly introduce the benchmarks used in our study and describe the details of our evaluation process.
Following standard LLM (Chu et al., 2024; Yang et al., 2024) evaluation practices, we assess LLaDA across four dimensions:
General ability: MMLU (Hendrycks et al., 2020), BBH (Suzgun et al., 2022), ARC-C (Clark et al., 2018), Hel-
laswag (Zellers et al., 2019), TruthfulQA (Lin et al., 2021), WinoGrande (Sakaguchi et al., 2021) and PIQA (Bisk
et al., 2020).
Math and science ability: GSM8K (Cobbe et al., 2021), Math (Hendrycks et al., 2021) and GPQA (Rein et al., 2023).
Code generation: HumanEval (Chen et al., 2021), HumanEval-FIM (Bavarian et al., 2022) and MBPP (Austin et al.,
2021b).
Chinese understanding: CMMLU (Li et al., 2023) and C-Eval (Huang et al., 2024a).
For all the aforementioned benchmarks, we follow the widely adopted evaluation process (Gao et al., 2024) used in LLM
assessments, primarily employing conditional likelihood estimation and conditional generation. Specifically, for certain
benchmarks, a prompt and multiple candidate answers are provided, and the model is required to compute each candidate’s

19
 Large Language Diffusion Models

Table 8. Inference Configuration for LLaDA 8B Instruct. We apply the low-confidence remasking strategy across all datasets. A block
length smaller than the answer length indicates the use of the semi-autoregressive remasking strategy; otherwise, it is not applied. The
shorter answer lengths for MMLU and HellaSwag result from their requirement to generate only a single token (i.e., A, B, C, or D) and
two special end tokens (i.e., end of the model response and end of the sentence).

Answer length Block length

MMLU 3 3
Hellaswag 3 3
GPQA 128 64
GSM8K 256 8
HumanEval 512 32
MBPP 512 32
Math 256 256
MMLU-pro 256 256
ARC-C 512 512

Table 9. Comparison on iGSM Dataset.

4 steps 5 steps 6 steps

LLaMA3 8B Base 38.0 35.0 34.0
LLaDA 8B Base 64.0 41.0 44.0

conditional likelihood. The candidate with the highest likelihood is then selected as the model’s final answer, and accuracy
is used as the evaluation metric. For the remaining benchmarks, the model generates responses based on the given prompt,
and performance is evaluated using metrics such as exact match and other relevant criteria.
For the base model, we use conditional likelihood estimation for MMLU, CMMLU, C-Eval, ARC-C, Hellaswag, TruthfulQA,
WinoGrande, PIQA, and GPQA, while the remaining benchmarks are evaluated using conditional generation. For the
instruct model, we evaluate all benchmarks using conditional generation. For datasets evaluated with conditional likelihood
estimation, we adopt the widely used lm-evaluation-harness (Gao et al., 2024) framework. However, for datasets evaluated
with conditional generation, we rely on an internal library for evaluation, as lm-evaluation-harness lacks implementations
for certain key metrics (e.g., HumanEval-FIM).
For benchmarks evaluated using conditional likelihood estimation, we use Monte Carlo estimation to approximate Eq. (6)
for conditional likelihood estimation. Since MMLU, CMMLU, and C-EVAL only require the likelihood of a single token, a
single Monte Carlo estimate is sufficient for these benchmarks. For all other benchmarks, we find that 128 Monte Carlo
samples are adequate to produce stable results. For ARC-C, Hellaswag, TruthfulQA, WinoGrande, PIQA, and GPQA, we
report the results with the optimal unsupervised classifier-free guidance scale searched in {0, 0.5, 1, 1.5, 2}.
For benchmarks evaluated using conditional generation, we apply the lowest confidence remasking strategy to the base model,
setting both the response length and the number of sampling steps to 1024. For the instruct model, we set the sampling steps
equal to the answer length and provide the inference hyperparameters in Table 8. Due to computational resource constraints,
we do not use the unsupervised classifier-free guidance for these benchmarks evaluated using conditional generation.

B.6. Analysis of Sampling Steps

In this section, we analyze the impact of sampling steps. For simplicity, we use the LLaDA 8B Base model and evaluate it
on the GSM8K and HumanEval datasets. Additionally, we fix the answer length at 1024. As shown in Fig. 5, as the number
of sampling steps increases to match the answer length, the model’s accuracy improves accordingly.

B.7. Evaluation on iGSM Dataset

To further assess the mathematical capabilities of LLaDA, we test its performance on iGSM (Ye et al., 2024), an infinite,
synthetic GSM8K-like dataset. iGSM is generated via specific rules, with parameters that control the difficulty of problems
(i.e., the number of solution steps). For evaluation consistency, we append ”#### $answer” to the final solution, adhering to

20
 Large Language Diffusion Models

60 30

HumanEval (0-shot)
GSM8K (4-shot)

40 20

20 10

64 128 256 512 1,024 64 128 256 512 1,024

NFEs NFEs
Figure 5. Analysis of Sampling Steps.

the GSM8K format. Below is an example with solution steps set to 4:

(Question) The number of each North Star Elementary’s Cultural Studies Classroom equals 1. The number of each
Westridge Elementary’s Dance Studio equals 3 times as much as the sum of each North Star Elementary’s Classroom
and each North Star Elementary’s Cultural Studies Classroom. How many Dance Studio does Westridge Elementary
have?
(Solution) Define North Star Elementary’s Cultural Studies Classroom as x; so x = 1.
Define North Star Elementary’s Classroom as m; so m = x = 1.
Define Westridge Elementary’s Dance Studio as n; w = m + x = 1 + 1 = 2;
so n = 3 * w = 3 * 2 = 1 #### 1

Since there are slight differences between GSM8K and iGSM (e.g., the use of a mod 5 algorithmic system), we follow (Ye
et al., 2024) and provide a system prompt along with four-shot question-answer pairs for each problem.

(Prompt) You’re an expert at solving elementary math problems involving addition, subtraction, and multiplication.
You solve all the problems in a uniform format. All calculations are done modulo 5. For example, 4 + 4 equals 3, 2 + 4
equals 1, 3 + 3 + 3 equals 4, 3 * 3 equals 4, and 2 * 2 equals 4. When providing your solution, please end with ’#### x.’
where x is your final answer, an integer between 0 and 4. You must solve all the problems using the same solution
format. Our scenarios involve up to four categories of objects: schools, classrooms, backpacks and stationeries. Each
school may contain classrooms, each classroom may contain backpacks, and each backpack may contain stationeries.
We can specify quantities, such as ẗhe number of dance studios at each Lakeshore High.Ässume that every entity
with the same name has an identical configuration; for example, each Lakeshore High contains the same number of
dance studios. Another guiding principle is that what is not mentioned does not exist: when we refer to classrooms at
Lakeshore High, we are only discussing the classrooms explicitly mentioned in our scenario. Furthermore, if Lakeshore
High is not even mentioned, any classroom within it is automatically considered to be non-existent (i.e. 0).

For solution steps ranging from 4 to 6, we generate 100 questions for each case and report the corresponding accuracy in
Table 9. As shown in the table, LLaDA Base demonstrates significant and consistent advantages over LLaMA3 on unseen
mathematical problems, aligning with the results in Table 1.

B.8. Poem Completion Tasks

In this section, we present examples from our poem completion dataset as follows.
Example 1:
Prompt: 窈窕淑女的下一句是什么？直接输出句子即可。
Answer: 君子好逑。

21
 Large Language Diffusion Models

Table 10. Visualization of the Semi-Autoregressive Sampling Process. In the response of LLaDA, darker colors indicate tokens
predicted in the later stages of sampling, while lighter colors correspond to earlier predictions.

User What are the benefits of regular exercise for physical and mental health?
LLaDA

Table 11. Visualization of the Multi-turn Dialogue. We employ random remasking strategy. In the response of LLaDA, darker colors
indicate tokens predicted in the later stages of sampling, while lighter colors correspond to earlier predictions.

User I want to have a pet, but I don’t know whether to get a cat or a dog. Do you have any
suggestions?
LLaDA

User I’m very busy at work, so it seems like a cat would be a more suitable option. Do you have
any suggestions for getting a cat?
LLaDA

Example 2:
Prompt: 不拘一格降人才的上一句是什么？直接输出句子即可。
Answer: 我劝天公重抖擞。

B.9. More Case Studies

In this section, we present additional case studies of LLaDA 8B Instruct. First, Tab. 10 illustrates the sampling process of
semi-autoregressive remasking, while Tab. 11 depicts the sampling process for multi-turn dialogues with random remasking.
Additionally, Tab. 12 and Tab. 13 provide further examples of single-turn and multi-turn dialogues. Finally, Tab. 14 presents
examples of poem reversal completions where the LLaDA 8B Instruct model succeeds, in contrast to the failure of GPT-4o.

22
 Large Language Diffusion Models

Table 12. Single-turn Dialogue Cases of LLaDA 8B Instruct.

Daily life
User What’s a piece of advice you’ve received that you’ve found helpful?
LLaDA One helpful piece of advice I’ve received is to always be open to learning and growth.
I’ve found that being willing to explore new ideas and experiences has helped me become
more resilient and adaptable. By embracing a growth mindset, I’ve been able to identify
opportunities for improvement and development, both personally and professionally.
Daily life
User Please recommend me three famous movies
LLaDA Certainly! Here are three famous movies that have achieved widespread popularity and
critical acclaim:
1. **The Shawshank Redemption** (1994) - Inspired by a novella and directed by Frank
Darabont, this film tells the story of Andy Dufresne, a man convicted of murdering his
wife, dealing with his harsh life in Shawshank Prison and forming an unlikely friendship
with a fellow inmate.
2. **The Godfather** (1972) - Directed by Francis Ford Coppola, this film is based
on Mario Puzo’s novel of the same name. It follows the story of the Corleone family,
particularly Michael Corleone, who rises to power and takes over the family’s criminal
empire.
3. **The Empire Strikes Back** (1980) - Directed by George Lucas, this film continues
the story of a young Skywalker in the Star Wars saga. It follows Luke Skywalker as
he helps the Rebel Alliance fight against the Galactic Empire and destroy the powerful
weapon, the Death Star.
All of these films are celebrated for their cinematic achievements, powerful storytelling,
and memorable characters.
Translation
User Please help me translate into Chinese: ‘What is now proved was once only imagined’
LLaDA 现在被证明的曾经只是想象
Translation
User Please help me translate into German: ‘What is now proved was once only imagined’
LLaDA Wer heute bestätigt ist, war einst nur immaginiert.
Math
User Emma decides to run 4 sprints 5 times a week. She runs 75 meters each sprint. How many
total meters does she run a week?
LLaDA Emma runs 4 sprints each time, so she runs a total of 4 x 5 = 20 sprints a week. Each sprint
is 75 meters long, so Emma runs a total of 20 x 75 = 1500 meters a week. The final result
is 1500
Code
User I am writing python code, I have an array, please help me sort it from largest to smallest
LLaDA Certainly! You can sort an array from the largest to smallest in Python using the ‘sort()’
method or the ‘sorted()‘ function. Here’s an example:
### Using ‘sort()’ method:
my array = [3, 1, 4, 1, 5, 2]
my [Link](reverse=True)
print(my array)

23
 Large Language Diffusion Models

Table 13. Multi-turn Dialogue Cases of LLaDA 8B Instruct.

Daily life
User How do I prepare for a job interview?
LLaDA To prepare for a job interview, you should research the company and the job, practice
common interview questions, and dress appropriately.
User What are some common interview questions?
LLaDA Some common interview questions include “Tell me about yourself” and “Why do you
want to work for us”?
Daily life
User How do I start learning programming?
LLaDA To learn programming, you can start by choosing a programming language that you are
interested in. Some popular languages include Python, Java, and C++.
User Which programming language should I learn first?
LLaDA It depends on your interests and goals. Python is a great language for beginners for its
simplicity and versatility.
User How long does it take to learn Python?
LLaDA It can take anywhere from a few weeks to several months to learn Python.
User What are some good Python projects for beginners?
LLaDA Some good Python projects for beginners include building a simple calculator, a game, and
a web application.
Math
User Natalia sold clips to 48 of her friends in April, and then she sold half as many clips in May.
How many clips did Natalia sell altogether in April and May?
LLaDA Natalia sold clips to 48 friends in April. She sold half as many clips in May, so she sold
48/2 = 24 clips in May. Altogether, Natalia sold 48 + 24 = 72 clips in April and May. The
final result is 72.
User Natalia sold three times as many items in June as she did in May. How many clips did
Natalia sell altogether in these three months?
LLaDA Natalia sold clips to 48 friends in April. She sold half as many clips in May, so she sold
48/2 = 24 clips in May. Natalia sold three times as many items in June as she did in May,
so she sold 24 * 3 = 72 clips in June. Altogether, Natalia sold 48 + 24 + 72 = 144 clips in
these three months. The final result is 144.

Table 14. Poem Reversal Completion Cases where LLaDA 8B Instruct Succeeds but GPT-4o Fails.

Question1
User ‘但闻人语响’的上一句是什么?
GPT-4o “但闻人语响”的上一句是“无可奈何花落去”,这两句出自唐代词人李清照的（武陵
春，春晚: 无可奈何花落去，但闻人语响。这首词表达了作者在春天时节的孤独与
感伤，借花落和人语声的描写，反映出词人内心的无亲与失落。
LLaDA “但闻人语响”的上一句是“空山不见人”
Question2
User ‘夜静春山空’的上一句是什么?
GPT-4o “夜静春山空”的上一句是“空山新雨后”.这两句出自唐代诗人王维的（山居秋
膜）:空山新雨后，夜静春山空。这首诗描绘了春天山中雨后的寂静与空灵，
表现出自然的宁静与和谐美。
LLaDA “夜静春山空”的上一句是“人闲桂花落”

24