DKDM: Data-Free Knowledge Distillation for

Diffusion Models with Any Architecture

Qianlong Xiang1 Miao Zhang1 Yuzhang Shang2 Jianlong Wu1 Yan Yan2 Liqiang Nie1,†
1
Harbin Institute of Technology (Shenzhen) 2 Illinois Institute of Technology
arXiv:2409.03550v1 [[Link]] 5 Sep 2024

Abstract
Diffusion models (DMs) have demonstrated exceptional generative capabilities
across various areas, while they are hindered by slow inference speeds and high
computational demands during deployment. The most common way to accelerate
DMs involves reducing the number of denoising steps during generation, achieved
through faster sampling solvers or knowledge distillation (KD). In contrast to
prior approaches, we propose a novel method that transfers the capability of large
pretrained DMs to faster architectures. Specifically, we employ KD in a distinct
manner to compress DMs by distilling their generative ability into more rapid
variants. Furthermore, considering that the source data is either unaccessible or
too enormous to store for current generative models, we introduce a new paradigm
for their distillation without source data, termed Data-Free Knowledge Distillation
for Diffusion Models (DKDM). Generally, our established DKDM framework
comprises two main components: 1) a DKDM objective that uses synthetic denois-
ing data produced by pretrained DMs to optimize faster DMs without source data,
and 2) a dynamic iterative distillation method that flexibly organizes the synthesis
of denoising data, preventing it from slowing down the optimization process as
the generation is slow. To our knowledge, this is the first attempt at using KD to
distill DMs into any architecture in a data-free manner. Importantly, our DKDM is
orthogonal to most existing acceleration methods, such as denoising step reduction,
quantization and pruning. Experiments show that our DKDM is capable of deriving
2× faster DMs with performance remaining on par with the baseline. Notably, our
DKDM enables pretrained DMs to function as “datasets” for training new DMs.

1 Introduction
The advent of Diffusion Models (DMs) [Sohl-Dickstein et al., 2015, Ho et al., 2020, Song et al.,
2021] heralds a new era in the generative modeling domain, garnering widespread acclaim for their
exceptional capability in producing samples of remarkable quality [Dhariwal and Nichol, 2021,
Nichol and Dhariwal, 2021, Rombach et al., 2022]. These models have rapidly ascended to a pivotal
role across a spectrum of generative applications, notably in the fields of image, video and audio
[Yang et al., 2023, Rombach et al., 2022, Blattmann et al., 2023, Huang et al., 2023]. However, the
generation via those models is significantly slow because the sampling process involves iterative
noise estimation over thousands of time steps, which poses a challenge for practical deployment,
particularly for consumer devices [Li et al., 2024, Zhao et al., 2023].
To accelerate diffusion models, as illustrated in Figure 1 (b)&(c), existing methods can be categorized
into two pathways: reducing denoising steps and speeding up inference process of denoising networks
for each step. Compared with the standard generation of DMs as shown in Figure 1 (a), existing
methods of the first category [Lu et al., 2022a,b, Luhman and Luhman, 2021, Salimans and Ho,
2022, Song et al., 2023, Gu et al., 2023, Sauer et al., 2023, 2024] focus on reducing denoising steps
of the lengthy sampling process. The second category focuses on reducing the inference time of

†
Corresponding Author.
 𝑇 Steps Less Steps

KD

Pruned / Quantized Any Architecture

(a) Standard Generation (b) Reduce Denoising Steps (c) Speed up Inference (d) DKDM (Ours)

Figure 1: Summary of previous acceleration methods and our DKDM: (a) The standard generation of DMs is to
utilize a neural network to denoise T steps to generate an image from the pure noise. (b) Reduce denoising steps.
(c) Compress the denoising net with the same architecture. (d) Our DKDM, transfer generative knowledge from
the large one to smaller ones with any architecture.

each denoising step, through quantization [Shang et al., 2023, Li et al., 2023, He et al., 2024, Wang
et al., 2024], pruning [Fang et al., 2024, Zhang et al., 2024], and so on. However, these studies often
overlook the efficiency in denoising network architecture, a critical factor in the generation speed
of DMs, which can be improved by efficient architecture design, i.e., Neural Architecture Search
[Elsken et al., 2019, Cheng et al., 2020].
Knowledge Distillation (KD) Hinton et al. [2015] is an effective method for transferring the capabili-
ties of large, cumbersome models to smaller ones, either with similar [Hao et al., 2022] or different
architectures [Hao et al., 2024]. The conventional KD methods typically require simultaneous access
to the dataset to sample the training data, which is then used to align the student model’s behavior
with that of the teacher model [Gou et al., 2021]. However, the vast data requirements of training
generative deep neural networks, such as GPT-4 [Achiam et al., 2023] and Stable Diffusion,1 present
significant challenges for traditional KD methods, as they often necessitate direct dataset access,
which complicates data storage and accessibility. Additionally, while weights of such models are
frequently released, the corresponding datasets may remain confidential due to privacy concerns.
Recent studies have explored using generative models to eliminate the demand for the source data in
KD [Lopes et al., 2017, Nayak et al., 2019, Binici et al., 2022, Yu et al., 2023]. Motivated by these
studies, in this paper, we explore a novel KD paradigm that distills generative ability of DMs without
source data, termed Data-Free Knowledge Distillation for Diffusion Models (DKDM).
The DKDM paradigm hinges on addressing two critical challenges. The first challenge involves
optimizing a student model through the synthetic denoising data, instead of the source data. The
second challenge involves flexibly organizing the synthesis of denoising data, preventing it from
becoming the main bottleneck in slowing the optimization process, as generation of DMs is inherently
slow. For the former, the optimization objective used in traditional DMs, as described by Ho
et al. [2020], is inappropriate due to the absence of the data. To address this, we have especially
designed a DKDM objective that aligns closely with the original DM optimization objective. For
the latter challenge, the most straightforward approach is to utilize the teacher DMs to generate a
comprehensive dataset, matching the size of the source dataset employed for training the teacher.
This dataset is then used to train the student model following the standard training algorithm [Ho
et al., 2020]. However, this method becomes impractical for extremely large datasets, like those
utilized by models such as Stable Diffusion, due to excessive computational and storage demands. To
overcome this, we introduce a dynamic iterative distillation method that efficiently collects denoising
data with varying noise levels, rather than generating real-like samples. This method significantly
reduces computation and storage requirements. Importantly, our approach is complementary to most
previously established methods, as summarized in Figure 1 (b)&(c). Our experiment results validate
that our DKDM is able to derive 2× faster DMs while still able generate high-quality samples.
Additionally, our method allows pretrained DMs to act as “dataset” for training new DMs, thereby
reducing the storage demands for further research on DMs.

1
[Link]

2
2 Preliminaries on Diffusion Models
In diffusion models [Ho et al., 2020], a Markov chain is defined to add noises to data and then
diffusion models learn the reverse process to generate data from noises.


Forward Process. Given a sample x0 ∼ q x0 from the data distribution, the forward process
iteratively adds Gaussian noise for T diffusion steps2 with the predefined noise schedule (β1 , . . . , βT ):
 p 
q xt |xt−1 = N xt ; 1 − βt xt−1 , βt I ,

T

Y
q x1:T |x0 = q xt |xt−1 ,


t=1
until a completely noise xT ∼ N (0, I) is obtained. According to Ho et al. [2020], adding noise t
times sequentially to the original sample x0 to generate a noisy sample xt can be simplified to a
one-step calculation as follows:
√
q xt |x0 = N xt ; ᾱt x0 , (1 − ᾱt ) I ,



√ √
xt = ᾱt x0 + 1 − ᾱt ϵ, (1)
Qt
where αt := 1 − βt , ᾱt := s=0 αs and ϵ ∼ N (0, I).
Reverse Process. The posterior q(xt−1 |xt ) depends on the data distribution, which is tractable
conditioned on x0 :  
q xt−1 |xt , x0 = N xt−1 ; µ̃ xt , x0 , β̃t I ,





where µ̃t xt , x0 and β̃t can be calculated by:
1 − ᾱt−1
β̃t := βt ,
1 − ᾱt
√ √
ᾱt−1 βt 0 αt (1 − ᾱt−1 ) t
µ̃t xt , x0 :=


x + x. (2)
1 − ᾱt 1 − ᾱt
Since x0 in the data is not accessible during generation, a neural network parameterized by θ is used
for approximation:
pθ xt−1 |xt = N xt−1 ; µθ xt , t , Σθ xt , t I .

Optimization. To optimize this network, the variational bound on negative log likelihood E[− log pθ ]
is estimated by:
Lvlb = Ex0 ,ϵ,t DKL (q(xt−1 |xt , x0 )||pθ (xt−1 |xt ) .
 
(3)
Ho et al. [2020] found that predicting ϵ is a more efficient way when parameterizing µθ (xt , t) in
practice, which can be derived by Equation (1) and Equation (2):
 
1 βt
µθ xt , t = √ xt − √ ϵθ xt , t .



αt 1 − ᾱt
Thus, a reweighted loss function is designed as the objective to optimize Lvlb :
h
2i
Lsimple = Ex0 ,ϵ,t ϵ − ϵθ xt , t . (4)

Improvement. In original DDPMs, Lsimple offers no signal for learning Σθ (xt , t) and Ho et al.
[2020] fixed it to βt or β̃t . Nichol and Dhariwal [2021] found it to be sub-optimal and proposed to
parameterize Σθ (xt , t) as a neural network whose output v is interpolated as:
 
Σθ xt , t = exp v log βt + (1 − v) log β̃t .


(5)

To optimize Σθ (xt , t), Nichol and Dhariwal [2021] use Lvlb , in which a stop-gradient is applied to
the µθ (xt , t) because it is optimized by Lsimple . The final hybrid objective is defined as:
Lhybrid = Lsimple + λLvlb , (6)
3
where λ is used for balance between the two objectives. Guided by (6), the process of training and
sampling are shown in Algorithm 2 and Algorithm 3 in Appendix A.
2
We set T to 1,000 for all our experiments.
3
We set λ to 1 for all our experiments.

3
3 Data-Free Knowledge Distillation for Diffusion Models
In this section, we introduce a novel paradigm, termed Data-Free Knowledge Distillation for Diffusion
Models (DKDM). Section 3.1 details the DKDM paradigm, focusing on two principal challenges:
the formulation of the optimization objective and the acquisition of denoising data for distillation.
Section 3.2 describes our proposed optimization objective tailored for DKDM. Section 3.3 details our
proposed method for effective collection of denoising data.

3.1 DKDM Paradigm

The DKDM paradigm represents a novel data-free KD approach for DMs. Unlike conventional KD
methods, DKDM aims to leverages KD to transfer the generative capabilities of DMs to models with
any architecture, while eliminating the need for access to large or proprietary datasets. This approach
poses two primary challenges: 1) optimizing DMs through synthetic denoising data instead of source
data, and 2) devising methods to flexibly collect denoising data for KD as the generation is slow.
In standard training of DMs, as depicted in Fig-
Derived DMs with any architecture Pre-trained DMs
ure 2 (a), a training sample x0 ∼ D is selected
along with a timestep t ∼ [1, 1000] and ran-
dom noise ϵ ∼ N (0, I). The input xt is com-
DKDM
puted using Equation (1), and the denoising net- Train Train
Objective
work is optimized according to Equation (6) to !
𝒙 ~𝒟 𝒙 !
+ ~𝒟 "
+$ , 𝑡 ~ℬ#
𝒙
generate outputs close to ϵ. However, without 𝑡~ 1,1000 𝑡~ 1,1000
dataset access, DKDM cannot obtain training 𝝐~𝒩 0, 𝑰 𝝐~𝒩 0, 𝑰
data (xt , t, ϵ) to employ this standard method. A
straightforward approach for DKDM, termed the Dataset 𝒟 Dataset 𝒟 " Collection of
intuitive baseline and depicted in Figure 2 (b), Denoising Data ℬ#
involves using DMs pretrained on D to generate
a synthetic dataset D′ ,4 which is then used to
train new DMs with varying architectures. De-
spite its simplicity, creating D′ is time-intensive (a) Standard Training (b) Intuitive Baseline (c) Our DKDM
and impractical for large datasets.
Figure 2: Illustration of DKDM Paradigm. (a): standard
We propose an effective framework for DKDM training of DMs. (b): an intuitive baseline. (c): our
paradigm, outlined in Figure 2 (c), which incor- proposed framework.
porates a DKDM Objective (described in Sec-
tion 3.2) and a strategy for collecting denoising data Bi during optimization (detailed in Section 3.3).
This framework addresses the challenges of distillation without source dataset and reduces the costs
associated with the intuitive baseline, since the synthetic Bi requires much less computation than D′ .

3.2 DKDM Objective

Given a dataset D, the original optimization objective for a diffusion model with parameters θ involves
minimizing the KL divergence Ex0 ,ϵ,t [DKL (q(xt−1 |xt , x0 )∥pθ (xt−1 |xt ))]. Our proposed DKDM
objective encompasses two primary goals: (1) eliminating the diffusion posterior q(xt−1 |xt , x0 )
and (2) removing the diffusion prior x0 ∼ q(xt |x0 ) from the KL divergence, since they both are
dependent on x0 from the dataset D.
Eliminating the diffusion posterior q(xt−1 |xt , x0 ). In our framework, we introduce a teacher
DM with parameters θ T , trained on dataset D. This model can generate samples that conform to the
learned distribution D′ . Optimized with the objective (6), the distribution D′ within a well-learned
teacher model closely matches D. Our goal is for a student DM, parameterized by θ S , to replicate D′
instead of D′ , thereby obviating the need for q during optimization.
Specifically, the pretrained teacher model is optimized via the hybrid objective in Equation (6), which
indicates that both the KL divergence DKL (q(xt−1 |xt , x0 )∥pθT (xt−1 |xt )) and the mean squared
error Ext ,ϵ,t [∥ϵ−ϵθT (xt , t)∥2 ] are minimized. Given the similarity in distribution between the teacher
model and the dataset, we propose a DKDM objective that optimizes the student model through
minimizing DKL (pθT (xt−1 |xt ))∥pθS (xt−1 |xt ))) and Ext [∥ϵθT (xt , t) − ϵθS (xt , t)∥2 ]. Indirectly,
4
The number of samples in the synthetic dataset D′ is equal to those in original dataset D

4
the DKDM objective facilitates the minimization of DKL (q(xt−1 |xt , x0 )∥pθS (xt−1 |xt ))) and
Ex0 ,ϵ,t [∥ϵ − ϵθS (xt , t)∥2 ], despite the inaccessibility of the posterior. Consequently, we propose the
DKDM objective as follows:
LDKDM = L′simple + λL′vlb , (7)
where L′simple guides the learning of µθS and L′vlb optimizes ΣθS , as defined in following equations:

L′simple = Ex0 ,ϵ,t ∥ϵθT (xt , t) − ϵθS (xt , t)∥2 ,

 

L′vlb = Ex0 ,ϵ,t DKL (pθT (xt−1 |xt )∥pθS (xt−1 |xt ) ,
 

where q(xt−1 |xt , x0 ) is eliminated whereas the term xt ∼ q(xt |x0 ) remains to be removed.
Removing the diffusion prior q(xt |x0 ). Considering the generative ability of the teacher model,
we utilize it to generate x̂t as a substitute for xt ∼ q(xt |x0 ). We define a reverse diffusion step
x̂t−1 ∼ pθT (x̂t−1 |xt ) through the equation x̂t−1 = gθT (xt , t). Next,we represent a sequence of
t reverse diffusion steps starting from T as GθT (t). Note that GθT (0) = ϵ where ϵ ∼ N (0, I).
For instance, GθT (2) yields x̂T −2 = gθT (gθT (ϵ, T ), T − 1). Consequently, x̂t is obtained by
x̂t = GθT (T − t) and the objective L′simple and L′vlb are reformulated as follows:

L′simple = Ex̂t ,t ϵθT (x̂t , t) − ϵθS (x̂t , t)∥2 ,

 
(8)

L′vlb = Ex̂t ,t DKL (pθT (x̂t−1 |x̂t )∥pθS (x̂t−1 |x̂t ) .

 
(9)

By this formulation, the necessity of x0 in LDKDM is removed by naturally leveraging the generative
ability of the teacher model. Optimized by the proposed LDKDM , the student progressively learns the
entire reverse diffusion process from the teacher model without reliance on the source datasets.
However, the removal of the diffusion posterior and prior in the DKDM objective introduces a
significant bottleneck, resulting in notably slow learning rates. As depicted in Figure 2 (a), standard
training for DMs enable straightforward acquisition of noisy samples xtii at an arbitrary diffusion step
t ∼ [1, T ] using Equation (1). These samples are compiled into a training data batch Bj = {xtii },
with j representing the training iteration. Conversely, our DKDM objective requires obtaining a
noisy sample x̂ti = GθT (T − ti ) through T − ti denoising steps. Consequently, considering the
denoising steps as the primary computational expense, the worst-case time complexity of assembling
a denoising data batch B̂j = {x̂tii } for distillation is O(T b), where b denotes the batch size. This
complexity significantly hinders the optimization process. To address this issue, we introduce a
method called dynamic iterative distillation, detailed in Section 3.3.

3.3 Efficient Collection of Denoising Data

In this section, we present our efficient strategy for gathering denoising data for distillation, illustrated
in Figure 3. We begin by introducing a basic iterative distillation method that allows the student model
to learn from the teacher model at each denoising step, instead of requiring the teacher to denoise
multiple times within every training iteration to create a batch of noisy samples for the student to learn
once. Subsequently, to enhance the diversity of noise levels within the batch samples, we develop an
advanced method termed shuffled iterative distillation, which allows the student to learn denoising
patterns across varying time steps. Lastly, we refine our approach to dynamic iterative distillation,
significantly augmenting the diversity of data in the denoising batch. This adaptation ensures that the
student model acquires knowledge from a broader array of samples over time, avoiding repetitive
learning from identical samples.
Iterative Distillation. We introduce a method called iterative distillation, which closely aligns the
optimization process with the generation procedure. In this approach, the teacher model consistently
denoises, while the student model continuously learns from this denoising. Each output from the
teacher’s denoising step is incorporated into some batch for optimization, ensuring the student model
learns from every output. Specifically, during each training iteration, the teacher performs gθT (xt , t),
which is a single-step denoising, instead of GθT (t), which would involve t-step denoising. Initially,
a batch B̂1 = {x̂Ti } is formed from a set of sampled noises x̂Ti ∼ N (0, I). After one step of
distillation, the batch B̂2 = {x̂Ti −1 } is used for training. This process is iterated until B̂T = {x̂1i }
is reached, indicating that the batch has nearly become real samples with no noise. The cycle then

5
 !"% ~𝒩(0, 𝑰)
𝒙 Batch set Batch set
Iterative Distillation
!!"
𝒙 !!!
𝒙
& &
!!!
𝒙
Batch Batch

!"#
𝒙 !#"
𝒙
&
!#"
𝒙
&
!#"
𝒙
& '!
!#"
𝒙
& '!

Shuffle Denoise Random Denoise Substitution

& &
!"$
𝒙 !$#
𝒙 !$#
𝒙 !$#
𝒙
& '! !"$
𝒙
Selection

!"$(!
𝒙 Teacher &#$!
!$(!
𝒙
&#$!
!$(!
𝒙 𝒙
&#$!
!$(!
'! &#$!
!$(!
𝒙
'!

Train
!")
𝒙 !)%
𝒙
& &
!)%
𝒙
Student

Figure 3: Dynamic Iterative Distillation: An enlarged batch set is initially constructed by sampling from a
Gaussian distribution. Subsequently, shuffle denoise is applied, wherein each sample is denoised random times.
A batch is then randomly selected from this enlarged set for optimizing the student model with the denoised
results substituting for their counterparts in the batch set. This process is repeated iteratively.

restarts with the resampling of noise to form a new batch B̂T +1 = {x̂Ti }. This method allows the
teacher model to provide an endless stream of data for distillation. To further improve the diversity of
the synthetic batch B̂j = {x̂tii }, we investigate from the perspectives of noise level ti and sample x̂i .
Shuffled Iterative Distillation. Unlike standard training, the t values in an iterative distillation batch
remain the same and do not follow a uniform distribution, resulting in significant instability during
distillation. To mitigate this issue, we integrated a method termed shuffle denoise into our iterative
distillation. Initially, a batch B̂0s = {x̂Ti } is sampled from a Gaussian distribution. Subsequently,
each sample undergoes random denoising steps, resulting in B̂1s = {x̂tii }, with ti following a uniform
distribution. This batch, B̂1s , initiates the iterative distillation process. By ensuring diversity in the ti
values within the batch, this method balances the impact of different t values during distillation.
Dynamic Iterative Distillation. There is a notable distinction between standard training and iterative
distillation regarding the flexibility in batch composition. Consider two samples, x̂1 and x̂2 , within a
batch without differentiating their noise level. During standard training, the pairing of x̂1 and x̂2 is
entirely random. Conversely, in iterative distillation, batches containing x̂1 almost always include x̂2 .
This departure from the principle of independent and identically distributed samples in a batch can
potentially diminish the model’s generalization ability.
To better align the distribution of the denoising
Algorithm 1 Dynamic Iterative Distillation
data with that of the standard training batch,
we propose a method named dynamic itera- Require: B̂0+ = {x̂Ti }
tive distillation. As shown in Figure 3 , this 1: Get B̂1+ = {x̂tii } with shuffle denoise, j = 0
method employs shuffle denoise to construct 2: repeat
an enlarged batch set B̂1+ = {x̂tii }, where size 3: j =j+1
|B̂j+ | = ρT |B̂js |, where ρ is a scaling factor. Dur- 4: get B̂js from B̂j+ through random selection
s
ing distillation, a subset B̂j is sampled from B̂j + 5: compute L⋆simple using (11)
through random selection for optimization. The 6: compute L⋆vlb using (5) and (12)
one-step denoised samples replace their coun- 7: take a gradient descent step on ∇θ LDKDM
+ +
terparts in B̂j+1 . This method only has a time 8: update B̂j+1
complexity of O(b) and significantly improves 9: until converged
distillation performance. The final DKDM ob-
jective is defined as:
L⋆DKDM = L⋆simple + λL⋆vlb , (10)
⋆ t t 2
 
Lsimple = E(x̂t ,t)∼B̂+ ϵθT (x̂ , t) − ϵθS (x̂ , t)∥ , (11)
⋆ t−1 t t−1 t
 
Lvlb = E(x̂t ,t)∼B̂+ DKL (pθT (x̂ |x̂ )∥pθS (x̂ |x̂ ) , (12)
where x̂t and t is produced by our proposed dynamic iterative distillation. The complete algorithm is
detailed in Algorithm 1.

6
Table 1: Performance of DMs with the same architecture as the teacher and those with 2× faster architecture
derived by baseline and our DKDM on three datasets.
CIFAR10 32 x 32 CelebA 64 x 64 ImageNet 32 x 32
IS↑ FID↓ sFID↓ IS↑ FID↓ sFID↓ IS↑ FID↓ sFID↓
Teacher 9.52 4.45 7.09 3.08 4.43 6.10 13.63 4.67 4.03
Same Architecture Student
Baseline 8.69 9.64 11.64 2.93 6.66 8.71 12.19 8.14 6.03
DKDM 8.69 6.85 8.01 2.93 6.03 8.26 12.20 6.97 5.64
2 ×-Speed, 1 /4 -Size Architecture Student
Baseline 8.28 12.06 13.23 2.87 7.62 10.17 10.50 12.32 8.90
DKDM 8.60 9.56 11.77 2.91 7.07 8.78 10.50 11.33 4.80

Table 2: Params and speed of architectures we test.

CIFAR10 32x32 CelebA 64x64 ImageNet 32x32
Params Speed Params Speed Params Speed
Teacher Architecture 57M 5.68s 295M 45.08s 57M 5.68s
Faster Architecture 14M 2.84s 57M 20.15s 14M 2.84s

4 Experiments

This section details extensive experiments that demonstrate the effectiveness of our proposed DKDM.
In Section 4.1, we establish appropriate metrics and baselines for evaluation. Section 4.2 compares
the performance of baselines and our DKDM with different architectures. Additionally, we show that
our DKDM can be combined with other methods to accelerate DMs. Finally, Section 4.3 describes
an ablation study that validates the effectiveness of our proposed dynamic iterative distillation.

4.1 Experiment Setting

Datasets and teacher diffusion models. Our DKDM paradigm inherently eliminates the necessity
for datasets. However, the pretrained teacher models employed are trained on specific datasets. We
utilize three distinct pretrained DMs as teacher models, following the configurations introduced by
Ning et al. [2023], and these models are all based on convolutional architecture. These models have
been pretrained on CIFAR10 at a resolution of 32 × 32 [Krizhevsky et al., 2009], CelebA at 64 × 64
[Liu et al., 2015] and ImageNet at 32 × 32 Chrabaszcz et al. [2017].
Metrics. The distance between the generated samples and the reference samples can be estimated
by the Fréchet Inception Distance (FID) score [Heusel et al., 2017]. In our experiments, we utilize
the FID score as the primary metric for evaluation. Additionally, we report sFID [Nash et al., 2021],
Inception Score (IS) Salimans et al. [2016] as secondary metrics. Following previous work [Ho
et al., 2020, Nichol and Dhariwal, 2021, Ning et al., 2023], we generate 50K samples for DMs and
we use the full training set in the corresponding dataset to compute the metrics. Without additional
contextual states, all the samples are generated through 50 Improved DDPM sampling steps [Nichol
and Dhariwal, 2021] and the speed is measured by the average time taken to generate 256 images on
a single NVIDIA A100 GPU. All of our metrics are calculated by ADM TensorFlow evaluation suite
[Dhariwal and Nichol, 2021].
Baseline. As the DKDM is a new paradigm proposed in this paper, previous methods are not suitable
to serve as baselines. Therefore, in the data-free scenario, we take the intuitive baseline depicted
in Figure 2 (a) as the baseline. Specifically, the teacher model consumes a lot of time to generate
a substantial number of high-quality samples, equivalent in quantity to the source dataset, through
1,000 DDPM denoising steps. These samples then serve as the synthetic dataset (D′ ) for the training
of randomly initialized student models, following the standard training (Algorithm 2 in Appendix A).
We use the performance obtained from this method as our baseline for comparative analysis.

7
4.2 Main Results 60 8 Speed (s)

Speed (s)

Speed (s)
50 6
40 4
Effectiveness. Table 1 shows the performance com- 30 2
parison of our DKDM and baselines. Our DKDM 57M 44M 32M 24M 14M
0
50 25 10
consistently outperforms the baselines, demonstrat- Baseline DKDM Baseline DKDM
ing superior generative quality when maintaining 12 Teacher
16
identical architectures for the derived DMs. This 10

FID

FID
12
performance validates the efficacy of our proposed 8
8
DKDM objective and dynamic iterative distillation 6 4
approach. The improvement over baselines is at- 57M 44M 32M 24M 14M 50 25 10
tributed to the complexity of the reverse diffusion Model Size DDIM Steps
process, which baselines struggle to learn, whereas (a) Different Arch. (b) DDIM
the knowledge from pretrained teacher models is eas-
ier to learn, highlighting the advantage of our DKDM. Figure 4: Performance and speed of DMs on
Additionally, DKDM facilitates the distillation of CIFAR10. (a) shows the result of models across
different architectures and (b) shows result of com-
generative capabilities into faster and more compact bining DDIM with DMs derived by our DKDM.
models, as evidenced by the 2× faster architectures
evaluated. The parameter count and generative speed
are detailed in Table 2, with further information on hyperparameters and network architecture avail-
able in Appendix B.1. Appendix C includes some exemplary generated samples, illustrating that the
student DMs, derived through DKDM, are capable of producing high-quality images. Nevertheless, a
limitation noted in Table 1 is that the performance of these student DMs falls behind their teacher
counterparts, which will be discussed further in Section 5.
We further evaluated the performance of these student models across a diverse range of architectures.
Specifically, we tested five different model sizes by directly specifying the architecture, bypassing
complex methods like neural architecture search. Both the teacher and student models employ Con-
volutional Neural Networks (CNNs) and the results are shown in Figure 4(a). Detailed descriptions
of these architectures are available in AppendixB.2. Typically, we distilled a 14M model from a 57M
teacher model, maintaining competitive performance and doubling the generation speed. Additionally,
the 44M and 33M student models demonstrated similar speeds, suggesting that DKDM could benefit
from integration with efficient architectural design techniques to further enhance the speed and quality
of DMs. This aspect, however, is beyond our current scope and is designated for future research.
Cross-Architecture Distillation. Our DKDM transcends spe- Table 3: FID scores on CIFAR10 for
cific model architectures, enabling the distillation of genera- cross-architecture distillation. Here the
tive capabilities from CNN-based DMs to Vision Transform- FID score of the CNN teacher is 4.45
ers (ViT) and vice versa. We utilized DiT [Peebles and Xie, and that of the ViT teacher is 11.30.
2023] for ViT-based DMs to further affirm the superiority of
our approach. Detailed structural descriptions are available CNN ViT
in Appendix B.3. For experimental purposes, we pretrained Teacher Teacher
a small ViT-based DM to serve as the teacher. As shown in CNN Student
Table 3, DKDM effectively facilitates cross-architecture distil- Baseline 9.64 44.62
lation, yielding superior performance compared to baselines. DKDM 6.85 13.17
Additionally, our results suggest that CNNs are more effective ViT Student
as compressed DMs than ViTs. Baseline 17.11 63.15
Combination with Orthogonal Methods. The DMs derived DKDM 17.11 17.86
by our DKDM are compatible with various orthogonal methods,
such as denoising step reduction, quantization, pruning and so
on. Here we conducted experiments to integrate the DDIM method, which reduces denoising steps,
as illustrated in Figure 4 (b). This integration demonstrates that DDIM can further accelerate our
derived student models, although with some performance trade-offs.

4.3 Ablation Study

To validate our designed paradigm DKDM, we tested FID score of our progressively designed
methods, including iterative, shuffled iterative, and dynamic iterative distillation, over 200K training
iterations. For the dynamic iterative distillation, the parameter ρ was set to 0.4. The results, shown
in Figure 5 (a), demonstrate that our dynamic iterative distillation strategy not only converges more
rapidly but also delivers superior performance. The convergence curve for our method closely

8
 DKDM Objective
Baseline Iterative Distillation 0.010 0.020 0.030
Shuffled Iterative Distillation 0.050 0.100 0.200
Dynamic Iterative Distillation 0.300 0.400 Baseline
300 Student 𝜽!
FID Linear Fitting
50
12
25 Distribution of 𝜽 " : 𝒟 #
40

FID − 𝒟
FID

FID
200 30

20
10 10
10
100 100 150 5 15

FID − 𝒟 !
8

0 50 100 150 200 10 15 20

Training Iterations (k) Training Iterations (k)
Real Dataset: 𝒟 Teacher 𝜽 "
(a) Ablation (b) Effect of ρ
Figure 6: The DKDM paradigm learns distri-
Figure 5: FID scores on CIFAR10. (a): Ablation on dynamic bution D′ integrated in teacher DMs, which
iterative distillation with ρ = 0.4. (b): Effect of different ρ. may deviate from the original dataset D.

matches that of the baseline, which confirms the effectiveness of the DKDM objective in alignment
with the standard optimization objective (6).
Further experiments explored the effects of varying ρ on the performance of dynamic iterative
distillation. As illustrated in Figure 5 (b), higher ρ values enhance the distillation process up to a
point, beyond which performance gains diminish. This outcome supports our hypothesis that dynamic
iterative distillation enhances batch construction flexibility, thereby improving distillation efficiency.
Beyond a certain level of flexibility, increasing ρ does not significantly benefit the distillation process.
Further stability analysis of our dynamic iterative distillation is available in Appendix D.

5 Discussion and Future Work

The primary concept of our proposed DKDM Table 4: Results on CIFAR10 calculated over the
paradigm is illustrated in Figure 6. In this paradigm, source D and that over distribution D′ of θT .
the teacher DM θ T , is trained on a real dataset D,
which follows the distribution D′ . There are two FID sFID FID′ sFID′
critical relationships between D and D′ . First, the Teacher 4.45 7.09 2.09 3.92
distribution D′ of a well-trained teacher closely ap- Baseline 9.64 11.64 4.13 5.19
proximates D. Second, the FID scores, when com- DKDM 6.85 8.01 2.60 4.14
puted using D as a reference, correlate with those
using D′ , as demonstrated by the linear fitting in Figure 6. This correlation underpins the effective-
ness of the DKDM paradigm. By transferring the distribution D′ from the teacher DM to a lighter
student DM, DKDM enables the student to generate data whose distribution closely approximates D.
However, in practice, there is invariably some discrepancy between D and D′ , limiting the perfor-
mance of the student model. We report these scores, denoted as FID′ and sFID′ , calculated over the
distribution D′ instead of D in Table 4. The results indicate that the FID′ and sFID′ scores of the
student closely mirror those of the teacher, suggesting effective optimization. Nevertheless, these
scores are inferior to those of the teacher, primarily due to the gap between D and D′ . A potential
solution to enhance DKDM involves improving the generative capabilities of the teacher, which we
leave as a direction for future work.

6 Conclusions

In this paper, we introduce DKDM, a novel paradigm designed to efficiently distill the generative
abilities of pretrained diffusion models into more compact and faster models with any architecture.
Our experiments demonstrate the effectiveness of DKDM across three datasets, showcasing its ability
to compress models to various sizes and architectures. A key advantage of our method is its ability to
perform efficient distillation without direct data access, significantly facilitating ongoing research
and development in the field. Moreover, our DKDM is compatible with most existing methods for
accelerating diffusion models and can be integrated with them.

9
References
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman,
Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report.
arXiv preprint arXiv:2303.08774, 2023.
Kuluhan Binici, Shivam Aggarwal, Nam Trung Pham, Karianto Leman, and Tulika Mitra. Robust
and resource-efficient data-free knowledge distillation by generative pseudo replay. In Proceedings
of the AAAI Conference on Artificial Intelligence, volume 36, pages 6089–6096, 2022.
Andreas Blattmann, Robin Rombach, Huan Ling, Tim Dockhorn, Seung Wook Kim, Sanja Fidler, and
Karsten Kreis. Align your latents: High-resolution video synthesis with latent diffusion models. In
Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages
22563–22575, 2023.
Xuelian Cheng, Yiran Zhong, Mehrtash Harandi, Yuchao Dai, Xiaojun Chang, Hongdong Li, Tom
Drummond, and Zongyuan Ge. Hierarchical neural architecture search for deep stereo matching.
Advances in neural information processing systems, 33:22158–22169, 2020.
Patryk Chrabaszcz, Ilya Loshchilov, and Frank Hutter. A downsampled variant of imagenet as an
alternative to the cifar datasets. arXiv preprint arXiv:1707.08819, 2017.
Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. Advances
in Neural Information Processing Systems, 34:8780–8794, 2021.
Thomas Elsken, Jan Hendrik Metzen, and Frank Hutter. Neural architecture search: A survey. Journal
of Machine Learning Research, 20(55):1–21, 2019.
Gongfan Fang, Xinyin Ma, and Xinchao Wang. Structural pruning for diffusion models. Advances in
neural information processing systems, 36, 2024.
Jianping Gou, Baosheng Yu, Stephen J Maybank, and Dacheng Tao. Knowledge distillation: A
survey. International Journal of Computer Vision, 129(6):1789–1819, 2021.
Jiatao Gu, Shuangfei Zhai, Yizhe Zhang, Lingjie Liu, and Josh Susskind. Boot: Data-free distillation
of denoising diffusion models with bootstrapping. arXiv preprint arXiv:2306.05544, 2023.
Zhiwei Hao, Jianyuan Guo, Ding Jia, Kai Han, Yehui Tang, Chao Zhang, Han Hu, and Yunhe Wang.
Learning efficient vision transformers via fine-grained manifold distillation. Advances in Neural
Information Processing Systems, 35:9164–9175, 2022.
Zhiwei Hao, Jianyuan Guo, Kai Han, Yehui Tang, Han Hu, Yunhe Wang, and Chang Xu. One-for-all:
Bridge the gap between heterogeneous architectures in knowledge distillation. Advances in Neural
Information Processing Systems, 36, 2024.
Yefei He, Luping Liu, Jing Liu, Weijia Wu, Hong Zhou, and Bohan Zhuang. Ptqd: Accurate post-
training quantization for diffusion models. Advances in Neural Information Processing Systems,
36, 2024.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans
trained by a two time-scale update rule converge to a local nash equilibrium. Advances in Neural
Information Processing Systems, 30, 2017.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network. arXiv
preprint arXiv:1503.02531, 2015.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in
Neural Information Processing Systems, 33:6840–6851, 2020.
Qingqing Huang, Daniel S Park, Tao Wang, Timo I Denk, Andy Ly, Nanxin Chen, Zhengdong Zhang,
Zhishuai Zhang, Jiahui Yu, Christian Frank, et al. Noise2music: Text-conditioned music generation
with diffusion models. arXiv preprint arXiv:2302.03917, 2023.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.

10
Xiuyu Li, Yijiang Liu, Long Lian, Huanrui Yang, Zhen Dong, Daniel Kang, Shanghang Zhang,
and Kurt Keutzer. Q-diffusion: Quantizing diffusion models. In Proceedings of the IEEE/CVF
International Conference on Computer Vision, pages 17535–17545, 2023.
Yanyu Li, Huan Wang, Qing Jin, Ju Hu, Pavlo Chemerys, Yun Fu, Yanzhi Wang, Sergey Tulyakov,
and Jian Ren. Snapfusion: Text-to-image diffusion model on mobile devices within two seconds.
Advances in Neural Information Processing Systems, 36, 2024.
Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild. In
Proceedings of the IEEE international conference on computer vision, pages 3730–3738, 2015.
Raphael Gontijo Lopes, Stefano Fenu, and Thad Starner. Data-free knowledge distillation for deep
neural networks. arXiv preprint arXiv:1710.07535, 2017.
Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu. Dpm-solver++: Fast
solver for guided sampling of diffusion probabilistic models. arXiv preprint arXiv:2211.01095,
2022a.
Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu. Dpm-solver: A fast
ODE solver for diffusion probabilistic model sampling in around 10 steps. In Advances in Neural
Information Processing Systems, 2022b.
Eric Luhman and Troy Luhman. Knowledge distillation in iterative generative models for improved
sampling speed. arXiv preprint arXiv:2101.02388, 2021.
Charlie Nash, Jacob Menick, Sander Dieleman, and Peter W. Battaglia. Generating images with
sparse representations. In International Conference on Machine Learning, 2021.
Gaurav Kumar Nayak, Konda Reddy Mopuri, Vaisakh Shaj, Venkatesh Babu Radhakrishnan, and
Anirban Chakraborty. Zero-shot knowledge distillation in deep networks. In International
Conference on Machine Learning, pages 4743–4751. PMLR, 2019.
Alexander Quinn Nichol and Prafulla Dhariwal. Improved denoising diffusion probabilistic models.
In International Conference on Machine Learning, 2021.
Mang Ning, Enver Sangineto, Angelo Porrello, Simone Calderara, and Rita Cucchiara. Input
perturbation reduces exposure bias in diffusion models. In International Conference on Machine
Learning, 2023.
William Peebles and Saining Xie. Scalable diffusion models with transformers. In International
Conference on Computer Vision, pages 4172–4182, 2023.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-
resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF confer-
ence on computer vision and pattern recognition, pages 10684–10695, 2022.
Tim Salimans and Jonathan Ho. Progressive distillation for fast sampling of diffusion models. In
International Conference on Learning Representations, 2022.
Tim Salimans, Ian Goodfellow, Wojciech Zaremba, Vicki Cheung, Alec Radford, Xi Chen, and
Xi Chen. Improved techniques for training gans. In Advances in Neural Information Processing
Systems, volume 29, 2016.
Axel Sauer, Dominik Lorenz, Andreas Blattmann, and Robin Rombach. Adversarial diffusion
distillation. arXiv preprint arXiv:2311.17042, 2023.
Axel Sauer, Frederic Boesel, Tim Dockhorn, Andreas Blattmann, Patrick Esser, and Robin Rombach.
Fast high-resolution image synthesis with latent adversarial diffusion distillation. arXiv preprint
arXiv:2403.12015, 2024.
Yuzhang Shang, Zhihang Yuan, Bin Xie, Bingzhe Wu, and Yan Yan. Post-training quantization on
diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern
Recognition, pages 1972–1981, 2023.

11
Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised
learning using nonequilibrium thermodynamics. In International Conference on Machine Learning,
2015.
Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben
Poole. Score-based generative modeling through stochastic differential equations. In International
Conference on Learning Representations, 2021.
Yang Song, Prafulla Dhariwal, Mark Chen, and Ilya Sutskever. Consistency models. In International
Conference on Machine Learning, 2023.
Haoxuan Wang, Yuzhang Shang, Zhihang Yuan, Junyi Wu, and Yan Yan. Quest: Low-bit diffusion
model quantization via efficient selective finetuning. arXiv preprint arXiv:2402.03666, 2024.
Ling Yang, Zhilong Zhang, Yang Song, Shenda Hong, Runsheng Xu, Yue Zhao, Wentao Zhang,
Bin Cui, and Ming-Hsuan Yang. Diffusion models: A comprehensive survey of methods and
applications. ACM Computing Surveys, 56(4):1–39, 2023.
Shikang Yu, Jiachen Chen, Hu Han, and Shuqiang Jiang. Data-free knowledge distillation via
feature exchange and activation region constraint. In Proceedings of the IEEE/CVF Conference on
Computer Vision and Pattern Recognition, pages 24266–24275, 2023.
Dingkun Zhang, Sijia Li, Chen Chen, Qingsong Xie, and Haonan Lu. Laptop-diff: Layer pruning and
normalized distillation for compressing diffusion models. arXiv preprint arXiv:2404.11098, 2024.
Yang Zhao, Yanwu Xu, Zhisheng Xiao, and Tingbo Hou. Mobilediffusion: Subsecond text-to-image
generation on mobile devices. arXiv preprint arXiv:2311.16567, 2023.

12
A Training and Sampling of DDPMs
In this section, we present the algorithms for training and sampling from standard DDPMs. The
specific details have been previously introduced in Section 2.

Algorithm 2 Diffusion Models Training

1: repeat
2: x0 ∼ q(x0 ), t ∼ [1, ..., T ], ϵ ∼ N (0, I)
3: obtain the noisy sample xt using (1)
4: compute Lsimple using (4)
5: compute Lvlb using (3) and (5)
6: take a gradient descent step on ∇θ Lhybrid
7: until converged

Algorithm 3 Diffusion Models Sampling

T
1: x̂ ∼ N (0, I)
2: for t := T, ..., 1 do
3: if t > 1 then z ∼ N (0, I), else z = 0 q
4: x̂t−1 = √1αt (x̂t − √1−α t
ϵ (x̂t , t)) + Σθ (x̂t , t)z
1−ᾱt θ
5: end for
0
6: return x̂

B Hyperparameters
B.1 Hyperparameters for DKDM

For experiments of both DKDM and baseline, we use the hyperparameters specified by Ning et al.
[2023], which are also in line with those adopted by Dhariwal and Nichol [2021], as reported in
Table 5. Settings of our training process are basically the same with Dhariwal and Nichol [2021] and
Ning et al. [2023], including mixed precision training, EMA and so on. All the models are trained on
8 NVIDIA A100 GPUs (with 40G memory).

Table 5: Hyperparameters for main results. “T. Arch.” refers to the student model whose architecture mirrors
that of the teacher. “Faster Arch.” denotes the smaller architecture we evaluated.
CIFAR10 32x32 CelebA 64x64 ImageNet 32x32
T. Arch. Faster Arch. T. Arch. Faster Arch. T. Arch. Faster Arch.
Model size 57M 14M 295M 57M 57M 14M
Diffusion steps 1,000 1,000 1000 1000 1000 1,000
Noise schedule cosine cosine cosine cosine cosine cosine
Channels 128 64 192 96 128 64
Residual blocks 3 3 3 2 3 3
Channels multiple 1,2,2,2 1,2,2,2 1,2,3,4 1,2,3,4 1,2,2,2 1,2,2,2
Heads channels 32 32 64 32 32 32
Attention resolution 16,8 16,8 32,16,8 32,16,8 16,8 16,8
BigGAN up/downsample True True True True True True
Dropout 0.3 0.3 0.1 0.1 0.3 0.3
Batch size 128 128 256 256 512 512
Distillation Iterations 200K 200K 100K 100K 500K 500K
Learning rate 1e-4 1e-4 1e-4 1e-4 1e-4 1e-4

B.2 Hyperparameters for Model Compression

Table 6 shows the architectures of different students trained in Section 4.2 for model compression.

13
 Table 6: Hyperparameters for CNN diffusion model compression.

Model Size 57M 44M 32M 24M 14M

Diffusion steps 1,000 1,000 1,000 1,000 1,000
Noise schedule cosine cosine cosine cosine cosine
Channels 128 128 96 96 64
Residual blocks 3 2 3 2 3
Channels multiple 1, 2, 2, 2 1, 2, 2, 2 1, 2, 2, 2 1, 2, 2, 2 1, 2, 2, 2
Heads channels 32 32 32 32 32
Attention resolution 16, 8 16, 8 16, 8 16, 8 16, 8
BigGAN up/downsample True True True True True
Dropout 0.3 0.3 0.3 0.3 0.3
Batch size 128 128 128 128 128
Distillation Iterations 200K 200K 200K 200K 200K
Learning rate 1e-4 1e-4 1e-4 1e-4 1e-4

B.3 Hyperparameters for Cross-Architecture Distillation

Similar to the configuration used by Peebles and Xie [2023], the hyperparameters employed for the
ViT-based diffusion model in Section 4.2 are presented in Table 7. This particular configuration was
chosen due to the characteristic of ViT-based diffusion models generally requiring more time for
image generation compared to their CNN-based counterparts. To illustrate this point, when generating
2,500 images on single A100 40GB GPU with 50 Improved DDPM steps, it takes approximately 57
seconds for a 57M CNN diffusion model, whereas a 19M ViT diffusion model requires 66 seconds.
Our final choice of this configuration was driven by the aim to achieve fairness in our experimentation
and analysis.

Table 7: Hyperparameters for ViT-based diffusion models.

Model Size Layers N Patch Size Hidden size d Heads

19M 7 2 384 6

C Exemplary Generated Samples

In this section, we demonstrate some exepmlary generated samples of our derived 2×-speed and
1/4-size student DMs.

(a) CIFAR10 32x32 (b) CelebA 64x64 (c) ImageNet 32x32

Figure 7: Exemplary samples generated by our distilled students.

14
D Analysis: Random Discard

During our exploration, we discovered that the utilization of the Random Discard technique proves
to be a straightforward yet highly effective approach for enhancing the distillation process. The idea
behind it involves the random elimination of some batch of noisy samples generated by the teacher
model during the iterative distillation. For instance, in iterative distillation during the initial five
training iterations, batches B̂1 , B̂3 , B̂4 may be discarded, while B̂2 , B̂5 are utilized for the student’s
learning. We present an analysis of the impact of random discarding in our devised methodologies.
Specifically, we introduce the parameter p to denote the probability of discarding certain noisy
samples. Subsequently, we apply varying discard probabilities to the iterative distillation, shuffled
iterative distillation, and dynamic iterative distillation, and assess their respective performance
alterations over a training duration of 200k iterations. The outcomes are presented in Figure 8.
It is noteworthy that both iterative distillation and shuffled iterative distillation face limitations in
constructing flexible batches, where random discard emerges as a noteworthy solution to enhance
their efficacy. Conversely, for dynamic iterative distillation, when ρ attains a sufficiently large value,
it becomes apparent that random discard fails to confer additional advantages. This observation
underscores the inherent stability of our dynamic iterative distillation method and we ultimately
omitted random discard from the final implementation. This is beneficial because of its inefficiency
in requiring the teacher model to prepare a larger number of noisy samples.

70 40

60 35

50 30

40 25
FID

FID

30 20

20 15

10 10

0 5
5 10 15 20 5 10 15 20
Training Iterations (k) Training Iterations (k)
(a) Iterative Distillation (b) Shuffled iterative distillation

18 18

16 16

14 14
FID
FID

12 12

10 10

8 8

6 6
5 10 15 20 5 10 15 20
Training Iterations (k) Training Iterations (k)

(c) Dynamic iterative distillation with ρ = 0.05 (d) Dynamic iterative distillation with ρ = 0.4
Figure 8: Effect of random discard on our iterative distillation, shuffled iterative distillation and dynamic
iterative distillation with different ρ. Random discard can not improve our dynamic iterative distillation with
ρ = 0.4 and thus removed from our framework finally.

15
E Social Impact
As an effective method for compressing and accelerating diffusion models, we believe that DKDM
has the potential to significantly reduce the deployment costs associated with these models, thereby
facilitating more widespread use of diffusion models for generating desired content. However, it
is imperative to acknowledge that, as generative models, diffusion models, while offering creative
applications across various scenarios, may also engender consequences such as the production of
dangerous or biased content.
Our DKDM is capable of mimicking the generative capabilities of a wide array of existing diffusion
models without accessing the source datasets, which leads to derived models inheriting the flaws and
limitations of these pre-existing models. For instance, if the training data of a pre-trained diffusion
model contains sensitive or personal information collected without explicit consent, derived models
may still risk leaking this data. Consequently, the potential societal harms of our approach primarily
hinge on the negative impacts brought about by the existing diffusion models themselves. Addressing
how to mitigate the adverse effects inherent in diffusion models remains a critical area of research.

16