Diffusion Models: Tutorial and Survey

Benyamin Ghojogh, Ali Ghodsi

To cite this version:

Benyamin Ghojogh, Ali Ghodsi. Diffusion Models: Tutorial and Survey. 2024. �hal-04642649�

HAL Id: hal-04642649

[Link]
Preprint submitted on 10 Jul 2024

HAL is a multi-disciplinary open access L’archive ouverte pluridisciplinaire HAL, est

archive for the deposit and dissemination of sci- destinée au dépôt et à la diffusion de documents
entific research documents, whether they are pub- scientifiques de niveau recherche, publiés ou non,
lished or not. The documents may come from émanant des établissements d’enseignement et de
teaching and research institutions in France or recherche français ou étrangers, des laboratoires
abroad, or from public or private research centers. publics ou privés.

HAL Authorization
 To appear as a part of an upcoming textbook on deep learning.

Diffusion Models: Tutorial and Survey

Benyamin Ghojogh, Ali Ghodsi {BGHOJOGH , ALI . GHODSI}@ UWATERLOO . CA

Waterloo, Ontario, Canada

Abstract
Diffusion models are a family of generative mod-
els which work based on a Markovian process. In
their forward process, they gradually add noise
to data until it becomes a complete noise. In the
backward process, the data are gradually gener-
ated out of noise. In this tutorial paper, the De-
noising Diffusion Probabilistic Model (DDPM)
is fully explained. Detailed simplification of the
Figure 1. The forward and backward processes of the diffusion
variational lower bound of its likelihood, param-
model. The credit of the used images is for Arash Vahdat.
eters of the distributions, and the loss function of
the diffusion model are discussed. Some mod-
ifications to the original DDPM, including non-
fixed covariance matrix, reducing the gradient
noise, improving the noise schedule, and non-
standard Gaussian noise distribution, and condi-
tional diffusion model are introduced. Finally,
continuous noise schedule by Stochastic Differ-
Figure 2. The diffusion model as an autoencoder.
ential Equation (SDE), where the noise schedule
is in a continuous domain, is explained.
are learned.
1. Introduction
Diffusion models take their inspiration from the principles
2. Denoising Diffusion Probabilistic Model
of non-equilibrium thermodynamics. They use a Markov (DDPM)
chain (Markovian process) to gradually introduce noise The first and original diffusion model is the Denoising Dif-
into data and then they learn to reverse this process to recre- fusion Probabilistic Model (DDPM), proposed in (Ho et al.,
ate the desired data from the noise. 2020). This section explains DDPM in details.
As shown in Fig. 1, a diffusion model has two processes The process of adding noise to data can be seen as a se-
with opposite directions. In the fixed forward diffusion pro- quence or chain {x0 , x1 , . . . , xT } where x0 ∈ Rd and
cess, it starts with the data and gradually adds noise to it xT ∈ Rd are the original data and data completely cor-
until it becomes a complete noise. Then, in the generative rupted with noise, respectively. Figure 3 depicts such a
reverse denoising process, it reverses that forward process chain. In this notation, xt+1 is xt plus some noise. As
and generates data out of the noise by denoising it gradu- the noise is random, there can be a conditional distribu-
ally. tion q(xt+1 |xt ) to model the probability of obtaining xt+1
As Fig. 2 depicts, the forward and reverse processes can from xt . The goal of diffusion models is to learn the con-
be modeled by the encoder and decoder of an autoencoder, ditional distribution q(xt |xt+1 ) modeling how to gener-
respectively. As the forward process merely needs to make ate data out of noise. This is because when q(xt |xt+1 ) is
data noisy, it does not involve any learning and thus is fixed. learned, t can be swept from T to 0 to generate a non-noisy
Therefore, only the decoder needs to be learned in diffusion data instance out of the complete noise.
models. This is one of the differences of diffusion models The distribution q(xt |xt+1 ) is intractable and it should be
from variational autoencoder (Kingma & Welling, 2014; approximated using a neural network. Let pθ (xt |xt+1 )
Ghojogh et al., 2022) in which both encoder and decoder denote the neural network with weights θ which approx-
 2

Figure 3. The conditional distributions in the forward and back-

ward processes of the diffusion model.

imates this conditional distribution. It is noteworthy that

q(xt |xt+1 , x0 ) is obviously tractable because it is con-
ditioning on the original data x0 . However, modeling
q(xt |xt+1 , x0 ) is not useful because the goal of diffusion Figure 4. The probabilistic graphical models of (a) the diffusion
models is to find x0 and the goal cannot be used in the model and (b) variational autoencoder (VAE).
model. In fact, that is a chicken and egg problem. Al-
though this distribution is not practical, but it will appear
diffusion model is illustrated in Fig. 4 where the states x1:T
later in the derivations of this chapter.
are the latent (or hidden) variables for the observed data x0 .
2.1. The Forward Process This figure also compares the diffusion model with the vari-
ational autoencoder where a variable z is the latent variable
The forward process is for making data noisy gradually:
for the observed data x.
x0 → x1 → · · · → xT . (1)
2.2. The Generative Backward Process
The forward process is defined by the following Gaussian The diffusion process, or the generative backward process,
conditional distribution: is the reverse process for generating data out of noise grad-
p ually:
q(xt |xt−1 ) = N (xt ; 1 − βt xt−1 , βt I), (2)
√ xT → xT −1 → · · · → x0 . (6)
where 1 − βt xt−1 is the mean and βt I is the covariance,
I is the identity matrix, and βt ∈ (0, 1) is the noise level
As the forward process has Gaussian distributions in Eq.
parameter in a pre-determined noise schedule:
(2), it makes sense to use Gaussian distributions for the
β1 < β2 < · · · < βT . (3) backward process, too. A generator machine learning
model, with parameters or weights θ, is used to model the
The βt parameter increases
√ gradually in the forward pro- backward process:
cess, so that the mean 1 − βt xt−1 deviates more and

more from the previously noisy data xt−1 and the variance pθ (xt−1 |xt ) = N xt−1 ; µθ (xt ; t), Σθ (xt ; t) , (7)
βt I increases. The closer the t gets to T , the more noisy
the data xt becomes. where µθ (xt ; t) ∈ Rd and Σθ (xt ; t) ∈ Rd×d are the mean
The forward process is considered as a Markov chain with and covariance of the distribution at iteration t, respec-
the first Markovian property. In the Markov chain, every tively. For simplicity, the covariance is usually set to be
state is merely dependant on its previous state and not the isotropic and diagonal:
entire previous sequence (Ghojogh et al., 2019):
Σθ (xt ; t) = σt2 I. (8)
q(xt |xt−1 , xt−2 , . . . , x0 ) = q(xt |xt−1 ). (4)
The goal of diffusion model is to learn the parameters of
Therefore, the joint distribution of the forward process is:
this Gaussian distribution, which are the mean and vari-
T
Y ance.
q(x1:T |x0 ) = q(xt |xt−1 ), (5) The generative process is also considered as a Markov
t=1 chain with the first Markovian property (Ghojogh et al.,
where x1:T represents all latent states from x1 to xT and 2019):
x0 is the initial state, i.e., the original data. The proba-
bilistic graphical model (Koller & Friedman, 2009) of the q(xt−1 |xt , xt+1 , . . . , xT ) = q(xt−1 |xt ). (9)
 3

Therefore, the joint distribution of the generative process els can be simplified as follows:
is:
 
L = Eq(x1:T |x0 ) log pθ (x0 |x1:T )
T

pθ (x0:T ) = p(xT )
Y
pθ (xt−1 |xt ), (10) − KL q(x1:T |x0 )∥pθ (x1:T )
t=1 (a)  
= Eq(x1:T |x0 ) log pθ (x0 |x1:T )
q(x1:T |x0 )
Z
where x0:T represents all states from xT to x0 and xT is − q(x1:T |x0 ) log dx1:T
pθ (x1:T )
the state corresponding to the complete noise. Usually, the (b)  
p(xT ) is defined to be the noise with standard Gaussian = Eq(x1:T |x0 ) log pθ (x0 |x1:T )
distribution: h q(x1:T |x0 ) i
− Eq(x1:T |x0 ) log
pθ (x1:T )
(c)
h q(x1:T |x0 ) i
p(xT ) = N (0, I). (11) = Eq(x1:T |x0 ) log pθ (x0 |x1:T ) − log
pθ (x1:T )

2.3. Variational Lower Bound of the Likelihood

As discussed before, variational autoencoders consider a
latent variable z as the latent factor for the observed data x. (d)
h pθ (x1:T ) i
= Eq(x1:T |x0 ) log pθ (x0 |x1:T ) + log
Likewise, the diffusion models consider the latent variables q(x1:T |x0 )
{x1 , x2 , . . . , xT } for the observed data x0 (see Fig. 4). (e)
h pθ (x0 |x1:T ) pθ (x1:T ) i
The likelihood of data in the variational autoencoders and = Eq(x1:T |x0 ) log
q(x1:T |x0 )
diffusion models are: (f )
h pθ (x0:T ) i
= Eq(x1:T |x0 ) log , (16)
q(x1:T |x0 )
Z
pθ (x) = pθ (x, z) dz, (12)
Z where (a) is because of the definition of the KL divergence,
pθ (x0 ) = pθ (x0 , x1:T ) dx1:T , (13) (b) is because of the definition of the expectation in the
second term of expression, (c) is because expectation is
a linear operator, (d) and (e) are because of the property
respectively. Recall that in variational inference, maxi- of logarithm, and (f ) is because pθ (x0 |x1:T ) pθ (x1:T ) =
mization of the likelihood was not tractable so a lower pθ (x0 , x1:T ) = pθ (x0:T ).
bound of the likelihood, named evidence lower bound The Eq. (16) can be simplified further:
(ELBO), was maximized instead (Ghojogh et al., 2021):
h pθ (x0:T ) i
L = Eq(x1:T |x0 ) log
q(x1:T |x0 )
 

Eq(z|x) log pθ (x|z) − KL q(z|x)∥pθ (z) ≤ pθ (x),
QT
(14) (10)
h p(xT ) t=1 pθ (xt−1 |xt ) i
= Eq(x1:T |x0 ) log
q(x1:T |x0 )
(a)  
where E[.] denotes expectation and KL(.∥.) denotes the = Eq(x1:T |x0 ) log p(xT )
Kullback-Leibler (KL) divergence (Kullback & Leibler, T
X
1951). See (Ghojogh et al., 2021; 2023) for the derivation
 
+ Eq(x1:T |x0 ) log pθ (xt−1 |xt )
of the ELBO in Eq. (14). t=1
 
Similarly, the diffusion models maximize the variational − Eq(x1:T |x0 ) log q(x1:T |x0 )
lower bound of the likelihood of data: (5)  
= Eq(x1:T |x0 ) log p(xT )
  T
L := Eq(x1:T |x) log pθ (x0 |x1:T )
X  

(15) + Eq(x1:T |x0 ) log pθ (xt−1 |xt )
− KL q(x1:T |x0 )∥pθ (x1:T ) ≤ pθ (x0 ). t=1
T
Y
 
− Eq(x1:T |x0 ) log q(xt |xt−1 )
The variational lower bound of likelihood in diffusion mod- t=1
 4
(b)
Plugging Eqs. (18) and (19) in Eq. (17) results in:
 
= Eq(x1:T |x0 ) log p(xT )
T
X  
+ Eq(x1:T |x0 ) log pθ (xt−1 |xt )

t=1
T
X L = Eq(x1:T |x0 ) log p(xT )
 
− Eq(x1:T |x0 ) log q(xt |xt−1 ) T 
t=1
X pθ (xt−1 |xt ) q(xt−1 |x0 )
+ log
(c)   q(xt−1 |xt , x0 ) q(xt |x0 )
= Eq(x1:T |x0 ) log p(xT ) t=1
T
 T
X pθ (xt−1 |xt )  (a) X pθ (xt−1 |xt )
= Eq(x1:T |x0 ) log p(xT ) + log

+ Eq(x1:T |x0 ) log
q(xt |xt−1 ) t=2
q(x t−1 |xt , x0 )
t=1
T T 
(d)
 X pθ (xt−1 |xt )
 X q(xt−1 |x0 ) pθ (x0 |x1 ) q(x0 |x0 )
= Eq(x1:T |x0 ) log p(xT ) + log , + log + log
q(xt |xt−1 ) t=2
q(xt |x0 ) q(x0 |x1 , x0 ) q(x1 |x0 )
t=1
(17)  T
(b) X pθ (xt−1 |xt )
= Eq(x1:T |x0 ) log p(xT ) + log
t=2
q(xt−1 |xt , x0 )
where (a) is because of the properties of logarithm and lin- T 
earity of the expectation, (b) is because logarithm of mul- X q(xt−1 |x0 ) pθ (x0 |x1 )
+ log + log , (20)
tiplication becomes summation of logarithms, (c) is for the t=2
q(xt |x0 ) q(x1 |x0 )
properties of logarithm, and (d) is because expectation is a
linear operator.
According to the definition of the KL divergence, the Eq.
(17), i.e., the variational lower bound, can be stated as fol- where (a) breaks down the logarithm and takes out the
lows: t = 1 from the summation, and (b) is because q(x0 |x0 ) =
q(x0 |x1 , x0 ) = 1. The third term in the expectation is a
  telescoping summation and can be simplified as follows:
L = Eq(x1:T |x0 ) log p(xT )


− KL q(xt |xt−1 )∥pθ (xt−1 |xt ) .
T
This is the variational lower bound which needs to be
X q(xt−1 |x0 )
log =
maximized; therefore, the KL divergence is minimized. t=2
q(xt |x0 )
Here, the KL divergence is between the forward process log q(xT −1 |x0 ) + · · · + log q(x2 |x0 ) + log q(x1 |x0 )
q(xt |xt−1 ) and the generative process pθ (xt−1 |xt ). Min-
− log q(xT |x0 ) − log q(xT −1 |x0 ) − · · · − log q(x2 |x0 )
imization of this KL divergence makes sense because the
distribution of the generative process should mimic the dis- q(x1 |x0 )
= − log q(xT |x0 ) + log q(x1 |x0 ) = log .
tribution of the forward process. q(xT |x0 )
As the sequence of states in the diffusion process is a
Markov chain, the following holds:
Therefore, the variational lower bound in Eq. (20) be-
q(xt |xt−1 ) = q(xt |xt−1 , x0 ). (18) comes:

According to the Bayes’ rule, there is:

 T
X pθ (xt−1 |xt )
q(xt−1 |xt , x0 ) q(xt |x0 ) L = Eq(x1:T |x0 ) log p(xT ) + log
q(xt |xt−1 , x0 ) = , (19) t=2
q(x t−1 |xt , x0 )
q(xt−1 |x0 ) 
q(x1 |x0 ) pθ (x0 |x1 )
+ log + log
q(xT |x0 ) q(x1 |x0 )
where q(xt−1 |xt , x0 ) is tractable in contrast to q(xt−1 |xt )
 T
because it is conditioning on the original data. In fact, Eqs. (a) p(xT ) X pθ (xt−1 |xt )
(18) and (19) are used to make q(xt−1 |xt ) tractable. The = Eq(x1:T |x0 ) log + log
q(xT |x0 ) t=2 q(xt−1 |xt , x0 )
tractable distribution q(xt−1 |xt , x0 ) is called the ground 
truth denoising distribution because it sees the original data + log pθ (x0 |x1 )
x0 .
 5
 
(b) p(xT ) Here, the reparameterization technique (Kingma &
= Eq(x1:T |x0 ) log
q(xT |x0 ) Welling, 2014) can be used. In this technique, for sam-
T   pling from a non-standard Gaussian distribution z ∼
X pθ (xt−1 |xt )
+ Eq(x1:T |x0 ) log N (µ, σ 2 I), one can sample from a standard distribution
t=2
q(xt−1 |xt , x0 ) instead, i.e., ϵ ∼ N (0, I) and obtain the sample z as
 
+ Eq(x1:T |x0 ) log pθ (x0 |x1 ) z = µ + σϵ. Here, using the reparameterization technique,
(c)
the ϵ ∼ N (0, I) is sampled and then {xt , xt−1 , . . . , x1 }
= −KL q(xT |x0 )∥p(xT ) are each obtained as:
T √
X √
xt = αt xt−1 + 1 − αt ϵ,


− KL q(xt−1 |xt , x0 )∥pθ (xt−1 |xt ) (25)
t=2 √ p
  xt−1 = αt−1 xt−2 + 1 − αt−1 ϵ, (26)
+ Eq(x1:T |x0 ) log pθ (x0 |x1 ) , √ p
xt−2 = αt−2 xt−3 + 1 − αt−2 ϵ, (27)
where (a) is because of the properties of logarithm, (b) is ..
.
because expectation is a linear operator, and (c) is because
of the definition of KL divergence. Plugging xt−1 , Eq. (26), in xt , Eq. (25), gives:
The variational lower bound of the likelihood should be
√
√ √ p
maximized with respect to the parameters θ. The first xt = αt αt−1 xt−2 + 1 − αt−1 ϵ + 1 − αt ϵ
KL divergence is constant with respect to θ so it can be √ p √
= αt αt−1 xt−2 + αt (1 − αt−1 )ϵ + 1 − αt ϵ.
dropped. Maximization of the lower bound is equivalent to
(28)
minimization of it multiplied by negative one:
Note that the ϵ in the second and third terms are two dif-
T
X
ferent samples from the standard Gaussian distributions.
minimize KL q(xt−1 |xt , x0 )∥pθ (xt−1 |xt )
θ Therefore, the second and third terms are two independent
t=2
  random Gaussian distributions:
− Eq(x1:T |x0 ) log pθ (x0 |x1 ) . (21)
p

αt (1 − αt−1 )ϵ ∼ N 0, αt (1 − αt−1 ) ,
Therefore, it can be the loss function of a neural net- | {z }
=αt −αt αt−1
work with weights θ. This loss function makes sense be- √
cause it learns the backward distribution pθ (xt−1 |xt ) to 1 − αt ϵ ∼ N (0, 1 − αt ),
become close to the ground truth denoising distribution
q(xt−1 |xt , x0 ) by minimizing their KL divergence. That is so their summation becomes a Gaussian distribution whose
why the focus of the loss function is on the KL divergence variance is summation of the two variances:
and the second term in Eq. (21) can be ignored. Remem- p √
αt (1 − αt−1 )ϵ + 1 − αt ϵ ∼ N (0, 1 − αt αt−1 ).
ber this loss function and we will get back to it later in the
chapter. As a result, Eq. (28) can be stated as:
2.4. Parameters of the Distributions √ p
xt = αt αt−1 xt−2 + 1 − αt αt−1 ϵ
2.4.1. S AMPLING IN F ORWARD P ROCESS AT
A RBITRARY I TERATIONS Plugging xt−2 , Eq. (27), in this expression gives:
Recall the Gaussian distributions of the forward process in √ √ p 
Eq. (2). That conditional distribution was for sampling xt = αt αt−1 αt−2 xt−3 + 1 − αt−2 ϵ
xt given xt−1 . However, it is also possible to sample xt p
at any arbitrary iteration t in a closed form. Consider the + 1 − αt αt−1 ϵ
√ p
following definitions: = αt αt−1 αt−2 xt−3 + αt αt−1 − αt αt−1 αt−2 ϵ
p
+ 1 − αt αt−1 ϵ. (29)
αt := 1 − βt , (22)
t
Y Again, the second and third terms are two independent ran-
ᾱt = αi . (23) dom Gaussian distributions:
i=1
p
αt αt−1 − αt αt−1 αt−2 ϵ
Therefore, Eq. (2) can be restated as:
∼ N (0, αt αt−1 − αt αt−1 αt−2 ),
√
p
q(xt |xt−1 ) = N xt ; αt xt−1 , (1 − αt )I . (24) 1 − αt αt−1 ϵ ∼ N (0, 1 − αt αt−1 ),
 6

so their summation becomes a Gaussian distribution whose ties in this Bayes’ rule are:
variance is summation of the two variances:

p p
αt αt−1 − αt αt−1 αt−2 ϵ + 1 − αt αt−1 ϵ
(18)
∼ N 0, 1 − αt αt−1 αt−2 ). q(xt |xt−1 , x0 ) = q(xt |xt−1 )
(22),(24) √
= N (xt ; αt xt−1 , βt I),
As a result, Eq. (29) can be stated as: √
1 (xt − αt xt−1 )2 
  
∝ exp − ,
2 βt
√ p
xt = αt αt−1 αt−2 xt−3 + 1 − αt αt−1 αt−2 ϵ. (31) √

q(xt−1 |x0 ) = N xt−1 ; ᾱt−1 x0 , (1 − ᾱt−1 )I ,
√
1 (xt−1 − ᾱt−1 x0 )2 
  
By induction, continuing this recursion until x0 gives xt in ∝ exp − ,
2 1 − ᾱt−1
terms of the original data x0 and the noise ϵ:
(31) √

q(xt |x0 ) = N xt ; ᾱt x0 , (1 − ᾱt )I
√
1 (xt − ᾱt x0 )2 
v v   
u t u t
uY u Y ∝ exp − .
xt = t αi x0 + 1 −
t αi ϵ 2 1 − ᾱt
i=1 i=1
(23) √ √
= ᾱt x0 + 1 − ᾱt ϵ, (30)
Therefore:
which states xt in terms of the original data x0 and noise
ϵ. As x0 is not random and ϵ has standard Gaussian dis-
tribution, the distribution of xt in this equation has mean
√
ᾱt x0 and covariance (1 − ᾱt )I. Therefore: q(xt−1 |xt , x0 )
√ √
1 (xt − αt xt−1 )2 (xt−1 − ᾱt−1 x0 )2
 
√
(33)
∝ exp − +
q(xt |x0 ) = N xt ; ᾱt x0 , (1 − ᾱt )I . (31) 2 βt 1 − ᾱt−1
√ 2

(xt − ᾱt x0 )
By this conditional distribution, the noisy data xt at any −
1 − ᾱt
iteration t can be sampled based on the original data x0 .   2 √
(a) 1 xt − 2 αt xt xt−1 + αt x2t−1
= exp −
2.4.2. T HE G ROUND T RUTH D ENOISING 2 βt
D ISTRIBUTION √
x2t−1 − 2 ᾱt−1 x0 xt−1 + ᾱt−1 x20
+
A neural network is required to learn the parameters, i.e., 1 − ᾱt−1
the mean and covariance, of the backward process in Eq. √
(xt − ᾱt x0 )2 

(7). Similar to Eq. (7), the ground truth denoising distribu- −
1 − ᾱt
tion can be a Gaussian distribution:  
(b) 1 αt 1
2
= exp − + x

2 βt 1 − ᾱt−1 t−1
q(xt−1 |xt , x0 ) = N xt−1 ; µ
e (xt , x0 ), Σ(x
e t , x0 ) , (32) √ √ 
αt ᾱt−1

−2 xt + x0 xt−1 + C(xt , x0 )
βt 1 − ᾱt−1
√
where µ e (xt , x0 ) ∈ Rd and Σ(x
e t , x0 ) ∈ Rd×d are the √
αt ᾱt−1 2
βt xt + 1−ᾱt−1 x0

mean and covariance of the distribution, respectively. For  xt−1 − αt 1

(c) 1 βt + 1−ᾱt−1
simplicity, similar to Eq. (8), the covariance is usually set ∝ exp − , (34)
to be isotropic and diagonal, i.e., Σ(x et2 I.
e t , x0 ) = σ 2 1/( αβtt + 1−ᾱ1t−1 )
By Bayes’ rule, we have:

q(xt |xt−1 , x0 ) q(xt−1 |x0 )

q(xt−1 |xt , x0 ) = . (33) where (a) and (c) are because of the binomial theorem, (b)
q(xt |x0 ) is because of factoring out the terms x2t−1 and xt−1 , and
C(xt , x0 ) is a function not including xt−1 and thus it can
According to Eqs. (18), (22), (24), and (31), the probabili- be dropped and ignored for optimization.
 7

2.4.3. M EAN OF THE G ROUND T RUTH D ENOISING where q(x) = N (x; µ e and p(x) = N (x; µ , Σθ ).
e , Σ) θ
D ISTRIBUTION There is a closed form expression for the KL divergence of
According to Eq. (34), the mean of the ground truth de- two Gaussian distributions:
noising distribution, in Eq. (32), is as follows: 1
KL(q∥p) = tr(Σ−1
θ Σ) + (µθ − µ
e e )⊤ Σ−1
θ (µθ − µ
e)
√
αt
√
ᾱt−1
√
αt
√
ᾱt−1
2
βt xt + 1−ᾱt−1 x0 βt xt + 1−ᾱt−1 x0 det(Σθ )

µ
e (xt , x0 ) = αt 1 = αt −αt ᾱt−1 +βt
− d + ln , (39)
βt + 1−ᾱt−1 det(Σ)
e
βt (1−ᾱt−1 )
√ √
αt ᾱt−1 where d is the dimensionality of x, and tr(.) and det(.)
(a) βt xt + 1−ᾱt−1 x0
= 1−ᾱt denote trace and determinant of matrix, respectively. For
βt (1−ᾱt−1 ) simplicity, assume the covariances of the two distributions
 √α √
ᾱt−1  1 − ᾱ are diagonal and equal, i.e., Σθ = Σq = λI (see Eq. (8)).
t t−1
= xt + x0 βt
βt 1 − ᾱt−1 1 − ᾱt Then, Eq. (39) is simplified to:
√ √
αt (1 − ᾱt−1 ) ᾱt−1 βt 1
= xt + x0 , (35) KL(q∥p) = e )⊤ Σ−1
(µ − µ θ (µθ − µ
e ),
1 − ᾱt 1 − ᾱt 2 θ
where (a) is because αt + βt = 1 and αt ᾱt−1 = ᾱt by which is the squared Mahalanobis distance between the
Eqs. (22) and (23). According to Eq. (30), we have: means of distributions. This makes sense because the two
distributions p and q are normal distributions and their co-
√ √
xt = ᾱt x0 + 1 − ᾱt ϵ variances – related to the second moment – are equal. So,
1 √ for making them the same, their first moments, i.e., their
=⇒ x0 = √ (xt − 1 − ᾱt ϵ). (36) means, should become the same.
ᾱt
As a result, the loss function can be written as the squared
Substituting Eq. (36) in Eq. (35) gives: Mahalanobis distance between the means of distributions:
√ h 1 i
αt (1 − ᾱt−1 ) ℓt := Ex0 ,ϵ 2 µ(xt , x0 ) − µθ (xt , t)∥22
∥e
µ
e (xt , x0 ) = xt 2∥Σθ (xt , t)∥2
1 − ᾱt
√ (a)

1 1  1 − αt 
ᾱt−1 βt 1 √ = Ex0 ,ϵ √ xt − √ ϵ
+ √ (xt − 1 − ᾱt ϵ) 2∥Σθ (xt , t)∥22 αt 1 − ᾱt
1 − ᾱt ᾱt
1  1 − αt  2
(a) 1  1 − αt  (22) 1  βt 
−√ xt − √ ϵθ
= √ xt − √ ϵ = √ xt − √ ϵ , αt 1 − ᾱt 2
αt 1 − ᾱt αt 1 − ᾱt
2
 
(37) (1 − αt ) 2
= Ex0 ,ϵ ∥ϵ − ϵθ (xt , t)∥2
2αt (1 − ᾱt )∥Σθ (xt , t)∥22
where (a) is because of simplification of the expression
(1 − αt )2

(30)
while noticing Eqs. (22) and (23). Eq. (37) is the mean = Ex0 ,ϵ ×
of the tractable backward distribution q(xt−1 |xt , x0 ) as in 2αt (1 − ᾱt )∥Σθ (xt , t)∥22
√ √

Eq. (32). 2
ϵ − ϵθ ( ᾱt x0 + 1 − ᾱt ϵ, t) 2 , (40)
Let µθ (xt , t) denote the mean of the distribution of the
backward process, i.e., pθ (xt−1 |xt ). This mean also has where (a) is because of Eqs. (37) and (38) while noticing
a similar format as the mean µ
e (xt , x0 ); therefore: Eq. (22).
1  βt  Ho et al. (Ho et al., 2020) discovered empirically that the
µθ (xt , t) = √ xt − √ ϵθ (xt , t) , (38) diffusion model produces better-quality images when using
αt 1 − ᾱt
a simplified loss function, omitting the weighting term:
where ϵθ (xt , t) is a function approximator (i.e., neural net- 
√ √

2
work) intended to predict ϵ from xt . In summary, the back- ℓ = Et∼[1,T ],x0 ,ϵ ϵ − ϵθ ( ᾱt x0 + 1 − ᾱt ϵ, t) 2 ,
ward process pθ (xt−1 |xt ) has a Gaussian distribution as in
(41)
the Eq. (7), whose mean and covariance are Eqs. (38) and
(8), respectively. where t ∼ [1, T ] means sampling t from the set
{1, 2, . . . , T } uniformly. Recall that ᾱt is calculated by
2.5. The Loss Function of the Diffusion Model Eq. (22) based on the selected noise schedule βt in Eq.
Recall the KL-divergence in the loss function of Eq. (21): (3). Note that the expectation can be approximated by aver-

age according to the Monte Carlo approximation (Ghojogh
KL q(xt−1 |xt , x0 )∥pθ (xt−1 |xt ) , et al., 2020).
 8

1 while not converged do 1 xT ∼ N (0, I)

2 x0 ∼ q0 (x0 ) 2 for t from T to 1 do
3 t ∼ Uniform{1, . . . , T } 3 ϵ ∼ N (0, I)
ϵ ∼ N (0, I)

4
4 xt−1 = √1 xt − √1−αt ϵθ (xt , t) + σt ϵ
αt 1−ᾱt
5 Do backpropagation by the gradient of the
loss function (41) 5 Return x0

Algorithm 1: Training algorithm in DDPM Algorithm 2: Inference algorithm for generating

data in DDPM

et al., 2020). We define:

2.6. Training Phase of DDPM
1 − ᾱt−1
The training algorithm of the DDPM is in Algorithm 1. In βet := βt . (43)
1 − ᾱt
every iteration, an image x0 is sampled from the training
dataset. Then, the time step t is sampled uniformly from Nichol et al. (Nichol & Dhariwal, 2021) found out em-
{1, . . . , T }. Then, for the sampled time step t, the noise pirically that the covariance Σθ (xt ; t) used in Eq. (40) is
ϵ is sampled from the standard Gaussian distribution. Fi- better to be learned as a linear combination of βt and βet in
nally, a backpropagation step is performed by the gradient the log domain:
of the loss function in Eq. (41). This makes the neural net- 

work learn the noise. In other words, it learns how much Σθ (xt ; t) = diag exp v log βt + (1 − v) log βet ,
noise it should have in the backward process to reverse the (44)
forward process. This procedure is repeated until conver-
gence of the weights θ of neural network. One of the advan- where diag(.) makes a diagonal matrix with its input as the
tages of DDPM compared to GAN (Generative Adversarial diagonal, exp is the exponential operator, 1 ∈ Rd is the
Network) (Goodfellow et al., 2014), is that this is a simple vector of ones, and v ∈ Rd is a vector outputted by the
algorithm to train and it is stable. This is while training network of DDPM which is learned by backpropagation.
GAN is hard and tricky. The loss function of the network is changed from Eq. (41)
to:
2.7. Inference (Sampling) Phase of DDPM T
X
In the inference phase, the network has been trained, mean- ℓ+λ ℓt , (45)
ing that the distribution pθ is learned. This distribution can t=0

be used to start with a complete noise and generate (or sam- where ℓt and ℓ are defined in Eqs. (40) and (41), respec-
ple) data out of it in a backward process. tively, and λ = 0.001 is the regularization hyperparameter.
The inference phase starts with the complete noise, i.e., Note that Σθ (xt ; t), defined by Eq. (44), is used in ℓt so its
xT ∼ N (0, I). Let σt I denote the covariance of xt . v is learned by backpropagation.
By use of the reparameterization technique (Kingma &
Welling, 2014) and according to Eq. (38) while noticing 3.2. Reducing the Gradient Noise
Eq. (22), we have: One might wonder why a regularized loss function, as in
Eq. (45), is required while ℓ is the simplification of sum-
mation of ℓt ’s; meaning that the information of the first
1  1 − αt 
term is contained in the second term of Eq. (45). By use
xt−1 = √ xt − √ ϵθ (xt , t) + σt ϵ, (42)
αt 1 − ᾱt of gradient noise scale (McCandlish et al., 2018), Nichol
et al. (Nichol & Dhariwal, 2021) empirically observed that
the gradient noise of ℓt is much more than that of ℓ. They
where ϵ ∼ N (0, I) and αt and ᾱt are calculated by Eqs.
observed that this gradient noise is caused by the uniform
(22) and (23) based on the selected noise schedule βt in Eq.
sampling of t in Algorithm 1. By replacing the uniform
(3). Note that ϵθ (xt , t) is outputted by the trained network.
sampling with importance sampling (Ghojogh et al., 2020),
Iterating over t from T to 1 gives the generated data x0 .
which records a history of previous values, they reduced
The inference phase of DDPM is in Algorithm 2.
this gradient [Link] using importance sampling, they
T
could use merely t=0 ℓt as the loss function.
3. Modifications to the Original DDPM
3.1. Non-fixed Covariance Matrix 3.3. Improving the Noise Schedule
Recall Eq. (8) in which Σθ is assumed to be fixed and Recall the noise schedule stated in Eq. (3). While lin-
diagonal, i.e., Σθ (xt ; t) = σt2 I, in the original DDPM (Ho ear noise schedule used in the original DDPM (Ho et al.,
 9

2020) works well on high-resolution images, Nichol et al. 1 xT ∼ N (0, I)

(Nichol & Dhariwal, 2021) observed that it does not work 2 for t from T to 1 do
properly on the small 32 × 32 or 64 × 64 images. This is 3 ϵ ∼ N (0, I)
because the linear noise schedule makes the xT too noisy 
4 xt−1 = √1 xt − √1−αt ϵθ (xt , t) + σt ϵ
at the end of the forward process. They proposed a cosine αt 1−ᾱt
√ √
noise schedule which works well also on the small images 5 y t−1 = ᾱt−1 y + 1 − ᾱt−1 ϵ
(Nichol & Dhariwal, 2021): 6 xt−1 ← xt−1 − ϕn (xt−1 ) + ϕn (y t−1 )
f (t) 7 Return x0
ᾱt = , (46)
f (0) Algorithm 3: Inference algorithm for generating
 t/T + s π 2
data in ILVR (conditional diffusion model)
f (t) := cos × , (47)
1+s 2
where s = 0.008 is the offset. Note that according to Eqs. which generalizes DDPM from Markov chains to non-
(22) and (23), we have: Markovian chains to make inference (sampling) faster. An-
ᾱt other improvement over DDPM is the Discrete Denois-
βt = 1 − αt = 1 − . (48) ing Diffusion Probabilistic Model (D3PM) (Austin et al.,
ᾱt−1
2021) which extends DDPM from generating continuous-
Using Eq. (46) in Eq. (48) gives βt at time t in the sched- state data to discrete-state data. In other words, D3PM
ule. works on categorical (discrete) data rather than the conven-
tional float data as in DDPM.
3.4. Non-Standard Gaussian Noise Distribution
The original DDPM (Ho et al., 2020) considers standard 4. Conditional Diffusion Model
Gaussian distribution for the noise ϵ. However, the corre- DDPM (Ho et al., 2020) is unconditional; meaning that it
sponding data distribution may be more complicated than generates any data, from the data distribution, in the infer-
the standard Gaussian distribution making this noise distri- ence (sampling) phase. Iterative Latent Variable Refine-
bution not necessarily suitable for the process. PriorGrad ment (ILVR) (Choi et al., 2021) is a conditional diffusion
(Lee et al., 2022) takes care of this concern and modifies model which can generate data from a specific category or
DDPM to use a data-dependent prior distribution. Suppose class of data chosen by the user. In this algorithm, the al-
the mean µ and the covariance Σ are obtained from some gorithm takes a reference data instance (e.g., a reference
data-dependent prior; for example, they can be the mean image) and generates a data instance similar to, or in the
and variance of the training data. In PriorGrad, the noise is category of, the reference. The training phase of ILVR is
sampled from the Gaussian distribution with covariance Σ, the same as training in DDPM. In fact, this algorithm mod-
i.e., ϵ ∼ N (0, Σ). It changes Eq. (30) to: ifies the inference (sampling) phase of DDPM to become a
√ √ conditional generator. The inference algorithm of ILVR is
xt = ᾱt (x0 − µ) + 1 − ᾱt ϵ, (49)
Algorithm 3.
in which the mean is deducted from the data x0 . This algo- Let ϕn (.) be a low-pass filter which downsamples data by
rithm also changes the loss function from Eq. (41) to: a factor of n and then upsamples it again by the factor n.
  This only preserves the low frequency of data and omits its
2 high frequency. For example, in images, it makes the image
ℓ = Et∼[1,T ],x0 ,ϵ ϵ − ϵθ (xt , t) Σ−1 , (50)
blurred. Obviously, if x ∈ Rd , then ϕn (x) ∈ Rd . Suppose
the unconditional sample xt−1 ∈ Rd has been calculated
where xt is obtained from Eq. (49) and ∥x∥2Σ−1 := by Eq. (42). Let the reference data instance be denoted by
x⊤ Σ−1 x. In the inference phase, it simply adds back the y ∈ Rd . Similar to Eq. (30), y t−1 can be obtained from y
subtracted mean µ to the generated data x0 . The main and the noise ϵ as:
ideas of PriorGrad (Lee et al., 2022) are two-fold: (1) sub-
√ p
tracting the mean of data in the training and adding it back y t−1 = ᾱt−1 y + 1 − ᾱt−1 ϵ. (51)
in sampling, and (2) using the inverse of covariance matrix
in the loss function. Therefore, it can use a non-standard The unconditional sample xt−1 can then be modified to be
Gaussian distribution in the diffusion model. similar to the noisy reference y t−1 at time slot (t − 1). For
this, the low frequency of xt−1 , i.e., ϕn (xt−1 ) ∈ Rd , is
3.5. Some Other Improvements subtracted from it and, instead, the low frequency of y t−1
An additional improvements over DDPM is the Denois- is added back to it. This replaces the low frequency of xt−1
ing Diffusion Implicit Model (DDIM) (Song et al., 2021a) with the low frequency of the noisy reference at time slot
 10

(t − 1). Performing this iteratively from t = T to t = 1 and the ∇x log pt (x) is the time-dependant gradient field,
gives the sample similar to the reference sample. called score briefly. As long as the score gets estimated,
It is noteworthy that n is a positive integer selected by the this reverse-time SDE can be solved and therefore the sam-
user. The smaller the n, the more similar the sample will ple can be generated out of noise in the backward process
be to the reference. This is because a small n makes the of diffusion model.
bandwidth of the low pass filter larger, replacing most of According to (Song et al., 2021b), the loss function of the
the frequencies of the sample with the reference data. original DDPM, for maximizing the ELBO, can be stated
in the form of a weighted sum of denoising score matching
5. Continuous Noise Schedule by Stochastic (Vincent, 2011):
Differential Equation T 
X
The noise schedule in DDPM (Ho et al., 2020) is in the ℓ= (1 − αt )Epdata (x) Eq(xt |x0 ) sθ (xt , t)
discrete domain; meaning that there are T discrete steps t=1
from the original data to the complete noise and backwards.



However, it is possible to have noise schedule of diffusion − ∇xt log q(xt |x0 ) , (55)
models in a continuous domain; meaning that t ∈ [0, T ] can
be continuous rather than being discrete t ∈ {1, 2, . . . , T }. where sθ (xt , t) is the output of the neural network with
The continuous noise schedule converts the Markov chain weights θ. Likewise, the loss function in the diffusion
of diffusion model to a Stochastic Differential Equation model with SDE is (Song et al., 2021b):
(SDE) (Song et al., 2021b). " 
Consider Eq. (25):
ℓ = Et λ(t) Ex(0) Ex(t)|x(0) sθ (xt , t)
√ √ (22) p p
xt = αt xt−1 + 1 − αt ϵ = 1 − βt xt−1 + βt ϵ, 

#
− ∇x(t) log q x(t)|x(0) , (56)
in DDPM in which t ∈ {1, 2, . . . , T }. If T → ∞, the num-
ber of steps in the noise schedule becomes huge making
the flow of time almost continuous. By T → ∞, Eq. (25) where λ : [0, T ] → R>0 is a positive weighting func-
converges to the following SDE (Song et al., 2021b): tion, t is sampled from the uniform distribution in range
[0, T ], the x(0) is a sampled data instance from the train-
1


p
dx = − β(t) x dt + β(t) dw, (52) ing dataset, and x(t) ∼ q x(t)|x(0) . This loss function
2 is minimized by backpropagation until the output of net-
work, i.e., sθ (xt , t), becomes a good estimator of the score
where w ∈ Rd is the standard Wiener process (Brownian ∇x(t) log q x(t)|x(0) .



motion), x ∈ Rd is the data variable, and dx and dw are
After training, the neural network becomes a function ap-
the differentials of x and w. As a result, DDPM (Ho et al.,
proximator for the score. Therefore, the reverse-time SDE,
2020) is a special case of diffusion model with SDE (Song
Eq. (54), can be solved by numerical SDE solvers to
et al., 2021b).
generate data in the backward process of diffusion model.
In a more general case, the forward process in the diffusion Various numerical methods exist for solving SDEs, some
model with SDE is the following SDE (Song et al., 2021b): of which are Euler-Maruyama and stochastic Runge-Kutta
(Kloeden & Platen, 1992). Any SDE solver can be used to
dx = f (x, t) dt + g(t) dw, (53)
solve the reverse-time SDE of the diffusion model to gen-
where w ∈ Rd is the standard Wiener process (Brownian erate data.
motion), f (., t) : Rd → Rd is a function named the drift It is noteworthy that continuous noise schedule by SDE
coefficient of x(t), and g(.) : R → R is a function called has been also extended for discrete-state (categorical) data
the diffusion coefficient of x(t). The backward process of (Campbell et al., 2022).
the diffusion model with SDE is the following reverse-time
SDE (Song et al., 2021b): 6. Conclusion
  This was a tutorial paper about diffusion models – a fam-
dx = f (x, t) − g(t)2 ∇x log pt (x) dt + g(t) dw̄, (54) ily of generative models in machine learning. The original
DDPM, its forward and backward processes, its variational
where w̄ is the standard Wiener process when time flows lower bound, and parameters of distributions were covered.
backwards from T to 0 and dt is the infinitesimal nega- Some modifications and improvements to DDPm were also
tive time. The operator ∇x is the derivative with respect to introduced. Then, conditional diffusion model for generat-
x, the pt (x) is the probability distribution of x at time t, ing data similar to a reference data instance was explained.
 11

Finally, the diffusion model using SDE was covered for Advances in neural information processing systems, 27,
having a continuous noise schedule. 2014.

Acknowledgement Ho, Jonathan, Jain, Ajay, and Abbeel, Pieter. Denoising

diffusion probabilistic models. Advances in neural in-
Some of the materials in this tutorial paper have been cov- formation processing systems, 33:6840–6851, 2020.
ered by Prof. Ali Ghodsi’s videos (Data Science Courses)
and Tushar Kumar’s videos (Explaining AI) on YouTube. Kingma, Diederik P and Welling, Max. Auto-encoding
variational Bayes. In International Conference on
References Learning Representations, 2014.
Austin, Jacob, Johnson, Daniel D, Ho, Jonathan, Tarlow,
Kloeden, Peter E and Platen, Eckhard. Stochastic differen-
Daniel, and van den Berg, Rianne. Structured denoising
tial equations. Springer, 1992.
diffusion models in discrete state-spaces. In Advances in
Neural Information Processing Systems, 2021. Koller, Daphne and Friedman, Nir. Probabilistic graphical
models: principles and techniques. MIT press, 2009.
Campbell, Andrew, Benton, Joe, De Bortoli, Valentin,
Rainforth, Tom, Deligiannidis, George, and Doucet, Ar- Kullback, Solomon and Leibler, Richard A. On informa-
naud. A continuous time framework for discrete denois- tion and sufficiency. The annals of mathematical statis-
ing models. In Advances in Neural Information Process- tics, 22(1):79–86, 1951.
ing Systems, 2022.
Lee, Sang-gil, Kim, Heeseung, Shin, Chaehun, Tan, Xu,
Choi, Jooyoung, Kim, Sungwon, Jeong, Yonghyun, Gwon, Liu, Chang, Meng, Qi, Qin, Tao, Chen, Wei, Yoon, Sun-
Youngjune, and Yoon, Sungroh. ILVR: Conditioning groh, and Liu, Tie-Yan. PriorGrad: Improving condi-
method for denoising diffusion probabilistic models. In tional denoising diffusion models with data-dependent
Proceedings of the IEEE/CVF International Conference adaptive prior. In International Conference on Learning
on Computer Vision, pp. 14367–14376, 2021. Representations, 2022.

Ghojogh, Benyamin, Karray, Fakhri, and Crowley, Mark. McCandlish, Sam, Kaplan, Jared, Amodei, Dario, and
Hidden Markov model: Tutorial. Engineering Archive, Team, OpenAI Dota. An empirical model of large-batch
2019. training. arXiv preprint arXiv:1812.06162, 2018.

Ghojogh, Benyamin, Nekoei, Hadi, Ghojogh, Aydin, Kar- Nichol, Alexander Quinn and Dhariwal, Prafulla. Im-
ray, Fakhri, and Crowley, Mark. Sampling algorithms, proved denoising diffusion probabilistic models. In In-
from survey sampling to Monte Carlo methods: Tutorial ternational Conference on Machine Learning, pp. 8162–
and literature review. arXiv preprint arXiv:2011.00901, 8171. PMLR, 2021.
2020.
Song, Jiaming, Meng, Chenlin, and Ermon, Stefano. De-
Ghojogh, Benyamin, Ghodsi, Ali, Karray, Fakhri, and noising diffusion implicit models. In International Con-
Crowley, Mark. Factor analysis, probabilistic princi- ference on Learning Representations, 2021a.
pal component analysis, variational inference, and vari-
Song, Yang, Sohl-Dickstein, Jascha, Kingma, Diederik P,
ational autoencoder: Tutorial and survey. arXiv preprint
Kumar, Abhishek, Ermon, Stefano, and Poole, Ben.
arXiv:2101.00734, 2021.
Score-based generative modeling through stochastic dif-
Ghojogh, Benyamin, Crowley, Mark, Karray, Fakhri, and ferential equations. In International Conference on
Ghodsi, Ali. Variational autoencoders. In Elements of Learning Representations, 2021b.
Dimensionality Reduction and Manifold Learning, pp. Vincent, Pascal. A connection between score matching
563–576. Springer, 2022. and denoising autoencoders. Neural computation, 23(7):
Ghojogh, Benyamin, Crowley, Mark, Karray, Fakhri, and 1661–1674, 2011.
Ghodsi, Ali. Factor analysis and probabilistic principal
component analysis. In Elements of Dimensionality Re-
duction and Manifold Learning, pp. 355–374. Springer,
2023.

Goodfellow, Ian, Pouget-Abadie, Jean, Mirza, Mehdi, Xu,

Bing, Warde-Farley, David, Ozair, Sherjil, Courville,
Aaron, and Bengio, Yoshua. Generative adversarial nets.