Diffusion Generative Models

CS 274E Guest Lecture

Gavin Kerrigan
2023 16 November
gavin.k@[Link]
Introduction

Diffusion Generative Models are a class of deep generative models

that generate data by iterative denoising.

1
Introduction

Unconditional generation: Images

Figure 1: [Dhariwal and Nichol, Diffusion Models Beat GANs on Image Synthesis,
NeurIPS 2021]
2
Introduction

Unconditional generation: Point clouds

Figure 2: [Cai et al., Learning Gradient Fields for Shape Generation, ECCV 2020]

3
Introduction

Conditional generation: Text → Image

• e.g. DALLE-2, Imagen, Stable Diffusion, etc.

Figure 3: “An oil painting of a cat wearing an ornate wizard hat and robe”

[Link]/dall-e-2

4
Introduction

Conditional generation: Image → Image

• e.g. Super-resolution

Figure 4: [Saharia et al., Image Super-Resolution via Iterative Refinement, ICCV

2021]

5
Introduction

Conditional generation: Image → Image

• Precipitation forecasting

[Asperti et al., 2023]

6
Introduction

Conditional generation: Graphs → 3D Molecules

Figure 6: [Xu et al., GeoDiff: A Geometric Diffusion Model for Molecular

Conformation Generation, ICLR 2022]

7
Introduction

Conditional generation: Graphs → 3D Molecules

Figure 7: [Corso et al., DiffDock: Diffusion Steps, Twists, and Turns for Molecular
Docking, ICLR 2023]

8
Denoising Diffusion Models

9
Denoising Diffusion Models

“Creating noise from data is easy; creating data from noise is

generative modeling”1 .
This talk: Denoising Diffusion Probabilistic Models (DDPM)

• Derivations from [Ho et al., DDPM, NeurIPS 2020] and

[Sohl-Dickstein et al., Deep Unsupervised Learning using
Nonequilibrium Thermodynamics, ICML 2015]
• Based in part on Arash Vahdat’s CVPR 2022 tutorial

1 Song et al., Score-Based Generative Modeling through Stochastic Differential

Equations, ICLR 2021

10
Denoising Diffusion Models

“Creating noise from data is easy; creating data from noise is

generative modeling”2 .

2 Song et al., Score-Based Generative Modeling through Stochastic Differential

Equations, ICLR 2021

11
Denoising Diffusion Models

Have samples from an unknown data distribution q(x0 )

12
Forward Process

Creating noise from data is easy:

• Choose an integer T (typically large) and a variance schedule βt

• βt is often a linear interpolation between β1 and βT
• Slowly make your data noisier over T steps
√ √
xt = 1 − βt xt−1 + βt ϵ ϵ ∼ N (0, I) (1)
√
q(xt | xt−1 ) = N (xt | 1 − βt xt−1 , βt I) (2)

This defines the forward (diffusion) process:

t=0 t = 50 t = 100 t = 200 t = 300

13
Forward Process

Forward Process:

√
q(xt | xt−1 ) = N (xt | 1 − βt xt−1 , βt I) (3)

• Gives you a joint distribution

∏
T
q(x1:T | x0 ) = q(xt | xt−1 ) (4)
t=1

Similar to a latent variable model / VAE:

• Encoder: q
• Latent variable(s): x1:T
• Observed variable: x0

14
Forward Process

Forward Process:

√
q(xt | xt−1 ) = N (xt |1 − βt xt−1 , βt I) (5)
∏t
Cheap to sample at any time t. Set γt = s=1 (1 − βs ). Then:
√ √
xt = γt x0 + 1 − γt ϵ ϵ ∼ N (0, I) (6)
√
q(xt | x0 ) = N (xt | γt x0 , (1 − γt )I ) (7)

Proof (sketch): Write out the densities and compute.

• Note that for large t, q(xt | x0 ) ≈ N (0, I)
t=0 t = 50 t = 100 t = 200 t = 300

15
Denoising Diffusion Models

“Creating noise from data is easy; creating data from noise is

generative modeling”3 .

3 Song et al., Score-Based Generative Modeling through Stochastic Differential

Equations, ICLR 2021

16
Reverse Process

Generate data by reversing the diffusion process

• Sample xT ∼ N (0, I)
• Iteratively sample

xt−1 ∼ q(xt−1 | xt ) t = T, T − 1, . . . , 1 (8)

17
Reverse Process

Generate data by reversing the diffusion process

• Sample xT ∼ N (0, I)
• Iteratively sample

xt−1 ∼ q(xt−1 | xt ) t = T, T − 1, . . . , 1 (9)

• Problem:
q(xt | xt−1 )q(xt−1 )
q(xt−1 | xt ) = (10)
q(xt )
• Know forward transitions, but marginals are intractable
• Variational approximation:

q(xt−1 | xt ) ≈ pθ (xt−1 | xt ) (11)

= N (xt−1 | µθ (xt , t), Σθ (xt , t)) (12)

18
Reverse Process

Variational approximation:
q(xt−1 | xt ) ≈ pθ (xt−1 | xt ) (13)
= N (xt−1 | µθ (xt , t), Σθ (xt , t)) (14)

How can we train such a model?

• Want to maximize the model likelihood pθ (x0 )
• Can think of pθ (xt−1 | xt ) as the decoder in a VAE

Figure 8: Image credit: Calvin Luo 19

Loss Analysis

The usual ELBO (treating x1:T as latents) is

[ ]
pθ (x0 , x1:T )
log pθ (x0 ) ≥ Ex1:T ∼q(−|x0 ) log (15)
q(x1:T | x0 )
= Ex1:T ∼q(−|x0 ) [log pθ (x0 | x1 )] − KL [q(x1:T | x0 ) || pθ (x1:T )]

Let’s analyze this to get something we can compute

Reminder: KL is the Kullback-Liebler divergence; a “distance”

between probability distributions

∫
q(x)
KL [q(x) || p(x)] = q(x)dx (16)
p(x)
20
KL Chain Rule

Chain rule for the KL divergence:

KL [p(x, y) | q(x, y)] = KL [p(x) || q(x)] + Ex∼p(x) KL [p(y|x) || q(y|x)] (17)

Proof (sketch):
Decompose the joint distributions into a product of marginal and
conditional distributions. Plug into the definition of the KL
divergence and compute.

21
KL Chain Rule

Chain rule for KL divergences:

KL [p(x, y) | q(x, y)] = KL [p(x) || q(x)] + Ex∼p(x) KL [p(y|x) || q(y|x)] (18)

Apply to the chain rule to condition on xT :

log pθ (x0 )
≥ Eq [log qθ (x0 | x1 )] − KL [q(x1:T | x0 ) || pθ (x1:T )]
= Eq [log qθ (x0 | x1 )] − KL [q(xT | x0 ) || pθ (xT )]
−Eq KL [q(x1:T−1 | x0 , xT ) || pθ (x1:T−1 | xT )]

22
KL Chain Rule

Chain rule for KL divergences:

KL [p(x, y) | q(x, y)] = KL [p(x) || q(x)] + Ex∼p(x) KL [p(y|x) || q(y|x)] (19)

Apply to the chain rule to condition on xT :

log pθ (x0 )
≥ Eq [log qθ (x0 | x1 )] − KL [q(x1:T | x0 ) || pθ (x1:T )]
= Eq [log qθ (x0 | x1 )] − KL [q(xT | x0 ) || pθ (xT )]
−Eq KL [q(x1:T−1 | x0 , xT ) || pθ (x1:T−1 | xT )]

Repeat to condition on xT−1 , xT−2 , . . . , x1 :

[
log pθ (x0 ) ≥Eq log pθ (x0 | x1 )−

∑
T ]
KL [q(xT | x0 ) || pθ (xT )] − KL [q(xt−1 | xt , x0 ) || pθ (xt−1 | xt )]
t=2

23
Loss Analysis

Three types of terms appear in the loss:

LT := KL [q(xT | x0 ) || pθ (xT )] (20)

L0 := Eq [log pθ (x0 | x1 )] (21)

Lt−1 := Eq KL [q(xt−1 | xt , x0 ) || pθ (xt−1 | xt )] (22)

24
Loss Analysis

LT := KL [q(xT | x0 ) || pθ (xT )] (23)

This measures the error at the end of the forward process, i.e. t = T.

• We typically choose pθ (xT ) = N (0, I)

• Note q(xT | x0 ) ≈ N (0, I) for T sufficiently large
• Hence, LT is negligible and is typically ignored during training

25
Loss Analysis

L0 := Eq [log pθ (x0 | x1 )] (24)

Measures the error at the end of the backwards process, i.e. t = 0.

• This is essentially a decoder log-likelihood

• Analogous to the reconstruction term in a VAE
• Cheap to compute

26
Loss Analysis

Lt−1 := Eq KL [q(xt−1 | xt , x0 ) || pθ (xt−1 | xt )] (25)

Measures the error between the at intermediate steps between

1. The model’s reverse transitions pθ (xt−1 | xt )

2. The true reverse transitions q(xt−1 | xt , x0 )

Important note: the true reverse transitions are conditioned on x0

• q(xt−1 | xt ) is intractible
• ... but we’ll see q(xt−1 | xt , x0 ) is known!

27
Loss Analysis

By Bayes’ rule:

q(xt | xt−1 , x0 )q(xt−1 | x0 )

q(xt−1 | xt , x0 ) = (26)
q(xt | x0 )

The right-hand side only involves the forward process

• ... so everything is known and Gaussian

After a tedious but straightforward calculation:

q(xt−1 | xt , x0 ) = N (µq (xt , x0 ), σq2 (t) I) (27)

√ √
γt−1 βt 1 − βt (1 − γt−1 ) 1 − γt−1
µq (xt , x0 ) = x0 + xt σq2 (t) = βt
1 − γt 1 − γt 1 − γt

28
Model Parametrization

The story so far:

Lt−1 := Eq KL [q(xt−1 | xt , x0 ) || pθ (xt−1 | xt )] (28)

q(xt−1 | xt , x0 ) = N (µq (xt , x0 ), σq2 (t) I) (29)

How should we parametrize the model pθ (xt−1 | xt )?

29
Model Parametrization

The story so far:

Lt−1 := Eq KL [q(xt−1 | xt , x0 ) || pθ (xt−1 | xt )] (30)

q(xt−1 | xt , x0 ) = N (µq (xt , x0 ), σq2 (t) I) (31)

How should we parametrize the model pθ (xt−1 | xt )?

Since q(xt−1 | xt , x0 ) is Gaussian, let’s assume pθ (xt−1 | xt ) is too:

pθ (xt−1 | xt ) = N (µθ (xt , t), Σθ (xt , t)) (32)

• Now, we parametrize µθ (xt , t) and Σθ (xt , t)

30
Model Parametrization

Lt−1 := Eq KL [q(xt−1 | xt , x0 ) || pθ (xt−1 | xt )] (33)

( )
q(xt−1 | xt , x0 ) = N µq (xt , x0 ), σq2 (t) I (34)
pθ (xt−1 | xt ) = N (µθ (xt , t), Σθ (xt , t)) (35)

Let’s make our lives easy and set

Σθ (xt , t) = σq (t)2 I (36)

The KL between Gaussians has a closed form:

[ ]
1 2
Lt−1 = Eq ||µθ (xt , t) − µq (xt , x0 )||2 + C (37)
2σq2 (t)

31
Model Parametrization

[ ]
1 [ ]
Lt−1 = Eq 2
||µθ (xt , t) − µq (xt , x0 )||22 (38)
2σq (t)

Variational mean µθ needs to predict the denoised mean µq .

How should we parametrize µθ (xt , t)?

• Most straightforward: just have network try to predict µq , since

this is known
• Can we do better?

32
Model Parametrization: Data Prediction

[ ]
1 [ ]
Lt−1 = Eq 2
||µθ (xt , t) − µq (xt , x0 )||22 (39)
2σq (t)
Idea: we can exploit the structure of µq to obtain a better
parametrization

√ √
γt−1 βt 1 − βt (1 − γt−1 )
µq (xt , x0 ) = x0 + xt
1 − γt 1 − γt
Since the model has xt as input, we can parametrize via

√ √
γt−1 βt 1 − βt (1 − γt−1 )
µθ (xt , t) = xθ (xt , t) + xt (40)
1 − γt 1 − γt
i.e. network needs to predict noise-free input from xt :
xθ (xt , t) ≈ x0 (41)
33
Model Parametrization: Data Prediction

[ ]
1 [ ]
Lt−1 = Eq 2
||µθ (xt , t) − µq (xt , x0 )||22 (42)
2σq (t)

√ √
γt−1 βt 1 − βt (1 − γt−1 )
µθ (xt , t) = xθ (xt , t) + xt (43)
1 − γt 1 − γt

xθ (xt , t) ≈ x0 (44)

Loss simplifies to
[ ]
Lt−1 = Eq Ct ||xθ (xt , t) − x0 ||2 (45)

1 γt−1 βt2
Ct = (46)
2σq (t) (1 − γt )2
2

34
Model Parametrization: Noise Prediction

[ ]
1 [ ]
Lt−1 = Eq 2
||µθ (xt , t) − µq (xt , x0 )||22 (47)
2σq (t)
An alternative parametrization
√ √
Since xt = γt x0 + 1 − γt ϵ for ϵ ∼ N (0, I):
√ √
γt−1 βt 1 − βt (1 − γt−1 )
µq (xt , x0 ) = x0 + xt
1 − γt 1 − γt
( )
1 βt
=√ xt − √ ϵ
1 − βt 1 − γt

We can thus parametrize µθ as:

( )
1 βt
µθ (xt , t) = √ xt − √ ϵθ (xt , t) (48)
1 − βt 1 − γt

i.e. network tries to predict noise added to xt

ϵθ (xt , t) ≈ ϵ (49)
35
Denoising Diffusion Models: Noise Prediction

This parametrization results in a fairly simple and interpretable loss:

[ ]
Lt−1 = Eϵ Ct ||ϵ − ϵθ (xt , t)||2 ϵ ∼ N (0, I) (50)

• Ct is a time-dependent constant; often dropped during training

for simplicity
βt2
Ct = (51)
2σq2 (t)(1 − βt )(1 − γt )
• ϵθ (xt , t) is a network that tries to predict the added noise from
the noisy input – i.e. it is denoising

36
Denoising Diffusion Models: Training

Putting everything together:

[ ]
∑
T
2
L = Eq log pθ (x0 | x1 ) − Ct Eϵ ||ϵ − ϵθ (xt , t)|| (52)
t=2

• Ct is ignored
• Likelihood term is assumed to be Gaussian
√ √
• Recall xt = γt x0 + 1 − γt ϵ – pseudocode uses αt = γt 37
Denoising Diffusion Models: Sampling

Sampling:

( )
1 βt
µθ (xt , t) = √ xt − √ ϵθ (xt , t) (53)
1 − βt 1 − γt
pθ (xt−1 | xt ) = N (µθ (xt , t), σq2 (t)I) (54)

• Notation: αt = 1 − βt and σt = σq (t)

38
Denoising Diffusion Models

Some practical details:

• For images, ϵθ (xt , t) is typically implemented via the U-Net
architecture
• Time input t is discrete integer – usually handled via (learnable)
embeddings
• Can tune forward process: number of steps T, variance schedule
βt

Figure 9: Image credit: Arash Vahdat

39
Denoising Diffusion Models

Some samples from a trained DDPM model

Figure 10: [Ho et al., DDPM, 2020]

40
Conditional Diffusion Models

41
Conditional Diffusion Models

Conditioning information c, e.g.

• text (embedding)
• class label
• image(s)

Condition reverse chain on c:

∏
T
pθ (x0:T | c) = pθ (xT ) pθ (xt−1 | xt , c) (55)
t=1

Loss can be derived in an analogous way:

[
log pθ (x0 |c) ≥Eq log pθ (x0 | x1 , c) − KL [q(xT | x0 ) || pθ (xT )]

∑
T ]
− KL [q(xt−1 | xt , x0 ) || pθ (xt−1 | xt , c)]
t=2

42
Conditional Diffusion Models

Condition reverse chain on c:

∏
T
pθ (x0:T | c) = pθ (xT ) pθ (xt−1 | xt , c) (56)
t=1

Loss can be derived in an analogous way:

[
log pθ (x0 |c) ≥Eq log pθ (x0 | x1 , c) − KL [q(xT | x0 ) || pθ (xT )]

∑
T ]
− KL [q(xt−1 | xt , x0 ) || pθ (xt−1 | xt , c)]
t=2

• Basic idea still holds for conditional models

• Challenge: building architectures to best make use of c

43
Conditional Diffusion Models

How do you model

pθ (xt−1 | xt , c)? (57)

Some examples:

• Scalar c (e.g. class labels, time): pass through small MLP; mix
with hidden layers

Figure 11: Image credit: Arash Vahdat

44
Conditional Diffusion Models

How do you model

pθ (xt−1 | xt , c)? (58)

Some examples:
• Image c: concatenate channel-wise with xt
• Can be easily combined with scalar information

Figure 12: [Saharia et al., Image Super-Resolution via Iterative Refinement, 2021]
45
Conditional Diffusion Models

Case study: Imagen text-to-image model

• Text prompt embedded into a latent space (via T5)
• Cascaded image-to-image super resolution models

Figure 13: [Saharia et al., Photorealistic Text-to-Image Diffusion Models..., 2022]]

46
Connections to Score-Based Models

47
Score-Based Models

Tweeedie’s Formula (1956):

z ∼ N (µz , Σz ) (59)
E[µz | z] = z + Σz ∇z log p(z) (60)

Given a sample z from a Gaussian, our best guess for the mean is to
perturb z in the direction that most increases the log density.

• The gradient ∇x log p(z) is called the score of p(z)

48
Score-Based Models

Tweeedie’s Formula:

z ∼ N (µz , Σz ) (61)
E[µz | z] = z + Σz ∇z log p(z) (62)

Suppose we have a noisy measurement

z=x+ϵ ϵ ∼ N (0, Σ) (63)

Then Tweedie’s formula says:

∫
E[x | z] = xp(x | z)dx = z + Σ∇x log p(z) (64)

If you know ∇z log p(z), you don’t need to know p(x | z)!

49
Score-Based Models

Tweeedie’s Formula:

z ∼ N (µz , Σz ) =⇒ E[µz | z] = z + Σz ∇z log p(z) (65)

Recall our forward process:

√
q(xt | x0 ) = N (xt | γt x0 , (1 − γt )I ) (66)
√ √
xt = γt x0 + 1 − γt ϵ ϵ ∼ N (0, I) (67)

By Tweedie’s formula:
1 ( √ )
E[x0 | xt ] = √ xt + 1 − γt ∇ log p(xt ) (68)
γt

50
Score-Based Models

Recall our setup:

Lt−1 := Eq KL [q(xt−1 | xt , x0 ) || pθ (xt−1 | xt )] (69)

q(xt−1 | xt , x0 ) = N (µq (xt , x0 ), σq2 (t) I) (70)
pθ (xt−1 | xt ) = N (µθ (xt , t), Σθ (xt , t)) (71)

By Tweedie’s formula (plug in for x0 ):

√ √
γt−1 βt 1 − βt (1 − γt−1 )
µq (xt , x0 ) = x0 + xt (72)
1 − γt 1 − γt
1 βt
=√ xt + √ ∇ log p(xt ) (73)
1 − βt 1 − βt
Thus we have an alternative parametrization:
1 βt
µθ (xt , t) = √ xt + √ sθ (xt , t) (74)
1 − βt 1 − βt
sθ (xt , t) ≈ ∇ log p(xt ) (75)
51
Score-Based Models

Further connections:

∫
sθ (xt , t) ≈ ∇ log p(xt ) = q(x0 )∇ log q(xt | x0 )dx0 (76)
∫ ( )
xt − x0
= q(x0 ) − dx0 (77)
1 − γt
( )
ϵ
= −Ex0 √ ϵ ∼ N (0, 1) (78)
1 − γt
ϵ
= −√ (79)
1 − γt

That is:
1
sθ (xt , t) ≈ − √ ϵθ (xt , t) (80)
1 − γt
Predicting the score is the same (up to a time-dependent constant) as
predicting the noise
52
Score-Based Models

Thus an alternative form of the loss is:

[ ]
Lt−1 = Eq C′t ||sθ (xt , t) − ∇ log p(xt )||22 (81)

i.e. we can predict the score rather than the added noise
Note that ∫
∇ log p(xt ) = q(x0 )∇p(xt | x0 )x0 (82)

is intractable as written

• Requires specialized techniques for score-matching

• Beyond the scope of this lecture

53
Conditional Diffusion Models

Condition reverse chain on c:

∏
T
pθ (x0:T | c) = pθ (xT ) pθ (xt−1 | xt , c) (83)
t=1

Loss can be derived in an analogous way:

[
log pθ (x0 |c) ≥Eq log pθ (x0 | x1 , c) − KL [q(xT | x0 ) || pθ (xT )]

∑
T ]
− KL [q(xt−1 | xt , x0 ) || pθ (xt−1 | xt , c)]
t=2

• Basic idea still holds for conditional models

• Challenge: building architectures to best make use of c
• Score-based models can be conditioned via guidance

54
Conclusions and Summary

55
Conclusions and Summary

Diffusion Generative Models are a class of deep generative models

that generate data by iterative denoising.

• Can be applied to a wide array of conditional and unconditional

generation tasks
• The forward process is a Markov chain that turns our data into
noise
• We learn to undo this procedure via a variational approximation
to the time-reversed chain

56
Conclusions and Summary

Diffusion Generative Models are a class of deep generative models

that generate data by iterative denoising.

There are many complementary perspectives on diffusion models:

• Hierarchical VAEs; Latent variable models

• From xt , predicting:
• Denoised input x0
• Added noise ϵ
• Score ∇ log p(xt )

57
Conclusions and Summary

Not covered today:

• A lot!
• Continuous-time perspectives via Stochastic Differential
Equations (SDEs)
• Improvements to forward process [Kingma et al., Variational
Diffusion Models, NeurIPS 2021]
• Techniques to speed up generation [Song et al., Denoising
Diffusion Implicit Models, ICLR 2021]
• Conditional generation methods [Ho et al., Classifier Free
Guidance, 2022]

58
Additional Resources

• CVPR 2022 tutorial:

[Link]
• Calvin Luo’s blog: [Link]/2022/08/26/
[Link]
• Yang Song’s blog: [Link]/blog/2021/score/

59
 Thanks!
gavin.k@[Link]
[Link]

60