Introduction to

Deep Generative Models

(Mô hình tạo sinh sâu)
 2
Contents

¡ Introduction

¡ Probabilistic models
¡ Generative models
¡ Variational auto-encoder
¡ Generative Adversarial Networks
Some successes: Text-to-image (2022) 3

¡ Draw pictures by descriptions

Midjourney

DALL-E 2
A bowl of soup

Imagen
Some successes: ChatGPT (2022) 4

¡ Human-level Chatting, Writing, QA,…

Why ChatGPT is
about to change
how you work,
like it or not?
- Forbes, 2/2023
Some successes: more 5

AlphaGeometry
≈ Olympiad-level student
 6
Generative Models

 Probabilistic models of data

 Sample: lấy mẫu dữ liệu (sinh/tạo ra dữ liệu)

 Evaluate likelihood: tính likelihood của dữ liệu cho trước

 Train: huấn luyện

 Representation: biểu diễn mới

 What if all we care about is sampling?

 Not in the training data, but the novel samples.
 7

Probabilistic models
Introduction
 8
Probabilistic model

 Our assumption on how the data samples were generated

(giả thuyết của chúng ta về quá trình mà các mẫu dữ liệu đã được sinh
ra như thế nào)
 Example: how a sentence is generated?
 We assume our brain does as follow:
 First choose the topic of the sentence
 Generate the words one-by-one to form the sentence
 How will TIM be drawn?
1. 2. 3. 4.

8. 7. 6. 5.

[Link]
 9
Probabilistic model

 A model sometimes consists of

 Observed variable (e.g., 𝒙) which models 𝛼 𝜙
the observation (data instance)
(biến quan sát được)
z x
 Hidden variable which describes the
N
hidden things (e.g., 𝑧, 𝜙)
(biến ẩn)
 Relations between the variables
 Each variable follows some probability distribution
(mỗi biến tuân theo một phân bố xác suất nào đó)
 10
Different types of models

¡ Probabilistic graphical model (PGM): Graph + Probability Theory

(mô hình đồ thị xác suất)
𝛼 𝜙
 Each vertex represents a random variable,
grey circle means “observed”,
white circle means “latent” z x
N
 Each edge represents the conditional
dependence between two variables

¡ Latent variable model: a PGM which has at least one latent variable

¡ Generative model: a model that enables us to generate data instances

 11
Univariate normal distribution

¡ We wish to know the average height of a person

 We had collected a dataset from 10 people in Hanoi:
D = {1.6, 1.7, 1.65, 1.63, 1.75, 1.71, 1.68, 1.72, 1.77, 1.62}

¡ Let x denote the random variable that represents the height of a person

¡ Assumption: x follows a Normal distribution (Gaussian) with the following

probability density function (PDF)
4
. 3 637 1
𝒩 𝑥 𝜇, 𝜎 + ) = 𝑒 151
+/01

 where {𝜇, 𝜎 + } are the mean and variance

¡ Note:

 𝒩 𝑥 𝜇, 𝜎 + ) represents the class of normal distributions

x
+
 This class is parameterized by 𝜽 = (𝜇, 𝜎 )

¡ Learning: we need to know specific values of {𝜇, 𝜎 +} 𝜇 𝜎+

 12
PGM: some well-known models

¡ Gaussian mixture model (GMM)

 Modeling real-valued data

¡ Latent Dirichlet allocation (LDA)

 Modeling the topics hidden in textual data

¡ Hidden Markov model (HMM)

 Modeling time-series, i.e., data with time stamps or sequential nature

¡ Conditional Random Field (CRF)

 for structured prediction

¡ Deep generative models

 Modeling the hidden structures, generating artificial data

 13
Probabilistic model: inference & learning

 Inference for a given instance 𝒙<

(Suy diễn/phán đoán đối với một quan sát cho trước)
 Recovery of the local variable (e.g., 𝑧< ), or 𝛼 𝜙
 The distribution of the local variables
(e.g., 𝑃 𝑧< 𝜙, 𝒙< ))
 Example: for GMM, we want to know 𝑧< z x
indicating which Gaussian did generate 𝒙< N

 Learning (estimation)
(Học/ước lượng mô hình)
 Given a training dataset, estimate the joint distribution of the variables
 E.g., estimate the density function 𝑝 𝜙, 𝑧., … , 𝑧< , 𝒙., … , 𝒙< 𝛼)
 E.g., estimate 𝑃 𝒙., … , 𝒙< 𝛼)
 E.g., estimate 𝛼
 Inference of local variables is often needed
 14
Generative model: sampling

 Sampling data
 Make novel data samples, given a trained model
(tạo ra dữ liệu mới từ mô hình đã có)

 Application:
 Entertainment (ngành giải trí): videos, images, musics, …
 Limited resources: khi khả năng thu thập được ít mẫu dữ liệu
 Fashion: tạo mẫu quần/áo thời trang
 Design: tạo mẫu trang thiết bị mới
 Materials: tạo các vật liệu mới
 …
 15

Generative models
Learning
 16
Learning a generative model

¡ Given a training set of examples, e.g., images of dogs

¡ We want to learn a probability distribution P(x) over images x
such that
 Generation: If we sample 𝒙𝑛𝑒𝑤 ∼ 𝑃(𝒙), 𝒙𝑛𝑒𝑤 should look like a dog
(sampling)
 Density estimation: P(x)
 Unsupervised representation learning: We should be able to learn what
these images have in common, e.g., ears, tail, etc. (features)
 17
The sampling distribution

¡ Dataset D = {x1, x2, …, xm}

¡ Hardness of the learning problem: Impossible

¡ P(x) in the space of all probability distributions

¡ In practice, we often find a 𝑃C (𝒙) to approximate 𝑃(𝒙)

𝑝C
 18
Hypothesis space

¡ Ussually, we can choose a restricted set ℋ of distributions 𝑃

¡ Parameterized by 𝜃 ∈ Θ

¡ A learner must find one 𝑃C ∈ ℋ ℋ

¡ Hypothesis space (model family):

a set ℋ of distributions, providing candidates for a learner
¡ Represents prior knowledge about a task

¡ Represents our inductive bias or preference

¡ Each 𝑃C is often called a “model”

¡ Gaussian family:
ℋ = 𝑃C : 𝑃C is the normal distribution with 𝜃 = 𝜇, 𝜎 , 𝜇 ∈ ℝ, 𝜎 ∈ ℝZ
 19
Learning goal

¡ Find a model 𝑃C that precisely captures the distribution P from

which our data was sampled
¡ Intractability:
¡ P(x) is in the space of all probability distributions
¡ The sampled data set is limited
¡ Computational reasons
¡ We want to select Pθ to be the ”best” approximation to the
underlying distribution P 𝑃

¡ What is “best”?
𝑃C
¡ Depends on specific task of interest

ℋ
 20
Learning as density estimation

¡ We want to learn the full distribution so that later we can answer

any probabilistic inference query
¡ In this setting we can view the learning problem as density
estimation
¡ We want to construct 𝑃C as ”close” as possible to P
(recall we assume we are given a dataset D of samples from P)

How do we
evaluate
”closeness”?
 21
KL-divergence

¡ How should we measure distance between distributions?

¡ The Kullback-Leibler divergence (KL-divergence) between two
distributions P and Q is defined as
𝑝(𝒙)
𝐾𝐿(𝑃| 𝑄 = 𝔼𝒙~a(𝒙) log
𝑞(𝒙)
¡ where p(𝒙) and q(𝒙) represents the densities of P and Q, respectively
¡ Note that:
¡ 𝐾𝐿(𝑃| 𝑄 ≥ 0 for any P and Q, and 𝐾𝐿(𝑃| 𝑃 = 0
¡ 𝐾𝐿(𝑃| 𝑄 ≠ 𝐾𝐿(𝑄| 𝑃
¡ It measures the loss (in bits) when describing distribution P by Q.
 22
Learning: a revisit

¡ We want to construct Pθ as ”close” as possible to P

(Given a dataset D of samples from P)
¡ Closeness by KL: 𝑃
𝑝(𝒙)
𝐾𝐿(𝑃| 𝑃C = 𝔼𝒙~a(𝒙) log 𝑃C∗
𝑝C (𝒙)
¡ Learning by minimizing 𝐾𝐿(𝑃| 𝑃C
ℋ
𝜃 ∗ = argmin 𝐾𝐿(𝑃| 𝑃C
C∈j
¡ Find the parameter 𝜃 ∗ that minimizes 𝐾𝐿(𝑃| 𝑃C
¡ 𝜃 ∗ provides the minimal loss when compressing P by 𝑃C∗
 23
Expected log-likelihood

¡ We can rewrite
𝑝 𝒙
𝐾𝐿(𝑃| 𝑃C = 𝔼𝒙~a 𝒙 log = 𝔼𝒙~a 𝒙 log 𝑝 𝒙 − 𝔼𝒙~a(𝒙) log 𝑝C (𝒙)
𝑝C 𝒙
¡ The first term does not depend on 𝜃
¡ Minimizing 𝐾𝐿 is equivalent to maximizing the Expected log-
likelihood 𝔼𝒙~a(𝒙) log 𝑝C (𝒙)
¡ Learning can be done by Maximum Likelihood Estimation (MLE)
𝜃 ∗ = argmax 𝔼𝒙~a(𝒙) log 𝑝C (𝒙)
C∈j

¡ In general, we do not know P

¡ So, we cannot access to the objective
 24
Maximum likelihood

¡ We approximate the expected log-likelihood 𝔼𝒙~a(𝒙) log 𝑝C (𝒙) by

1
𝔼𝒙∈𝑫 log 𝑝C (𝒙) = p log 𝑝C (𝒙)
𝑚
𝒙∈𝑫
¡ Sometimes known as Empirical log-likelihood
(note the similarity with empirical loss in ML)
¡ MLE is the formulated as
1
𝜃∗ = argmax p log 𝑝C (𝒙)
C∈j 𝑚
𝒙∈𝑫
¡ This is equivalent to maximizing the likelihood 𝑃 𝒙. , … , 𝒙q = ∏q
st. 𝑃 𝒙s for

i.i.d. samples
 25
MLE: Gaussian example (1)

¡ We wish to estimate the height of a person in the world.

¡ Use a dataset D = {1.6, 1.7, 1.65, 1.63, 1.75, 1.71, 1.68, 1.72, 1.77, 1.62}
 Let x be the random variable representing the height of a person.

 Model: assume that x follows a Gaussian distribution with unknown mean 𝜇

and variance 𝜎 +

 Learning: estimate (𝜇, 𝜎) from the given data 𝑫 = {𝑥., … , 𝑥.u}.

¡ Let 𝑓(𝑥|𝜇, 𝜎) be the density function of the Gaussian family,

parameterized by (𝜇, 𝜎).
 𝑓(𝑥< |𝜇, 𝜎) is the likelihood of instance 𝑥< .

 𝑓(𝑫|𝜇, 𝜎) is the likelihood function of D.

¡ Using MLE, we will find

𝜇∗ , 𝜎∗ = arg max 𝑓(𝑫|𝜇, 𝜎)
7,0
 26

MLE: Gaussian example (2)

¡ i.i.d assumption: we assume that the data are independent and
identically distributed (dữ liệu được sinh ra một cách độc lập)
 As a result, we have 𝑃 𝑫 𝜇, 𝜎 = 𝑃 𝑥. , … , 𝑥.u 𝜇, 𝜎 = ∏.u
st. 𝑃 𝑥s 𝜇, 𝜎

¡ Using this assumption, MLE will be

.u
.u 1 . 1
3 1 6z 37
𝜇∗, 𝜎∗ = arg max w 𝑓 𝑥s 𝜇, 𝜎 = arg max w 𝑒 +0
7,0 st. 7,0
st.
2𝜋𝜎 +
.u
1 .
3 1 6z 37 1
= arg max log w 𝑒 +0 Log trick,
7,0
st.
2𝜋𝜎 + log ≝ ln
.u
1 +
= arg max p − + 𝑥s − 𝜇 − log 2𝜋𝜎 +
7,0 2𝜎
st.

¡ Using gradients (w.r.t 𝜇, 𝜎), we can find

1 .u 1 .u
𝜇∗ = p 𝑥s = 1.683, +
𝜎∗ = p (𝑥s −𝜇∗ )+ ≈ 0.0015
10 st. 10 st.
 27

Generative models
Approximation by
mixture models
Learning the data distribution 28

¡ Dataset D = {x1, x2, …, xm}

 Images about dogs
¡ Hardness of the learning
problem:
 P(x) is in the space of all
probability distributions
¡ In practice, we often find a
𝑃C (𝒙) to approximate 𝑃(𝒙)
¡ How to choose a good model
family?
 Gaussian family?
=> too simple
 29
Gaussian mixture model (GMM)

 GMM: we assume that the data are samples from K Gaussian

distributions.
 Each instance x is generated from one of those K Gaussians by the
following generative process:
 Take the component index 𝑧 ~ 𝐶𝑎𝑡𝑒𝑔𝑜𝑟𝑖𝑐𝑎𝑙(𝝓)
 Generate 𝒙 ~ 𝑁𝑜𝑟𝑚𝑎𝑙( 𝝁• , 𝜮• )

 The density function is

•
𝑞(𝒙|𝝁, 𝜮, 𝝓) = p 𝜙• 𝒩 𝒙 𝝁• , 𝜮• )
•t.

 𝝓 = (𝜙. , … , 𝜙• ) represents the weights of the Gaussians: ∑•

•t. 𝜙• = 1, 𝜙’ ≥ 0, ∀𝑗

. .
 Each Gaussian has density 𝒩 𝒙 𝝁, 𝜮) = exp − 𝒙 − 𝝁 ˜ 𝜮3. 𝒙 − 𝝁
•–—(+/𝜮) +

¡ Note: z is an unobserved (latent) variable, x is observable

 30
GMM: approximation ability

¡ The density 𝑞(𝒙|𝝁, 𝜮, 𝝓) = ∑•

•t. 𝜙• 𝒩 𝒙 𝝁• , 𝜮• )
¡ Gaussian model: 𝐾 = 1 component
¡ A larger K produces a more complex model Q

GMM with 2 components GMM with 3 components

¡ GMMs are universal approximators

¡ Any smooth density can be approximated arbitrarily well by a GMM
with enough components
Dalal, S. R., and W. J. Hall. "Approximating Priors by Mixtures of Natural Conjugate Priors." J.
of the Royal Statistical Society. Series B (Methodological), vol. 45, no. 2, 1983, pp. 278–286.
 31
Infinite GMM

 Mixture of an infinite number of Gaussians: we assume that the

data are samples from an infinite number of Gaussians
 Each instance x is generated from one of those Gaussians by the
following generative process:

 Choose 𝒛 ~ 𝑁𝑜𝑟𝑚𝑎𝑙(0, 𝑰) P(z)

 Generate 𝒙 ~ 𝑁𝑜𝑟𝑚𝑎𝑙(𝝁C (𝒛), 𝜮C (𝒛)) P(x|z)

 Where 𝝁C , 𝜮C are neural networks, parameterized by 𝜃

¡ Universal approximator?
¡ Each component is simple, but the marginal P(x) is very complex
 32

Variational auto-encoder
Variational inference,
Amortized inference,
Sampling
 33
Learning for GMM

¡ Learning by MLE:
1
𝜃∗ = argmax p log 𝑝C (𝒙)
C 𝑚
𝒙∈𝑫
. .
¡ where 𝑝C 𝒙 = ∑•
•t. 𝜙• exp − 𝒙 − 𝝁• ˜ 𝜮3.
• 𝒙 − 𝝁• , 𝜃 = (𝝓, 𝝁, 𝜮)
•–—(+/𝜮› ) +

¡ Evaluation of log 𝑝C (𝒙) is hard in general, since

log 𝑝C (𝒙) = log p 𝑝C (𝒙, 𝒛)

œ•• žŸ ¡¢•– £¤•¥– Ÿ¦ 𝒛
¡ E.g., for 𝒛 ∈ 0,1 .uu , the sum has 2100 terms

¡ It is even harder for more complex models

Approximation is needed
 34
Evidence Lower Bound

¡ Note
𝑞(𝒛) 𝑝C (𝒙, 𝒛)
log 𝑝C (𝒙) = log p 𝑝C (𝒙, 𝒛) = log p 𝑝 (𝒙, 𝒛) = log 𝔼¨(𝒛)
𝑞(𝒛) C 𝑞(𝒛)
𝒛∈𝓩 𝒛∈𝓩
¡ Since log is concave, Jensen Inequality suggests
𝑝C (𝒙, 𝒛) 𝑝C 𝒙, 𝒛
log 𝔼¨(𝒛) ≥ 𝔼¨ 𝒛 log = 𝔼¨ 𝒛 log 𝑝C 𝒙, 𝒛 − 𝔼¨ 𝒛 log 𝑞 𝒛
𝑞(𝒛) 𝑞 𝒛
¡ This is called the Evidence Lower Bound (ELBO)
¡ For any 𝑞 𝒛
log 𝑝C (𝒙) ≥ 𝐸𝐿𝐵𝑂
¡ For ELBO = 𝔼¨ 𝒛 log 𝑝C 𝒙, 𝒛 − 𝔼¨ 𝒛 log 𝑞 𝒛
¡ When 𝑞 𝒛 = 𝑝C 𝒛 𝒙
log 𝑝C (𝒙|𝜽) = 𝔼°±(𝒛|𝒙) log 𝑝C 𝒙, 𝒛 𝜽 − 𝔼°±(𝒛|𝒙) log 𝑝C 𝒛 𝒙 = 𝑬𝑳𝑩𝑶
 35
Variational inference

¡ When the posterior 𝑝C 𝒛 𝒙 is easy to compute, we can learn the

model by maximizing
1 1
p log 𝑝C (𝒙) = p 𝔼°± 𝒛 𝒙 log 𝑝C 𝒙, 𝒛 𝜽 − 𝔼°± 𝒛 𝒙 log 𝑝C 𝒛 𝒙
𝑚 𝑚
𝒙∈𝑫 𝒙∈𝑫
¡ E.g., for the case of GMM
¡ What if the posterior 𝑝C 𝒛 𝒙 is intractable to compute?
¡ Variational inference (VI):
¡ choose a family of simple distributions 𝑞¶ (𝒛),
parameterized by 𝜑 (variational parameters) 𝑝C
¡ then find 𝜑 ∗ so that 𝑞¶∗ (𝒛) is as close as
possible to 𝑝C 𝒛 𝒙
𝑞¶
 36
VI and KL

¡ Maximize the ELBO

q
1
p 𝔼¨¸ 𝒛 log 𝑝C 𝒙s , 𝒛 𝜽 − 𝔼¨¸ 𝒛 log 𝑞¶z 𝒛
𝑚 z z
st.
¡ given a training set D = {x1, x2, …, xm}

¡ Maximizing ELBO is equivalent to Minimizing KL, due to

log 𝑝C (𝒙) = 𝐸𝐿𝐵𝑂 + 𝐾𝐿(𝑞¶ 𝒛 ||𝑝C 𝒛 𝒙 )

¡ Jointly optimize over

¡ 𝜑. , … , 𝜑q (variational parameters)
¡ 𝜃 (model parameters)
 37
VI: some properties

¡ Pros: ¡ Variational inference (VI):

¡ choose a family of simple
¡ Easy to be used in a large class of models
distributions 𝑞¶ (𝒛),
¡ Efficient in practice parameterized by 𝜑

¡ Cons: ¡ find 𝜑∗ so that 𝑞¶∗ (𝒛) is as

close as possible to 𝑝C 𝒛 𝒙
¡ Hard to choose a good variational family
¡ When we do not know the explicit form for the posterior 𝑝C 𝒛 𝒙
¡ For inference, given model param 𝜃 and instance 𝒙, we estimate the
posterior 𝑝C 𝒛 𝒙 by solving an optimization problem:
max 𝔼¨¸ 𝒛 log 𝑝C 𝒙, 𝒛 𝜽 − 𝔼¨¸ 𝒛 log 𝑞¶ 𝒛 Expensive
¶

¡ Require too many variational parameters

¡ Each instance xi requires one specific 𝜑s  O(m) parameters
¡ GMM needs O(mKn 2) params, where K is #components, n is #dims
 38
Amortized inference

1 1
max p log 𝑝C (𝒙) ≥ max p L 𝒙s ; 𝜃, 𝜑
C 𝑚 C,¶4 ,…,¶º 𝑚
𝒙∈𝑫 𝒙z ∈𝑫

¡ Where L 𝒙s ; 𝜃, 𝜑 = 𝔼¨¸ 𝒛 log 𝑝C 𝒙s , 𝒛 𝜽 − 𝔼¨¸ 𝒛 log 𝑞¶z 𝒛

z z

¡ VI uses 𝜑s for each point xi.

¡ May not scale well with large datasets; prone to overfitting
¡ Amortization: we learn a single neural network 𝑓¼ : 𝒙 ⟼ 𝜑 that
maps each input x to a set of (good) variational parameters
¡ 𝑓¼ has a trainable parameter w
¡ For a given input xi, 𝑓¼ will produce the parameter 𝜑s = 𝑓¼ (𝒙s ) of the
variational distribution 𝑞¶z 𝒛
¡ Amortized inference: feed instance x to the trained network to
get the variational parameter 𝜑 = 𝑓¼ (𝒙)
¡ No optimization  cheap Kingma, D. P. & Welling, M. (2014).
Auto-Encoding Variational Bayes. ICLR.
 39
Learning with amortized inference

¡ We can use using stochastic gradient descent to solve

max p L 𝒙s ; 𝜃, 𝜑
C,¶4 ,…,¶º
𝒙z ∈𝑫

¡ Initialize 𝜃 u , 𝜑(u)
¡ At iteration 𝑗 ≥ 1:
¡ Randomly sample a data point xi from D
¡ Compute 𝛻C 𝐿 𝒙s ; 𝜃 ’3. , 𝜑 (’3.) and 𝛻¶ 𝐿 𝒙s ; 𝜃 ’3. , 𝜑 (’3.)
¡ Update 𝜃 ’ , 𝜑 (’) in the gradient direction

¡ How to compute the gradients?

¡ L 𝒙s ; 𝜃, 𝜑 = 𝔼¨¸ 𝒛 log 𝑝C 𝒙s , 𝒛 𝜽 − 𝔼¨¸ 𝒛 log 𝑞¶z 𝒛

z z

¡ The expectation complicates gradient computation for 𝜑

 40
Reparameterization trick

¡ Consider z being continuous, and we want to compute a gradient

with respect to 𝜑 of

𝔼¨¸ 𝒛 𝑟 𝒛 = ¿ 𝑞¶ 𝒛 𝑟 𝒛 𝑑𝒛

¡ Suppose 𝑞¶ 𝒛 = 𝒩 (𝝁, 𝜎 + 𝑰) is Gaussian with parameters 𝜑 = (𝝁, 𝜎)

¡ Since 𝒛~𝑞¶ 𝒛 , there exists representation 𝒛 = 𝝁 + 𝜎𝝐 where 𝝐~𝒩(0, 𝑰)
¡ We can write
𝔼𝒛~¨¸ 𝒛 𝑟 𝒛 = 𝔼𝝐~𝒩 (u,𝑰) 𝑟 𝝁 + 𝜎𝝐
𝛻¶ 𝔼¨¸ 𝒛 𝑟 𝒛 = 𝛻¶ 𝔼𝝐 𝑟 𝝁 + 𝜎𝝐 = 𝔼𝝐 𝛻¶ 𝑟 𝝁 + 𝜎𝝐
¡ Easy to estimate via Monte Carlo if r is differentiable w.r.t. 𝜑,
since 𝝐 is easy to sample
.
¡ 𝔼𝝐 𝛻¶ 𝑟 𝝁 + 𝜎𝝐 ≈ ∑•
’t. 𝛻¶ 𝑟 𝝁 + 𝜎𝝐’ , where 𝝐. , … , 𝝐• ~𝒩(0, 𝑰)
•
 41
Variational auto-encoder (VAE)

¡ Since 𝑞¶ 𝒛 approximates the posterior 𝑝C 𝒛 𝒙 , we can write it as

𝑞¶ 𝒛|𝒙 and

L 𝒙; 𝜃, 𝜑 = 𝔼¨¸ 𝒛|𝒙 log 𝑝C 𝒙, 𝒛 𝜃 − 𝔼¨¸ 𝒛|𝒙 log 𝑞¶ 𝒛|𝒙

= 𝔼¨¸ 𝒛|𝒙 log 𝑝C 𝒙, 𝒛 𝜃 − log 𝑝C 𝒛 + log 𝑝C 𝒛 − log 𝑞¶ 𝒛|𝒙
= 𝔼¨¸ 𝒛|𝒙 log 𝑝C 𝒙 𝒛 − 𝐾𝐿(𝑞¶ 𝒛|𝒙 ||𝑝C 𝒛 )
¡ Maximize L: maximize 𝑝C 𝒙 𝒛 and push 𝑞¶ 𝒛|𝒙 close to 𝑝C 𝒛
¡ Encoder:
¡ Maps each data point 𝒙 to a latent vector 𝒛Â , a sample from a Gaussian
(𝑞¶ 𝒛|𝒙 ) with parameter 𝜇, 𝜎 = 𝐸𝑛𝑐𝑜𝑑𝑒𝑟¶ (𝒙)
¡ Decoder:
¡ Reconstruct 𝒙
à from a latent vector 𝒛 , i.e., pick a sample from a
Gaussian (𝑝C 𝒙 𝒛Â ) with parameter 𝐷𝑒𝑐𝑜𝑑𝑒𝑟C (Â𝒛)
Kingma, D. P. & Welling, M. (2014). Auto-Encoding Variational Bayes. ICLR.
 42
VAE

L 𝒙; 𝜃, 𝜑 = 𝔼¨¸ 𝒛|𝒙 log 𝑝C 𝒙 𝒛 − 𝐾𝐿(𝑞¶ 𝒛|𝒙 ||𝑝C 𝒛 )

¡ Maximizing L:
¡ The first term encourages accurate reconstruction 𝒙
Ã≈𝒙
¡ The KL term encourages 𝒛Â to have a distribution similar to the prior 𝑝C 𝒛
¡ Training: SGD + reparameterization trick

Image from
Stefano Ermon

𝐸𝑛𝑐𝑜𝑑𝑒𝑟¶ 𝐷𝑒𝑐𝑜𝑑𝑒𝑟C
𝒙 𝒛Â Ã
𝒙
𝑝C (𝒙) 𝑞¶ 𝒛|𝒙 𝑝C 𝒙 𝒛
 43
VAE: some properties

¡ Pros:
¡ Efficient inference
¡ Flexible and expressive (Universal approximator)
¡ Good diversity of the synthetic samples
¡ Cons:
¡ Blur images VAE (2014)

VQ-VAE (2017)
 44

Generative
Adversarial Networks
Introduction

(Adapted from a lecture by Pieter Abbeel, Xi (Peter) Chen, Jonathan Ho, Aravind Srinivas, Alex Li, Wilson Yan, UC Berkeley, 2020)
 45
Generative Adversarial Networks

min max 𝔼𝒙~°ÇÈÉÈ log 𝐷 𝒙 + 𝔼𝒛~° 𝒛 log 1 − 𝐷 𝐺 𝒛

Å Æ

 Two player minimax game between generator (G) and discriminator (D)

 D tries to maximize the log-likehood for the binary classification

problem (D cố gắng cực đại hoá hàm log-likehood của bài toán phân loại nhị phân)
 Data: real (1)
 Generated: fake (0)

 G tries to minimize the log-probability of its samples being classified as

“fake” by the discriminator D
(G cố gắng cực tiểu hoá xác suất để D phân loại chính xác các mẫu dữ liệu do G tạo ra)
 46
Generative Adversarial Networks

Figure from NeurIPS 2016

GAN Tutorial (Goodfellow)
 47
Representation for the players

 D and G can be represented as two neural networks

 Discriminator:
𝐷 𝒙 = 𝑁𝑁(𝒙; 𝜃Ë )
 𝜃Ë is the weight of the neural network which takes a sample x as input.
 Output is a value in [0, 1].
(biểu diễn D bằng một mạng nơron với trọng số 𝜃Ë , với đầu vào x thì trả về
một giá trị thuộc [0, 1])
 Generator:
𝐺 𝒛 = 𝑁𝑁(𝒛; 𝜃Ì )
 𝜃Ì is the weight of the neural network which takes a noise z as input.
 z often follows a simple distribution, and is of low dimensionality.
 Output is a fake sample 𝒙 = 𝐺 𝒛 .
(biểu diễn G bằng một mạng nơron với trọng số 𝜃Ì , với đầu vào z thì trả về
một mẫu dữ liệu x)
 48
GANs: pseudocode for training

[Goodfellow et al., NeurIPS 2014]

 49
GAN

 See it in action: [Link]

 50
GAN samples from 2014

Figure from [Goodfellow et al., NeurIPS 2014]

 51
Generative Adversarial Networks

 Key pieces of GAN

 Fast sampling
 No inference
 Notion of optimizing directly for what you care about
– perceptual samples
 52
GAN: Bayes optimal discriminator

 What’s the optimal discriminator given generated and true distributions?

𝑉 𝐺, 𝐷 = 𝔼6~°ÇÈÉÈ log 𝐷 𝑥 + 𝔼•~° • log 1 − 𝐷 𝐺 𝑧

= ∫6 𝑝ËÏÐÏ 𝑥 log 𝐷 𝑥 𝑑𝑥 + ∫• 𝑝 𝑧 log 1 − 𝐷 𝐺 𝑧 𝑑𝑧

= ∫6 𝑝ËÏÐÏ 𝑥 log 𝐷 𝑥 𝑑𝑥 + ∫6 𝑝Ì 𝑥 log 1 − 𝐷 𝑥 𝑑𝑥

= ∫6 𝑝ËÏÐÏ 𝑥 log 𝐷 𝑥 + 𝑝Ì 𝑥 log 1 − 𝐷 𝑥 𝑑𝑥

Ï
∇Ò 𝑎 log 𝑦 + 𝑏 log 1 − 𝑦 = 0 ⟹ 𝑦∗ = ∀ (𝑎, 𝑏) ∈ ℝ+ \(0,0)
ÏZÖ

∗ °ÇÈÉÈ 6
⟹𝐷 𝑥 =
°ÇÈÉÈ 6 Z°Ø 6
 53
GAN: Bayes optimal discriminator

Data distribution
Discriminator

Model /
Generator
distribution

[Figure Source: Goodfellow

NeurIPS 2016 Tutorial on GANs]
 54
Generator Objective under Optimal Discriminator

𝑉 𝐺, 𝐷 ∗ = 𝔼6~°ÇÈÉÈ log 𝐷 ∗ 𝑥 + 𝔼•~°Ø log 1 − 𝐷 ∗ 𝑥

°ÇÈÉÈ 6 °ÇÈÉÈ 6
= 𝔼6~°ÇÈÉÈ log +𝔼•~°Ø log
°ÇÈÉÈ 6 Z °Ø 6 °ÇÈÉÈ 6 Z °Ø 6

°ÇÈÉÈ Z °Ø °ÇÈÉÈ Z °Ø
= − log 4 + 𝐾𝐿 𝑝ËÏÐÏ ∥ + 𝐾𝐿 𝑝Ì ∥
+ +

Û–Ü –Ü3ÝÞ¤ÜÜŸÜ ß¡£–àá–Üâ– ÛÝß Ÿ¦ °ÇÈÉÈ ¤Ü• °Ø ã u

(𝐾𝐿(p||q) is the Kullback-Leibler divergence between p and q)

𝑉 𝐺 ∗ , 𝐷 ∗ = − log 4 when 𝑝Ì = 𝑝ËÏÐÏ

 Given the Bayes-optimal D*, solving for G is equivalent to minimizing the

JSD divergence between pdata and pg
 55
Behaviors across divergence measures

[“A note on the evaluation of generative models” -- Theis, Van den Oord, Bethge 2015]
 56
KL and JSD

For given 𝑝(𝑥), find 𝑞∗(𝑥) that minimizes the divergence between them
 57
Mode covering vs Mode seeking: Tradeoffs

 For compression, one would prefer to ensure all points in the data
distribution are assigned probability mass.

 For generating good samples, blurring across modes spoils

perceptual quality because regions outside the data manifold are
assigned non-zero probability mass.

 Picking one mode without assigning probability mass on points

outside can produce “better-looking” samples.

 Caveat: More expressive density models can place probability mass

more accurately.
 58
Mode Collapse

Standard GAN training collapses when the true distribution is a

mixture of gaussians (Figure from Metz et al 2016)
 59
More?

¡ Diffusion models
¡…

Xiao, Z., Kreis, K., & Vahdat, A. Tackling the Generative

Learning Trilemma with Denoising Diffusion GANs. In ICLR, 2022.
60