Generative Models

Supervised Learning
• Data in form of input-label pairs
• {(x1, y1), {(x2, y2),… {(xn, yn) }
• Goal: Learn a mapping between input to label f(x) = y
Unsupervised Learning
• Only data, no labels
• Goal: Learn hidden or underlying structure of data
• Since no labels required, training data easy to obtain

Clustering Dimensionality Reduction

What are Generative Models
• Discriminative models: learn differentiating features between
various classes in a dataset
• Generative models: learn underlying distribution of each class in a
dataset
Introduction to Generative Models
• Intelligent agents are constantly generating, acquiring, processing
data
• images captured on phones
• text messages shared with friends
• graphs modeling interactions on social media
• videos recording important events, etc.
• Natural agents excel at discovering patterns, extracting knowledge,
and performing complex reasoning based on data they observe
• How can we build artificial learning systems to do the same?
Generative Models: Applications

Image super-resolution Image Colourization

Cross domain Image Translation Generating realistic face dataset

Generative Models
• Model learns good generalizable latent features
• Can be used for other downstream tasks
• Augment small datasets
• Challenges:
• A typical image from a modern phone camera has a resolution of approximately
700*1400 pixels
• Each pixel has three channels: R(ed), G(reen) and B(lue) and each channel can take a
value between 0 to 255 → number of possible images is given by 256700*1400*3 ~ 10800000
• In contrast, Imagenet, consists of only about 15 million images → learning a generative
model with such a limited dataset is a highly underdetermined problem
• Can hope the model learns the underlying structure directly from data
Introduction to Generative Models
• Given any kind of observed data, say 𝐷, as a finite set of samples
from an underlying distribution, say 𝑝𝑑𝑎𝑡𝑎

• Goal: approximate this data distribution, given access to dataset D

• If able to learn a good generative model:

• Can use learned model for downstream inference
• Generate data similar to samples in training set
Generative Models
Generative Models
• Primarily interested in parametric approximations to the data
distribution
• Summarize all information about dataset D in a finite set of parameters
• Parametric models scale more efficiently with large datasets
• But limited in the family of distributions they can represent

• In the parametric setting:

• Select parameters within a family of model distributions that minimizes some
notion of distance between model distribution and data distribution
Generative Models: How to learn?
• Goal: Minimize some notion of distance between 𝑝𝐷 (𝑥) and 𝑝𝜃 (𝑥)
• Given a dataset 𝑋 = {𝑥1 , 𝑥2 , … 𝑥𝑛 } comes from an underlying
distribution 𝑝𝐷 (𝑥)
• Consider an approximating distribution 𝑝𝜃 (𝑥) coming from a family of
distributions M
• Need to find best distribution in M, paramaterized by θ, which minimizes
distance between 𝑝𝐷 (𝑥) and 𝑝𝜃 (𝑥)

𝜃 ∗ = arg min 𝑑𝑖𝑠𝑡(𝑝𝜃 , 𝑝𝐷 )

𝜃𝜖𝑀
Generative Models: How to learn?
• What distance to choose?
• If KL-divergence is used:
𝜃 ∗ = arg min 𝑑𝑖𝑠𝑡(𝑝𝜃 , 𝑝𝐷 )
𝜃𝜖𝑀

𝜃 ∗ = arg min 𝐷𝐾𝐿 (𝑝𝐷 ∥ 𝑝𝜃 )

𝜃𝜖𝑀

𝜃 ∗ = arg min Ε𝑥~𝑝𝐷 [log 𝑝𝐷 (𝑥) − 𝑙𝑜𝑔 𝑝𝜃 𝑥 ]

𝜃𝜖𝑀

𝜃 ∗ = arg min Ε𝑥~𝑝𝐷 [−𝑙𝑜𝑔 𝑝𝜃 𝑥 ]

𝜃𝜖𝑀
(𝜃 parameters are independent of log 𝑝𝐷 (𝑥))
Maximum likelihood problem becomes minimizing negative log likelihood
𝑝
𝐷𝐾𝐿 𝑝 ∥ 𝑞 = 𝑝 𝑙𝑜𝑔 𝑞
Generative Models
• Discriminative classifier directly models
the posterior 𝑝 𝑦 𝑥
• What is probability of 𝑦 given 𝑥

• Generative classifier models joint

distribution 𝑝(𝑥, 𝑦)
• If data point 𝑥 assigned a class label 𝑦, what is
the probability of having generated this data
point

• Recall: posterior and joint are related as:

𝑝(𝑥, 𝑦) 𝑝 𝑥 𝑦 𝑝(𝑦)
𝑝 𝑦𝑥 = = Binary classification problem of classifying
𝑝(𝑥) 𝑝(𝑥) images of 1s and 0s
Generative Models
• Explicit Density Estimation • Implicit Density Estimation
• Write explicit function 𝑝 𝑥 = 𝑓(𝑥, 𝜃) • No estimation of explicit form of
Input: Image 𝑥 𝑝 𝑥
Output: Likelihood value for image • Aim only to sample images from
Parameter: Weights 𝜃
model
• Assign explicit likelihood to images
• High quality samples generated
• Can enable outlier detection but
compromise generated image • Ex. GANs
quality
• Ex. VAEs
Variational Autoencoder
Limitations of AE
• Maps input image via encoder to a deterministic latent code
• Decoder maps latent code to reconstruct input image
Limitations of AE
• AE creates latent space which is not continuous
• For generation, we need latent space from which we can smoothly
sample and get realistic reconstructions
• AE does not allow easy interpolation in latent space
Limitations of AE

AE has never seen such data….

Does not know what to create
Latent space of autoencoder
Limitations of AE

Expected

Smooth interpolation
Latent space
between set of vectors
 Variational Autoencoders (VAE)
• Proposed in 2013 by Diederik P. Kingma and Max Welling
• Provides a probabilistic manner for describing an observation in latent
space
• Trained to learn a mapping from input data to a probability distribution in a
lower-dimensional latent space
• Generate new samples from this distribution
• Commonly used in image and text generation tasks
• Latent space is continuous
• Decoder can generate new data points that interpolate among training data
points

Kingma, Diederik P. and Welling, Max , An Introduction to Variational Autoencoders, Foundations and Trends® in Machine Learning, volume 12, number 4, 307–392, 2019
Variational Auto Encoder (VAE)
• VAE provides a statistical manner for describing samples of dataset in
latent space
• Encoder outputs a probability distribution in bottleneck layer instead
of a single output value
AE vs. VAE
AE vs VAE
VAE
Architecture of VAE
• Mean and standard deviation the AE learns can put distributions of
different classes far apart
• Clusters might be isolated
• Interpolation might be difficult
• To ensure they are close to each other, impose a prior distribution
• For each category, mean and standard deviation should be close to prior
distribution - usually of zero mean and unit variance
• Try to minimize KL divergence (distance) between these two distributions
• Ensures distribution of different categories closely resemble target
Gaussian distribution – encodings clustered around center of latent space
Architecture of VAE
• Only KL loss - results in latent space encodings near center of
latent space
• No regard to similarity/dissimilarity of input samples
• Decoder cannot decode as structural property of classes is not present
• Requires reconstruction loss for maintain structural property of classes

• Use KL divergence along with reconstruction loss

• Only KL-divergence – encodings near center of latent space
• Only reconstruction loss – compactness of distributions is not maintained
 Optimizing reconstruction loss + KL divergence
Generates latent space which maintains similarity of nearby encodings
and globally is packed around center

[Link]
Architecture of VAE
• Reconstruction loss - compels model to accurately reconstruct
input
• Regularization term - encourages latent space to adhere to chosen
distribution
• Preventing overfitting and promoting generalization
• Iteratively adjust these parameters during training
• Optimized latent code encapsulates underlying features and
structures of data, facilitating precise reconstruction
• Probabilistic nature of latent space enables generation of novel
samples by drawing random points from learned distribution
Architecture of VAE
• Regularization done by enforcing distributions to be close to a
standard normal distribution (centered and reduced)
Architecture of VAE
• Encoder network takes raw input data and transforms it into a
probability distribution within latent space
• Latent code generated is a probabilistic encoding
• Allows VAE to express not just a single point in latent space but a distribution
of potential representations
• Decoder network takes a sampled point from latent distribution and
reconstructs it back into data space
• During training - model refines both encoder and decoder parameters
to minimize reconstruction loss
• Goal: to achieve accurate reconstruction and to regularize latent space,
ensuring it conforms to a specified distribution
 KL Divergence
• Assume there is a latent (unobserved) variable 𝑧 generating observed
variable 𝑥 z
• Our aim: Compute posterior probability:
𝑃 𝑥 𝑧 𝑃(𝑧)
𝑃 𝑧𝑥 = x
𝑃(𝑥)
• For 𝑥 of different categories, the probability of getting latent vector 𝑧 will be
different
• 𝑃 𝑥 = ‫𝑥(𝑃 ׬ ׮‬, 𝑧1 , 𝑧2 , . . 𝑧𝑑 )𝑑𝑧1 , 𝑑𝑧2 , . . 𝑑𝑧𝑑 → intractable as 𝑧 is multi-
dimensional and may have many dimensions
• Requires multiple integrations
 KL Divergence
• Assume a tractable distribution 𝑄 such that 𝑃 𝑧 𝑥 ~𝑄(z)
• 𝑄 can be Gaussian
• Minimize KL divergence of 𝑃 𝑧 𝑥 with respect to 𝑄(z)

𝑄(z)
Minimize KL(Q(𝑧)| 𝑃 𝑧 𝑥 = σ𝑥 𝑄 z 𝑙𝑜𝑔
𝑃(z|𝑥)

Check: KL(P ∥ Q) = −0.5 ∗ σ(1 + log(sigma2 ) − mu2 − sigma2 )

[Link]
872f77de0d79#:~:text=To%20calculate%20the%20KL%20divergence,%2C%20%2D1.36%2C%20%2D0.09%5D
KL Divergence
𝑄(𝑥)
• KL(Q(x)| 𝑃(x) = σ𝑥 𝑄 𝑥 𝑙𝑜𝑔
𝑃(𝑥)

x 0 1 2
P(x) 0.36 0.48 0.16
Q(x) 0.33 0.33 0.33
KL Divergence
𝑃(𝑥)
• KL(𝑃||𝑄) = σ𝑥 𝑃(𝑥)𝑙𝑜𝑔 𝑄(𝑥)
0.36 0.48 0.16
= 0.36 ∗ 𝑙𝑜𝑔 + 0.48 ∗ 𝑙𝑜𝑔 + 0.16 ∗ 𝑙𝑜𝑔
0.33 0.33 0.33
= 0.0414 x 0 1 2
P(x) 0.36 0.48 0.16
𝑄(𝑥)
• KL(𝑄||𝑃) = σ𝑥 𝑄(𝑥)𝑙𝑜𝑔 𝑃(𝑥) Q(x) 0.33 0.33 0.33
0.33 0.33 0.33
= 0.33 ∗ 𝑙𝑜𝑔 0.36 + 0.33 ∗ 𝑙𝑜𝑔 0.48 + 0.33 ∗ 𝑙𝑜𝑔 0.16
= 0.0375

→KL divergence is not symmetric – not a true distance metric

→KL divergence value ≥ 0
→If KL divergence is high: distributions are dissimilar, if low: distributions are similar
 KL Divergence
• Minimize: KL(Q(𝑧)| 𝑃 𝑧 𝑥
𝑄(𝑧)
• KL(Q(𝑧) ∥ P(z|x)) = − σ𝑧 𝑄 𝑧 𝑙𝑜𝑔
𝑃(𝑧|𝑥)
𝑄(𝑧)
= − 𝑧 𝑄 𝑧 𝑙𝑜𝑔
σ
𝑃 𝑥,𝑧 /𝑃(𝑥)
𝑄 𝑧 .𝑃(𝑥)
= − σ𝑧 𝑄 𝑧 {𝑙𝑜𝑔 }
𝑃(𝑥,𝑧)
= − σ𝑧 𝑄 𝑧 {𝑙𝑜𝑔𝑄(𝑧) + 𝑙𝑜𝑔𝑃(𝑥) − 𝑙𝑜𝑔 𝑃(𝑥, 𝑧)}
=
= − σ𝑧 𝑄 𝑧 𝑙𝑜𝑔Q(z) + σ𝑧 𝑄 𝑧 𝑙𝑜𝑔P(x, z) + 𝑙𝑜𝑔 𝑃(𝑥)

log 𝑃(𝑥) is independent of z and Σ 𝑄(𝑧|𝑥) over z is 1

 KL Divergence
KL(Q(𝑧) ∥ P(z|x))=− σ𝑧 𝑄 𝑧 𝑙𝑜𝑔Q(z) + σ𝑧 𝑄 𝑧 𝑙𝑜𝑔P(x, z) + 𝑙𝑜𝑔 𝑃(𝑥)
log 𝑃(𝑥) = KL(Q(𝑧) ∥ P(z|x))+σ𝑧 𝑄 𝑧 𝑙𝑜𝑔Q(z) − σ𝑧 𝑄 𝑧 𝑙𝑜𝑔P(x, z)

• Since 𝑥 is given, LHS is constant

• Aim → minimize KL(Q(𝑧) ∥ P(z|x))
• New aim → maximize σ𝑧 𝑄 𝑧 𝑙𝑜𝑔Q(z) − σ𝑧 𝑄 𝑧 𝑙𝑜𝑔P(x, z) (ELBO)
• Evidence = KL divergence + ELBO
• As KL divergence ≥ 0 → ELBO ≤ log P(x)
• Minimizing KL Divergence → Maximizing ELBO
Variational Lower Bound
• log 𝑃(𝑥) = KL(Q(𝑧) ∥ P(z|x)) + ELBO
• Key: ELBO is computable
• ELBO = σ𝑧 𝑄 𝑧 𝑙𝑜𝑔Q(z) − σ𝑧 𝑄 𝑧 𝑙𝑜𝑔P(x, z)

We control From our model

VAE: Network Realization

After VAE has been trained, Encoder is not required; Decoder can be used to generate data
[Link]
VAE: Reparameterization
• Problem: Have to sample z from latent vector distribution to be
fed to decoder
• Sampling operator breaks computation graph and hinders gradient
descent based optimization
• Randomly sample ε from a unit Gaussian; Shift randomly sampled ε by
latent distribution's mean μ and scale it by latent distribution's variance σ

Sample from Normal distribution;

sampling operator disassociated
from network
Reparameterizing
• Enables:
• Optimization of distribution parameters (through gradient descent)
• Still maintaining ability to randomly sample from that distribution
GANs
 Generative Adversarial Networks (GANs)*
• Use an adversarial process in which two models (“players”) are
trained simultaneously
• Also referred to as a two-player game
• Generator G: tries to capture the data distribution of real samples
• Discriminator D: estimates if sample comes from data distribution
• Goal of G: maximize probability of D making a mistake

*Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, Yoshua Bengio, Advances in
Neural Information Processing Systems, 2014
 Real Samples

𝑥~𝑝𝑑𝑎𝑡𝑎 (𝑥)

Discriminator
Network Real/
Fake
𝐷 𝑥; 𝜃
Generated
Generator Fake Samples Use the discriminator that
Network differentiates between real
𝑧~𝑝𝑧 (𝑧) samples 𝑥 ~ 𝑝𝑑𝑎𝑡𝑎 (class 1)
𝑥~𝑝𝑚𝑜𝑑𝑒𝑙 (𝑥) and 𝑥 ~ 𝑝𝑚𝑜𝑑𝑒𝑙 (class 0);
𝐺 𝑧; 𝜙 Assume sigmoid activation
Latent variable 𝑧 in discriminator
with simple prior Sample 𝑧 ~ 𝑝(𝑧),
𝑝(𝑧) pass it through
(e.g. Gaussian) generator
Generative Adversarial Networks (GANs)
• Goal:
• Build good sampler that allows to draw high quality samples from
𝑝𝑚𝑜𝑑𝑒𝑙 (𝑥)
• Not explicitly compute 𝑝(𝑥) but ensure 𝑝𝑚𝑜𝑑𝑒𝑙 (𝑥) is close to
𝑝𝑑𝑎𝑡𝑎 (𝑥)
• Outputs samples similar but not exactly same as training data
 Real Samples

𝑥~𝑝𝑑𝑎𝑡𝑎 (𝑥)

Discriminator
Network Real/
Fake
𝐷 𝑥; 𝜃
Generated
Generator Fake Samples
𝑧~𝑝𝑧 (𝑧) Network
𝑥~𝑝𝑚𝑜𝑑𝑒𝑙 (𝑥)
𝐺 𝑧; 𝜙

Define a mapping to data space as 𝑮 𝒛; 𝜽𝒈 for generator

Train generator G such that discriminator D misclassifies generated sample 𝑥 into class 1
→Can no more differentiate between 𝑥 ~ 𝑝𝑑𝑎𝑡𝑎 and 𝑥~𝑝𝑚𝑜𝑑𝑒𝑙 (𝑥)
 Real Samples

𝑥~𝑝𝑑𝑎𝑡𝑎 (𝑥)

Discriminator
Network Real/
Fake
𝐷 𝑥; 𝜃
Generated
Generator Fake Samples
𝑧~𝑝𝑧 (𝑧) Network
𝑥~𝑝𝑚𝑜𝑑𝑒𝑙 (𝑥)
𝐺 𝑧; 𝜙 Define a mapping 𝑫(𝒛; 𝜽𝒅 ) that outputs a
scalar - denoting probability of 𝒙 coming
from real data distribution 𝒑𝒅𝒂𝒕𝒂

Job of discriminator is to separate real from fake samples

 Real Samples

𝑥~𝑝𝑑𝑎𝑡𝑎 (𝑥)
Discriminator
Network Real/
Fake
𝐷 𝑥; 𝜃
Generated
Generator Fake Samples
𝑧~𝑝𝑧 (𝑧) Network
𝑥~𝑝𝑚𝑜𝑑𝑒𝑙 (𝑥)
𝐺 𝑧; 𝜙

Train G and D networks such that:

• Generator output realistic samples and
• Discriminator not able to distinguish between real samples and generated samples
→Generated sample distribution 𝑝𝑚𝑜𝑑𝑒𝑙 (𝑥) will be a good approximation of real data
distribution 𝑝𝑑𝑎𝑡𝑎 (𝑥)

Goal: Optimize the objective function max min ℱ(𝐺, 𝐷)

𝜃 𝜙
Training the Generator
• Goal of G: Output realistic samples Error
Function
• G does not have access to real data Update G

samples “Fake”
• GAN is “indirectly” trained through the “Real” Discriminator D frozen;
discriminator Cannot be
updated
Fake

𝑎𝑟𝑔 𝑚𝑖𝑛𝜙 log(1 − 𝐷𝜃 𝐺𝜙 𝑧 𝑖 ) Generator Updated G

Random Vector
 Training the Discriminator
Error Error
Function Function
Update D
Update D
“Fake”
“Fake”
“Real” “Real”
Discriminator
Discriminator

Real Fake

Real Generator
samples

Random Vector

𝑎𝑟𝑔 𝑚𝑎𝑥𝜃 [log(𝐷𝜃 𝑥 𝑖 + log(1 − 𝐷𝜃 𝐺𝜙 𝑧 𝑖 )

GANs
• Training objective:
𝑚𝑖𝑛𝜙 𝑚𝑎𝑥𝜃 (Ε𝑥~𝑝𝑑𝑎𝑡𝑎 [log 𝐷𝜃 (𝑥)] + Ε𝑧~𝑝𝑧 [log(1 − 𝐷𝜃 𝐺𝜙 𝑧 )])

Expectation E: losses averaged over a batch of samples

GANs Training Strategy
• Idea: First train D completely, then train G
• Problem: If D is initially very confident that samples from G are fake,
when 𝑥 is obtained from G;
𝐷𝜃 (𝑥) or 𝜎 𝑥 = 0 → log(1 − 𝐷𝜃 𝐺𝜙 𝑧 = log(1- 𝜎 𝑥 ) = 0
𝜕(log(1−𝜎 𝑥 )
→ =0
𝜕𝑥
→ G gets no gradients to train
• Solution: Alternate between Discriminator and Generator
Training the GAN
Step 1: Gradient Ascent on Discriminator (maximize w.r.t. θ):
𝑚𝑎𝑥𝜃 (Ε𝑥~𝑝𝑑𝑎𝑡𝑎 [log 𝐷𝜃 (𝑥)] + Ε𝑧~𝑝𝑧 [log(1 − 𝐷𝜃 𝐺𝜙 𝑧 )])
• D receives two kinds of inputs:
• Real data samples from the training dataset
• Data samples generated by G
• D analyzes each input and determines whether it is real or fake
Step 2: Gradient Descent on Generator (minimize w.r.t. φ):
𝑚𝑖𝑛𝜙 𝐸𝑧~𝑝𝑧 [log(1 − 𝐷𝜃 𝐺𝜙 𝑧 )]
• G takes random noise vector as input
• Contains random values and acts as starting point for G
• G transforms noise into a new data sample, like a generated image
Training the GAN
3. Adversarial Learning:
• D correctly classifies real data as real and fake data as fake - it strengthens
its ability
• G successfully fools the discriminator - D is penalized
4. Generator’s improvement:
• Over multiple iterations, G produces more convincing synthetic samples
5. Discriminator’s adaptation:
• D continuously refines its ability to distinguish real from fake
• Eventually it struggles to distinguish as training progresses
6. At this point, G can be used to generate high-quality synthetic data
for various applications
Training Strategy
• Why Discriminator first?
• At the beginning, the generator produces very poor samples
• The discriminator needs a reasonable ability to judge real vs fake
• If D is too weak, G will not get meaningful gradients

• Balance is Key
• If D becomes too strong → G gets vanishing gradients
• If G becomes too strong → D cannot learn properly
Optimal Discriminator
• Proposition: For any given generator G, the optimal discriminator D is:
∗
𝑝𝑑𝑎𝑡𝑎 (𝑥)
𝐷𝐺 𝑥 =
𝑝𝑑𝑎𝑡𝑎 𝑥 + 𝑝𝑚𝑜𝑑𝑒𝑙 (𝑥)

• If 𝑝𝑑𝑎𝑡𝑎 (𝑥) ≫ 𝑝𝑚𝑜𝑑𝑒𝑙 (𝑥), then 𝐷∗ 𝑥 ≈ 1 (high probability of being real)

• If 𝑝𝑑𝑎𝑡𝑎 (𝑥) ≪ 𝑝𝑚𝑜𝑑𝑒𝑙 (𝑥), then 𝐷∗ 𝑥 ≈ 0 (high probability of being fake)
• If 𝑝𝑑𝑎𝑡𝑎 𝑥 = 𝑝𝑚𝑜𝑑𝑒𝑙 (𝑥), then 𝐷∗ 𝑥 = 0.5
• Discriminator cannot distinguish between real and fake samples
• Indicates generator has learned to produce realistic samples
Global Optimality
• Theorem: Global minimum of virtual training criterion is achieved if and
only if 𝑝𝑚𝑜𝑑𝑒𝑙 = 𝑝𝑑𝑎𝑡𝑎 :

𝑉 𝐷𝐺∗ , 𝐺 = Ε𝑥~𝑝𝑑𝑎𝑡𝑎 [log 𝐷𝐺∗ (𝑥)] + Ε𝑥~𝑝𝑚𝑜𝑑𝑒𝑙 [log(1 − 𝐷𝐺∗ 𝑥 )]

𝑝𝑑𝑎𝑡𝑎 (𝑥) 𝑝𝑚𝑜𝑑𝑒𝑙 (𝑥)

= Ε𝑥~𝑝𝑑𝑎𝑡𝑎 log + Ε𝑥~𝑝𝑚𝑜𝑑𝑒𝑙 log
𝑝𝑑𝑎𝑡𝑎 𝑥 + 𝑝𝑚𝑜𝑑𝑒𝑙 (𝑥) 𝑝𝑑𝑎𝑡𝑎 𝑥 + 𝑝𝑚𝑜𝑑𝑒𝑙 (𝑥)

When 𝑝𝑚𝑜𝑑𝑒𝑙 = 𝑝𝑑𝑎𝑡𝑎 , where 𝐷𝐺∗ = 0.5 and 𝑉 𝐷𝐺∗ , 𝐺 = −𝑙𝑜𝑔4 ≈ 1.386
• Fundamental result in GAN theory
• Establishes the conditions under which the generator and discriminator reach equilibrium
Vanilla GAN
• Simplest type of GAN
• G and D, both built using multi-layer perceptrons (MLPs)
• Model optimizes using stochastic gradient descent (SGD)
• Serve as foundation for more advanced GAN models
• Often struggles with issues like mode collapse and unstable
training
Limitations of GANs
Failure to Converge
• Imbalance between generator and discriminator due to
adversarial training
• Any one network may learn too quickly
• If discriminator becomes too strong:
• Can reject generator outputs – leading to mode collapse and vanishing
gradients
• If generator becomes powerful:
• Might produce poor quality samples and discriminator does not provide
useful feedback
 Non-saturating GAN loss
• When starting training of generator,
discriminator is already trained
once
• Initially, generator will not generate
good images
• For fake images, score of
discriminator will be close to zero
• Curve of loss function is flat
• Gradient is going to be zero
• Generator will find it hard to learn
Non-saturating GAN loss
• In practice, the trick is:
• Instead of minimizing likelihood
of discriminator being correct,
maximize likelihood of
discriminator being wrong
• Step 2 now becomes
ascending gradient descent of
generator
−𝐸𝑧~𝑝𝑧 [log(𝐷𝜃 𝐺𝜙 𝑧 )]
Mode Collapse

10 digits, 10 modes
Mode Collapse
• Generator produces limited number of outputs, regardless of
latent input vector
• Limited set of outputs fail to capture the full diversity of the real data
distribution
• Leading to a collapse in modes of the data distribution
• Generator tries to minimize Ε𝑧~𝑝𝑧 log 1 − 𝐷𝜃 𝐺𝜙 𝑧
• Generator generates fake image, 𝑥 ∗ , so that 𝑥 ∗ = arg max 𝐷𝜃 (𝑥)
𝑥
• This objective function does not force generator to create a unique image
𝑥 ∗ for different values of input latent vector
Mode Collapse
• G and D are trained simultaneously
• As D gets better for one mode in distinguishing between real and
fake:
• G will focus on other modes where it can easily fool D
• OR G will try to make better samples of that mode only
• If G can fool D, it gets the same reward whether it is generating for
one mode or for multiple modes
Mode Collapse
• Solution: Use an objective function which compares real and fake
data distributions with a different distance

• Wasserstein loss (Earth-Mover distance):

• Measures minimum transportation cost for transforming one distribution
into another distribution
Earth-Mover (EM) distance/ Wasserstein Metric
• Example:
• Have 6 boxes → want to move them from left to locations marked by dotted
square on right
• For box #1: move it from location 1 to location 7
• Moving cost equals to its weight times the distance. Assume, weight = 1
• So cost to move box #1 = 1*(7–1) = 6
Earth-Mover (EM) distance/ Wasserstein Metric
• Two different moving plans:
• Plan 1: move 2 boxes from location 1 to location 10 → entry γ(1, 10) = 2
• Total transport cost of either plan below is 42.

Wasserstein distance is the cost

of the cheapest transport plan

[Link]
WGAN
• Measures distance between real and generated data distribution
• Vs. GAN: minimizing probabilities of real and generated data samples

𝑊 𝑃𝑑𝑎𝑡𝑎 , 𝑃𝑚𝑜𝑑𝑒𝑙 = inf 𝐸 𝑥,𝑦 ~𝛾 [∥ 𝑥 − 𝑦 ∥]

𝛾𝜖𝜋(𝑃𝑑𝑎𝑡𝑎 ,𝑃𝑚𝑜𝑑𝑒𝑙 )

• infimum gives minimum transportation costs over all possible transport

plans 𝛾
• Computation intractable in high dimensions (e.g. images)
• Rewritten using Kantorovich-Rubinstein duality:
𝑊 𝑃𝑑𝑎𝑡𝑎 , 𝑃𝑚𝑜𝑑𝑒𝑙 = sup 𝐸𝑥~𝑃𝑑𝑎𝑡𝑎 [𝑓 𝑥) − 𝐸𝑥~𝑃𝑚𝑜𝑑𝑒𝑙 [𝑓 𝑥 ]
∥𝑓∥𝐿 ≤1
WGAN
• Kantorovich-Rubinstein duality:
𝑊 𝑃𝑑𝑎𝑡𝑎 , 𝑃𝑚𝑜𝑑𝑒𝑙 = sup 𝐸𝑥~𝑃𝑑𝑎𝑡𝑎 [𝑓 𝑥) − 𝐸𝑥~𝑃𝑚𝑜𝑑𝑒𝑙 [𝑓 𝑥 ]
∥𝑓∥𝐿 ≤1
• Instead of searching over transport plans 𝛾 , search over functions
𝑓(x) that are 1-Lipschitz continuous; for regularization of discriminator
• Keeps check on discriminator gradients
• Measure how much discriminator function changes with small changes in input
• WGAN uses:
• Replaces 𝑓(x) with a neural network (called critic)
• Enforces Lipschitz constraint
WGAN-GP
• Use Lipschitz Constant:
• Example: 𝑓 𝑥 = 2𝑥 + 3 Let, 𝑦1 = 𝑓 𝑥1 and 𝑦2 = 𝑓 𝑥2
𝑓 𝑥1 −𝑓 𝑥2 2𝑥1 +3−2𝑥2 −3
• Gradient = = =2
𝑥1 −𝑥2 𝑥1 −𝑥2
→ Lipschitz constant = 2 → referred to as 2- Lipschitz continuity
→ If input changes by 1 unit, output will change by 2 units
→ Function is Lipschitz continuous
WGAN
𝑊 𝑃𝑑𝑎𝑡𝑎 , 𝑃𝑚𝑜𝑑𝑒𝑙 = sup 𝐸𝑥~𝑃𝑑𝑎𝑡𝑎 [𝑓 𝑥) − 𝐸𝑥~𝑃𝑚𝑜𝑑𝑒𝑙 [𝑓 𝑥 ]
∥𝑓∥𝐿 ≤1
• 𝑓 𝑥 is a 1-Lipschitz function (for output score of discriminator (critic))
→ satisfies 𝑓 𝑥 − 𝑓 𝑦 ≤ 𝑥 − 𝑦
• Expectation terms → expected outputs of function under real and generated
distributions
• supremum finds largest possible difference between these expectations, while
ensuring 𝑓 remains a 1-Lipschitz function
• 𝑓 𝑥 acts as a critic that scores how distinguishable real and generated
distributions are
• critic tries to maximize the difference between scores given to real and fake
samples
WGAN-GP

• Discriminator (critic) produces real-valued score for real/fake image

• Ex., may give score 10 to real image and -10 to fake
• Provides more meaningful gradients to G on how much work is to be done to
match degree of real image
• 𝐷𝑙𝑜𝑠𝑠 = 𝐷𝑚𝑒𝑎𝑛 𝑟𝑒𝑎𝑙 − 𝐷𝑚𝑒𝑎𝑛 𝑔𝑒𝑛𝑒𝑟𝑎𝑡𝑒𝑑 + 𝜆 ∗ 𝑝𝑒𝑛𝑎𝑙𝑡𝑦
• 𝑝𝑒𝑛𝑎𝑙𝑡𝑦: to avoid discriminator becoming too powerful/weak
• 𝜆: hyperparameter to control 𝑝𝑒𝑛𝑎𝑙𝑡𝑦
• Regularize by penalizing gradients if they cross threshold
• 𝑝𝑒𝑛𝑎𝑙𝑡𝑦 = ∥ ∇𝑥 𝐷 𝑥 ∥2 −1 2

• Discriminator change bounded not to exceed 1

[Link]
Earth-Mover (EM) distance/ Wasserstein Metric

Plot for value of D(x) for

both GAN and WGAN
• GAN: areas with
diminishing or
exploding gradients
• WGAN: gradient is
smoother everywhere
and learns better
GAN Frameworks
Unconditional vs. Conditional GANs
Unconditional GAN Conditional GAN
Training Data Training data does not need to be Training data needs to be labeled
labeled
Input to Generator Random noise vector z Random noise vector z and condition
vector y
Input to Discriminator Data sample (either real or generated) Data sample and condition vector y
Output Control Generates data without control over Generates data with control over
specific characteristics of the output specific characteristics based on
condition vector y
Modelling Models joint probability P(X, Y) Models conditional probability P(X | Y)
Example An unconditional MNIST GAN would Conditional MNIST GAN would let you
produce random digits specify which digit the GAN should
generate
Unconditional GAN
Deep Convolutional GAN (DCGAN)
• Unconditional GAN Framework
• Focus on stable training
• Use Convolutional Neural Networks (CNNs) instead of simple
multi-layer perceptrons (MLPs)
• Highly successful in generating high-quality images
• For discriminator, any CNN classifier with one class (real) can be
used
• VGG, ResNet etc.
DCGAN
• Architecture guidelines for stable DCGAN:
• Use fractional-strided (transpose) convolutions in generator (for
upsampling)
• Replace pooling layers replaced with strided convolutions in discriminator –
allows to learn spatial downsampling
• Use BatchNorm in both G and D → stable architecture
• Use ReLU activation in generator for all layers except for output, which uses
tanh
• Use LeakyReLU activation in discriminator for all layers
 Left: Generator (Redford et al. (2015) Right: Discriminator (Yeh et al. (2016)

• As per original DCGAN paper, all the weights are initialized randomly from normal distribution with 0 mean
and 0.02 standard deviation.
• In LeakyReLU slope of leak is set to 0.2 in all layers.
• Adam optimizer is used for both Generator and Discriminator model with learning rate of 0.0002 and β1 = 0.5.
• Fixed Batch of latent vectors are generated from Gaussian distribution. Batch size is set to 128.
StyleGAN
• A powerful generative model for creating high-quality, realistic
images, especially human faces
• Advanced version of GANs introduced by Nvidia researchers
• Controls image generation through “styles” at different layers
• Instead of feeding noise directly into the generator:
• Transforms noise into a style representation
• Injects this style into different layers of the network

Tero Karras and Samuli Laine and Timo Aila , A Style-Based Generator Architecture for Generative Adversarial Networks, arXiv, 2019
StyleGAN: Key Components
1. Mapping network:
• Input: random noise vector 𝑧
• Output: intermediate latent vector 𝑤
• Separates latent space of input randomness into style space, a more
disentangled representation
2. Adaptive Instance Normalization (AdaIN)
• Each layer of generator is controlled by style – feature map is normalized and
scaled and shifted using style 𝑤
• Style modifies mean and variance of feature maps
• Control over features like pose, hair style, lighting, facial structure

Tero Karras and Samuli Laine and Timo Aila , A Style-Based Generator Architecture for Generative Adversarial Networks, arXiv, 2019
StyleGAN: Key Components
3. Noise Injection
• Small random noise added at each layer
• Controls fine details like skin texture, freckles, hair strands etc.

• Style Mixing: can combine styles from different latent vectors

• Early layers – coarse features (face shape)
• Middle layers – structure (eyes, nose)
• Late layers – fine details (texture)
• Mapping network f consists of 8 layers
• Synthesis network g consists of 18 layers -
two for each resolution (42 − 10242 )
• Gaussian noise added after each convolution,
before evaluating the nonlinearity
• “A” stands for a learned affine transform
• Applied to further modify features
• Specialize 𝑤 → (𝑦𝑠 , 𝑦𝑏 ) to styles that control
adaptive instance normalization
• 𝑦𝑠 : scale (style strength); 𝑦𝑏 : bias (style shift);
• “B” applies learned per-channel scaling
factors to the noise input

Use of Baseline Progressive Growing GANs – Size of generated image increases gradually from a very low
resolution (4*4) to high resolution (1024*1024) through bilinear upsampling
• AdaIN adjusts feature maps by
normalizing, scaling and biasing them
using style vector
• Gives StyleGAN more control over
image style at different layers
• AdaIN operation defined as:
𝑥𝑖 − 𝜇(𝑥𝑖 )
𝐴𝑑𝑎𝐼𝑁 𝑥𝑖 , 𝑦 = 𝑦𝑠,𝑖 + 𝑦𝑏,𝑖
𝜎(𝑥𝑖 )
• Each feature map 𝑥𝑖 normalized separately
• Scaled and biased using corresponding
scalar components from style y
Conditional GAN
Conditional GAN (CGAN)
Working of CGANs:
• A conditional variable (y) fed into both generator and discriminator
• This ensures that generator creates data corresponding to the given condition
(e.g., generating images of specific objects)
• Discriminator also receives labels to help distinguish between real and fake
data
• Loss function:
𝑚𝑖𝑛𝜙 𝑚𝑎𝑥𝜃 (Ε𝑥~𝑝𝑑𝑎𝑡𝑎 [log 𝐷𝜃 (𝑥|𝑦)] + Ε𝑧~𝑝𝑧 [log(1 − 𝐷𝜃 𝐺𝜙 𝑧|𝑦 )])
[Link]