Generating Images
CMPUT 328
Nilanjan Ray
Generative neural networks
CNN
Random
noise image
Picture
2
Three types of image generators
Picture: [Link]
3
Generative Adversarial Networks (GANs)
Real pictures from a
database
Discriminator
Real picture or
CNN
…
fake?
Generator CNN
Random
noise Fake
pictures
…
…
4
Generative Adversarial Networks (GANs)…
Two-player game between
Real pictures
from a
database
the generator CNN and
the discriminator CNN
Discriminat
Real
or CNN
picture or
… fake?
Generat
Random
or CNN
noise
Fake D We need to train both
pictures
these CNNs so that they
…
become very good at their
…
G
own trade!
𝑚𝑖𝑛𝐺 𝑚𝑎𝑥𝐷 𝐸𝐼~𝑅𝑒𝑎𝑙 log 𝐷 𝐼 + 𝐸𝑧~𝑁𝑜𝑖𝑠𝑒 [log(1 − 𝐷(𝐺 𝑧 ))]
5
GANs training algorithm
After training, use
generator CNN to
generate pictures!
Source: [Link] 6
GANs tutorial
• DCGANs: [Link]
• [Link]
• And if you like PyTorch lightning: [Link]
[Link]/en/stable/notebooks/lightning_examples/ba
[Link]
7
GANs training..
• Very unstable, as we are trying to minimize one part and maximize another part of the loss
function in an alternating fashion; stabilization of training is an active area of research
• Some recommendations:
• Freeze layers: during training frequently one player gets stronger than the other; when it happens,
freeze its parameters, and let the other player catch up
• Minibatch discrimination: have an entire minibatch composed of either real or fake images. Don’t mix
these two types within a single minibatch.
• Batch normalization seems to help
• Averaging of gradients over time seems to help
• Lebel smoothing (replace 1 by 0.9 in the one-hot vector) also helps
• Experience replay also helps: keep generated images from last 50 or so epochs and show those to the
discriminator. This prevents the discriminator from becoming too greedy.
8
[Link]
GANs training
• Lots of trick on modifying loss functions
([Link]
collections)
• We will study “Wasserstein GANs” ([Link]
algorithm
9
Wasserstein GANs algorithm
[Link] [Link] 10
CycleGAN: Unpaired Image-to-Image
Translation using GANs
[Link] 11
Unpaired image translation
12
CycleGAN schematic
[Link] 13
CycleGAN loss
Cycle consistency loss:
Adversarial losses:
Total loss:
14
CycleGAN training algorithm
[Link]
csc321_2018/assignments/[Link]
15
How do you evaluate accuracies of GANs?
• Measuring accuracy of unsupervised learning is often difficult
• Computer vision community started using a score called FID (Fréchet
Inception Distance)
• FID measures distance between two distributions - real images and
generated images
• So, a lower FID implies a better GANs performance
• An interesting paper: backpropagating through FID
([Link]
16
Normalizing flows
Can you turn a simple distribution (like Gaussian) into a more
complex distribution (and vice-versa) using invertible functions?
Picture: [Link]
17
Variational autoencoder
Cost function has two components:
Reconstruction + constraint for and
The constraint: the encoded distribution
should look like a zero-mean, unit variance
Gaussian.
The advantage is that you can generate
data (images) that look like the training
images.
MNIST_Variational_AE.ipynb
Normalizing flows…
Source distribution target distribution
Learning Learning
Invertible DNN
Generation Generation
Learning can be done by maximum likelihood estimation – when the target
distribution can be expressed by a mathematical formula (aka closed form)
19
Normalizing flows
• [Link]
g-
flows/blob/master/example/real_nvp_colab.ipynb#scrollTo=BUbS_Ol
PgXNb
• [Link]
• There is also great tutorial available here:
[Link]
20
Takeaway
• By far GANs can generate high resolution and realistic pictures
compared to VAEs or NFs
• Reasons
• Adversarial training seems superior
• GANs can employ very deep and flexible models for both generators and
discriminators
• Models in NFs tend to have weaker expressive power because these models
need to be invertible
• VAEs inherits some limitations of ELBO (evidence lower bound) optimization
• Training GANs still remains tricky
21