Goal
● Goes through basic concepts in Computer Vision
○ Convolution
○ Pooling
○ Upsampling
○ Interpolation
○ Transposed Convolution
● Goes through details of VAE
○ Architecture
○ Math
○ Training & Sampling
○ More
Note: Diffusion is similar with VAE, so consider this as prequel of Diffusion!
Autoencoder Recap
● Encoder will compress the original
image to a lower dimension latent
space. The latent dimension is
supposed to capture the essential
features of original image.
● Decoder will reconstruct the
original image from the lower
dimension latent space.
How Is It Done Exactly?
Image from IBM VAE blog
Encoder Architecture Details
● Convolution Layer: Extract features
● ReLU:
○ Allow model to learn non-linear
complex relationship
○ Help vanishing gradients
○ Efficient computation
● Pooling:
○ Downsample feature maps
○ Reduce spatial dimensions and
computational complexity while
retaining important information,
○ Make the model more robust to
variations in feature position
● Fully connected: Flatten input and map it
to probability distribution we need
Typical CNN example from [Link]
Convolution Layer Intuition
● For single image, we first identify simple
micro patterns like straight lines, small
curves
● Then based on these patterns’
relationship with each other, identify
shapes like circle, oval, squares etc
● Then based on these shapes’
relationship with each other, identify
more complex shapes and patterns, like
legs, wheels etc
● The shape identifiers should ideally be
reusable
Image from Medium (sharathmanjunath)
Convolution Layer Intuition
● Use shape identifiers to identify patterns
● The shape identifiers are called filters or
kernels
Image from Medium (sharathmanjunath)
Convolution Operation with Kernel
● Values in the Kernel are all
trainable parameters
● Convolution is to use the
Kernel to scan through the
image and compute matrix
sum product
● Kernel values (learnable
parameters of CNN) stay
unchanged throughout the
image, so the number of
model’s parameter does not
depend on image size
Image from Medium (patale_akhil)
Neural Network Activation Function
Activation functions allow neural networks to
learn nonlinear relationships:
● Sigmoid and tanh suffers from vanishing
gradient. When value is too big or too
small, the gradient will become 0.
● ReLU improves vanishing gradient since
the gradient is 1 for all positive values. The
computation is also more efficient since
gradient constant (1 for positive and 0 for
negative). ReLU can suffer if value is
constantly negative causing 0 gradient,
model will stop learning.
Image from Medium (shrutijadon)
● Leaky ReLU improve on top of ReLU,
where negative values still have non-zero
positive gradient.
Pooling Layer
Pooling layer’s purpose:
● Downsample feature maps
● Reduce spatial dimensions and
computational complexity while retaining
important information
● Make the model more robust to variations
in feature position
Type of Pooling:
● Max Pooling (keep most apparent feature)
● Average Pooling (get average in a region)
How it works:
● 2x2 pooling, with stride of 2, reduces both
length and width of feature-map by 50%.
● No new parameters needed, just get the
maximum or average
● Often added after convolution layer
How Does Decoder Work?
● In Autoencoder, decoder reconstruct the original image from the lower dimension latent space.
● Basically reverse the process of what Encoder does.
○ Encoder: Convolution + Downsample
○ Decoder: Convolution + Upsample
Decoder Increase Image Size
Convolution
● We can remember the pooling
22 0 0 48 indices and use it to do
Pre Max Pooling Features
45 55 91 96
upsampling
15 31 6 39 0 65 87 91 Bilinear ● After upsampling and
Interpolation
45 12 3 96 0 32 43 0 interpolation, usually followed
by convolutions to learn how to
54 65 87 1 REVERSE
rebuild simple and complex
4 5 66 77 0 0 0 0 patterns with latent space
45 0 0 96 features
Upsampling
0 65 87 0
● Interpolation is used to
45 96 2 3
0 0 0 0 estimate or predict values
between known data points.
65 87 1 0
Common ways include bilinear
(closest 4 neighbors average),
Post Max Pooling Features Pooling Indices
45 96 2 3 polynomial etc
65 87 1 0
Pooling Features Pooling Indices
Decoder Increase Image Size
1 1 1
1 3
1 -1 1
2 4
1 1 1
● We can also do transposed
Pooling Features
Kernel (with learnable parameters) convolution
1 1 1 1 1+3 1+3 3
● With the features map after
-1+
1 -1 1 1
3
1-3 3
1 4 4 3
encoder’s pooling layer, and a
1 1 1 1 1+3 1+3 3
Kernel (with learnable
3 8 4 7 parameters) aiming to
reconstruct image with latent
Step 1 3 6 2 7
Step 2 space features, we can bring
2 6 6 4 back the image to its original
1 1+3 1+3 3 1 1+3 1+3 3 dimension
-1+3 1-3 Output
1+2 +2 +2
3 1+2
-1+3 1-3+2
3+4
+2+4 +4
1+3 1+3
1+2
-2 +2
3 1+3- 1+3
1+2
2+4 +2-4
3+4
2 2 2
2 2+4 2+4 4
Step 3
Step 4
Autoencoder Recap
● During training, autoencoder’s loss function is comparing pixel wise difference between output image
and original image. That’s also where the `auto` in the name comes from.
● Autoencoder is great in denoising image
Training
Process
Inference
Diffusion Recap
● Diffusion’s mechanism is also based on using model to predict noise for noisy image, so we have the
capability to iteratively remove noises and generate good images.
● Many overlap in architecture, underlying math, training & sampling etc. Stay tuned!
`Number 4` Embedding
Update Weights
Diffusion
Training UNet Loss
Predicted Noise
Actual Noise
From Autoencoders to Variational Autoencoders (VAE)
● Goal of autoencoder is to reconstruct the original image. However, for a generative model, what we
want is `Given observed samples x from a distribution of interest, learn to model the true data
distribution p(x)`
● In plain words, we want to generate new samples that resemble the original input. That’s why AE is not good
for Generative task. Other reasons are:
○ Autoencoder are fixed, deterministic mapping and discrete on the latent space
If sample a random point from this 2-D space and feed it to the decoder,
will it generate a new image that is “similar” to one of the 0-9 digit images
in the training set?
Probably will generate a good `1`
???
Probably will generate a good `0`
Latent space of MNIST data (number classification)
Variational Autoencoders (VAE)
● VAES are probabilistic models. VAEs encode latent variables of training data not as a fixed discrete
value z, but as a continuous range of possibilities expressed as a probability distribution p(z),
represented with mean vector (μ) and a std-deviation / variance vector (𝜎).
Conditional VAE
● With VAE, we can sample a random point in the latent space and generate new images similar to
training set. However, we can’t generate an new on demand (say generate a specific number with the
MNIST example)
● With Conditional VAE, we will be able to achieve this
○ CVAE has an extra input to both the encoder and the decoder, usually an one-hot vector.
Same point in the latent space,
different y vector
x μ
Encoder
z
σ Decoder z
x̂
Decoder x̂
y [0,0,0,0,1,0,0,0,0,0]
y y
[0,0,0,0,1,0,0,0,0,0] [0,0,0,0,0,0,0,1,0,0]
to represent `4`
Different numbers generated
Training Time Inference Time
Underlying Math
● Assumptions
○ We have observed data
○ We assume the data is generated from a latent variable
○ 𝛳 represents parameters of the model
● Goal
○ Maximize the log likelihood of observed data
■ Same as maximize likelihood directly (log function is monotonic), log is used for
convenience to make math easier
● Given , can we calculate this directly?
○ This is computationally intractable
○ The prior distribution is often chosen to be a simple distribution, such as a standard
Gaussian
○ However, is typically a complex, non-linear function defined modeled by neural network
with parameters θ.
● So what to do now? Use ELBO (Evidence Lower Bound)
Underlying Math
● We introduce the approximate posterior , then use Jensen's inequality to derive a lower
bound.
○ 𝟇 represents the approximation function’s parameters
Lower bound of original
log likelihood
● Expand the right hand side of equation
● Rewriting integrals in terms of expectations
Underlying Math
● Rearrange orders
● Recall Kullback-Leibler Divergence (KL Divergence) between p(x) and q(x) is defined as:
● Rewrite lower bound expression
● Intuition of ELBO
○ Reconstruction Loss: encourages the decoder to reconstruct the
input X well from the latent representation Z.
○ KL Divergence Loss: encourages the approximate posterior to be
close to the prior p(Z), typically a standard Gaussian distribution.
● Note that we only care about decoder parameters of 𝛳, and encoder parameters of 𝟇
Underlying Math
● Reconstruction Loss is typically computed by Mean Squared Error (MSE) between the original and
reconstructed images:
○ f𝛳 is the decoder function
○ g𝟇 is the encoder function
● Practically, we usually assume the prior distribution p(Z) is standard normal distribution. Thus the KL
Divergence Loss is:
○ is the Gaussian distribution of latent space defined by the mean and standard deviation.
● Then we have the final loss function formula:
● We can now use gradient descent to maximize ELBO, with one last obstacle :<
Underlying Math
● Recap
○ We plan to use gradient descent to maximize ELBO:
○ The parameters we are updating is decoder parameters of 𝛳, and
encoder parameters of 𝟇
● Problem:
○ Although ELBO is a function of encoder parameter 𝟇, it’s not
differentiable of 𝟇 since Zi is randomly picked from the latent
space represented by 𝞵 and 𝞼
● Reparameterization Trick to the rescue
○ The essence is change how sampling is executed. Instead of
random sampling directly from ,introduce a random
auxiliary variable 𝝴 from a distribution that is not parameterized by
𝟇, usually from standard normal distribution, then pass it through
g𝟇(X, 𝝴) to sample
■ Since 𝝴 is fixed and not dependent on 𝟇, now ELBO
differentiable against 𝟇
○ For example g𝟇(X, 𝝴) = 𝞵 + 𝝴 𝞼 Reparameterization Pseudocode
Underlying Math
Original Reparameterized
Back Back
Propagation f Propagation f
Fixed
∂f/∂Z Z ∂f/∂Z Z
X Random
∂Z/∂𝞵 ∂Z/∂𝞼
𝞵 𝞼 𝞵 𝞼 𝝴
Now we are able to do gradient descent to find parameters that can maximize ELBO :D
Mission Accomplished!