16-08-2025

Generative Models
(Unsupervised learning)

-GAN
-VAE
-Diffusion Models
GAN original paper: Goodfellow, I. J., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., ... &
Bengio, Y. (2014). Generative adversarial nets. Advances in neural information processing systems, 27.
VAE original paper: Davidson, T. R., Falorsi, L., De Cao, N., Kipf, T., & Tomczak, J. M. (2018). Hyperspherical
variational auto-encoders. arXiv preprint arXiv:1804.00891.
Kingma, D. P., & Welling, M. (2013). Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114.
DDPM/Diffusion Models original paper: Ho, J., Jain, A., & Abbeel, P. (2020). Denoising diffusion probabilistic
models. Advances in neural information processing systems, 33, 6840-6851.

1
 16-08-2025

GAN
• GAN was introduced by Ian Goodfellow and his team in 2014 and they have transformed
how computers generate images, videos, music and more.
• Unlike traditional models that only recognize or classify data, they take a creative way by
generating entirely new content that closely resembles real-world data.
• A GAN is called adversarial because it trains two different networks which compete against
each other to generate more authentic new data from a given training dataset.
• One network generates new data by taking an input data sample and modifying it as much
as possible.
• The other network tries to predict whether the generated data output belongs in the
original dataset.
• In other words, the predicting network determines whether the generated data is fake or
real.
• The system generates newer, improved versions of fake data values until the predicting
network can no longer distinguish fake from original.

Review of Bayes’ theorem

2
 16-08-2025

Product Rule : probability P(A,B) of a conjunction of

two events A and B:
P ( A, B )  P ( A | B ) P ( B )  P ( B | A) P ( A)
● The two joint probabilities on occurrence of same two
events are always equal.
● P(B,A) = P(A,B)
● P(B)* P(A|B)= P(A)*P(B|A)

 P(A | B) = P(B|A) * P(A) / P(B) (Baye’s Rule)

 P(A | B) = P(A,B) / P(B)

Bayesian classifier
• A Bayesian classifier such as Naive Bayes is a type of generative model that
learns the joint probability distribution P(X,Y) over the input data X and
class labels Y
• From prob distribu on X → Y or features → class_label, make predictions
P(Y ∣ X).
• It estimates how likely it is to see each data point X given a
class Y (i.e., P(X∣Y)), and the overall chance of each class P(Y).
• Using Bayes' theorem, it computes the conditional probability P(Y∣X) for
classification tasks.
• Informally, this can be viewed as
posteriori = likelihood * prior/evidence

3
 16-08-2025

Classification is to Derive the Maximum Posteriori

• Let D be a training set of tuples and their associated class labels, and each tuple is
represented by an n-D attribute vector X = (x1, x2, …, xn)
• Suppose there are m classes C1, C2, …, Cm.
• Classification is to derive the maximum posteriori, i.e., the maximal P(Ci|X)
• This can be derived from Bayes’ theorem P(X | C )P(C )
P(C | X)  i i
i P(X)
• Since P(X) is constant for all classes, only numerator needs to be maximized

P(C | X)  P(X | C )P(C )

i i i
• Predicts X belongs to Ci iff the probability P(Ci|X) is the highest among all the P(Ck|X) for
all the k classes

weather example: classifying X

• An unseen sample X = <rain, hot, high, weak>

• P(X|yes)·P(yes) =
P(rain|yes)·P(hot|yes)·P(high|yes)·P(weak|yes)·P(yes) =
3/9·2/9·3/9·6/9·9/14 = 0.010582

• P(X|no)·P(no) =
P(rain|no)·P(hot|no)·P(high|no)·P(weak|no)·P(no) =
2/5·2/5·4/5·2/5·5/14 = 0.018286

• Sample X is classified in class no (don’t play)

4
 16-08-2025

Generative models

Discriminative Models v/s Generative Models

Generative models take random noise along with a class as input

e.g. a car, and from these inputs its goal is to generate a set of
features X that look like a realistic car.
 Why we need this noise?
 The noise is there to ensure that what's generated isn't actually,
the same car each time.

5
 16-08-2025

Discriminative v/s Generative Models

• Discriminative models take a set of features X, and from these features
determine a category whether the image is of a person or a car.
• They try to model the probability of class Y given a set of features X in
order to predict label of a test sample belonging to the same distribution.
• On the other hand, generative models try to learn how to make a realistic
representation of some class; for example, a realistic picture of a car or a
person.
• Generative Models take some random input represented by the noise to
produce an output fake sample
• Generative models try to mimic the distribution of Training data.

Adversarial nature of GAN

6
 16-08-2025

MiniMax Game tree

• MiniMax is a depth first, depth limited, recursive search procedure.

• This method is used for playing games in which there are 2 players taking
turns to play moves.

• Physically it is just a tree of all possible moves.

• MiniMax game tree are best suited for games in which both players can see
the entire game situation.

• The strategy behind MiniMax Algorithm is that it assumes that both player
will play to the best of their ability.
• Works for Zero-sum, perfect information games.

2 sub-models of GAN: Generator & Discriminator

• GANs are composed of two models that compete against each other in
order to reach a point where realistic examples are produced by the
generator.
• In VAE, Encoder part tries to produce a good distribution over the latent
space, a little deviation from latent space vector generates real output by
decoder
• The job of “Generator” is to generate fake samples in order to fool
“Discriminator” or in turn “humans”.
• A Discriminator receives both types of samples: fake samples generated by
“Generator” and “Real samples from the domain”.
• The job of “Discriminator” is to figure out whether the input sample (fake
sample by Generator or Real sample from the domain) is Real or Fake??

7
 16-08-2025

GAN Architecture

Source: What is a GAN? - Generative Adversarial Networks Explained - AWS

2 sub-models of GAN: Generator & Discriminator

• For image generation, both generator and discriminators can be “deep conv
net” e.g. Inception Net v3.
• It begins with random initial weights.
• Generator is trained to output very convincing fake samples from input
“random noise”.
• While Discriminator is trained to classify its input as real or fake? Uses BCE
loss to adjust its weights using Error backpropagation.
• This creates a contest based on data distributions, where the generator
learns to map from a latent space/Noise to the true data distribution,
aiming to produce candidates that the discriminator cannot distinguish
from real data.
• The discriminator’s goal is to correctly identify these candidates, but as the
generator improves, its task becomes more challenging, increasing the
discriminator’s error rate.

8
 16-08-2025

Error Backpropagation (source: original GAN paper)

• The fake data is constructed by first sampling a random vector ‘z’ from a prior distribution
over latent variables of the model. The generator is then used to produce a sample x =
G(z).
• The function G is simply a function represented by a neural network that transforms the
random, unstructured z vector into structured data, intended to be statistically
indistinguishable from the training data.
• The discriminator then classifies this fake data. The discriminator is trained to assign this
data to the “fake” class.
• The backpropagation algorithm makes it possible to use the derivatives of loss using the
discriminator’s output with respect to the discriminator’s input to train the generator.
• The training process for the discriminator is thus much the same as for any other binary
classifier with the exception that the data for the “fake” class comes from a distribution
that changes constantly as the generator learns rather than from a fixed distribution.
• The learning process for the generator is somewhat unique, because it is not given
specific targets for its output, but rather simply given a reward for producing outputs that
fool its (constantly changing) opponent.

Error Backpropagation for training of GAN

The original work on GANs offered two versions of the cost for the
generator.
Original version of GAN; Most common implementation
Discriminator’s Loss (BCE i.e. discriminator is trained just like a regular
binary classifier.)
Mini-Max GAN (M-GAN)
Generator’s loss= - Discriminator’s Loss (negative of D’s Loss)
 “Min-Max Loss”
Min-Max Loss ensures both player maintain same level gaming.
• Requires finding Nash Equilibrium (end of game point) during training

9
 16-08-2025

Generator’s Loss Function

The generator tries to minimize this loss:

where
JG measure how well the generator is fooling the discriminator.
G(zi) is the generated sample from random noise zi
D(G(zi)) is the discriminator’s estimated probability that the generated
sample is real.
• Each time the discriminator mistakes fake data for real, the
generator learns from this success.
• Through many iterations, the generator improves and creates
more convincing fake samples.

Discriminator’s Loss Function

where
JD measures how well the discriminator classifies real and fake samples.
xi is a real data sample.
G(zi) is a fake sample from the generator.
D(xi) is the discriminator’s probability that xi is real.
D(G(zi)) is the discriminator’s probability about the fake sample is fake.

10
 16-08-2025

Cross Entropy Loss

• Cross-entropy loss, or log loss, measures
the performance of a classification model
whose output is a probability value
between 0 and 1.
• Cross-entropy loss is always greater than
zero.
Both logarithm terms are negative as
their operand is lying between 0 to 1
being sigmoidal output.
There is negative sign in front.

Cross Entropy Loss

• If neuron’s actual output is close to desired output for all training
examples then cross entropy will be close to zero.
• Calculate log loss function for some input x from training set D at these 2
cases (here Y is target label and a is computed output by model) :
Y = 0 and a ≈ 0, C = 0
Y = 1 and a ≈ 1, C = 0
At Y = 1, C = -log a
At Y = 0, C = -log(1-a)

11
 16-08-2025

Log loss when true label = 1

C = -log a
• This graph shows the range of possible loss values given a true
observation (label= 1).
• As the predicted probability approaches 1, log loss slowly decreases.
• As the predicted probability decreases, however, the log loss increases rapidly.
• Log loss penalizes both types of errors, but especially those predictions
that are confident and wrong!
• Cross-entropy loss increases as the predicted probability diverges from
the actual label.
• So predicting a probability of .012
when the actual observation label is 1
would be bad and result in a high loss value.
• A perfect model would have a log loss of 0.

CE loss when true label = 0

C = -log(1-a)
• This graph shows the range of possible loss values given a true
observation (label= 0).
As the predicted probability approaches 0, log loss slowly decreases.
As the predicted probability increases, however, the log loss increases rapidly.
Comparison of MSE with CE Loss:
Let’s say Y=1 (desired label)
and
a=0.2 (computed output by
model)
MSE=(1-0.2)2
=0.64

Log Loss=-log(0.2)
=1.609

12
 16-08-2025

MinMax Loss

• Here the term ‘x’ represents the real data i.e. train set from the domain
• Term D(x) represents prediction of Discriminator; whether ‘x’ is real or
fake;
 returns a value between 0 to 1, a value near zero indicating input is fake,
a value near 1 indicates input is real
• G(z) represents the generated output
• Term D(G(z)) represents Discriminator’s output on Generator’s fake sample
i.e. what class Discriminator assigned to G(z)
 The goal of Generator model is to minimize this function.
 The goal of Discriminator model is to maximize this function.
 The first part of formula tells how good the Discriminator is
 The second part of equation tells how good the Generator is

Visualizing Loss

13
 16-08-2025

Role of Generator & Discriminator after training

• After many epochs and achievement of generalization
criteria, once the GAN model is trained we have 2 separate
models with us which can function independently with
learned weights:
• A Generator model which is good enough to generate “fake
samples”
• A Discriminator model which is good enough to detect “fake
samples” from the real ones

Generate images Applications of GAN

• They can help create realistic and immersive visual experiences in video games and digital
entertainment. It can create realistic faces, characters, and animals for animation and video.
GAN can also edit images
• converting a low-resolution image to a high resolution or turning a black-and-white image to color.
Generate synthetic training data for other models
• In machine learning (ML), data augmentation artificially increases the training set by creating
modified copies of a dataset using existing data. E.g. , it can generate fraudulent transaction data
that you then use to train another fraud-detection ML system. This data can teach the system to
accurately distinguish between suspicious and genuine transactions.
Complete missing information
• to accurately guess and complete some missing information in a dataset.
Generate 3D models from 2D data
• GAN can generate 3D models from 2D photos or scanned images, e.g. in healthcare, GAN
combines X-rays and other body scans to create realistic images of organs for surgical planning and
simulation.

14
 16-08-2025

Improvements in GAN from 2014 to 2017

An illustration of progress in GAN capabilities over the course of approximately three years following the introduction of
GANs. GANs have rapidly become more capable, due to changes in GAN algorithms, improvements to the underlying
deep learning algorithms, and improvements to underlying deep learning software and hardware infrastructure.
This rapid progress means that it is infeasible for any single document to summarize the state-of-the-art GAN
capabilities or any specific set of best practices; both continue to evolve rapidly enough that any comprehensive survey
quickly becomes out of date.

Source: Karras, T., Aila, T., Laine, S., Lehtinen, J. Progressive growing of GANs for improved quality, stability, and variation.
CoRR, abs/1710.10196 (2017).

Output of GAN

 Major companies are using GAN

for cartoon/animation generation
 It has improved a lot from 2014
(black & white images) to 2018
(realistic colored) images using
Style GAN2 model.

This image is a sample from a GAN depicting a person who does not exist but was “imagined” by a GAN
after training on photos of celebrities.

15
 16-08-2025

Autoencoders

Autoencoder architecture

• The encoder maps the input data to a lower-dimensional latent space,

often referred to as the bottleneck, or latent space distribution
• while the decoder attempts to reconstruct the original input from the
encoded representation.

16
 16-08-2025

Unsupervised Autoencoder
• An autoencoder is a type of artificial neural network architecture designed
for learning compact and meaningful representations of input data.
• Its structure consists of two primary components: the encoder and
the decoder.
• During training, the autoencoder minimizes the mean square difference
between the input and its reconstruction at output layer; to optimize the
weights and biases in the network.

Unsupervised Autoencoder
• This learned representation captures essential features and structures
present in the dataset, facilitating subsequent tasks like dimensionality
reduction, denoising, anomaly detection, or feature extraction.
• There’s no need of labelled data for training an autoencoder.
• The training patterns are arranged as Input-Output pair, with output
vector being same as input vector.
• This kind of setting makes it suitable for unsupervised learning though it
uses Backpropagation similar to supervised learning.

17
 16-08-2025

Error-backpropagation Training
• In a typical autoencoder setup, the input and output pair shares identical
values since the primary goal of the autoencoder is to learn a condensed
yet representative encoding of the input itself.

• Thus, the output reconstructed by the decoder must closely match the
original input in terms of values.

• However, note that exceptions arise in cases like Denoising Autoencoders

or Variational Autoencoders, where the input and output pairs may vary
intentionally.

Training Loss
• An autoencoder trains by minimizing the difference between its
input and the reconstructed output using backpropagation and
stochastic gradient descent or any equivalent optimization
strategy.
Initially, random weights are assigned to the network. Then, the input
passes through the encoder, producing a lower-dimensional latent
space representation, or code.
Subsequently, the decoder regenerates the original input from this
code.

18
 16-08-2025

Training Loss
• Throughout the training phase, the network iteratively adjusts its weights
and biases to reduce the discrepancy between the actual input and the
decoded output.

• Commonly, this divergence is evaluated using a reconstruction loss

metric, such as mean squared error (MSE) (for real-valued data) or
binary cross-entropy (for categorical data), depending on the nature of
the input data.

Autoencoder uses
• A trained autoencoder captures essential features and structures
present in the dataset, facilitating subsequent tasks like dimensionality
reduction, denoising, anomaly detection, or feature extraction.

• Once trained, the autoencoder retains the ability to encode incoming

instances to a reduced dimension, then decode the resulting
representations back to the original space.

• This behavior permits exploiting the autoencoder for downstream

applications, such as dimensionality reduction, anomaly detection,
generative modelling, or feature learning.

19
 16-08-2025

Autoencoder for Anomaly detection

Train on Normal Data:
• An autoencoder is a neural network architecture trained to reconstruct
its input data.
• You train the autoencoder on a large dataset of normal examples
relevant to your task.
• During training, the autoencoder learns a compressed representation of
the normal data that captures its essential features.

Autoencoder for Anomaly detection

Reconstruction Error:
• Once trained, the autoencoder can reconstruct "normal" data with a
high degree of accuracy.
• However, when presented with anomalous data (data that deviates
significantly from the normal patterns), the reconstruction process will
generate a higher reconstruction error.

20
 16-08-2025

Autoencoder for Anomaly detection

Anomaly Detection Threshold:
• By analyzing the reconstruction error for each data point, you can identify
potential anomalies.
• A threshold is set based on the distribution of reconstruction errors
observed in the normal data.
• Data points exceeding this threshold are flagged as anomalies.

Autoencoder summary
• Autoencoders are a powerful tool for anomaly detection, particularly
when dealing with unlabeled data.
• By effectively training the autoencoder and setting appropriate
thresholds, one can gain valuable insights into anomalies within your
data.
• They are well-suited for detecting point anomalies (data points deviating
from the norm).
• For context-based anomalies (anomalies dependent on surrounding data
points or time series data), one needs to explore other techniques such
as Variational autoencoders (VAEs), Transformer, etc.

21
 16-08-2025

Normal Autoencoders v/s Variational Autoencoders

Normal Autoencoder
learns a hidden representation of input
cannot generate new data
Learns by minimizing the reconstruction loss

Variational Autoencoder
Learns a hidden representation of input
Also Learns to generate new data
Minimize reconstruction loss + latent loss
Latent vectors are sampled from Gaussian Mixture

Kullback-Leibler (KL) divergence

• KL Divergence is a fundamental concept in information
theory and statistics, used to measure the difference
between two probability distributions.
• In the context of machine learning, it is often used to
compare the predicted probability distribution of a model
with the true distribution of the data.

22
 16-08-2025

VAE Latent Vector

• To generate images, the decoder samples from the latent space.
• Sampling from the specific points in the latent space representing the original inputs in the
training data would replicate those original inputs.
• To generate new images, the VAE must be able to sample from anywhere in the latent
space between the original data points. For this to be possible, the latent space must
exhibit two types of regularity:
Continuity: Nearby points in latent space should yield similar content when decoded.
Completeness: Any point sampled from the latent space should yield meaningful content
when decoded.
• A simple way to implement both continuity and completeness in latent space is to help
ensure that it follows a standard normal distribution, called a Gaussian distribution.
• But minimizing only reconstruction loss doesn't incentivize the model to organize the
latent space in any particular way, because the “in-between” space is not relevant to the
accurate reconstruction of the original data points. This is where the KL divergence
regularization term

VAE KL Divergence
• Minimizing the KL divergence between the learned
distribution of latent variables and a simple Gaussian
distribution whose values range from 0 to 1 forces the
learned encoding of latent variables to follow a normal
distribution.
• This allows for smooth interpolation of any point in latent
space, and thereby the generation of new images.

23
 16-08-2025

VAE v/s GAN

• Like VAEs, GANs are a joint architecture combining two neural networks
Encoder and Decoder
Generator and Discriminator
In VAE, feedback of Decoder through reconstruction loss and Latent loss is used to learn
latent vector by Encoder.
The two sub-networks are trained adversarially in a zero-sum game in GAN: the
feedback from the discriminator is used to improve the output of the generator until the
discriminator is no longer able to discern between real and fake samples.
• For image synthesis, both have upsides and downsides: GANs produce
clearer images but, due to the adversarial tradeoffs between the two
composite models, are unstable in training. VAEs are easier to train but, due
to the nature of producing images from the “average” features of training
data, tend to produce blurrier images.

Diffusion Models
Denoising diffusion probabilistic
models (DDPM)
Ho, J., Jain, A., & Abbeel, P. (2020). Denoising diffusion probabilistic
models. Advances in neural information processing systems, 33, 6840-6851.

24
 16-08-2025

Output of DDPM
Source: Original
Paper
2006.11239

It’s used by
commercial s/w
DALL-E for image
generation.

Diffusion models
• The intuition behind diffusion models is inspired by physics,
treating pixels like the molecules of a drop of ink spreading
out in a glass of water over time.
• Much like how the random movement of the ink molecules
will eventually lead to their even dispersal in the glass, the
random introduction of noise into an image will eventually
result in what looks like TV static.
• By modeling that diffusion process, then somehow learning
to reverse it, an artificial intelligence model can generate new
images by simply “denoising” samples of random noise.

25
 16-08-2025

Diffusion models Applications

• Diffusion models are a type of generative model that learn to generate new
data samples by reversing a process of gradually adding noise to existing
data.

• They are particularly known for their ability to generate high-quality images
and other data types.

• Conditional Diffusion Models incorporate learning word embeddings to be

able to generate new data (images) on a given text prompt.

• Diffusion models are most prominently associated with image generation

and other image processing tasks such as inpainting and super-resolution,
but their applications extend to other domains including audio generation,
drug design, and molecule generation.

Diffusion models
• Forward Diffusion:
The model starts with real data (like an image) and progressively
adds random noise until it becomes pure noise.
This is like gradually blurring an image until it's just random pixels.
• Reverse Diffusion:
The model learns to reverse this process with a neural network,
starting from pure noise and gradually removing the noise to
reconstruct the original data or generate new data.
starting from a blurry image and gradually sharpening it.

26
 16-08-2025

Background of Gaussian Filters

from DIP book by Gonzalez

Averaging/Low-pass filters
• Smoothing filters are used for blurring and for noise reduction.
• Blurring/averaging is used in pre-processing tasks, such as removal of small details from
an image prior to large object extraction, and bridging of small gaps in lines or curves.
• Note that the sum of all elements of this matrix is 1.0. This is important. If the sum is
not exactly one, the resultant image will be brighter or darker.

27
 16-08-2025

Convolve 3 by 3 averaging filter on an input image 6 by 6

90 90 90 0 0 0
90 90 90 0 0 0 1/9 1/9 1/9

90 90 90 0 0 0 *
1/9 1/9 1/9 = ?

90 90 90 0 0 0
1/9 1/9 1/9
90 90 90 0 0 0
90 90 90 0 0 0
• Convolution operation with padding=‘same’ option results in an output image of size same
as input.
• In diffusion model, Gaussian noise is added in a controlled manner using Markov Chain
method.

Idea behind smoothing linear filters

•By replacing value of every pixel in image by average of intensity levels in neighborhood
defined by filter mask, this process results in image with reduced sharp transitions in
intensities.
•As random noise typically consists of sharp transitions in intensity levels, the most
obvious application of smoothing is noise reduction.
•A major use of averaging filters is in reduction of irrelevant detail in an image.
• These operations have two key features: they are shift-invariant, and they are linear.
Shift-invariant means that we perform the same operation at every
point in the image.
Linear means that this operation is linear, that is, we replace every
pixel with a linear combination of its neighbors.

28
 16-08-2025

Result of smoothing
with square
averaging filter
masks of varying
sizes

This slide gives you an idea how square averaging filters

of various sizes affect original image. Here applying
averaging filter to original image is same as convolving
filter on the original image with some choice of padding
to get output image same as input image.

Blurring with Spatial Averaging

• We need to blur image for purpose of getting a gross representation of
objects of interest, such that intensity of smaller objects blends with
background and larger objects become bloblike and easy to detect.

29
 16-08-2025

Constructing a Filter from a Continuous Function

• An important continuous function used for image smoothing, the
Gaussian.
• A one-dimensional Gaussian is:

• This is also known as a Normal distribution.

Here μ is the mean value, and σ is the variance.
• Here’s a plot of a Gaussian:

• The mean, μ, gives the location of the peak of the function.

The parameter,σ, controls how wide the peak is.
As ‘σ’ gets smaller the peak becomes narrower.
• σ serves as a parameter that allows to control how much we smooth
the image (that is, how big a neighborhood we use for averaging).
• The bigger σ is, the more we smooth the image.

Effect of standard deviation on Gaussian Filter

• If the standard deviation of the Gaussian is very small
• e.g., smaller than one pixel; the smoothing will have little effect
because the weights for all pixels off the center will be very small.
• For larger than a pixel standard deviation,
• the neighbouring pixels will have larger weights in the weighted
average, which in turn means that the average will be strongly biased
toward a consensus of the neighbours.
• This will be a good estimate of a pixel’s value, and the noise will
largely disappear at the cost of some blurring.

• Finally, a kernel/filter/mask that has a large standard deviation will cause

much of the image detail to disappear, along with the noise.

30
 16-08-2025

Gaussian Filter
• The averaging filter that we introduced earlier replaces each pixel with
the average gray value of its neighbours.
• This means that nearby pixels all play an equal role in the average, and
more distant pixels play no role.
• It is more appealing to use the Gaussian to replace each pixel with a
weighted average of its neighbors.
• In this way, the nearest pixels influence the average more, and more
distant pixels play a smaller and smaller role.
• This is more elegant, because we have a smooth and continuous drop-
off in the influence of pixels on the result, instead of a sudden,
discontinuous change.

Gaussian Filter
• In order to build a discrete filter that looks like a Gaussian: is
done by evaluating the Gaussian function values at discrete
locations.

• That is, although G is defined for any continuous value of x,

we will just use its values at discrete locations (… -3, -2, -1, 0,
1, 2, 3…).

31
 16-08-2025

Gaussian filter
• Example of a Gaussian filter and its effect on image

Review of UNET & Transposed

Convolution

32
 16-08-2025

A typical UNet for Medical Image Segmentation

Add Dropout layers

in between
convolution layers to
avoid overfitting.

Transposed convolution with a 2X2 kernel

• The shaded portions are a portion of an intermediate tensor as well as the input and
kernel tensor elements used for the computation.
• In Transposed Conv, decreasing padding will increase the output feature map size.
O=(I−1) × S + K−2P

33
 16-08-2025

Markov Chain & Diffusion Models

• A Markov chain of adding noise to the input image in diffusion
models refers to the mathematical framework used in diffusion-
based generative models
• This process is a fundamental aspect of how diffusion models
learn and generate images.
• It learns to predict and remove the noise step by step,
effectively reconstructing the original data or generating new
samples.
• This reverse process is trained by learning to predict the noise at
each step, allowing the model to gradually denoise the data.

Forward Diffusion (Markov Chain)

 noise is incrementally added to the data over a series of steps in a

controlled manner using a Markov chain where each step slightly
degrades the data by adding Gaussian noise
 addition of noise to an image at each step depends only on the
previous step, not the entire history

34
 16-08-2025

Reverse Diffusion (Markov Chain)

Original authors:
 To represent the reverse process, authors used a U-Net backbone similar
to an unmasked PixelCNN++ with group normalization throughout.
 Parameters are shared across time, which is specified to the network
using the Transformer sinusoidal position embedding.
 They used self-attention at the 16 X 16 feature map resolution.

Reverse Diffusion (CNN based UNET)

• It starts with a image of random noise at random time ‘t’, learns to
predict the noise that was added to this image at previous time step.
• From forward diffusion, we already know how much actual noise
was added from previous step at ‘t-1’ to result in Xt
• UNET trains by minimizing the MSE between predicted Noise and
the Actual noise.
• If your model learns to predict very correct noise for input Xt, this
noise can be subtracted from Xt to obtain denoised image Xt-1.
• After training, for a noise input, if we repeat this process T times,
step by step, it will convert noise into it’s original denoised image.

35
 16-08-2025

Broader Impact of Diffusion Models

• Diffusion model advanced efforts to improve the sample quality of GANs,
flows, autoregressive models, etc.
• Diffusion model paper represented progress in making diffusion models a
generally useful tool in this family of techniques, so it may serve to
amplify any impacts that generative models have had (and will have) on
the broader world.
• Unfortunately, there are numerous well-known malicious uses of
generative models.
• Sample generation techniques can be employed to produce fake images
and videos of high profile figures for political purposes.
• While fake images were manually created long before software tools were
available, generative models such as Diffusion Model make the process
easier.

Commercial use of DM
• Diffusion models are among the neural network architectures
at the forefront of generative AI, most notably represented
by popular text-to-image models including Stability AI’s
Stable Diffusion, OpenAI’s DALL-E (beginning with DALL-E-2),
Midjourney and Google’s Imagen.
• They improve upon the performance and stability of
other machine learning architectures used for image
synthesis such as variational autoencoders (VAEs), generative
adversarial networks (GANs) and autoregressive models such
as PixelCNN.

36
 16-08-2025

Limitations of Diffusion Models

• High memory consumption during training due to the need
to store multiple intermediate steps.
• Requires careful tuning of noise schedules and other
hyperparameters to achieve optimal performance while fine
tuning.
• High demand for GPUs or TPUs, making them less accessible
for small-scale research or applications with limited
resources.

37