Module-2

Generative Adversarial Network

● Generative Adversarial Networks (GANs) were developed in 2014 by

Ian Goodfellow and his teammates.

● GAN is basically an approach to generative modeling that generates

a new set of data based on training data that look like training data.

● To understand the term GAN let’s break it into separate three parts

○ Generative – To learn a generative model, which describes

how data is generated in terms of a probabilistic model. In

simple words, it explains how data is generated visually.

○ Adversarial – The training of the model is done in an

adversarial setting.

○ Networks – use deep neural networks for training purposes.

Architecture of GAN

● GANs have two main blocks(two neural networks) which compete

with each other and are able to capture, copy, and analyze the

variations in a dataset.

● The two models are usually called Generator and Discriminator

● The generator network

○ Role: The generator's role is to create new data instances that

resemble the training data.

○ Input: It takes random noise as input and generates samples

that ideally cannot be distinguished from real data.

○ The generator tries to create realistic samples to fool the

discriminator.

● The discriminator network:

○ It tries to distinguish between real and generated samples.

○ It is trained with real samples from the training data and

generated samples from the generator.

○ The discriminator’s objective is to correctly classify real data as

real and generated data as fake.

○ The discriminator is trained to improve its ability to differentiate

between real and fake samples.

● The training process:

○ involves an adversarial game between the generator and the

discriminator.
 ○ The generator aims to produce samples that fool the

discriminator, while the discriminator tries to improve its ability

to distinguish between real and generated data.

○ This adversarial training pushes both networks to improve over

time.

The Discriminator Model

● The discriminator in a GAN is simply a classifier.

● It tries to distinguish real data from the data created by the generator.
 ● It could use any network architecture appropriate to the type of data it's

classifying.

Figure1: Backpropagation in discriminator training.

Discriminator Training Data

The discriminator's training data comes from two sources:

● Real data instances, such as real pictures of people. The discriminator uses

these instances as positive examples during training.

● Fake data instances created by the generator. The discriminator uses these

instances as negative examples during training.

● In Figure 1, the two "Sample" boxes represent these two data sources feeding

into the discriminator.

● During discriminator training the generator does not train. Its weights remain

constant while it produces examples for the discriminator to train on.

Training the Discriminator

● The discriminator connects to two loss functions.

● During discriminator training, the discriminator ignores the generator loss and just

uses the discriminator loss.

● We use the generator loss during generator training, as described in the next

section.

During discriminator training:

1. The discriminator classifies both real data and fake data from the generator.

2. The discriminator loss penalizes the discriminator for misclassifying a real

instance as fake or a fake instance as real.

3. The discriminator updates its weights through backpropagation from the

discriminator loss through the discriminator network.

The Generator

● The generator part of a GAN learns to create fake data by incorporating feedback

from the discriminator.

● It learns to make the discriminator classify its output as real.

Generator training requires tighter integration between the generator and the

discriminator than discriminator training requires. The portion of the GAN that trains the

generator includes:

● random input

● generator network, which transforms the random input into a data instance

● discriminator network, which classifies the generated data

● discriminator output

● generator loss, which penalizes the generator for failing to fool the discriminator

Figure 2: Backpropagation in generator training.

Random Input

● In its most basic form, a GAN takes random noise as its input.

● The generator then transforms this noise into a meaningful output.

 ● By introducing noise, we can get the GAN to produce a wide variety of data,

sampling from different places in the target distribution.

● Experiments suggest that the distribution of the noise doesn't matter much, so we

can choose something that's easy to sample from, like a uniform distribution. For

convenience the space from which the noise is sampled is usually of smaller

dimension than the dimensionality of the output space.

Note: Some GANs use non-random input to shape the output.

Using the Discriminator to Train the Generator

● To train a neural net, we alter the net's weights to reduce the error or loss of its

output.

● In our GAN, however, the generator is not directly connected to the loss that

we're trying to affect. The generator feeds into the discriminator net, and the

discriminator produces the output we're trying to affect.

● The generator loss penalizes the generator for producing a sample that the

discriminator network classifies as fake.

● This extra chunk of network must be included in backpropagation.

○ Backpropagation adjusts each weight in the right direction by calculating

the weight's impact on the output — how the output would change if you

changed the weight.

 ○ But the impact of a generator weight depends on the impact of the

discriminator weights it feeds into.

○ So backpropagation starts at the output and flows back through the

discriminator into the generator.

At the same time, we don't want the discriminator to change during generator training.

Trying to hit a moving target would make a hard problem even harder for the generator.

So we train the generator with the following procedure:

1. Sample random noise.

2. Produce generator output from sampled random noise.

3. Get discriminator "Real" or "Fake" classification for generator output.

4. Calculate loss from discriminator classification.

5. Backpropagate through both the discriminator and generator to obtain gradients.

6. Use gradients to change only the generator weights.

GAN Training

Because a GAN contains two separately trained networks, its training algorithm must

address two complications:

 ● GANs must juggle two different kinds of training (generator and discriminator).

● GAN convergence is hard to identify.

Alternating Training

The generator and the discriminator have different training processes. So how do we

train the GAN as a whole?

GAN training proceeds in alternating periods:

1. The discriminator trains for one or more epochs.

2. The generator trains for one or more epochs.

3. Repeat steps 1 and 2 to continue to train the generator and discriminator

networks.

● We keep the generator constant during the discriminator training phase. As

discriminator training tries to figure out how to distinguish real data from fake, it

has to learn how to recognize the generator's flaws. That's a different problem for

a thoroughly trained generator than it is for an untrained generator that produces

random output.
 ● Similarly, we keep the discriminator constant during the generator training phase.

Otherwise the generator would be trying to hit a moving target and might never

converge.

It's this back and forth that allows GANs to tackle otherwise intractable generative

problems. We get a toehold in the difficult generative problem by starting with a much

simpler classification problem. Conversely, if you can't train a classifier to tell the

difference between real and generated data even for the initial random generator

output, you can't get the GAN training started.

Convergence

● Convergence in the context of Generative Adversarial Networks (GANs) refers to

the stabilization of the training process where the generator produces high-

quality and diverse samples, and the discriminator is no longer able to distinguish

between real and generated data effectively.

● Achieving convergence in GANs can be challenging due to various issues, such

as mode collapse, training instability, and vanishing/exploding gradients.

● As the generator improves with training, the discriminator performance gets

worse because the discriminator can't easily tell the difference between real and

fake.

● If the generator succeeds perfectly, then the discriminator has a 50% accuracy.

In effect, the discriminator flips a coin to make its prediction.

 ● This progression poses a problem for convergence of the GAN as a whole: the

discriminator feedback gets less meaningful over time.

● If the GAN continues training past the point when the discriminator is giving

completely random feedback, then the generator starts to train on junk feedback,

and its own quality may collapse.

Note: GAN convergence is an active area of research, and there is ongoing work

to develop more stable and efficient training techniques

Loss Function
● GANs try to replicate a probability distribution.

● They should therefore use loss functions that reflect the distance between the

distribution of the data generated by the GAN and the distribution of the real

data.

● minimax loss: The loss function used in the original paper.

One Loss Function or Two?

● A GAN can have two loss functions: one for generator training and one for

discriminator training.

● How can two loss functions work together to reflect a distance measure between

probability distributions?
 ● In the loss schemes we'll look at here, the generator and discriminator losses

derive from a single measure of distance between probability distributions.

● In both of these schemes, however, the generator can only affect one term in the

distance measure: the term that reflects the distribution of the fake data. So

during generator training we drop the other term, which reflects the distribution of

the real data.

● The generator and discriminator losses look different in the end, even though

they derive from a single formula.

Minimax Loss

In this, the generator tries to minimize the following function while the discriminator tries

to maximize it:

In this function:

● D(x) is the discriminator's estimate of the probability that real data instance x is

real.

● Ex is the expected value over all real data instances.

 ● G(z) is the generator's output when given noise z.

● D(G(z)) is the discriminator's estimate of the probability that a fake instance is

real.

● Ez is the expected value over all random inputs to the generator (in effect, the

expected value over all generated fake instances G(z)).

● The formula derives from the cross-entropy between the real and generated

distributions.

The generator can't directly affect the log(D(x)) term in the function, so, for the

generator, minimizing the loss is equivalent to minimizing log(1 - D(G(z))).

Different Types of GAN Models
● Vanilla GAN
● DCGAN
● WGAN
● Conditional GAN
● Cycle GAN

Vanilla GAN

● This is the simplest type of GAN.

● Here, the Generator and the Discriminator are simple multi-layer
perceptrons.
● In vanilla GAN, the algorithm is really simple, it tries to optimize the
mathematical equation using stochastic gradient descent.
Architecture:

The architecture of a Vanilla Generative Adversarial Network (GAN) is relatively simple

and consists of two main components: a generator and a discriminator. Both the
generator and discriminator are neural networks, typically implemented using fully
connected layers. Here's a general outline of the architecture:

Generator Architecture:
 Input Layer:
● The generator starts with a low-dimensional random noise vector, often
sampled from a normal distribution.
Fully Connected Layer:
● The noise vector is passed through a fully connected layer to transform it
into a higher-dimensional representation.
Activation Function (e.g., ReLU or Sigmoid):
● A non-linear activation function, such as Rectified Linear Units (ReLU) or
Sigmoid, is applied to introduce non-linearity into the network.
Fully Connected Layer(s):
● One or more additional fully connected layers follow, gradually increasing
the dimensionality.
Output Layer:
● The final layer produces the generated data, which could be in the form of
images, vectors, or any other data type depending on the application.

Discriminator Architecture:
Input Layer:
● The discriminator takes as input either real data (e.g., images) or
generated data from the generator.
Fully Connected Layer(s):
● The input data is passed through one or more fully connected layers to
produce a single output value.
Activation Function (e.g., Sigmoid):
● A sigmoid activation function is often applied to the output layer to produce
a probability score indicating whether the input data is real or generated.

Training Process:
● The generator and discriminator are trained simultaneously in an adversarial
manner.
● The generator tries to generate realistic data to fool the discriminator, while the
discriminator tries to distinguish between real and generated data.
● The loss function involves minimizing the generator's loss and maximizing the
discriminator's loss.
Challenges and Enhancements:
● While the Vanilla GAN architecture is conceptually simple, training GANs can be
challenging due to issues such as mode collapse, training instability, and the
potential for the generator and discriminator to become imbalanced.
● Researchers have introduced various enhancements and modifications to
address these challenges, leading to the development of more sophisticated
GAN variants like DCGAN (Deep Convolutional GAN), WGAN (Wasserstein
GAN), and others.
● These enhancements often involve using convolutional layers, batch
normalization, and advanced activation functions to improve stability and
generate higher-quality samples.

Deep Convolutional GAN (DCGAN)

● DCGAN is one of the most popular and also the most successful
implementations of GAN.
● It is composed of ConvNets in place of multi-layer perceptrons.
● The ConvNets are implemented without max pooling, which is in fact
replaced by convolutional stride.
● Also, the layers are not fully connected.
Figure: The generator of DCGAN with four sequential fractionally strided
convolutional layers.

● DCGAN generator used for LSUN scene modeling.

● A 100 dimensional uniform distribution Z is projected to a small spatial extent
convolutional representation with many feature maps.
● A series of four fractionally-strided convolutions (in some recent papers, these
are wrongly called deconvolutions) then convert this high level representation
into a 64 × 64 pixel image.
● Notably, no fully connected or pooling layers are used.

Note:

What is Fractionally-Strided Convolution?

● Fractionally-strided convolution is essentially the reverse of a standard

convolution operation.
● While standard convolutional layers reduce the spatial dimensions of the input
data (downsampling), fractionally-strided convolutions aim to increase the spatial
dimensions (upsampling).
● This is achieved by reversing the forward and backward passes of a convolution.
 ● In a standard convolution, an input is convolved with a filter to produce an output
feature map.
● In fractionally-strided convolution, the goal is to learn a convolution operation
that, given an output feature map, can produce an input feature map of a larger
spatial size.
● This is done by applying a stride in the input space that is a fraction of the
convolutional filter size, hence the name "fractionally-strided."

● where the dashed white cells are zero rows/columns padded between the input cells
(blue)

How Does It Work?

● The process of fractionally-strided convolution involves inserting zeros between

the entries of the input feature map, effectively increasing its dimensions.
● This expanded map is then convolved with a learned filter (or kernel), resulting in
an output that is larger than the original input.
 ● The spacing between the input values is determined by the stride, which, in this
case, is fractional. For example, a stride of 1/2 would double the spatial
dimensions of the input.

It is important to note that the term "deconvolution" can be somewhat misleading.

Deconvolution implies an exact reversal of the convolution process, which is not strictly
what happens in fractionally-strided convolution. Instead, this operation should be
thought of as a learnable upsampling that uses convolutional principles.

Architectural guidelines for Stable DCGANs:

● DCGANs were proposed with some structural changes to the original

GANs which were unstable to train in the sense that the network could
collapse after certain epochs, where the generator produced nonsensical
outputs.
● There were five significant modifications done as follows:
1. The first change was to use only convolutional layers, instead of
the traditional alternating convolution and max-pooling layers
followed by full connection with ANNs.

This was done because a convolutional layer with augmented stride

can replace a max-pooling layer without any loss in accuracy.

This allowed the generator to learn its own spatial upsampling

(through transposed convolutional layers or fractionally strided
convolutional layers) and similarly for the discriminator to learn its
own spatial downsampling.

2. Secondly, the full connection on top of the highest convolutional

features was eliminated.
 The highest convolutional features were connected to the input of
the generator and the output of the discriminator.

Generator G takes a noise distribution z as input which can be called

a full connection (because it is only a matrix multiplication) and as for
discriminator D, the last convolutional layer is flattened to be fed into
a single sigmoid output.

3. The third change was to use batch normalization in both

networks (except for the G output layer and D input layer to avoid
sample oscillation and model instability).

Batch normalization normalizes data to have zero mean and unit

variance, which averts training problems arising from poor
initialization and helps to have strong gradients in deep networks.

This change showed that G could initially learn better with stronger
gradients and also avoided it to collapse later on where all the
generated samples would be the same and not make any sense
(however, this can still happen with DCGANs.

4. The fourth and fifth changes made respectively were to use

rectified linear unit (reLU) as activation function for the G
(except the final layer which had tanh activation, allowing the model
to learn quicker to convergence and utilize the whole spectrum of
the colours from the training data) and to use leaky reLU for D for
higher resolution modeling which incorporates a negative slope in
the negative domain of the function to achieve better results in
neural networks.
 ● DCGANs have been used for CNN-based image recognition , automatic
sketch colourization , gesture recognition, object regeneration , infrared
image colourization, etc.

Key Features of DCGAN:

Use of Convolutional Layers:

● Unlike the original GAN, which uses fully connected layers, DCGAN

implements convolutional layers, making it more suitable for image data.

Batch Normalization:

● Applied to both the Generator and Discriminator to stabilize training, help

with gradient flow, and prevent the model from collapsing to a single

mode.

Activation Functions:

● The Generator typically uses ReLU activation for all layers except for the

output, which uses tanh.

● The Discriminator uses LeakyReLU activation to provide a path for

gradients when the unit is not active.

Strided Convolutions:
 ● The Discriminator uses strided convolutions to reduce the spatial

dimensions of the image, while the Generator uses fractional-strided

convolutions for upsampling.

No Pooling Layers:

● DCGANs use strided convolutions for downsampling in the Discriminator

and fractional-strided convolutions for upsampling in the Generator,

avoiding pooling layers.

Challenges and Solutions:

● Mode Collapse: A common challenge in GANs where the Generator starts

producing a limited variety of outputs. DCGANs address this to some extent

through its architecture and batch normalization.

● Training Stability: DCGANs are more stable than vanilla GANs but still require

careful tuning of hyperparameters and training regimen.

Applications:
● Image Generation: High-quality and diverse image generation.

● Feature Learning: DCGANs can learn useful representations of images, which

can be used in various supervised and unsupervised learning tasks.

Conclusion:
● DCGANs represent a significant advancement in the GAN architecture,
 especially for tasks involving image data.

● By incorporating convolutional layers and other architectural improvements,

DCGANs not only generate higher quality images but also offer a more stable

training process compared to the original GAN framework.

● They have opened the door to numerous applications in image generation,

super-resolution, and more, and have laid the groundwork for further innovations

in the field of generative models.

Wasserstein GAN (WGAN)

● Wasserstein Generative Adversarial Network (Wasserstein GAN or
WGAN) is a type of generative model introduced by Martin Arjovsky,
Soumith Chintala, and Léon Bottou in 2017.
([Link]
● WGAN addresses some of the issues associated with traditional
GANs, such as mode collapse and training instability.
● The key idea behind WGAN is the use of Wasserstein distance (also
known as Earth Mover's distance) as the metric to measure the
dissimilarity between the generated and real data distributions.
● Wasserstein distance provides a more stable and meaningful
gradient for training the generator, compared to the Jensen-Shannon
divergence or the Kullback-Leibler divergence used in traditional
GANs.

Key components of Wasserstein GAN include:

➢ Wasserstein Distance:

○ The Wasserstein distance measures the minimum cost of

turning one probability distribution into another.
○ In the context of GANs, it provides a more continuous and
meaningful measure of the difference between the generated
and real data distributions.
○ In traditional GAN, the discriminator produces a probability
score indicating whether a given sample is real or fake.
○ In WGAN the discriminator’s output is not a probability but
rather the estimate of a distance between the generated
sample’s distribution and the real data [Link] distance
is measured using Wasserstein distance, which represents the
minimum cost of transforming one distribution into another.
○ The Wasserstein distance for the real data distribution Pr

and the generated data distribution Pg is mathematically

defined as the greatest lower bound (infimum) for any

transport plan (i.e. the cost for the cheapest plan):

○ Π(Pr, Pg) denotes the set of all joint distributions γ(x, y)

whose marginals are respectively Pr and Pg.

 Wasserstein GAN (WGAN) proposes a new cost function
using Wasserstein distance that has a smoother gradient
everywhere.

○ The above diagram below repeats a similar plot on the

value of D(X) for both GAN and WGAN.
○ For GAN (the red line), it fills with areas with diminishing or
exploding gradients.
○ For WGAN (the blue line), the gradient is smoother
everywhere and learns better even if the generator is not
producing good images.
➢ Wasserstein Loss:
○ However, the equation for the Wasserstein distance is

highly intractable.
○ Using the Kantorovich-Rubinstein duality, we can simplify the

calculation to

○ where sup is the least upper bound and f is a 1-Lipschitz

function following this constraint :

So to calculate the Wasserstein distance, we just need to

find a 1-Lipschitz function.

○ The network design is almost the same except the critic

does not have an output sigmoid function. The major

difference is only on the cost function:

● However, there is one major thing missing. f has to be a 1-

Lipschitz function.
● To enforce the constraint, WGAN applies a very simple clipping

to restrict the maximum weight value in f, i.e. the weights of the

discriminator must be within a certain range controlled by the

hyperparameters c.

➢ Lipschitz Continuity:

○ In Wasserstein Generative Adversarial Networks (WGANs), the

concept of Lipschitz continuity plays a crucial role in ensuring
the stability and convergence of the training process.
○ The Wasserstein distance, which is the basis of the WGAN loss
function, involves the optimization over a space of functions
that are 1-Lipschitz continuous.
 ● To enforce Lipschitz continuity in WGANs, a weight clipping technique is
commonly used.
● Weight clipping involves constraining the weights of the critic network to lie within
a certain range, typically [-c, c], where c is a hyperparameter.
● This clipping ensures that the Lipschitz constant of the critic remains bounded by
1.

The WGAN loss is formulated to maximize the difference between the average critic
scores for real and generated samples. By enforcing Lipschitz continuity, WGANs aim
to find a stable and meaningful Wasserstein distance between the real and generated
distributions, leading to improved training stability and better convergence properties
compared to traditional GANs.

➢ Stability and Training Improvement:

○ WGANs are known for their improved stability during training

and their ability to generate higher quality samples.
○ They are less prone to mode collapse and exhibit better
convergence properties compared to traditional GANs.
➢ Weight Clipping vs. Gradient Penalty:

○ Initially, WGANs used weight clipping to enforce the Lipschitz

constraint. However, it was later discovered that weight clipping
can lead to training issues.
○ The gradient penalty was proposed as an alternative method to
enforce Lipschitz continuity without the drawbacks of weight
clipping.
Wasserstein GANs have been influential in advancing the field of
generative models and have inspired further research on GAN variants and
improvements. It's important to note that research in this field is dynamic,
and newer developments may have occurred since my last update in
January 2022.

Key Concepts of WGAN:

1. Wasserstein Distance:
● The WGAN uses the Wasserstein distance (also known as the Earth

Mover's Distance, or EMD) as a loss function, which provides a more

meaningful and smooth gradient everywhere.

● It measures the distance between two probability distributions in a

way that translates to the cost of transporting mass in converting one

distribution to another.

2. Critic Instead of Discriminator:

● In traditional GANs, the Discriminator classifies inputs as real or fake.

In WGAN, this role is replaced by a Critic.

● The Critic estimates the Wasserstein distance between the

distributions of real and generated data, rather than classifying inputs

as real or fake.

3. Weight Clipping:
 ● To enforce a Lipschitz constraint (a mathematical condition needed

for Wasserstein distance), WGAN introduces weight clipping. This

means the weights of the Critic are clipped to a fixed range after each

gradient update.

Training Process of:

GAN
Training a Wasserstein Generative Adversarial Network (WGAN) involves optimizing
the parameters of both the generator and the critic (discriminator) networks to minimize
the Wasserstein distance, as defined by the WGAN loss function. Here's a step-by-step
guide to WGAN training:

Define the Generator and Critic Networks:

● Define the architecture of the generator (G) and critic (D) networks.
● Choose an appropriate activation function for the generator's output layer
(commonly tanh for image generation).
Define the WGAN Loss Function:
● Define the WGAN loss function, which involves the Wasserstein distance
and the weight clipping constraint for Lipschitz continuity.
Initialize the Generator and Critic Networks:
● Initialize the weights of the generator and critic networks.
Compile the Critic Model:
● Compile the critic model with an optimizer (commonly RMSprop or Adam)
and the WGAN loss function.
Compile the Combined Model:
● Compile the combined model (generator + critic) with an optimizer, but
freeze the critic's weights during the combined model training.
Training Loop:
● For each training iteration:
● Sample a batch of real data from the dataset.
● Generate a batch of fake samples using the generator.
● Train the critic:
● Compute the critic loss using real and fake samples.
 ● Update the critic's weights using gradient descent or another
optimization algorithm.
● Clip the critic's weights to enforce Lipschitz continuity.
● Train the generator:
● Generate a new batch of fake samples.
● Compute the generator loss using the critic's output.
● Update the generator's weights using gradient ascent.
Evaluate and Monitor:
● Periodically evaluate the generator on a fixed set of seed noise inputs to
monitor the quality of generated samples.
● Monitor the Wasserstein distance during training to assess convergence.
Repeat Until Convergence:
● Repeat the training loop until the generator produces samples that closely
match the real data distribution and the Wasserstein distance converges
to a stable value.

Remember that tuning hyperparameters, such as the learning rates, weight clipping
parameter, and the architecture of the networks, is essential for successful training.
Experimentation and monitoring are crucial to achieve the desired results with WGANs.

Advantages of WGAN:

Improved Training Stability:

● The Wasserstein distance provides more stable and meaningful

gradients. This leads to a more stable training process

compared to traditional GANs.

Reduced Mode Collapse:

● WGANs are less prone to mode collapse, where the Generator

starts producing a limited variety of outputs.

More Meaningful Loss Metric:

● The Wasserstein distance correlates better with the quality of

generated images. In standard GANs, the loss does not always

 correlate well with the perceptual quality of generated samples.

Challenges and Modifications:

Weight Clipping Problem:

● Weight clipping in WGAN can lead to optimization difficulties

and convergence issues.

● This led to the development of WGAN with Gradient Penalty

(WGAN-GP), which replaces weight clipping with a gradient

penalty term, addressing these issues and improving model

performance.

Computational Cost:

● Training WGANs, especially the Critic, can be computationally

intensive due to the need for more frequent updates.

Applications of WGAN:

● Image Synthesis: Producing high-quality and diverse images.

● Data Augmentation: Generating realistic data samples for training

machine learning models.

● Domain Adaptation: Adapting data from one domain to have

characteristics of another domain.

In conclusion, WGANs represent a significant improvement over traditional

GANs, particularly in terms of training stability and the quality of the

generated data. The introduction of the Wasserstein distance as a loss

function and the concept of a Critic instead of a Discriminator have

influenced subsequent developments in the field of generative models.

CONDITIONAL GAN

● In some situations you wanted your GAN to generate a specific type

of data but did not have sufficient control over GANs outputs.

● For example,

○ assume you used a broad spectrum of flower images to train a

GAN capable of producing fake pictures of flowers.

○ While you can use your model to generate an image of a

random flower, you cannot instruct it to create an image of,

say, a tulip or a sunflower.

● Conditional GAN (cGAN) allows us to condition the network with

additional information such as class labels.

● It means that during the training, we pass images to the network with

their actual labels (rose, tulip, sunflower etc.) for it to learn the

difference between them.

 ● That way, we gain the ability to ask our model to generate images of

specific flowers.

● This type of GAN allows the generator to create data that is

conditioned on a set of input data.

Basic GAN model:

In the case of a Conditional GAN,

● We want to condition both the Generator and the Discriminator

so they know which type they are dealing with.

● Say we use our GAN to create synthetic data containing house

prices in London and Madrid.

● To make it conditional, we need to tell the Generator which city

to generate the data for each time.

● We also need to inform the Discriminator whether the example

passed to it is for London or Madrid.

So the Conditional GAN model architecture would look like this:

 ● A Conditional Generative Adversarial Network (cGAN) is an extension of the

basic Generative Adversarial Network (GAN) framework, which allows the

generation of more specific or conditioned outputs.

● Introduced by Mirza and Osindero in 2014, cGANs enable the model to generate

data conditioned on certain auxiliary information, such as class labels or other

data types. This conditioning can be applied to both the Generator and the

Discriminator.

Key Components of cGAN:

1. Generator (G):
● Input: A random noise vector (from the latent space) and a conditional variable

(like a class label).

● Output: A generated sample that not only looks realistic but also aligns with the

conditional input.
2. Discriminator (D):
● Input: Real or fake data samples along with the conditional variable.

● Output: A probability score indicating whether the input sample is real or fake,

considering the conditioning.

How cGAN Works:

● Conditional Variable Integration:

○ The conditional variable (such as a class label or any other relevant data)

is integrated into both the Generator and the Discriminator. This variable

can be concatenated with the input noise vector for the Generator and

with the input data for the Discriminator.

● Training the Generator:

○ The Generator learns to produce data that not only appears real but also

aligns with the conditioning variable. For instance, in the case of image

generation, if the condition is a class label, the Generator learns to

produce images corresponding to specific classes.

● Training the Discriminator:

○ The Discriminator learns to distinguish between real and fake data,

considering the conditioning. It evaluates not just the authenticity of the

data but also how well the data aligns with the given condition.

Loss function of cGAN:

● Generative adversarial nets can be extended to a conditional model if both the

generator and discriminator are conditioned on some extra information y.

● y could be any kind of auxiliary information, such as class labels or data from

other modalities.

● We can perform the conditioning by feeding y into both the discriminator and

generator as an additional input layer.

● In the generator the prior input noise pz(z), and y are combined in joint hidden

representation, and the adversarial training framework allows for considerable

flexibility in how this hidden representation is composed.

● In the discriminator x and y are presented as inputs and to a discriminative

function (embodied again by a MLP in this case).

● The objective function of a two-player minimax game would be as:

The structure of a simple conditional adversarial net using MLP:

Advantages of cGAN:
Controlled Data Generation:

● cGANs allow for the generation of data that is conditioned on certain

attributes, enabling more control over the output.

Diversity in Generation:

● By varying the conditional variables, cGANs can generate a diverse range

of outputs within the same framework.

Improved Relevance:

● The conditional aspect often leads to the generation of more relevant and

targeted data.

Challenges and Considerations:

 Quality of Conditioning Data:

● The effectiveness of a cGAN strongly depends on the quality and

relevance of the conditional data.

Mode Collapse:

● While cGANs offer more control, they can still suffer from mode collapse,

where the Generator produces limited varieties of outputs.

Complexity in Training:

● Integrating and properly utilizing conditional variables can add complexity

to the training process.

Applications of cGAN:
● Image-to-Image Translation: For tasks like turning sketches into colored images,

satellite images to maps, etc.

● Text-to-Image Generation: Generating images from descriptive text.

● Style Transfer: Generating images in a particular style conditioned on a reference

image.

● Personalized Data Generation: Generating data tailored to specific conditions or

user preferences.

In summary, Conditional GANs represent a significant step towards more controlled and

diverse data generation in the field of generative models. By incorporating auxiliary

information, cGANs offer enhanced capabilities compared to traditional GANs,

especially in tasks where specific output characteristics are desired.

CYCLE GAN

● a range of application domains, most notably translating photographs of

horses to zebra, and the reverse.

● The main idea behind CycleGAN is the use of cycle consistency.

● CYCLE GAN is an approach for learning to translate an image from a

source domain X to a target domain Y in the absence of paired examples.

● Our goal is to learn a mapping G : X → Y such that the distribution of

images from G(X) is indistinguishable from the distribution Y using an

adversarial loss.

● Because this mapping is highly under-constrained, we couple it with an

inverse mapping F : Y → X and introduce a cycle consistency loss to

enforce F(G(X)) ≈ X (and vice versa).

Image-to-image translation involves generating a new synthetic version of a

given image with a specific modification, such as translating a summer landscape

to winter.

● Training a model for image-to-image translation typically requires a large

dataset of paired examples. These datasets can be difficult and expensive

to prepare, and in some cases impossible, such as photographs of

paintings by long dead artists.

● The CycleGAN is a technique that involves the automatic training of

image-to-image translation models without paired examples.

Figure : Paired training data (left) consists of training examples {xi , yi} N i=1, where the

correspondence between xi and yi exists. We instead consider unpaired training data (right),

consisting of a source set {xi} N i=1 (xi ∈ X) and a target set {yj}M j=1 (yj ∈ Y ), with no information

provided as to which xi matches which yj .

● In Cycle GAN the models are trained in an unsupervised manner using a

collection of images from the source and target domain that do not need to

be related in any way.

● This simple technique is powerful, achieving visually impressive results on

from Y to X.

Key Components of CycleGAN:

Two Generators:

● G: Translates images from domain X to domain Y.

 ● F: Translates images from domain Y to domain X.

Two Discriminators:

● Dx: Differentiates between images from domain X and translated images from Y

to X.

● Dy: Differentiates between images from domain Y and translated images

from X to Y.

How CycleGAN Works:

Unpaired Training Data:

● CycleGAN learns to translate between two domains without paired

examples. It only requires a collection of images from each domain.

Cycle Consistency Loss:

● This is a crucial part of CycleGAN. It ensures that an image translated

from one domain to the other and then back again should resemble the

original image. This cycle consistency loss is mathematically formulated to

minimize the difference between the original and the cycled image.

Adversarial Training:

● Similar to traditional GANs, each Generator is trained to produce images

that look similar to the authentic images from the target domain, while

each Discriminator tries to distinguish between real and generated

 images.

The Training Process:

● Training G (X → Y) and Dy:

○ G attempts to generate images that look like they're from domain Y from

input images in domain X, while Dy learns to differentiate between the real

Y images and the fake Y images produced by G.

● Training F (Y → X) and Dx:

○ F attempts to generate images that look like they're from domain X from

input images in domain Y, while Dx learns to differentiate between the real

X images and the fake X images produced by F.

● Applying Cycle Consistency:

○ For a given X image, when passed through G to get G(X) and then F to

get F(G(X)), the final result should be close to the original X. The same

applies in the reverse direction for Y.

Loss Function of Cycle GAN:
● Our goal is to learn mapping functions between two domains X and Y given

training samples {xi} N i=1 where xi ∈ X and {yj}M j=1 where yj ∈ Y .

● We denote the data distribution as x ∼ pdata(x) and y ∼ pdata(y). As illustrated in

Figure 3 (a), our model includes two mappings G : X → Y and F : Y → X.

● In addition, we introduce two adversarial discriminators DX and DY , whereDX

aims to distinguish between images {x} and translated images {F(y)}; in the same

way, DY aims to discriminate between {y} and {G(x)}.

● Our objective contains two types of terms:

○ adversarial losses for matching the distribution of generated images to

the data distribution in the target domain;

○ cycle consistency losses to prevent the learned mappings G and F from

contradicting each other.

● Adversarial Loss

○ We apply adversarial losses to both mapping functions.

○ For the mapping function G : X → Y and its discriminator DY , we express

 the objective as:

● where G tries to generate images G(x) that look similar to images from

domain Y , while DY aims to distinguish between translated samples G(x)

and real samples y.

● G aims to minimize this objective against an adversary D that tries to

maximize it, i.e., minG maxDY LGAN(G, DY , X, Y ).

● We introduce a similar adversarial loss for the mapping function F : Y →

X and its discriminator DX as well: i.e., minF maxDX LGAN(F, DX, Y, X).

Cycle Consistency Loss:

● A network can map the same set of input images to any random permutation of

images in the target domain, where any of the learned mappings can induce an

output distribution that matches the target distribution.

● Thus, adversarial losses alone cannot guarantee that the learned function can

map an individual input xi to a desired output yi .

● To further reduce the space of possible mapping functions, we argue that the

learned mapping : as shown in Figure 3 (b), for each image x from domain X,

the image translation cycle should be able to bring x back to the original image,
 i.e., x → G(x) → F(G(x)) ≈ x. We call this forward cycle consistency.

● Similarly, as illustrated in Figure 3 (c), for each image y from domain Y , G and

F should also satisfy backward cycle consistency: y → F(y) → G(F(y)) ≈ y. We

incentivize this behavior using a cycle consistency loss:

Full Objective:

where λ controls the relative importance of the two objectives.

We aim to solve:

Advantages of CycleGAN:
Eliminates Need for Paired Data:

● It can learn to translate between domains without the need for

corresponding pairs of images in the two domains.

Diverse Applications:

● Useful in various tasks like photo enhancement, art-to-real translation,

season transfer, object transfiguration, and more.

 Impressive Results in Style Transfer:

● Capable of capturing and transferring styles between completely different

sets of images.

Challenges and Limitations:

Preserving Key Attributes:

● CycleGAN may alter or lose important details of the original images,

especially when there are significant differences between domains.

Dependency on Data Quality:

● The quality of the generated images heavily depends on the diversity and

representativeness of the training datasets.

Computational Intensity:

● Requires significant computational resources due to the complexity of

training four networks (two Generators and two Discriminators).

Common Problems with GAN

GANs have a number of common failure modes. All of these common

problems are areas of active research. While none of these problems have
been completely solved, we'll mention some things that people have tried.

1. Vanishing Gradients
 ● Research has suggested that if your discriminator is too good, then
generator training can fail due to vanishing gradients.
● In effect, an optimal discriminator doesn't provide enough information
for the generator to make progress.

Attempts to Remedy

● Wasserstein loss: The Wasserstein loss is designed to prevent

vanishing gradients even when you train the discriminator to

optimality.
● Modified minimax loss: The original GAN paper proposed a

modification to minimax loss to deal with vanishing gradients.

Wasserstein Loss

● This loss function depends on a modification of the GAN scheme

(called "Wasserstein GAN" or "WGAN") in which the discriminator
does not actually classify instances.
● For each instance it outputs a number. This number does not have
to be less than one or greater than 0, so we can't use 0.5 as a
threshold to decide whether an instance is real or fake. Discriminator
training just tries to make the output bigger for real instances than for
fake instances.
● Because it can't really discriminate between real and fake, the WGAN
discriminator is actually called a "critic" instead of a "discriminator".
● This distinction has theoretical importance, but for practical purposes
we can treat it as an acknowledgement that the inputs to the loss
functions don't have to be probabilities.

The loss functions themselves are deceptively simple:

Critic Loss: D(x) - D(G(z))

 The discriminator tries to maximize this function. In other words, it tries to
maximize the difference between its output on real instances and its output
on fake instances.

Generator Loss: D(G(z))

The generator tries to maximize this function. In other words, It tries to

maximize the discriminator's output for its fake instances.

In these functions:

2. D(x) is the critic's output for a real instance.

3. G(z) is the generator's output when given noise z.
4. D(G(z)) is the critic's output for a fake instance.
5. The output of critic D does not have to be between 1 and 0.
6. The formulas derive from the earth mover distance between the real
and generated distributions.

2. Mode Collapse

● Usually you want your GAN to produce a wide variety of outputs.

● You want, for example, a different face for every random input to your
face generator.
● However, if a generator produces an especially plausible(possible,
thinkable) output, the generator may learn to produce only that
output. In fact, the generator is always trying to find the one output
that seems most plausible to the discriminator.
● If the generator starts producing the same output (or a small set of
outputs) over and over again, the discriminator's best strategy is to
learn to always reject that output. But if the next generation of
discriminator gets stuck in a local minimum and doesn't find the best
 strategy, then it's too easy for the next generator iteration to find the
most plausible output for the current discriminator.
● Each iteration of generator over-optimizes for a particular
discriminator, and the discriminator never manages to learn its way
out of the trap. As a result the generators rotate through a small
set of output types. This form of GAN failure is called mode
collapse.
● Mode collapse happens when the generator maps multiple
distinct inputs to the same output, which means that the
generator produces samples of low diversity.

Attempts to Remedy

The following approaches try to force the generator to broaden its scope by
preventing it from optimizing for a single fixed discriminator:

● Wasserstein loss: The Wasserstein loss alleviates mode collapse by

letting you train the discriminator to optimality without worrying about

vanishing gradients. If the discriminator doesn't get stuck in local
minima, it learns to reject the outputs that the generator stabilizes on.
So the generator has to try something new.
● Unrolled GANs: Unrolled GANs use a generator loss function that

incorporates not only the current discriminator's classifications, but

also the outputs of future discriminator versions. So the generator
can't over-optimize for a single discriminator.

3. Failure to Converge

● GANs frequently fail to converge.

● Convergence problems refer to difficulties in training the model to
generate high-quality and diverse samples consistently.
● The generator tries to find the best image to fool the discriminator.
The “best” image keeps changing when both networks counteract
 their opponent. However, the optimization can turn too greedy and fall
into a never-ending cat-and-mouse game. This is one of the
scenarios where the model does not converge and mode collapses.

Attempts to Remedy

Researchers have tried to use various forms of regularization to improve

GAN convergence, including:

● Adding noise to discriminator inputs: therefore smoothening the

distribution of the probability mass.

● Penalizing discriminator weights: