Module-2

Generative Adversarial Network

● Generative Adversarial Networks (GANs) were developed in 2014 by

Ian Goodfellow and his teammates.

● GAN is basically an approach to generative modeling that generates

a new set of data based on training data that look like training data.

● To understand the term GAN let’s break it into separate three parts

○ Generative – To learn a generative model, which describes

how data is generated in terms of a probabilistic model. In

simple words, it explains how data is generated visually.

○ Adversarial – The training of the model is done in an

adversarial setting.

○ Networks – use deep neural networks for training purposes.

Architecture of GAN

● GANs have two main blocks(two neural networks) which compete

with each other and are able to capture, copy, and analyze the

variations in a dataset.

● The two models are usually called Generator and Discriminator

● The generator network

○ Role: The generator's role is to create new data instances that

resemble the training data.

○ Input: It takes random noise as input and generates samples

that ideally cannot be distinguished from real data.

○ The generator tries to create realistic samples to fool the

discriminator.

● The discriminator network:

○ It tries to distinguish between real and generated samples.

○ It is trained with real samples from the training data and

generated samples from the generator.

○ The discriminator’s objective is to correctly classify real data as

real and generated data as fake.

○ The discriminator is trained to improve its ability to differentiate

between real and fake samples.

● The training process:

○ involves an adversarial game between the generator and the

discriminator.
 ○ The generator aims to produce samples that fool the

discriminator, while the discriminator tries to improve its ability

to distinguish between real and generated data.

○ This adversarial training pushes both networks to improve over

time.

The Discriminator Model

● The discriminator in a GAN is simply a classifier.

● It tries to distinguish real data from the data created by the generator.
 ● It could use any network architecture appropriate to the type of data it's

classifying.

Figure1: Backpropagation in discriminator training.

Discriminator Training Data

The discriminator's training data comes from two sources:

● Real data instances, such as real pictures of people. The discriminator uses

these instances as positive examples during training.

● Fake data instances created by the generator. The discriminator uses these

instances as negative examples during training.

● In Figure 1, the two "Sample" boxes represent these two data sources feeding

into the discriminator.

● During discriminator training the generator does not train. Its weights remain

constant while it produces examples for the discriminator to train on.

Training the Discriminator

● The discriminator connects to two loss functions.

● During discriminator training, the discriminator ignores the generator loss and just

uses the discriminator loss.

● We use the generator loss during generator training, as described in the next

section.

During discriminator training:

1. The discriminator classifies both real data and fake data from the generator.

2. The discriminator loss penalizes the discriminator for misclassifying a real

instance as fake or a fake instance as real.

3. The discriminator updates its weights through backpropagation from the

discriminator loss through the discriminator network.

The Generator

● The generator part of a GAN learns to create fake data by incorporating feedback

from the discriminator.

● It learns to make the discriminator classify its output as real.

Generator training requires tighter integration between the generator and the

discriminator than discriminator training requires. The portion of the GAN that trains the

generator includes:

● random input

● generator network, which transforms the random input into a data instance

● discriminator network, which classifies the generated data

● discriminator output

● generator loss, which penalizes the generator for failing to fool the discriminator

Figure 2: Backpropagation in generator training.

Random Input

● In its most basic form, a GAN takes random noise as its input.

● The generator then transforms this noise into a meaningful output.

 ● By introducing noise, we can get the GAN to produce a wide variety of data,

sampling from different places in the target distribution.

● Experiments suggest that the distribution of the noise doesn't matter much, so we

can choose something that's easy to sample from, like a uniform distribution. For

convenience the space from which the noise is sampled is usually of smaller

dimension than the dimensionality of the output space.

Note: Some GANs use non-random input to shape the output.

Using the Discriminator to Train the Generator

● To train a neural net, we alter the net's weights to reduce the error or loss of its

output.

● In our GAN, however, the generator is not directly connected to the loss that

we're trying to affect. The generator feeds into the discriminator net, and the

discriminator produces the output we're trying to affect.

● The generator loss penalizes the generator for producing a sample that the

discriminator network classifies as fake.

● This extra chunk of network must be included in backpropagation.

○ Backpropagation adjusts each weight in the right direction by calculating

the weight's impact on the output — how the output would change if you

changed the weight.

 ○ But the impact of a generator weight depends on the impact of the

discriminator weights it feeds into.

○ So backpropagation starts at the output and flows back through the

discriminator into the generator.

At the same time, we don't want the discriminator to change during generator training.

Trying to hit a moving target would make a hard problem even harder for the generator.

So we train the generator with the following procedure:

1. Sample random noise.

2. Produce generator output from sampled random noise.

3. Get discriminator "Real" or "Fake" classification for generator output.

4. Calculate loss from discriminator classification.

5. Backpropagate through both the discriminator and generator to obtain gradients.

6. Use gradients to change only the generator weights.

GAN Training

Because a GAN contains two separately trained networks, its training algorithm must

address two complications:

 ● GANs must juggle two different kinds of training (generator and discriminator).

● GAN convergence is hard to identify.

Alternating Training

The generator and the discriminator have different training processes. So how do we

train the GAN as a whole?

GAN training proceeds in alternating periods:

1. The discriminator trains for one or more epochs.

2. The generator trains for one or more epochs.

3. Repeat steps 1 and 2 to continue to train the generator and discriminator

networks.

● We keep the generator constant during the discriminator training phase. As

discriminator training tries to figure out how to distinguish real data from fake, it

has to learn how to recognize the generator's flaws. That's a different problem for

a thoroughly trained generator than it is for an untrained generator that produces

random output.
 ● Similarly, we keep the discriminator constant during the generator training phase.

Otherwise the generator would be trying to hit a moving target and might never

converge.

It's this back and forth that allows GANs to tackle otherwise intractable generative

problems. We get a toehold in the difficult generative problem by starting with a much

simpler classification problem. Conversely, if you can't train a classifier to tell the

difference between real and generated data even for the initial random generator

output, you can't get the GAN training started.

Convergence

● Convergence in the context of Generative Adversarial Networks (GANs) refers to

the stabilization of the training process where the generator produces high-

quality and diverse samples, and the discriminator is no longer able to distinguish

between real and generated data effectively.

● Achieving convergence in GANs can be challenging due to various issues, such

as mode collapse, training instability, and vanishing/exploding gradients.

● As the generator improves with training, the discriminator performance gets

worse because the discriminator can't easily tell the difference between real and

fake.

● If the generator succeeds perfectly, then the discriminator has a 50% accuracy.

In effect, the discriminator flips a coin to make its prediction.

 ● This progression poses a problem for convergence of the GAN as a whole: the

discriminator feedback gets less meaningful over time.

● If the GAN continues training past the point when the discriminator is giving

completely random feedback, then the generator starts to train on junk feedback,

and its own quality may collapse.

Note: GAN convergence is an active area of research, and there is ongoing work

to develop more stable and efficient training techniques

Loss Function
● GANs try to replicate a probability distribution.

● They should therefore use loss functions that reflect the distance between the

distribution of the data generated by the GAN and the distribution of the real

data.

● minimax loss: The loss function used in the original paper.

One Loss Function or Two?

● A GAN can have two loss functions: one for generator training and one for

discriminator training.

● How can two loss functions work together to reflect a distance measure between

probability distributions?
 ● In the loss schemes we'll look at here, the generator and discriminator losses

derive from a single measure of distance between probability distributions.

● In both of these schemes, however, the generator can only affect one term in the

distance measure: the term that reflects the distribution of the fake data. So

during generator training we drop the other term, which reflects the distribution of

the real data.

● The generator and discriminator losses look different in the end, even though

they derive from a single formula.

Minimax Loss

In this, the generator tries to minimize the following function while the discriminator tries

to maximize it:

In this function:

● D(x) is the discriminator's estimate of the probability that real data instance x is

real.

● Ex is the expected value over all real data instances.

 ● G(z) is the generator's output when given noise z.

● D(G(z)) is the discriminator's estimate of the probability that a fake instance is

real.

● Ez is the expected value over all random inputs to the generator (in effect, the

expected value over all generated fake instances G(z)).

● The formula derives from the cross-entropy between the real and generated

distributions.

The generator can't directly affect the log(D(x)) term in the function, so, for the

generator, minimizing the loss is equivalent to minimizing log(1 - D(G(z))).

Different Types of GAN Models
● Vanilla GAN
● DCGAN
● WGAN
● Conditional GAN
● Cycle GAN

Vanilla GAN

● This is the simplest type of GAN.

● Here, the Generator and the Discriminator are simple multi-layer
perceptrons.
● In vanilla GAN, the algorithm is really simple, it tries to optimize the
mathematical equation using stochastic gradient descent.
Architecture:

The architecture of a Vanilla Generative Adversarial Network (GAN) is relatively simple

and consists of two main components: a generator and a discriminator. Both the
generator and discriminator are neural networks, typically implemented using fully
connected layers. Here's a general outline of the architecture:

Generator Architecture:
 Input Layer:
● The generator starts with a low-dimensional random noise vector, often
sampled from a normal distribution.
Fully Connected Layer:
● The noise vector is passed through a fully connected layer to transform it
into a higher-dimensional representation.
Activation Function (e.g., ReLU or Sigmoid):
● A non-linear activation function, such as Rectified Linear Units (ReLU) or
Sigmoid, is applied to introduce non-linearity into the network.
Fully Connected Layer(s):
● One or more additional fully connected layers follow, gradually increasing
the dimensionality.
Output Layer:
● The final layer produces the generated data, which could be in the form of
images, vectors, or any other data type depending on the application.

Discriminator Architecture:
Input Layer:
● The discriminator takes as input either real data (e.g., images) or
generated data from the generator.
Fully Connected Layer(s):
● The input data is passed through one or more fully connected layers to
produce a single output value.
Activation Function (e.g., Sigmoid):
● A sigmoid activation function is often applied to the output layer to produce
a probability score indicating whether the input data is real or generated.

Training Process:
● The generator and discriminator are trained simultaneously in an adversarial
manner.
● The generator tries to generate realistic data to fool the discriminator, while the
discriminator tries to distinguish between real and generated data.
● The loss function involves minimizing the generator's loss and maximizing the
discriminator's loss.
Challenges and Enhancements:
● While the Vanilla GAN architecture is conceptually simple, training GANs can be
challenging due to issues such as mode collapse, training instability, and the
potential for the generator and discriminator to become imbalanced.
● Researchers have introduced various enhancements and modifications to
address these challenges, leading to the development of more sophisticated
GAN variants like DCGAN (Deep Convolutional GAN), WGAN (Wasserstein
GAN), and others.
● These enhancements often involve using convolutional layers, batch
normalization, and advanced activation functions to improve stability and
generate higher-quality samples.

Deep Convolutional GAN (DCGAN)

● DCGAN is one of the most popular and also the most successful
implementations of GAN.
● It is composed of ConvNets in place of multi-layer perceptrons.
● The ConvNets are implemented without max pooling, which is in fact
replaced by convolutional stride.
● Also, the layers are not fully connected.
Figure: The generator of DCGAN with four sequential fractionally strided
convolutional layers.

● DCGAN generator used for LSUN scene modeling.

● A 100 dimensional uniform distribution Z is projected to a small spatial extent
convolutional representation with many feature maps.
● A series of four fractionally-strided convolutions (in some recent papers, these
are wrongly called deconvolutions) then convert this high level representation
into a 64 × 64 pixel image.
● Notably, no fully connected or pooling layers are used.

Note:

What is Fractionally-Strided Convolution?

● Fractionally-strided convolution is essentially the reverse of a standard

convolution operation.
● While standard convolutional layers reduce the spatial dimensions of the input
data (downsampling), fractionally-strided convolutions aim to increase the spatial
dimensions (upsampling).
● This is achieved by reversing the forward and backward passes of a convolution.
 ● In a standard convolution, an input is convolved with a filter to produce an output
feature map.
● In fractionally-strided convolution, the goal is to learn a convolution operation
that, given an output feature map, can produce an input feature map of a larger
spatial size.
● This is done by applying a stride in the input space that is a fraction of the
convolutional filter size, hence the name "fractionally-strided."

● where the dashed white cells are zero rows/columns padded between the input cells
(blue)

How Does It Work?

● The process of fractionally-strided convolution involves inserting zeros between

the entries of the input feature map, effectively increasing its dimensions.
● This expanded map is then convolved with a learned filter (or kernel), resulting in
an output that is larger than the original input.
 ● The spacing between the input values is determined by the stride, which, in this
case, is fractional. For example, a stride of 1/2 would double the spatial
dimensions of the input.

It is important to note that the term "deconvolution" can be somewhat misleading.

Deconvolution implies an exact reversal of the convolution process, which is not strictly
what happens in fractionally-strided convolution. Instead, this operation should be
thought of as a learnable upsampling that uses convolutional principles.

Architectural guidelines for Stable DCGANs:

● DCGANs were proposed with some structural changes to the original

GANs which were unstable to train in the sense that the network could
collapse after certain epochs, where the generator produced nonsensical
outputs.
● There were five significant modifications done as follows:
1. The first change was to use only convolutional layers, instead of
the traditional alternating convolution and max-pooling layers
followed by full connection with ANNs.

This was done because a convolutional layer with augmented stride

can replace a max-pooling layer without any loss in accuracy.

This allowed the generator to learn its own spatial upsampling

(through transposed convolutional layers or fractionally strided
convolutional layers) and similarly for the discriminator to learn its
own spatial downsampling.

2. Secondly, the full connection on top of the highest convolutional

features was eliminated.
 The highest convolutional features were connected to the input of
the generator and the output of the discriminator.

Generator G takes a noise distribution z as input which can be called

a full connection (because it is only a matrix multiplication) and as for
discriminator D, the last convolutional layer is flattened to be fed into
a single sigmoid output.

3. The third change was to use batch normalization in both

networks (except for the G output layer and D input layer to avoid
sample oscillation and model instability).

Batch normalization normalizes data to have zero mean and unit

variance, which averts training problems arising from poor
initialization and helps to have strong gradients in deep networks.

This change showed that G could initially learn better with stronger
gradients and also avoided it to collapse later on where all the
generated samples would be the same and not make any sense
(however, this can still happen with DCGANs.

4. The fourth and fifth changes made respectively were to use

rectified linear unit (reLU) as activation function for the G
(except the final layer which had tanh activation, allowing the model
to learn quicker to convergence and utilize the whole spectrum of
the colours from the training data) and to use leaky reLU for D for
higher resolution modeling which incorporates a negative slope in
the negative domain of the function to achieve better results in
neural networks.
 ● DCGANs have been used for CNN-based image recognition , automatic
sketch colourization , gesture recognition, object regeneration , infrared
image colourization, etc.

Key Features of DCGAN:

Use of Convolutional Layers:

● Unlike the original GAN, which uses fully connected layers, DCGAN

implements convolutional layers, making it more suitable for image data.

Batch Normalization:

● Applied to both the Generator and Discriminator to stabilize training, help

with gradient flow, and prevent the model from collapsing to a single

mode.

Activation Functions:

● The Generator typically uses ReLU activation for all layers except for the

output, which uses tanh.

● The Discriminator uses LeakyReLU activation to provide a path for

gradients when the unit is not active.

Strided Convolutions:
 ● The Discriminator uses strided convolutions to reduce the spatial

dimensions of the image, while the Generator uses fractional-strided

convolutions for upsampling.

No Pooling Layers:

● DCGANs use strided convolutions for downsampling in the Discriminator

and fractional-strided convolutions for upsampling in the Generator,

avoiding pooling layers.

Challenges and Solutions:

● Mode Collapse: A common challenge in GANs where the Generator starts

producing a limited variety of outputs. DCGANs address this to some extent

through its architecture and batch normalization.

● Training Stability: DCGANs are more stable than vanilla GANs but still require

careful tuning of hyperparameters and training regimen.

Applications:
● Image Generation: High-quality and diverse image generation.

● Feature Learning: DCGANs can learn useful representations of images, which

can be used in various supervised and unsupervised learning tasks.

Conclusion:
● DCGANs represent a significant advancement in the GAN architecture,
 especially for tasks involving image data.

● By incorporating convolutional layers and other architectural improvements,

DCGANs not only generate higher quality images but also offer a more stable

training process compared to the original GAN framework.

● They have opened the door to numerous applications in image generation,

super-resolution, and more, and have laid the groundwork for further innovations

in the field of generative models.