DEEP LEARNING

23DS5PCDLG
• Representation Learning using • Generative Adversarial Networks
Autoencoders: (GANs):
Stacked Autoencoders Difficulties of training GANs
Convolutional Autoencoders Deep Convolutional GANs
Recurrent Autoencoders Progressive Growing of GANs
Denoising Autoencoders
StyleGANs
Sparse Autoencoders
Applications of Autoencoders
Transfer Learning, and Domain
Adaptation.
Generative Adversarial Networks
(GANs)
• Proposed in 2014 by Ian Goodfellow et al.
• Concept:
• Two neural networks compete against each other, improving through
this competition.
• Components: A GAN consists of two networks:
1. Generator: Creates fake data.
2. Discriminator: Tries to distinguish between real and fake data.
• Generator:
• Input: Random distribution (usually Gaussian).
• Output: Generated data (e.g., an image).
• Function: Similar to a decoder in a variational autoencoder.
• Goal: Generate new images by feeding in Gaussian noise.
• Discriminator
• Takes either a fake image from the generator or a real image
from the training set as input, and must guess whether the input
image is fake or real.
• During training, the generator and the discriminator have opposite
goals:
• The discriminator tries to tell fake images from real images
• The generator tries to produce images that look real enough to trick
the discriminator.
• Because the GAN is composed of two networks with different
objectives, it cannot be trained like a regular neural network.
• Each training iteration is divided into two phases:
• Phase 1: Train Discriminator • Phase 2: Train Generator
• Use real and fake images. • Generate fake images.
• Labels: 1 for real, 0 for fake. • Discriminator tries to classify
• Train the discriminator with them as real (labels set to 1).
binary cross-entropy. • Freeze discriminator's weights.
• Only update the discriminator's • Update generator's weights to
weights. improve its ability to fool the
discriminator.
• The generator never sees real images directly.
• It learns by using gradients from the discriminator.
• As the discriminator improves, these gradients carry richer
information about real images, helping the generator produce more
realistic fakes over time.
A simple GAN for Fashion MNIST
• Generator: • Discriminator: A binary classifier.
• Functions like an autoencoder's • Input: Image.
decoder. • Output: Single unit with sigmoid
• Takes random noise as input and activation for real (1) or fake (0).
generates data (e.g., images).

GAN Model: Combines generator and

discriminator. Used during the second training
phase.
• Model Compilation Steps:
• Discriminator:Use binary cross-entropy loss (binary classifier).
• Generator: No direct compilation; trained only through the GAN
model.
• Set the discriminator to non-trainable before compiling GAN to freeze
its weights during GAN training.
• Training loop is unusual - cannot use the regular fit() method.
• So,
1. First, create a Dataset to iterate through the images
2. Then, write a custom training loop
Write the training loop, wrap it in a train_gan() function
• Two Phases per Training Iteration:
• Phase One (Train the Discriminator):
• Input: Gaussian noise to the generator → fake
images.
• Combine fake images with real images into a batch.
• Targets (y1): 0 for fake, 1 for real.
• Train the discriminator on this batch.
• Set [Link] = True to avoid warnings.
• Phase Two (Train the Generator through GAN):
• Input: Gaussian noise to the GAN.
• GAN generates fake images → discriminator
classifies them.
• Targets (y2): All set to 1 (make discriminator believe
fakes are real).
• Set [Link] = False to avoid
warnings.
Generated Output

• Training Challenges in GANs:

• After the first epoch, generated images may resemble noisy versions
of the target dataset (e.g., Fashion MNIST).
• Images often fail to improve significantly beyond a point.
• The GAN may even "forget" previously learned patterns during some
epochs.
• Reason: GAN training is inherently challenging and prone to instability.
Difficulties of training GANs
• Training GANs: The Zero-Sum Game
• The generator and discriminator compete in a zero-sum game, trying to
outsmart each other.
• Training may converge to a Nash equilibrium, where neither network
benefits from changing its strategy unless the other changes.
• Example:
• Everyone driving on the same side of the road is a Nash equilibrium—no one
benefits by switching.
• Predators chasing prey and prey escaping is another, with competing strategies in
balance.
• The outcome depends on initial conditions and training dynamics.
How does this apply to GANs?
• GAN Nash Equilibrium
• A GAN reaches Nash equilibrium when:
• The generator creates perfectly realistic images.
• The discriminator can only guess at random (50% real, 50% fake).
• Training long enough could lead to a perfect generator.
• Challenge: Reaching this equilibrium is not guaranteed, even with
extended training.
• Mode Collapse in GANs
• Definition: The generator's outputs lose diversity, focusing on one or
a few classes.
• Example:
• The generator gets better at creating shoes than other classes.
• It produces more shoes, forgetting other classes.
• The discriminator only sees fake shoes and forgets to detect other fakes.
• Result:
• The GAN cycles between classes (e.g., shoes → shirts) without mastering any.
• Other Challenges in Training GANs
1. Oscillations:
Generator and discriminator constantly compete.
Parameters may oscillate, causing instability.
2. Sudden Divergence:
Training may start well but can diverge unexpectedly.
3. Hyperparameter Sensitivity:
GANs are highly sensitive to hyperparameters.
Extensive fine-tuning is often required.
• Addressing GAN Training Challenges
• Numerous papers propose solutions to stabilize training and tackle mode
collapse.
• Key Techniques:
1. Experience Replay:
Store generated images in a buffer.
Train the discriminator using real images and buffered fakes.
Prevents discriminator overfitting to the latest generator outputs.
2. Mini-Batch Discrimination:
Measures similarity within a batch.
Encourages diversity by rejecting non-diverse fake batches.
• Advancements:
• Novel cost functions (efficacy debated).
• Specialized architectures with better performance.
Deep Convolutional GANs(DCGANs)
• Origin:
• Inspired by the 2014 GAN paper, which used convolutional layers for small
images.
• Researchers struggled with instability for larger images.
• Breakthrough:
• Alec Radford et al. (2015) succeeded with deeper convolutional GANs after
extensive experimentation.
• Impact:
• Introduced DCGANs, paving the way for stable training with larger images.
• The main guidelines proposed for building stable convolutional
GANs:
1. Replace any pooling layers with strided convolutions (in the
discriminator) and transposed convolutions (in the generator).
2. Use Batch Normalization in both the generator and the
discriminator, except in the generator’s output layer and the
discriminator’s input layer.
3. Remove fully connected hidden layers for deeper architectures.
4. Use ReLU activation in the generator for all layers except the output
layer, which should use tanh.
5. Use leaky ReLU activation in the discriminator for all layers.
• These guidelines will work in many cases, but not always, so need to
experiment with different hyperparameters
Example-DCGAN with Fashion MNIST:
1. Input: 4. Upsampling (2nd Transposed
• Takes a coding vector of size 100. Convolution):
2. First Layer: • Stride: 2.
• Projects the coding to 6272 • Upscales tensor from 14 × 14 to
dimensions (7 × 7 × 128). 28 × 28.
• Reshapes it into a 7 × 7 × 128 tensor. • Reduces depth from 64 to 1.
• Applies batch normalization. 5. Output Layer:
3. Upsampling (1st Transposed • Uses the tanh activation function.
Convolution):
• Outputs values in the range [-1, 1].
• Stride: 2.
• Upscales tensor from 7 × 7 to 14 × 6. Data Preprocessing:
14. • Rescale the training set to the
• Reduces depth from 128 to 64. range [-1, 1] before training the
GAN.
• Batch normalization applied.
• Reshape it to add the channel dimension:
• reshaping and adding the channel dimension allows the generator to create
images with the correct structure, ensuring each pixel has the proper color
values.

• The discriminator is similar to a CNN for binary classification, but with key
differences:
• Strided Convolutions: Used instead of max pooling to downsample (stride=2).
• Leaky ReLU: Activation function used to prevent dying neurons.
• Modifications:
• Dropout: Replaces Batch Normalization in the discriminator for stability.
• SELU: Replaces ReLU in the generator for better training and self-normalization.
• Hyperparameter Sensitivity:
• The architecture is sensitive to hyperparameters, especially the learning rates of the
generator and discriminator.
Nice-to-know
• When working with images in neural networks, especially in deep learning models like
GANs, adding a channel dimension is essential because:
• Image Structure: Images are typically represented in 3D arrays (height × width ×
channels). The channels represent color information:
• Grayscale images have a single channel (1 channel).
• RGB images have 3 channels (Red, Green, Blue).
• RGBA images have 4 channels (Red, Green, Blue, Alpha/Transparency).
• Neural Network Compatibility: Convolutional neural networks (CNNs) and GANs expect
the input and output images to include this channel dimension. Without it, the network
would not know how to process or generate color information.
• Proper Representation:
• When reshaping the output of the generator, you're constructing the final image from latent
vectors. The channel dimension is added to specify the depth of the image, telling the network
how many values per pixel to output (e.g., 1 for grayscale, 3 for RGB).
• In the final layer of the generator, the output shape needs to explicitly include this channel
dimension (e.g., 28x28x1 for grayscale, or 28x28x3 for RGB) to represent a complete image.
• Earlier code: build the dataset,compile and train model.
• After 50 epochs of training, the generator produces images like those
shown in Figure
• Scaling up the DCGAN architecture and training it on a large face dataset
produces realistic images. (Fig in next slide)
• Summary of the results:
• Latent Space Representation: Averaging codings of specific categories (e.g.,
men with glasses, women without glasses) generates meaningful images.
• Latent Space Arithmetic: By performing operations like "men with glasses -
men without glasses + women without glasses," you can generate new
images, such as a woman with glasses.
• Semantic Interpolation: DCGANs can interpolate between images by adding
noise to the latent space vector, showcasing their ability to perform
arithmetic on faces.
 A Conditional GAN (CGAN)
adds the class of each image
as extra input to both the
generator and discriminator.

This allows the generator to

control the class of images it
produces, based on the
provided class label.
• DCGANs can struggle with generating large images, often resulting in local
features looking convincing but global inconsistencies, like uneven sleeves.
• To fix this, you can use more advanced architectures or techniques.
Progressive Growing of GANs
• In 2018, Nvidia researchers proposed a technique where small images
are generated initially, then gradually larger images are produced by
adding convolutional layers.
• This resembles greedy layer-wise training in stacked autoencoders.
• New layers are added to the generator and discriminator, while
previously trained layers stay trainable.
• Growing Generator Outputs from 4x4 to 8x8:
1. Add an upsampling layer (nearest neighbor filtering) to expand
feature maps to 8x8.
2. New convolutional layer follows, using "same" padding and strides
of 1, outputting 8x8.
3. Output convolutional layer (kernel size 1) projects to desired color
channels (e.g., 3).
4. Weighted sum of original and new layers:
Original layer weight = (1 - α)
New layer weight = α
Gradually increase α from 0 to 1.
5. Similar fade-in/fade-out for the discriminator.
 Progressively growing GAN:
a GAN generator outputs 4 × 4 color images
(left);
we extend it to output 8 × 8 images (right)
Techniques to increase the diversity of the outputs
(to avoid mode collapse) and making training
more stable:
1. Minibatch standard deviation layer
• Adding near the end of the discriminator
• Compute standard deviation across all channels and instances in the batch: s
= [Link].reduce_std(inputs, axis=[0, -1])
• Average standard deviations across all points to get a single value: v
= tf.reduce_mean(S)
• Add extra feature map to each batch instance, filled with the computed
value:
[Link]([inputs, [Link]([batch_size, height, width, 1], v)], axis=-1)
• Benefit:
Helps the discriminator detect low diversity in generated images.
Encourages the generator to produce more diverse outputs, reducing mode collapse.
2. Equalized learning rate
• Initial weights: Gaussian distribution with mean 0 and standard
deviation 1 (instead of He initialization).
• Rescaling at runtime: Weights are divided by sqrt(2/ninputs)​, where
ninputs​is the number of inputs to the layer.
• Benefit:
Ensures a consistent dynamic range for all parameters during training.
Prevents instability and speeds up training.
Works well with optimizers like RMSProp or Adam.
3. Pixelwise normalization layer
• Added after each convolutional layer in the generator
• It normalizes each activation based on all the activations in the same
image and at the same location, but across all channels.
• In TensorFlow code
inputs / [Link](tf.reduce_mean([Link](X), axis=-1, keepdims=True) +
1e-8)
• the smoothing term 1e-8 is needed to avoid division by zero.
• This technique avoids explosions in the activations due to excessive
competition between the generator and the discriminator.
• Evaluation Challenge in GANs:
1. Convincing Images: Hard to define; quality is subjective.
2. Diversity Evaluation: Easier to measure automatically.
3. Quality Evaluation: Requires human raters, which is costly
and time-consuming.
4. Proposed Solution: Measure similarity between local
image structure of generated images and training images at
multiple scales.
5. Outcome: Led to the development of StyleGANs.
StyleGANs
• StyleGAN (2018) by Nvidia:
• Advancement: Improved high-resolution image
generation.
• Technique: Used style transfer in the generator to
match local structure at every scale.
• Discriminator & Loss Function: Not modified.
• Architecture: Composed of two networks: Mapping
and Synthesis.
StyleGAN’s generator architecture (part of figure 1 from the StyleGAN paper)
Paper link: [Link]
Mapping network
• An eight-layer MLP that maps the latent representations z (i.e., the
codings) to a vector w.
• This vector is then sent through multiple affine transformations (i.e.,
Dense layers with no activation functions, represented by the “A”
boxes in Figure), which produces multiple vectors.
• These vectors control the style of the generated image at different
levels, from fine-grained texture (e.g., hair color) to high-level
features (e.g., adult or child).
• In short, the mapping network maps the codings to multiple style
vectors.
 Synthesis network
• Responsible for generating the images.
• It has a constant learned input (this input will be constant after training, but
during training it keeps getting tweaked by backpropagation).
• It processes this input through multiple convolutional and upsampling layers, but

First, some noise is added to the input and to all the outputs of the
convolutional layers (before the activation function).
Second, each noise layer is followed by an Adaptive Instance Normalization
(AdaIN) layer:
• it standardizes each feature map independently (by subtracting the feature map’s mean
and dividing by its standard deviation),
• then it uses the style vector to determine the scale and offset of each feature map (the
style vector contains one scale and one bias term for each feature map).
• Adding Noise in StyleGAN: • Benefits:
• Noise is added independently from 1. Avoids wasting coding space for
the codings. randomness.
• Noise is unique for each level. 2. No need for noise to pass through
• Each noise input is a Gaussian noise the network to final layers,
feature map. speeding up training.
• The noise is scaled using learned per- 3. Prevents visual artifacts from
feature scaling factors. reusing noise at different levels.
• The scaled noise is added to each 4. Generator doesn’t waste
feature map at the level. resources creating pseudorandom
• represented by the “B” boxes in Figure noise, leading to better results.
• Mixing Regularization (Style Mixing):
• Two codings (c1, c2) are used for style generation.
• Codings are processed through the mapping network to produce two
style vectors (w1, w2).
• The generator creates the image using w1 for the first levels and w2
for the remaining levels.
• The cutoff level is selected randomly, preventing style correlation at
adjacent levels.
• This technique encourages locality, ensuring that each style vector
impacts only specific traits of the image.
---XXX---