DEEP LEARNING

Course Code: 2344106

Comprehensive Study Notes – All Units

Programme [Link] Semester 7

Branch CSE (Data Science) Category Professional Core

Credits 3 L–T–P 3–0–0

COURSE UNITS
Unit 1 – Introduction to Deep Learning (9 Hrs)
Unit 2 – Convolutional Neural Networks (9 Hrs)
Unit 3 – Recurrent Neural Networks (9 Hrs)
Unit 4 – Generative Adversarial Networks (9 Hrs)
Unit 5 – Auto-Encoders (9 Hrs)

Ravi Kiran | REC | 2024–25

Total Contact Hours: 45
 SYLLABUS
Deep Learning (2344106) | [Link] CSE | Professional Core

PREAMBLE
This course covers the foundations and modern techniques of Deep Learning including neural networks, CNNs,
RNNs, GANs, and Autoencoders, with applications in computer vision, natural language processing, and
generative modelling.

PREREQUISITES
Basic knowledge of Machine Learning, Linear Algebra, and Python programming is required.

COURSE OBJECTIVES
• To understand the basic concepts and techniques of Deep Learning and the need of Deep Learning
techniques in real-world problems.
• To understand CNN algorithms and the way to evaluate performance of the CNN architectures.
• To apply RNN and LSTM to learn, predict and classify the real-world problems in the paradigms of Deep
Learning.
• To understand, learn and design GANs for the selected problems.
• To understand the concept of Auto-encoders and enhancing GANs using auto-encoders.

COURSE CONTENTS
Unit Title Topics Covered Hrs

Historical Trends, ANN, XOR/XNOR, Single/Multiple Layer

Introduction to Deep Perceptron, Feed Forward Networks, Stochastic Gradient-Based
Unit 1 9
Learning Learning, Hidden Units, Architecture Design, Back-Propagation,
TensorFlow/Keras, PyTorch.

CNNs and applications in computer vision, CNN architecture,

Convolutional Neural Activation functions (Sigmoid, Tanh, ReLU, Softmax), Pooling
Unit 2 9
Networks (CNN) layers, Training in TensorFlow, VGG/GoogLeNet/ResNet,
Dropout, Batch Normalization, Data Augmentation.

RNNs and sequential data analysis, BPTT, Vanishing Gradient

Recurrent Neural
Unit 3 Problem, Gradient Clipping, LSTM Networks, GRU, Bidirectional 9
Networks (RNN)
LSTMs and RNNs.

Generative models, GAN concept and principles, Generator and

Generative Adversarial Discriminator architecture, Discriminative vs. Generative models,
Unit 4 9
Networks (GANs) Types: DCGAN, CGAN, LSGAN, CycleGAN, SRGAN,
Applications.

Architecture (Encoder, Bottleneck, Decoder), Training for

compression and reconstruction, Variants: Denoising, Variational
Unit 5 Auto-Encoders 9
(VAE), Convolutional Autoencoders, Relationship with GANs,
Hybrid Encoder-Decoder GANs.

Total Contact Hours 45

COURSE OUTCOMES
CO Description BT Level

Understand the basic concepts and techniques of Deep Learning and the need
CO 1 Understanding
of Deep Learning techniques in real-world problems.
 CO Description BT Level

Understand CNN algorithms and evaluate performance of the CNN

CO 2 Understanding
architectures.

Apply RNN and LSTM to learn, predict and classify real-world problems in the
CO 3 Applying
paradigms of Deep Learning.

CO 4 Understand, learn and design GANs for selected problems. Applying

Understand the concept of Auto-encoders and enhancing GANs using auto-

CO 5 Applying
encoders.

TEXT BOOKS
1. Deep Learning — Ian Goodfellow, Yoshua Bengio, Aaron Courville, MIT Press.
2. Neural Networks and Deep Learning — Michael Nielsen, Determination Press, 2015.
3. Neural Networks: A Classroom Approach — Satish Kumar, Tata McGraw-Hill Education, 2004.

REFERENCE BOOKS
1. Deep Learning with Python — François Chollet, Manning Publications, 2018.
2. Advanced Deep Learning with Keras — Rowel Atienza, PACKT Publications, 2018.
3. Hands-On Machine Learning with Scikit-Learn, Keras & TensorFlow — Aurélien Géron, O'Reilly, 2019.
4. Generative Deep Learning — David Foster, O'Reilly Media, 2019.

ASSESSMENT PATTERN
Test Remembering (K1) Understanding (K2) Applying (K3) Total

MID-1 6% 9% 85% 100%

MID-2 6% 9% 85% 100%

SEE 10% 10% 80% 100%

* ±3% may be varied

 UNIT 1
Introduction to Deep Learning
Deep Learning (2344106) • [Link] CSE • 9 Contact Hours

Topics Covered in Unit 1

1. Historical Trends in Deep Learning
2. Why Deep Learning is Growing
3. Artificial Neural Networks (ANN)
4. Non-linear Classification — XOR/XNOR Example
5. Single & Multiple Layer Perceptron
6. Feed Forward Networks & Deep Feed-Forward Networks
7. Stochastic Gradient-Based Learning
8. Hidden Units & Architecture Design
9. Back-Propagation Algorithm
10. Deep Learning Frameworks — TensorFlow/Keras & PyTorch
1. Historical Trends in Deep Learning
Deep Learning has a rich and evolving history spanning more than seven decades. Understanding its
historical trajectory helps us appreciate why it has become one of the most transformative technologies
of the 21st century. The field did not emerge overnight — it went through multiple phases of excitement,
neglect, and revival before achieving its current dominance.
1.1 The Three Waves of Neural Networks
Wave 1 — Cybernetics (1940s–1960s): The first wave began with the pioneering work of McCulloch
and Pitts in 1943, who introduced a simplified mathematical model of a neuron. Frank Rosenblatt's
Perceptron (1957) created immense excitement as it could learn to classify linearly separable patterns.
However, Minsky and Papert's 1969 book 'Perceptrons' exposed fundamental limitations — the single-
layer perceptron could not solve XOR. This caused the first 'AI Winter'.
Wave 2 — Connectionism (1980s–1990s): Interest revived with the discovery of the Backpropagation
algorithm (Rumelhart, Hinton & Williams, 1986), which could train multi-layer networks. The Universal
Approximation Theorem (Cybenko, 1989) proved that a neural network with even one hidden layer
could theoretically approximate any continuous function. However, vanishing gradients and
computational limits led to a second stagnation by the late 1990s. Support Vector Machines (SVMs)
and shallow methods dominated.
Wave 3 — Deep Learning (2006–Present): Hinton et al.'s breakthrough paper in 2006 on Deep Belief
Networks showed how to pre-train deep networks layer by layer. The true watershed moment came in
2012 when AlexNet — a deep convolutional neural network — won the ImageNet competition with a
top-5 error of 15.3%, nearly halving the competition's second-best result. This ignited the modern deep
learning revolution.

Key Historical Milestones

1943 — McCulloch & Pitts: Mathematical model of the neuron
1957 — Rosenblatt: Perceptron algorithm
1969 — Minsky & Papert: 'Perceptrons' book — First AI Winter
1986 — Rumelhart, Hinton & Williams: Backpropagation
1989 — Cybenko: Universal Approximation Theorem
1998 — LeCun: LeNet-5 for digit recognition (CNN)
2006 — Hinton: Deep Belief Networks — revival begins
2012 — Krizhevsky: AlexNet wins ImageNet — Modern Deep Learning era
2014 — Goodfellow: Generative Adversarial Networks (GANs)
2017 — Vaswani et al.: Transformer architecture
2022 — OpenAI: ChatGPT — LLMs enter mainstream
2. Why Deep Learning is Growing
Deep Learning's explosive growth is driven by the convergence of three critical factors, often referred to
as the "DDD Triangle": Data, Compute, and Algorithms. When all three came together around 2012,
it triggered a paradigm shift.
2.1 Big Data
Modern deep learning models are inherently data-hungry. The internet age has created an
unprecedented explosion of labeled and unlabeled data. ImageNet contains over 14 million labeled
images. Social platforms generate billions of text posts, images, and videos daily. Sensors, IoT devices,
and scientific instruments continuously produce structured and unstructured data. This abundance of
data allows deep networks — which have millions or even billions of parameters — to generalize
without overfitting, something that was impossible in the data-scarce era of early neural networks.
2.2 Computational Power — GPUs
Graphics Processing Units (GPUs), originally designed for rendering 3D games, turned out to be
perfectly suited for the massively parallel matrix operations that underlie neural network training. A
modern GPU can contain thousands of cores operating simultaneously. The key insight by Raina et al.
(2009) and the landmark AlexNet paper (2012) demonstrated that GPU-accelerated training could
reduce training time from weeks to days. Today, specialized hardware like Google's Tensor Processing
Units (TPUs) and NVIDIA's H100 GPUs further accelerate deep learning.
2.3 Algorithmic Improvements
Beyond data and hardware, numerous algorithmic advances have made training deep networks more
reliable and efficient:
• ReLU Activation: Replaces sigmoid/tanh and largely solves the vanishing gradient problem
• Dropout Regularization: Prevents overfitting in deep networks
• Batch Normalization: Stabilizes and accelerates training
• Residual Connections (ResNets): Enable training of networks 100+ layers deep
• Adam Optimizer: Adaptive learning rate optimizer that converges faster than SGD
• Transfer Learning: Pre-trained models on large datasets can be fine-tuned for specific tasks
2.4 Open-Source Ecosystem
The democratization of deep learning has been accelerated by open-source frameworks like
TensorFlow (Google, 2015), PyTorch (Facebook, 2016), and Keras. These tools abstract away low-
level GPU programming and allow researchers and engineers to build sophisticated models with tens of
lines of Python code. Pre-trained model repositories like Hugging Face Hub and TensorFlow Hub have
further lowered entry barriers.
3. Artificial Neural Network (ANN)
An Artificial Neural Network (ANN) is a computational model loosely inspired by the structure and
function of biological neural networks in the brain. It consists of interconnected processing units called
neurons (or nodes), organized in layers, that collectively learn to map input data to desired outputs by
adjusting the strength of their connections during training.
3.1 Biological vs. Artificial Neuron
Biological Neuron Artificial Neuron
• Dendrites receive signals from other • Inputs (x1, x2, ..., xn) receive signals
neurons
• Weighted sum computes net input
• Cell body (soma) sums the inputs
• Activation function controls output
• Axon transmits the output signal
• Weights model synaptic strength
• Synapse is the connection point
• Continuous or discrete output
• All-or-nothing firing (action potential)
• Networks have millions of nodes
• Brain has ~86 billion neurons

3.2 Mathematical Model of a Neuron

A single artificial neuron computes:
z = w₁x₁ + w₂x₂ + ... + wₙxₙ + b = Σ(wᵢxᵢ) + b (Weighted Sum)
ŷ = f(z) (Activation Function applied to z)
Where: x = input features, w = weights (learned parameters), b = bias (threshold), f = activation
function, ŷ = predicted output.

┌────────────────────────────────────────────────────────
─────┐
│ Artificial Neuron │
│ │
│ x₁ ──(w₁)──┐ │
│ x₂ ──(w₂)──┤──► [ Σ wᵢxᵢ + b ] ──► [ f(z) ] ──► ŷ │
│ x₃ ──(w₃)──┘ │
│ ▲ │
│ bias (b) │

└────────────────────────────────────────────────────────
─────┘

3.3 Structure of an ANN

A typical ANN is organized into three types of layers:
• Input Layer: Receives the raw input features. Each node corresponds to one feature. No
computation occurs here — data simply flows in.
• Hidden Layers: One or more intermediate layers where computation happens. Neurons here
detect increasingly abstract features of the data. The 'deep' in deep learning refers to having
many hidden layers.
 • Output Layer: Produces the final predictions. For binary classification, a single sigmoid neuron;
for multi-class, softmax over as many neurons as there are classes; for regression, a linear
neuron.

INPUT LAYER HIDDEN LAYER 1 HIDDEN LAYER 2 OUTPUT LAYER

x₁ ──┐ ○ ──┐ ○ ──┐

├──────────► ○ ├────────────► ○ ├──────────────► ŷ
x₂ ──┤ ○ ──┤ ○ ──┤
├──────────► ○ │ ○ │
x₃ ──┘ ○ ──┘ ○ ──┘

(3 inputs) (4 neurons) (4 neurons) (1 output)

4. Non-Linear Classification: XOR/XNOR Example

The XOR problem is historically significant because it was the key example used by Minsky and Papert
(1969) to demonstrate that a single-layer perceptron cannot solve non-linearly separable problems.
This limitation drove researchers to explore multi-layer networks.
4.1 The XOR Truth Table
x₁ x₂ XOR XNOR
0 0 0 1
0 1 1 0
1 0 1 0
1 1 0 1

When we plot these four points on a 2D plane, we see that XOR outputs 1 when inputs differ and 0
when they match. The two classes (0s and 1s) cannot be separated by a single straight line — the
problem is non-linearly separable. No single-layer perceptron can solve this.
4.2 Neural Network Solution to XOR
A two-layer network solves XOR by learning an intermediate feature representation in the hidden
layer that transforms the original space into one that is linearly separable. The network effectively
learns two hidden neurons: one that activates for (0,0) and (1,1), and another that encodes the XOR
relationship. The output neuron then linearly separates the hidden representations.

XOR Network Architecture (2-2-1):

x₁ ──────┬──(w=-1,b=1.5)──► h₁ = σ(−x₁−x₂+1.5) ─┐
│ ├──► y
x₂ ──────┼──(w=-1,b=0.5)──► h₂ = σ(x₁+x₂−0.5) ─┘
└

Output y = σ(−2h₁ + 2h₂ − 1) [with threshold activation]

Key Insight: The network learns an intermediate representation (feature space) that makes the
problem linearly separable. This ability to learn hierarchical representations is the core power of deep
learning.

5. Single and Multiple Layer Perceptron

5.1 Single Layer Perceptron (SLP)
The Perceptron, invented by Frank Rosenblatt in 1957, is the simplest form of a neural network. It
consists of a single layer of weights connecting inputs directly to an output neuron. It is a binary linear
classifier — it can only correctly classify data that is linearly separable.
Perceptron Learning Algorithm:
4. Initialize all weights w and bias b to 0 (or small random values)
5. For each training sample (x, y_true):
Compute: ŷ = step(Σwᵢxᵢ + b)
Update: wᵢ ← wᵢ + η(y_true − ŷ)xᵢ and b ← b + η(y_true − ŷ)
6. Repeat until all samples classified correctly or max epochs reached
Where η (eta) is the learning rate (typically 0.01–0.1). The perceptron is guaranteed to converge if the
data is linearly separable (Perceptron Convergence Theorem).

Perceptron Limitations
• Cannot solve XOR or any non-linearly separable problem
• Only one layer, so it can only learn a linear decision boundary
• Uses a step/threshold activation — not differentiable
• No probabilistic output — just hard 0 or 1
• These limitations motivated the development of Multi-Layer Perceptrons (MLPs)

5.2 Multi-Layer Perceptron (MLP)

A Multi-Layer Perceptron (MLP) adds one or more hidden layers between the input and output,
enabling it to learn non-linear decision boundaries. This is achieved by:
• Non-linear activation functions (sigmoid, tanh, ReLU) that introduce non-linearity at each
layer
• Backpropagation algorithm that efficiently computes gradients through all layers
• Universal approximation capability — any continuous function can be approximated given
sufficient hidden neurons
Example: A 3-layer MLP (input → hidden → output) with sigmoid activations can perfectly classify the
XOR problem, as shown in the previous section. Adding more layers and neurons allows the network to
model increasingly complex decision surfaces — curved boundaries, spirals, and multi-modal
distributions.

MLP Example: 3-2-1 Network for Binary Classification

Input: x = [x₁, x₂, x₃]

Hidden: h = σ(W₁x + b₁) where W₁ is 2×3 matrix
Output: ŷ = σ(W₂h + b₂) where W₂ is 1×2 matrix
 Total trainable parameters: 3×2 + 2 + 2×1 + 1 = 11
6. Feed Forward Networks & Deep Feed-Forward Networks
6.1 Feed Forward Neural Network
A Feed Forward Neural Network (FFNN) is the simplest type of artificial neural network where
information flows in one direction only — from the input layer, through the hidden layers, to the output
layer. There are no cycles or loops; no neuron's output feeds back as its own input. This distinguishes
it from Recurrent Neural Networks (RNNs).
Mathematical Forward Pass: For a network with L layers, the forward pass computes:
Layer 1: a⁽¹⁾ = f(W⁽¹⁾x + b⁽¹⁾)
Layer 2: a⁽²⁾ = f(W⁽²⁾a⁽¹⁾ + b⁽²⁾)
...
Layer L: ŷ = f(W⁽ᴸ⁾a⁽ᴸ⁻¹⁾ + b⁽ᴸ⁾)
Where W⁽ˡ⁾ is the weight matrix for layer l, b⁽ˡ⁾ is the bias vector, and f is the activation function applied
element-wise.
6.2 Deep Feed-Forward Networks
A Deep Feed-Forward Network (also called Deep Neural Network or DNN) is simply a feed-forward
network with many hidden layers — typically 3 or more. The 'depth' refers to the number of layers in
the network, which is the chain of transformations a signal undergoes from input to output.
The key advantage of depth is the ability to learn hierarchical representations. For example, in image
recognition:
• Layer 1 detects low-level features: edges, gradients, corners
• Layer 2 combines edges into textures and shapes
• Layer 3 combines shapes into object parts (e.g., a wheel, an eye)
• Layer 4 combines parts into full objects (e.g., a car, a face)
This hierarchy of abstractions means that a deep network with fewer neurons can often represent more
complex functions than a wide but shallow network, making depth a form of computational efficiency.

Why Depth Matters — Exponential Expressiveness

A shallow network needs exponentially many neurons to represent what a deep network
can with polynomially many neurons.

Example (Counting functions): With k layers and n neurons per layer, a deep ReLU network
can represent O(n^k) linear regions, while a shallow network with the same total neurons
represents only O(n) regions.

Key insight: Depth provides a multiplicative (not just additive) boost in representational
power.

7. Stochastic Gradient-Based Learning

7.1 The Learning Problem
Training a neural network is fundamentally an optimization problem: we want to find the weights W
and biases b that minimize a loss function L(W, b) which measures how far the network's predictions
are from the true labels. For a training set of m examples:
L(W) = (1/m) Σᵢ loss(ŷᵢ, yᵢ)
Common loss functions: Mean Squared Error (MSE) for regression — L = (1/m) Σ(ŷᵢ − yᵢ)²; Cross-
Entropy for classification — L = −(1/m) Σ yᵢ log(ŷᵢ)
7.2 Gradient Descent
Gradient Descent is the core optimization algorithm. The gradient ∇W L points in the direction of
steepest increase of the loss. By moving in the opposite direction (negative gradient), we descend
towards a minimum:
W ← W − η · ∇_W L
Where η (eta) is the learning rate. If η is too large, training diverges (overshooting); if too small, training
converges too slowly.
7.3 Variants of Gradient Descent
Variant Batch Size Key Characteristic
Batch GD All m samples Accurate gradient but very slow per
update; can't escape local minima
Stochastic GD 1 sample Fast updates, noisy gradient — acts as
(SGD) regularizer; escapes local minima
Mini-Batch SGD 32–512 samples Best of both: vectorized efficiency +
gradient noise. Most widely used in
practice
7.4 Advanced Optimizers
Pure SGD has limitations (noisy updates, sensitivity to learning rate). Advanced optimizers address
these:
• Momentum: Adds a velocity term v ← βv + ∇L, then W ← W − ηv. Accumulates gradient in
consistent directions, dampens oscillations.
• RMSProp: Adapts learning rate per parameter by dividing by the root mean square of recent
gradients. Good for non-stationary objectives.
• Adam (Adaptive Moment Estimation): Combines momentum and RMSProp. Maintains
moving averages of both gradients (m) and squared gradients (v). Default optimizer for most
deep learning tasks. Update rule: W ← W − η·m̂ /(√v̂ + ε)

8. Hidden Units & Activation Functions

Hidden units are the neurons in the hidden layers of a deep network. Their behavior is entirely
determined by their activation function — the non-linear transformation applied to the weighted sum
of inputs. The choice of activation function significantly affects training dynamics, convergence speed,
and model capacity.
8.1 Sigmoid (Logistic) Activation
Formula: σ(z) = 1 / (1 + e⁻ᶻ)
Output Range: [0, 1]
The sigmoid function squashes any real number into [0, 1], which makes it ideal for the output layer of
binary classification models (interpreting output as a probability). However, as a hidden unit
activation, it suffers from the vanishing gradient problem: for very large or very small inputs, the
gradient approaches zero, making learning slow in deep networks. Additionally, sigmoid outputs are not
zero-centered, causing gradient updates to always be either all positive or all negative, slowing
convergence.
8.2 Tanh (Hyperbolic Tangent)
Formula: tanh(z) = (eᶻ − e⁻ᶻ) / (eᶻ + e⁻ᶻ)
Output Range: [-1, 1]
Tanh is a zero-centered version of sigmoid — its outputs are symmetric around 0. This means
gradients from hidden units are balanced, leading to slightly faster convergence than sigmoid.
However, tanh still saturates at extreme values, causing vanishing gradients. It is commonly used as
the activation for hidden layers in RNNs.
8.3 ReLU — Rectified Linear Unit
Formula: ReLU(z) = max(0, z)
Output Range: [0, ∞)
ReLU is the most widely used activation function in modern deep learning. It is extremely simple to
compute, creates sparse activations (negative inputs become zero), and does not saturate for positive
values — so gradients flow freely during backpropagation. This largely solves the vanishing gradient
problem. However, ReLU can suffer from the "dying ReLU" problem: if a neuron's input is always
negative during training, it will always output zero and its gradient will always be zero — effectively
'dying' and never contributing to the network again.
8.4 Variants of ReLU
• Leaky ReLU: f(z) = max(0.01z, z). Allows a small, non-zero gradient for negative inputs (0.01z),
preventing neurons from dying.
• Parametric ReLU (PReLU): f(z) = max(αz, z) where α is a learned parameter.
• ELU (Exponential Linear Unit): f(z) = z if z>0, else α(eᶻ−1). Smooth, zero-centered, avoids
dead neurons.
• GELU (Gaussian Error Linear Unit): f(z) = z·Φ(z). Used in Transformers and BERT. Smooth
approximation of ReLU.
• Swish: f(z) = z·σ(z). Proposed by Google Brain. Often outperforms ReLU on deep networks.
8.5 Softmax (Output Layer)
Formula: softmax(zᵢ) = eᶻⁱ / Σⱼ eᶻʲ
Softmax converts a vector of raw scores (logits) into a probability distribution over multiple classes —
all outputs are non-negative and sum to 1. It is used exclusively in the output layer for multi-class
classification. Paired with cross-entropy loss, it enables the network to output calibrated class
probabilities.

Activation Function Quick Comparison

Sigmoid : Range [0,1] | Vanishing gradient | Output layers (binary)
Tanh : Range [-1,1] | Vanishing gradient | RNN hidden layers
ReLU : Range [0,∞) | Dying ReLU | Default for CNNs/FFNs
Leaky ReLU: Range (-∞,∞) | Avoids dying ReLU | When dying ReLU is a concern
 Softmax : Sums to 1 | N/A | Output layers (multi-class)
GELU : Smooth | Efficient | Transformers, BERT, GPT

9. Architecture Design
Designing the architecture of a neural network — deciding how many layers, how many neurons per
layer, what activation functions to use, and how layers are connected — is as much an art as a
science. There are guiding principles derived from empirical research and theoretical understanding.
9.1 Width vs. Depth
A network can be made more powerful by increasing its width (neurons per layer) or its depth (number
of layers). While both approaches increase capacity, depth is generally more efficient:
• Deep narrow networks generalize better and are more parameter-efficient
• Width mainly increases memorization, not abstraction
• Practical guideline: start deeper (3–5 layers) before going wider
9.2 Layer Sizing
Common practice is to use a funnel shape: wider early layers (closer to input) and progressively
narrower layers towards the output. This mirrors the intuition of going from a high-dimensional raw input
to low-dimensional learned representations. For example: 784 → 512 → 256 → 128 → 10 for MNIST.
9.3 Choosing the Output Layer Design
• Binary Classification: 1 neuron, sigmoid activation, binary cross-entropy loss
• Multi-Class Classification: K neurons (K = number of classes), softmax activation, categorical
cross-entropy loss
• Regression: 1 neuron (or multiple for multi-output), linear activation (no activation), MSE or
MAE loss
• Multi-Label Classification: K neurons with sigmoid (independent probabilities per class),
binary cross-entropy
9.4 Regularization Considerations
• Dropout: Randomly zeros neurons during training with probability p (typically 0.2–0.5). Acts as
an ensemble of subnetworks, preventing co-adaptation of neurons.
• L2 Regularization (Weight Decay): Adds ||W||² penalty to loss, shrinking weights towards zero,
preventing overfitting.
• Batch Normalization: Normalizes layer inputs to reduce internal covariate shift. Allows higher
learning rates and makes training more stable.
• Early Stopping: Monitor validation loss; stop training when it starts increasing even if training
loss continues to decrease.

10. Back-Propagation Algorithm

Backpropagation (short for "backward propagation of errors") is the algorithm used to compute the
gradient of the loss function with respect to every weight in the network. It is the foundation that makes
training deep networks feasible. Without backpropagation, computing gradients for millions of
parameters would be computationally intractable.
10.1 The Chain Rule — Foundation of Backprop
Backpropagation is essentially an efficient application of the chain rule of calculus. For a composition
of functions f(g(h(x))), the chain rule gives:
d/dx [f(g(h(x)))] = f'(g(h(x))) · g'(h(x)) · h'(x)
In a neural network, the output is a chain of matrix multiplications and non-linear transformations.
Backprop applies the chain rule layer by layer, starting from the output and working backwards to the
input.
10.2 Forward Pass
In the forward pass, input data flows from left to right through the network, computing the output. At
each layer l, we compute:
zˡ = Wˡaˡ⁻¹ + bˡ (pre-activation)
aˡ = f(zˡ) (post-activation)
At the output layer, we compute the loss L by comparing the network output with the true label.
10.3 Backward Pass — Gradient Computation
In the backward pass, we compute gradients of L with respect to all weights. Starting from the output:
δˡ = ∂L/∂zˡ (error signal at layer l, called 'delta')
The backpropagation equations (for sigmoid activation f):
Output layer delta: δᴸ = (aᴸ − y) ⊙ f'(zᴸ)
Hidden layer delta: δˡ = ((Wˡ⁺¹)ᵀ δˡ⁺¹) ⊙ f'(zˡ)
Weight gradient: ∂L/∂Wˡ = δˡ (aˡ⁻¹)ᵀ
Bias gradient: ∂L/∂bˡ = δˡ
Where ⊙ is element-wise multiplication.

BACKPROPAGATION OVERVIEW:

Forward: x → [L1] → a¹ → [L2] → a² → [L3] → ŷ → Loss

Backward: ∂L/∂W³ ← ∂L/∂W² ← ∂L/∂W¹

(Compute gradient layer by layer using chain rule)

Update: W ← W − η·∂L/∂W for each layer

10.4 Vanishing and Exploding Gradients

A key challenge in training deep networks with backprop is the vanishing gradient problem. When
gradients are repeatedly multiplied (via the chain rule) across many layers using sigmoid/tanh
activations, they can shrink exponentially towards zero — making the early layers of the network learn
extremely slowly or not at all.
The opposite can also occur: exploding gradients, where gradients grow exponentially, causing
numerical overflow and unstable training. Solutions include:
• ReLU activations: Gradient is 1 for positive inputs, preventing shrinkage
• Residual connections (skip connections): Add shortcuts that bypass layers, ensuring
gradient can flow unimpeded
 • Gradient clipping: Cap gradient magnitude at a threshold value (e.g., 1.0) to prevent explosion
• Careful weight initialization: He initialization for ReLU, Glorot/Xavier for tanh — ensures
appropriate initial gradient magnitudes
• Batch Normalization: Normalizes layer inputs, preventing extreme activation values

11. Deep Learning Frameworks and Libraries

Deep learning frameworks provide high-level abstractions that allow researchers and engineers to
build, train, and deploy neural networks without implementing matrix operations and gradient
computation from scratch. They handle GPU acceleration, automatic differentiation, and optimized
numerical routines under the hood.
11.1 TensorFlow and Keras
TensorFlow is an open-source deep learning framework developed by Google Brain, released in 2015.
It uses computational graphs — networks of nodes representing mathematical operations — to
represent and execute computations. This allows TensorFlow to optimize the computation graph, run it
on GPUs/TPUs efficiently, and support distributed training across multiple machines.
Keras is a high-level neural network API designed for fast experimentation. Originally a standalone
library, Keras is now fully integrated into TensorFlow as [Link]. It provides intuitive abstractions for
building models using the Sequential API (for simple stack-of-layers) or the Functional API (for complex
multi-input/multi-output architectures).

# TensorFlow/Keras Example: Building a Simple MLP

import tensorflow as tf
from tensorflow import keras

model = [Link]([
[Link](128, activation='relu', input_shape=(784,)),
[Link](0.2),
[Link](64, activation='relu'),
[Link](10, activation='softmax')
])

[Link](optimizer='adam',
loss='categorical_crossentropy',
metrics=['accuracy'])

[Link](x_train, y_train, epochs=10, batch_size=32, validation_split=0.1)

11.2 PyTorch
PyTorch is an open-source deep learning framework developed by Facebook AI Research (FAIR),
released in 2016. It has become the dominant framework in academic research, known for its dynamic
computation graph (also called eager execution) — unlike TensorFlow's original static graph
approach, PyTorch builds the computation graph on-the-fly as code executes. This makes debugging
intuitive using standard Python tools (pdb, print statements).
Key advantages of PyTorch include its Pythonic design, seamless NumPy integration, and the
autograd module which automatically computes gradients for any tensor operation. PyTorch is
particularly preferred for research due to its flexibility.
 # PyTorch Example: Building a Simple MLP
import torch
import [Link] as nn
import [Link] as optim

class MLP([Link]):
def __init__(self):
super().__init__()
self.fc1 = [Link](784, 128)
self.fc2 = [Link](128, 64)
self.fc3 = [Link](64, 10)
[Link] = [Link](0.2)

def forward(self, x):

x = [Link](self.fc1(x))
x = [Link](x)
x = [Link](self.fc2(x))
return [Link](self.fc3(x), dim=1)

model = MLP()
optimizer = [Link]([Link](), lr=0.001)
criterion = [Link]()

11.3 TensorFlow vs. PyTorch Comparison

Aspect TensorFlow / Keras PyTorch
Developer Google Brain Facebook AI Research (Meta)
Computation Graph Static (default) / Dynamic (eager) Dynamic (eager execution)
Debugging Can be complex with graph mode Pythonic, use pdb/print easily
Deployment TF Serving, TFLite, [Link] TorchServe, TorchScript, ONNX
Popularity Strong in industry/production Dominant in research & academia
Hardware GPU, TPU, multi-device GPU, XLA (limited TPU)
High-level API Keras (built-in) PyTorch Lightning (third-party)
Unit 1 Summary & Important Questions

Key Concepts at a Glance

Unit 1 — Core Takeaways
1. Deep Learning evolved through 3 waves; modern era began with AlexNet (2012)
2. Growth is driven by Big Data, GPU computing, and algorithmic improvements
3. ANN: Input Layer → Hidden Layers → Output Layer. Neuron: z = Wx + b, output = f(z)
4. XOR problem proved single-layer perceptrons can't solve non-linear problems
5. MLP adds hidden layers + non-linear activations to solve non-linear problems
6. Feed Forward Network: data flows in one direction only — no loops
7. Depth creates hierarchical feature representations — very efficient
8. Stochastic Gradient Descent (SGD) with mini-batches is the standard training approach
9. Activation functions: ReLU (default), Sigmoid (binary output), Softmax (multi-class output)
10. Backpropagation: uses chain rule to compute gradients from output backwards
11. Vanishing gradients: solved by ReLU, residual connections, batch normalization
12. TensorFlow/Keras: industry-focused. PyTorch: research-focused

Important Examination Questions

1. Define Deep Learning. Explain the historical evolution of neural networks through the three
waves of AI.
2. Why is Deep Learning growing? Discuss the role of big data, GPU computing, and algorithmic
advancements.
3. Explain the mathematical model of an artificial neuron with a neat diagram. Compare it with
a biological neuron.
4. Why can't a single-layer perceptron solve the XOR problem? Explain with a diagram and
show how a multi-layer network solves it.
5. Explain the architecture of a Multi-Layer Perceptron (MLP). Define each layer and its
function.
6. What is a Feed Forward Neural Network? Differentiate between shallow and deep networks.
Why is depth important?
7. Explain Stochastic Gradient Descent. Compare BGD, SGD, and Mini-Batch SGD with their
advantages and disadvantages.
8. Discuss different activation functions — Sigmoid, Tanh, ReLU, Leaky ReLU, Softmax —
with formulas and when to use each.
9. Explain the Backpropagation algorithm in detail with the forward pass and backward pass
steps. Derive the weight update rule.
10. What is the Vanishing Gradient Problem? How do ReLU activations, residual connections,
and batch normalization address it?
11. Compare TensorFlow/Keras and PyTorch with respect to computation graph, debugging,
deployment, and use cases.
12. Explain architecture design principles for deep neural networks — width vs depth, output
layer design, regularization strategies.
 UNIT 2
Convolutional Neural Networks (CNN)
Deep Learning (2344106) • [Link] CSE • 9 Contact Hours

Topics Covered in Unit 2

1. Introduction to CNNs and Applications in Computer Vision
2. CNN Basic Architecture — Convolution, Pooling, Fully Connected Layers
3. Activation Functions — Sigmoid, Tanh, ReLU, Softmax
4. Types of Pooling Layers — Max, Average, Global Pooling
5. Training CNN in TensorFlow
6. Popular CNN Architectures — VGG, GoogLeNet (Inception), ResNet
7. Dropout Regularization
8. Batch Normalization
9. Data Augmentation
1. Introduction to Convolutional Neural Networks
A Convolutional Neural Network (CNN) is a specialized deep learning architecture designed to
process grid-structured data — most notably images and videos. While a standard Multi-Layer
Perceptron (MLP) connects every neuron to every other neuron in adjacent layers (fully connected), a
CNN exploits the spatial structure of images by using local receptive fields, shared weights, and
spatial pooling. This makes CNNs dramatically more efficient and effective for visual data.
The concept was pioneered by Yann LeCun in 1998, who developed LeNet-5 to recognize handwritten
digits on bank cheques. For over a decade, CNNs were considered computationally too expensive for
large-scale use. The real breakthrough came in 2012 with AlexNet, which trained a deep CNN on
GPUs and won the ImageNet Large Scale Visual Recognition Challenge (ILSVRC) by a wide margin,
triggering the deep learning revolution in computer vision.
1.1 Why Not Use MLP for Images?
Consider a modest 224×224 colour image (224 × 224 × 3 = 150,528 pixels). Feeding this into an MLP's
first hidden layer with just 1,000 neurons would require 150 million weights in that single layer alone!
Such networks would:
• Require enormous memory and compute resources
• Overfit dramatically on typical training datasets
• Fail to exploit the spatial locality and translation invariance of image features
• Treat spatially close pixels identically to pixels far apart — losing structural information
CNNs solve all of these problems through three key ideas:
• Local Connectivity: Each neuron connects only to a small region of the input (the receptive
field), drastically reducing parameters.
• Weight Sharing: All neurons in a feature map share the same filter weights, detecting the same
feature everywhere in the image. This provides translation invariance — a cat detector works
regardless of where the cat is in the image.
• Pooling (Spatial Subsampling): Reduces spatial dimensions progressively, building
robustness to small shifts and distortions.
1.2 Applications of CNNs in Computer Vision
Application Domain Examples
Image Classification Identifying objects: cat, dog, car. ImageNet,
CIFAR-10
Object Detection Locating & classifying objects: YOLO, Faster
R-CNN
Semantic Segmentation Pixel-wise labeling: autonomous driving
scene understanding
Face Recognition FaceNet, DeepFace (Facebook), FaceID
Medical Imaging Tumor detection, retinal disease diagnosis,
X-ray analysis
OCR & Document Analysis Reading text from images, cheque
processing
Video Analysis Action recognition, surveillance, sports
analytics
 Generative Models GANs for image synthesis, style transfer

2. CNN Basic Architecture

A CNN is composed of a sequence of layers, each transforming a 3D volume of activations into
another. The standard CNN architecture consists of alternating Convolutional Layers and Pooling
Layers, followed by one or more Fully Connected (Dense) Layers for the final classification or
regression output.

CNN ARCHITECTURE OVERVIEW:

Input Image Conv Layer 1 Pool Layer 1 Conv Layer 2 Pool Layer 2 FC Layers
Output
[224x224x3] ──► [222x222x32] ──► [111x111x32] ──► [109x109x64] ──►
[54x54x64] ──► [512] ──► [10]
(3x3 filters) (2x2 Max Pool) (3x3 filters) (2x2 Max Pool) (Dense) (Softmax)

Features become: edges → textures → shapes → object parts → objects

2.1 Convolutional Layer

The convolutional layer is the core building block of a CNN. It applies a set of learnable filters
(kernels) to the input volume, producing a set of feature maps. Each filter is a small matrix (e.g., 3×3
or 5×5) that slides across the entire input, computing a dot product at each position. This operation
detects specific features — edges, corners, textures — wherever they appear in the input.
2.1.1 Convolution Operation
Mathematical definition: For a 2D input X and filter W of size k×k, the output feature map S at position
(i,j) is:
S(i, j) = Σₘ Σₙ X(i+m, j+n) · W(m, n) + b
For a colour image with C input channels and F filters, the output has F feature maps — one per filter.

3×3 Filter applied to 5×5 Input (padding=0, stride=1):

Input: Filter: Output (3×3):

1 2 3 0 1 1 0 -1 (1×1 + 2×0 + 3×-1) + ...
0 1 2 3 0 1 0 -1
1 0 1 2 1 1 0 -1 This filter detects vertical
2 1 0 1 2 edges (like a Sobel filter).
0 1 2 0 1

2.1.2 Key Hyperparameters

• Filter Size (k×k): Determines the local region each neuron looks at. Common sizes: 1×1
(channel mixing), 3×3 (most popular), 5×5, 7×7. Larger filters capture more context but have
more parameters.
• Number of Filters (F): Each filter learns to detect a different feature. More filters → richer
representations but more computation. Typical: 32, 64, 128, 256, 512.
• Stride (S): The step size of the sliding filter. Stride=1: filter moves one pixel at a time (dense
sampling). Stride=2: filter skips every other pixel (downsampling by 2, reduces computation).
 • Padding (P): Zeros added around the border of the input. 'Valid' padding (P=0): output is
smaller than input. 'Same' padding: output size = input size. Preserves border information.
Output Size Formula: For input W×W, filter k×k, padding P, stride S:
Output Size = ⌊(W − k + 2P) / S⌋ + 1
Example: 28×28 input, 3×3 filter, padding=1, stride=1 → ⌊(28 − 3 + 2) / 1 ⌋ + 1 = 28. Output is same
size!
2.1.3 Parameter Sharing
A critical efficiency gain: instead of each output position having its own unique weights, all positions in
a feature map share the same filter weights. A 3×3×3 filter (3×3 spatial, 3 input channels) has only
27 + 1 = 28 parameters regardless of the input image size. Compare this to a fully connected neuron
on a 224×224×3 input which would need 150,528 parameters. Parameter sharing exploits the
stationarity of statistics in images — an edge detector that works in the top-left also works in the
bottom-right.
2.2 Pooling Layer
The pooling layer reduces the spatial dimensions (width and height) of feature maps, retaining the
most important information while discarding fine-grained positional detail. This achieves two goals: (1)
dimensionality reduction, which reduces computation in subsequent layers, and (2) translation
invariance, making the network robust to small shifts of features in the input.
2.2.1 Max Pooling
Max Pooling takes the maximum value in each pooling window. For a 2×2 window with stride 2, it
selects the strongest activation from each 2×2 region. This is the most commonly used pooling
operation.

Max Pooling (2×2, stride=2):

Input (4×4): Output (2×2):

1 3 2 4 3 4
5 6 1 2 ──► 6 4
7 8 5 3 8 5
2 4 3 5 4 5

Takes MAX from each 2×2 region

2.2.2 Average Pooling

Average Pooling computes the arithmetic mean of values in the pooling window. It retains more
information than max pooling (doesn't throw away non-maximum activations) but results in blurrier
feature maps. Used in some architectures (e.g., GoogLeNet) and for global pooling.
2.2.3 Global Average Pooling (GAP)
Global Average Pooling averages all values in each entire feature map into a single number. For a
feature map of size H×W×C, GAP produces a vector of length C. It serves as a parameter-free
alternative to fully connected layers at the end of a network, drastically reducing model size and
overfitting. Used in ResNet, GoogLeNet, and most modern architectures.
2.3 Fully Connected Layer
After the convolutional and pooling layers have extracted spatial features, the feature maps are
flattened into a 1D vector and fed into one or more fully connected (dense) layers. These layers
perform high-level reasoning by combining the features into final class scores. The last FC layer
typically uses softmax activation to produce class probabilities. Modern architectures minimize FC
layers (they account for most parameters) — some use Global Average Pooling instead.
3. Activation Functions in CNNs
Activation functions introduce non-linearity into the network, enabling CNNs to learn complex, non-
linear mappings from inputs to outputs. Without non-linear activations, stacking multiple layers would be
equivalent to a single linear transformation — no matter how deep the network. In CNNs, activation
functions are applied element-wise after each convolutional layer.
3.1 Sigmoid
Formula: σ(z) = 1 / (1 + e⁻ᶻ)
Range: [0, 1] | Derivative: σ'(z) = σ(z)(1 − σ(z))
Sigmoid was one of the first activation functions used in neural networks. It maps any input to a
smooth, bounded output between 0 and 1, making it intuitively appealing as a 'probability-like'
activation. However, it suffers from several problems in deep CNNs: the gradient at saturated regions (z
<< 0 or z >> 0) approaches zero, causing vanishing gradients in deep networks. Additionally, sigmoid
outputs are always positive (not zero-centered), leading to zig-zagging gradients during optimization. It
is now primarily used in the output layer of binary classification networks.
3.2 Tanh
Formula: tanh(z) = (eᶻ − e⁻ᶻ) / (eᶻ + e⁻ᶻ)
Range: [−1, 1] | Derivative: tanh'(z) = 1 − tanh²(z)
Tanh is a zero-centered activation — its outputs are symmetric around 0, which makes optimization
somewhat easier than sigmoid (gradients from different examples can cancel, allowing more balanced
updates). It still saturates at extremes, causing vanishing gradients, but is generally preferred over
sigmoid for hidden layers when a bounded activation is needed. It is commonly used in RNN and
LSTM hidden layers (see Unit 3) but rarely in CNNs.
3.3 ReLU — Rectified Linear Unit
Formula: ReLU(z) = max(0, z)
Range: [0, ∞) | Derivative: 1 if z > 0, else 0
ReLU is the default activation function in CNNs. Its advantages are compelling: it is computationally
trivial (a single comparison), it does not saturate for positive inputs so gradients flow freely, and it
promotes sparse activations — typically about 50% of neurons are inactive (output 0) for any given
input, creating efficient representations. ReLU accelerated training of very deep networks dramatically
compared to sigmoid/tanh.
The main drawback is the "dying ReLU" problem: neurons with consistently negative pre-activations
will always output 0 and receive no gradient — they become permanently inactive. Solutions include
careful weight initialization, using Adam optimizer, or switching to Leaky ReLU.
3.4 Softmax Layer
Formula: softmax(zᵢ) = eᶻⁱ / Σⱼ eᶻʲ
Output: Probability distribution: all outputs ∈ (0,1), sum = 1
Softmax is used exclusively in the final output layer for multi-class classification. It converts a
vector of raw scores (logits) from the last fully connected layer into class probabilities. The class with
the highest probability is the predicted class. Softmax is numerically stable when combined with the
cross-entropy loss function — together they form the standard classification objective. For a CNN
classifying among K classes, the output layer has K neurons with softmax activation.
 Activation Function Summary for CNNs
Convolutional layers: ReLU (standard), Leaky ReLU (if dying ReLU occurs)
Binary output: Sigmoid (1 output neuron)
Multi-class output: Softmax (K output neurons)
Regression output: Linear (no activation)
Avoid in CNN hidden: Sigmoid and Tanh (vanishing gradient in deep nets)

4. Types of Pooling Layers

Pooling layers are a crucial component of CNNs that progressively reduce the spatial size of the
representation, reducing the number of parameters and computation in the network, thereby helping to
control overfitting. Pooling also makes the representation approximately invariant to small
translations — a critical property for object recognition, where we want to detect an object regardless
of its exact pixel location.
4.1 Max Pooling
Max Pooling selects the maximum activation within each pooling window. This is motivated by the
idea that the presence of a feature is more important than its exact location. If a particular feature (e.g.,
a vertical edge) is detected anywhere within the receptive field, max pooling will propagate that signal
forward, regardless of where precisely within the window it occurred.
Properties:
• Window size: typically 2×2 with stride 2 (reduces spatial dimensions by half)
• Retains the strongest activation — most discriminative signal
• Introduces translation invariance within the pooling window
• Gradients during backprop flow only through the maximum element
• Most widely used pooling type in AlexNet, VGG, and similar architectures

Max Pooling in detail (2×2 pooling, stride 2):

Feature map section: After Max Pool (2×2):

┌───┬───┐ ┌──────────┐
│ 2 │ 8 │ max=8 │ 8 │
├───┼───┤ ────► └──────────┘
│4│6│
└───┴───┘

Input 4×4 → Output 2×2 (spatial reduction by 2×)

4.2 Average Pooling

Average Pooling computes the arithmetic mean of all activations within the pooling window. Unlike
max pooling, it considers all values rather than just the maximum. This produces smoother, more
diffuse representations that can sometimes help with fine-grained visual recognition tasks where global
context matters.
Properties:
• Retains information from all positions (not just the dominant activation)
 • Produces smoother feature maps — less discriminative than max pooling for object detection
• Gradients flow back to all elements in the pooling window (not just the max element)
• Less commonly used for spatial pooling; more common as global pooling at network end
• Used in GoogLeNet's Inception modules and some ResNet variants

Average Pooling (2×2, stride 2):

┌───┬───┐ ┌──────────┐
│ 2 │ 8 │ avg=(2+8+4+6)/4 = 5 │ 5 │
├───┼───┤ ────► └──────────┘
│4│6│
└───┴───┘

4.3 Global Average Pooling (GAP)

Global Average Pooling averages all spatial positions in each feature map into a single value. Given
a tensor of shape H×W×C, it produces a vector of shape 1×1×C = C. It completely removes spatial
information, retaining only 'what' was detected, not 'where'. GAP was introduced by Lin et al. (2013) as
a structural regularizer to prevent overfitting — it has no learnable parameters.
Key advantages:
• Eliminates large fully connected layers — massive parameter reduction
• More interpretable: each channel directly corresponds to a class confidence
• Input-size agnostic: can process images of any size after convolutional layers
• Reduces overfitting by removing the dense layer bottleneck
• Standard in modern architectures: ResNet, GoogLeNet, MobileNet

Global Average Pooling:

Input: feature map of shape [7 × 7 × 512]

→ Average all 7×7 = 49 values per channel
→ Output: vector of shape [512]

Then: Dense(512 → 1000, softmax) for ImageNet classes

Pooling Type Key Characteristic Typical Use

Max Pooling Takes maximum in window Spatial feature extraction in
AlexNet, VGG
Average Pooling Takes mean in window GoogLeNet, fine-grained
recognition
Global Avg Pool Mean of entire feature map Replaces FC layers in ResNet,
MobileNet
5. Training a CNN in TensorFlow
TensorFlow (with the Keras API) provides a clean, intuitive pipeline for building and training CNNs. The
process follows a standard sequence: define the model architecture, compile it with loss and optimizer,
load and preprocess data, then train and evaluate.
5.1 Building a CNN with Keras
import tensorflow as tf
from tensorflow import keras
from [Link] import layers

# Define CNN architecture

model = [Link]([
# Block 1: Convolution + Activation + Pooling
layers.Conv2D(32, (3,3), activation='relu', padding='same',
input_shape=(32, 32, 3)),
layers.MaxPooling2D((2,2)),

# Block 2
layers.Conv2D(64, (3,3), activation='relu', padding='same'),
layers.MaxPooling2D((2,2)),

# Block 3
layers.Conv2D(128, (3,3), activation='relu', padding='same'),
layers.MaxPooling2D((2,2)),

# Classifier Head
layers.GlobalAveragePooling2D(), # or Flatten()
[Link](256, activation='relu'),
[Link](0.5),
[Link](10, activation='softmax') # 10 classes
])

[Link]() # Print architecture

5.2 Compiling the Model

[Link](
optimizer=[Link](learning_rate=0.001),
loss='sparse_categorical_crossentropy', # use 'categorical_crossentropy'
# if labels are one-hot encoded
metrics=['accuracy']
)

5.3 Data Loading and Preprocessing

from [Link] import cifar10
from [Link] import ImageDataGenerator
 # Load CIFAR-10 dataset
(x_train, y_train), (x_test, y_test) = cifar10.load_data()

# Normalize pixel values to [0, 1]

x_train = x_train.astype('float32') / 255.0
x_test = x_test.astype('float32') / 255.0

# Data augmentation pipeline

datagen = ImageDataGenerator(
rotation_range=15,
width_shift_range=0.1,
height_shift_range=0.1,
horizontal_flip=True,
)
[Link](x_train)

5.4 Training
# Callbacks
callbacks = [
[Link](patience=5, restore_best_weights=True),
[Link](factor=0.5, patience=3),
[Link]('best_model.h5', save_best_only=True),
]

# Train with augmented data

history = [Link](
[Link](x_train, y_train, batch_size=64),
epochs=50,
validation_data=(x_test, y_test),
callbacks=callbacks
)

# Evaluate
test_loss, test_acc = [Link](x_test, y_test)
print(f'Test accuracy: {test_acc:.4f}')

5.5 Training Pipeline Overview

TRAINING LOOP (each epoch):

─────────────────────────────────────────────────────────
────
for batch in training_data:
1. FORWARD PASS: predictions = model(batch_x)
2. COMPUTE LOSS: loss = cross_entropy(predictions, batch_y)
 3. BACKWARD PASS: gradients = [Link](loss, [Link])
4. UPDATE: optimizer.apply_gradients(gradients)

─────────────────────────────────────────────────────────
────
After each epoch: validate on val_set, log metrics, check callbacks

6. Popular CNN Architectures

As CNNs became the dominant approach in computer vision, researchers designed increasingly
sophisticated architectures to improve accuracy and efficiency. Each landmark architecture introduced
key innovations that influenced all subsequent work.
6.1 LeNet-5 (1998)
Authors: Yann LeCun, Léon Bottou, Yoshua Bengio, Patrick Haffner
Task: Handwritten digit recognition (MNIST)
LeNet-5 was the first successful CNN — a 7-layer architecture consisting of two convolutional layers
with average pooling, followed by three fully connected layers. It used tanh and sigmoid activations
and was trained with backpropagation. Though small by today's standards (60K parameters), LeNet-5
demonstrated that CNNs could outperform all other methods on digit recognition and was deployed
commercially for cheque reading at US banks.

LeNet-5 Architecture:
Input(32×32×1) → Conv(5×5,6) → AvgPool(2×2) → Conv(5×5,16)
→ AvgPool(2×2) → Flatten → FC(120) → FC(84) → FC(10,softmax)
Parameters: ~60K

6.2 AlexNet (2012)

Authors: Alex Krizhevsky, Ilya Sutskever, Geoffrey Hinton
Achievement: Won ILSVRC 2012 with 15.3% top-5 error (runner-up: 26.2%)
AlexNet was the watershed moment that triggered the deep learning revolution. It was the first to train a
deep CNN on GPUs (two GTX 580s), enabling a network 5× deeper than previous winners. Key
innovations that AlexNet introduced to the field:
• ReLU activations: Replaced tanh/sigmoid — 6× faster convergence in experiments
• Dropout: 0.5 dropout in FC layers to prevent overfitting on the 1.2M image dataset
• Data augmentation: Random cropping, horizontal flipping, color jittering
• Local Response Normalization (LRN): Lateral inhibition (later replaced by batch norm)
• GPU training: Split across 2 GPUs — first GPU parallel training strategy
• Overlapping pooling: 3×3 max pooling with stride 2 (windows overlap)

AlexNet Architecture:
Input(224×224×3)
→ Conv(11×11,96,stride=4) → LRN → MaxPool(3×3,stride=2)
→ Conv(5×5,256,pad=2) → LRN → MaxPool(3×3,stride=2)
 → Conv(3×3,384,pad=1) → Conv(3×3,384,pad=1) → Conv(3×3,256,pad=1)
→ MaxPool(3×3,stride=2) → Flatten
→ FC(4096,ReLU,Dropout) → FC(4096,ReLU,Dropout) → FC(1000,softmax)
Parameters: ~60M

6.3 VGGNet (2014)

Authors: Karen Simonyan, Andrew Zisserman (Oxford VGG group)
Achievement: 1st runner-up ILSVRC 2014 — pioneered depth with small filters
VGGNet's central insight was that using only 3×3 convolutions (the smallest possible kernel that
captures both horizontal/vertical and diagonal features) stacked deeply could match and exceed the
performance of architectures with larger filters. Two consecutive 3×3 convolutions have the same
receptive field as a single 5×5 convolution, but with fewer parameters (2×3×3=18 vs 5×5=25) and an
extra non-linearity. VGGNet demonstrated that depth is the key factor in network quality.

VGG-16 Architecture (16 weight layers):

Input(224×224×3)
Block 1: [Conv(3×3,64)] × 2 → MaxPool(2×2)
Block 2: [Conv(3×3,128)] × 2 → MaxPool(2×2)
Block 3: [Conv(3×3,256)] × 3 → MaxPool(2×2)
Block 4: [Conv(3×3,512)] × 3 → MaxPool(2×2)
Block 5: [Conv(3×3,512)] × 3 → MaxPool(2×2)
→ FC(4096) → FC(4096) → FC(1000,softmax)
Parameters: ~138M Top-5 Error: 7.3%

VGGNet's uniform architecture — all 3×3 convolutions, doubling feature maps after each pooling —
made it very easy to understand and adapt. It remains a popular baseline and feature extractor for
transfer learning. However, its 138M parameters (largely in the FC layers) make it memory-intensive.
VGG-16 and VGG-19 are the two most used variants.
6.4 GoogLeNet / Inception (2014)
Authors: Christian Szegedy et al. (Google)
Achievement: Winner ILSVRC 2014 — 6.67% top-5 error with 12× fewer parameters than AlexNet
GoogLeNet's key insight was that the optimal local network structure is uncertain — should we use a
1×1, 3×3, or 5×5 convolution at each layer? The Inception module answers: use them all in parallel
and let the network learn which representations are most useful. The outputs are then concatenated
along the channel dimension.

Inception Module:

Input ──┬──► 1×1 Conv ─────────────────────────────┐

├──► 1×1 Conv ──► 3×3 Conv ─────────────── ┤
├──► 1×1 Conv ──► 5×5 Conv ─────────────── ┤
└──► MaxPool(3×3) ─► 1×1 Conv ────────────► Concatenate

1×1 Convolutions: bottleneck to reduce channel dimensions

before expensive 3×3 and 5×5 operations
The 1×1 convolution bottleneck is a key innovation: applying 1×1 filters reduces the channel depth
(e.g., from 256 to 64 channels) before applying the expensive 3×3 or 5×5 convolutions. This is called
dimensionality reduction and dramatically reduces computation. GoogLeNet uses 22 layers and only
4M parameters — vs AlexNet's 60M — showing that wider, more parallel architectures are more
efficient than simply stacking layers.
6.5 ResNet — Residual Networks (2015)
Authors: Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun (Microsoft Research)
Achievement: Won ILSVRC 2015 with 3.57% top-5 error — surpassed human-level performance
(~5%)
The central problem ResNet addressed: as networks become very deep (>20 layers), accuracy
degrades — not due to overfitting, but because very deep networks are harder to optimize. The
counterintuitive observation was that a 56-layer network performed worse than a 20-layer network on
both training and test data. This degradation problem is distinct from overfitting.
The Residual Block — Skip Connections: ResNet introduces a shortcut connection (also called
skip or residual connection) that bypasses one or more layers and adds the input directly to the output:
y = F(x, {Wᵢ}) + x
Where F(x, {Wᵢ}) is the residual mapping learned by the stacked layers. Instead of learning the full
desired mapping H(x), the layers learn the residual F(x) = H(x) − x. If the optimal mapping is close to
the identity (i.e., the layer doesn't need to change the representation), it's easier to push F(x) → 0 than
to learn H(x) = x directly.

Residual Block:

x ────────────────────────────────────┐
│ │
└──► Conv(3×3) → BN → ReLU │
→ Conv(3×3) → BN ──►(+)──► ReLU ◄─┘
▲
shortcut
(identity or 1×1 conv)

Output: F(x) + x (residual + shortcut)

The shortcut connection has two important effects: (1) Gradients can flow directly through the shortcut
during backpropagation, bypassing many layers and preventing vanishing gradients. (2) The network
can easily learn identity mappings — if the best action is to do nothing, it simply sets F(x) = 0. This
enabled training of extremely deep networks: ResNet-50, ResNet-101, ResNet-152, and even
ResNet-1000+ layer networks.

Popular ResNet Variants:

ResNet-18: 18 layers, 11M params, fast, good for small datasets
ResNet-34: 34 layers, 21M params
ResNet-50: 50 layers, 25M params, uses bottleneck blocks (1×1, 3×3, 1×1)
ResNet-101: 101 layers, 44M params
ResNet-152: 152 layers, 60M params, deepest common variant
 Architecture Year Depth Parameters ImageNet Top-5 Error
LeNet-5 1998 7 ~60K N/A (MNIST)
AlexNet 2012 8 ~60M 15.3%
VGG-16 2014 16 ~138M 7.3%
GoogLeNet 2014 22 ~4M 6.67%
ResNet-50 2015 50 ~25M 5.25%
ResNet-152 2015 152 ~60M 3.57%

7. Dropout Regularization
Dropout, introduced by Srivastava et al. in 2014, is a powerful regularization technique for neural
networks. During training, it randomly 'drops out' (sets to zero) a fraction p of neurons in each layer
on every forward pass. The remaining neurons are scaled up by 1/(1-p) to preserve the expected sum.
During inference, all neurons are active (no dropout), but their weights are scaled by (1-p) — this is
equivalent to averaging over the ensemble of all thinned networks.
7.1 Motivation — Why Dropout Works
In a standard fully connected layer, neurons can develop complex co-adaptations — one neuron's
activation becomes highly dependent on specific other neurons. This fragile co-dependence is a form of
overfitting. Dropout prevents this by ensuring that no neuron can rely on specific other neurons always
being present. Each neuron must learn to be useful independently and in combination with random
subsets of other neurons.
Conceptually, dropout is equivalent to training an ensemble of 2ⁿ different sub-networks (where n is
the number of neurons), all sharing weights. At test time, the full network with scaled weights
approximates the average prediction of the entire ensemble.
7.2 Dropout in Practice
• Placement: Applied after fully connected layers (and sometimes after convolutional layers).
Most commonly used in the FC head of a CNN.
• Dropout rate p: Fraction of neurons dropped. Typically 0.5 for FC layers, 0.2–0.3 for
convolutional layers.
• Training vs Inference: Dropout is active during training. During evaluation/inference, it must be
disabled (Keras handles this automatically via [Link] vs [Link]).
• Spatial Dropout: For convolutional layers, entire feature maps (channels) are dropped rather
than individual pixels — called SpatialDropout2D in Keras.

Dropout in Keras:

model = [Link]([
layers.Conv2D(256, (3,3), activation='relu'),
layers.SpatialDropout2D(0.2), # drops entire feature maps
[Link](),
[Link](512, activation='relu'),
[Link](0.5), # drops individual neurons
[Link](10, activation='softmax')
 ])
# During [Link](): dropout is active
# During [Link](): dropout is disabled automatically

7.3 Effect of Dropout

• Reduces overfitting — validation loss and training loss diverge less
• Forces the network to learn more robust, distributed features
• Increases training time (because weight updates are noisier)
• Can be combined with other regularization methods (weight decay, batch norm)
• High dropout (p=0.5) on large FC layers is a common recipe for CNNs on small datasets

8. Batch Normalization
Batch Normalization (BN), introduced by Ioffe and Szegedy in 2015, is one of the most impactful
techniques in deep learning. It normalizes the inputs to each layer across the current mini-batch,
reducing internal covariate shift — the phenomenon where the distribution of each layer's inputs
changes during training as the parameters of the previous layers change. This instability forces the
network to use very small learning rates and careful initialization to avoid divergence.
8.1 How Batch Normalization Works
For a mini-batch B = {x₁, x₂, ..., xₘ}, Batch Normalization normalizes each feature dimension:
Step 1: Compute mini-batch mean: μ_B = (1/m) Σ xᵢ
Step 2: Compute mini-batch var: σ²_B = (1/m) Σ (xᵢ − μ_B)²
Step 3: Normalize: x̂ ᵢ = (xᵢ − μ_B) / √(σ²_B + ε)
Step 4: Scale and shift: yᵢ = γ x̂ ᵢ + β
Where γ (scale) and β (shift) are learnable parameters. This means the network can restore the
original distribution if needed — BN never forces the activations to have zero mean and unit variance
permanently. ε is a small constant (e.g., 1e-5) for numerical stability.
8.2 Benefits of Batch Normalization
• Higher learning rates: BN smooths the loss landscape, allowing larger learning rates without
diverging. Training is typically 3–5× faster.
• Reduced sensitivity to initialization: Networks with BN converge well even with somewhat
careless weight initialization.
• Implicit regularization: The noise introduced by using mini-batch statistics (rather than full-
batch statistics) acts as a regularizer. BN often makes Dropout unnecessary.
• Allows deeper networks: By stabilizing the gradient signal through normalization, BN enables
stable training of very deep networks.
• Reduces covariate shift: Each layer receives inputs from a more stable distribution, making
individual layer learning more predictable.
8.3 Placement in CNN
Standard practice: place BN after the convolution but before the activation function:

Standard order: Conv2D → BatchNorm → ReLU

 # In Keras:
x = layers.Conv2D(64, (3,3), use_bias=False)(x) # bias=False: BN has its own β
x = [Link]()(x)
x = [Link]('relu')(x)

# ResNet blocks use this exact pattern

8.4 Training vs Inference Behavior

During training: BN normalizes using the current mini-batch's mean and variance. During inference:
uses running averages of mean and variance accumulated during training (since batch size might be
1 at inference and mini-batch statistics would be unreliable). This is handled automatically by Keras and
PyTorch.

9. Data Augmentation
Data Augmentation is the practice of artificially expanding the training dataset by applying label-
preserving transformations to existing training images. Since deep CNNs have millions of parameters
and require large datasets to generalize, augmentation is a critical technique — especially when
labeled data is scarce. Augmented samples expose the network to a wider variety of inputs, making it
more robust to variations in the test data.
Crucially, augmentation is applied only to the training set, never to the validation or test sets. The
transformations are applied on-the-fly during training (in real-time, on each epoch), meaning the model
sees a slightly different version of each image every epoch — effectively seeing more unique samples
than the original dataset contains.
9.1 Geometric Transformations
• Random Horizontal Flip: Mirror the image left-right. Valid for most object recognition tasks
(cats look like cats flipped). Not valid for tasks where orientation matters (e.g., reading text,
recognizing asymmetric logos).
• Random Crop / Random Resized Crop: Randomly crop a region of the image and resize to
the target size. Exposes the network to different scales and positions of the object. Used in
AlexNet (cropping 224×224 patches from 256×256 images).
• Rotation: Rotate the image by a small random angle (e.g., ±15°). Helps with rotated object
detection.
• Translation (Shift): Shift the image horizontally or vertically by a few pixels. Builds positional
invariance.
• Zoom: Randomly zoom in/out. Helps with scale invariance.
• Shear: Apply a shear transformation — useful for document and handwriting recognition.
9.2 Photometric / Color Transformations
• Brightness Jitter: Randomly adjust image brightness. Simulates different lighting conditions.
• Contrast Jitter: Vary the contrast. Helps with images from different cameras or under different
lighting.
• Saturation Jitter: Vary color saturation. Makes the network less sensitive to color intensity.
• Hue Jitter: Shift the color hue by a small amount. Helps with color variation in real-world
settings.
• Gaussian Noise / Blur: Add random noise or apply Gaussian blur. Builds robustness to image
degradation.
 • Grayscale Conversion: Randomly convert to grayscale with probability p. Makes the model
less reliant on color.
9.3 Advanced Augmentation Techniques
• Cutout: Randomly mask out square regions of the input image during training. Forces the
network to rely on multiple features rather than one dominant region.
• Mixup: Creates convex combinations of pairs of training examples and their labels. If x₁ and x₂
are images, the augmented image is λx₁ + (1−λ)x₂ with label λy₁ + (1−λ)y₂. Improves
calibration and robustness.
• CutMix: Combines Cutout and Mixup — cuts and pastes patches from one image to another,
mixing labels proportionally. State-of-the-art augmentation for ImageNet classification.
• RandAugment: Automatically selects from a set of augmentation operations and magnitudes
— eliminates manual augmentation policy design. Used in EfficientNet training.
• AutoAugment: Google's method that uses reinforcement learning to discover optimal
augmentation policies for specific datasets.
9.4 Data Augmentation in TensorFlow/Keras
from [Link] import ImageDataGenerator

# Classic augmentation pipeline

train_datagen = ImageDataGenerator(
rescale=1./255,
rotation_range=20,
width_shift_range=0.2,
height_shift_range=0.2,
shear_range=0.2,
zoom_range=0.2,
horizontal_flip=True,
fill_mode='nearest'
)

# Modern Keras layers approach (applied inside the model)

data_augmentation = [Link]([
[Link]('horizontal'),
[Link](0.1),
[Link](0.1),
[Link](0.1),
])

# Include augmentation in model (runs on GPU)

inputs = [Link](shape=(224, 224, 3))
x = data_augmentation(inputs) # only active during training
# ... rest of CNN ...
Unit 2 Summary & Important Questions

Key Concepts at a Glance

Unit 2 — Core Takeaways
1. CNN exploits local connectivity, weight sharing, and pooling — far more efficient than
MLP for images
2. Conv Layer: filter slides over input computing dot products → feature maps
3. Output size: ⌊(W − k + 2P) / S⌋ + 1
4. ReLU is the default activation in CNN hidden layers; Softmax for multi-class output
5. Max Pooling: takes max → sharp features. Average Pooling: takes mean → smoother.
GAP: removes all spatial dims
6. TF/Keras pipeline: define → compile(optimizer, loss) → fit(data, epochs) → evaluate
7. VGGNet: uniform 3×3 convolutions stacked deep. Simple but 138M parameters
8. GoogLeNet: Inception module — parallel 1×1, 3×3, 5×5 convolutions concatenated
9. ResNet: skip connections (residual blocks) — F(x)+x. Solved degradation. Enabled 152+
layer networks
10. Dropout: randomly zeros neurons (p=0.5) → prevents co-adaptation → implicit
ensemble
11. Batch Normalization: normalize each layer's input → faster training, higher LR, less
overfitting
12. Data Augmentation: flips, crops, rotations, color jitter — expand training set artificially

Important Examination Questions

1. What is a Convolutional Neural Network? Explain why CNNs are preferred over MLPs for
image processing. Describe the three key ideas that make CNNs efficient.
2. Explain the CNN basic architecture with a neat block diagram. Describe the role of the
convolutional layer, pooling layer, and fully connected layer.
3. Derive the output size formula for a convolutional layer and apply it to an example. Explain
the effect of padding and stride.
4. Compare Sigmoid, Tanh, ReLU, and Softmax activation functions with formulas, output
ranges, advantages and disadvantages, and when each is used in CNNs.
5. Explain Max Pooling, Average Pooling, and Global Average Pooling with diagrams. What is
the effect of pooling on feature maps and why is it important?
6. Explain the Inception module in GoogLeNet. What problem does it solve? What is the role of
1×1 convolutions?
7. What is the degradation problem in very deep networks? Explain how ResNet's
residual/skip connections solve it. Draw a residual block.
8. What is Dropout? Explain its mechanism, why it prevents overfitting, and how to apply it in
TensorFlow/Keras.
9. Explain Batch Normalization — the full algorithm including scale and shift parameters. What is
internal covariate shift and how does BN address it?
10. What is Data Augmentation? List any six augmentation techniques and explain how each
helps with generalization. Why is augmentation applied only on training data?
11. Write TensorFlow/Keras code to build, compile, and train a CNN for CIFAR-10 classification
with dropout and data augmentation.
 UNIT 3
Recurrent Neural Network (RNN)
Deep Learning (2344106) • [Link] CSE • 9 Contact Hours

Topics Covered in Unit 3

1. Introduction to RNNs and their Applications in Sequential Data Analysis
2. Back Propagation Through Time (BPTT)
3. Vanishing Gradient Problem in RNNs
4. Gradient Clipping
5. Long Short-Term Memory (LSTM) Networks
6. Gated Recurrent Units (GRU)
7. Bidirectional LSTMs
8. Bidirectional RNNs
1. Introduction to Recurrent Neural Networks (RNNs)
Recurrent Neural Networks (RNNs) are a class of artificial neural networks specifically designed to
handle sequential data — data where order and temporal context matter. Unlike Feed Forward Neural
Networks (FFNNs) that process each input independently, RNNs maintain a hidden state (memory)
that carries information from previous time steps, allowing the network to capture patterns across
sequences of arbitrary length.

1.1 Why Standard Neural Networks Fail for Sequential Data

Standard feed-forward networks struggle with sequential data for three fundamental reasons:
7. No memory: Each input is processed in isolation — no information is retained from previous
inputs.
8. Fixed input size: They require inputs of a fixed dimension; sequences can vary in length.
9. No temporal context: The order of inputs does not influence computation — but in language
and time series, order is everything.

For example, to translate 'The cat sat on the mat' — each word depends on the context of all previous
words. An FFNN has no mechanism to capture this dependency.

1.2 The RNN Architecture — Recurrence Relation

The defining feature of an RNN is the recurrence: the network's output at each time step depends on
both the current input and the previous hidden state.

RNN Core Equations:

hₜ = f(Wₕ · hₜ₋₁ + Wₓ · xₜ + b) (Hidden State Update)
yₜ = Wᵧ · hₜ + bᵧ (Output at time step t)
Where: xₜ = input at time t | hₜ = hidden state at time t | h ₜ₋₁ = previous hidden state | f = activation
(tanh or ReLU) | Wₕ, Wₓ, Wᵧ = shared weight matrices

A critical property: the same weights (Wₕ, Wₓ, Wᵧ) are shared across all time steps. This is called
parameter sharing — it means the model learns patterns that can be applied at any position in the
sequence, and it keeps the number of parameters manageable regardless of sequence length.

1.3 Unrolling an RNN Through Time

RNN Unrolled (3 time steps):
t=1: x₁ ──► [RNN Cell] ──► h₁ ──► y₁
▲
h₀=0
t=2: x₂ ──► [RNN Cell] ──► h₂ ──► y₂
▲
h₁
t=3: x₃ ──► [RNN Cell] ──► h₃ ──► y₃
▲
h₂
The same RNN cell (same weights) is applied at each step. h₀ is typically initialized to zeros.
1.4 Types of RNN Architectures
RNNs are highly versatile and can be configured for different input-output structures:

Architecture
Input → Output
Use Case Example
One-to-One
Single input → Single output
Standard classification (no sequence)
One-to-Many
Single input → Sequence output
Image captioning — one image, many words
Many-to-One
Sequence input → Single output
Sentiment analysis — many words, one label
Many-to-Many (equal)
Sequence → Sequence (same length)
POS tagging — label each word
Many-to-Many (unequal)
Encoder-Decoder sequence
Machine translation (e.g., English to French)

1.5 Applications of RNNs in Sequential Data Analysis

RNNs and their advanced variants (LSTM, GRU) are applied across a wide range of real-world
domains:
10. Natural Language Processing (NLP): Language modeling, machine translation, text
generation, named entity recognition, sentiment analysis.
11. Speech Recognition: Converting audio waveforms (sequences of sound frames) into text
transcriptions.
12. Time Series Forecasting: Stock price prediction, weather forecasting, energy demand
prediction — all involve predicting the next value in a sequence.
13. Music Generation: Learning patterns in musical sequences to generate new compositions.
14. Video Analysis: Processing sequences of video frames for action recognition or video
captioning.
15. Handwriting Recognition: Recognizing handwritten text from sequences of pen strokes.
2. Back Propagation Through Time (BPTT)
Back Propagation Through Time (BPTT) is the algorithm used to train RNNs. It is the standard
backpropagation algorithm adapted for sequences — gradients are computed by unrolling the RNN
across all time steps and then applying the chain rule backwards through time, just as regular
backpropagation applies the chain rule backwards through layers.

2.1 Forward Pass in BPTT

The forward pass processes the entire input sequence from t=1 to t=T, computing hidden states and
outputs at each step:

BPTT Forward Pass (for t = 1 to T):

hₜ = tanh(Wₕ · hₜ₋₁ + Wₓ · xₜ + bₕ) ← Hidden state
ŷₜ = softmax(Wᵧ · hₜ + bᵧ) ← Output prediction
Lₜ = CrossEntropy(ŷₜ, yₜ) ← Loss at step t
L = Σₜ Lₜ ← Total loss

2.2 Backward Pass in BPTT

After the forward pass, gradients of the total loss L are computed with respect to all weights (W ₕ, Wₓ,
Wᵧ) by unrolling through time. The gradient of L with respect to the hidden state at time t involves
contributions from the current step and all future steps:

BPTT Backward Pass (for t = T down to 1):

∂L/∂Wᵧ = Σₜ ∂Lₜ/∂Wᵧ
∂L/∂hₜ = ∂Lₜ/∂hₜ + Wₕᵀ · δₜ₊₁
δₜ = (∂L/∂hₜ) ⊙ (1 − hₜ²) [tanh derivative]
∂L/∂Wₕ = Σₜ δₜ · hₜ₋₁ᵀ
∂L/∂Wₓ = Σₜ δₜ · xₜᵀ

Key Insight: Because the same weight matrix Wₕ is reused at every time step, the gradient ∂L/∂W ₕ is
the sum of gradient contributions from all time steps. Gradients must flow backwards through all T time
steps — through multiplications of Wₕ and tanh' at each step. This repeated multiplication is the root
cause of the Vanishing Gradient Problem.

2.3 Truncated BPTT

Full BPTT through very long sequences (T = 1000) is computationally expensive and memory-
intensive. Truncated BPTT is a practical approximation:
16. The sequence is split into chunks of fixed length k (e.g., k=50 time steps).
17. Forward pass runs for k steps; backward pass runs only k steps back.
18. Hidden state is carried across chunks (not reset), so long-range context is still approximately
maintained.
19. Trade-off: Gradients only propagate k steps back — gradients from very early in the sequence
are cut off, but training is much faster.
3. Vanishing Gradient Problem in RNNs
The Vanishing Gradient Problem is the most fundamental challenge in training standard RNNs. It
severely limits their ability to learn long-range dependencies — patterns that span many time steps.

3.1 Mathematical Origin

The gradient of the loss at time T with respect to the hidden state at time t (where t << T) involves a
chain of matrix multiplications through time:

Gradient Chain Across Time Steps:

∂L_T/∂hₜ = ∂L_T/∂h_T × ∏ᵢ₌ₜᵀ⁻¹ (∂hᵢ₊₁/∂hᵢ)
∂hᵢ₊₁/∂hᵢ = Wₕᵀ × diag(tanh'(zᵢ))
For tanh: |tanh'(z)| ≤ 1. If |λ_max(Wₕ)| < 1, the product of (T−t) such terms → 0 exponentially fast as the
sequence grows longer.

If the largest singular value of Wₕ is less than 1, and the tanh derivative is also less than 1, then the
product of these terms across many time steps decays exponentially towards zero. The gradient
effectively disappears for time steps far in the past.

3.2 Effect on Learning

Consequence of Vanishing Gradients:
• Weights connected to early time steps receive near-zero gradient updates.
• The network cannot adjust these weights meaningfully — it cannot learn long-range dependencies.
• In practice, vanilla RNNs can only reliably learn dependencies across ~10–20 time steps.
• Example: in a long paragraph, predicting the last word based on context from the first sentence fails.
• The 'short-term memory' problem: the network quickly forgets distant past information.

3.3 Exploding Gradients

Exploding gradients occur when the gradient grows exponentially (when |λ_max(Wₕ)| > 1). The
gradients become astronomically large, causing numerical overflow (NaN) and destabilizing training.
While less subtle than vanishing gradients (the training loss will visibly diverge), exploding gradients are
equally damaging. The primary solution is Gradient Clipping (Section 4).

Problem Cause Symptom Solution

Vanishing Gradients
|λ_max(Wₕ)| < 1 repeatedly multiplied
Network cannot learn long-range patterns; loss plateau
LSTM, GRU, residual connections
Exploding Gradients
|λ_max(Wₕ)| > 1 repeatedly multiplied
NaN loss, training diverges suddenly
Gradient clipping
4. Gradient Clipping
Gradient Clipping is a simple yet highly effective technique to prevent exploding gradients. Before the
weight update step, the gradients are examined and if their magnitude exceeds a chosen threshold,
they are rescaled so that the gradient norm equals the threshold. This prevents any single update from
being catastrophically large while preserving the direction of the gradient.
4.1 Gradient Clipping by Norm (Most Common)

Gradient Clipping by Norm Algorithm:

1. Compute all gradients g = ∂L/∂W
2. Compute global gradient norm: ‖g‖ = √(Σᵢ gᵢ²)
3. If ‖g‖ > threshold (e.g., 1.0 or 5.0):
g ← g × (threshold / ‖g‖) [rescale gradient]
4. Apply gradient update: W ← W − η·g
This preserves the direction of the gradient while bounding its magnitude — the network still moves in
the correct direction, just with a controlled step size.

4.2 Gradient Clipping by Value

An alternative simpler method clips each individual gradient element independently:

gᵢ ← clip(gᵢ, −clip_value, +clip_value)

Example: clip_value = 1.0 → all gradients clamped to [-1.0, +1.0]. Simpler but changes gradient
direction, which is generally less desirable than norm clipping.

4.3 Implementation in Frameworks

# PyTorch — Gradient Clipping by Norm
[Link]()
[Link].clip_grad_norm_([Link](), max_norm=1.0)
[Link]()
# TensorFlow/Keras — Clip in optimizer
optimizer = [Link](clipnorm=1.0)

Choosing the threshold: A threshold of 1.0 or 5.0 is commonly used. Too small a threshold can slow
convergence; too large makes clipping ineffective. In practice, monitor the gradient norm during training
— if it frequently spikes, clipping is needed.

5. Long Short-Term Memory (LSTM) Networks

Long Short-Term Memory (LSTM) networks were introduced by Hochreiter and Schmidhuber in 1997
to directly solve the vanishing gradient problem. LSTMs replace the simple RNN cell with a
sophisticated gated memory cell that can selectively remember, forget, and output information across
long sequences — potentially hundreds or even thousands of time steps.

5.1 The LSTM Cell — Two States

The fundamental innovation of LSTM over vanilla RNN is the introduction of a second state:
20. Cell State (Cₜ): The 'long-term memory' — a separate vector that flows through the network
largely unchanged, allowing gradients to flow through without multiplication. Modifications are
made via additive operations (not multiplicative), preventing gradient decay.
21. Hidden State (hₜ): The 'short-term memory' / working memory — the output of the cell, used for
predictions and passed to the next time step.

5.2 The Three Gates of LSTM

The LSTM controls information flow through three learned gates, each producing values between 0 and
1 (via sigmoid). A gate output of 0 means 'block everything'; 1 means 'let everything through'.

Gate Formula Purpose

Forget Gate (fₜ)
fₜ = σ(Wf · [hₜ₋₁, xₜ] + bf)
Decides what to erase from Cell State. 0=forget, 1=keep
Input Gate (iₜ)
iₜ = σ(Wi · [hₜ₋₁, xₜ] + bi)
Decides what new info to store in Cell State
Candidate Values (C̃ ₜ)
C̃ ₜ = tanh(Wc · [hₜ₋₁, xₜ] + bc)
New candidate values to potentially add to Cell State
Output Gate (oₜ)
oₜ = σ(Wo · [hₜ₋₁, xₜ] + bo)
Decides what to output from Cell State

5.3 LSTM State Update Equations

Complete LSTM Update (at each time step t):

fₜ = σ(Wf · [hₜ₋₁, xₜ] + bf) ← Forget Gate
iₜ = σ(Wi · [hₜ₋₁, xₜ] + bi) ← Input Gate
C̃ ₜ = tanh(Wc · [hₜ₋₁, xₜ] + bc) ← Candidate
Cₜ = fₜ ⊙ Cₜ₋₁ + iₜ ⊙ C̃ ₜ ← Cell State Update
oₜ = σ(Wo · [hₜ₋₁, xₜ] + bo) ← Output Gate
hₜ = oₜ ⊙ tanh(Cₜ) ← Hidden State Output

Why does this solve vanishing gradients? The Cell State update Cₜ = fₜ ⊙ Cₜ₋₁ + iₜ ⊙ C̃ ₜ is an
additive operation. Gradients can flow through this addition unchanged across many time steps,
bypassing the multiplicative decay of vanilla RNNs. The forget gate controls which gradients are
preserved — if fₜ ≈ 1, gradients flow freely; if fₜ ≈ 0, old information is discarded.

5.4 Intuitive Example — Language Modeling

Sentence: 'The clouds in the sky are dark, so it will rain.'
• Forget Gate: After reading 'sky are dark', the LSTM can forget that we were talking about 'clouds'
(specific noun) and focus on the overall situation.
• Input Gate: When reading 'dark', the LSTM writes 'dark weather' concept into the cell state as a
relevant cue for future prediction.
• Output Gate: When predicting the word after 'will', the LSTM outputs the hidden state rich with
'rain/weather' context, making 'rain' the most likely next word.
• Cell State: Carries the 'dark weather' concept across many words — a long-range dependency.

5.5 LSTM in Practice — Code

# PyTorch LSTM Example
import [Link] as nn
lstm = [Link](input_size=100, hidden_size=256, num_layers=2, batch_first=True)
output, (hn, cn) = lstm(x) # x: [batch, seq_len, input_size]
# hn: final hidden state, cn: final cell state
# TensorFlow/Keras LSTM Example
from [Link] import LSTM
[Link](LSTM(256, return_sequences=True, input_shape=(seq_len, features)))

6. Gated Recurrent Units (GRU)

Gated Recurrent Units (GRU) were introduced by Cho et al. in 2014 as a simplified, computationally
efficient alternative to LSTM. GRU achieves similar performance to LSTM on most tasks but with fewer
parameters and a simpler architecture — it uses two gates instead of three and has a single hidden
state instead of LSTM's separate cell state and hidden state.

6.1 GRU Architecture and Equations

GRU Gate Equations (at time step t):

zₜ = σ(Wz · [hₜ₋₁, xₜ] + bz) ← Update Gate
rₜ = σ(Wr · [hₜ₋₁, xₜ] + br) ← Reset Gate
h̃ ₜ = tanh(W · [rₜ ⊙ hₜ₋₁, xₜ] + b) ← Candidate Hidden State
hₜ = (1 − zₜ) ⊙ hₜ₋₁ + zₜ ⊙ h̃ ₜ ← Final Hidden State

6.2 Understanding the GRU Gates

22. Update Gate (zₜ): Controls how much of the previous hidden state is carried forward. When zₜ ≈
1, the new candidate state h̃ ₜ is used; when zₜ ≈ 0, the old state hₜ₋₁ is preserved unchanged.
This gate simultaneously performs the role of both the forget gate and input gate of LSTM.
23. Reset Gate (rₜ): Controls how much of the previous hidden state is used to compute the new
candidate h̃ ₜ. When rₜ ≈ 0, the candidate ignores the previous hidden state and is computed only
from the current input — effectively 'resetting' the memory.

The key formula: hₜ = (1−zₜ) ⊙ hₜ₋₁ + zₜ ⊙ h̃ ₜ is a linear interpolation between the old state and the
new candidate. This interpolation is smooth and differentiable — gradients flow through it easily, solving
the vanishing gradient problem in the same spirit as LSTM's additive cell state update.

6.3 GRU vs. LSTM Comparison

Aspect LSTM GRU
Gates
3 (Forget, Input, Output)
2 (Update, Reset)
States
2 (Cell State + Hidden State)
1 (Hidden State only)
Parameters
More (~4× base)
Fewer (~3× base)
Training Speed
Slower (more computation)
Faster (simpler computation)
Long-range memory
Excellent (explicit cell state)
Good (slightly shorter range)
When to prefer
Complex tasks, very long sequences
Limited data, faster iteration
Performance
Slightly better on some tasks
Comparable on most tasks

Rule of thumb: Start with GRU for a faster baseline. If performance is insufficient, switch to LSTM. On
many practical NLP and time series tasks, GRU and LSTM perform comparably.

6.4 GRU Code Example

# PyTorch GRU Example
gru = [Link](input_size=100, hidden_size=256, num_layers=2, batch_first=True)
output, hn = gru(x) # Note: only hidden state, no separate cell state
7. Bidirectional LSTMs
Bidirectional LSTMs (BiLSTMs) extend the standard LSTM by processing the sequence in both
directions simultaneously — one LSTM processes the sequence from left to right (forward), and a
second LSTM processes it from right to left (backward). At each time step, the hidden states of both
LSTMs are concatenated, giving the network access to both past and future context.

7.1 Architecture of BiLSTM

BiLSTM Architecture (for sequence x₁, x₂, x₃, x₄):

Forward LSTM →: x₁ → x₂ → x₃ → x₄ → h→₁, h→₂, h→₃, h→₄
Backward LSTM ←: x₄ → x₃ → x₂ → x₁ → h←₄, h←₃, h←₂, h←₁
Combined output: yₜ = [h→ₜ ; h←ₜ] (concatenation)
yₜ has dimension 2×H (twice the hidden size), capturing context from both directions.

7.2 Why Bidirectionality Helps

In many NLP tasks, the meaning of a word depends on both what came before and what comes after:

Example: Context determines meaning

Sentence: 'I saw the bat in the cave.'
The word 'bat' could mean a flying mammal or a baseball bat.
Forward LSTM: sees 'I saw the bat' — ambiguous.
BiLSTM: also sees 'in the cave' (from backward pass) — resolves ambiguity: cave → flying mammal.

7.3 Applications of BiLSTM

24. Named Entity Recognition (NER): Labeling each word as PERSON, LOCATION,
ORGANIZATION, etc. — requires both left and right context.
25. Machine Translation (Encoder): The encoder uses BiLSTM to build rich representations of the
source sentence.
26. Sentiment Analysis: Capturing negations ('not bad') requires seeing both the negation and the
sentiment word.
27. Part-of-Speech Tagging: Grammatical role depends on surrounding words.
 28. Question Answering: Finding the answer span in a passage requires understanding context
from both sides.

7.4 Limitation of BiLSTM

Key constraint: BiLSTMs require the entire sequence to be available before processing. They
cannot be used for real-time or streaming tasks where future inputs are not yet available (e.g., live
speech recognition, real-time language generation). For such tasks, unidirectional LSTMs or causal
models are required.

7.5 BiLSTM Code

# PyTorch BiLSTM
bilstm = [Link](input_size=100, hidden_size=256, bidirectional=True, batch_first=True)
output, (hn, cn) = bilstm(x) # output has shape [batch, seq_len, 512]
# Keras BiLSTM
[Link](Bidirectional(LSTM(256, return_sequences=True)))

8. Bidirectional RNNs
Bidirectional RNNs (BiRNNs) generalize the concept of bidirectionality from LSTMs to any RNN
variant — including vanilla RNNs, GRUs, or LSTMs. The principle is identical: process the input
sequence in both directions and combine the hidden states from both passes.

8.1 General BiRNN Architecture

BiRNN Forward Pass Equations:

Forward direction: h→ₜ = f(W→ₕ · h→ₜ₋₁ + W→ₓ · xₜ + b→)
Backward direction: h←ₜ = f(W←ₕ · h←ₜ₊₁ + W←ₓ · xₜ + b←)
Output: yₜ = g(Wᵧ · [h→ₜ ; h← ₜ] + bᵧ)

The forward and backward RNNs have completely separate weight matrices (W→ₕ, W→ₓ vs W← ₕ,
W←ₓ) — they are trained independently via BPTT in their respective directions. The combined output yₜ
aggregates both directional contexts.

8.2 Deep Bidirectional RNNs

Bidirectional RNNs can be stacked into multiple layers to form Deep Bidirectional RNNs:
29. Layer 1: Processes raw inputs bidirectionally, outputs h₁→ and h₁← at each step.
30. Layer 2: Takes concatenated [h₁→ₜ ; h₁←ₜ] as input, processes bidirectionally again.
31. Layer k: Each additional layer learns higher-level bidirectional features.
32. Output: The final layer's concatenated states are passed to the task-specific output layer.

Deep BiRNNs are powerful but computationally expensive — training time scales with both depth and
sequence length. Modern architectures like BERT use Transformer encoders instead, which achieve
similar bidirectional context with more efficient parallelization.

8.3 Comparison: Vanilla RNN vs. BiRNN vs. BiLSTM

Feature Vanilla RNN Bidirectional RNN Bidirectional LSTM
Context access
Past only (→)
Past & Future (← →)
Past & Future (← →)
Gradient issue
Severe vanishing
Severe vanishing
Largely solved
Long-term memory
Very limited (~10 steps)
Very limited
Excellent (100s of steps)
Parameters
Fewest
~2× vanilla RNN
~4× vanilla RNN (per direction)
Speed
Fastest
Moderate
Slowest
Best for
Simple sequences, real-time
Short sequences needing context
Complex NLP tasks

8.4 Transition to Modern Architectures

While Bidirectional RNNs and BiLSTMs were the dominant approach for sequence modeling from
~2015–2018, they have largely been superseded in NLP by Transformer-based models (BERT, GPT,
T5). However, BiLSTMs remain important in:
33. Resource-constrained environments where Transformer models are too large to deploy
(mobile, edge devices).
34. Structured prediction tasks like NER and POS tagging, where BiLSTMs combined with CRF
layers remain competitive.
35. Time series forecasting and scientific data, where the sequential inductive bias of RNNs is
beneficial.
36. Educational and research contexts as the building blocks for understanding modern
sequence models.
Unit 3 Summary & Important Questions

Unit 3 — Core Takeaways

1. RNNs maintain a hidden state (memory) to process sequential data — unlike FFNNs that treat each
input independently.
2. Recurrence: hₜ = f(Wₕ·hₜ₋₁ + Wₓ·xₜ + b). Same weights shared across all time steps (parameter
sharing).
3. BPTT unrolls the RNN through time and applies backpropagation — gradients flow backwards
through all T time steps.
4. Vanishing Gradients: gradients decay exponentially through repeated multiplication of W ₕ × tanh' —
RNN forgets long-range patterns.
5. Exploding Gradients: gradients grow exponentially — solved by Gradient Clipping (rescale if ‖g‖ >
threshold).
6. LSTM: 3 gates (Forget, Input, Output) + separate Cell State. Additive cell update prevents vanishing
gradients.
7. GRU: 2 gates (Update, Reset) — simpler and faster than LSTM, comparable performance on most
tasks.
8. Bidirectional LSTM/RNN: processes sequence in both directions — provides past AND future context
at each step.
9. BiLSTM limitation: requires the entire sequence — cannot be used for real-time/streaming tasks.
10. Deep BiRNNs stack multiple bidirectional layers for richer hierarchical representations.

Important Examination Questions

1. Define RNN. Explain its architecture with recurrence equations and a diagram of the unrolled
RNN across time steps.
2. Why do standard Feed-Forward Networks fail for sequential data? Explain how RNNs
address this limitation.
3. Explain BPTT in detail. Derive the gradient equations for Wₕ, Wₓ, and Wᵧ. What is Truncated
BPTT and why is it used?
4. What is the Vanishing Gradient Problem in RNNs? Explain its mathematical cause using the
chain of gradient multiplications. How does it affect long-range learning?
5. Explain Gradient Clipping. Describe both norm-based and value-based clipping with
algorithms. When is clipping by norm preferred?
6. Explain the LSTM architecture in detail. Describe all three gates (Forget, Input, Output), the
Cell State, and the Hidden State with complete update equations. Why does LSTM solve the
vanishing gradient problem?
7. Compare LSTM and GRU. Explain the GRU gates (Update and Reset) with equations. When
would you prefer GRU over LSTM?
8. What is a Bidirectional LSTM? Explain its architecture with a diagram. List its applications and
mention its key limitation.
9. What is a Bidirectional RNN? Write the forward pass equations. How does adding
bidirectionality improve sequence understanding?
10. Compare Vanilla RNN, Bidirectional RNN, and Bidirectional LSTM in terms of context
access, gradient issues, long-term memory, and speed.
 UNIT 4
Generative Adversarial Networks (GANs)
Deep Learning (2344106) • [Link] CSE • 9 Contact Hours

Topics Covered in Unit 4

1. Generative Models
2. Concept and Principles of GANs
3. Architecture of GANs — Generator and Discriminator
4. Discriminative vs Generative Models
5. Types of GANs — DCGAN, CGAN, LSGAN, CycleGAN, SRGAN
6. Applications of GANs
 1. Generative Models
Definition: A generative model is a class of statistical and machine-learning models that learn the
underlying probability distribution of training data so that they can generate new synthetic data
samples that are statistically indistinguishable from the original data. In formal terms, given a dataset of
samples x drawn from an unknown distribution p_data(x), a generative model tries to approximate
p_data(x) as closely as possible.
Generative models are fundamentally different from predictive or discriminative models. A classifier like
a CNN is trained to answer "what class does this sample belong to?" — it learns the conditional
distribution P(y|x). A generative model instead asks "what does a plausible sample from this domain
look like?" — it learns the joint or marginal distribution P(x).
1.1 Real-World Intuition
Analogy: Consider a painter and an art critic. A discriminative model is like the critic — it looks at a
painting and decides whether it is a genuine Rembrandt or a forgery. A generative model is like the
painter — it studies hundreds of Rembrandt paintings and then creates entirely new paintings that
capture the same style, texture, and composition. Both roles are valuable, but they require
fundamentally different skills.
1.2 Types of Generative Models
The major families of generative models in deep learning are:
• Generative Adversarial Networks (GANs): Use an adversarial game between two networks to
learn implicit distributions. Produce extremely sharp, photorealistic samples.
• Variational Autoencoders (VAEs): Use a probabilistic encoder-decoder framework with a
structured latent space. Samples tend to be smoother but sometimes blurrier.
• Diffusion Models: Learn to reverse a gradual noising process. State-of-the-art for image and
audio generation (e.g., DALL-E 2, Stable Diffusion).
• Autoregressive Models: Generate samples one element at a time, conditioned on previous
elements (e.g., PixelCNN for images, GPT for text).

1.3 Discriminative vs Generative Models — Detailed Comparison

Aspect Discriminative Model Generative Model Core question
What class does x belong to?
What does a sample from P(x) look like?
Learns
Conditional distribution P(y|x)
Joint/marginal distribution P(x) or P(x,y)
Output
Class label / probability of class
New synthesized data sample
Training
Supervised — requires labelled data
Often unsupervised or self-supervised
Examples
Logistic Regression, SVM, CNN classifier, BERT
GAN, VAE, Diffusion Model, GPT
Goal during test
Classify unseen samples correctly
Generate novel, realistic samples
Evaluation metric
Accuracy, F1-score, AUC-ROC
FID score, Inception Score, perceptual quality

📌 Key Insight: Discriminative models are task specialists — they excel at classification but cannot
create. Generative models are creative — they understand the structure of data deeply enough to
produce new instances of it.

2. Concept and Principles of GANs

Generative Adversarial Networks (GANs) were introduced by Ian Goodfellow, Yoshua Bengio, and
collaborators in 2014 in the landmark paper "Generative Adversarial Nets". The paper proposed a
novel training framework in which two neural networks — a Generator and a Discriminator — are
trained simultaneously through a competitive (adversarial) game, driving each other to improve until
the Generator produces perfectly realistic data.
The GAN framework is one of the most elegant ideas in machine learning: instead of defining an
explicit loss function that measures image quality (which is hard — MSE produces blurry images), we
let another neural network learn to judge quality. This adversarial signal produces much sharper,
more realistic outputs than reconstruction-based approaches.
2.1 The Counterfeiter–Detective Analogy
Analogy: Think of the Generator as a counterfeiter who makes fake currency, and the Discriminator
as a detective who examines banknotes and classifies them as real or fake. Initially, the counterfeiter
produces obvious fakes and the detective easily catches them. Over time, the counterfeiter improves
their technique in response to the detective's feedback. The detective in turn sharpens their skills. After
many rounds of this competition, the counterfeiter becomes so skilled that the detective can no longer
reliably distinguish fake notes from real ones — at which point the Generator has effectively learned the
data distribution.
2.2 The Minimax Game — Mathematical Formulation
GAN training is formally a two-player minimax game. The objective function that both players optimize
is:

Minimax Objective:

min_G max_D V(D, G)

V(D, G) = E_{x ~ p_data(x)} [ log D(x) ]

+ E_{z ~ p_z(z)} [ log(1 − D(G(z))) ]

where:
x = real data sample from training distribution p_data(x)
z = random noise vector sampled from prior p_z (e.g., N(0,I))
G(z) = fake sample generated by Generator given noise z
D(x) = probability that Discriminator assigns to x being real
D(G(z))= probability that Discriminator assigns to G(z) being real

Breaking it down term by term:

• E[log D(x)]: Discriminator is rewarded for outputting high probability (close to 1) for real
samples x. Maximising this makes D better at recognising real data.
• E[log(1 − D(G(z)))]: Discriminator is rewarded for outputting low probability (close to 0) for fake
samples G(z). Generator is punished for producing detectable fakes.
 • Generator minimises: G tries to minimise log(1 − D(G(z))), i.e., it wants D(G(z)) → 1 — it
wants to fool D into thinking generated samples are real.
• Discriminator maximises: D tries to maximise the entire expression — correctly classifying
both real and fake samples.

📌 Nash Equilibrium: GAN training converges to a Nash Equilibrium when the Generator perfectly
captures the true data distribution: p_G(x) = p_data(x). At this point, D(x) = 0.5 for every input — D can
do no better than random guessing because real and fake samples are identical. In practice,
convergence to the true equilibrium is hard to guarantee.

2.3 GAN Training Algorithm — Step by Step

Each training iteration alternates between updating D (holding G fixed) and updating G (holding D
fixed):

GAN Training Algorithm (one iteration):

Step 1: Sample a mini-batch of real data: {x⁽¹⁾, ..., x ⁽ᵐ ⁾} ~ p_data(x)

Step 2: Sample a mini-batch of noise: {z⁽¹⁾, ..., z ⁽ᵐ ⁾} ~ p_z(z)
Step 3: Generate fake samples: {G(z ⁽¹ ⁾), ..., G(z ⁽ᵐ ⁾)}

─── Update Discriminator (maximize V) ───

Step 4: Compute D loss:
L_D = −(1/m) Σ [ log D(x⁽ⁱ⁾) + log(1 − D(G(z ⁽ⁱ⁾))) ]
Step 5: Update D weights: θ_D ← θ_D − η · ∇_θ_D L_D
(Repeat steps 4-5 for k steps, typically k=1)

─── Update Generator (minimize V) ───

Step 6: Sample new noise: {z⁽¹⁾, ..., z⁽ᵐ⁾} ~ p_z(z)
Step 7: Compute G loss:
L_G = −(1/m) Σ log D(G(z⁽ⁱ⁾)) ← non-saturating variant
Step 8: Update G weights: θ_G ← θ_G − η · ∇_θ_G L_G

Repeat until convergence.

💡 Non-Saturating Loss: The original G loss log(1−D(G(z))) saturates early in training — when D is
strong and D(G(z)) ≈ 0, the gradient is very small. In practice, the non-saturating variant maximises log
D(G(z)) instead, providing stronger gradients early in training.

3. Architecture of GANs — Generator and Discriminator

A GAN consists of two separate neural networks with opposing objectives. Their architecture is task-
dependent — for image generation, both use convolutional layers; for text generation, recurrent or
transformer layers are used. We describe the canonical image-generation architecture below, followed
by a concrete worked example.
3.1 Generator Network
The Generator (G) is a neural network that takes a random noise vector z sampled from a prior
distribution (typically N(0,I) — standard Gaussian) and transforms it into a synthetic data sample of the
target domain. For image generation, the Generator must upsample from a low-dimensional noise
vector to a full-resolution image.
Key design principles of the Generator:
• Input: Noise vector z of fixed dimension (e.g., 100). This is the "seed" — different z values
produce different images.
• Upsampling layers: Transposed Convolutions (also called deconvolutions or fractionally
strided convolutions) progressively increase spatial resolution: 4×4 → 8×8 → 16×16 → 32×32
→ 64×64.
• Batch Normalisation: Applied after each layer (except the output) to stabilise training and
prevent internal covariate shift.
• Activation: ReLU after each hidden layer. Tanh at the output layer — scales pixel values to
[−1, 1], matching normalised real images.
• No sigmoid in hidden layers: Sigmoid saturates and kills gradients — the Generator needs
rich gradients from D to learn.

Generator Architecture — Image Generation (64×64 RGB output):

Input: z ∈ R^100 (random noise)

Layer 1: Dense(100 → 4×4×512) + Reshape → feature map 4×4×512

BatchNorm + ReLU

Layer 2: ConvTranspose2D(512 → 256, kernel=4, stride=2, padding=1)

Output: 8×8×256 | BatchNorm + ReLU

Layer 3: ConvTranspose2D(256 → 128, kernel=4, stride=2, padding=1)

Output: 16×16×128 | BatchNorm + ReLU

Layer 4: ConvTranspose2D(128 → 64, kernel=4, stride=2, padding=1)

Output: 32×32×64 | BatchNorm + ReLU

Layer 5: ConvTranspose2D(64 → 3, kernel=4, stride=2, padding=1)

Output: 64×64×3 | Tanh activation

Final: G(z) ∈ R^(64×64×3) — a synthesized RGB image

3.2 Discriminator Network

The Discriminator (D) is a neural network that takes an image — either a real sample from the training
set or a fake sample produced by G — and outputs a single scalar probability P(image is real). It is
essentially a binary classifier trained to distinguish real data from generated data. Its architecture is the
mirror image of the Generator — it uses strided convolutions to progressively downsample the input
image into a compact feature vector.
Key design principles of the Discriminator:
• Input: Image of target resolution (e.g., 64×64×3). Pixel values normalised to [−1, 1].
• Strided Convolutions: Downsample spatial dimensions without pooling layers. Strided
convolutions are learnable and often preferred over max pooling in GANs.
• LeakyReLU: Used instead of ReLU to prevent dying neurons — allows small gradients for
negative inputs (slope 0.2). Critical for GAN Discriminators.
• No Batch Norm on first layer: The first layer of D operates on raw pixel values — batch norm
here can hurt performance.
 • Output: Single neuron with Sigmoid activation → probability ∈ [0, 1]. D(x) ≈ 1 means "likely
real"; D(x) ≈ 0 means "likely fake".

Discriminator Architecture — Image Classifier (64×64 RGB input):

Input: Image ∈ R^(64×64×3)

Layer 1: Conv2D(3 → 64, kernel=4, stride=2, padding=1)

Output: 32×32×64 | LeakyReLU(0.2) [No BatchNorm here]

Layer 2: Conv2D(64 → 128, kernel=4, stride=2, padding=1)

Output: 16×16×128 | BatchNorm + LeakyReLU(0.2)

Layer 3: Conv2D(128 → 256, kernel=4, stride=2, padding=1)

Output: 8×8×256 | BatchNorm + LeakyReLU(0.2)

Layer 4: Conv2D(256 → 512, kernel=4, stride=2, padding=1)

Output: 4×4×512 | BatchNorm + LeakyReLU(0.2)

Layer 5: Flatten → Dense(4×4×512 → 1)

Sigmoid activation

Final: D(x) ∈ [0,1] — probability that x is a real image

GAN Architecture Overview:

┌─────────────┐
z ~ N(0,I) ──────►│ GENERATOR │──── G(z) (fake image) ──┐
(noise vector) └─────────────┘ │
▼
Real image
───────────────────────────────────────►┌───────────────┐
x ~ p_data │ DISCRIMINATOR │──► D(·) ∈ [0,1]
└───────────────┘
│
┌───────────────────────────────────┘
│ Gradient signal
▼
D is updated: maximise [ log D(x) + log(1−D(G(z))) ]
G is updated: maximise [ log D(G(z)) ] (non-saturating)

3.3 The Generator-Discriminator Interaction — Worked Example

Example: Training a GAN to generate handwritten digit images (MNIST, 28×28 grayscale).
37. Iteration 1 (Early training): G produces random grey blobs. D easily classifies them as fake.
D(G(z)) ≈ 0.02. G's gradient is large — it updates aggressively to fool D.
38. Iteration 100: G starts producing vague digit shapes. D is confused on some samples. D(G(z))
≈ 0.2. Both networks improve.
39. Iteration 1000: G produces recognisable digits with varied strokes. D accuracy drops to ~65%.
G has learned strokes and curves.
 40. Iteration 5000 (near convergence): G produces sharp, realistic digits indistinguishable from
real MNIST. D(G(z)) ≈ 0.48–0.52. Training dynamics stabilise.

4. Comparison: Discriminative vs Generative Models

Having studied both model types in depth, we now compare them across several important dimensions:

Dimension
Discriminative Models
Generative Models
Mathematical formulation
Learns P(y|x) — conditional distribution
Learns P(x) or P(x,y) — generative distribution
Training objective
Minimize classification/regression loss
Reconstruct data, maximize likelihood, or adversarial game
Data requirements
Requires labelled (x, y) pairs
Can train unsupervised on unlabelled x alone
Primary output
Class label or score
New synthesized data sample
Sample quality
N/A — not designed to generate
GANs: very sharp; VAEs: slightly blurry; Diffusion: best
Interpretability
Decision boundary is interpretable
Latent space can encode semantically meaningful features
Training stability
Stable — single loss to minimize
GANs are notoriously unstable; VAEs/Diffusion are stable
Computational cost
Moderate
High — GANs and Diffusion models need many GPU hours
Typical tasks
Classification, detection, segmentation
Image synthesis, data augmentation, style transfer
State-of-the-art examples
ResNet, ViT, BERT, YOLO
GANs (StyleGAN), VAE, Stable Diffusion, DALL-E

📌 Complementary Roles: Modern deep learning systems often combine both. For example, a GAN
itself contains a discriminative model (D) and a generative model (G). Semi-supervised learning uses a
generative model to augment labelled data for a discriminative classifier. These two families are not
opposites — they are partners.

5. Types of GANs
Since the original GAN paper in 2014, hundreds of GAN variants have been proposed to address
different problems — training instability, mode collapse, lack of output control, low resolution, and more.
The syllabus covers five major types: DCGAN, CGAN, LSGAN, CycleGAN, and SRGAN.
5.1 DCGAN — Deep Convolutional GAN
Proposed by: Alec Radford, Luke Metz, and Soumith Chintala — "Unsupervised Representation
Learning with Deep Convolutional Generative Adversarial Networks" (2015).
Motivation: The original GAN used fully connected (dense) layers, which are parameter-inefficient for
images and do not exploit the spatial structure of image data. DCGAN replaced all dense layers with
convolutional and transposed convolutional layers, establishing a stable and effective GAN
architecture for image generation that became the default starting point for most image GAN work.
Core Architectural Rules of DCGAN
• Replace pooling with strided convolutions: In D, use strided Conv2D (stride=2) instead of
max pooling. In G, use ConvTranspose2D (stride=2) instead of upsampling + conv. This lets the
network learn its own spatial downsampling/upsampling.
• Batch Normalisation everywhere: Applied in both G and D, except on the output layer of G
and the input layer of D.
• ReLU in Generator: All G hidden layers use ReLU; output layer uses Tanh.
• LeakyReLU in Discriminator: All D hidden layers use LeakyReLU(α=0.2) for better gradient
flow; output uses Sigmoid.
• No fully connected hidden layers: Only the first layer of G (projecting noise z) and the last
layer of D (classification) involve dense connections.

Component Original GAN DCGAN

G hidden layers
Dense (FC) layers
Transposed Conv2D + BatchNorm + ReLU
D hidden layers
Dense (FC) layers
Strided Conv2D + BatchNorm + LeakyReLU
Downsampling
Not applicable
Strided convolutions (learnable)
Upsampling
Not applicable
Transposed convolutions (learnable)
Spatial structure
Ignored
Explicitly modelled
Training stability
Often unstable
Significantly more stable
Image quality
Low resolution, poor quality
Much sharper, higher resolution

Example — DCGAN on CelebA: Trained on 200,000 celebrity face images (CelebA dataset), DCGAN
generates photorealistic 64×64 face images of people who do not exist. The Generator learns facial
attributes such as eyes, nose, hair, and skin tone from scratch through the adversarial signal alone —
no labels are used. The network even organises the latent space such that z_smiling − z_neutral ≈ a
"smiling direction" that can be applied to other faces (latent vector arithmetic).

5.2 CGAN — Conditional GAN

Proposed by: Mehdi Mirza and Simon Osindero — "Conditional Generative Adversarial Nets" (2014).
Problem with Vanilla GAN: Standard GANs generate samples randomly from the learned distribution.
You cannot control what gets generated. If trained on MNIST, the GAN produces random digits — you
cannot say "generate a 7". CGAN solves this by conditioning both networks on additional information.
Core Idea: Both G and D receive an extra conditioning variable y (which can be a class label, text
description, or any auxiliary information). The Generator produces G(z|y) — a sample conditioned on y.
The Discriminator judges whether (x, y) is a real pair — i.e., whether the image x actually belongs to
class y.

CGAN Objective (Conditional Minimax):

min_G max_D V(D, G)

V(D, G) = E_{x,y ~ p_data} [ log D(x|y) ]

+ E_{z ~ p_z, y ~ p_y} [ log(1 − D(G(z|y)|y)) ]

Implementation:
G input: concat(z, y_embedding) → generates sample of class y
D input: concat(x, y_embedding) → judges if x is real given class y

y is typically embedded (one-hot or learned embedding) before concat.

Example — CGAN on MNIST:

41. Training: Pairs (image, label) are fed to both G and D. G receives (z, label=3) and must
generate a "3". D receives (image, label) and must verify the image matches the label.
42. Inference: To generate a "7", feed z ~ N(0,I) and y=7 into G. To generate a "0", feed y=0.
Complete class control at test time.
43. Mismatched pairs: D is also trained on mismatched pairs (real image of "3" paired with label
"7") — these should be classified as fake. This forces D to check semantic consistency, not just
visual quality.

Broader Applications of CGAN:

• Text-to-Image: Condition on text embedding; G generates image matching the description ("a
red apple on a white background").
• Age Progression: Condition on target age; G transforms a face photo to look younger or older.
• Medical Imaging: Condition on diagnosis code; generate synthetic MRI scans of specific
pathologies for data augmentation.
• Fashion: Condition on clothing type and colour; generate product images for e-commerce.

5.3 LSGAN — Least Squares GAN

Proposed by: Xudong Mao et al. — "Least Squares Generative Adversarial Networks" (2017).
Problem: The original GAN Discriminator uses a Sigmoid activation + Binary Cross-Entropy loss. The
gradient of BCE loss with respect to the Discriminator output saturates when samples are far from the
decision boundary. Concretely: if D correctly classifies a fake sample with very high confidence
(D(G(z)) ≈ 0), the gradient flowing back to G is nearly zero — G receives no useful signal to improve,
even though its output may still look very different from real data.

Saturation problem in original GAN:

Original G loss: L_G = log(1 − D(G(z)))

If D(G(z)) ≈ 0 (D is very confident fake):

∂L_G/∂D(G(z)) = 1/(1−D(G(z))) ≈ 1/(1−0) = 1 [large enough]
BUT: ∂D/∂(G(z)) ≈ 0 since Sigmoid saturates at 0
Combined gradient → ≈ 0 → G stops learning!

Result: Fake samples that are confidently classified as fake

receive no gradient → G cannot improve from these samples

LSGAN Solution: Replace the Sigmoid+BCE discriminator with a linear output + Least Squares
(MSE) loss. Real samples are assigned a target of 1, fake samples a target of 0. The Least Squares
loss penalises the distance from the decision boundary — even correctly classified samples far from
the boundary receive gradient, pulling them closer to the boundary and improving quality continuously.

LSGAN Objective:

Discriminator loss:
L_D = (1/2) · E_{x~p_data} [(D(x) − 1)²]
+ (1/2) · E_{z~p_z} [(D(G(z)) − 0)²]

Generator loss:
L_G = (1/2) · E_{z~p_z} [(D(G(z)) − 1)²]

D output: linear (no Sigmoid)

Interpretation: D outputs a real-valued score, not a probability.
Real target = 1, fake target = 0, G wants D to output 1 for its fakes.

Property Vanilla GAN LSGAN

D output
Sigmoid → probability ∈ [0,1]
Linear → real-valued score
Loss function
Binary Cross-Entropy (BCE)
Mean Squared Error (MSE / L2)
Gradient at boundary
Saturates — vanishes for confident predictions
Always non-zero — proportional to distance from boundary
Training stability
Prone to oscillations, mode collapse
More stable training dynamics
Image quality
Good baseline
Visually higher quality, sharper details
Theoretical basis
Jensen-Shannon divergence
Pearson χ² divergence

Example: LSGAN trained on bedroom images (LSUN dataset) produces sharper textures — straight
lines in furniture, edges of windows — compared to vanilla GAN, which tends to produce wavy, blurry
edges. The continuous MSE signal corrects even "good enough" fakes that BCE would neglect.

5.4 CycleGAN — Cycle-Consistent GAN

Proposed by: Jun-Yan Zhu, Taesung Park, et al. — "Unpaired Image-to-Image Translation using
Cycle-Consistent Adversarial Networks" (2017).
Problem — Paired data is expensive: Image-to-image translation tasks (e.g., converting a horse
photo to a zebra photo) are useful, but collecting paired training images is often impossible or
prohibitively expensive. You cannot photograph the same horse and zebra from exactly the same angle
and lighting to create a training pair. CycleGAN enables image translation using only two unpaired
sets of images — for example, 1,000 horse photos and 1,000 zebra photos, with no correspondence
between them.
Core Idea — Cycle Consistency: CycleGAN uses two generators and two discriminators. If we
learn G: X→Y (horse to zebra), we simultaneously learn F: Y→X (zebra to horse). The key constraint is
cycle consistency: translating an image to the other domain and back should return the original
image. This constraint alone — enforced as a loss — is sufficient to learn meaningful translation without
paired data.

CycleGAN Architecture:

Generators: G: X → Y (e.g., horse → zebra)

F: Y → X (e.g., zebra → horse)

Discriminators: D_Y: distinguishes real Y from G(X) [real zebra vs. fake zebra]
D_X: distinguishes real X from F(Y) [real horse vs. fake horse]

Cycle-Consistency Constraint:
Forward cycle: F(G(x)) ≈ x for x ∈ X (horse → zebra → horse ≈ original horse)
Backward cycle: G(F(y)) ≈ y for y ∈ Y (zebra → horse → zebra ≈ original zebra)

Cycle-Consistency Loss:
L_cyc(G, F) = E_x [ ||F(G(x)) − x||₁ ]
+ E_y [ ||G(F(y)) − y||₁ ] (L1 norm)

Total Loss:
L_total = L_GAN(G, D_Y, X, Y) [G adversarial loss]
+ L_GAN(F, D_X, Y, X) [F adversarial loss]
+ λ · L_cyc(G, F) [cycle loss, λ=10 typically]

Why Cycle Consistency Works: Without any constraint between G and F, the Generator can
hallucinate any mapping that fools D — it might convert every horse to the same random zebra. Cycle
consistency forces the mapping to be information-preserving — the content (pose, background,
structure) must be retained while only the style is changed. This acts as a powerful self-supervised
regulariser.

Real-World Examples of Cycle GAN:

Domain X → Domain Y
Application
Horse → Zebra
Changing animal stripe pattern without changing pose or background
Summer → Winter
Seasonal conversion of landscape photos
Monet painting → Real photo
Unpaired artistic style to photorealistic conversion
Real photo → Monet painting
Photorealistic to artistic style conversion (any artist)
MRI → CT scan
Medical image modality translation (avoids costly paired collection)
Day → Night
Scene lighting transformation for autonomous driving data augmentation
Aerial map → Street map
Satellite-to-map translation for navigation
Apple → Orange
Object texture translation preserving shape

📌 Key Advantage: CycleGAN requires no paired training data — only two collections of images from
the two domains. This dramatically reduces data collection costs and makes the framework applicable
to problems where pairing is physically impossible.

5.5 SRGAN — Super Resolution GAN

Proposed by: Christian Ledig et al. — "Photo-Realistic Single Image Super-Resolution Using a
Generative Adversarial Network" (2016).
Problem — Traditional Super Resolution: Classical super-resolution methods (bicubic interpolation,
SRCNN) minimise pixel-wise MSE loss. This produces images that score well on PSNR (Peak Signal-
to-Noise Ratio) and SSIM (Structural Similarity Index), but look perceptually blurry because MSE
averages over many plausible high-resolution details, producing the average of all possibilities — a
blurry image. SRGAN replaces MSE with a combination of perceptual and adversarial losses to recover
sharp, photorealistic high-frequency details.
Goal: Given a low-resolution (LR) image, generate a corresponding high-resolution (HR) image with a
magnification factor of 4× (e.g., 64×64 → 256×256) that is perceptually indistinguishable from a real
HR image.

SRGAN Architecture:

Generator (SRResNet):
Input: LR image (e.g., 64×64×3)
Core: 16 Residual Blocks (Conv → BatchNorm → PReLU → Conv → BatchNorm + skip)
Output: HR image (256×256×3) via sub-pixel convolution (PixelShuffle) upsampling

Discriminator:
 Input: Image (HR real or SR generated) — 256×256×3
Core: 8 Conv layers with strided convolutions + LeakyReLU
Output: Probability real ∈ [0, 1]

SRGAN Loss Function:

L_SR = L_content + λ · L_adversarial

L_content (Perceptual Loss):

= ||φ(SR) − φ(HR)||² where φ = VGG-19 feature maps (not pixel space)
Compares high-level feature representations, not raw pixels.

L_adversarial:
= −log D(G(LR)) (Generator tries to fool D)

Key insight: Perceptual loss uses a pre-trained VGG-19 network to compare

feature maps, producing perceptually sharp results that MSE cannot achieve.

Why VGG Perceptual Loss? MSE in pixel space is equivalent to asking "are the exact pixel values the
same?" — but two images that look identical to a human eye may have very different pixel values if one
is shifted by 1 pixel. VGG feature space compares semantic content (textures, edges, object parts),
which correlates much better with human perception of image quality.

Method Loss Used PSNR

Perceptual Quality
Bicubic Interpolation
— (no training)
Low
Blurry, no detail
SRCNN
MSE (pixel)
Good
Slightly blurry
SRResNet (no GAN)
MSE (pixel)
Best PSNR
Still over-smooth
SRGAN
Perceptual + Adversarial
Lower than SRResNet
Sharp, photorealistic — best perceptual quality

Example: An SRGAN trained on face images can recover fine details — individual hair strands, pores,
eyelashes — from a blurry 64×64 crop that are completely absent in the bicubic upsampled version.
Although PSNR is lower (because the network fills in plausible but not necessarily pixel-accurate
details), human observers consistently prefer SRGAN outputs in perceptual quality studies (Mean
Opinion Score — MOS).

5.6 Summary Comparison of GAN Variants

GAN Variant Year
Key Innovation
Training Data Required
Primary Application
Vanilla GAN
2014
Adversarial training framework; minimax loss
Unlabelled images
Proof of concept; random image generation
DCGAN
2015
Convolutional layers; architectural guidelines (BatchNorm, LeakyReLU, strided conv)
Unlabelled images
Stable image generation (faces, bedrooms)
CGAN
2014
Conditioning on class label y — control over output
Labelled (x, y) pairs
Class-conditional generation, text-to-image
LSGAN
2017
Least-Squares loss (MSE) instead of BCE — no gradient saturation
Unlabelled images
Higher quality generation; more stable training
CycleGAN
2017
Cycle-consistency loss; two G + two D; no paired data needed
Two unpaired image sets
Unpaired image-to-image translation
SRGAN
2016
Perceptual (VGG) + adversarial loss; ResNet generator
LR-HR image pairs
4× single image super-resolution

6. Applications of GANs
GANs have found remarkable applications across a wide range of domains. They represent one of the
most practically impactful developments in deep learning, enabling capabilities — such as generating
photorealistic images — that were previously thought to require human creativity or were
computationally infeasible.
6.1 Computer Vision & Image Processing
• Photorealistic Image Synthesis: StyleGAN2 (NVIDIA) and BigGAN generate 1024×1024 high-
fidelity faces, animals, and objects ([Link]). Generated images are used
for synthetic datasets and creative applications.
• Image-to-Image Translation: Pix2Pix (paired) and CycleGAN (unpaired) convert sketches to
photos, aerial maps to street maps, day to night, and summer to winter, enabling dataset
creation where real paired data is scarce.
 • Image Super-Resolution: SRGAN, ESRGAN enhance images 2×–8× beyond their native
resolution with photorealistic detail. Used in medical imaging, satellite analysis, digital forensics,
and video streaming (upscaling HD to 4K).
• Image Inpainting: Context-aware image completion — removing objects from photos and filling
with contextually appropriate backgrounds. Used in photo editing software (Adobe Photoshop's
Content-Aware Fill).
• Face Editing & Age Progression: Modify facial attributes (add/remove glasses, change hair
colour, age progression). Used in entertainment, forensics, and personalised advertising.
6.2 Medical Imaging
• Synthetic Medical Data Generation: GANs generate synthetic MRI, CT, and X-ray images to
augment training datasets for medical AI models — critical in domains where real patient data is
scarce or privacy-protected.
• Cross-Modality Translation: CycleGAN converts MRI scans to CT scans and vice versa —
valuable when a patient has only one modality available but both are needed for diagnosis.
• Tumour Segmentation Augmentation: GANs generate varied tumour appearances in different
organs, improving segmentation model robustness.
• Histology Image Enhancement: Enhance low-quality biopsy slide images for better pathologist
diagnosis accuracy.
6.3 Natural Language Processing
• Text Generation: GANs applied to text sequences (SeqGAN, TextGAN) generate realistic
sentences, paragraphs, and dialogue. Combined with reinforcement learning to handle discrete
token generation.
• Text-to-Image Synthesis: AttnGAN and StackGAN generate images described by natural
language captions. Predecessor to DALL-E and Stable Diffusion.
6.4 Audio & Music
• Speech Synthesis: WaveGAN generates raw audio waveforms; GAN-TTS generates natural-
sounding speech with diverse voice characteristics.
• Music Generation: MuseGAN generates polyphonic music with multiple instruments in MIDI
format.
• Voice Conversion: StarGAN-VC converts speech from one speaker's voice to another —
preserving content while changing vocal identity.
6.5 Data Augmentation & Anomaly Detection
• Rare Class Augmentation: GANs generate synthetic training examples for underrepresented
classes in imbalanced datasets — e.g., rare disease cases, rare vehicle types in autonomous
driving datasets.
• Anomaly Detection: Train a GAN on normal data only. At inference, compute D(x) — low
discriminator confidence on a test sample indicates it falls outside the learned distribution,
flagging it as an anomaly. Used in fraud detection, industrial defect detection, and network
intrusion detection.
6.6 Autonomous Driving & Robotics
• Synthetic Scene Generation: GANs generate diverse urban driving scenes (different weather,
lighting, traffic) to supplement real driving data for training perception systems.
• Sim-to-Real Transfer: GANs convert synthetic simulation images (from game engines) to look
photorealistic, enabling models trained on synthetic data to transfer to real-world sensors.
6.7 Drug Discovery & Chemistry
• Molecular Generation: GroundTruth GAN and ORGAN use GANs to generate novel molecular
structures in SMILES notation with desired chemical properties — massively accelerating drug
discovery by exploring chemical space more efficiently than random search.
 • Protein Structure Prediction: Generative models generate candidate protein folding
configurations for drug target analysis.

Application Domain GAN Variant / System Real-World Example

Face generation
StyleGAN2
[Link] — unlimited photorealistic faces
Image translation (unpaired)
CycleGAN
Horse↔Zebra; Monet↔Photos; Summer↔Winter
Image translation (paired)
Pix2Pix
Sketch→Photo; Segmentation map→Real scene
Super-resolution
SRGAN, ESRGAN
Enhance satellite, medical, CCTV images 4×–8×
Medical imaging
MedGAN, CycleGAN
Synthetic MRI/CT generation for training data
Text-to-image
AttnGAN, StackGAN
"A small red bird with black wings" → Image
Anomaly detection
Vanilla GAN + D score
Credit card fraud detection, industrial QA
Data augmentation
CGAN
Generate rare pathology images for medical AI
Audio synthesis
WaveGAN, GAN-TTS
Natural-sounding speech without human recordings
Drug discovery
ORGAN
Generate molecules with desired pharmacological properties

7. GAN Training Challenges and Solutions

Despite their power, GANs are notoriously difficult to train. Understanding the failure modes is essential
for working with them in practice.

Challenge Description Symptoms Solutions

Mode Collapse
G learns to produce only a few types of samples, ignoring the diversity of the data distribution. Nash
equilibrium is not reached.
All generated samples look similar; low diversity; G ignores z
Mini-batch discrimination; Unrolled GANs; WGAN; diverse training batches
Vanishing Gradients
D becomes too strong early — perfectly classifies all samples. G receives near-zero gradients.
G loss plateaus; generated images stagnate; training stalls
Non-saturating loss; LSGAN; train D fewer steps (k=1); label smoothing
Training Instability
G and D loss oscillate wildly; neither converges
Loss plots spike; generated images degrade after initial improvement
Gradient penalty (WGAN-GP); spectral normalisation; lower learning rate
Evaluation Difficulty
No single loss value indicates the quality of generated images — loss and quality are decoupled
Hard to know when to stop training
FID score; Inception Score (IS); human evaluation
Hyperparameter Sensitivity
GAN performance highly sensitive to LR, architecture, batch size
Works on one dataset, fails on another
Extensive grid search; follow DCGAN guidelines; use proven architectures

FID Score (Fréchet Inception Distance) — Standard GAN Evaluation Metric:

FID = ||μ_r − μ_g||² + Tr(Σ_r + Σ_g − 2(Σ_r·Σ_g)^(1/2))

where:
μ_r, Σ_r = mean and covariance of real image features (from Inception-v3)
μ_g, Σ_g = mean and covariance of generated image features

Interpretation:
FID = 0 → Generated distribution identical to real distribution (perfect)
FID low → High quality and high diversity of generated samples
FID high → Poor quality, low diversity, or mode collapse

Example FID scores:

Random noise: FID ≈ 300+
Vanilla GAN (CIFAR-10): FID ≈ 70
DCGAN (CIFAR-10): FID ≈ 37
StyleGAN2 (FFHQ): FID ≈ 2.84 (near-photographic quality)

Unit 4 Summary & Important Examination Questions

Key Concepts at a Glance
Unit 4 — Core Takeaways
1. Generative models learn P(x) and generate new data; Discriminative models learn P(y|x) and
classify
2. GANs (Goodfellow, 2014) use a minimax game: G minimises, D maximises V(D,G)
3. Generator: noise z → upsampling ConvTranspose layers → Tanh → fake image
4. Discriminator: image → strided Conv layers → LeakyReLU → Sigmoid → real/fake score
5. Nash equilibrium: p_G = p_data, D(x) = 0.5 everywhere
6. DCGAN replaced FC layers with convolutions; established stable GAN training guidelines
7. CGAN adds conditioning variable y to control what G generates (class label, text)
8. LSGAN uses MSE loss instead of BCE to prevent gradient saturation
9. CycleGAN enables unpaired image-to-image translation via cycle consistency constraint
10. SRGAN uses perceptual (VGG) + adversarial loss for photorealistic super-resolution
11. Major challenges: mode collapse, vanishing gradients, training instability
12. FID score is the standard evaluation metric — lower is better

Important Examination Questions

• Define a Generative Model. Explain the difference between discriminative and generative
models with examples and a comparison table.
• What is a Generative Adversarial Network (GAN)? Explain the minimax objective function in
detail, breaking down each term.
• Describe the GAN training algorithm step by step. What is the non-saturating loss and why is
it preferred over the original formulation?
• Explain the Generator and Discriminator architectures for image generation with layer-by-
layer details. Why is LeakyReLU preferred in D and Tanh at the output of G?
• What is DCGAN? State the architectural guidelines of DCGAN and explain why each guideline
improves training stability and image quality.
• What is CGAN? How does conditioning work in both G and D? Illustrate with a class-
conditional MNIST example.
• Explain LSGAN. What is the gradient saturation problem in vanilla GAN and how does Least
Squares loss solve it? Write the LSGAN objective.
• What is CycleGAN? Define cycle consistency. Write the cycle-consistency loss equation and
explain why it enables unpaired image-to-image translation.
• Explain SRGAN. Why does MSE loss produce blurry super-resolved images? How does the
perceptual (VGG) loss address this? Write the SRGAN loss function.
• What is mode collapse in GANs? Define it formally and list three techniques to mitigate it.
• What is the FID score? Write its formula, explain each term, and interpret what a low vs. high
FID score means.
• List and explain six real-world applications of GANs across different domains — computer
vision, medical imaging, audio, and drug discovery.
 UNIT 5
Auto-Encoders
Deep Learning (2344106) • [Link] CSE • 9 Contact Hours

Topics Covered in Unit 5

1. Introduction to Auto-Encoders
2. Architecture and Components (Encoder, Code/Bottleneck, Decoder)
3. Training an Auto-Encoder for Data Compression and Reconstruction
4. Variant: Denoising Auto-Encoders
5. Variant: Variational Auto-Encoders (VAE)
6. Variant: Convolutional Auto-Encoders
7. Relationship between Auto-Encoders and GANs
8. Hybrid Models: Encoder-Decoder GANs
1. Introduction to Auto-Encoders
An Auto-Encoder (AE) is an unsupervised deep learning model that learns to compress (encode) input
data into a compact lower-dimensional representation, and then reconstruct (decode) the original input
from that representation. The defining characteristic is that the network learns to be its own teacher: the
target output is the same as the input.

1.1 Definition
Definition:
An Auto-Encoder is a type of artificial neural network used for learning efficient codings of unlabelled
data. It consists of two parts: an Encoder that maps the input to a latent (compressed) space, and a
Decoder that reconstructs the input from that latent space. The network is trained by minimizing a
reconstruction loss — the difference between the input and the reconstructed output.

1.2 Key Intuition

Think of an auto-encoder as a bottleneck: the network is forced to learn the most important features of
the data in order to squeeze it through the narrow middle layer, and then recover the full input on the
other side. This is analogous to data compression — like a ZIP file algorithm, but learned automatically
from data.

Example: Suppose you have 784-pixel grayscale images of handwritten digits (28×28). A naive
approach would store all 784 values. An auto-encoder trained on this data can learn to represent each
digit using only 32 numbers — capturing the essence of the digit (stroke pattern, curvature, etc.) without
storing every pixel explicitly. The decoder can then regenerate a close replica of the original image from
just those 32 numbers.

1.3 Why Auto-Encoders Matter

Application What Auto-Encoder Does Data Compression
Compresses high-dimensional data to a compact code for efficient storage/transmission
Dimensionality Reduction
Replaces PCA with a non-linear alternative that captures complex patterns
Anomaly Detection
Normal data reconstructs well; anomalies show high reconstruction error
Denoising
Remove noise from corrupted images, audio, or sensor data
Generative Modeling
Variational AEs can generate new, plausible data samples
Pre-training
Learn useful feature representations from unlabeled data for downstream tasks

2. Architecture and Components of Auto-Encoders

An auto-encoder has a symmetric architecture with three core components: the Encoder, the Code
(Bottleneck/Latent Space), and the Decoder. Together they form an hourglass-shaped network.
2.1 Overall Architecture Diagram

INPUT (x) RECONSTRUCTION (x̂ )

[784 neurons] [784 neurons]
│ │
↓ ENCODER DECODER ↑
[512 neurons] [512 neurons]
↓ ↑
[256 neurons] [256 neurons]
↓ ↑
[128 neurons] [128 neurons]
↓ ↑
└─────── CODE / LATENT SPACE ───────┘
[32 neurons]
(compressed representation)

Objective: Minimize Reconstruction Loss L(x, x̂ ) = ||x - x̂ ||^2

2.2 Component 1: The Encoder

The Encoder is a function f that maps the input x (which lives in a high-dimensional space) to a lower-
dimensional latent code z:
z = f_θ(x) where f is the encoder function parameterized by θ
Typically: z = σ(W_enc · x + b_enc) (with possible multiple layers)

The encoder progressively compresses the input through a series of hidden layers, each with fewer
neurons than the last. Each layer learns increasingly abstract representations — for images, early
layers might detect edges, middle layers detect shapes, and the final encoder layer produces the
compact latent code.

Encoder — Key Properties

• The encoder performs a non-linear dimensionality reduction.
• Each layer: z^(l) = σ(W^(l) · z^(l-1) + b^(l)) where σ is a non-linear activation (ReLU, tanh).
• The encoder 'compresses' data — discards noise while preserving structure.
• For an input of dimension D and code of dimension d: d << D (e.g., 784 → 32).
• The encoder learns which features matter by being trained end-to-end.

2.3 Component 2: The Code (Bottleneck / Latent Space)

The Code (also called the Bottleneck or Latent Representation) z is the output of the encoder — the
compressed form of the input. It is the most critical layer: its size (dimensionality) controls the tradeoff
between compression and reconstruction quality.
z ∈ ℝ^d where d << D (d is the code dimension, D is the input dimension)
Example: Input is 784-dimensional, code is 32-dimensional → Compression ratio = 784/32 =
24.5x
The bottleneck forces the network to learn an information bottleneck — to identify and retain the most
essential structure in the data while discarding irrelevant details (noise). The quality of the latent space
determines whether the auto-encoder can generalize: a well-organized latent space means similar
inputs will have similar codes.

Example: For MNIST digits, the latent space may organize itself such that all '3's cluster together, all
'7's cluster together, etc. — even though the network was never told class labels!

2.4 Component 3: The Decoder

The Decoder is a function g that maps the latent code z back to a reconstruction of the original input:
x̂ = g_φ(z) where g is the decoder function parameterized by φ
Typically: x̂ = σ(W_dec · z + b_dec) (with possible multiple layers)

The decoder is typically the mirror image of the encoder — it has the same number of layers but in
reverse order, progressively expanding the representation back to the original input dimension. The
final decoder layer uses sigmoid activation (for normalized pixel values between 0 and 1) or linear
activation (for real-valued outputs).

Decoder — Key Properties

• The decoder performs a non-linear expansion: code dimension d → input dimension D.
• Architecture is symmetric to encoder: mirror of encoder layers.
• Final activation: sigmoid for image pixels [0,1], linear for regression.
• The decoder learns to 'hallucinate' missing details from the compressed code.
• Quality of decoder output is measured by the reconstruction loss.

2.5 Mathematical Formulation

For a simple single-hidden-layer auto-encoder, the complete computation is:
ENCODER: z = σ_1(W_1 · x + b_1)
DECODER: x̂ = σ_2(W_2 · z + b_2)
LOSS: L = (1/N) Σ_i ||x_i - x̂ _i||^2 (Mean Squared Error)
or L = -(1/N) Σ_i [x_i log(x̂ _i) + (1-x_i) log(1-x̂ _i)] (Binary Cross-Entropy)
LEARNING: θ, φ ← argmin_{θ,φ} L(x, g_φ(f_θ(x)))

The network is trained end-to-end via backpropagation — the gradients flow from the reconstruction
loss through the decoder, through the bottleneck, and into the encoder. Both encoder and decoder
weights are jointly optimized.
3. Training an Auto-Encoder
Training an auto-encoder involves choosing a suitable architecture, defining a reconstruction loss, and
optimizing the network using gradient descent. The training process is self-supervised: no labels are
needed — the network's own input serves as the target output.

3.1 Step-by-Step Training Process

• Initialize encoder weights W_enc and decoder weights W_dec (Xavier/He initialization).
• Feed a mini-batch of inputs {x_1, x_2, ..., x_B} through the encoder to get codes {z_1, ..., z_B}.
• Feed each code z_i through the decoder to get reconstructions {x̂ _1, ..., x̂ _B}.
• Compute the reconstruction loss: L = (1/B) Σ ||x_i - x̂ _i||^2.
• Backpropagate the loss through decoder and encoder jointly.
• Update all weights: W ← W - η · ∂L/∂W using an optimizer (Adam, SGD).
• Repeat for all mini-batches across many epochs until loss converges.

3.2 Reconstruction Loss Functions

Loss Function Formula When to Use
MSE (L2) (1/N)Σ||x - x̂ ||^2 Continuous values, images
MAE (L1) (1/N)Σ|x - x̂ | Sparse data, outlier-robust
Binary Cross-Entropy -Σ[x log(x̂ ) + (1-x) log(1-x̂ )] Pixel values in [0,1]

3.3 Use Case: Data Compression and Reconstruction

The primary traditional application of auto-encoders is lossy data compression. After training:
44. Compression: Run the encoder f on any input x to get code z. Store/transmit z instead of x.
45. Reconstruction: On the receiving end, run the decoder g on z to get x̂ ≈ x.
46. The compression ratio is D/d (e.g., 784/32 = 24.5x smaller).
47. The reconstruction is an approximation — some information is lost through the bottleneck.

Concrete Example: MNIST Digit Compression

Input: 28×28 grayscale digit image = 784 float values

Code: 32-dimensional vector (compression ratio = 24.5x)
After training 10 epochs on 60,000 images:
• Reconstruction loss (MSE) drops from ~0.12 to ~0.015
• Reconstructed images look visually very similar to originals
• Digit class information is largely preserved in the 32 code values
• Using code size = 2 gives maximum compression but blurry reconstructions
• Using code size = 128 gives near-perfect reconstructions but less compression
4. Denoising Auto-Encoders (DAE)
A Denoising Auto-Encoder (DAE) is a variant of the standard auto-encoder that is trained to recover a
clean input from a corrupted (noisy) version of it. Rather than trying to simply reconstruct the exact
input, the network learns to capture the underlying true structure of the data.

4.1 Definition
Definition (Denoising Auto-Encoder):
A Denoising Auto-Encoder is trained with input x̃ (a corrupted version of x) but target output
x (the original, clean input). It learns to map noisy inputs back to clean data, thereby
learning a robust representation that captures the true data manifold.

4.2 How It Works

During training, corruption is artificially added to clean training samples:

TRAINING PROCESS:

Clean Input (x) → [Add Noise] → Corrupted Input (x̃ )

↓
ENCODER
↓
Latent Code (z)
↓
DECODER
↓
Reconstruction (x̂ )
↓
Loss = ||x - x̂ ||^2 (compare to CLEAN x!)

Key: Input is corrupted x̃ , but loss is computed against clean x

4.3 Types of Noise/Corruption

Corruption Type Description Example
Gaussian Noise Add random Gaussian noise: Salt-and-pepper in photos
x̃ = x + ε, ε~N(0,σ^2)
Masking Noise Randomly set a fraction of Missing pixels in an image
inputs to zero
Salt-and-Pepper Randomly set pixels to 0 or 1 Random black/white specks
(min/max)
Dropout Noise Randomly drop input features Missing words in text
 (set to 0)

4.4 Why Denoising Helps

Forcing the network to reconstruct clean inputs from noisy ones has a regularizing effect: the encoder
cannot simply memorize inputs or learn trivial identity mappings. It must learn a representation that
captures the true underlying structure of the data — not the noise. This leads to more robust,
generalizable latent representations.

Example Application: In medical imaging, X-ray or MRI scans are often noisy due to hardware
limitations. A DAE trained on pairs of (noisy scan, clean scan) learns to remove imaging artifacts,
improving diagnostic quality without requiring complex hand-crafted filters.

5. Variational Auto-Encoders (VAE)

A Variational Auto-Encoder (VAE) is a generative variant of the auto-encoder that introduces a
probabilistic framework to the latent space. Unlike standard auto-encoders (which map inputs to fixed
point vectors), VAEs map inputs to probability distributions in the latent space, enabling the model to
generate new, realistic data samples.

5.1 Definition
Definition (VAE):
A Variational Auto-Encoder is a generative model that learns a probabilistic mapping from
data to a continuous, structured latent space. The encoder outputs distribution parameters
(μ, σ^2) instead of a fixed code, samples z from that distribution, and the decoder
reconstructs the input from z. It is trained by maximizing the Evidence Lower Bound
(ELBO).

5.2 Standard AE vs. VAE: Key Difference

Aspect Standard Auto-Encoder Variational Auto-Encoder
Encoder output Fixed vector z Mean μ and variance σ^2
Latent space Unstructured, can have gaps Continuous, smooth,
structured
Sampling Deterministic Stochastic: z ~ N(μ, σ^2)
Generation Poor (gaps in latent space) Excellent (smooth
interpolation)
Loss Function Reconstruction loss only Reconstruction + KL
Divergence

5.3 VAE Architecture

 Input x
↓
[Encoder Network]
↓ ↓
μ (mean) log(σ^2) (log-variance) ← Encoder outputs TWO vectors
│ │
└───────────┘
↓
Reparameterization Trick:
z = μ + σ ⊙ ε where ε ~ N(0, I) ← Enables backpropagation through sampling!
↓
[Decoder Network]
↓
Reconstruction x̂

5.4 The Reparameterization Trick

Sampling z from a distribution (z ~ N(μ, σ^2)) is a non-differentiable operation — gradients cannot flow
through a random node. The reparameterization trick solves this by expressing the sample as a
deterministic function of the parameters plus independent noise:

Instead of: z ~ N(μ, σ^2) (non-differentiable sampling)

Use: z = μ + σ ⊙ ε, where ε ~ N(0, I) (differentiable!)

μ and σ are outputs of the encoder network → gradients can now flow through them

5.5 VAE Loss Function (ELBO)

The VAE is trained by maximizing the Evidence Lower BOund (ELBO), equivalently minimizing:

L_VAE = Reconstruction Loss + KL Divergence

L_VAE = E[||x - x̂ ||^2] + β · KL[ N(μ, σ^2) || N(0, I) ]

KL term = -½ Σ_j (1 + log(σ_j^2) - μ_j^2 - σ_j^2)

β is a weighting factor (standard VAE: β = 1; β-VAE: β > 1 for disentanglement)

The KL Divergence term acts as a regularizer that forces the latent distribution to be close to a standard
Normal N(0,I). This ensures the latent space is smooth and continuous — so that sampling random
points from N(0,I) and decoding them produces meaningful, realistic outputs.

VAE Example Application: Face Generation

 Train VAE on 200,000 celebrity face images (CelebA dataset).
Encoder learns: μ and σ for each face in 128-dimensional latent space.
Latent space organizes along meaningful axes (smile, age, gender, etc.).
Generation: Sample z ~ N(0, I) and decode → get a brand new realistic face!
Interpolation: Blend between two faces by linearly interpolating their latent codes.
Editing: Shift a face along the 'smile' axis in latent space → face smiles more.

6. Convolutional Auto-Encoders (CAE)

A Convolutional Auto-Encoder (CAE) replaces the fully-connected layers of a standard auto-encoder
with convolutional layers, making it highly suited for image data. The encoder uses convolution and
pooling to progressively reduce spatial resolution, while the decoder uses transposed convolutions
(deconvolutions) and upsampling to recover the original spatial dimensions.

6.1 Definition
Definition (CAE):
A Convolutional Auto-Encoder is an auto-encoder where both the encoder and decoder are
built using convolutional layers (and optionally pooling/upsampling layers). It exploits the
spatial structure of image data through translation-invariant filters, learning compact feature
maps as the latent representation.

6.2 Architecture

CONVOLUTIONAL AUTO-ENCODER (for 28×28 input images):

Input: 28×28×1
↓ Conv(32, 3×3) + ReLU
28×28×32
↓ MaxPool(2×2) ENCODER
14×14×32
↓ Conv(64, 3×3) + ReLU
14×14×64
↓ MaxPool(2×2)
7×7×64 ←───── Latent Feature Maps (Bottleneck)
↓ ConvTranspose(32, 3×3) + ReLU
14×14×32
↓ Upsample(2×2) DECODER
28×28×32
↓ ConvTranspose(1, 3×3) + Sigmoid
28×28×1 ←───── Reconstruction
6.3 Why Convolutions for Images?
Property Benefit
Parameter sharing Convolutional filters are shared across spatial locations
→ far fewer parameters than FC layers
Translation invariance Filters detect features (edges, textures) regardless of
where they appear in the image
Spatial structure Preserves the 2D layout of the image, leading to better-
quality reconstructions
Scalability Works on high-resolution images (256×256, 1024×1024)
where FC layers would be infeasible

6.4 Transposed Convolution (Deconvolution)

The decoder uses Transposed Convolutions (sometimes called deconvolutions) to upsample feature
maps. Unlike standard convolutions that reduce spatial size, transposed convolutions increase it:

Standard Conv: Input H×W → Output (H/s)×(W/s) (s = stride, typically reduces size)
Transposed Conv: Input H×W → Output (H×s)×(W×s) (increases size)
Also can use: Upsample (nearest-neighbor or bilinear) followed by Conv

CAE Application: Image Colorization

Input: Grayscale image (1 channel, H×W)

Encoder: Extracts features (shapes, textures, object boundaries)
Bottleneck: Compact representation of scene content
Decoder: Predicts color channels (2 channels in LAB color space)
Output: Colorized image (3 channels, H×W)
Training: Pairs of (grayscale, color) images
Famous use: Automatic photo colorization of historical black-and-white images

7. Relationship between Auto-Encoders and GANs

Auto-Encoders and Generative Adversarial Networks (GANs) are two major paradigms for
unsupervised/generative deep learning. While they approach the problem differently, they share deep
connections and complement each other's strengths and weaknesses.

7.1 Comparison: Auto-Encoders vs. GANs

Aspect Auto-Encoders (AE/VAE) GANs
Core idea Encode input to latent z, Generator vs. Discriminator
decode to reconstruct x adversarial training
Training objective Minimize reconstruction loss (+ Minimax game: G minimizes,
 KL for VAE) D maximizes
Latent space Explicit encoder maps x → z No encoder; z is sampled from
prior
Sample quality Blurry (especially VAE due to Sharp, realistic (especially
MSE loss) DCGAN, StyleGAN)
Training stability Stable, straightforward Unstable, mode collapse risk
optimization
Encoding ability Can encode any input to latent Standard GAN has no encoder
space
Key use cases Compression, anomaly Image synthesis, super-
detection, denoising resolution

7.2 Strengths and Weaknesses

Auto-Encoders — Strengths
48. Stable training: Direct reconstruction loss is well-behaved.
49. Explicit latent space: Can encode, interpolate, and manipulate latent codes.
50. Anomaly detection: High reconstruction error signals out-of-distribution samples.
51. Versatile: Works for compression, denoising, dimensionality reduction.

Auto-Encoders — Weaknesses
52. Blurry reconstructions: MSE loss encourages averaging over pixel distributions → blurry
outputs.
53. Limited generation quality: VAE samples are less sharp than GAN outputs.
54. No adversarial sharpness: No discriminator to ensure perceptual quality.

GANs — Strengths
55. Photorealistic generation: Discriminator pushes generator to produce perceptually sharp
images.
56. High visual quality: State-of-the-art for image, video, and audio synthesis.

GANs — Weaknesses
57. No encoder: Cannot map a real input to a latent code without modifications.
58. Unstable training: Balancing G and D is difficult; mode collapse is common.
59. No explicit latent representation for downstream tasks.

7.3 The Complementary Relationship

The insight that led to hybrid models is that Auto-Encoders and GANs are complementary: AEs provide
stable training and a structured latent space, while GANs provide a perceptual quality mechanism (the
discriminator). By combining them, we get models that are both stable to train and produce high-quality
outputs.
 Hybrid insight:
Auto-Encoder (AE/VAE) → structured latent space + stable training
GAN Discriminator → perceptual quality + sharpness
Combined (VAE-GAN) → structured generation + photorealistic quality

8. Hybrid Models: Encoder-Decoder GANs

Hybrid Encoder-Decoder GAN models combine the encoder-decoder structure of auto-encoders with
the adversarial training of GANs. These hybrids leverage the strengths of both architectures to achieve
high-quality generation with structured, interpretable latent spaces.

8.1 VAE-GAN
The VAE-GAN (Variational Auto-Encoder GAN) combines a VAE with a GAN discriminator. The VAE
provides a structured latent space and a reconstruction-based training signal; the GAN discriminator
provides perceptual quality guidance by distinguishing between real and reconstructed/generated
images.

Architecture

Real Image x
↓
[VAE Encoder: x → μ, σ]
↓ (reparameterize: z = μ + σ⊙ε)
Latent Code z
┃
┣━━━━[VAE Decoder: z → x̂ ] → x̂ (Reconstructed)
┃ ↓
┃ z ~ N(0,I) → [GAN Generator: z → x̃ ] → x̃ (Generated)
┃ ↓
┗━━━━━━━━━[GAN Discriminator: is this real?]
┓
D(x)=real ┣ D(x̂ )=fake ┣ D(x̃ )=fake

Combined Loss: L = L_reconstruct + L_KL + λ · L_adversarial

The combined training loss for VAE-GAN has three terms: the reconstruction loss (how well the
decoder reconstructs x from z), the KL divergence (how well the latent distribution matches N(0,I)), and
the adversarial loss (how well the discriminator is fooled by reconstructed/generated images). Together
they ensure both structural accuracy and perceptual quality.

8.2 BiGAN / ALI (Adversarially Learned Inference)

BiGAN (Bidirectional GAN) and ALI jointly train an encoder, a generator, and a discriminator. The
discriminator receives pairs (x, z) and must distinguish between real pairs (x, E(x)) and generated pairs
(G(z), z). This trains an encoder alongside the generator.

Real pair: x → [Encoder E] → z_E → (x, z_E) → [Discriminator D] → Real/Fake

Generated pair: z → [Generator G] → x_G → (x_G, z) → [Discriminator D] → Real/Fake

D must distinguish: real pair (x, E(x)) vs. generated pair (G(z), z)

8.3 AEGAN and Other Variants

Model Key Idea Application
VAE-GAN VAE encoder + GAN High-quality image generation
generator/discriminator
BiGAN/ALI Train encoder alongside Unsupervised feature learning
generator in GAN framework
AEGAN AE regularization + GAN Image-to-image translation
training for style transfer
BEGAN Use AE as discriminator; Stable, high-quality generation
balance training via energy
VGAN VAE prior structure + GAN Semi-supervised learning
adversarial training

8.4 Encoder-Decoder Architecture in Other Contexts

The encoder-decoder structure is not limited to auto-encoders — it is a general architectural pattern
used throughout deep learning:
60. U-Net: Encoder-decoder with skip connections for biomedical image segmentation.
61. Seq2Seq: Encoder encodes input sequence (e.g., English sentence), decoder generates output
sequence (e.g., French translation) — used in machine translation.
62. Image Captioning: CNN encoder encodes image; LSTM decoder generates caption.
63. Pix2Pix: Encoder-decoder GAN for paired image-to-image translation (sketch → photo).
64. DALL-E (early version): Uses VQVAE (Vector Quantized VAE) as the encoder-decoder
backbone.

Key Insight: Why Hybrid Models Win

Auto-Encoders alone produce blurry outputs because MSE loss averages pixel uncertainty.
GANs alone produce sharp images but have no encoder and training is unstable.
Hybrid models (VAE-GAN, BiGAN) combine:
• AE strength: stable optimization, structured latent space, encodable inputs
• GAN strength: adversarial sharpness, photorealistic quality
Result: Models that are both trainable and produce high-perceptual-quality outputs.
Modern architectures like VQVAE-2 and Stable Diffusion's VAE use these principles.
Unit 5 Summary & Important Questions
Key Concepts at a Glance
Unit 5 — Core Takeaways

1. Auto-Encoder: Unsupervised model with Encoder → Bottleneck → Decoder. Input =

Target output.
2. Encoder: Maps x to latent code z. z = σ(W_enc · x + b). Reduces dimensionality.
3. Bottleneck: Low-dimensional code z << D. Controls compression vs. quality tradeoff.
4. Decoder: Maps z back to x̂ . Mirror of encoder. Loss = ||x - x̂ ||^2.
5. Denoising AE: Train with corrupted input x̃ , target is clean x. Learns robust features.
6. VAE: Encoder outputs (μ, σ^2). Sample z = μ + σ⊙ε. Loss = Reconstruction + KL
divergence.
7. Reparameterization trick: Enables gradients to flow through stochastic sampling.
8. Convolutional AE: Uses Conv layers for image data. Encoder uses pooling; decoder uses
transposed convolutions.
9. AE vs. GAN: AEs are stable but blurry; GANs are sharp but unstable, no encoder.
10. VAE-GAN hybrid: Combines VAE's structured latent space with GAN's adversarial
sharpness.
11. ELBO (Evidence Lower Bound): Objective maximized by VAE = reconstruction +
regularization.
12. Applications: Compression, denoising, anomaly detection, generation, image
colorization.

Important Examination Questions

• Define Auto-Encoder. Explain its architecture with a neat diagram showing the Encoder,
Bottleneck, and Decoder components.
• What is the reconstruction loss in an auto-encoder? Compare MSE and Binary Cross-Entropy
losses.
• Explain how an auto-encoder is trained end-to-end. Describe the role of backpropagation.
• What is a Denoising Auto-Encoder? How does it differ from a standard auto-encoder? Describe
the types of noise used.
• Define a Variational Auto-Encoder (VAE). Explain the reparameterization trick and why it is
needed.
• Derive the VAE loss function (ELBO). What is the role of the KL divergence term?
• Compare Standard Auto-Encoders and Variational Auto-Encoders in terms of latent space,
generation quality, and loss functions.
• Explain the architecture of a Convolutional Auto-Encoder. What is a Transposed Convolution?
Why are convolutions preferred for images?
• Compare Auto-Encoders and GANs: their objectives, training stability, latent space, and
generation quality.
• What is a VAE-GAN? Explain its architecture and combined loss function. Why do hybrid
models outperform each component alone?
• Write a short note on BiGAN/ALI and its role in unsupervised feature learning.
• List five real-world applications of Auto-Encoders and briefly describe each.
Formula Sheet for Quick Revision
Standard AE: z = fθ(x), x̂ = gφ(z)
AE Loss: L = ||x - x̂ ||^2 (MSE)

VAE Encoder: outputs μ and log(σ^2)

Reparam. Trick: z = μ + σ ⊙ ε, ε ~ N(0,I)
ELBO Loss: L_VAE = E[||x-x̂ ||^2] + β · KL[N(μ,σ^2) || N(0,I)]
KL Term: = -½ Σ(1 + logσ_j^2 - μ_j^2 - σ_j^2)

Denoising AE: Train on (x̃, x): x̃ = x + noise, target = clean x

Compression: Code dim d << Input dim D (ratio = D/d)