M.

Sc (IT) Sem IV | Deep Learning | Module 2: Deep Networks + CNNs

DEEP LEARNING
Module 2 — Deep Networks & Convolutional Neural Networks
[Link] (IT) | Year II / Semester IV | SVKM's UPG College | 2024-25
Duration: 15 Lectures · Exam Weightage: ~25%

Part A — Deep Networks: Deep Feedforward Networks, MLPs, Regularisation (Dropout, Weight
Decay), Optimisation Algorithms (SGD, Adam)
Part B — CNNs: Convolutional Layers, Pooling Layers, CNN Architectures, CNN Applications

Page 1 | SVKM's UPG College | Deep Learning Notes 2024-25

[Link] (IT) Sem IV | Deep Learning | Module 2: Deep Networks + CNNs

PART A — DEEP NETWORKS

1. Deep Feedforward Networks

📌 Deep Feedforward Network: Also called a feedforward neural network or multilayer perceptron
(MLP). A neural network where information flows only in one direction — from input to output —
with no cycles or feedback connections. 'Deep' means it has multiple hidden layers.
Feedforward networks are the quintessential deep learning model. They form the basis of almost
all modern deep learning architectures — CNNs, RNNs, and Transformers all contain feedforward
sub-networks.

1.1 Why 'Feedforward'?

• Information flows forward: Input layer → Hidden layers → Output layer. No loops.
• Contrast with recurrent networks: RNNs have feedback connections (output feeds back as
input for next step).
• Mathematical form: A composition of functions — f(x) = fₙ(fₙ₋₁(...f₂(f₁(x))...)).
• Each fᵢ is a layer transformation: fᵢ(x) = activation(Wᵢx + bᵢ).

1.2 Universal Approximation Theorem

📌 Universal Approximation Theorem: A feedforward network with a single hidden layer
containing a sufficient (possibly exponential) number of neurons can approximate any continuous
function to arbitrary precision, given a non-linear activation function.
• Proved by Cybenko (1989) for sigmoid activations. Extended by Hornik (1991) to general
activations.
• Key implications:
– Theoretical justification: MLPs are powerful enough to represent any function.
– Width vs depth trade-off: A single wide layer CAN approximate anything — but
exponentially many neurons needed. Multiple narrower layers (depth) achieve the same
with far fewer parameters.
– Does NOT tell you how to train the network — only that it exists in principle.
• Why depth helps: Deep networks learn hierarchical representations — each layer abstracts
the previous. Edges → corners → shapes → objects (in vision). This compositional
structure matches real-world data.
✎ Example: Image classification: Layer 1 detects edges, Layer 2 detects corners and curves, Layer
3 detects object parts (wheel, eye), Layer 4 detects whole objects (car, face). No single layer could
learn all this efficiently.

1.3 Network Architecture Notation

• L: Total number of layers (including output, excluding input). 'Depth' = L.
• nˡ: Number of neurons in layer l. 'Width' of layer l.
• Wˡ ∈ ℝⁿˡˣⁿˡ⁻¹: Weight matrix connecting layer l-1 to layer l.
• bˡ ∈ ℝⁿˡ: Bias vector for layer l.
• zˡ = Wˡaˡ⁻¹ + bˡ: Pre-activation (linear combination) at layer l.
• aˡ = g(zˡ): Post-activation (apply activation function g element-wise).
• aˡ is the input to layer l+1.
Forward pass: a⁰=x → z¹=W¹a⁰+b¹ → a¹=g(z¹) → z²=W²a¹+b² → ... → aᴸ=ŷ

Page 2 | SVKM's UPG College | Deep Learning Notes 2024-25

[Link] (IT) Sem IV | Deep Learning | Module 2: Deep Networks + CNNs

💡 EXAM TIP: Deep feedforward networks: know the forward pass formula layer by layer,
Universal Approximation Theorem (any function, sufficient width), and why depth matters
(hierarchical features, parameter efficiency).

2. Multilayer Perceptrons (MLPs)

The MLP is the classic architecture of deep learning — a stack of fully connected layers with non-
linear activations. Every modern deep learning model builds upon this fundamental structure.

2.1 Single Neuron — The Building Block

📌 Neuron (Perceptron): Computes a weighted sum of inputs, adds a bias, then applies a non-
linear activation function: output = g(wᵀx + b).
z = w₁x₁ + w₂x₂ + ... + wₙxₙ + b = wᵀx + b [pre-activation]
a = g(z) [post-activation, also called activation]
• w: Weight vector — learnable parameters determining input importance.
• b: Bias — learnable scalar shifting the activation threshold.
• g: Activation function — introduces non-linearity (essential for learning complex patterns).
• Without activation: stacking linear layers = one linear layer. Cannot learn non-linear
functions.

2.2 Activation Functions — Complete Guide

Sigmoid (Logistic):
σ(z) = 1 / (1 + e⁻ᶻ) Range: (0, 1) Derivative: σ(z)(1 - σ(z))
• Output interpretable as probability (0 to 1).
• Problems: Vanishing gradient — σ'(z) max = 0.25. For |z| > 4, derivative ≈ 0. Gradient
signal dies in deep networks. Also not zero-centred — slow convergence.
• Use: Output layer for binary classification (final sigmoid → P(y=1|x)). Not recommended for
hidden layers.
Tanh (Hyperbolic Tangent):
tanh(z) = (eᶻ - e⁻ᶻ)/(eᶻ + e⁻ᶻ) Range: (-1, 1) Derivative: 1 -
tanh²(z)
• Zero-centred (unlike sigmoid) — better gradient flow.
• Still has vanishing gradient problem for large |z|.
• Use: RNN hidden states, some classification tasks. Preferred over sigmoid for hidden
layers.
ReLU (Rectified Linear Unit) — Most Popular:
ReLU(z) = max(0, z) Range: [0, ∞) Derivative: 1 if z>0, 0 if z<0
• Computationally cheap — just a threshold operation.
• No vanishing gradient for positive z (gradient = 1).
• Sparse activation — negative inputs give 0. Creates sparse representations.
• Problem — Dying ReLU: If a neuron always gets negative input (e.g., large negative bias),
its gradient is always 0 — neuron permanently 'dead', never learns.
• Use: Default activation for hidden layers in most deep networks. CNNs, MLPs,
Transformers FFN.
Leaky ReLU:

Page 3 | SVKM's UPG College | Deep Learning Notes 2024-25

[Link] (IT) Sem IV | Deep Learning | Module 2: Deep Networks + CNNs

LeakyReLU(z) = max(αz, z) where α=0.01 Never zero gradient

• Fixes dying ReLU — small gradient (α) for negative inputs instead of zero.
• Parametric ReLU (PReLU): α is learnable per neuron.
ELU (Exponential Linear Unit):
ELU(z) = z if z>0, α(eᶻ-1) if z≤0 α typically = 1
• Smooth for all z. Negative outputs possible → closer to zero mean → faster learning.
• More expensive than ReLU (exponential computation).
GELU (Gaussian Error Linear Unit) — Modern Standard:
GELU(z) = z · Φ(z) where Φ is standard normal CDF
Approximation: GELU(z) ≈ 0.5z(1 + tanh[√(2/π)(z + 0.044715z³)])
• Smooth, probabilistic gating — weights inputs by their Gaussian probability.
• Use: BERT, GPT, all modern Transformers. Outperforms ReLU on many NLP tasks.
Softmax — Output Layer for Classification:
softmax(z)ᵢ = exp(zᵢ) / Σⱼ exp(zⱼ) Outputs sum to 1
• Converts raw scores (logits) to a probability distribution over K classes.
• Use: ONLY in output layer for multi-class classification. Never in hidden layers.
• Temperature: softmax(z/τ). τ→0: hard argmax. τ→∞: uniform distribution.

Activation Range Deep Learning Use

Sigmoid (0,1) Output: binary classification.
Avoid in hidden layers
(vanishing grad)
Tanh (-1,1) RNN hidden states. Better
than sigmoid (zero-centred)
ReLU [0,∞) Default hidden layers. CNNs,
MLPs. Fast, sparse
Leaky ReLU (-∞,∞) Fixes dying ReLU.
Recommended over ReLU
when ReLU fails
ELU (-α,∞) Smooth negative, zero-mean
outputs. Regularising effect
GELU ≈(-0.17,∞) Modern Transformers (BERT,
GPT). Best on NLP tasks
Softmax (0,1), sum=1 Output layer multi-class
classification only

2.3 MLP Forward Pass — Full Example

Consider a 3-layer MLP for classifying handwritten digits (MNIST: 784 inputs, 10 classes):
Layer 0 (Input): a⁰ = x ∈ ℝ⁷⁸⁴ (784 pixel values, flattened)
Layer 1 (Hidden): z¹ = W¹a⁰ + b¹ W¹ ∈ ℝ⁵¹²ˣ⁷⁸⁴, b¹ ∈ ℝ⁵¹²
a¹ = ReLU(z¹) a¹ ∈ ℝ⁵¹²
Layer 2 (Hidden): z² = W²a¹ + b² W² ∈ ℝ²⁵⁶ˣ⁵¹²

Page 4 | SVKM's UPG College | Deep Learning Notes 2024-25

[Link] (IT) Sem IV | Deep Learning | Module 2: Deep Networks + CNNs

a² = ReLU(z²) a² ∈ ℝ²⁵⁶
Layer 3 (Output): z³ = W³a² + b³ W³ ∈ ℝ¹⁰ˣ²⁵⁶
ŷ = softmax(z³) ŷ ∈ ℝ¹⁰ (class
probabilities)
• Total parameters: (784×512+512) + (512×256+256) + (256×10+10) = 532,490 ≈ 530K
parameters.
• Loss: Cross-entropy L = -Σₖ yₖ log(ŷₖ) where y is one-hot true label.

2.4 Backpropagation — How MLPs Learn

📌 Backpropagation: Algorithm for efficiently computing gradients of the loss w.r.t. all parameters
using the chain rule of calculus, propagating error signals backward from output to input.
Key Insight — Chain Rule:
∂L/∂W¹ = (∂L/∂a³)(∂a³/∂z³)(∂z³/∂a²)(∂a²/∂z²)(∂z²/∂a¹)(∂a¹/∂z¹)(∂z¹/∂W¹)
• Backward pass computes these partial derivatives layer by layer, reusing intermediate
results.
• Error signal δˡ = ∂L/∂zˡ — how much the pre-activation at layer l contributes to loss.
• Output layer: δᴸ = ŷ - y (for cross-entropy + softmax — elegant simplification).
• Hidden layer l: δˡ = (Wˡ⁺¹)ᵀ δˡ⁺¹ ⊙ g'(zˡ) — propagate error back, weight by activation
derivative.
• Weight gradient: ∂L/∂Wˡ = δˡ (aˡ⁻¹)ᵀ — outer product of error and input.
• Bias gradient: ∂L/∂bˡ = δˡ.
Vanishing Gradient Problem:
• Each layer multiplies by g'(zˡ). Sigmoid derivative max = 0.25.
• For L layers: gradient ∝ (0.25)ᴸ → exponentially small for deep networks.
• Early layers learn extremely slowly or not at all.
Solutions to Vanishing Gradients:
• ReLU activation: gradient = 1 for positive z → no repeated multiplication by small number.
• Residual connections (skip connections): Highway for gradients — bypass layers.
• Batch Normalisation: Keeps activations in a range where gradients are non-zero.
• Careful initialisation: Xavier/Glorot or He initialisation maintains variance across layers.
💡 EXAM TIP: MLP exam: Draw architecture (input→hidden→output layers), write forward
pass equations layer by layer. Explain backprop chain rule formula. Vanishing gradient: why
sigmoid fails (derivative 0.25), why ReLU solves it (derivative=1). Activation functions: know
all 6 with use cases.

3. Regularisation Techniques for Deep Learning

Regularisation methods reduce overfitting — the tendency of deep networks to memorise training
data rather than learning generalizable patterns. Overfitting = low training error, high validation
error.

3.1 The Bias-Variance Trade-off

📌 Bias: Error from wrong assumptions. High bias = underfitting. Model too simple to capture the
pattern.
📌 Variance: Error from sensitivity to training data fluctuations. High variance = overfitting. Model
too complex, memorises noise.

Page 5 | SVKM's UPG College | Deep Learning Notes 2024-25

[Link] (IT) Sem IV | Deep Learning | Module 2: Deep Networks + CNNs

Total Error = Bias² + Variance + Irreducible Noise

• Underfitting (high bias): Training AND validation error both high. Model too simple.
• Overfitting (high variance): Training error low, validation error high. Large train-val gap.
• Goal: Find the sweet spot — complex enough to capture signal, not so complex as to
memorise noise.
✎ Example: Polynomial regression: degree-1 (straight line) underfits complex data. Degree-50
overfits perfectly on training points but wild oscillations elsewhere. Degree-5 — just right.

3.2 Dropout
📌 Dropout: During training, randomly set each neuron's output to 0 with probability p (dropout
rate). During inference, use all neurons but scale outputs by (1-p). Effectively trains an ensemble of
2ⁿ sub-networks.
Training: h̃ᵢ = mᵢ · hᵢ
where mᵢ ~ Bernoulli(1-p) [mask m: 1=keep,
0=drop]
Inference: h̃ᵢ = (1-p) · hᵢ [scale to match expected training output]
Inverted Dropout (modern standard):
Training: h̃ᵢ = (mᵢ · hᵢ) / (1-p) [scale UP during training]
Inference: h̃ᵢ = hᵢ [no scaling needed at test time]
• Typical dropout rates: p=0.5 for fully connected layers. p=0.1-0.2 for convolutional layers.
Why Dropout Works:
• Ensemble interpretation: Each forward pass uses different sub-network (different neurons
dropped). Inference = geometric mean of all 2ⁿ sub-network predictions.
• Co-adaptation prevention: Neurons cannot rely on specific other neurons always being
present → must learn robust, independent features.
• Noise injection: Stochasticity acts as regulariser — like data augmentation on activations.
• Sparsity: 50% of neurons off → sparse representations, similar to L1 regularisation effect.
Where to Apply:
• After fully connected (dense) layers: p=0.3-0.5. Most effective here.
• After convolutional layers: p=0.1-0.2, or use Spatial Dropout (drop entire feature maps).
• NOT inside BatchNorm layers (they conflict).
• NOT in small datasets where more regularisation needed — use lower p or combine with
other methods.
✎ Example: Without dropout: model learns 'if neuron A fires AND neuron B fires, output cat'. With
dropout: 'if neuron A fires alone, probably cat'. More robust, generalises better.

3.3 Weight Decay (L2 Regularisation)

📌 Weight Decay: Adds a penalty proportional to the squared magnitude of all weights to the loss
function. Discourages large weights, promotes simpler models.
L_total = L_original + (λ/2) Σⱼ wⱼ² = L_original + (λ/2)||w||₂²
• Gradient with weight decay: ∂L_total/∂w = ∂L/∂w + λw.
• Update rule: w ← w - α(∂L/∂w + λw) = w(1 - αλ) - α∂L/∂w.
• The (1-αλ) factor shrinks weights toward zero at each step — hence 'weight decay'.
• λ (regularisation strength): Hyperparameter. Typical: 1e-4 to 1e-2. Cross-validate.
L2 vs L1 Regularisation:
• L2 (weight decay): Penalty = λΣwⱼ². Gradient = λwⱼ. Shrinks weights proportionally. Most
common. Smooth gradient.

Page 6 | SVKM's UPG College | Deep Learning Notes 2024-25

[Link] (IT) Sem IV | Deep Learning | Module 2: Deep Networks + CNNs

•L1 (Lasso): Penalty = λΣ|wⱼ|. Gradient = λ sign(wⱼ). Produces sparse weights (many exactly
zero) — automatic feature selection.
• Elastic Net: Combination: λ₁Σ|wⱼ| + λ₂Σwⱼ². Gets sparsity (L1) + grouping effect (L2).
AdamW (Weight Decay with Adam):
• Standard Adam + L2 regularisation is NOT the same as weight decay in Adam due to
adaptive scaling.
• AdamW (Loshchilov & Hutter 2019): Decouple weight decay from gradient update. Apply
decay directly to weights: w ← w - αλw, separately from Adam's gradient step.
• AdamW is the standard optimiser for large language models (BERT, GPT, LLaMA).

3.4 Batch Normalisation

📌 Batch Normalisation (BN): Normalise the inputs to each layer across the mini-batch — zero
mean, unit variance — then scale and shift with learned parameters γ and β.
μ_B = (1/B) Σᵢ zᵢ [batch mean]
σ²_B = (1/B) Σᵢ (zᵢ - μ_B)² [batch variance]
z̃ᵢ = (zᵢ - μ_B) / √(σ²_B + ε) [normalise, ε=1e-5 for stability]
yᵢ = γ z̃ᵢ + β [scale and shift with learnable γ, β]
• γ (scale) and β (shift): Learnable parameters that restore representational power —
network can 'undo' normalisation if needed.
• At inference: Use running mean and variance (accumulated during training via exponential
moving average), not mini-batch statistics.
Why BatchNorm Works:
• Reduces internal covariate shift: Stabilises distribution of layer inputs, allowing higher
learning rates.
• Gradient flow: Keeps activations in a non-saturating region — reduces vanishing gradient.
• Acts as regulariser: Adds noise (batch statistics vary per batch) — reduces need for
dropout.
• Allows higher learning rates → faster training.
Variants:
• Layer Normalisation (LN): Normalise across features for one sample. Used in Transformers
(no dependence on batch size). LN works with batch size = 1.
• Instance Normalisation (IN): Normalise per channel per sample. Used in style transfer.
• Group Normalisation (GN): Normalise within groups of channels. Works for small batches.

3.5 Early Stopping

📌 Early Stopping: Monitor validation loss during training. Stop when validation loss starts
increasing (even if training loss continues to decrease). Save the checkpoint with the best
validation performance.
• Most conceptually simple regulariser. Stops training before overfitting begins.
• Patience: Number of epochs to wait after validation loss stops improving before stopping.
Typical: 5-20 epochs.
• Restore best weights: Don't use final weights — restore weights from the epoch with lowest
validation loss.
• Equivalent to L2 regularisation under some conditions (via gradient descent trajectory
analysis).

3.6 Data Augmentation

📌 Data Augmentation: Artificially expand the training dataset by applying label-preserving
transformations to existing examples. The most effective regulariser when data is limited.
Standard Image Augmentations:

Page 7 | SVKM's UPG College | Deep Learning Notes 2024-25

[Link] (IT) Sem IV | Deep Learning | Module 2: Deep Networks + CNNs

•Geometric: Random horizontal flip, random crop, random rotation (±15°), random zoom,
translation.
• Colour: Random brightness, contrast, saturation, hue jitter. Grayscale conversion.
• Noise: Gaussian noise, random erasing (CutOut — random rectangles set to zero/mean).
Advanced Augmentations:
• MixUp: Blend two images and their labels: x̃ = λxᵢ + (1-λ)xⱼ, ỹ = λyᵢ + (1-λ)yⱼ. λ~Beta(α,α).
• CutMix: Cut a patch from one image and paste into another. Labels mixed proportionally to
area.
• RandAugment: Randomly sample from a library of augmentations (autocontrast, equalize,
rotate, sharpness...) with magnitude M and N operations.
• AutoAugment: Learn optimal augmentation policy using reinforcement learning.

3.7 Other Regularisation Techniques

Label Smoothing:
ỹᵢ = (1-ε)·yᵢ + ε/K [smooth one-hot labels]
• Instead of hard 0/1 targets, use soft targets: correct class gets (1-ε) + ε/K ≈ 0.9, others get
ε/K ≈ 0.01 (K=10, ε=0.1).
• Prevents model from becoming overconfident. Improves calibration and generalisation.
• Used in: image classification (Inception, ViT), machine translation.
Max-Norm Constraint:
• Constrain ||wⱼ||₂ ≤ c for each neuron j. Project weights to sphere of radius c after each
update.
• More stable than weight decay — doesn't shrink weights but caps maximum size.
Gradient Clipping:
if ||∇L|| > threshold: ∇L ← ∇L × (threshold / ||∇L||)
• Clip gradient norm to maximum value. Essential for training RNNs and LSTMs. Prevents
exploding gradients.

Regularisation Method Best Used When

Dropout (p=0.5) Fully connected layers, large models with
enough data
Weight Decay (λ=1e-4) Almost always. Default regulariser, add to all
layers
Batch Normalisation Deep networks, CNNs. Also speeds training
significantly
Early Stopping Always monitor validation loss. Free
regularisation
Data Augmentation Limited labeled data, image/text tasks
Label Smoothing When model overconfident on training data.
Classification
Gradient Clipping RNNs, LSTMs, unstable training, exploding
gradients

💡 EXAM TIP: Regularisation = at least 5 marks. Know: dropout (formula, why it works,
inverted dropout), weight decay (formula, update rule, AdamW), BatchNorm (4 formulas, γ/β,

Page 8 | SVKM's UPG College | Deep Learning Notes 2024-25

[Link] (IT) Sem IV | Deep Learning | Module 2: Deep Networks + CNNs

LN for Transformers), early stopping (patience), data augmentation (MixUp, CutMix). Bias-
variance tradeoff.

4. Optimisation Algorithms for Training Deep Models

The goal of optimisation in deep learning is to find model parameters θ that minimise the loss
function L(θ). This section covers gradient-based optimisation algorithms from SGD to modern
adaptive methods.

4.1 Stochastic Gradient Descent (SGD)

📌 Stochastic Gradient Descent: Approximates the true gradient using a random mini-batch of B
samples. Applies the gradient update after each mini-batch. Standard algorithm for deep learning
training.
g ← (1/B) Σᵢ∈B ∇θ L(f(xᵢ;θ), yᵢ) [mini-batch gradient estimate]
θ ← θ - α g [parameter update]
•α: Learning rate. Most critical hyperparameter. Too high → diverge. Too low → slow.
•Mini-batch size B: Typically 32-256. Larger B → more accurate gradient, less noisy, but
less regularisation benefit.
SGD Challenges:
• Same learning rate for ALL parameters — suboptimal for parameters with different gradient
magnitudes.
• Slow convergence in narrow, elongated loss valleys (zig-zag behaviour).
• Sensitive to learning rate choice — requires careful tuning.
• Can get stuck in saddle points (gradient ≈ 0 but not a minimum).

4.2 SGD with Momentum

📌 Momentum: Accumulates a velocity vector in directions of persistent gradient. Accelerates
learning in consistent directions, dampens oscillations.
v ← β·v - α·g [update velocity: decay old + add new gradient]
θ ← θ + v [update parameters using velocity]
• β: Momentum coefficient. Typically 0.9 — retains 90% of previous velocity.
• Effect: Like a ball rolling downhill — builds speed in consistent gradient directions, smooths
out noise.
• Nesterov Momentum (NAG): Look-ahead before computing gradient: g = ∇L(θ + βv)
instead of ∇L(θ). Slightly better convergence guarantees.
✎ Example: Without momentum: gradient points left, then right (oscillating in valley). With
momentum: leftward and rightward cancel, downward accumulates → faster descent along valley
floor.

4.3 AdaGrad (Adaptive Gradient Algorithm)

📌 AdaGrad: Adapts the learning rate per parameter — parameters with historically large
gradients get smaller LR; parameters with small gradients get larger LR.
G ← G + g² [accumulate squared gradients element-wise]
θ ← θ - (α / √(G+ε)) · g
• ε = 10⁻⁸: Small constant prevents division by zero.

Page 9 | SVKM's UPG College | Deep Learning Notes 2024-25

[Link] (IT) Sem IV | Deep Learning | Module 2: Deep Networks + CNNs

• Problem: G only grows (never decays). Learning rate decays to near-zero and training
stalls.
• Good for sparse gradients (NLP tasks with sparse word embeddings). Bad for dense tasks.

4.4 RMSprop
📌 RMSprop: Fixes AdaGrad's diminishing LR by using exponential moving average of squared
gradients instead of accumulation.
v ← ρ·v + (1-ρ)·g² [exponential moving avg of squared gradients,
ρ≈0.9]
θ ← θ - (α / √(v+ε)) · g
• v: Running estimate of gradient magnitude. Forgets old gradients (via ρ).
• Effectively normalises each gradient by its recent magnitude — adaptive LR.
• Good for non-stationary objectives: RNN training, reinforcement learning.

4.5 Adam (Adaptive Moment Estimation) — The Standard

📌 Adam: Combines momentum (1st moment) with RMSprop (2nd moment). Includes bias
correction. Most widely used optimiser in deep learning.
m ← β₁·m + (1-β₁)·g [1st moment: mean of gradients]
v ← β₂·v + (1-β₂)·g² [2nd moment: uncentred variance of
gradients]
m̂ = m/(1-β₁ᵗ), v̂ = v/(1-β₂ᵗ) [bias correction for first t steps]
θ ← θ - α · m̂ / (√v̂ + ε)
•Default hyperparameters: β₁=0.9, β₂=0.999, ε=10⁻⁸, α=10⁻³.
•Why bias correction? At t=1: m = (1-β₁)g ≈ 0.1g — underestimates gradient. m̂ = 0.1g/(1-
0.9¹) = g. Corrects initial bias.
• Effective LR per parameter: α/√v̂ . Parameters with large gradient history → small effective
LR.
Adam Variants:
• AdamW: Decouple weight decay from gradient update. w ← w(1-αλ) - α·m̂ /√v̂ . Better
generalisation. Standard for LLMs.
• NAdam: Adam + Nesterov momentum. Look-ahead for better convergence.
• AdaBelief: Use (g-m)² instead of g² for 2nd moment — adapts to curvature of loss
landscape. Better for very deep networks.
• Adan: Uses gradient differences for better momentum. State-of-the-art for vision models.

4.6 Learning Rate Schedules

The learning rate should decrease over training — large LR initially for fast progress, small LR later
for fine-tuning near minimum.
Step Decay:
α(t) = α₀ × γ^⌊t/k⌋ [multiply by γ<1 every k epochs]
• Common: Multiply by 0.1 every 30 epochs. Simple and effective.
Exponential Decay:
α(t) = α₀ × e^(-λt) [smooth continuous decay]
Cosine Annealing:
α(t) = α_min + ½(α_max - α_min)(1 + cos(πt/T))
• Smoothly decreases from α_max to α_min following cosine curve. Very popular.

Page 10 | SVKM's UPG College | Deep Learning Notes 2024-25

[Link] (IT) Sem IV | Deep Learning | Module 2: Deep Networks + CNNs

• Cosine with warm restarts (SGDR): Periodically reset LR to α_max — can escape local
minima.
Linear Warmup:
α(t) = α_target × t/T_warmup for t < T_warmup [gradually increase
LR]
• Prevents instability at the start of training when gradients are large and noisy.
• Standard in Transformer training: warmup for 4,000-10,000 steps, then cosine decay.
Cyclical Learning Rate:
• Oscillate between LR_min and LR_max in cycles. Can escape local minima by periodically
using high LR.
1cycle Policy ([Link]):
• Warmup → peak → cosine anneal to very small LR in one cycle. Super-convergence —
trains faster than step decay.

4.7 Comparison of Optimisers

Optimiser Formula Summary Best Use Case

SGD θ ← θ - α∇L Fine-tuned CV models
(ResNet). Generalises better
than Adam sometimes
SGD + Momentum v ← βv - α∇L; θ ← θ+v Most CV training. β=0.9.
Faster than SGD alone
AdaGrad G+=g²; θ ← θ-αg/√G Sparse features, NLP with
word counts
RMSprop v ← ρv+(1-ρ)g²; θ ← θ-αg/√v RNNs, RL. Non-stationary
objectives
Adam m,v moments + bias Default for DL. Best all-
correction rounder
AdamW Adam + decoupled weight LLMs (BERT, GPT, LLaMA).
decay Better generalisation
Nadam Adam + Nesterov momentum Slightly better convergence
than Adam

💡 EXAM TIP: Optimisation = guaranteed 10 marks. Know: SGD update rule, Momentum
formula (velocity), Adam full 4-formula derivation (m,v,m̂ ,v̂ ,θ update). Default hyperparams:
β₁=0.9, β₂=0.999, α=0.001. Learning rate schedules: cosine annealing, warmup. AdamW for
LLMs.

Page 11 | SVKM's UPG College | Deep Learning Notes 2024-25

[Link] (IT) Sem IV | Deep Learning | Module 2: Deep Networks + CNNs

PART B — CONVOLUTIONAL NEURAL NETWORKS

(CNNs)

5. Introduction to CNNs
Convolutional Neural Networks (CNNs) are the dominant architecture for processing grid-
structured data — primarily images. They were inspired by the visual cortex and introduced by
LeCun et al. (1989). Their success in the 2012 ImageNet challenge (AlexNet) launched the deep
learning revolution in computer vision.

5.1 Why Not Plain MLPs for Images?

Consider a modest 224×224 RGB image (common input size):
• Total pixels: 224 × 224 × 3 = 150,528 input features.
• MLP with first hidden layer of 4,096 neurons: 150,528 × 4,096 = 617 million parameters —
just for ONE layer!
• Problems with MLPs for images:
– 1. Too many parameters — memory and computation infeasible. Severe overfitting.
– 2. No spatial awareness — MLP treats pixel at (0,0) and pixel at (223,223) as
independent. Loses spatial structure.
– 3. Not translation invariant — if cat shifts 10 pixels right, MLP sees completely different
input.
– 4. Doesn't exploit locality — a pixel's meaning is determined by its neighbourhood
(context).
• CNNs solve all these with three key ideas: local connectivity, weight sharing, and spatial
pooling.

5.2 Key Principles of CNNs

1. Local Connectivity:
• Each neuron connects to only a small local region of the input — the receptive field (e.g.,
3×3 or 5×5 pixels).
• Exploits spatial locality: nearby pixels are more related than distant ones.
2. Weight Sharing (Parameter Sharing):
• The SAME set of weights (filter/kernel) is applied at every spatial position.
• A single 3×3 filter with 9 weights detects the same feature (e.g., horizontal edge)
everywhere in the image.
• Massive parameter reduction: One filter has 3×3×C_in weights regardless of image size.
3. Translation Equivariance:
• If the input shifts, the feature map shifts by the same amount. The filter 'sees' the feature
wherever it appears.
• Combining with pooling → translation invariance (approximate).

Page 12 | SVKM's UPG College | Deep Learning Notes 2024-25

[Link] (IT) Sem IV | Deep Learning | Module 2: Deep Networks + CNNs

6. Convolutional Layers — Deep Dive

📌 Convolution Operation: Slide a learnable filter (kernel) across the input, computing the dot
product between filter weights and the local input patch at each position. Result is a feature map
(activation map).

6.1 2D Convolution — Mathematics

(I * K)[i,j] = Σₘ Σₙ I[i+m, j+n] · K[m,n]
• I: Input image/feature map. K: Kernel/filter. *: Convolution operation.
• At each position (i,j): Multiply filter values with the overlapping input patch element-wise,
sum all products.
• For deep learning (cross-correlation, not true convolution): filter is NOT flipped. PyTorch/TF
use cross-correlation by convention.
✎ Example: Input: [[1,2,3],[4,5,6],[7,8,9]], Kernel: [[1,0],[0,1]]. At position (0,0):
1×1+2×0+4×0+5×1=6. At (0,1): 2×1+3×0+5×0+6×1=8. Output: [[6,8],[12,14]].

6.2 Convolution with Volume (3D Input)

Real images have multiple channels (RGB = 3 channels). A filter must match the input depth:
Filter K ∈ ℝᶠˣᶠˣᶜⁱⁿ (F×F spatial, C_in channels)
Output pixel: z[i,j] = Σc Σm Σn I[i+m, j+n, c] · K[m,n,c] + b
• One filter → ONE output channel (one feature map).
• For C_out output channels: Use C_out different filters. Each produces one feature map.
Output volume: C_out feature maps, each of size H_out × W_out
• Parameters per conv layer: (F×F×C_in + 1) × C_out (+1 for bias per filter).
✎ Example: Conv layer: input 3-channel RGB, 64 filters of 3×3. Params = (3×3×3 + 1) × 64 = 28 ×
64 = 1,792. Much less than MLP!

6.3 Output Size Formula

H_out = ⌊(H_in + 2P - F) / S⌋ + 1
W_out = ⌊(W_in + 2P - F) / S⌋ + 1
• H_in, W_in: Input height and width.
• F: Filter size (square filter assumed).
• P: Padding — zeros added around border of input.
• S: Stride — step size of filter movement.
Padding types:
• Valid padding (P=0): No padding. Output smaller than input. H_out = ⌊(H_in-F)/S ⌋+1.
• Same padding: P=(F-1)/2 (for S=1). Output same size as input. Most common default.
Stride:
• S=1: Filter moves one pixel at a time. Standard.
• S=2: Filter skips every other position. Output size halved. Alternative to pooling for
downsampling.
✎ Example: Input: 32×32, Filter: 5×5, P=0, S=1. H_out = (32+0-5)/1+1 = 28. Output: 28×28. ✓ Like
MNIST LeNet.
✎ Example: Input: 224×224, Filter: 3×3, P=1, S=1. H_out = (224+2-3)/1+1 = 224. Same size!
✎ Example: Input: 224×224, Filter: 3×3, P=0, S=2. H_out = (224-3)/2+1 = 112. Halved!

Page 13 | SVKM's UPG College | Deep Learning Notes 2024-25

[Link] (IT) Sem IV | Deep Learning | Module 2: Deep Networks + CNNs

6.4 Receptive Field

📌 Receptive Field: The region of the input image that affects a particular neuron's activation.
Deeper neurons have larger receptive fields.
• Single conv layer with 3×3 filter: receptive field = 3×3.
• Two 3×3 conv layers: receptive field = 5×5. Three: 7×7.
• Three 3×3 layers use fewer params than one 7×7 layer AND have more non-linearities:
3×(3²×C) = 27C < 7²×C = 49C. This is why modern CNNs stack small filters.
• Dilated/Atrous convolution: Introduce gaps in filter to expand receptive field without losing
resolution. Used in segmentation (DeepLab).
Dilated conv: z[i,j] = Σm Σn I[i+d·m, j+d·n] · K[m,n] where d=dilation
rate

6.5 1×1 Convolution

• Filter size F=1. Acts as a learned linear combination across channels at each spatial
position.
• Does NOT look at spatial context (window size=1). Only mixes channels.
• Used for: Channel dimension reduction (bottleneck) — reduce C_in to smaller C_out before
expensive 3×3 conv. Network-in-Network (NiN), Inception modules, ResNet bottleneck.
✎ Example: Input: 256 channels → 1×1 conv with 64 filters → 64 channels → 3×3 conv with 64
filters (cheaper) → 1×1 with 256 filters → 256 channels. This is a ResNet bottleneck block.

💡 EXAM TIP: Convolutional layer: Must know output size formula. Show for any given
input/filter/padding/stride. Parameters formula: (F²×C_in+1)×C_out. 1×1 conv for channel
reduction. Receptive field grows with depth. Stride=2 replaces pooling.

7. Pooling Layers
📌 Pooling Layer: Down-samples spatial dimensions (H, W) of feature maps by summarising
patches with a single value. Reduces computation, provides translation invariance, controls
overfitting.

7.1 Max Pooling

MaxPool[i,j] = max over F×F window centered at (i·S, j·S)
• Takes the maximum value in each F×F window. Retains most prominent feature activation.
• Most common: 2×2 max pool with stride S=2 → halves both H and W → reduces feature
map to 1/4 size.
• No learnable parameters! Pure downsampling.
• Biological analogy: Similar to complex cells in visual cortex responding to the most
prominent stimulus in a region.
✎ Example: Input [[1,3,2,4],[5,6,1,2],[3,2,4,5],[2,1,3,4]], 2×2 MaxPool, S=2: Top-left: max(1,3,5,6)=6.
Top-right: max(2,4,1,2)=4. Bottom-left: max(3,2,2,1)=3. Bottom-right: max(4,5,3,4)=5. Output: [[6,4],
[3,5]].
• Why max pooling works: If a feature (edge, corner) exists somewhere in the pooling
window, max pooling detects it regardless of exact location → approximate translation
invariance.

Page 14 | SVKM's UPG College | Deep Learning Notes 2024-25

[Link] (IT) Sem IV | Deep Learning | Module 2: Deep Networks + CNNs

7.2 Average Pooling

AvgPool[i,j] = (1/F²) Σ over F×F window
• Computes the mean of each F×F window.
• Smoother than max pooling — doesn't ignore non-maximum values.
• Less common in intermediate layers (max pooling preferred). Used as Global Average
Pooling.

7.3 Global Average Pooling (GAP)

📌 Global Average Pooling: Averages each feature map to a single value. If C channels,
produces C-dim vector regardless of spatial size. Replaces final fully connected layers.
GAP(Fk) = (1/(H×W)) ΣᵢΣⱼ Fk[i,j] for channel k
• Input: C × H × W feature volume → Output: C-dim vector.
• NO parameters. Dramatically reduces parameter count and overfitting.
• Forces each feature map to represent one concept — interpretable.
• Used in: ResNet, MobileNet, EfficientNet (replace fully connected layers).
• Allows variable-size input images (no fixed FC layer requiring fixed input size).
✎ Example: ResNet-50 final layer: C=2048 feature maps, each H×W → GAP → 2048-dim vector →
single FC layer (2048→1000 classes). Only 2048×1000=2M params vs. traditional FC with millions.

7.4 Pooling vs Strided Convolution

Pooling (MaxPool/AvgPool) Strided Convolution (S=2)

No learnable parameters Learnable parameters (filter weights)
Fixed operation (max or average) Learned downsampling operation
Computationally cheap Slightly more expensive
Max pooling: good translation invariance Learned to be task-specific
Traditional approach (pre-2015) Modern approach (ResNet, all-conv nets)
Cannot learn optimal downsampling Can learn optimal feature selection during
downsampling

Modern trend: Replace max pooling with strided convolution (S=2). Networks like ResNet and
modern architectures use Conv(S=2) for downsampling — more expressive.
💡 EXAM TIP: Pooling layers: Know max pooling formula and worked example. GAP = average
each feature map to 1 value → C-dim vector. Why use GAP: fewer params, variable input size.
Difference between max pool (translation invariance) vs strided conv (learnable).

8. CNN Architectures — From LeNet to Modern Networks

8.1 LeNet-5 (1998) — The Pioneer
Yann LeCun's LeNet-5 is the first successful CNN for handwritten digit recognition. Simple by
modern standards but introduced all key CNN concepts.

Page 15 | SVKM's UPG College | Deep Learning Notes 2024-25

[Link] (IT) Sem IV | Deep Learning | Module 2: Deep Networks + CNNs

Input(32×32) → Conv(6@5×5) → AvgPool(2×2) → Conv(16@5×5) → AvgPool(2×2)

→ FC(120) → FC(84) → Output(10)
• ~60,000 parameters. Trained on MNIST (28×28 grayscale digits).
• Introduced: conv → pool → conv → pool → FC pattern. Sigmoid/tanh activations.
• Significance: First practical CNN demonstrating end-to-end learning for visual recognition.

8.2 AlexNet (2012) — The Revolution

Krizhevsky, Sutskever & Hinton's AlexNet won ImageNet 2012 with 15.3% top-5 error (vs 26.2%
second place). This single result launched the deep learning era in computer vision.
Input(224×224×3) → Conv(96@11×11,S=4) → MaxPool → Conv(256@5×5) →
MaxPool
→ Conv(384@3×3) → Conv(384@3×3) → Conv(256@3×3) → MaxPool → FC(4096) →
FC(4096) → FC(1000)
• ~60 million parameters. 5 conv layers + 3 FC layers.
• Key innovations:
– ReLU activations: 6× faster training than tanh/sigmoid.
– Dropout (p=0.5) in FC layers: First use of dropout in CNNs.
– Data augmentation: Random crops, horizontal flips, colour jitter.
– GPU training: Two GTX 580 GPUs with model parallelism.
– Local Response Normalisation (LRN): Lateral inhibition (later replaced by BatchNorm).
• Significance: Proved that deep CNNs trained on GPUs dominate traditional computer
vision. Started the ImageNet race.

8.3 VGGNet (2014) — Simplicity and Depth

Simonyan & Zisserman's VGG showed that network depth with small (3×3) filters is key to
performance.
• Architecture: Stacks of 3×3 convolutions with max pooling, ending in 3 FC layers.
• VGG-16: 13 conv layers + 3 FC = 16 weight layers. ~138M parameters.
• VGG-19: 16 conv layers + 3 FC. Deeper, slightly better.
Key insight:
• Two 3×3 conv layers have the same receptive field as one 5×5 layer but: fewer parameters
(2×3²=18 vs 25) and two non-linearities (more representational power).
• Three 3×3 layers equivalent to one 7×7: params 27 vs 49 (45% fewer).
• Weaknesses: Very slow to train (~2-3 weeks on single GPU). Huge memory footprint. FC
layers dominate parameter count.
• Legacy: VGG-16/19 still widely used as feature extractors for transfer learning. Architecture
is easy to understand and implement.

8.4 GoogLeNet / Inception (2014) — Wider and Deeper

Szegedy et al.'s GoogLeNet won ImageNet 2014 with 6.67% top-5 error. Introduced the Inception
module — a more efficient multi-scale architecture.
📌 Inception Module: Applies 1×1, 3×3, and 5×5 convolutions AND 3×3 max pooling in
PARALLEL on the same input, then concatenates their outputs channel-wise. Captures features at
multiple scales simultaneously.
• Problem with naive inception: 5×5 conv on 256-channel input is very expensive. Solution:
1×1 conv BEFORE expensive convolutions for channel reduction (bottleneck).
• Architecture: 9 stacked inception modules. No FC layers at end (Global Average Pooling).
22 layers deep.
• ~6.8M parameters — 20× fewer than AlexNet, yet much more accurate.

Page 16 | SVKM's UPG College | Deep Learning Notes 2024-25

[Link] (IT) Sem IV | Deep Learning | Module 2: Deep Networks + CNNs

• InceptionV3, V4: Further refinements. Factored convolutions (5×5 → two 3×3, 7×7 → 1×7 +
7×1).
• Significance: Showed that network width (parallel paths) is as important as depth.

8.5 ResNet (2015) — Residual Learning Solves Depth

He et al.'s ResNet won ImageNet 2015 with 3.57% top-5 error — better than human performance
(~5%). Solved the degradation problem: very deep networks were HARDER to train than shallower
ones (not just overfitting).
📌 Residual Connection (Skip Connection): Instead of learning H(x) directly, learn the residual
F(x) = H(x) - x. Output: y = F(x) + x. The identity shortcut allows gradients to flow directly through
the network.
y = F(x, {Wᵢ}) + x [residual block output]
F(x) = W₂·ReLU(W₁·x) [typically 2-3 layer transformation]
Why Residual Connections Work:
• Gradient highway: Gradient flows directly through + x path without passing through layer
weights. Vanishing gradient largely eliminated.
• Identity shortcut: If the optimal function is close to identity (H(x) ≈ x), easier to learn F(x) ≈ 0
than H(x) ≈ x.
• Ensemble effect: ResNet can be interpreted as an ensemble of many sub-networks of
different depths.
ResNet Variants:
• ResNet-18, 34: Basic residual blocks (two 3×3 convs).
• ResNet-50, 101, 152: Bottleneck blocks (1×1→3×3→1×1 conv). More efficient for deeper
networks.
• ResNet-50 architecture: 5 stages, each stage with multiple bottleneck blocks, Global
Average Pooling, single FC.
✎ Example: ResNet bottleneck block: 256-dim input → 1×1(64 ch, reduce) → 3×3(64 ch) →
1×1(256 ch, expand) + identity shortcut → 256-dim output. Only 3×3 conv on 64 channels (not 256)
→ efficient.
• ResNet-152 trains easily. Plain 152-layer networks (without residuals) fail to converge!

8.6 Modern CNN Architectures

DenseNet (2017):
• Each layer receives feature maps from ALL previous layers: xˡ = H([x₀, x₁,...,xˡ ⁻¹]).
• Feature reuse, strong gradient flow, parameter efficient.
• Better than ResNet on some tasks with fewer parameters.
MobileNet (2017) — Efficient CNNs:
• Depthwise separable convolutions: Decompose standard conv into depthwise (per-channel
3×3) + pointwise (1×1). 8-9× fewer operations.
• MobileNetV2: Inverted residuals + linear bottlenecks. Used in mobile deployment.
• MobileNetV3: Neural architecture search (NAS) + hard-swish activation.
EfficientNet (2019):
• Compound scaling: Simultaneously scale depth, width, and resolution using compound
coefficient φ.
• EfficientNet-B0 to B7. B7 achieves 84.4% top-1 on ImageNet with 66M params.
• State-of-the-art accuracy per parameter count.
Vision Transformer (ViT, 2020):
• Splits image into 16×16 patches. Linearly embed patches as tokens. Feed to standard
Transformer encoder.

Page 17 | SVKM's UPG College | Deep Learning Notes 2024-25

[Link] (IT) Sem IV | Deep Learning | Module 2: Deep Networks + CNNs

• No convolutions. Pure attention-based processing. Outperforms CNNs at scale (large

datasets).
• Hybrid architectures (ConvNeXt, Swin Transformer) combine conv inductive biases with
attention.

Architecture Year Key Innovation

LeNet-5 1998 First successful CNN.
Conv+Pool pattern
AlexNet 2012 Deep CNN on GPU. ReLU +
Dropout. Started DL era
VGGNet 2014 Depth via stacked 3×3 convs.
Simple, effective
GoogLeNet 2014 Inception modules. Multi-
scale parallel convs. GAP
ResNet 2015 Residual/skip connections.
Train 100-1000 layers
DenseNet 2017 Dense connections. Feature
reuse
MobileNet 2017 Depthwise separable convs.
Mobile deployment
EfficientNet 2019 Compound scaling
depth+width+resolution
ViT 2020 Pure Transformer for images.
Patches as tokens
Swin Transformer 2021 Hierarchical Vision
Transformer. SOTA on
multiple tasks

💡 EXAM TIP: CNN architectures = guaranteed 5-10 marks. Know LeNet (pioneer), AlexNet
(ReLU+GPU revolution), VGGNet (3×3 stacking rationale), ResNet (residual connection formula
+ why it works, bottleneck block). For each: year, key innovation, significance. EfficientNet for
modern context.

9. CNN Applications
CNNs are the backbone of computer vision — the field of enabling computers to 'see' and
understand images and video.

9.1 Image Classification

The foundational CV task: assign one class label to an entire image.
• Benchmark: ImageNet Large Scale Visual Recognition Challenge (ILSVRC) — 1.2M
images, 1000 classes.
• Human performance: ~5.1% top-5 error. ResNet (2015): 3.57% — super-human!
• Applications: Medical image diagnosis (CXR, CT, MRI classification), quality control in
manufacturing, content moderation, agricultural disease detection.

Page 18 | SVKM's UPG College | Deep Learning Notes 2024-25

[Link] (IT) Sem IV | Deep Learning | Module 2: Deep Networks + CNNs

• Transfer learning: Pre-train on ImageNet, fine-tune on domain-specific dataset. State of the

art for most classification tasks with limited data.
✎ Example: CovidNet: CNN trained to classify chest X-rays as Normal/COVID-19/Pneumonia. Fine-
tuned from ImageNet-pretrained VGG/ResNet. 93% sensitivity achieved.

9.2 Object Detection

Detect and localise multiple objects in an image — output: bounding boxes + class labels for each
object.
• R-CNN (2014): Selective search → ~2000 region proposals → CNN feature extraction per
region → SVM classification. Accurate but very slow (~47s/image).
• Fast R-CNN: Single CNN pass on full image → ROI Pooling extracts fixed-size features per
proposal. 200× faster than R-CNN.
• Faster R-CNN: Region Proposal Network (RPN) — fully convolutional network that
proposes object regions from CNN features. Near real-time on GPU.
• YOLO (You Only Look Once): Single CNN predicts all bounding boxes + classes in one
pass. Real-time (30+ FPS). YOLOv8 (2023) is state of the art for speed-accuracy trade-off.
• Applications: Autonomous vehicles (pedestrian/vehicle/sign detection), surveillance, retail
analytics, satellite imagery analysis.

9.3 Semantic Segmentation

Classify every pixel in the image with a class label — pixel-level understanding.
• FCN (Fully Convolutional Network, 2015): Replace FC layers with 1×1 convolutions → can
take variable-sized input. Upsampling via transposed convolutions.
• U-Net (2015): Encoder-decoder with skip connections. Encoder: conv+pool
(downsampling). Decoder: transposed conv (upsampling). Skip connections from encoder
to decoder. Excellent for medical image segmentation (small datasets).
• DeepLab: Dilated/atrous convolutions to increase receptive field without reducing
resolution. ASPP (Atrous Spatial Pyramid Pooling) for multi-scale context.
• SegFormer: Transformer-based segmentation. Hierarchical encoder + lightweight MLP
decoder.
• Applications: Autonomous driving (road, pedestrian, vehicle segmentation), medical
imaging (organ/tumour delineation), satellite land use mapping.

9.4 Face Recognition

• DeepFace (Facebook, 2014): 9-layer CNN achieving 97.35% on LFW (Labeled Faces in
the Wild). Uses 3D face alignment.
• FaceNet (Google, 2015): Triplet loss training. Maps face images to 128-dim embedding.
Same person → close embeddings. Different person → far apart. Used in Google Photos.
• ArcFace: Additive Angular Margin Loss. State-of-the-art face verification. Used in law
enforcement, border control, mobile unlock.

9.5 Medical Image Analysis

• Diabetic retinopathy detection: Google's CNN matches ophthalmologist performance (95%
sensitivity) on retinal fundus images.
• Skin cancer classification: Stanford study — CNN matches dermatologist on 130,000 skin
lesion images.
• Radiology: Chest X-ray pathology detection (ChexNet, CheXpert). CT/MRI tumour
segmentation (nnU-Net).
• Pathology: Whole slide image (WSI) analysis — detect cancer in gigapixel histology slides.

Page 19 | SVKM's UPG College | Deep Learning Notes 2024-25

[Link] (IT) Sem IV | Deep Learning | Module 2: Deep Networks + CNNs

9.6 Other CNN Applications

• Style transfer: Gram matrix (feature map correlations) captures artistic style. Optimise
content image to match style of a painting (Gatys 2015).
• Image generation: CNN Discriminator + Transposed CNN Generator in GANs. DCGAN
generates realistic faces.
• Video understanding: 3D convolutions (C3D, I3D) — extend 2D conv to temporal dimension
for action recognition.
• Super-resolution: SRCNN — upscale low-resolution images. Used in satellite imagery
enhancement, video streaming.
• Text recognition (OCR): CNN extracts visual features from text images. Combined with
RNN/Transformer for sequence decoding.
• Drug discovery: CNN on molecular graph representations predicts drug-protein binding
affinity.

Application Area CNN Models Used

Image Classification AlexNet, VGG, ResNet, EfficientNet, ViT
Object Detection Faster R-CNN, YOLO (v3/v5/v8), SSD, DETR
Semantic Segmentation FCN, U-Net, DeepLab, SegFormer
Instance Segmentation Mask R-CNN (Faster R-CNN + mask head)
Face Recognition DeepFace, FaceNet, ArcFace
Medical Imaging U-Net (segmentation), DenseNet, nnU-Net
Image Generation DCGAN, StyleGAN (discriminator is CNN)
Video Analysis 3D-CNN, I3D, SlowFast, VideoMAE

💡 EXAM TIP: CNN applications: List at least 5 application domains with a specific model for
each. Know: image classification (ResNet on ImageNet), object detection (YOLO vs R-CNN
speed trade-off), segmentation (U-Net for medical), face recognition (FaceNet triplet loss).

MODULE 2 — COMPLETE QUICK REVISION SUMMARY

DEEP NETWORKS — Key Formulas:
• Forward pass: aˡ = g(Wˡaˡ⁻¹ + bˡ) for each layer l.
• Backprop: δᴸ = ŷ-y (output). δˡ = (Wˡ⁺¹)ᵀδˡ⁺¹ ⊙ g'(zˡ). ∂L/∂Wˡ = δˡ(aˡ ⁻¹)ᵀ.
• Vanishing gradient: sigmoid derivative max=0.25 → (0.25)ᴸ → 0. Fix: ReLU, residual
connections.
• Activations: Sigmoid (binary output), Tanh (RNN), ReLU (default hidden), GELU
(Transformers), Softmax (multi-class output).
• Dropout: h̃ = m⊙h/(1-p) training. h̃ = h inference. p=0.5 for FC, 0.1-0.2 for Conv.
• Weight decay: L_total = L + (λ/2)||w||₂². Update: w ← w(1-αλ) - α∂L/∂w.
• BatchNorm: Normalise → z̃ =(z-μ)/√(σ²+ε) → scale+shift: y=γz̃ +β. γ,β learnable.
• SGD: θ←θ-αg. Momentum: v←βv-αg, θ←θ+v. β=0.9.
• Adam: m=β₁m+(1-β₁)g, v=β₂v+(1-β₂)g², m̂ =m/(1-β₁ᵗ), v̂ =v/(1-β₂ᵗ), θ←θ-αm̂ /√(v̂ +ε). β₁=0.9,
β₂=0.999, α=0.001.

Page 20 | SVKM's UPG College | Deep Learning Notes 2024-25

[Link] (IT) Sem IV | Deep Learning | Module 2: Deep Networks + CNNs

CNNs — Key Facts:

• Output size: H_out = ⌊(H_in+2P-F)/S⌋+1. Params: (F²×C_in+1)×C_out.
• MaxPool 2×2,S=2: halves H and W (quarters area). No learnable params.
• GAP: Average each feature map → C-dim vector. Replaces FC layers.
• LeNet(1998): first CNN. AlexNet(2012): deep learning era, ReLU+GPU+Dropout.
• VGGNet: 3×3 stacking (2 conv→5×5 RF with fewer params). ResNet: y=F(x)+x, gradient
highway.
• ResNet bottleneck: 1×1→3×3→1×1 + skip. EfficientNet: compound scaling.
• Applications: Classification (ResNet), Detection (YOLO/Faster RCNN), Segmentation (U-
Net), Medical AI.

Topic Likely Exam Question Key Answer Points

MLP & Forward Pass 5-10 marks — Explain MLP Layer equations aˡ=g(Wˡaˡ⁻¹+bˡ),
with example backprop chain rule, parameter
count
Activation Functions 5 marks — Compare 4 Sigmoid/Tanh/ReLU/GELU
activation functions formulas, ranges, uses,
vanishing gradient issue
Dropout 5 marks — Explain dropout Formula (training vs inference),
regularisation inverted dropout, why it works
(ensemble)
Batch Normalisation 5 marks — Explain BatchNorm 4 formulas (μ,σ²,z̃ ,y=γz̃ +β), γ/β
purpose, LN for Transformers
Adam Optimiser 5-10 marks — Explain Adam m,v,m̂ ,v̂ , update rule, default
with formulas hyperparams, AdamW for LLMs
Convolutional Layer 5 marks — Explain conv with Convolution formula, output
output size size formula, parameter count,
receptive field
Pooling Layers 5 marks — Explain max pool MaxPool formula + example,
and GAP GAP (average per channel), no
parameters
CNN Architectures 5-10 marks — Compare Year, innovation, key formula
LeNet/AlexNet/ResNet (ResNet: y=F(x)+x),
significance
CNN Applications 5 marks — Applications of CNN 5 areas: classification,
detection, segmentation, face
recognition, medical

— END OF DEEP LEARNING MODULE 2 NOTES —

15 Lectures · 9 Major Topics · Part A: Deep Networks (4 topics) + Part B: CNNs (5 topics)
Good luck with your Deep Learning examination! 🎓

Page 21 | SVKM's UPG College | Deep Learning Notes 2024-25