AD-601 Deep Learning

Exam-Ready Visual Notes — Units I, II, III, IV

[Link] VI Sem, AI & Data Science, RGPV Bhopal | Based on syllabus + PYQs (2023, May 2024, June 2025)

Star rating = PYQ importance DON'T CONFUSE = common mix-up MEMORY TRICK DRAW THIS IN EXAM
 UNIT I — Introduction to Deep Learning

1. Introduction to Deep Learning ⭐⭐⭐⭐⭐

PYQ: 2023 Q1a (8), 2023 Q6a "Role of DL in Data Science" (9), 2023 Q8d "AI vs DL" (14), 2024 Q1b "ML vs AI" (5), 2024 Q6a "Types of DL" (8),
2025 Q6b "AI vs ML vs DL"

Definition

Deep Learning is a subfield of Machine Learning that uses artificial neural networks with many layers to automatically learn hierarchical
feature representations directly from raw data, without manual feature engineering.

Keywords

Neural Network, Layers, Weights, Bias, Activation Function, Forward Propagation, Loss Function, Backpropagation,
Hierarchical Feature Learning, Representation Learning

Why do we need it?

Traditional ML needs a human to hand-craft features (e.g. edges, colors for image tasks). This does not scale to complex data like
images, speech, text.
Deep Learning learns the features itself, layer by layer — early layers learn simple patterns (edges), deeper layers learn complex
patterns (faces, objects).
Works very well when there is a large amount of data and high computing power (GPUs).

AI vs ML vs DL

DRAW THIS IN EXAM — nested circles diagram

______________________________________
| Artificial Intelligence (AI) |
| ___________________________ |
| | Machine Learning (ML) | |
| | ______________________ | |
| | | Deep Learning (DL) | | |
| | |______________________| | |
| |___________________________| |
|______________________________________|

Feature AI ML DL

Definition Machines mimicking human intelligence Systems that learn patterns from data ML using multi-layer neural networks

Feature extraction N/A Manual (human designed) Automatic (learned)

Data need Varies Works on smaller data Needs large data

Hardware Varies CPU sufficient Needs GPU/TPU

Example Chess-playing computer Spam filter (Naive Bayes, SVM) Self-driving car vision, ChatGPT

Visualization — Neural Network Architecture

DRAW THIS IN EXAM

Input Layer Hidden Layer 1 Hidden Layer 2 Output Layer

x1 o ------\ o ---------\ o -----------\
x2 o -------> o ----------->o --------------> o -----> y (output)
x3 o ------/ o ---------/ o -----------/
(weights w, bias b, activation f at each neuron)

Input layer: receives raw features (x1, x2, x3 ...)

Hidden layers: do the actual "deep" computation; more hidden layers = more "depth"
Output layer: gives final prediction (class label / number)
Weights (w): strength/importance of each connection
Bias (b): shifts the activation, allows flexibility
Activation function: introduces non-linearity (Sigmoid, ReLU, tanh) so the network can learn complex, non-linear patterns

Working (Step by Step)

Raw Data (Input)

|
v
Forward Propagation (data flows layer to layer, weighted sums + activation)
|
v
Prediction (output y_hat)
 |
v
Loss Function (compares y_hat with actual y)
|
v
Backpropagation (error sent backward, gradients computed)
|
v
Weight Update (Gradient Descent)
|
v
Repeat for many iterations (epochs) until loss is minimum

Mathematics

z = (w1*x1 + w2*x2 + ... + wn*xn) + b and y = f(z)

where w=weights, x=inputs, b=bias, f=activation function, z=weighted sum (pre-activation), y=neuron output.

Advantages
Automatic feature learning — no manual feature engineering
Handles unstructured data (images, audio, text) very well
Accuracy keeps improving as data grows (scales well)

Limitations
Needs huge labeled data and heavy compute (GPU)
"Black box" — hard to interpret why a decision was made
Prone to overfitting on small datasets
Long training time

Applications

Image processing (object detection), Speech recognition, Natural Language Processing (chatbots, translation), Self-driving cars, Medical
diagnosis, Recommendation systems.

EXAM-READY ANSWER (7–8 marks): "What is Deep Learning? / Significance of depth in DL"
1. Start with the 2–4 line definition above.
2. Draw the AI ⊃ ML ⊃ DL nested-circle diagram.
3. Draw the Input→Hidden→Hidden→Output neural network diagram, label weights/bias/activation.
4. Explain forward propagation → loss → backpropagation → weight update as a cycle.
5. Write the equations z = Σwx + b, y = f(z) and explain each symbol.
6. State 2 advantages, 2 limitations, and 2–3 applications.
7. Conclude: "Deep Learning's power comes from depth — each layer builds increasingly abstract, hierarchical features, which is
why increasing layers (depth) generally improves performance until overfitting/vanishing-gradient issues appear."

QUICK REVISION: DL = ML with multi-layer neural nets that learn features automatically. AI ⊃ ML ⊃ DL. Flow: Input → Forward Prop →
Prediction → Loss → Backprop → Weight Update. Formula: z=Σwx+b, y=f(z).

2. Biological Neuron ⭐⭐⭐⭐

PYQ: 2023 Q1a "Structure and units of Biological Neuron" (8), 2025 Q1b "Compare biological vs artificial neural networks", 2024 Q1a (McCulloch-
Pitts references neuron)

Definition

A biological neuron is the basic unit of the human brain and nervous system that receives, processes, and transmits information using
electrical and chemical signals.

Keywords

Dendrites, Cell Body (Soma), Axon, Synapse, Nucleus, Firing, Threshold

Visualization

DRAW THIS IN EXAM

Dendrites ---> Cell Body (Soma) ---> Axon ---> Synapse ---> (next neuron's dendrites)
(receive (processes/sums (carries (transmits signal
signals) the signals) signal) to next neuron)

Working
 Dendrites receive electrical/chemical signals from neighbouring neurons.
Cell body (Soma) sums up (integrates) all incoming signals.
If the summed signal crosses a threshold, the neuron "fires" — generates an electrical pulse.
The Axon carries this pulse away from the cell body.
The Synapse is the junction where the signal passes to the next neuron (with some strength — this inspired the idea of "weight").

Biological Neuron → Artificial Neuron Mapping

MEMORY TRICK: "D-S-F-A" = Dendrite(Input) → Soma(Sum) → Firing(Activation) → Axon(Output)

Dendrites ----> Inputs (x1, x2, x3...)

Synapses ----> Weights (w1, w2, w3...)
Cell Body ----> Summation (z = Σwx + b)
Firing ----> Activation Function f(z)
Axon ----> Output (y)

Feature Biological Neuron Artificial Neuron

Basic unit Cell in brain/nervous system Mathematical/software unit

Input Dendrites x1, x2, ... xn

Connection strength Synapse Weight (w)

Processing Soma sums signals Weighted summation z=Σwx+b

Firing rule Fires if signal > threshold Activation function f(z)

Output path Axon Output value y

Speed Slow (ms) but massively parallel Fast (ns) per operation, needs many units for same power

Learning Synaptic plasticity (biological learning) Weight update via backpropagation/gradient descent

Scale ~86 billion neurons in human brain Millions to billions of parameters in large models

Limitations of current DL models vs human brain

Brain learns from very few examples; DL models need thousands/millions of labeled examples.
Brain is extremely energy efficient (~20W); DL training needs huge power (GPU clusters).
Brain generalizes across tasks easily; DL models are usually narrow/task-specific.
Brain has continual/lifelong learning; DL models suffer "catastrophic forgetting" when learning new tasks.

EXAM-READY ANSWER (7–8 marks): "Compare biological and artificial neural networks"
1. Define biological neuron (2 lines) and artificial neuron (2 lines).
2. Draw Dendrites→Soma→Axon→Synapse diagram AND the artificial neuron diagram (x→w→Σ→f→y).
3. Give the mapping table (Dendrite=Input, Synapse=Weight, Soma=Summation, Firing=Activation, Axon=Output).
4. Give the full comparison table above.
5. Conclude with 3–4 limitations of current DL models vs the human brain (data efficiency, energy, generalization, continual
learning).

QUICK REVISION: Dendrite=Input, Synapse=Weight, Soma=Sum, Firing=Activation, Axon=Output. Brain is data/energy efficient and
generalizes; DL models need huge data/compute and are task-specific.

3. Computational Unit / Artificial Neuron ⭐⭐⭐

Foundation topic — appears embedded inside MCP/Perceptron questions every year.

Definition

A computational unit (artificial neuron) is the basic building block of a neural network that takes multiple weighted inputs, sums them with
a bias, and passes the result through an activation function to produce an output.

Keywords

Inputs, Weights, Bias, Summation, Activation Function, Output

Visualization

DRAW THIS IN EXAM

x1 --w1--\
x2 --w2---> [ Summation Σ(wi xi) + b ] ---> [ Activation f(z) ] ---> Output y
x3 --w3--/

Mathematics
z = Σ(wi * xi) + b y = f(z)

xi = i-th input, wi = weight of i-th input (importance), b = bias (shifts decision boundary), z = net input/pre-activation, f = activation
function (Sigmoid/ReLU/tanh/Step), y = final output of the neuron.

Example

If x1=1, x2=0, w1=0.6, w2=0.8, b=-0.5, then z = (0.6*1 + 0.8*0) - 0.5 = 0.1. If activation is step function (fires if z>0), output y=1.

Applications

Basic unit for every neural network — perceptrons, MLPs, CNNs, RNNs are all built by connecting many computational units in layers.

QUICK REVISION: z = Σwx+b, y=f(z). Neuron = weighted sum + activation.

4. McCulloch–Pitts (MCP) Neural Model ⭐⭐⭐⭐⭐

PYQ: 2024 Q1a "Define Neuron. Explain McCulloch-Pitts Neural Model in detail" (9 marks) — directly asked, very high priority.

Definition

The McCulloch–Pitts (MCP) neuron (1943) is the earliest mathematical model of a neuron. It takes binary inputs (0/1), sums excitatory
and inhibitory inputs, and fires (outputs 1) only if the sum meets or exceeds a fixed threshold, and there is no inhibitory input active.

Keywords

Binary inputs, Threshold (θ), Excitatory input, Inhibitory input, Step activation, Fixed weights

Why do we need it?

It was the first attempt to show that a simple neuron-like unit could perform logical operations (AND, OR, NOT) — proving that networks of
such units could, in principle, compute anything a digital computer could.

Visualization

DRAW THIS IN EXAM

Excitatory inputs (x1, x2 ...) ----\

>--- Σ(inputs) ---> compare with threshold θ ---> Output (0 or 1)
Inhibitory inputs (y1, y2 ...) -----/ |
if ANY inhibitory input = 1 => output forced to 0

Architecture / Rule
Inputs are binary (0 or 1) only.
Two types of inputs: Excitatory (normal, positive contribution) and Inhibitory (if even ONE inhibitory input is 1, neuron output is
forced to 0, no matter what).
All excitatory weights are equal (usually taken as 1, fixed, not learned).
Neuron fires (y=1) only if: sum of excitatory inputs ≥ threshold θ AND no inhibitory input is active.

Mathematics

y = 1 if (Σ xi ≥ θ) AND (no inhibitory input is 1)

y = 0 otherwise

Logic Gates using MCP Neuron

DRAW ALL THREE GATE DIAGRAMS IN EXAM

Gate Weights Threshold (θ) Working

AND (x1,x2) w1=1, w2=1 θ=2 Fires only when x1=1 AND x2=1 (sum=2 ≥ 2)

OR (x1,x2) w1=1, w2=1 θ=1 Fires when at least one input is 1 (sum ≥ 1)

NOT (x1) w1 = −1 (inhibitory) θ=0 x1=1 (inhibitory) forces output 0; x1=0 gives output 1

Numerical Example (AND gate)

x1 x2 Sum = x1+x2 Sum ≥ θ(2)? Output y

0 0 0 No 0

0 1 1 No 0

1 0 1 No 0

1 1 2 Yes 1
Limitations
Only works with binary inputs/outputs — cannot handle real-valued data.
Weights and threshold are fixed (hand-set), not learned from data.
Cannot solve non-linearly separable problems like XOR.
All excitatory weights must be equal — cannot give different importance to different inputs.
DON'T CONFUSE: MCP neuron has fixed, non-learnable weights/threshold set by a human. Perceptron has weights that are learned
automatically from training data. This is the single most important difference between the two.

EXAM-READY ANSWER (7–9 marks): "Define Neuron. Explain McCulloch-Pitts Model in detail"
1. Give the general neuron definition (from Topic 3) in 2 lines.
2. Give the MCP definition (2–3 lines).
3. Draw the excitatory/inhibitory input diagram.
4. Write the firing rule: y=1 if Σxi≥θ and no inhibitory input active.
5. Show AND, OR, NOT gate tables with weights and threshold (pick at least AND with the truth table).
6. List 3–4 limitations, especially "cannot learn weights" and "cannot solve XOR".
7. Conclude: "MCP neuron was the foundation of neural computing, but its fixed, binary, non-learning nature led to development of
the trainable Perceptron."

QUICK REVISION: MCP = binary in/out, fixed weights (usually 1), threshold θ, one active inhibitory input ⇒ output 0. Can implement
AND/OR/NOT. Cannot learn, cannot do XOR.

5. Linear Perceptron ⭐⭐⭐⭐

PYQ: 2024 Q2b "Write about Linear and Multilayer Perceptron in detail" (7 marks), 2025 Q2a "What is MLP? overcome limitations of single-layer
perceptrons"

Definition

A linear perceptron is a single-layer neural network that computes a weighted sum of inputs and passes it through a step (threshold)
activation function to classify inputs into one of two classes, and whose weights can be learned from training data.

Keywords

Weighted sum, Threshold activation, Decision boundary, Linear separability, Learnable weights

Visualization

DRAW THIS IN EXAM

x1 --w1--\
x2 --w2---> Σ(wi xi) + b ---> Step Function ---> Output (0 or 1)
x3 --w3--/

Working
Compute z = Σ(wi xi) + b (same as any artificial neuron).
Pass through a step function: output 1 if z ≥ 0, else 0.
Geometrically, the perceptron draws a straight line/hyperplane (decision boundary) that separates two classes.
Works only if data is linearly separable (a single straight line can separate the classes).

MCP Neuron vs Perceptron

Feature MCP Neuron Perceptron

Inputs Binary only Real-valued

Weights Fixed, equal, hand-set Learnable (different, via training)

Threshold Fixed, manually chosen Learned as part of bias

Learning algorithm None Perceptron Learning Rule

Inhibitory concept Yes (hard veto) No — uses signed weights instead

QUICK REVISION: Perceptron = weighted sum + step function, learnable weights, draws a linear decision boundary, works only for
linearly separable data.

6. Perceptron Learning (Algorithm) ⭐⭐⭐⭐⭐

Core algorithm topic — base for MLP/Backprop questions asked every year.
Definition

Perceptron Learning is a supervised algorithm that adjusts the perceptron's weights step by step, based on the error between predicted
and actual output, until the perceptron correctly classifies all (linearly separable) training examples.

Algorithm (Step by Step)

1. Initialize weights w and bias b to small random values (or 0)

2. For each training example (x, target t):
a. Compute output: y = f(Σ wi xi + b)
b. Compute error: e = t - y
c. Update weights: wi(new) = wi(old) + η * e * xi
d. Update bias: b(new) = b(old) + η * e
3. Repeat step 2 for all examples, for many epochs,
until error becomes 0 (or max epochs reached)

Mathematics — Weight Update Rule

w(new) = w(old) + η (t − y) x

η (eta) = learning rate (controls step size), t = target/desired output, y = predicted output, x = input, (t−y) = error.

Small Training Example (AND gate, η=1)

Epoch x1,x2 Target t w1,w2,b (before) y (predicted) Error (t-y) w1,w2,b (after)

1 0,0 0 0,0,0 0 0 0,0,0 (no change)

1 0,1 0 0,0,0 0 0 0,0,0

1 1,0 0 0,0,0 0 0 0,0,0

1 1,1 1 0,0,0 0 1 1,1,1 (weights updated because prediction was wrong)

Training continues over more epochs, refining weights until all 4 AND-gate examples are classified correctly.

Why single-layer Perceptron cannot solve XOR ⭐⭐⭐⭐⭐

DON'T CONFUSE: This is one of the most commonly asked conceptual questions.

XOR output is 1 only when inputs differ (0,1)→1 and (1,0)→1, but (0,0)→0 and (1,1)→0. Plotting these points, no single straight line can
separate the 1-outputs from the 0-outputs — XOR is not linearly separable. Since a single-layer perceptron can only draw one straight
decision boundary, it fundamentally cannot represent XOR. This limitation (highlighted by Minsky & Papert, 1969) led directly to the
development of Multi-Layer Perceptrons (MLPs), which use hidden layers to create non-linear decision boundaries.

x2
1 | (0,1)=1 (1,1)=0
|
0 | (0,0)=0 (1,0)=1
+------------------------ x1
0 1
No single straight line separates the 1's from the 0's!

EXAM-READY ANSWER (7–8 marks): "Perceptron Learning Algorithm"

1. Define Perceptron Learning (2 lines).
2. Draw the perceptron diagram (inputs→weights→sum→step→output).
3. Write the algorithm steps 1–3 exactly as above.
4. Write the weight update rule and explain each symbol (η, t, y, x).
5. Show a mini worked example (like the AND-gate table).
6. Explain the XOR limitation with the plotted diagram, and conclude that this motivated MLPs.

QUICK REVISION: w(new)=w(old)+η(t−y)x. Update only when prediction is wrong. Single-layer perceptron = 1 straight decision
boundary ⇒ cannot solve XOR (not linearly separable) ⇒ need MLP.

7. Feed Forward Neural Network ⭐⭐⭐

PYQ: 2023 Q8c "Feed forward Network" (in choice of 2, 14 marks total), 2025 Q4a "Diff CNN and RNN" (uses FFN as base concept)

Definition

A Feed Forward Neural Network (FFNN) is a neural network in which information moves in only one direction — from input layer, through
hidden layer(s), to the output layer — with no loops or cycles.

Keywords
Unidirectional flow, No cycles/loops, Single-layer FFN, Multi-layer FFN (MLP)

Visualization

DRAW THIS IN EXAM

Input Layer --> Hidden Layer(s) --> Output Layer

(data flows only forward; never goes back to a previous layer)

Why does information move only forward?

Because there are no feedback/recurrent connections — each layer's output becomes the next layer's input, and information is never sent
back to an earlier layer. This is different from RNNs, which do have feedback loops (Unit IV).

Single-layer vs Multi-layer FFN

Feature Single-Layer FFN (Perceptron) Multi-Layer FFN (MLP)

Hidden layers None (only input→output) One or more hidden layers

Can solve XOR? No Yes

Decision boundary Linear (1 straight line) Non-linear (complex boundary)

Training Perceptron Learning Rule Backpropagation

Applications

Basic classification/regression problems, base building block used inside CNNs (fully-connected layers) and many other networks.

QUICK REVISION: FFN = one-directional Input→Hidden→Output, no loops. Single-layer = perceptron (linear only). Multi-layer = MLP
(non-linear, solves XOR).

8. Back Propagation Network ⭐⭐⭐⭐⭐

PYQ: 2024 Q2b "Backpropagation algorithm in detail" (7 marks) — extremely important, base of Unit II as well.

Definition

Backpropagation is a supervised learning algorithm that trains a multi-layer neural network by computing the gradient of the loss function
with respect to each weight (using the chain rule), and propagating this error backward from the output layer to the input layer to update
the weights.

Keywords

Forward pass, Loss/Error, Backward pass, Chain rule, Gradient, Gradient Descent, Weight update, Learning rate

Why do we need it?

A single perceptron cannot learn complex, non-linear patterns (like XOR). MLPs (with hidden layers) can — but we need an efficient way to
figure out how much each weight, in every layer, contributed to the final error. Backpropagation solves this using calculus (chain
rule).

Visualization — Full Cycle

DRAW THIS IN EXAM — very frequently expected

Input
|
v
Forward Propagation (compute outputs layer by layer)
|
v
Prediction (y_hat)
|
v
Loss / Error (compare y_hat with actual y)
|
v
Backward Propagation (send error backward through layers)
|
v
Gradient Calculation (chain rule: dLoss/dWeight for every weight)
|
v
Weight Update (Gradient Descent: w = w - eta * dLoss/dw)
|
 v
Repeat (many epochs, until loss is minimum)

Working (Step by Step)

1. Forward Pass: Input passes through each layer (weighted sum + activation) to produce a final prediction.
2. Loss Calculation: Compare prediction with actual target using a loss function (e.g. Mean Squared Error, Cross-Entropy).
3. Backward Pass: Starting from the output layer, compute how much each weight contributed to the error, moving backward layer by
layer.
4. Chain Rule (intuitive): Error at output is "chained" backward — the gradient for an earlier weight is the product of gradients of all
the layers after it. This is just repeated application of derivative chain rule from calculus.
5. Gradient Descent: Once gradients are known, update every weight in the direction that reduces the loss.
6. Repeat for many epochs until the loss stops decreasing (converges).

Mathematics

Weight update rule (Gradient Descent): w(new) = w(old) − η * (∂L / ∂w)

L = Loss function, ∂L/∂w = gradient of loss w.r.t. that weight (how much loss changes if weight changes slightly), η = learning rate.

MEMORY TRICK: "F-L-B-G-W-R" = Forward → Loss → Backward → Gradient → Weight update → Repeat.

Advantages
Efficiently computes gradients for every weight in deep networks (reuses computation via chain rule)
Enables training of multi-layer, non-linear networks (solves XOR-type problems)
Works for any differentiable activation/loss function

Limitations
Can suffer from vanishing gradient problem in very deep networks
Can get stuck in local minima
Computationally expensive for very large networks
Requires labeled data (supervised)

EXAM-READY ANSWER (7–9 marks): "Explain Backpropagation Algorithm"

1. Definition (2–3 lines).
2. Draw the full cycle diagram (Input→Forward→Prediction→Loss→Backward→Gradient→Weight Update→Repeat).
3. Explain the 5 working steps (forward pass, loss, backward pass, chain rule, gradient descent).
4. Write the weight update equation, explain η and ∂L/∂w.
5. Mention that it needs a differentiable activation function (mention sigmoid/ReLU).
6. Give 2 advantages and 2 limitations (especially vanishing gradient).
7. Conclude: "Backpropagation, combined with Gradient Descent, is what makes training deep multi-layer networks practically
possible."

QUICK REVISION: Forward pass → Loss → Backward pass (chain rule) → Gradients → w = w − η(∂L/∂w) → Repeat. Solves the "how
much did each weight cause the error" problem.

UNIT I — EXAM STRATEGY

A. MUST-DO: McCulloch-Pitts Model (2024 PYQ, 9 marks), Backpropagation (2024 PYQ, 7 marks), Perceptron Learning + why XOR fails,
Biological vs Artificial Neuron comparison (2023 & 2025 PYQ).
B. HIGH-PRIORITY: AI vs ML vs DL, Feed Forward Network, MLP overcoming single-layer limits (2025 PYQ).
C. LOW-PRIORITY: Detailed numerical perceptron training beyond 1-2 iterations.

D. DEFINITIONS SHEET: Neuron, MCP Neuron, Perceptron, Weight, Bias, Activation Function, Threshold, Backpropagation, Gradient
Descent, Learning Rate.
E. DIAGRAM SHEET: Neuron architecture, Biological neuron, AI/ML/DL circles, MCP AND/OR/NOT gates, Perceptron diagram, XOR non-
separability plot, Backprop full cycle.
F. FORMULA SHEET: z=Σwx+b, y=f(z), MCP firing rule, w(new)=w(old)+η(t−y)x, w(new)=w(old)−η(∂L/∂w).
G. COMPARISON SHEET: Biological vs Artificial Neuron; MCP Neuron vs Perceptron; Single vs Multi-layer FFN; AI vs ML vs DL.
 UNIT II — Feedforward Networks (MLP, Gradient Descent, Backpropagation,
ERM, Regularization, Autoencoders)

Syllabus (from your course file): Multilayer Perceptron, Gradient Descent, Backpropagation, Empirical Risk Minimization, Regularization,
Autoencoders.

1. Multilayer Perceptron (MLP) ⭐⭐⭐⭐⭐

PYQ: 2024 Q2b "Linear and Multilayer Perceptron in detail" (7), 2025 Q2a "What is MLP? How does it overcome limitations of single-layer
perceptrons?"

Definition

A Multilayer Perceptron is a feed-forward neural network with one or more hidden layers between the input and output layers, where
each neuron uses a non-linear activation function, enabling it to learn complex, non-linearly separable patterns.

Visualization

DRAW THIS IN EXAM

Input Layer Hidden Layer(s) Output Layer

x1 o o o
x2 o --> o o --> o (y)
x3 o o o
(each connection has a weight; each neuron has bias + activation)

How MLP overcomes single-layer limits

A single-layer perceptron can only draw one straight line → fails on XOR (see Unit I).
MLP stacks multiple layers with non-linear activation functions (Sigmoid/ReLU/tanh) → this lets it combine several straight lines
into a curved, complex decision boundary.
MLP is trained with Backpropagation + Gradient Descent, which single-layer perceptron learning rule cannot do across hidden
layers.

MEMORY TRICK: "Depth + Non-linearity = the power of MLP over Perceptron."

Feature Single-Layer Perceptron Multilayer Perceptron

Hidden layers 0 ≥1

Solves XOR? No Yes

Activation Step function Sigmoid/ReLU/tanh (differentiable)

Training algorithm Perceptron rule Backpropagation

Decision boundary Linear Non-linear

QUICK REVISION: MLP = FFN with ≥1 hidden layer + non-linear activations, trained via backpropagation, overcomes linear-
separability limit of single perceptron.

2. Gradient Descent ⭐⭐⭐⭐⭐

PYQ: 2023 Q2a "Importance of Gradient Descent" (7), 2025 Q2b "Optimization techniques: GD, SGD, Adam" (7), 2025 Q5a "Batch GD vs SGD" (7),
2024 Q6b (ERM, related)

Definition

Gradient Descent is an optimization algorithm that iteratively updates a model's weights in the direction opposite to the gradient of the
loss function, in order to reach the minimum loss (best-fit weights).

Why do we need it?

After backpropagation gives us the gradient (∂L/∂w) for every weight, we need a rule to actually change the weights so that the loss
decreases. Gradient Descent is that rule — think of it as walking downhill on the "loss surface" toward the lowest point (minimum error).

Visualization

DRAW THIS IN EXAM — bowl-shaped loss curve with descent arrows

Loss
|\ /|
| \ / |
| \ start (high loss) / |
| \ o / |
 | \ \ step 1 / |
| \ v / |
| \ o / |
| \ \ step2 / |
| \ v o(minimum) /
|_________\_______________ ____/______ weight (w)

Mathematics

w(new) = w(old) − η * (∂L / ∂w)

η (learning rate) = step size; too large → overshoots minimum, too small → very slow convergence. ∂L/∂w = slope of loss curve at current
weight (tells direction of steepest increase; we move opposite to it).

Types of Gradient Descent

Feature Batch Gradient Descent Stochastic Gradient Descent (SGD) Mini-Batch GD

Data used per update Entire training dataset One random sample Small batch (e.g. 32, 64 samples)

Speed per update Slow Fast Medium

Convergence path Smooth Noisy/zig-zag Moderately smooth

Memory needed High Low Moderate

Used in practice? Rarely (large data) Sometimes Most common in deep learning

Adam Optimizer (brief)

Adam = Adaptive Moment Estimation. It combines Momentum (keeps track of past gradients to smooth updates) and RMSProp (adapts
the learning rate for each weight individually). It generally converges faster and more reliably than plain SGD, and is the most widely used
optimizer in modern deep learning.

EXAM-READY ANSWER (7 marks): "Batch GD vs SGD" / "Optimization techniques"

1. Define Gradient Descent (2 lines).
2. Draw the bowl-shaped loss curve with descent steps.
3. Write the weight update formula and explain η and gradient.
4. Give the Batch vs SGD vs Mini-batch comparison table.
5. Briefly mention Adam as an adaptive, faster-converging optimizer.
6. Conclude: mini-batch GD (often with Adam) is the practical standard in deep learning — balances speed and stability.

QUICK REVISION: w=w−η(∂L/∂w). Batch=whole data (slow,smooth), SGD=1 sample (fast,noisy), Mini-batch=small group (best of
both). Adam = momentum + adaptive learning rate.

3. Backpropagation (Feed-Forward context) ⭐⭐⭐⭐⭐

See Unit I Topic 8 for full detail — same algorithm, asked repeatedly (2024, 2025).

In the Feedforward Networks unit, Backpropagation is studied as the training algorithm for MLPs: it uses the chain rule to propagate
the error gradient from the output layer back to every hidden layer, so that Gradient Descent can update all the weights. See Unit I →
"Back Propagation Network" for the full diagram, steps, and equations — they are identical to what is expected here, just applied
specifically to MLP training.

QUICK REVISION: Backprop = chain-rule gradient computation; Gradient Descent = the update step that uses those gradients. They
always work together.

4. Empirical Risk Minimization (ERM) ⭐⭐⭐⭐

PYQ: 2024 Q6b "Objective of Empirical Risk Minimization, explain principle" (6 marks)

Definition

Empirical Risk Minimization is the principle of training a model by minimizing the average loss (empirical risk) calculated over the
available training data, as a practical stand-in for minimizing the true (but unknown) risk over the entire data distribution.

Keywords

Empirical risk, True risk, Loss function, Training data, Overfitting, Generalization

Why do we need it?

Ideally we want a model that performs well on all possible data (true risk), but we only have a limited training sample. ERM says: since
we cannot compute true risk, minimize the average error on the training sample we do have — this is what most ML/DL training actually
does in practice.

Mathematics

R_emp(w) = (1/N) Σ L(y_i, f(x_i; w))

N = number of training examples, L = loss function, y_i = true label, f(x_i;w) = model prediction for example i, w = model weights.
Training = find w that minimizes R_emp(w).

Visualization

True Risk (over ALL possible data) -- unknown, cannot compute directly
|
| approximate using training sample
v
Empirical Risk (average loss over N training examples) -- ERM minimizes THIS
|
v
Model learns weights w that minimize training loss

Limitations
Minimizing empirical risk too aggressively can cause overfitting — model memorizes training data but performs poorly on
new/unseen data.
This is exactly why Regularization (next topic) is added to the ERM objective.

EXAM-READY ANSWER (6–7 marks): "Empirical Risk Minimization"

1. Define ERM (2–3 lines).
2. Explain true risk vs empirical risk with the diagram above.
3. Write the formula R_emp(w) = (1/N)ΣL(y_i,f(x_i;w)) and explain each symbol.
4. State that training = minimizing R_emp(w).
5. Mention the overfitting risk and link to regularization as the solution.

QUICK REVISION: ERM = minimize average training loss (1/N)ΣL(y,f(x)) as a proxy for true risk. Risk: overfitting → solved by
regularization.

5. Regularization ⭐⭐⭐

Definition

Regularization is a set of techniques that add a penalty or constraint to the training process to reduce overfitting and improve a model's
ability to generalize to unseen data.

Keywords

Overfitting, Underfitting, L1/L2 regularization, Dropout, Early stopping, Generalization

Visualization

Without Regularization: Loss(w) = R_emp(w) -> may overfit

With Regularization: Loss(w) = R_emp(w) + λ * Penalty(w) -> generalizes better

Common Techniques

Technique Idea

L1 Regularization (Lasso) Adds sum of |weights| to loss — pushes some weights to exactly 0 (feature selection)

L2 Regularization (Ridge/Weight Adds sum of weights² to loss — keeps weights small and smooth
Decay)

Dropout Randomly "turns off" a fraction of neurons during each training step, forcing the network not to rely too heavily
on any one neuron

Early Stopping Stop training when validation loss starts increasing (even if training loss still decreasing)

Data Augmentation Artificially increase training data variety (flips, crops, noise) so the model generalizes better

Applications

Used in almost every deep network (MLP, CNN, RNN) to prevent overfitting, especially when training data is limited.

QUICK REVISION: Regularization = penalty added to loss to avoid overfitting. Key methods: L1 (sparsity), L2 (small weights), Dropout
 (random neuron drop), Early stopping.

6. Autoencoders ⭐⭐⭐⭐

PYQ: 2024 Q2a "Explain different types of Autoencoders" (7), 2024 Q5b "Diff Autoencoders and RBM" (7), 2023 Q8a "Deep Generic Models" choice,
2025 Q8d "Autoencoder Architecture" choice

Definition

An Autoencoder is an unsupervised neural network that learns to compress input data into a smaller latent representation (encoding) and
then reconstruct the original input from that representation (decoding), with the goal of learning efficient, meaningful features.

Keywords

Encoder, Decoder, Latent space/Bottleneck, Reconstruction loss, Unsupervised learning

Visualization

DRAW THIS IN EXAM

Input (x) --> [ ENCODER ] --> Latent/Bottleneck (z, small) --> [ DECODER ] --> Output (x_hat)
|____________________________________________________________________________________|
Goal: x_hat should be as close to x as possible
(Reconstruction Loss = difference between x and x_hat)

Working
1. Encoder compresses input x into a lower-dimensional latent vector z (forces the network to learn only the most important features).
2. Decoder reconstructs the original input from z, producing x_hat.
3. Network is trained to minimize reconstruction loss (e.g. MSE between x and x_hat) — no labels needed, so training is
unsupervised.

Types of Autoencoders

Type Idea

Vanilla/Basic Autoencoder Simple encoder-decoder, bottleneck smaller than input

Denoising Autoencoder Trained to reconstruct clean input from a noisy/corrupted version — learns robust features

Sparse Autoencoder Adds a sparsity penalty so only a few neurons activate at a time

Convolutional Autoencoder Uses convolution layers instead of fully-connected — good for images

Variational Autoencoder (VAE) Learns a probability distribution over the latent space (not a fixed point) — can generate new data by sampling

Autoencoders vs RBM

Feature Autoencoder RBM (Restricted Boltzmann Machine)

Structure Encoder + Decoder (feed-forward, deterministic) Bipartite graph of visible + hidden units, stochastic

Training Backpropagation (minimize reconstruction loss) Contrastive Divergence / Gibbs Sampling

Nature Deterministic mapping Probabilistic/generative energy-based model

Direction One-directional (encoder→decoder) Bidirectional connections between visible & hidden layers

Applications

Dimensionality reduction, denoising, anomaly detection, feature learning, image compression, pretraining for deep networks.

EXAM-READY ANSWER (7 marks): "Different types of Autoencoders"

1. Define Autoencoder (2–3 lines).
2. Draw the Input→Encoder→Latent→Decoder→Output diagram.
3. Explain the 3-step working (encode, decode, minimize reconstruction loss).
4. List and briefly explain 3–5 types (vanilla, denoising, sparse, convolutional, VAE) from the table.
5. Mention 2–3 applications.

QUICK REVISION: Autoencoder = Encoder (compress) + Decoder (reconstruct), unsupervised, minimizes reconstruction loss. Types:
vanilla, denoising, sparse, convolutional, VAE (generative).

UNIT II — EXAM STRATEGY

A. MUST-DO: MLP overcoming single-layer limits (2024 & 2025 PYQ), Gradient Descent + Batch vs SGD (2023 & 2025 PYQ),
Autoencoder types (2024 PYQ).
B. HIGH-PRIORITY: Empirical Risk Minimization (2024 PYQ), Autoencoders vs RBM (2024 PYQ), Regularization.
C. LOW-PRIORITY: Deep mathematical derivation of Adam optimizer internals.

D. DEFINITIONS SHEET: MLP, Gradient Descent, SGD, Adam, Empirical Risk, Regularization, Dropout, Autoencoder, Encoder, Decoder,
Latent space.
E. DIAGRAM SHEET: MLP architecture, Gradient descent bowl curve, ERM flow, Autoencoder encoder-decoder diagram.
F. FORMULA SHEET: w=w−η(∂L/∂w); R_emp(w)=(1/N)ΣL(y_i,f(x_i;w)); Regularized loss = R_emp(w)+λPenalty(w).
G. COMPARISON SHEET: Single vs Multi-layer Perceptron; Batch vs SGD vs Mini-batch; Autoencoder vs RBM; L1 vs L2 regularization.
 UNIT III — Convolutional Networks

1. Convolutional Networks / The Convolution Operation ⭐⭐⭐⭐⭐

PYQ: 2023 Q7a "Convolution Operation, explain in detail" (8), 2024 Q7a "Why convolution network in DL" (7), 2025 Q3b "Significance of
convolution operation in CNNs" (7), 2025 Q3a "Briefly discuss CNN with Deep Learning" (7)

Definition

A Convolutional Neural Network (CNN) is a deep learning architecture designed for grid-like data (especially images) that uses the
convolution operation to automatically extract spatial features (edges, textures, shapes) using small filters that scan across the input.

Keywords

Kernel/Filter, Feature Map, Local Connectivity, Weight Sharing, Stride, Padding, Pooling, ReLU, Flatten, Fully Connected
Layer

Why CNN for images?

A normal MLP flattens the image into a long vector — this destroys spatial structure (which pixels are near each other) and
creates too many parameters for large images.
CNN uses small filters that slide over the image — this preserves spatial relationships and drastically reduces parameters via weight
sharing (same filter used at every location).
Local connectivity: each neuron only looks at a small local region (receptive field) instead of the whole image, just like how our
eyes focus on small regions at a time.

Visualization — CNN Architecture

DRAW THIS IN EXAM

Image
|
v
Convolution (filters slide over image)
|
v
Feature Map (highlights detected patterns)
|
v
ReLU (adds non-linearity, removes negative values)
|
v
Pooling (downsamples, keeps important info, reduces size)
|
v
(Repeat Conv+ReLU+Pool blocks several times)
|
v
Flatten (convert 2D feature maps to 1D vector)
|
v
Fully Connected Layer (like a normal MLP)
|
v
Output (class probabilities)

Numerical Example — Convolution Operation

3×3 input, 2×2 filter [[1,0],[0,1]], stride=1, no padding → output is 2×2:

Input: Filter: Output Feature Map:

1 2 3 1 0 (1*1+2*0+4*0+5*1)=6 (2*1+3*0+5*0+6*1)=8
4 5 6 0 1
7 8 9 (4*1+5*0+7*0+8*1)=12 (5*1+6*0+8*0+9*1)=14

Mathematics

S(i,j) = ΣΣ I(i+m, j+n) * K(m,n)

I = input image, K = kernel/filter, S = output feature map (sum of element-wise multiplication of filter with the region it covers, slid across
the whole image).

Advantages
Far fewer parameters than fully-connected MLP for images (weight sharing)
 Preserves spatial structure
Automatically learns hierarchical features (edges → shapes → objects)
Translation invariance (can detect a feature anywhere in the image)

Limitations
Needs large labeled datasets and heavy compute
Not naturally rotation/scale invariant (needs augmentation)
Struggles with very long-range spatial relationships (large receptive field needed)

Applications

Image classification, object detection, face recognition, medical image analysis, self-driving car vision.

EXAM-READY ANSWER (7–8 marks): "What is Convolution Operation? / Why CNN for Deep Learning?"
1. Define CNN and convolution (2–3 lines).
2. Explain why plain MLP fails on images (destroys spatial info, too many params).
3. Draw the CNN architecture pipeline diagram (Image→Conv→ReLU→Pool→...→FC→Output).
4. Show a small numerical convolution example (like above).
5. Write the convolution formula, explain I, K, S.
6. Mention weight sharing + local connectivity as key ideas.
7. Give 2 advantages, 2 limitations, 2 applications.

QUICK REVISION: CNN pipeline: Image→Conv→ReLU→Pool→Flatten→FC→Output. Convolution = filter slides + element-wise multiply +
sum. Key ideas: local connectivity + weight sharing.

2. Variants of the Basic Convolution Function ⭐⭐⭐⭐⭐

PYQ: 2023 Q3b "Explain variants of the basic convolution function in detail" (7) — directly asked, must-do.

Definition

The basic convolution operation can be modified in several ways — padding, stride, dilation, and connectivity pattern — to control the
output size, computation cost, and receptive field of a CNN.

Variant Definition Purpose

Padding Adding extra (usually zero) pixels around the input Controls output size; preserves border information that
border before convolution would otherwise be lost

Valid Convolution No padding used — output is smaller than input Simplest, but shrinks feature map every layer and loses
border info

Same Convolution Padding added so output size = input size Keeps spatial dimensions constant across layers

Stride Number of pixels the filter moves at each step Larger stride = more downsampling, less computation

Strided Convolution Convolution with stride > 1 Reduces output size while extracting features (acts like
pooling)

Dilated Convolution Filter has "gaps" between its weights (dilation rate > 1) Increases receptive field without increasing
parameters/computation

Locally Connected / Like convolution, but each location uses a different filter Useful when different image regions have very different
Unshared Convolution (no weight sharing) statistics (e.g. face parts)

Tiled Convolution A middle ground: a set of filters is learned and cycled Balances parameter efficiency of shared convolution
across locations (partial weight sharing) with flexibility of unshared convolution

Visualization — Padding, Stride, Dilation

DRAW ALL THREE IN EXAM

PADDING (zero border added): STRIDE (filter jump = 2): DILATION (filter with gaps):
0 0 0 0 0 X . X . X X . X
0 1 2 3 0 <- zeros added (filter moves 2 steps . . . <- filter "spread out"
0 4 5 6 0 around border at a time, skips 1 pixel) X . X covers wider area,
0 7 8 9 0 same # of weights
0 0 0 0 0

Output Size Formula

Output = ((Input + 2*Padding − Kernel) / Stride) + 1

EXAM-READY ANSWER (7 marks): "Variants of the basic convolution function"

1. State that basic convolution can be modified via padding, stride, dilation and connectivity pattern.
 2. For each of Padding (valid/same), Stride, Dilated, Locally-connected, Tiled: give a 1-line definition + purpose (use the table).
3. Draw the padding/stride/dilation diagrams.
4. Write the output-size formula.
5. Conclude: these variants let CNN designers trade off receptive field size, output resolution, and computation cost.

QUICK REVISION: Padding=border pixels added (valid=none, same=output size preserved). Stride=step size of filter. Dilation=filter
with gaps (bigger receptive field, same params). Locally-connected=no weight sharing. Tiled=partial sharing.

3. Structured Outputs ⭐⭐⭐⭐

PYQ: 2024 Q8a & 2025 Q8a "Structured output in Convolutional Network" (choice question, 7–14 marks)

Definition

Structured output refers to CNN tasks where the model must output a structured, high-dimensional prediction (like a label for every
pixel) instead of just a single class label for the whole image.

Ordinary Classification vs Structured Output

DRAW THIS IN EXAM

Ordinary classification: Image ---> CNN ---> ONE label (e.g. "cat")

Structured output: Image ---> CNN ---> label for EVERY pixel/region
(e.g. pixel-wise segmentation map)

Key Ideas
Pixel-wise prediction: CNN outputs a full 2D map, same spatial size as input, where each pixel has its own predicted class.
Semantic Segmentation: classic structured-output task — every pixel labeled (e.g. "road", "car", "sky").
Spatial relationships between output elements must be respected (e.g. neighbouring pixels usually belong to the same object) —
this makes structured output harder than simple classification.
CNNs are well suited because their convolutional feature maps are already spatially organized — they naturally preserve the "where"
information needed for pixel-wise output.

Applications

Semantic segmentation (self-driving cars, medical imaging), pose estimation, image captioning (structured sequence output).

QUICK REVISION: Structured output = one label per pixel/region (not one label for whole image). Main example: semantic
segmentation. CNNs work well because feature maps preserve spatial layout.

4. Efficient Convolution Algorithms ⭐⭐⭐

Why is convolution computationally expensive?

A direct/naive convolution requires sliding the filter across every position of a (potentially large) input and computing element-wise
multiply + sum at each position — for large images, many channels, and many filters, this becomes a very large number of multiply-add
operations, especially in deep CNNs with many layers.

Key Approaches

Approach Idea

im2col + Matrix Rearrange overlapping image patches into columns of a matrix, then convolution becomes a single big matrix
Multiplication multiplication — which GPUs are extremely optimized for

FFT-based Convolution in the spatial domain = multiplication in the frequency domain. Convert input & filter using Fast Fourier
Convolution Transform, multiply, then inverse-transform — much faster for large filters

Separable Split a 2D filter into two 1D filters (one horizontal, one vertical) applied sequentially — reduces multiplications from O(k²) to
Convolution O(2k) per pixel

Visualization

im2col approach:
Image patches ---> flattened into columns ---> Matrix Multiply (with filter matrix) ---> Feature Map
(fast on GPU hardware)

Separable convolution:
2D filter (k x k) = 1D vertical filter (k x 1) followed by 1D horizontal filter (1 x k)
Why efficient convolution matters

Modern CNNs (like AlexNet, ResNet) have millions of parameters and are trained on millions of images — without efficient convolution
algorithms, training would be computationally infeasible even with GPUs.

QUICK REVISION: Convolution is expensive due to sliding-window multiply-adds. Solutions: im2col+matmul (GPU-friendly), FFT-based
(fast for large filters), separable convolution (splits 2D filter into two 1D filters).

5. Random or Unsupervised Features ⭐⭐⭐

Definition

Instead of learning all convolution filters through backpropagation on labeled data, CNN filters can also be obtained using randomly
initialized (and fixed) filters or via unsupervised feature learning methods that don't require labels.

Key Ideas
Random features: surprisingly, even randomly generated filters (never trained) can extract somewhat useful low-level features
(like edge detectors) — useful as a fast baseline.
Unsupervised feature learning: learn filters from unlabeled data using clustering (e.g. k-means on patches), sparse coding, or
autoencoders — useful since labeled data is expensive but unlabeled data is abundant.

Feature Supervised Unsupervised Random

Needs labels? Yes No No

Training required? Yes (backprop) Yes (e.g. clustering/autoencoder) No (fixed at initialization)

Quality of features Best (task-specific) Good (general-purpose) Weak but surprisingly non-zero

Cost High Medium Very low

Why unlabeled data is useful

Labeled data is expensive and limited, while unlabeled data (raw images, text) is abundant. Unsupervised/random feature methods let us
leverage this abundant unlabeled data to build useful initial representations, which can then be fine-tuned with a smaller labeled dataset.

QUICK REVISION: Random features = fixed, untrained filters (surprisingly usable as a baseline). Unsupervised features = learned
without labels (clustering, sparse coding, autoencoders). Motivation: unlabeled data is cheap and abundant.

6. LeNet ⭐⭐⭐⭐⭐

PYQ: 2024 Q3b "What is LeNet? Explain the usage of LeNet in object recognition" (7), 2023 Q3a "Role of LeNet in Convolutional Networks" (7)

Definition

LeNet (LeNet-5, by Yann LeCun, 1998) is one of the earliest CNN architectures, originally designed to recognize handwritten digits (e.g.
postal codes, bank cheques), consisting of alternating convolution and subsampling (pooling) layers followed by fully connected layers.

Visualization — LeNet-5 Architecture

DRAW THIS IN EXAM

32x32 Input
|
v
C1: Convolution -> 6@28x28
|
v
S2: Subsampling/Pooling -> 6@14x14
|
v
C3: Convolution -> 16@10x10
|
v
S4: Subsampling/Pooling -> 16@5x5
|
v
C5: Convolution (fully connected) -> 120
|
v
F6: Fully Connected -> 84
|
v
Output -> 10 classes (digits 0-9)
 MEMORY TRICK: "6 gets Sweet 16, makes 120 Rupees in 84 Ten-paisa coins" → 6 → 16 → 120 → 84 → 10

Layer-by-layer Explanation
Input: 32×32 grayscale image
C1 (Conv): 6 filters → produces 6 feature maps of size 28×28 (extracts basic features like edges)
S2 (Pooling): subsamples to 6@14×14 (reduces size, keeps important info)
C3 (Conv): 16 filters → 16@10×10 (extracts more complex features)
S4 (Pooling): subsamples to 16@5×5
C5 (Conv, acts as FC): 120 feature maps (essentially fully connected at this point since spatial size is 1×1 after this)
F6 (Fully Connected): 84 neurons
Output: 10 neurons (digit classes 0–9), using softmax/RBF

Usage in Object Recognition

LeNet was used to automatically read handwritten/printed digits for postal mail sorting and bank cheque processing — demonstrating that
CNNs could outperform traditional hand-crafted feature methods for image recognition, and set the template (Conv+Pool blocks → FC
layers) followed by almost all later CNNs.

EXAM-READY ANSWER (7 marks): "What is LeNet? Usage in object recognition"

1. Definition (2–3 lines) with year/author (LeCun, 1998).
2. Draw the full architecture diagram with dimensions at each stage.
3. Explain each layer briefly (C1, S2, C3, S4, C5, F6, Output).
4. Use the memory trick to recall the numbers 6-16-120-84-10.
5. Explain its usage: handwritten digit recognition, postal/cheque processing, and its historical importance as the template for
modern CNNs.

QUICK REVISION: LeNet-5: 32x32 input → C1(6@28x28) → S2(6@14x14) → C3(16@10x10) → S4(16@5x5) → C5(120) → F6(84) →
Output(10). Used for handwritten digit recognition.

7. AlexNet ⭐⭐⭐⭐⭐

PYQ: 2024 Q8b & 2025 Q8b "AlexNet" (choice question, 7–14 marks) — repeatedly asked in choice questions.

Definition

AlexNet (Krizhevsky, Sutskever, Hinton, 2012) is a deep CNN that won the ImageNet Large Scale Visual Recognition Challenge (ILSVRC)
2012 by a huge margin, and is widely credited with sparking the modern deep learning revolution in computer vision.

Architecture
5 Convolutional layers + 3 Fully Connected layers
Uses ReLU activation (instead of sigmoid/tanh) — trains much faster
Uses Max Pooling layers to downsample feature maps
Uses Dropout in fully-connected layers to reduce overfitting
Uses Data Augmentation (crops, flips) to artificially increase training data
Trained on GPUs (2 GPUs in parallel) — showed GPUs could make deep CNN training practical
DRAW THIS IN EXAM — simplified block diagram

Input Image (224x224x3)

|
v
Conv1 -> ReLU -> Max Pool
|
v
Conv2 -> ReLU -> Max Pool
|
v
Conv3 -> ReLU
|
v
Conv4 -> ReLU
|
v
Conv5 -> ReLU -> Max Pool
|
v
FC1 (Dropout) -> FC2 (Dropout) -> FC3
|
v
 Output (1000 ImageNet classes, Softmax)

MEMORY TRICK: "AlexNet = 2012 + ImageNet + 5 Conv + 3 FC + ReLU + Max Pooling + Dropout + GPU"

LeNet vs AlexNet

Feature LeNet (1998) AlexNet (2012)

Depth ~7 layers (2 conv) 8 layers (5 conv + 3 FC)

Activation Sigmoid/tanh ReLU (faster training)

Dataset Small (MNIST digits) Large (ImageNet, 1.2M images, 1000 classes)

Regularization None specific Dropout + Data Augmentation

Hardware CPU GPU (2 GPUs in parallel)

Application Handwritten digit recognition General large-scale image classification

Impact First successful CNN template Triggered the deep learning boom in computer vision

EXAM-READY ANSWER (7 marks): "Explain AlexNet"

1. Definition with year and ImageNet 2012 win (2–3 lines).
2. Draw the block diagram (5 conv + 3 FC pipeline).
3. List the key innovations: ReLU, Max Pooling, Dropout, Data Augmentation, GPU training.
4. Use the memory line to recall all key facts.
5. Give the LeNet vs AlexNet comparison table.
6. Conclude: AlexNet proved deep CNNs + GPUs + large data = breakthrough performance, launching the deep learning era.

QUICK REVISION: AlexNet (2012, ImageNet winner): 5 Conv + 3 FC, ReLU, Max Pooling, Dropout, Data Augmentation, trained on
GPUs. Sparked the deep learning revolution.

UNIT III — EXAM STRATEGY

A. MUST-DO: Convolution Operation (2023 & 2024 & 2025 PYQ), Variants of convolution – padding/stride/dilation (2023 PYQ), LeNet
(2023 & 2024 PYQ), AlexNet (2024 & 2025 PYQ, appears every year in choice questions).
B. HIGH-PRIORITY: Structured Outputs (2024 & 2025 PYQ choice question), Pooling layers/max&avg pooling (2025 PYQ).
C. LOW-PRIORITY: Deep math derivation of FFT-based convolution.

D. DEFINITIONS SHEET: Convolution, Kernel/Filter, Feature Map, Padding, Stride, Dilation, Pooling, Structured output, im2col, LeNet,
AlexNet.
E. DIAGRAM SHEET: CNN pipeline, padding/stride/dilation diagrams, ordinary vs structured output, LeNet-5 full architecture, AlexNet
block diagram.
F. FORMULA SHEET: S(i,j)=ΣΣI(i+m,j+n)K(m,n); Output=((Input+2Padding−Kernel)/Stride)+1.
G. COMPARISON SHEET: Valid vs Same convolution; LeNet vs AlexNet; Supervised vs Unsupervised vs Random features.
 UNIT IV — Recurrent Neural Networks

0. Basic RNN (Foundation) ⭐⭐⭐⭐⭐

PYQ: 2024 Q4a "Diff CNN and RNN" (7), 2025 Q4a "Diff CNN and RNN" (7) — appears every year

Definition

A Recurrent Neural Network (RNN) is a neural network designed for sequential data, where the output of a neuron at one time step is fed
back as input at the next time step, giving the network a form of "memory" of past information.

Keywords

Sequential data, Hidden state, Memory, Recurrent connection, Temporal dependency, Weight sharing across time,
Unrolling, BPTT, Vanishing/Exploding gradient

Why ordinary Feed-Forward Networks are insufficient

FFNs treat every input independently — they have no memory of previous inputs. But for sequences (text, speech, time-series), the order
and context matter (e.g. meaning of a word depends on previous words). RNNs solve this by maintaining a hidden state that carries
information forward through time.

Visualization — Unrolled RNN

DRAW THIS IN EXAM

x1 x2 x3
| | |
v v v
[h0]-->[h1]-->[h2]-->[h3] (hidden state passed forward through time)
| | |
v v v
y1 y2 y3

Compact (folded) view:

x_t
|
v
h(t-1) --> [ RNN cell ] --> h_t --> y_t
^
| (loop back into itself for next time step)

Mathematics

h_t = tanh(W_xh * x_t + W_hh * h_(t-1) + b)

x_t = input at time t, h_(t-1) = previous hidden state (memory), h_t = new hidden state, W_xh = weight matrix from input to hidden,
W_hh = weight matrix from previous hidden to current hidden (same weights reused at every time step — weight sharing), b = bias, tanh
= activation function.

Working
1. At each time step t, RNN takes current input x_t AND previous hidden state h_(t-1).
2. Combines them to compute new hidden state h_t (the "memory").
3. h_t can be used to produce output y_t, and is also passed to the next time step.
4. Same weights (W_xh, W_hh) are reused at every time step (weight sharing across time).
5. Training uses Backpropagation Through Time (BPTT) — the network is "unrolled" across time steps and normal backpropagation
is applied across this unrolled graph.

Vanishing / Exploding Gradient & Long-Term Dependency Problem

DON'T CONFUSE: this problem is why LSTM was invented — make sure to link RNN's weakness directly to LSTM's motivation in your answer.

Long Sequence
|
v
Gradients multiplied repeatedly through many time steps (chain rule)
|
v
Gradients shrink toward 0 (Vanishing) or grow huge (Exploding)
|
v
RNN forgets distant/old information (can't learn long-term dependencies)
|
v
 Solution: LSTM (adds Cell State + Gates for better long-term memory)

CNN vs RNN

Feature CNN RNN

Best suited for Spatial data (images) Sequential data (text, speech, time-series)

Key operation Convolution (filters over space) Recurrence (loop over time)

Memory of past No (each region processed independently) Yes (hidden state carries memory)

Weight sharing Shared across spatial locations Shared across time steps

Parallelization Easy (all positions independent) Harder (sequential dependency)

Common problem Overfitting on small datasets Vanishing/exploding gradient over long sequences

EXAM-READY ANSWER (7 marks): "Difference between CNN and RNN"

1. Give 1-line definitions of CNN and RNN.
2. Draw the CNN pipeline (Unit III) and the unrolled RNN diagram side by side.
3. Give the full comparison table above.
4. Conclude: "CNN exploits spatial locality, RNN exploits temporal/sequential dependency — they solve different classes of
problems."

QUICK REVISION: h_t = tanh(W_xh x_t + W_hh h_(t-1) + b). RNN has memory via hidden state, trained via BPTT. Long sequences →
vanishing gradient → forgets old info → motivates LSTM.

1. Bidirectional RNN ⭐⭐⭐⭐

PYQ: 2023 Q4a "What are Bidirectional RNNs? Explain in detail" (7), 2024 Q7b "Diff RNN and Bidirectional RNN" (7)

Definition

A Bidirectional RNN processes a sequence in both directions — one RNN reads left-to-right (forward, using past context) and another
reads right-to-left (backward, using future context) — and combines both hidden states to make predictions.

Visualization

DRAW THIS IN EXAM

Forward RNN: Past --------->

Current
|
v (combine/concatenate)
^
|
Backward RNN: Future ---------> Current
(reads sequence in reverse)

Working
Forward RNN processes the sequence from start to end, capturing past context.
Backward RNN processes the sequence from end to start, capturing future context.
At each time step, the forward hidden state and backward hidden state are combined (usually concatenated) to form the final
representation, so each output depends on the entire sequence (both before and after).

Feature RNN Bidirectional RNN

Context used Only past (left-to-right) Both past and future (left-to-right + right-to-left)

Number of RNN passes 1 2 (forward + backward)

Real-time/streaming use Yes (works as data arrives) No (needs entire sequence available first)

Accuracy for offline tasks Lower (missing future context) Higher (full context)

Advantages

Captures full context (past + future) → better accuracy for tasks like translation, named entity recognition, speech recognition.

Limitations

Cannot be used for real-time/streaming prediction (needs the whole sequence first); more computation (2 RNNs).

Applications

Machine translation, Named Entity Recognition, speech recognition, sentiment analysis.

 QUICK REVISION: Bidirectional RNN = Forward RNN (past) + Backward RNN (future), combined at each step. More accurate but needs
full sequence upfront (not real-time).

2. Deep Recurrent Networks ⭐⭐⭐⭐

PYQ: 2023 Q4b "Define Recurrent Network? Discuss about Deep Recurrent Networks" (7)

Definition

A Deep Recurrent Network stacks multiple RNN layers on top of each other, so that the output (hidden state sequence) of one RNN layer
becomes the input sequence to the next RNN layer — adding depth in the "vertical" direction in addition to the usual "horizontal" flow
through time.

Visualization

DRAW THIS IN EXAM

h2_1 --> h2_2 --> h2_3 (Layer 2 - higher level features)

^ ^ ^
| | |
h1_1 --> h1_2 --> h1_3 (Layer 1 - lower level features)
^ ^ ^
| | |
x1 x2 x3 (Input sequence)

Working
Horizontal flow: within each layer, hidden state flows through time (t=1,2,3...) like a normal RNN.
Vertical flow: at each time step, the hidden state of layer 1 becomes the input to layer 2, and so on — building hierarchical,
increasingly abstract sequence representations.

Advantages

Learns more abstract/complex temporal patterns; generally better performance on complex sequence tasks than a single-layer RNN.

Limitations

Harder to train (deeper network = more prone to vanishing gradient); more computation and parameters.

QUICK REVISION: Deep RNN = multiple RNN layers stacked; horizontal flow=through time, vertical flow=through layers. More
abstract features but harder to train.

3. Recursive Neural Networks ⭐⭐⭐

Definition

A Recursive Neural Network applies the same set of weights repeatedly over a tree-structured input (instead of a linear sequence),
combining child node representations to build a parent representation, recursively, up to the root of the tree.

DON'T CONFUSE: Recurrent NN operates over a linear sequence (chain, time steps). Recursive NN operates over a tree structure
(hierarchical). "Recurrent" repeats over TIME; "Recursive" repeats over TREE STRUCTURE.

Visualization — Tree Structure

DRAW THIS IN EXAM

Root (final representation)

/ \
Node P Node Q
/ \ / \
w1 w2 w3 w4 (leaf words/inputs combined recursively upward)

Rule: parent_vector = f(W * [child1_vector ; child2_vector] + b)

(same W reused at every merge point in the tree)

RNN vs Recursive NN

Feature Recurrent NN Recursive NN

Input structure Linear sequence (chain) Tree structure (hierarchical)

Processing direction Left to right (through time) Bottom-up (leaves to root)

 Weight sharing Across time steps Across tree nodes/merge operations

Typical use case Speech, time-series, general text Parsing sentence structure, sentiment on parse trees (NLP)

Applications

Natural Language Processing tasks that use parse trees — sentence structure analysis, sentiment analysis with syntactic trees.

QUICK REVISION: Recursive NN = same weights applied recursively over a TREE (not a sequence). parent = f(W[child1;child2]+b).
Used in NLP parse-tree tasks.

4. Long Short-Term Memory (LSTM) ⭐⭐⭐⭐⭐

PYQ: 2024 Q4b "What is Long Short Term Memory? Explain" (7), 2025 Q4b "LSTM explain" (7) — asked EVERY year, top priority.

Motivation

Basic RNN
|
v
Vanishing Gradient (over long sequences)
|
v
Long-Term Dependency Problem (forgets old information)
|
v
LSTM (solves this using a separate Cell State + Gates)

Definition

LSTM is a special type of RNN that introduces a cell state (a long-term memory highway) and three gates (Forget, Input, Output) that
control what information is added, removed, or output at each time step — allowing it to retain important information over long sequences
and avoid the vanishing gradient problem.

Keywords

Cell state, Hidden state, Forget gate, Input gate, Candidate cell state, Output gate, Sigmoid gate

Visualization — LSTM Cell

DRAW THIS IN EXAM — very frequently expected

C(t-1) ----------[x]-------[+]---------- C(t) (Cell State - long term memory)

| ^ ^ |
| | | |
Forget Input Candidate
Gate Gate Cell State
f_t i_t C~_t
\ | /
\ | /
h(t-1), x_t (combined input)
|
v
Output Gate (o_t)
|
v
h_t = o_t * tanh(C_t) (Hidden State - short term output)

MEMORY TRICK: F-I-O — Forget: "What should I remove?" Input: "What should I store?" Output: "What should I expose/output?"

Working (Gate by Gate, in Simple Words)

1. Forget Gate (f_t): Looks at h(t-1) and x_t, decides what fraction of the OLD cell state to keep vs throw away (0=forget completely,
1=keep completely).
2. Input Gate (i_t): Decides what NEW information is important enough to store in the cell state.
3. Candidate Cell State (C~_t): Creates new candidate values that could be added to the cell state.
4. Cell State Update (C_t): Combines: keep some of the old cell state (via forget gate) + add some new candidate info (via input
gate).
5. Output Gate (o_t): Decides what part of the (updated) cell state should be exposed as the hidden state/output at this time step.

Mathematics

f_t = σ(W_f[h(t-1), x_t] + b_f) (Forget Gate)

i_t = σ(W_i[h(t-1), x_t] + b_i) (Input Gate)
 C~_t = tanh(W_c[h(t-1), x_t] + b_c) (Candidate Cell State)
C_t = f_t * C_(t-1) + i_t * C~_t (Cell State Update)
o_t = σ(W_o[h(t-1), x_t] + b_o) (Output Gate)
h_t = o_t * tanh(C_t) (Hidden State / Output)

σ = sigmoid function (squashes values 0–1, acts as a "gate"/filter), [h(t-1),x_t] = concatenation of previous hidden state and current
input, W, b = learnable weights/biases for each gate, * = element-wise multiplication.

Basic RNN vs LSTM

Feature Basic RNN LSTM

Memory Single hidden state only Cell state (long-term) + Hidden state (short-term)

Gates None Forget, Input, Output gates

Long-term dependency Poor (vanishing gradient) Good (cell state acts as a gradient "highway")

Complexity Simple, fewer parameters More complex, more parameters

Training speed Faster per step Slower per step (more computation)

EXAM-READY ANSWER (7 marks): "What is LSTM? Explain"

1. Start with the motivation chain diagram (RNN → Vanishing Gradient → Long-term dependency problem → LSTM).
2. Give the definition (2–3 lines).
3. Draw the LSTM cell diagram with Forget/Input/Output gates and Cell state.
4. Use the F-I-O memory trick and explain each gate in simple words.
5. Write all 6 equations, briefly noting σ and tanh's role.
6. Give the Basic RNN vs LSTM comparison table.
7. Conclude: "The cell state, protected by gates, allows LSTM to carry relevant information across many time steps without the
gradient vanishing, solving RNN's core weakness."

QUICK REVISION: LSTM = Cell state + 3 gates (F-I-O). f_t=σ(...), i_t=σ(...), C~_t=tanh(...), C_t=f_t*C(t-1)+i_t*C~_t, o_t=σ(...),
h_t=o_t*tanh(C_t). Solves vanishing gradient / long-term dependency problem.

5. Other Gated RNNs (GRU) ⭐⭐⭐⭐

PYQ: 2023 Q7b "Write about Gated RNNs" (6)

Definition

GRU (Gated Recurrent Unit) is a simplified alternative to LSTM that uses only two gates (Update gate and Reset gate) and merges the
cell state and hidden state into one, achieving similar performance to LSTM with fewer parameters and faster training.

Keywords

Update gate, Reset gate, Single hidden state (no separate cell state)

Visualization

DRAW THIS IN EXAM

h(t-1), x_t
|
+----> Reset Gate (r_t) -- controls how much past info to forget when computing candidate
|
+----> Update Gate (z_t) -- controls balance between old hidden state and new candidate
|
v
h_t = (1 - z_t) * h(t-1) + z_t * h~_t (blend of old and new)

Why GRU is simpler than LSTM

GRU has 2 gates vs LSTM's 3 gates — fewer parameters to learn.
GRU has no separate cell state — hidden state alone carries both short and long-term memory.
Because of fewer computations, GRU trains faster and needs less data, while still handling long-term dependencies reasonably well.

RNN vs LSTM vs GRU

Feature RNN LSTM GRU

Gates None 3 (Forget, Input, Output) 2 (Update, Reset)

Memory type Hidden state only Cell state + Hidden state Hidden state only (merged)

Long-term Poor Very good Good

dependency
Parameters Fewest Most Fewer than LSTM

Training speed Fastest Slowest Faster than LSTM

When to prefer Very short Complex, long sequences, more data Good middle ground; smaller datasets/faster training
sequences available needed

EXAM-READY ANSWER (6–7 marks): "Write about Gated RNNs / GRU"

1. Define Gated RNNs in general (LSTM and GRU are the two main types), then focus on GRU.
2. Draw the GRU cell diagram (Reset gate, Update gate, blended hidden state).
3. Explain Update gate and Reset gate roles in simple words.
4. Write the hidden-state blend formula.
5. Give the RNN vs LSTM vs GRU comparison table.
6. Conclude: GRU is a lighter, faster alternative to LSTM with comparable performance on many tasks.

QUICK REVISION: GRU = 2 gates (Update, Reset), no separate cell state, simpler/faster than LSTM. h_t = (1−z_t)h(t-1) + z_t*h~_t.

UNIT IV — EXAM STRATEGY

A. MUST-DO: LSTM (2024 & 2025 PYQ, every year), CNN vs RNN (2024 & 2025 PYQ), Bidirectional RNN (2023 & 2024 PYQ).
B. HIGH-PRIORITY: Deep Recurrent Networks (2023 PYQ), Gated RNNs/GRU (2023 PYQ).
C. LOW-PRIORITY: Recursive Neural Networks (not directly asked yet, but do not skip — syllabus topic, could appear).

D. DEFINITIONS SHEET: RNN, Hidden state, BPTT, Bidirectional RNN, Deep RNN, Recursive NN, LSTM, Cell state, Forget/Input/Output
gate, GRU, Update/Reset gate.
E. DIAGRAM SHEET: Unrolled RNN, Bidirectional RNN (forward+backward), Deep RNN (stacked layers), Recursive NN tree, LSTM cell
(full gates), GRU cell.
F. FORMULA SHEET: h_t=tanh(W_xh x_t+W_hh h_(t-1)+b); LSTM's 6 equations (f_t, i_t, C~_t, C_t, o_t, h_t); GRU's h_t=(1−z_t)h(t-
1)+z_t h~_t.
G. COMPARISON SHEET: CNN vs RNN; RNN vs Bidirectional RNN; Recurrent vs Recursive NN; RNN vs LSTM vs GRU.

End of Notes — Units I to IV. Good luck for AD-601 Deep Learning!