Deep Learning (PCCAIML602) — Complete Study Guide

For Sayan | MAKAUT Semester VI

Zero-to-Exam-Ready Notes

How To Use This Guide

CHAPTER 1: INTRODUCTION TO DEEP LEARNING
1.1 What is Deep Learning?
1.2 Paradigms of Learning Problems
1.3 Issues / Perspectives in Deep Learning
1.4 Review of Fundamental Learning Techniques (Quick Recap)
1.5 Important One-Line Facts (frequently asked as True/False or fill-in-blank)
CHAPTER 2: FEEDFORWARD NEURAL NETWORKS
2.1 The Biological Inspiration (in one line)
2.2 The Artificial Neuron (Perceptron)
2.3 Single Layer Perceptron vs Logistic Regression (PYQ favorite: Q7a)
2.4 Activation Functions
2.5 Multi-Layer Neural Network (MLP) Architecture
2.6 Counting Weights and Biases (VERY common numerical — PYQ Q5)
2.7 A Simpler Worked Example (matches PYQ Q6 & Q9 style)
2.8 MNIST Digit Classification Architecture (PYQ Q3 — asked almost every year)
2.9 Deep NN Architecture for NLP / Text Classification (PYQ Q4)
CHAPTER 3: TRAINING NEURAL NETWORKS
3.1 The Core Training Loop
3.2 Loss Functions (Risk Minimization)
3.3 Gradient Descent — The Optimization Engine
3.4 Types of Gradient Descent (PYQ Q1-XII)
3.5 Backpropagation — How the Gradient is Actually Calculated
3.6 Vanishing Gradient Problem (Asked almost every year — PYQ Q1-IX-related, Q8e,
Q11i)
3.7 Regularization (Preventing Overfitting)
3.8 Model Selection & Evaluation Metrics
3.9 Other Common Conceptual Questions in This Chapter
CHAPTER 4: CONDITIONAL RANDOM FIELDS (CRF) & RELATED SEQUENCE MODELS
4.1 Why We Need These Models
4.2 Markov Network & Markov Property
4.3 Hidden Markov Model (HMM)
4.4 Linear Chain CRF
4.5 Partition Function
4.6 Belief Propagation
4.7 Training CRFs
4.8 Entropy (PYQ Q1-XI relates conceptually)
CHAPTER 5: DEEP LEARNING ARCHITECTURES
5.1 Deep Feedforward Networks (Recap + Extension)
5.2 Convolutional Neural Networks (CNN)
5.3 Recurrent Neural Networks (RNN)
5.4 Deep Belief Network (DBN)
5.5 Global Minima and Gradient Descent Certainty (PYQ Q8e — second half)
CHAPTER 6: DEEP LEARNING RESEARCH & APPLICATIONS
6.1 Object Recognition
 6.2 Sparse Coding
6.3 Computer Vision
6.4 Natural Language Processing (NLP)
QUICK-REVISION MASTER TABLE (Everything at a glance)
HOW TO ANSWER IN THE EXAM (Strategy Tips)

How To Use This Guide

This guide follows your official syllabus chapter order and is written for zero prior knowledge. Every
topic includes: plain-language explanation → formula/diagram (where needed) → worked example → the
exact type of exam question MAKAUT asks about it (based on the 2025 PYQ paper). Read it top to
bottom once, then use it as revision material.

CHAPTER 1: INTRODUCTION TO DEEP LEARNING

1.1 What is Deep Learning?

Machine Learning (ML) = teaching a computer to find patterns from examples instead of writing
explicit rules.

Deep Learning (DL) = a branch of ML that uses Artificial Neural Networks with many layers
(“deep” = many layers stacked) to automatically learn patterns directly from raw data.

Key difference from traditional ML:

Traditional ML Deep Learning

Human manually designs “features” (e.g., “check for

Network learns features automatically from raw data
word ‘free’ in email”)

Works well with small/medium data Needs large amounts of data to perform well

Fast to train Needs more compute (GPUs) and time

Example: Logistic Regression, SVM, Decision Tree Example: CNN, RNN, Transformer

1.2 Paradigms of Learning Problems

There are three broad ways a model can learn:

1. Supervised Learning – You give the model input AND the correct output (labels). It learns the
mapping input → output. Example: showing images labelled “cat”/“dog”.
2. Unsupervised Learning – You give only inputs, no labels. The model finds structure/groups on
its own. Example: clustering customers by purchase behaviour.
3. Reinforcement Learning – The model (agent) learns by taking actions in an environment and
getting rewards/penalties. Example: a game-playing AI.

Most of this syllabus (and exam) focuses on Supervised Learning using neural networks.

1.3 Issues / Perspectives in Deep Learning

Common challenges you should be aware of (asked conceptually):

Needs huge labelled data — Deep networks have millions of parameters; without enough data
they overfit (memorize instead of generalizing).
Computationally expensive — needs GPUs (see below).
Black box problem — hard to explain why a deep network made a decision.
Vanishing/Exploding gradients — a training problem in very deep networks (explained fully in
Chapter 3/5).
Overfitting — the model performs great on training data but poorly on new/unseen data.

1.4 Review of Fundamental Learning Techniques (Quick Recap)

Before DL, ML used algorithms like: - Linear Regression — fits a straight line to predict continuous
values. - Logistic Regression — despite the name, used for classification (yes/no type problems);
outputs a probability between 0 and 1 using the sigmoid function. - Decision Trees, SVM, k-NN —
other classical ML classifiers.

Deep Learning models are, in a sense, “logistic regression stacked in layers with non-linear
transformations added between them” — this idea will make much more sense after Chapter 2.

1.5 Important One-Line Facts (frequently asked as True/False or fill-in-

blank)

GPU = Graphics Processing Unit. It has thousands of small cores that do matrix multiplications
in parallel — this is exactly what neural networks need, so GPUs make training dramatically faster
than CPUs.
Big Data and Deep Learning are strongly related: DL models need very large datasets to
reach high accuracy; with small data, classical ML often outperforms DL.
Deep Learning is popular now (rather than in the 1980s when neural nets were invented) mainly
because of: (1) availability of Big Data, (2) powerful GPUs, (3) better algorithms.

CHAPTER 2: FEEDFORWARD NEURAL NETWORKS

This is the most important chapter — nearly every numerical question in the exam comes from here.

2.1 The Biological Inspiration (in one line)

A neural network is loosely inspired by brain neurons: a neuron receives signals, combines them, and
“fires” if the combined signal is strong enough. Artificial neurons do the same mathematically.

2.2 The Artificial Neuron (Perceptron)

A single artificial neuron does exactly 3 steps:

1. Take inputs and multiply each by a weight (a number that represents “importance”).
2. Sum them up and add a bias (a constant that shifts the result).
3. Apply an activation function to decide the final output.

Mathematically:
 z = (w1*x1 + w2*x2 + ... + wn*xn) + b
a = activation(z)

Where: - x1, x2, ..., xn = inputs - w1, w2, ..., wn = weights (learned during training) - b =
bias (learned during training) - z = weighted sum (“pre-activation”) - a = output of the neuron
(“activation”)

Analogy: Think of a neuron as a judge scoring a talent show. Each input (singing, dancing, stage
presence) is multiplied by how much the judge cares about it (weight), summed up, and a bias
represents the judge’s general strictness/leniency. The final score then passes through a rule
(activation function) that decides pass/fail.

2.3 Single Layer Perceptron vs Logistic Regression (PYQ favorite: Q7a)

They are mathematically almost identical!

A single-layer perceptron computes z = wx + b , then applies a step function (output is exactly

0 or 1).
Logistic Regression computes z = wx + b , then applies a sigmoid function (output is a
probability between 0 and 1).

The only real difference is the activation function used (step vs sigmoid) and how they’re trained
(perceptron uses the perceptron learning rule; logistic regression uses gradient descent on a
probabilistic loss). Structurally, both are “one neuron” models — so yes, there IS a difference, but it
lies specifically in the activation function and the training/loss approach, not in the overall structure.

2.4 Activation Functions

An activation function decides whether/how strongly a neuron “fires.” Without activation functions,
a neural network — no matter how many layers — would behave like a single linear equation
(stacking linear functions just gives another linear function). Activation functions introduce non-
linearity, which lets networks learn complex patterns.

(a) Sigmoid Function

g(z) = 1 / (1 + e^(-z))

Output range: (0, 1) — always positive, good for representing probabilities.

Shape: S-curve.
Derivative: g'(z) = g(z) * (1 - g(z))
Maximum gradient point (PYQ Q10b): The derivative is maximum when g(z) = 0.5 , i.e., at
z = 0 . At that point, g'(z) = 0.5 * 0.5 = 0.25 . So the maximum possible gradient of sigmoid
is 0.25, occurring exactly at z=0.
Problem: For very large positive or negative z, gradient becomes nearly 0 → causes vanishing
gradient (explained in 2.7).

(b) Tanh (Hyperbolic Tangent)

tanh(z) = (e^z - e^(-z)) / (e^z + e^(-z))

Output range: (-1, 1) (PYQ Q8d asks this directly).

Zero-centered (better than sigmoid for hidden layers), but still suffers from vanishing gradient at
 extreme values.

(c) ReLU (Rectified Linear Unit)

ReLU(z) = max(0, z)

Output range: [0, ∞)

If z > 0 → output = z. If z ≤ 0 → output = 0.
Why ReLU causes sparse activation maps (PYQ Q8a): Since ReLU outputs exactly 0 for any
negative input, a large fraction of neurons output 0 for a given input. This means at any time, only
a sparse (few, scattered) subset of neurons are actually “active” (non-zero), which is
computationally efficient and reduces overfitting.
Very popular in modern deep networks because it avoids vanishing gradient for positive values
and is computationally cheap (no exponentials).
Problem: “Dying ReLU” — if a neuron’s weights update such that z is always negative, that
neuron always outputs 0 and stops learning.

(d) Softmax

Used in the output layer for multi-class classification. Converts a vector of raw scores into
probabilities that sum to 1.

softmax(zi) = e^(zi) / Σ e^(zj) [sum over all classes j]

Example: raw scores [2, 1, 0.1] → softmax converts to something like [0.66, 0.24, 0.10] (all positive,
sum = 1).

Quick Comparison Table (great for revision)

Activation Range Used In Main Issue

Sigmoid (0,1) Binary classification output Vanishing gradient

Tanh (-1,1) Hidden layers (older networks) Vanishing gradient

ReLU [0,∞) Hidden layers (modern default) Dying ReLU

Softmax (0,1), sums to 1 Multi-class output layer —

2.5 Multi-Layer Neural Network (MLP) Architecture

A real neural network stacks many neurons into layers:

1. Input Layer — just holds the raw input features (no computation happens here, no weights of its
own into it).
2. Hidden Layer(s) — layers between input and output; this is where the “learning” of complex
patterns happens. Can be one or many (more layers = “deeper” network).
3. Output Layer — produces the final prediction.

Notation used in your PYQ (Q9): a^[1] means the activation (output) of layer 1, W^[1] means the
weight matrix connecting input to layer 1, and so on.

How to read a network diagram

If a network is drawn as: x → a[1] → a[2] → a[3] → a[4] , this means: - Input x feeds into layer 1,
producing a[1] - a[1] feeds into layer 2, producing a[2] - and so on until the final output a[4]
Why weights are matrices, not single numbers (PYQ Q9d)

Because a layer usually has multiple neurons, and each neuron needs its own weight for every input
coming into it. So instead of one number, you need a whole matrix of numbers — rows for output
neurons, columns for input neurons. This lets you compute the entire layer’s output in a single matrix
multiplication instead of looping neuron by neuron.

2.6 Counting Weights and Biases (VERY common numerical — PYQ Q5)

Rule: - Number of biases in a layer = number of neurons in that layer (every neuron has exactly
one bias). - Number of weights between two layers = (neurons in previous layer) × (neurons in
current layer).

Worked Example (exact PYQ pattern)

Network: 3 input neurons → Hidden Layer 1 (8 neurons) → Hidden Layer 2 (8 neurons) → Output layer
(3 neurons)

Biases: - Input layer: 0 biases (input layer never has biases — it doesn’t compute anything) - Hidden
Layer 1: 8 biases - Hidden Layer 2: 8 biases - Output layer: 3 biases - Total biases = 8 + 8 + 3 = 19

Weights: - Input(3) → Hidden1(8): 3 × 8 = 24 - Hidden1(8) → Hidden2(8): 8 × 8 = 64 - Hidden2(8) →

Output(3): 8 × 3 = 24 - Total weights = 24 + 64 + 24 = 112

Which loss/activation for output layer? Since there are 3 output neurons, this is likely a multi-
class classification problem → use Softmax activation at the output layer with Categorical Cross-
Entropy loss.

General formula to remember:

Total weights = Σ (neurons in layer i) × (neurons in layer i+1), for each

consecutive pair of layers
Total biases = Σ (neurons in each layer, EXCLUDING input layer)

2.7 A Simpler Worked Example (matches PYQ Q6 & Q9 style)

Suppose: 3 input units → Hidden Layer 1 → Hidden Layer 2 → 1 output unit.

Weight matrix W[1]: connects input to hidden layer 1. If hidden layer 1 has, say, 5 neurons, W[1]
size = 5×3 (5 neurons, each needing 3 weights — one per input).
Weight matrix W[2]: connects hidden layer 1 (5 neurons) to hidden layer 2 (say 4 neurons) → size
4×5
Weight matrix W[3]: connects hidden layer 2 (4 neurons) to output (1 neuron) → size 1×4

General rule for matrix size: If layer L-1 has n neurons and layer L has m neurons, then W[L] has
size m × n (rows = current layer size, columns = previous layer size).

Given z = Wx + b (PYQ Q10e style)

If z and a have dimension 3×1, and x has dimension n×1: - Since z = Wx + b , and matrix
multiplication requires: (rows of W × columns of W) times (rows of x × columns of x) = (rows of z ×
columns of z) - W must be 3 × n (so that W (3×n) times x (n×1) = 3×1, matching z) - b must be 3 × 1
(same shape as z, since it’s added directly to Wx)

2.8 MNIST Digit Classification Architecture (PYQ Q3 — asked almost every

year)

MNIST = a dataset of handwritten digit images (0–9), each image is 28×28 pixels (grayscale).

Suggested simple deep network architecture: 1. Input layer: Flatten the 28×28 image into a
single vector of 28×28 = 784 neurons. 2. Hidden Layer 1: e.g., 128 neurons, ReLU activation. 3.
Hidden Layer 2: e.g., 64 neurons, ReLU activation. 4. Output Layer: 10 neurons (one per digit 0–9),
Softmax activation (since it’s 10-class classification). 5. Loss function: Categorical Cross-Entropy.

(If asked for a CNN-based architecture instead: Conv layer → Pooling → Conv layer → Pooling → Flatten
→ Dense layer → Softmax output. Use this if the question specifically mentions “deep” or “CNN.”)

2.9 Deep NN Architecture for NLP / Text Classification (PYQ Q4)

Tokenisation = the process of breaking a sentence/text into smaller pieces called tokens (usually
words, sometimes sub-words or characters). This is the FIRST step before feeding text into any neural
network, because networks only understand numbers, not raw text.

Example: “I love deep learning” → tokens: [“I”, “love”, “deep”, “learning”] → then each token is
converted to a number/vector (via one-hot encoding or word embeddings).

Typical Deep NN architecture for NLP: 1. Tokenisation — split text into tokens. 2. Embedding
Layer — converts each token (word) into a dense vector of numbers (captures meaning; similar words
get similar vectors). 3. Sequential Layer — RNN / LSTM / GRU (to capture the order and context of
words, since text is sequential — word order matters!) — or a Transformer in modern systems. 4.
Dense (fully connected) output layer — final classification (e.g., Softmax for sentiment classes:
positive/negative/neutral).

CHAPTER 3: TRAINING NEURAL NETWORKS

This chapter is about HOW a network actually learns — i.e., how weights and biases get adjusted to
make correct predictions.

3.1 The Core Training Loop

1. Forward Pass: Feed input through the network → get prediction

2. Compute Loss: Compare prediction to actual/true answer → get error
3. Backward Pass (Backpropagation): Calculate how much each weight contributed to
the error
4. Update weights: Adjust weights slightly to reduce the error (Gradient Descent)
5. Repeat for many iterations (epochs) until error is small

3.2 Loss Functions (Risk Minimization)

A loss function measures how wrong the model’s prediction is compared to the actual answer. “Risk
minimization” simply means: training = trying to minimize the average loss over all training
examples.

(a) Mean Squared Error (MSE) — for Regression

 MSE = (1/n) * Σ (actual - predicted)^2

Used when output is a continuous number (e.g., predicting house price).

Why MSE is NOT used for binary classification with ANN (PYQ Q1-VIII → answer: TRUE, we
cannot/should not use MSE here): When combined with sigmoid activation, MSE creates a loss
surface that is non-convex and causes very slow learning (the gradient becomes tiny when the
prediction is very wrong, which is the opposite of what we want). Cross-Entropy loss doesn’t have this
problem and is mathematically the correct choice derived from probability theory (Maximum
Likelihood Estimation) for classification.

(b) Binary Cross-Entropy — for 2-class classification

Loss = -[y*log(p) + (1-y)*log(1-p)]

Where y = actual label (0 or 1), p = predicted probability.

(c) Categorical Cross-Entropy — for multi-class classification (>2 classes, one-hot encoded)

Loss = -Σ yi * log(pi) [sum over all classes i]

Since the true label is one-hot encoded (only one class = 1, rest = 0), this formula simplifies to just:
negative log of the predicted probability for the CORRECT class.

Worked Example (exact PYQ Q8b pattern)

4-class classification, one-hot encoded. True class = 3rd class. Softmax output = [0.3, 0.02, 0.6, 0.08]

Since the true class is the 3rd one, only that term survives:

Loss = -log(0.6) = -(-0.5108) = 0.5108

(Using natural log, ln(0.6) ≈ -0.5108, so Loss ≈ 0.511)

Rule to remember: Cross-entropy loss = -log(predicted probability assigned to the TRUE

class) . Ignore all other class probabilities completely.

(d) Which loss for outputs given as ordinal categories like {1,2,3…} (PYQ Q1-II)

If outputs are ordered/discrete category labels (not one-hot, but integer class labels), we use Sparse
Categorical Cross-Entropy (mathematically same as categorical cross-entropy, just a different input
label format — integers instead of one-hot vectors).

3.3 Gradient Descent — The Optimization Engine

Goal: find the weight values that minimize the loss function.

Idea (analogy): Imagine you’re standing on a hilly landscape (the loss surface) in thick fog, and you
want to reach the lowest valley (minimum loss). You can’t see far, but you can feel the slope under
your feet. So you take small steps downhill (in the direction opposite to the slope) repeatedly until you
reach the bottom.

Formula (weight update rule):

 w(t+1) = w(t) - η * dJ/dw

Where: - w(t) = current weight value - η (eta) = learning rate — how big a step to take - dJ/dw =
gradient (slope) of the loss/cost function J with respect to weight w

Worked Example (exact PYQ Q8c pattern)

Cost function: J(w) = 2w² - 4w + 2 , learning rate η = 0.01. Find weight update rule at step t+1.

Step 1: Find the derivative (gradient):

dJ/dw = 4w - 4

Step 2: Apply the update rule:

w(t+1) = w(t) - 0.01 * (4*w(t) - 4)

w(t+1) = w(t) - 0.04*w(t) + 0.04
w(t+1) = 0.96*w(t) + 0.04

This final expression is the weight update rule — you’d plug in the current w(t) value to get the next
weight.

What if learning rate = 0? (PYQ Q10a)

If η = 0, the update becomes w(t+1) = w(t) - 0 = w(t) . The weight never changes — the model
never learns anything, no matter how many iterations you run, because every step size is zero.

Effect of learning rate on weight changes (PYQ Q10d)

If weight changes are observed to be small across successive iterations, the possible causes are: -
Possibility 2 (learning rate small): YES, valid cause — a small η directly means small steps η *
gradient , hence small weight changes. - Possibility 3 (weight change small): This is circular/same
as what’s observed, not an independent “cause” — not a valid justification on its own. - Possibility 1
(learning rate large) and Possibility 4 (weight change large) do NOT explain small observed weight
changes — they’d cause the opposite effect (large jumps, possibly overshooting or diverging). -
Correct answer: Possibility 2 is the primary valid cause (a small learning rate directly and correctly
explains small weight updates). (Note: gradient being near-zero, e.g. near a minimum or due to
vanishing gradients, is also a valid real-world cause even though not listed as an explicit “possibility”
here.)

3.4 Types of Gradient Descent (PYQ Q1-XII)

Type How it works Pros/Cons

Uses ALL training examples to

Batch (Full-batch) Gradient Accurate but very slow for large
compute gradient before each
Descent datasets, memory-heavy
weight update

Stochastic Gradient Descent Uses just ONE training example per Fast but very noisy/unstable path to
(SGD) update minimum

Best of both worlds — faster than

Uses a small batch (e.g., 32, 64
Mini-batch Gradient Descent full-batch, more stable than pure
examples) per update
SGD

True/False: Mini-batch gradient descent is advantageous compared to full-batch gradient

descent → TRUE. It’s faster (doesn’t wait to process the entire dataset for one update), needs less
memory, and the slight “noise” in mini-batches can actually help escape shallow local minima.

3.5 Backpropagation — How the Gradient is Actually Calculated

Backpropagation is the algorithm that computes dJ/dw for EVERY weight in the network, layer by
layer, working backward from the output layer to the input layer, using the Chain Rule of Calculus.

The Big Idea

The loss depends on the output, the output depends on the last hidden layer, that depends on the
previous hidden layer, and so on back to the input. To find how much a weight deep inside the network
affects the final loss, you multiply together (“chain”) all the intermediate derivatives connecting that
weight to the loss.

Chain Rule for a deep network (PYQ Q10c — general formula)

If: J = loss, y = predicted output, d = desired/true output, a = activation (nonlinearity), f = the

network’s linear combination (weighted sum, i.e., z):

The derivative of the loss with respect to any weight w is computed by chaining together partial
derivatives from the loss all the way back to that weight:

dJ/dw = (dJ/dy) * (dy/da) * (da/df) * (df/dw)

In words: (how loss changes with output) × (how output changes with activation) × (how activation
changes with the pre-activation z) × (how z changes with the weight itself). Each of these is an “easy”
local derivative — the magic of backprop is efficiently chaining/reusing them layer by layer instead of
recomputing from scratch.

Delta Rule (PYQ Q7b)

The Delta Rule is a specific, simple learning rule (a special case of gradient descent) used to update
weights, especially historically for single-layer networks:

Δw = η * (target - output) * input

i.e., Δw = η * error * x

In words: the weight change is proportional to the learning rate, the error (how wrong the prediction
was), and the input value itself. This is literally gradient descent applied to a simple squared-error loss
for a linear/single neuron model.

Worked Backprop Example — MLP with 2 hidden layers, 1 output, 3 inputs (PYQ Q6 pattern)

Network: Input(3) → Hidden1 → Hidden2 → Output(1)

To find dJ/dW2 (weights from Hidden Layer 2 to Output), you chain:

dJ/dW2 = dJ/da_out * da_out/dz_out * dz_out/dW2

dJ/da_out = derivative of loss w.r.t. output activation (depends on loss function used)
da_out/dz_out = derivative of the output activation function (e.g., sigmoid derivative = a*(1-a))
dz_out/dW2 = simply the activation from Hidden Layer 2 (since z_out = W2 * a_hidden2 + b)

This “single step” — from output back to the 2nd hidden layer — is exactly what’s being asked when a
question says “show backpropagation for a single step from output to 2nd hidden layer.” You compute
the three pieces above and multiply them.

3.6 Vanishing Gradient Problem (Asked almost every year — PYQ Q1-IX-
related, Q8e, Q11i)

What it is: In deep networks, when backpropagation multiplies MANY small numbers together (chain
rule across many layers), the gradient becomes exponentially smaller as it moves backward toward
earlier layers. Eventually, the gradient becomes so close to zero that early layers stop learning
entirely — their weights barely update.

Why it happens: Sigmoid and Tanh activation functions have derivatives that are always less than 1
(sigmoid’s max derivative is only 0.25, as we calculated earlier). When you multiply many numbers
each less than 1 together (one per layer), the product shrinks toward zero very fast — like multiplying
0.2 × 0.2 × 0.2 × 0.2 = 0.0016.

Example: In a 10-layer network using sigmoid, the gradient reaching layer 1 could be smaller than
0.25^10 ≈ 0.00000095 — essentially zero, so layer 1’s weights never meaningfully update.

How to manage/prevent it (PYQ Q11i explicitly asks “suggest one network which can help”):
1. Use ReLU activation instead of sigmoid/tanh (ReLU’s derivative is exactly 1 for all positive inputs
— no shrinking). 2. Use architectures specifically designed to combat this: LSTM (Long Short-
Term Memory) or GRU for sequence models — they use special “gates” that allow gradients to flow
backward without shrinking. (This is the specific network to name if asked “suggest one
network.”) 3. Residual/Skip connections (ResNet-style) — allow gradients to “skip” layers directly,
bypassing the shrinking chain. 4. Batch Normalization — keeps activations in a well-behaved range,
indirectly helping gradient flow. 5. Proper weight initialization (e.g., Xavier/He initialization).

3.7 Regularization (Preventing Overfitting)

Overfitting = model performs great on training data but poorly on new/unseen (test) data — it has
“memorized” training data instead of learning general patterns.

Regularization = techniques that discourage the model from becoming overly complex/memorizing.

(a) L1 / L2 Regularization

Add a penalty term to the loss function based on the size of the weights, discouraging very large
weight values (large weights often indicate overfitting to specific data points).

Loss_new = Loss_original + λ * Σ|w| (L1)

Loss_new = Loss_original + λ * Σw² (L2)

(b) Dropout (PYQ Q1-I asks this directly)

During training, randomly “turn off” (set to zero) a fraction of neurons in each forward pass
(e.g., 50% of neurons dropped randomly each iteration).

What dropout prevents: It prevents neurons from co-adapting too much (i.e., relying heavily on
specific other neurons always being present). This forces the network to learn more robust,
redundant, and generalizable features rather than memorizing the training data — effectively, it
prevents overfitting.

Think of it like training a sports team where random players sit out each practice — every remaining
player has to learn to perform well regardless of who else is on the field, making the whole team more
robust.

(Note: Dropout is only applied during training; during testing/inference, all neurons are used, typically
with outputs scaled down accordingly.)

(c) Data Augmentation (PYQ Q1-VI)

Artificially creating more training examples by applying realistic transformations to existing data —
e.g., for images: rotation, flipping, cropping, brightness change, adding slight noise.

Trick question — recognizing digit “6” with augmentation, which augmentation to AVOID:
180° rotation (or vertical flip) should NOT be used, because rotating a “6” by 180° makes it look
exactly like a “9”! This would create wrongly-labeled training data and confuse the model. (Similarly,
horizontal flipping can confuse digits too — always think about whether a transformation changes the
meaning/identity of the label.)

(d) Early Stopping

Stop training once performance on a validation set starts getting worse (even if training loss keeps
improving) — this is a sign of overfitting beginning.

3.8 Model Selection & Evaluation Metrics

Confusion Matrix

A table comparing actual vs predicted classes. For binary classification:

Predicted Positive Predicted Negative

Actual Positive True Positive (TP) False Negative (FN)

Actual Negative False Positive (FP) True Negative (TN)

“All off-diagonal elements are zero” (PYQ Q7g): This means there are ZERO False Positives and
ZERO False Negatives — every single prediction matched the actual label exactly. Inference: the
classifier achieved 100% (perfect) accuracy on this dataset — it made no misclassifications at all.

Accuracy vs F1-Score (PYQ Q7f)

Accuracy = (TP+TN) / (TP+TN+FP+FN)

Precision = TP / (TP+FP)
Recall = TP / (TP+FN)
F1-Score = 2 * (Precision*Recall) / (Precision+Recall)

Why F1-score is often better than accuracy: Accuracy can be misleading on imbalanced
datasets. Example: if 95% of emails are “not spam” and only 5% are “spam,” a lazy model that always
predicts “not spam” gets 95% accuracy — but it’s useless (catches 0% of actual spam). F1-score, being
the harmonic mean of Precision and Recall, punishes models that ignore the minority class, giving
a much more honest picture of performance, especially when classes are imbalanced or both false
positives and false negatives matter.

3.9 Other Common Conceptual Questions in This Chapter

If logistic regression overfits, what do you do? (PYQ Q7c) Apply regularization (L1/L2), reduce
model complexity/features, get more training data, or use cross-validation to tune the regularization
strength.
Dividing weight vector W by 2 (no bias) — effect on accuracy? (PYQ Q1-VII) For logistic
regression, halving W scales down z = Wx by half, which changes the predicted probability (moves it
closer to 0.5, less confident), BUT since the decision boundary (where z=0, i.e., where we switch
from predicting class 0 to class 1) doesn’t change — z=0 stays z=0 even after scaling — the actual
classification decisions for every point remain the same. So the test accuracy A remains
UNCHANGED.

Nonlinearly separable data with linear decision boundary — possible? (PYQ Q7d) No, not
possible directly — a single linear boundary (straight line/hyperplane) cannot correctly separate data
that isn’t linearly separable, by definition. However, you CAN handle it by: transforming data into a
higher-dimensional space where it becomes linearly separable (kernel trick, like in SVM), or by using a
multi-layer neural network with non-linear activation functions, which can learn non-linear decision
boundaries.

Combining 5 logistic regression models into a strong classifier? (PYQ Q7e) Yes, possible —
this is the idea behind ensemble methods (e.g., bagging, boosting, or simple majority voting). Even if
each individual logistic regression model is a “weak” learner, combining their predictions (e.g.,
averaging probabilities or majority voting) typically produces a stronger, more robust classifier than
any single model alone — this is the core principle behind Random Forests and boosting algorithms
like AdaBoost.

10 sets of non-linearly separable, UNLABELLED data — how to classify? (PYQ Q7h) Since there
are no labels, this is NOT a classification problem in the supervised sense — it’s an unsupervised
learning / clustering problem. Since the data is non-linearly separable, use a non-linear
clustering approach such as: Kernel K-Means, DBSCAN (density-based), Spectral Clustering, or an
Autoencoder (deep learning approach) to first learn a non-linear feature representation, then cluster
in that transformed space.

CHAPTER 4: CONDITIONAL RANDOM FIELDS (CRF) &

RELATED SEQUENCE MODELS

This chapter deals with sequence/structured prediction problems — where the output isn’t a single
label but a sequence of labels (e.g., tagging each word in a sentence with its part of speech).

4.1 Why We Need These Models

Regular classifiers treat each prediction independently. But in sequences (like sentences, DNA,
speech), neighboring elements are related — e.g., in POS tagging, if the previous word was “the” (a
determiner), the next word is very likely a noun or adjective, not a verb. CRFs and HMMs are designed
to model this sequential dependency.

4.2 Markov Network & Markov Property

Markov Property: The future depends only on the present state, not on the entire past history. In
sequence terms: “the label at position t depends only on the label at position t-1” (for a 1st-order
Markov model), not on labels way back at t-5, t-6, etc.

A Markov Network (a.k.a. Markov Random Field) is a graphical model representing dependencies
between random variables using an undirected graph — connections don’t have a “cause → effect”
direction like Bayesian Networks; instead, they just represent mutual dependency/compatibility.
4.3 Hidden Markov Model (HMM)

An HMM models sequences where: - There’s a sequence of hidden states (not directly observable) —
e.g., true POS tags. - Each hidden state produces an observable output — e.g., the actual words.

Key components: 1. Transition probabilities — probability of moving from one hidden state to
another (e.g., P(Noun → Verb)). 2. Emission probabilities — probability of a hidden state producing a
particular observation (e.g., P(“dog” | Noun)). 3. Initial state probabilities — probability of starting
in each state.

Three Classic HMM Problems and Their Algorithms

Problem Question Answered Algorithm Used

What is the probability of an

Forward Algorithm (PYQ Q1-III
Likelihood/Evaluation observed sequence given the
asks this)
model?

What is the most likely hidden state

Decoding Viterbi Algorithm
sequence given observations?

How do we estimate the model’s

Learning Baum-Welch Algorithm (EM-based)
parameters from data?

PYQ Q1-III: “Which algorithm is used for likelihood computation in HMM?” → Answer:
Forward Algorithm (sometimes Forward-Backward algorithm is also referenced for related
computations, but pure likelihood computation = Forward Algorithm).

PYQ Q1-IX: “Viterbi Algorithm is used for decoding, i.e., to find hidden sequence. True or
False?” → Answer: TRUE. The Viterbi Algorithm is a dynamic programming algorithm that efficiently
finds the single most probable sequence of hidden states given the observed sequence, instead of
naively checking every possible combination (which would be exponentially expensive).

4.4 Linear Chain CRF

A linear-chain CRF is a specific, simpler structure of CRF used heavily in NLP, where labels are
arranged in a straight sequence (a “chain”) — each label depends on its neighboring labels only,
matching how words appear one after another in a sentence.

Key difference from HMM: HMM is a generative model (models P(observations, hidden states) —
how the data was “generated”). CRF is a discriminative model (directly models P(hidden states |
observations) — doesn’t try to model how observations were generated, just directly learns to predict
the labels given the data). This generally makes CRFs more accurate for labeling tasks since they don’t
waste effort modeling the observation distribution.

4.5 Partition Function

In CRF, the partition function (Z) is a normalization constant that ensures all the probabilities for
different possible label sequences sum to 1. It’s computed by summing the “compatibility scores” of
every possible label sequence:

Z = Σ (over all possible label sequences y) exp(score(y, x))

It’s often computationally expensive to calculate directly (since there are exponentially many possible
sequences), so dynamic programming (similar in spirit to the Forward algorithm) is used to compute it
efficiently.
4.6 Belief Propagation

An algorithm used to perform inference (compute probabilities/marginals) in graphical models like

Markov Networks, by passing “messages” (belief estimates) between connected nodes in the graph
until the beliefs converge/stabilize. Useful for computing marginal probabilities without exhaustively
enumerating every possible combination of states.

4.7 Training CRFs

CRFs are trained by maximizing the (log-)likelihood of the correct label sequences in the training data,
typically using gradient-based optimization (like gradient descent) — similar in spirit to how neural
networks are trained, but the gradient computation involves the partition function and message-
passing algorithms (like belief propagation) rather than simple backpropagation.

4.8 Entropy (PYQ Q1-XI relates conceptually)

Entropy measures the “uncertainty” or “randomness” in a probability distribution.

H(X) = -Σ p(x) * log(p(x))

High entropy = very uncertain/unpredictable (e.g., a fair coin flip — 50/50 — has maximum
entropy for 2 outcomes).
Low entropy = very predictable/certain (e.g., a biased coin that’s 99% heads has low entropy).

In the context of CRFs/HMMs and classification generally, entropy relates closely to cross-entropy loss
(Chapter 3) — cross-entropy essentially measures how “surprised” the model is by the true labels,
given its predicted probability distribution.

CHAPTER 5: DEEP LEARNING ARCHITECTURES

5.1 Deep Feedforward Networks (Recap + Extension)

Simply feedforward networks (Chapter 2) with many hidden layers stacked together. “Deep” refers
specifically to having multiple hidden layers, allowing the network to learn a hierarchy of features —
early layers learn simple patterns (edges in an image), later layers combine these into complex
patterns (shapes → objects).

5.2 Convolutional Neural Networks (CNN)

CNNs are specialized deep networks designed primarily for image data (though also used for other
grid-like data).

Why not just use a regular (fully-connected) network for images?

A 28×28 image has 784 pixels; a fully-connected layer connecting this to even 100 hidden neurons
needs 78,400 weights — and this explodes further for realistic, larger images (e.g., 224×224×3 color
images = 150,528 inputs!). This is computationally wasteful AND doesn’t respect the spatial
structure of images (nearby pixels are related; a fully connected layer ignores this and treats every
pixel independently).
Key CNN Components

(a) Convolution Layer — Instead of connecting every input to every neuron, a small filter/kernel
(e.g., 3×3 grid of weights) slides across the image, computing a weighted sum at each position. This
filter detects a specific local pattern (like a vertical edge) anywhere in the image (this property is
called “weight sharing” — PYQ Q1-IV — the SAME filter weights are reused at every position,
dramatically reducing the number of parameters compared to a fully connected layer. Weight sharing
being a procedure to reduce parameters → TRUE).

(b) Stride — how many pixels the filter moves each step (stride=1 moves 1 pixel at a time, stride=2
skips every other pixel, producing a smaller output). Stride and number of filters are treated as
hyperparameters (PYQ Q1-X) → TRUE (they’re set by the designer before training, not learned
automatically like weights).

(c) Pooling Layer (PYQ Q2 — asked almost every year, know this WELL) Pooling reduces the
spatial size (width/height) of the feature map, which reduces computation and helps make the network
more robust to small shifts/distortions in the image.

Max Pooling: Takes the MAXIMUM value from each small window (e.g., 2×2 region).
Example: window = [[1,3],[2,4]] → Max Pooling output = 4 (the largest value)
Average Pooling: Takes the AVERAGE value from each small window.
Example: window = [[1,3],[2,4]] → Average Pooling output = (1+3+2+4)/4 = 2.5

Max pooling is more commonly used in practice since it tends to preserve the strongest/most important
features (like edges) better than averaging.

Full CNN Architecture Pattern

Input Image → [Convolution → Activation(ReLU) → Pooling] × several times → Flatten →

Fully Connected Layer(s) → Softmax Output

5.3 Recurrent Neural Networks (RNN)

RNNs are specialized for sequential data (text, time series, speech) where order matters and there
can be dependencies across time.

Why not a standard feedforward network for sequences? (PYQ Q11c)

A standard fully-connected network treats each input as independent and requires a fixed input size.
But sequences (like sentences) can have variable length, and crucially, standard networks have no
memory — they cannot use information from earlier in the sequence to inform predictions about later
elements. Since word order and context matter enormously in language (and time matters in time-
series), a standard network cannot capture these sequential dependencies.

How RNN solves this (PYQ Q11d)

An RNN processes a sequence one element at a time, and crucially, it maintains a hidden state
(memory) that gets updated at every time step and is passed forward to the next step. This hidden
state acts like a summary of everything seen so far in the sequence.

RNN Architecture (standard, same input & output length):

x<1> → [RNN cell] → h<1> → y<1>

↓ (hidden state passed forward)
x<2> → [RNN cell] → h<2> → y<2>
 ↓
x<3> → [RNN cell] → h<3> → y<3>
... and so on

At each time step t : h<t> = activation(Wxh * x<t> + Whh * h<t-1> + b) , and this same set of
weights (Wxh, Whh) is reused (shared) at every time step — similar in spirit to weight sharing in
CNNs.

One-Hot Encoding for Words (PYQ Q11a, Q11b)

To feed words into a network, each word is represented as a one-hot vector: a vector as long as the
vocabulary size, with a 1 at the position of that word’s index, and 0 everywhere else.

Worked Example: Vocabulary size = 10,000. Word “Aaron” is at position 2, “Bengio” at position 1229,
“Goodfellow” at position 2048.

If sentence is: “The best book of deep learning is written by Goodfellow, Aaron and Bengio” and we’re
building x<10>, x<11>, x<12> (the 10th, 11th, 12th words in this sentence — matching positions of
Goodfellow, Aaron, Bengio respectively based on word order): - x<10> = one-hot vector of size 10,000,
with 1 at position 2048 (Goodfellow), 0 elsewhere. - x<11> = one-hot vector of size 10,000, with 1 at
position 2 (Aaron), 0 elsewhere. - x<12> = one-hot vector of size 10,000, with 1 at position 1229
(Bengio), 0 elsewhere.

(Match each word in the sentence to its position in the vocabulary, then place a 1 at exactly that index
in an otherwise all-zero vector of length = vocabulary size.)

Named Entity Recognition Output Vector (PYQ Q11a-related)

For a task like recognizing person names in a sentence, the RNN typically outputs a binary vector
(same length as the number of words in the sentence) where each position is 1 if that word is part of
a person’s name, 0 otherwise.

Example sentence: “The best book of deep learning is written by Goodfellow, Aaron and Bengio”
Output vector (word by word): [0,0,0,0,0,0,0,0,0,1,1,0,1] (1s at the positions of “Goodfellow,”
“Aaron,” and “Bengio” — the actual names).

RNN Limitations & LSTM

Plain (vanilla) RNNs suffer badly from the vanishing gradient problem (Chapter 3.6) over long
sequences — they struggle to remember information from many steps back (“long-term
dependencies”). LSTM (Long Short-Term Memory) networks solve this using special gates (forget
gate, input gate, output gate) that control what information to keep, update, or discard from the
memory — allowing gradients to flow much better across long sequences.

5.4 Deep Belief Network (DBN)

A Deep Belief Network is a generative deep learning model composed of multiple stacked layers of
Restricted Boltzmann Machines (RBMs). It’s trained in two stages: 1. Pre-training
(unsupervised): Each RBM layer is trained one at a time, layer-by-layer, to learn to reconstruct its
input — this initializes the weights sensibly before… 2. Fine-tuning (supervised): the whole network
is fine-tuned together using labeled data (e.g., with backpropagation) for the final task (like
classification).

DBNs were historically important (helped revive interest in deep learning in the mid-2000s by solving
the “how do we train deep networks at all” problem before better techniques like ReLU/better
initialization made this less necessary), though they’re less commonly used in cutting-edge systems
today compared to CNNs/RNNs/Transformers.

5.5 Global Minima and Gradient Descent Certainty (PYQ Q8e — second
half)

When is gradient descent CERTAIN to find the global minimum? Gradient descent is guaranteed
to find the global minimum only when the loss/cost function is convex (has a single bowl shape,
with only ONE minimum — no other local dips). Linear regression’s MSE loss, for example, is convex.
However, deep neural networks have highly non-convex loss surfaces (many hills, valleys, and
saddle points), so standard gradient descent is generally NOT guaranteed to find the true global
minimum — it might get stuck in a local minimum or saddle point instead. (In practice, for large deep
networks, most local minima found tend to be “good enough,” and techniques like momentum, Adam
optimizer, and proper learning rate schedules help escape poor local minima/saddle points.)

CHAPTER 6: DEEP LEARNING RESEARCH &

APPLICATIONS

This chapter is more conceptual/applied — covers WHERE deep learning is used in the real world.

6.1 Object Recognition

The task of identifying what objects are present in an image (and often, WHERE they are — via
bounding boxes, called “object detection”). Powered primarily by CNNs. Modern architectures:
ResNet, YOLO (You Only Look Once, for real-time detection), Faster R-CNN.

6.2 Sparse Coding

A technique (also relevant to representation learning generally) where data is represented using only
a few active (non-zero) elements out of a large set of possible basis elements/features — i.e., a
“sparse” representation. This is related conceptually to why ReLU (which produces many zero
activations) is useful — sparse representations tend to be more efficient and can capture the “true”
underlying structure of data more cleanly (similar to how, in language, a sentence uses just a few
relevant words from a huge vocabulary).

6.3 Computer Vision

The broad field of enabling computers to “see” and interpret visual information from images/videos.
Deep Learning (mainly CNNs, and increasingly Vision Transformers) has become the dominant
approach for tasks like: image classification, object detection, image segmentation (labeling every
pixel), face recognition, and image generation (GANs, diffusion models).

6.4 Natural Language Processing (NLP)

The field of enabling computers to understand, interpret, and generate human language. Deep
Learning approach evolved: RNN/LSTM (sequential processing) → Transformers (modern state-of-the-
art, use “attention” mechanisms to look at all words in a sentence simultaneously rather than
sequentially, capturing long-range dependencies much better than RNNs). Applications: machine
translation, sentiment analysis, chatbots, text summarization, named entity recognition (as covered in
Chapter 5).

QUICK-REVISION MASTER TABLE (Everything at a

glance)

Concept One-Line Answer

Dropout prevents Overfitting (co-adaptation of neurons)

Sigmoid range (0, 1)

Tanh range (-1, 1)

ReLU range [0, ∞)

Sigmoid max gradient 0.25, at z = 0

Weight sharing reduces parameters? True

MSE for binary classification with ANN Should NOT be used (True, we cannot)

Viterbi algorithm purpose Decoding — finds most likely hidden state sequence

Forward algorithm purpose Likelihood/evaluation of an observed sequence

Stride & filter count Both are hyperparameters (True)

GPU stands for Graphics Processing Unit

Big Data & DL relation Strongly related — DL needs large data

Mini-batch vs full-batch Mini-batch is advantageous (True)

ReLU causes sparse activation True — negative inputs become exactly 0

Same structure; differ mainly in activation function

Single perceptron vs logistic regression
(step vs sigmoid)

F1-score vs Accuracy F1 better for imbalanced classes

Confusion matrix, off-diagonal = 0 Perfect (100%) classifier

Learning rate = 0 effect No learning occurs at all

Sigmoid/tanh derivatives < 1, multiplied across many

Vanishing gradient caused by
layers

Vanishing gradient fixed by ReLU, LSTM/GRU, residual connections

Weights matrix size (layer L) (neurons in L) × (neurons in L-1)

Biases count per layer = number of neurons in that layer (input layer = 0)

HOW TO ANSWER IN THE EXAM (Strategy Tips)

1. Group A (1 mark, True/False & one-liners): Don’t overthink — these test whether you know
the fact. Revise the Quick-Revision Table above thoroughly.
2. Group B (5 marks, short answer): Always structure as: Definition → Formula/Diagram →
Small example. Pooling and MNIST architecture questions repeat almost every year — know
them cold.
 3. Group C (15 marks, choose any 3 — usually sub-parts of 1-6 marks each): These are
usually numerical + conceptual mixes. Show ALL working steps for numericals (weight/bias
counting, cross-entropy loss, gradient descent updates) — partial marks are given for correct
method even if the final number is slightly off. For conceptual sub-parts (like vanishing gradient,
overfitting fixes), always explain WHY, not just WHAT.

Good luck, Sayan — you’ve got this. Once you’ve read through this, we can do a rapid-fire Q&A drill
using the exact PYQ questions to test retention.