DEEP LEARNING

PCCAIML602 | MAKAUT Exam-Ready Question Bank

All 6 Chapters · LAQs · SAQs · Numericals · MCQs · Mock Paper
Target: 85+ Marks | Prepared by: LastMinuteEngineering

EXAM PATTERN (70 Marks, 3 Hours)

Group Questions Choice Marks

A – MCQ 10 objective questions All compulsory 10 × 1 = 10

B – Short Ans 5 questions set Answer any 3 3 × 5 = 15

C – Long Ans 5 questions set Answer any 3 3 × 15 = 45

Total 70

Chapter Weights: Ch1=5M · Ch2=10M · Ch3=15M · Ch4=15M · Ch5=15M · Ch6=10M

High-weight focus: Chapter 3 (Backprop, Optimizers), Chapter 4 (CRF, HMM), Chapter 5 (CNN, RNN, DBN)
CHAPTER 1 — INTRODUCTION TO DEEP LEARNING [3h · 5M]

Q1.1 What are the various paradigms of machine learning? [LAQ – 15M]
A. Overview
Machine learning paradigms define how a model learns from data. There are four main paradigms:

Paradigm Learning Signal Example Algorithms Example Tasks

Supervised Labeled (x, y) pairs Linear Regression, SVM, DNN Classification, Regression

Unsupervised No labels — find structure K-Means, PCA, Autoencoders Clustering, Gen. models

Reinforcement Reward / penalty signal Q-Learning, PPO, A3C Games, Robotics

Semi-supervised Small labeled + large unlabeled

Label propagation, VAT Web classification

B. Supervised Learning
Goal: learn h: X → Y from training set {(x1,y1), ..., (xn,yn)}. Minimize empirical risk: R = (1/n) Σ L(yi, h(xi)).
Two sub-types: Regression (continuous y, e.g. MSE loss) and Classification (discrete y, e.g.
cross-entropy loss).

C. Unsupervised Learning
No labels. Goal: discover hidden structure. Types: (1) Clustering — K-Means, DBSCAN, Hierarchical. (2)
Dimensionality Reduction — PCA, t-SNE, Autoencoders. (3) Density Estimation — GMM, Normalizing
Flows. (4) Generative Modeling — GAN, VAE.

D. Reinforcement Learning
Agent in environment. At step t: observe state st, take action at, receive reward rt, transition to st+1. Goal:
maximize cumulative reward Gt = Σ γk rt+k. Bellman equation: Q(s,a) = r + γ · maxa' Q(s',a').

E. Deep Learning vs. Shallow ML

Aspect Traditional ML Deep Learning

Feature Engineering Manual (domain expert) Automatic (learned)

Data Requirement Works with small data Needs large datasets

Performance ceiling Saturates with more data Improves with more data

Interpretability Often interpretable Black box

Hardware CPU sufficient GPU / TPU required

Best domain Tabular, structured data Images, text, audio, video

Q1.2 What are the key perspectives and challenges in deep learning? [LAQ – 15M]
A. Why Deep Learning Works
Hierarchy of features: Each layer learns increasingly abstract representations. Layer 1: edges/corners.
Layer 2: shapes/textures. Layer 3: object parts. Final: object classes. Universal Approximation Theorem:
An MLP with one hidden layer can approximate any continuous function.

B. Key Challenges
1. Vanishing Gradient: Gradients shrink exponentially in deep networks (sigmoid/tanh saturate). Fix:
ReLU, batch norm, residual connections.
2. Overfitting: Model memorizes training data. Fix: Dropout, L1/L2 reg, data augmentation, early
stopping.
3. Hyperparameter Tuning: Learning rate, architecture, batch size. Fix: Grid/random search, Bayesian
optimization, AutoML.
 4. Data Hunger: Deep models need millions of samples. Fix: Transfer learning, data augmentation,
self-supervised pre-training.
5. Interpretability: Hard to explain decisions. Fix: LIME, SHAP, Grad-CAM, attention visualization.
6. Computational Cost: Training can take weeks. Fix: Distributed training, mixed precision (FP16),
model pruning.
7. Adversarial Examples: Tiny perturbations fool models. Fix: Adversarial training, certified defenses.

Q1.3 Distinguish between deep learning and machine learning. [SAQ – 5M]
Parameter Machine Learning Deep Learning

Feature extraction Manual — domain knowledge required Automatic — learned from raw data

Model complexity Shallow (1-2 layers typical) Deep (10-1000+ layers)

Data volume Works well with 1K-100K samples Needs 100K-1M+ samples

Performance on unstructured
Limited
data State-of-the-art

Training time Minutes to hours Hours to weeks

Best used for Tabular, small datasets Images, NLP, speech

Examples SVM, Random Forest, KNN CNN, RNN, BERT, GPT

Chapter 1 — Multiple Choice Questions

[MCQ 1.1] Which is NOT a paradigm of machine learning?
A) Supervised B) Unsupervised C) Reinforcement D) Deterministic
Answer: D

[MCQ 1.2] The Bellman equation is used in which paradigm?

A) Supervised B) Unsupervised C) Reinforcement D) Semi-supervised
Answer: C

[MCQ 1.3] Deep learning models require which hardware for efficient training?
A) CPU only B) RAM C) GPU/TPU D) SSD
Answer: C

[MCQ 1.4] Which technique helps when labeled data is scarce?

A) Overfitting B) Transfer learning C) Higher learning rate D) Removing hidden layers
Answer: B
CHAPTER 2 — FEED FORWARD NEURAL NETWORK [6h · 10M]

Q2.1 What is an Artificial Neural Network? Explain its structure and working. [LAQ –
15M]
A. Definition
An Artificial Neural Network (ANN) is a computational model inspired by the biological nervous system. It
consists of interconnected processing units (neurons) organized in layers, communicating via weighted
connections.

B. Biological vs. Artificial Neuron

Biological Neuron Artificial Neuron

Dendrites receive signals Input values x₁, x₂, ..., x_n

Synaptic weights modulate signals Weights w₁, w₂, ..., w_n

Cell body sums inputs Weighted sum: z = Σ w_ix_i + b

Axon fires if threshold exceeded Activation: a = f(z)

Output signal sent to next neuron Output passed to next layer

C. Network Architecture (Diagram)

INPUT LAYER HIDDEN LAYER OUTPUT LAYER

x1 ----w11----> [H1] ---

| \ \------> [O1] --> y-hat
x2 --w21--> [H2] -------/
| / /------> [O2]
x3 ----w31----> [H3] ---

Each node: z = sum(w*x) + b, output = f(z)

D. Mathematical Formulation
For layer l (l = 1, 2, ..., L):
z(l) = W(l) · a(l-1) + b(l) (linear transformation)
a(l) = f(z(l)) (non-linear activation)
Final output: ŷ = a(L). Forward pass computes output for given input x = a(0).

E. Types of Layers
Layer Type Role Notes

Input layer Receives raw features No computation, just passes x

Hidden layer(s) Learns representations One or more; deeper = more abstract features

Output layer Produces prediction Softmax for classification, linear for regression

Q2.2 Explain all major activation functions. Compare their properties. [LAQ – 15M]
A. Purpose of Activation Functions
Without activation functions, a multi-layer network collapses to a single linear transformation: stacking
linear layers gives another linear layer. Activation functions introduce non-linearity, enabling networks to
learn complex, non-linear mappings.

B. Activation Function Comparison

Function Formula Range Derivative Key Issue

Step 1 if z>0 else 0 {0,1} 0 everywhere Not differentiable — cannot use gradient descent

Sigmoid 1/(1+e<super>-z</super>) (0,1) σ(1-σ) Vanishing gradient; not zero-centered

Tanh (e<super>z</super>-e<super>-z</super>)/(e<super>z</super>+e<super>-z</super>)
(-1,1) 1-tanh² Still vanishes; zero-centered (better than sigmoid)

ReLU max(0,z) [0,∞) 0 or 1 Dying ReLU (neuron stuck at 0)

Leaky ReLU max(αz,z), α=0.01 (-∞,∞) α or 1 Fixes dying ReLU; α is hyperparameter

ELU z if z>0; α(e<super>z</super>-1)(-α,∞)

else 1 or αe<super>z</super>
Smooth; mean activation ≈ 0

Softmax e<super>zi</super>/Σe<super>zj</super>
(0,1), sum=1 complex Only for output; multi-class prob

C. Vanishing Gradient in Sigmoid

Sigmoid derivative σ'(z) = σ(z)(1-σ(z)) ≤ 0.25. In a 10-layer network: gradient shrinks by 0.2510 ≈ 10-6.
Early layers learn almost nothing. ReLU derivative = 1 for positive inputs — no shrinkage.

D. Dying ReLU & Fix

If a ReLU neuron receives negative input always, it outputs 0 and gradient = 0 — it never updates (dies).
Fix: Leaky ReLU f(z) = max(0.01z, z) keeps a small gradient for negative inputs. Alternatively, He
initialization prevents most neurons from dying initially.

Q2.3 Explain the Perceptron model and its learning rule. Compare SLP with MLP. [LAQ –
15M]
A. Perceptron (Rosenblatt, 1958)
Simplest neural network: single layer, binary output. Computes: ŷ = step(wTx + b). Decision boundary:
wTx + b = 0 (a hyperplane in input space).

B. Perceptron Learning Algorithm

Initialize: w = 0, b = 0
Repeat until convergence:
For each training example (x_i, y_i):
y_hat = step(w^T * x_i + b)
if y_hat != y_i: # misclassified
w = w + eta*(y_i - y_hat)*x_i
b = b + eta*(y_i - y_hat)
Converges ONLY if data is linearly separable

C. Limitation: XOR Problem

Perceptron cannot solve XOR (Minsky & Papert, 1969): XOR is not linearly separable. Truth table:
(0,0)→0, (0,1)→1, (1,0)→1, (1,1)→0. No single line can separate 0s from 1s.

D. SLP vs MLP
Aspect Single-Layer Perceptron Multi-Layer Perceptron

Layers Input + Output only Input + ≥1 Hidden + Output

Decision boundary Single hyperplane (linear) Non-linear, arbitrary shape

XOR solvable? No Yes (1 hidden layer, 2 neurons)

Learning rule Perceptron rule (delta rule) Backpropagation

Expressiveness Linear functions only Universal approximator

Activation function Step function ReLU, Sigmoid, Tanh

Use case Linearly separable tasks only Any classification/regression

E. Universal Approximation Theorem

Statement: An MLP with one hidden layer of sufficient width and a non-linear activation can approximate
any continuous function on a compact subset of Rn to arbitrary precision (Hornik, 1989). Implication:
MLPs are theoretically capable of solving any problem — the challenge is finding the right weights.

NUMERICAL 2.1 Forward Pass Computation [Exam Favourite]

A 2-layer neural network has: x = [1, 2], W1 = [[0.5, 0.3],[0.2, 0.4]], b1 = [0.1, 0.1], W2 = [[0.6, 0.5]], b2 =
[0.2]. Activation: sigmoid. Compute the output.

Solution:
Step 1: z1 = W1*x + b1
z1[0] = 0.5*1 + 0.3*2 + 0.1 = 0.5+0.6+0.1 = 1.2
z1[1] = 0.2*1 + 0.4*2 + 0.1 = 0.2+0.8+0.1 = 1.1

Step 2: a1 = sigmoid(z1)
a1[0] = 1/(1+e^-1.2) = 1/(1+0.301) = 0.769
a1[1] = 1/(1+e^-1.1) = 1/(1+0.333) = 0.750

Step 3: z2 = W2*a1 + b2
z2 = 0.6*0.769 + 0.5*0.750 + 0.2 = 0.461+0.375+0.2 = 1.036

Step 4: output = sigmoid(z2) = 1/(1+e^-1.036) = 1/(1+0.355) = 0.738

Final output: y_hat = 0.738

Chapter 2 — Multiple Choice Questions

[MCQ 2.1] Which activation function is zero-centered and avoids vanishing gradient best?
A) Sigmoid B) Step C) ReLU D) Softmax
Answer: C

[MCQ 2.2] What is the derivative of sigmoid σ(z)?

A) σ(z) B) 1-σ(z) C) σ(z)(1-σ(z)) D) σ(z)²
Answer: C

[MCQ 2.3] XOR problem proved that single-layer perceptron cannot solve:
A) Linear problems B) Nonlinearly separable problems C) Binary problems D) All problems
Answer: B

[MCQ 2.4] The Universal Approximation Theorem states that an MLP with 1 hidden layer can:
A) Train faster B) Approximate any continuous function C) Avoid overfitting D) Learn without data
Answer: B

[MCQ 2.5] Which initialization is recommended for layers with ReLU activation?
A) Zero init B) All-ones init C) Xavier init D) He init
Answer: D
CHAPTER 3 — TRAINING NEURAL NETWORKS [6h · 15M] HIGH
WEIGHT

Q3.1 Explain the backpropagation algorithm in detail with mathematical derivation.

[LAQ – 15M]
A. Overview
Backpropagation (Rumelhart, Hinton & Williams, 1986) efficiently computes gradients of the loss with
respect to all network parameters using the chain rule of calculus. It consists of two passes: Forward
(compute output) and Backward (compute gradients).

B. Forward Pass
For layer l = 1, 2, ..., L:
z(l) = W(l) · a(l-1) + b(l)
a(l) = f(l)(z(l))
Store all z(l) and a(l) for use in backward pass.

C. Backward Pass
Output layer error (δL):
δ(L) = L f'(z(L))
a
Hidden layer error (δl) — propagate backward:
δ(l) = (W(l+1))T · δ(l+1) f'(z(l))
Gradients of parameters:
∂L/∂W(l) = δ(l) · (a(l-1))T
∂L/∂b(l) = δ(l)
Parameter update (gradient descent):
W(l) ← W(l) − η · ∂L/∂W(l)

D. Chain Rule Derivation

For a 2-layer network with MSE loss L = (y - ŷ)2:
∂L/∂W(1) = ∂L/∂ŷ · ∂ŷ/∂a1 · ∂a1/∂z1 · ∂z1/∂W1
Each term is a local gradient; backprop multiplies them in reverse order (chain rule).

E. Backpropagation Through Time (BPTT) for RNNs

For RNNs, gradients flow backward through time steps. Each step multiplies by the recurrent weight Wh.
If ||Wh|| < 1: gradients vanish. If ||Wh|| > 1: gradients explode. LSTM solves this by learning when to
preserve gradients via the forget gate.

NUMERICAL 3.1 Backpropagation Worked Example [Most Common in MAKAUT]

Network: 1 input, 1 hidden neuron (sigmoid), 1 output neuron (sigmoid), MSE loss. Given: x=0.5, y=0.8,
w1=0.4, w2=0.6, b1=0, b2=0, η=0.5. Perform one forward + backward pass and update weights.
FORWARD PASS:
z1 = w1*x = 0.4*0.5 = 0.2
a1 = sigmoid(0.2) = 1/(1+e^-0.2) = 0.5498
z2 = w2*a1 = 0.6*0.5498 = 0.3299
y_hat = sigmoid(0.3299) = 1/(1+e^-0.3299) = 0.5817
Loss L = (y - y_hat)^2 = (0.8 - 0.5817)^2 = 0.0476

BACKWARD PASS:
dL/dy_hat = -2*(y - y_hat) = -2*(0.2183) = -0.4366
sigmoid_deriv(z2) = y_hat*(1-y_hat) = 0.5817*0.4183 = 0.2433
delta2 = dL/dy_hat * sigmoid_deriv(z2) = -0.4366*0.2433 = -0.1062
dL/dw2 = delta2 * a1 = -0.1062 * 0.5498 = -0.0584
 dL/da1 = delta2 * w2 = -0.1062 * 0.6 = -0.0637
sigmoid_deriv(z1) = a1*(1-a1) = 0.5498*0.4502 = 0.2475
delta1 = dL/da1 * sigmoid_deriv(z1) = -0.0637*0.2475 = -0.0158
dL/dw1 = delta1 * x = -0.0158 * 0.5 = -0.0079

WEIGHT UPDATE (eta=0.5):

w2_new = 0.6 - 0.5*(-0.0584) = 0.6 + 0.0292 = 0.6292
w1_new = 0.4 - 0.5*(-0.0079) = 0.4 + 0.0040 = 0.4040

Q3.2 Explain regularization techniques: L1, L2, Dropout, Batch Normalization. [LAQ –
15M]
A. Need for Regularization
Overfitting: model performs well on training data but poorly on test data (high variance). Regularization
adds constraints or noise to prevent the model from memorizing training data.

B. L2 Regularization (Weight Decay)

Modified loss: Lreg = L + λ · Σ wi2
Effect: penalizes large weights → shrinks all weights toward zero (but rarely exactly zero). Gradient
update: w ← w(1 − 2ηλ) − η·∂L/∂w. Equivalent to Gaussian prior P(w) ~ N(0, 1/2λ) on weights.

C. L1 Regularization (Lasso)
Modified loss: Lreg = L + λ · Σ |wi|
Effect: many weights become exactly zero → sparse model, automatic feature selection. Gradient:
sign(wi). L1 is non-differentiable at w=0 (use subgradient).

D. Dropout (Srivastava et al., 2014)

Training: Each neuron dropped with probability p (typically 0.5)
Randomly set activations to 0; scale others by 1/(1-p)
Test time: Use all neurons; multiply outputs by (1-p)
Effect: Ensemble of 2^n different networks
Forces redundant representations

E. Batch Normalization (Ioffe & Szegedy, 2015)

For mini-batch B = {x1,...,xm}:
μB = (1/m) Σ xi (batch mean)
σB2 = (1/m) Σ (xi − μB)2 (batch variance)
x̂i = (xi − μB) / √(σB2 + ε) (normalize)
yi = γ·x̂i + β (scale and shift — learnable γ, β)
Benefits: stabilizes training, allows higher LR, mild regularization effect, reduces sensitivity to
initialization, speeds convergence 2-10×.

F. Comparison
Method What it controls Key param Produces sparse weights?

L2 Weight magnitude (squared) λ No — shrinks toward 0

L1 Weight magnitude (absolute) λ Yes — exact zeros

Dropout Neuron co-adaptation p (drop prob) No

Batch Norm Layer input distribution ε, γ, β No

Early Stopping Training duration Patience epochs No

Q3.3 Explain optimization algorithms: SGD, Momentum, RMSProp, Adam. [LAQ – 15M]
A. Vanilla Gradient Descent
w ← w − η · wL. Batch GD: use all n samples (stable, slow). SGD: use 1 sample (fast, noisy).
Mini-batch GD: use 32-256 samples (best trade-off, standard in practice).

B. SGD with Momentum

Adds velocity vector v: v ← βv + η L; w ← w − v. Accumulates gradient in consistent directions,
dampens oscillations. β = 0.9 typically. Analogy: ball rolling down hill — builds speed in consistent
direction.

C. RMSProp
Adapts learning rate per parameter. Maintains running average of squared gradients:
E[g2] ← ρ·E[g2] + (1−ρ)·g2
w ← w − (η / √(E[g2] + ε)) · g
Parameters with large gradients get smaller effective LR. Good for RNNs. ρ = 0.9, ε = 10-8.

D. Adam (Adaptive Moment Estimation — Kingma & Ba, 2015)

Combines Momentum (1st moment m) + RMSProp (2nd moment v):
m = beta1*m + (1-beta1)*g # 1st moment (momentum)
v = beta2*v + (1-beta2)*g^2 # 2nd moment (RMSProp)
m_hat = m / (1 - beta1^t) # bias correction
v_hat = v / (1 - beta2^t) # bias correction
w = w - eta * m_hat / sqrt(v_hat + eps)

Defaults: beta1=0.9, beta2=0.999, eps=1e-8, eta=0.001

Why bias correction? m and v are initialized to 0 — they are biased toward 0 in early iterations. Dividing
by (1 − βt) corrects this.

E. Optimizer Comparison
Optimizer Adaptive LR? Memory Convergence Best For

SGD No O(d) Slow Convex, carefully tuned

SGD+Momentum No O(2d) Faster General DL with tuning

AdaGrad Yes (decreasing) O(d) Good early, stalls late Sparse features, NLP

RMSProp Yes O(d) Good RNNs, non-stationary

Adam Yes O(2d) Very fast Default for most DL tasks

AdamW Yes + weight decayO(2d) Very fast Transformers, BERT, GPT

Q3.4 Explain loss functions used in neural networks. [SAQ – 5M]

Loss Function Formula Task Notes

MSE (1/n)Σ(y-ŷ)² Regression Sensitive to outliers

MAE (1/n)Σ|y-ŷ| Regression Robust to outliers

Binary Cross-Entropy -[y·log(ŷ)+(1-y)·log(1-ŷ)] Binary classification With sigmoid output

Categorical CE -Σ y_i·log(ŷ_i) Multi-class With softmax output

Hinge max(0, 1-y·ŷ) SVM / classification Maximum-margin loss

Huber MSE if |e|≤δ else MAE Regression Best of MSE + MAE

Q3.5 What is model selection? Explain cross-validation. [SAQ – 5M]

A. Model Selection
Choosing the best model architecture and hyperparameters. Split data into: Train (learn parameters) →
Validation (select hyperparameters) → Test (final evaluation). Never use the test set during model
selection (data leakage).
B. k-Fold Cross-Validation
Split data into k equal folds (typically k=5 or k=10)
For i = 1 to k:
Use fold i as validation, train on remaining k-1 folds
Record validation error E_i
Final estimate = mean(E_1, ..., E_k)

Advantage: uses all data for both training and validation

k=n: Leave-One-Out CV (expensive but low bias)

Chapter 3 — Short Answer Questions (SAQs)

[SAQ 3.1] What is the vanishing gradient problem and how is it solved?
Gradients shrink exponentially when backpropagating through sigmoid/tanh layers (derivative ≤ 0.25).
Early layers learn very slowly. Solutions: (1) ReLU activation (gradient=1 for positive inputs). (2) Residual
connections (skip gradients). (3) Batch normalization (stable gradient flow). (4) He/Xavier initialization.

[SAQ 3.2] What is the learning rate? How does it affect training?
Learning rate η scales the gradient step: w ← w − η· L. Too high: overshoots minimum, training
diverges. Too low: extremely slow convergence, may get stuck. Solutions: learning rate schedulers (step
decay, cosine annealing, warmup), or adaptive optimizers (Adam adjusts LR per parameter
automatically).

[SAQ 3.3] What is early stopping?

Monitor validation loss during training. Stop when validation loss starts increasing (even if training loss
still decreasing). Save model at best validation checkpoint. Prevents overfitting without explicit
regularization penalty. Cheap to implement.

[SAQ 3.4] Define empirical risk minimization (ERM).

ERM: minimize training loss as proxy for true risk. True risk R(f) = E[L(y,f(x))] over data distribution.
Empirical risk R_emp = (1/n)Σ L(y_i, f(x_i)) — average training loss. ERM finds f* = argmin R_emp.
Generalization gap = R(f*) - R_emp(f*).

[SAQ 3.5] What is the difference between batch, mini-batch, and stochastic gradient
descent?
Batch GD: gradient over entire training set — accurate, very slow per update, memory intensive. SGD:
gradient from 1 random sample — fast updates, noisy, may oscillate. Mini-batch GD: gradient from m
samples (32-256) — best of both: vectorized GPU ops, reasonable noise. Mini-batch is the standard in
practice.

Chapter 3 — Multiple Choice Questions

[MCQ 3.1] Backpropagation uses which mathematical rule to compute gradients?
A) Product rule B) Chain rule C) Power rule D) Quotient rule
Answer: B

[MCQ 3.2] Which optimizer combines momentum and adaptive learning rates?
A) SGD B) AdaGrad C) Adam D) Batch GD
Answer: C

[MCQ 3.3] L1 regularization produces what kind of solution?

A) All large weights B) Dense weights C) Sparse weights (zeros) D) Negative weights
Answer: C

[MCQ 3.4] Batch normalization normalizes input of each layer across:

 A) All training data B) The mini-batch C) Each neuron independently D) The test set
Answer: B

[MCQ 3.5] Which loss function is used for multi-class classification with softmax output?
A) MSE B) Hinge C) Categorical Cross-Entropy D) MAE
Answer: C

[MCQ 3.6] In dropout with p=0.5, what fraction of neurons are active during training?
A) 0.25 B) 0.5 C) 0.75 D) 1.0
Answer: B
CHAPTER 4 — CONDITIONAL RANDOM FIELDS & HMM [9h · 15M] HIGH
WEIGHT

Q4.1 What is a Conditional Random Field (CRF)? Explain Linear Chain CRF with
applications. [LAQ – 15M]
A. Definition
Conditional Random Field (CRF) (Lafferty et al., 2001) is a discriminative probabilistic graphical model
for structured prediction. It models the conditional distribution P(y|x) directly (unlike HMMs which model
the joint P(x,y)). CRFs capture dependencies between output labels without restrictive independence
assumptions on inputs.

B. Linear Chain CRF

Simplest CRF — the output graph is a chain. Given input sequence x = (x1,...,xn), predicts label
sequence y = (y1,...,yn).
x1 x2 x3 x4 <-- input tokens (observed)
| | | |
y1 -y2 -y3 -y4 <-- labels (chain: each yi linked to yi-1)

C. Probability Model
P(y|x) = (1/Z(x)) · exp( Σt Σk λk · fk(yt, yt-1, x, t) )
Where: fk = feature functions (hand-crafted or learned), λk = learned weights, Z(x) = partition function
(normalizer).

D. Partition Function Z(x)

Z(x) = Σy exp(Σt Σk λkfk(yt,yt-1,x,t))
Sums over all possible label sequences — exponential in sequence length! Computed efficiently using
the forward algorithm in O(n · |Y|²) time.

E. Training CRFs
Maximize log-likelihood: L(λ) = Σi log P(yi|xi) − (λ²/2σ²) [L2 reg]. Gradient: ∂L/∂λk = Edata[fk] − Emodel[fk].
Optimize with L-BFGS or gradient descent.

F. Inference (Decoding)
Find y* = argmaxy P(y|x). Solved using Viterbi algorithm (dynamic programming): O(n · |Y|²).

G. Applications
Application Input x Output y

NER (Named Entity Recognition) Word tokens PERSON / ORG / LOC / O

POS Tagging Words in sentence NOUN / VERB / ADJ etc.

Handwriting recognition Pixel segments Character labels

Gene sequence labeling DNA bases Exon / Intron labels

Chunking Words Phrase types (NP, VP, ...)

Q4.2 Explain the Hidden Markov Model (HMM). State the three fundamental problems.
[LAQ – 15M]
A. HMM Definition
An HMM is a generative probabilistic model for sequences. States are hidden (unobserved);
observations are emitted from states. Two key assumptions:
Markov Property: P(qt|q1,...,qt-1) = P(qt|qt-1). Future depends only on present state.
Output Independence: P(xt|q1,...,qn,x1,...,xn) = P(xt|qt). Observation depends only on current state.

B. HMM Parameters λ = (π, A, B)

 Parameter Symbol Definition

Initial distribution π π_i = P(q₁=i), probability of starting in state i

Transition matrix A A_ij = P(q_t=j | q_t-1=i), prob of moving from i to

Emission matrix B B_jk = P(x_t=k | q_t=j), prob of observing k in sta

C. Three Fundamental Problems

Problem Algorithm Complexity Purpose

1. Evaluation: P(x|λ)=? Forward Algorithm O(n·|Q|²) How likely is observed sequence?

2. Decoding: y*=argmax P(y|x,λ)

Viterbi Algorithm O(n·|Q|²) Best hidden state sequence

3. Learning: λ*=argmax P(x|λ)

Baum-Welch (EM) O(n·|Q|²) per iter Estimate parameters from data

D. Forward Algorithm
αt(i) = P(x1,...,xt, qt=i | λ)
Initialization: α1(i) = πi · Bi(x1)
Recursion: αt(j) = [Σi αt-1(i) · Aij] · Bj(xt)
Final: P(x|λ) = Σi αT(i)

E. Viterbi Algorithm
δt(i) = max probability of ending in state i at time t.
Initialization: δ1(i) = πi · Bi(x1)
Recursion: δt(j) = maxi[δt-1(i) · Aij] · Bj(xt)
Backtrack: q* = argmaxi δT(i), trace back pointers.

NUMERICAL 4.1 Viterbi Algorithm [Common in MAKAUT]

HMM with 2 states (Hot=H, Cold=C) and 2 observations (3 ice-creams, 1 ice-cream). π = [0.8, 0.2], A =
[[0.7,0.3],[0.4,0.6]], B(H,3)=0.4, B(H,1)=0.2, B(C,3)=0.1, B(C,1)=0.5. Observation sequence: x = [3, 1].
Find most likely state sequence.
t=1, x1=3:
delta_1(H) = pi(H)*B(H,3) = 0.8*0.4 = 0.320
delta_1(C) = pi(C)*B(C,3) = 0.2*0.1 = 0.020

t=2, x2=1:
delta_2(H) = max[delta_1(H)*A(H,H), delta_1(C)*A(C,H)] * B(H,1)
= max[0.320*0.7, 0.020*0.4] * 0.2
= max[0.224, 0.008] * 0.2 = 0.224 * 0.2 = 0.0448 (from H)

delta_2(C) = max[delta_1(H)*A(H,C), delta_1(C)*A(C,C)] * B(C,1)

= max[0.320*0.3, 0.020*0.6] * 0.5
= max[0.096, 0.012] * 0.5 = 0.096 * 0.5 = 0.0480 (from H)

Best final state: max(0.0448, 0.0480) = 0.0480 -> C at t=2

Backtrack: state at t=1 that led to C = H
Most likely sequence: [Hot, Cold]

Q4.3 Explain Markov Networks and Belief Propagation. [LAQ – 15M]

A. Markov Random Field (MRF) / Markov Network
An undirected probabilistic graphical model representing a joint distribution as a product of potential
functions over cliques:
P(X) = (1/Z) · ΠC φC(XC)
where C = cliques in the graph, φC ≥ 0 are potential functions, Z is the partition function.
Example MRF (pairwise):
 X1 -- X2 -- X3
| |
X4 ----------X5
P(X) = (1/Z)*phi(X1,X2)*phi(X2,X3)*phi(X3,X5)*phi(X1,X4)*phi(X4,X5)

B. Belief Propagation (Message Passing)

Exact on trees; Loopy BP (approximate) on graphs with cycles.
Message from node i to neighbor j: μi→j(xj) = Σxi φi(xi) · φij(xi,xj) · Πk μk→i(xi)
N(i)\j
Belief at node j: bj(xj) φj(xj) · Πk μk→j(xj)
N(j)
C. HMM vs CRF vs MRF
Aspect HMM CRF MRF

Graph type Directed (DAG) Undirected (chain) Undirected (general)

Models P(x,y) — generative P(y|x) — discriminative P(X) — joint

Independence Strong (output indep) No restriction on x Markov blanket only

Features Emission probs only Rich feature functions Potential functions

Training Baum-Welch (EM) Gradient descent MCMC or EM

Q4.4 Explain Shannon Entropy, KL Divergence, and Cross-Entropy. [LAQ – 15M]

A. Shannon Entropy
H(X) = −Σi P(xi) · log2 P(xi) [bits]
Interpretation: Average number of bits needed to encode outcomes of X. High entropy = high uncertainty
= more bits needed. Properties: H(X) ≥ 0; H = 0 iff X is deterministic; H maximized by uniform
distribution: Hmax = log2|X|.

B. KL Divergence (Relative Entropy)

KL(P || Q) = Σi P(xi) · log(P(xi) / Q(xi))
Interpretation: Information lost when Q approximates P. Properties: KL(P||Q) ≥ 0 (Gibbs inequality);
equals 0 iff P=Q; NOT symmetric (KL(P||Q) ≠ KL(Q||P)); not a true distance metric.

C. Cross-Entropy
H(P, Q) = −Σi P(xi) · log Q(xi) = H(P) + KL(P||Q)
In DL: P = true labels (one-hot), Q = model predictions (softmax). Minimizing cross-entropy ≡ minimizing
KL(P||Q) ≡ Maximum Likelihood Estimation (MLE).

D. Mutual Information
I(X;Y) = H(X) − H(X|Y) = H(Y) − H(Y|X) = KL(P(X,Y) || P(X)P(Y))
Measures how much X and Y share information. I(X;Y)=0 iff X and Y are independent.

E. Summary Table
Measure Formula Range DL Use

Entropy H(X) −Σ P log P [0, log|X|] Measure uncertainty

Joint Entropy H(X,Y) −ΣΣ P(x,y) log P(x,y) [0,∞) Source coding

Cross-Entropy H(P,Q) −Σ P log Q (0,∞) Classification loss function

KL Divergence Σ P log(P/Q) [0,∞) VAE loss, distribution matching

Mutual Info I(X;Y) H(X)−H(X|Y) [0,min(H(X),H(Y))] Feature selection, IB theory

NUMERICAL 4.2 Entropy Calculation

A fair coin: P(H)=0.5, P(T)=0.5. A biased coin: P(H)=0.9, P(T)=0.1. Compute entropy for both.
Fair coin:
 H = -(0.5*log2(0.5) + 0.5*log2(0.5))
= -(0.5*(-1) + 0.5*(-1)) = -(-0.5-0.5) = 1.0 bit
Maximum entropy -- completely uncertain

Biased coin:
H = -(0.9*log2(0.9) + 0.1*log2(0.1))
= -(0.9*(-0.152) + 0.1*(-3.322))
= -(-0.137 - 0.332) = 0.469 bits
Low entropy -- mostly predictable (usually Heads)

Chapter 4 — Short Answer Questions

[SAQ 4.1] Differentiate HMM and CRF.
HMM: generative model, models P(x,y) jointly, assumes output independence given states, strong
independence on inputs, trained with Baum-Welch (EM). CRF: discriminative, models P(y|x) directly, no
independence assumption on inputs, supports rich overlapping feature functions, trained with gradient
descent/L-BFGS. CRFs outperform HMMs on most NLP tasks given sufficient data.

[SAQ 4.2] What is the partition function and why is it difficult to compute?
Z(x) = Σ_y exp(Σ λ_k f_k(y,x)) normalizes P(y|x) to sum to 1 over all label sequences. Difficult because
|y| is exponential in sequence length n (|Y|^n possible sequences). For linear-chain CRFs, efficiently
computed in O(n·|Y|²) using the forward algorithm (dynamic programming, similar to HMM forward
algorithm).

[SAQ 4.3] State the Markov property.

First-order Markov: P(X_t | X_1,...,X_{t-1}) = P(X_t | X_{t-1}). Future depends only on present, not all
past. k-th order: P(X_t|past) = P(X_t|X_{t-1},...,X_{t-k}). Key property enabling efficient algorithms
(forward, Viterbi) by factoring the joint probability into a product of local transition probabilities.

Chapter 4 — Multiple Choice Questions

[MCQ 4.1] CRF is a ________ model that directly models P(y|x):
A) Generative B) Discriminative C) Parametric D) Non-parametric
Answer: B

[MCQ 4.2] Which algorithm solves the decoding problem in HMM?

A) Baum-Welch B) Forward algorithm C) Viterbi D) Belief propagation
Answer: C

[MCQ 4.3] Entropy of a fair die with 6 outcomes equals (log base 2):
A) 1 bit B) 2 bits C) 2.585 bits D) 6 bits
Answer: C

[MCQ 4.4] Belief Propagation is exact on:

A) Any graph B) Fully connected graphs C) Tree-structured graphs D) Dense graphs
Answer: C

[MCQ 4.5] KL divergence KL(P||Q) equals 0 when:

A) P and Q are independent B) P = Q everywhere C) P is uniform D) Q is uniform
Answer: B
CHAPTER 5 — DEEP LEARNING ARCHITECTURES [6h · 15M] HIGH
WEIGHT

Q5.1 Explain Convolutional Neural Networks (CNN) with architecture, operations, and
applications. [LAQ – 15M]
A. Motivation for CNN
A fully connected layer for a 224×224×3 image = 150,528 inputs → millions of parameters. CNNs exploit
spatial locality and parameter sharing (same filter across all positions) to dramatically reduce parameters
while capturing spatial patterns.

B. Core Operations
Operation Description Purpose

Convolution Slide filter w×h×C over input; output = dot product

Detect
+ bias
local features (edges, curves)

ReLU max(0, z) after each conv Non-linearity

Pooling (Max) Take max in each k×k window (stride k) Spatial downsampling, translation invariance

Flatten Convert 3D feature maps to 1D vector Connect to fully connected layers

Fully Connected Dense layer → classification Final decision

C. Architecture Diagram
Input [H x W x C]
| Conv(K filters, f x f)
Feature Maps [H1 x W1 x K]
| ReLU
| MaxPool (2x2, stride 2)
Feature Maps [H1/2 x W1/2 x K]
| (more Conv+Pool blocks)
| Flatten
Vector [n]
| FC Layers
Output [num_classes] + Softmax

D. Output Size Formula

Wout = (Win − f + 2P) / S + 1, Hout = (Hin − f + 2P) / S + 1
Win=input size, f=filter size, P=padding, S=stride. Parameters per conv layer = f × f × Cin × K + K
(biases).

E. Famous Architectures
Architecture Year Key Innovation Depth

LeNet-5 1998 First CNN for digit recognition (MNIST) 7 layers

AlexNet 2012 Deep CNN on GPU; ReLU; Dropout 8 layers

VGGNet 2014 Stack of 3×3 conv filters; simple uniform architecture 16/19 layers

GoogLeNet/Inception2014 Inception modules (parallel multi-scale filters) 22 layers

ResNet 2015 Residual (skip) connections; solves vanishing gradient 50/101/152

DenseNet 2017 Dense connections (each layer to all later layers) 121-264

NUMERICAL 5.1 CNN Output Size Calculation [Very Common in MAKAUT]

Input: 32×32×3 image. Apply Conv layer: 16 filters, 5×5, padding=0, stride=1. Then MaxPool: 2×2,
stride=2. Find output size and number of parameters.
After Conv:
W_out = (32 - 5 + 2*0) / 1 + 1 = 28
H_out = (32 - 5 + 2*0) / 1 + 1 = 28
Depth = 16 filters
 Output size: 28 x 28 x 16
Parameters: (5*5*3)*16 + 16 = 1200 + 16 = 1216

After MaxPool (2x2, stride=2):

W_out = (28 - 2) / 2 + 1 = 14
H_out = (28 - 2) / 2 + 1 = 14
Depth = 16 (pooling doesn't change depth)
Output size: 14 x 14 x 16
Parameters: 0 (pooling has no parameters)

Q5.2 Explain Recurrent Neural Networks (RNN), LSTM and GRU. [LAQ – 15M]
A. Need for RNNs
Standard FNNs process fixed-size inputs independently — no memory of sequence context. RNNs
maintain a hidden state ht that encodes history of the sequence.

B. Vanilla RNN
ht = tanh(Wh·ht-1 + Wx·xt + b)
ŷt = Wy·ht + by
xt
|
ht-1 --[RNN]-- ht --> yt
|_ same W reused every time step (parameter sharing)

C. LSTM (Hochreiter & Schmidhuber, 1997)

Adds a cell state Ct (long-term memory) controlled by 3 gates. All gates use sigmoid output (0=closed,
1=open):

Gate Formula Role

Forget gate f_t

σ(W_f·[h_t-1,x_t]+b_f)
How much of old cell state to forget

Input gate i_tσ(W_i·[h_t-1,x_t]+b_i)

How much new info to store

Candidate C̃_t
tanh(W_c·[h_t-1,x_t]+b_c)
New candidate cell values

Cell update C_t

f_t C_t-1 + i_t
Updated cell
C̃<(long-term
sub>t</sub>
memory)

Output gate o_t

σ(W_o·[h_t-1,x_t]+b_o)
What to output from cell

Hidden state h_t

o_t tanh(C_t) Filtered cell state → output

D. GRU (Cho et al., 2014)

Simpler LSTM: 2 gates instead of 3, no separate cell state. Comparable performance, fewer parameters.
rt = σ(Wr·[ht-1,xt]) (reset gate)
zt = σ(Wz·[ht-1,xt]) (update gate)
h̃t = tanh(W·[rt ht-1,xt])
ht = (1−zt) ht-1 + zt h̃t

E. RNN vs LSTM vs GRU

Aspect Vanilla RNN LSTM GRU

Memory type Short-term only Long + short term Intermediate

Gates None 3 (forget, input, output) 2 (reset, update)

Parameters Fewest Most (4× of RNN) ~75% of LSTM

Long-range dependency Cannot capture Excellent Good

Vanishing gradient Severe problem Largely solved Largely solved

Training speed Fastest Slowest Moderate

Q5.3 What is a Deep Belief Network (DBN)? Explain RBM and greedy layer-wise training.
[LAQ – 15M]
A. Restricted Boltzmann Machine (RBM)
Undirected bipartite graph: visible layer v (observed data) and hidden layer h (features). No intra-layer
connections. Energy function:
E(v,h) = − vTWh − bTv − cTh
Joint probability: P(v,h) = (1/Z)·exp(−E(v,h))
Conditional distributions factorize (bipartite):
P(hj=1|v) = σ(cj + Σi Wijvi)
P(vi=1|h) = σ(bi + Σj Wijhj)

B. RBM Training: Contrastive Divergence (CD-k)

CD-1 Algorithm:
1. Positive phase: sample h ~ P(h|v_data) [data statistics]
2. Reconstruct: sample v' ~ P(v|h) [model reconstruction]
3. Negative phase: sample h' ~ P(h|v') [model statistics]
4. Update: W += eta * (v*h^T - v'*h'^T)
b += eta * (v - v')
c += eta * (h - h')

C. Deep Belief Network (DBN)

Stack multiple RBMs. Train greedily layer-by-layer (Hinton et al., 2006):
Step 1: Train RBM1 on raw input v --> learn h1
Step 2: Treat h1 as input, train RBM2 --> learn h2
Step 3: Treat h2 as input, train RBM3 --> learn h3
Step 4: Add classification layer on top of h3
Step 5: Fine-tune entire network with backpropagation

Pre-training initializes weights in a good region

Historical importance: reignited deep learning (2006)

D. Significance
Historical: DBN pre-training (Hinton, 2006) showed for the first time that deep networks can be trained
effectively — this triggered the modern deep learning renaissance. Modern context: Less used today
(superseded by ReLU + BatchNorm + dropout + residuals), but conceptually important for generative
modeling.

Chapter 5 — Short Answer Questions

[SAQ 5.1] What are residual connections (ResNet)? Why do they help?
Residual connections (He et al., 2015) add identity shortcuts: output = F(x) + x. The network learns the
residual F(x) = desired_output - x. Benefits: (1) Gradients flow directly through skip connections — no
vanishing. (2) Training very deep networks (50, 101, 152 layers) becomes feasible. (3) Easier to learn
identity mapping (set F(x)=0). ResNet-152 achieved 3.57% top-5 error on ImageNet 2015.

[SAQ 5.2] Compare CNN and RNN.

CNN: grid-structured data (images/video), uses local convolutions + pooling, translation invariant, no
sequential memory, highly parallelizable, O(k) per layer. RNN: sequential data (text/audio/time-series),
maintains hidden state across timesteps, captures temporal dependencies, not parallelizable
(sequential), vanishing gradient. CNN for spatial; RNN/LSTM for temporal/sequential patterns.

[SAQ 5.3] What is dropout? Give its training and test procedure.
Dropout (Srivastava, 2014): during training, each neuron set to 0 with probability p (usually 0.5). Forces
network to learn redundant representations, prevents co-adaptation. Test time: use all neurons but scale
by (1-p) to match training expectation. Equivalent to averaging predictions of exponentially many thinned
networks. Implemented with a random binary mask: a_drop = a * mask / (1-p).

[SAQ 5.4] What is an autoencoder? Describe its architecture.

Autoencoder: unsupervised neural network that learns to compress and reconstruct data. Architecture:
Input x → Encoder → bottleneck z (latent code) → Decoder → x_hat ≈ x. Loss: reconstruction error ||x -
x_hat||². Applications: dimensionality reduction (z), denoising, anomaly detection, feature learning.
Variational AE (VAE): z ~ N(μ, σ²); ELBO loss = reconstruction + KL(q(z|x)||p(z)).

Chapter 5 — Multiple Choice Questions

[MCQ 5.1] Which operation in CNN provides translation invariance?
A) Convolution B) ReLU C) Pooling D) Batch Norm
Answer: C

[MCQ 5.2] LSTM solves the vanishing gradient problem using:

A) Bigger learning rate B) Cell state with gates C) More hidden layers D) Sofmax output
Answer: B

[MCQ 5.3] ResNet introduced which key architectural innovation?

A) Inception modules B) Depthwise convolution C) Residual skip connections D) Attention
Answer: C

[MCQ 5.4] In a CNN, number of parameters in a conv layer with 32 filters, 3x3 kernel, 64 input
channels equals:
A) 32 B) 18,432 C) 18,464 D) 64
Answer: C

[MCQ 5.5] GRU compared to LSTM has:

A) More parameters B) 3 gates C) Fewer parameters and 2 gates D) No recurrent connection
Answer: C

[MCQ 5.6] DBN stands for:

A) Deep Binary Network B) Deep Belief Network C) Dense Batch Norm D) Deep Backprop Node
Answer: B
CHAPTER 6 — DEEP LEARNING RESEARCH [6h · 10M]

Q6.1 Explain object recognition using deep learning. Describe YOLO. [LAQ – 15M]
A. Computer Vision Task Hierarchy
Task Output Difficulty Method

Image Classification 1 class label for whole image Easy VGG, ResNet

Object Localization Class + one bounding box Medium CNN + regression head

Object Detection Multiple boxes + classes Hard YOLO, Faster R-CNN, SSD

Semantic Segmentation Class per pixel Harder FCN, DeepLab, U-Net

Instance Segmentation Class + unique instance per pixel Hardest Mask R-CNN

B. YOLO — You Only Look Once (Redmon et al., 2016)

YOLO frames detection as a single regression problem. One forward pass gives all detections.
Extremely fast (real-time, 45+ FPS).
Divide image into S x S grid (e.g. 7x7)
Each cell predicts:
- B bounding boxes: (x_center, y_center, width, height, confidence)
- C class probabilities
Output tensor: S x S x (B*5 + C)
Post-process with NMS (Non-Maximum Suppression)

C. IoU and mAP Metrics

IoU = Area(Pred ∩ GT) / Area(Pred GT). Detection correct if IoU ≥ 0.5.
mAP (mean Average Precision): average of AP across all classes, where AP = area under
Precision-Recall curve. Standard benchmark metric (COCO, PASCAL VOC).

D. Faster R-CNN vs YOLO

Aspect Faster R-CNN YOLO

Approach 2-stage: propose + classify 1-stage: detect directly

Speed ~5 FPS 45 FPS (YOLOv1), 60+ FPS (v3+)

Accuracy Higher (especially small objects) Slightly lower for small objects

Key innovation Region Proposal Network (RPN) Single unified regression

Use case Accuracy-critical applications Real-time applications

Q6.2 Explain sparse coding and its connection to deep learning. [LAQ – 15M]
A. Sparse Coding
Sparse coding: represent input x as sparse linear combination of dictionary atoms D = [d1,...,dK]:
x ≈ D·a, minimize: ||x − D·a||2 + λ·||a||1
where a is sparse (most coefficients = 0). Learn both D and a jointly.

B. Biological Motivation
Olshausen & Field (1996): sparse coding of natural image patches learns Gabor-like filters (oriented
edge detectors) — identical to V1 simple cells in visual cortex. Suggests the brain represents information
sparsely and efficiently.

C. Connection to Deep Learning

Sparse Coding Concept Deep Learning Equivalent

Dictionary atoms D Learned conv filters / weight columns

Sparse activations (L1 on a) ReLU (outputs 0 for negatives → sparse)

 Sparse autoencoder Autoencoder with L1 penalty on bottleneck z

Hierarchical sparse coding Deep CNN: each layer learns sparse features of features

Basis pursuit L1 regularization in neural networks

Q6.3 Explain deep learning for Natural Language Processing (NLP). [LAQ – 15M]
A. Word Embeddings
Model Key Idea Strength

Word2Vec (CBOW) Predict word from context words Captures semantic similarity

Word2Vec (Skip-gram) Predict context from word Word analogies: king-man+woman=queen

GloVe Ratio of co-occurrence probabilities Global + local statistics

FastText Character n-grams + word vectors Handles out-of-vocabulary words

BERT Bidirectional Transformer pre-training Contextual embeddings (word meaning varies by context)

B. Sequence-to-Sequence Architecture
Input: [x1, x2, x3, x4] (e.g., English sentence)
| Encoder (LSTM or Transformer)
Context: c (fixed-size summary of input)
| Decoder (LSTM or Transformer)
Output: [y1, y2, y3] (e.g., French translation)
(input and output lengths can differ)

C. Attention Mechanism (Bahdanau et al., 2015)

Fixed context vector is bottleneck for long sequences. Attention lets decoder attend to all encoder states:
ct = Σi αti · hi
where αti = softmax(score(st, hi)) — attention weights.
Each decoder step focuses on relevant input positions. Enables translation of long sentences.

D. Transformer
Vaswani et al. (2017) — "Attention is All You Need." Eliminates RNNs entirely. Key: self-attention.
Attention(Q,K,V) = softmax(QKT / √dk) · V
Multi-head: run h parallel attention heads → concatenate → project. Captures different relationship
types. Fully parallelizable → much faster training than RNNs.

E. BERT and GPT

Model Architecture Pre-training Best For

BERT Transformer Encoder Masked LM + Next Sentence Prediction

Classification, NER, QA

GPT Transformer Decoder Next word prediction (autoregressive) Text generation

T5 Encoder-Decoder Text-to-text on all tasks Translation, summarization

GPT-3/4 Large Transformer DecoderWeb-scale text General NLP, chatbots

Chapter 6 — Short Answer Questions

[SAQ 6.1] What is transfer learning in computer vision?
Reuse a CNN pre-trained on ImageNet for a new task. Two strategies: (1) Feature extractor: freeze all
layers, replace + train only final layer. (2) Fine-tuning: freeze early layers (general features), fine-tune
later layers (task-specific). Works because early layers learn universal features (edges, textures,
shapes) transferable across vision tasks. Reduces data + compute requirements.

[SAQ 6.2] What is a GAN (Generative Adversarial Network)?

Two networks trained adversarially (Goodfellow et al., 2014): Generator G(z): maps random noise z to
fake samples. Discriminator D(x): classifies real vs. fake. Minimax objective: min_G max_D E[log D(x)] +
E[log(1-D(G(z)))]. At equilibrium: G produces samples indistinguishable from real data. Applications:
image synthesis, super-resolution, deepfakes, data augmentation.

[SAQ 6.3] List four important deep learning research applications.

(1) Object detection: YOLO, Faster R-CNN for real-time detection. (2) NLP: BERT/GPT for translation,
question answering, generation. (3) Medical imaging: tumor detection, retinal disease diagnosis. (4)
Speech recognition: deep RNN/Transformer-based ASR (Whisper). (5) Autonomous driving: CNN for
lane detection, object avoidance. (6) Drug discovery: protein structure prediction (AlphaFold).

Chapter 6 — Multiple Choice Questions

[MCQ 6.1] YOLO divides the input image into a grid and predicts:
A) One bounding box per image B) Bounding boxes and classes per grid cell C) Only class labels D)
Pixel-level labels
Answer: B

[MCQ 6.2] Self-attention in Transformer computes:

A) Conv over sequence B) Attention(Q,K,V) = softmax(QK^T/sqrt(dk))V C) LSTM over tokens D) CRF
over labels
Answer: B

[MCQ 6.3] BERT uses which pre-training objective?

A) Next word prediction only B) Masked Language Modeling + Next Sentence Prediction C) Image
reconstruction D) Contrastive learning
Answer: B

[MCQ 6.4] mAP in object detection stands for:

A) Maximum Approximate Prediction B) Mean Average Precision C) Multi-Anchor Pooling D)
Minimum Anchor Point
Answer: B
MOCK EXAM PAPER — PCCAIML602 DEEP LEARNING [70 Marks | 3
Hours]
Instructions: Group A (all 10, 1M each). Group B (any 3 of 5, 5M each). Group C (any 3 of 5, 15M each).

GROUP A — Objective (10 × 1 = 10 marks)

[A1] The activation function that solves vanishing gradient and creates sparse activations
is:
a) Sigmoid b) Tanh c) ReLU d) Step
Answer: c)

[A2] Backpropagation uses the ________ rule to compute gradients.

a) Power b) Chain c) Quotient d) Product
Answer: b)

[A3] Which optimizer uses both first and second moment estimates?
a) SGD b) Momentum c) AdaGrad d) Adam
Answer: d)

[A4] In CRF, the probability model requires computing:

a) Likelihood b) Posterior c) Partition function d) Prior
Answer: c)

[A5] The Viterbi algorithm solves which problem in HMM?

a) Evaluation b) Learning c) Decoding d) Encoding
Answer: c)

[A6] Max pooling in CNN provides:

a) Parameter sharing b) Translation invariance c) Gradient vanishing d) Sparse gradients
Answer: b)

[A7] LSTM uses a cell state to store:

a) Short-term memory only b) Long-term memory c) Batch statistics d) Gradient info
Answer: b)

[A8] Entropy of a uniform distribution over n outcomes equals:

a) 0 b) n c) log n d) 1/n
Answer: c)

[A9] Dropout is applied only during:

a) Test time b) Training time c) Both train and test d) Inference only
Answer: b)

[A10] Word2Vec Skip-gram predicts:

a) Word from context b) Context from word c) Sentence embedding d) POS tag
Answer: b)

GROUP B — Short Answer (5 × 5 = 25 marks — Answer any 3)

[B1] Compare L1 and L2 regularization. When would you prefer L1? [5M]
L1: penalty = λΣ|w|; produces sparse weights (exact zeros) → feature selection. L2: penalty = λΣw²;
shrinks all weights toward 0 (no exact zeros). Prefer L1 when: (1) feature selection needed, (2)
high-dimensional sparse features (NLP), (3) interpretability required. Prefer L2 for most DL tasks
(smooth, differentiable everywhere). ElasticNet combines both: L = L_task + λ1||w||1 + λ2||w||2².

[B2] Explain the forward algorithm for HMM. [5M]

Forward variable: α_t(i) = P(x1,...,xt, q_t=i | λ). Initialize: α_1(i) = π_i · B_i(x_1). Recurse: α_t(j) = [Σ_i
α_{t-1}(i)·A_ij] · B_j(x_t). Final: P(x|λ) = Σ_i α_T(i). Time complexity: O(n·|Q|²). Enables efficient
likelihood computation via DP.

[B3] What are the advantages of batch normalization? [5M]

(1) Reduces internal covariate shift — stable distribution at each layer input. (2) Allows higher learning
rates → faster convergence (2-10x). (3) Reduces sensitivity to weight initialization. (4) Acts as mild
regularizer → reduces need for dropout. (5) Learnable parameters γ,β restore representational power.
Applied after linear transform, before activation.

[B4] Differentiate semantic segmentation from instance segmentation. [5M]

Semantic segmentation: assigns a class label to every pixel; two objects of the same class get same
label (cannot distinguish them). Instance segmentation: assigns both class AND unique instance ID to
each pixel; each object instance separated even if same class. Example: crowd image — semantic: all
"person" pixels labeled same; instance: person 1, person 2, ... labeled separately. Methods:
Semantic=FCN,DeepLab; Instance=Mask R-CNN.

[B5] Explain the attention mechanism in seq2seq models. [5M]

Problem: fixed-size context vector is a bottleneck for long sequences. Attention (Bahdanau, 2015): at
each decoder step t, compute alignment score e_ti = score(s_t, h_i) for all encoder states h_i. Normalize:
α_ti = softmax(e_ti). Context: c_t = Σ_i α_ti · h_i (weighted sum of encoder states). Decoder attends to
different input positions at each step. Enables translation of long sentences and interpretable attention
maps.

GROUP C — Long Answer (5 × 15 = 75 marks — Answer any 3)

[C1] (a) Explain the backpropagation algorithm with mathematical derivation [8M]. (b)
Describe optimization algorithms: SGD, Momentum, Adam with comparison [7M]. [15M]
Refer to the corresponding LAQ answers above.

[C2] (a) What is a CRF? Explain Linear Chain CRF and training [8M]. (b) Explain HMM
with its three fundamental problems [7M]. [15M]
Refer to the corresponding LAQ answers above.

[C3] (a) Explain CNN architecture with operations and output size formula [8M]. (b)
Explain LSTM with all gate equations and diagram [7M]. [15M]
Refer to the corresponding LAQ answers above.

[C4] (a) Explain regularization techniques: L1, L2, Dropout, Batch Norm [8M]. (b) Explain
entropy, KL divergence, and cross-entropy with formulas [7M]. [15M]
Refer to the corresponding LAQ answers above.

[C5] (a) Explain Deep Belief Network and Restricted Boltzmann Machine [8M]. (b)
Explain object detection with YOLO and evaluation metrics [7M]. [15M]
Refer to the corresponding LAQ answers above.
QUICK REFERENCE — FORMULAS & CHEATSHEET
KEY FORMULAS
Formula Expression

Neuron output a = f(w<super>T</super>x + b)

Sigmoid σ(z) = 1/(1+e<super>-z</super>), σ'(z) = σ(z)(1-σ(z))

Tanh tanh(z) = (e<super>z</super>-e<super>-z</super>)/(e<super>z</super>+e<super>-z</super

ReLU f(z) = max(0,z), f'(z)=1 if z>0 else 0

Softmax σ(z)_i = e<super>zi</super>/Σe<super>zj</super>

MSE Loss L = (1/n)Σ(y - ŷ)²

Binary CE L = -[y·log(ŷ) + (1-y)·log(1-ŷ)]

Categorical CE L = -Σ y_i·log(ŷ_i)

L2 reg L_reg = L + λ·Σw_i², update: w ← w(1-2ηλ) - η·∂L/∂w

Backprop hidden δ<super>(l)</super> = (W<super>(l+1)</super>)<super>T</super>·δ<super>(l+1)</super>

Adam m̂ = m/(1-β₁<super>t</super>); v̂ = v/(1-β₂<super>t</super>); w -=

Entropy H(X) = -Σ P(x)·log₂P(x) [bits]

Cross-Entropy H(P,Q) = -Σ P(x)·log Q(x)

KL Divergence KL(P||Q) = Σ P(x)·log(P(x)/Q(x)) ≥ 0

CNN output size W_out = (W_in - f + 2P)/S + 1

CRF probability P(y|x) = (1/Z(x))·exp(Σ_tΣ_k λ_kf_k(y<sub>t<

IoU Area(Pred∩GT) / Area(Pred GT)

Transformer Attn softmax(QK<super>T</super>/√d_k)·V

ARCHITECTURE QUICK COMPARISON

Architecture Input Key Layer Strength Limitation

MLP Fixed vector Dense FC Universal approximator No spatial/temporal structure

CNN Grid (image) Convolution Spatial pattern detection Fixed input size

RNN Sequence Recurrent cell Temporal dependencies Vanishing gradient

LSTM Sequence LSTM cell Long-range dependencies Slow, many parameters

Transformer Sequence Self-attention Parallelizable, global context Quadratic in sequence length

Autoencoder Any Bottleneck z Unsupervised features No discriminative signal

GAN Noise z Generator+Discriminator

Realistic generation Training instability

DBN Vector Stacked RBMs Generative pre-training Complex, slow to train

IMPORTANT PAPERS TO KNOW

Paper Authors Year Contribution

Perceptron Rosenblatt 1958 First trainable neural model

Backpropagation Rumelhart, Hinton, Williams 1986 Efficient gradient computation

UAT Hornik 1989 MLP = universal approximator

LSTM Hochreiter & Schmidhuber 1997 Solves vanishing gradient in RNNs

Sparse Coding Olshausen & Field 1996 Biological basis of feature learning
DBN / RBM Hinton et al. 2006 Greedy pre-training, deep learning revival

ReLU / AlexNet Krizhevsky et al. 2012 Modern deep learning era begins

Dropout Srivastava et al. 2014 Regularization via random neuron masking

Batch Norm Ioffe & Szegedy 2015 Stable deep network training

ResNet He et al. 2015 152-layer network with skip connections

CRF Lafferty et al. 2001 Discriminative sequence labeling

Attention Bahdanau et al. 2015 Context-sensitive encoder-decoder

Transformer Vaswani et al. 2017 Self-attention, no RNNs

BERT Devlin et al. 2018 Bidirectional Transformer pre-training

YOLO Redmon et al. 2016 Real-time object detection