Advanced Practice Questions on Deep Learning

CNNs, RNN/LSTM/GRU, GANs, VAEs, Autoencoders, Losses, Optimization, Training

and Inference
Each question is intended to be of roughly 3–5 marks.

A. Convolutional Neural Networks

Q1. An input image of size 64 × 64 × 3 is passed through the following network:

Conv(5 × 5, 16, s = 1, p = 2) → MaxPool(2 × 2, s = 2)

→ Conv(3 × 3, 32, s = 1, p = 1)
→ Conv(3 × 3, 64, s = 2, p = 1).

Compute: (i) the spatial size and number of channels after each layer, (ii) the number of
trainable parameters in each convolution layer (including bias), and (iii) the receptive field of
one unit in the final feature map with respect to the input image.

Q2. Consider a 1D convolutional layer with input

x = [1, −1, 2, 0],

kernel
w = [2, −1],
stride 1, and no padding. The forward operator is the usual deep-learning cross-correlation.
Let the loss be
1X 2
L= yi ,
2
i
where y is the output of the convolution. Compute the forward output y, and then compute
∂L ∂L
∂w and ∂x .

Q3. A standard convolution takes an input tensor of size 32 × 32 × 64 and produces 128 output
channels using a 3 × 3 kernel. Compute the total number of parameters for: (i) a standard
convolution, and (ii) a depthwise-separable convolution (depthwise 3 × 3 followed by pointwise
1 × 1), both with bias ignored. Also compute the parameter reduction factor.

Q4. Two convolutional layers are stacked, both with stride 1 and chosen padding so that the spatial
size is preserved. The first layer uses a 3 × 3 kernel with dilation 1 and the second uses a 3 × 3
kernel with dilation 2. Compute the receptive field of one output activation with respect to
the original input. Compare it with the receptive field obtained if both layers had dilation 1.

B. RNN, LSTM and GRU

Q5. Consider the scalar recurrent network

ht = tanh(aht−1 + bxt ),

1
 ∂L
and suppose the loss is L = hT . Derive an expression for ∂h t
in product form. Using your
expression, explain mathematically why gradients vanish when |a(1 − h2k )| < 1 over many
steps and explode when it is consistently greater than 1. Further, if a = 1.5 and hk ≈ 0 for
all intermediate steps from t to T − 1, approximate the gradient amplification factor over 5
steps.

Q6. In a scalar LSTM at time t, the gate values are

ft = 0.8, it = 0.3, ot = 0.9, c̃t = 0.5,

and the previous cell state is ct−1 = 1.2. Compute: (i) the new cell state ct , (ii) the hidden state
ht = ot tanh(ct ), and (iii) ∂c∂ct−1
t
. If the forget gate remained fixed at 0.8 for four consecutive
∂ct
steps, what would ∂ct−4 be?

Q7. A GRU has input dimension dx = 20 and hidden dimension dh = 50. Compute the number of
trainable parameters in the GRU, including biases. Then compute the corresponding number
for an LSTM with the same dimensions, and state which one is more parameter-efficient.

Q8. A sequence model trained with teacher forcing assigns the following conditional probabilities
to the correct next token of a target sequence of length 4:

0.9, 0.8, 0.6, 0.5.

Compute: (i) the negative log-likelihood of the sequence, and (ii) the perplexity of the se-
quence. Write one mathematically informed sentence explaining why this training setup can
still suffer from exposure bias at inference time.

C. Generative Adversarial Networks

Q9. For the original GAN objective

   
V (D, G) = Ex∼pdata log D(x) + Ez∼p(z) log(1 − D(G(z))) ,

derive the optimal discriminator D∗ (x) for a fixed generator G.

Q10. By substituting the optimal discriminator into the GAN value function, show that

V (D∗ , G) = − log 4 + 2 JS(pdata ∥pg ) .

State clearly what this implies about the global optimum of the minimax GAN objective.

Q11. Let a denote the discriminator logit on a generated sample, and let D = σ(a). For the
saturating generator loss
Lsat = log(1 − σ(a)),
and the non-saturating generator loss

Lns = − log(σ(a)),

derive ∂L∂asat and ∂L

∂a . Evaluate both when D = σ(a) = 0.01, and comment on why the
ns

non-saturating loss is preferred in practice.

2
Q12. In WGAN-GP, the critic is regularized by

λ (∥∇x̂ f (x̂)∥2 − 1)2 .

Suppose for one interpolated sample x̂, the gradient norm is 1.8 and λ = 10. Compute
the gradient penalty contribution of this sample. If the critic’s average score on a batch of
real samples is 2.3 and on fake samples is 0.7, what is the regularized critic objective value
contributed by these batch means and this single penalty term?

D. Variational Autoencoders

Q13. Starting from

log pθ (x) = L(x; θ, ϕ) + DKL (qϕ (z|x) ∥ pθ (z|x)) ,
derive the evidence lower bound (ELBO)

L(x; θ, ϕ) = Eqϕ (z|x) [log pθ (x|z)] − DKL (qϕ (z|x) ∥ p(z)) .

Interpret both terms in words.

Q14. Let
qϕ (z|x) = N µ, diag(σ 2 ) ,


p(z) = N (0, I),
with
µ = (1, −1), log σ 2 = (ln 4, ln 0.25).
Compute the KL divergence
DKL (qϕ (z|x)∥p(z)).

Q15. Consider a 1D latent VAE with reparameterization

z = µ + σϵ, ϵ ∼ N (0, 1),

and a Gaussian decoder with mean W z. For one Monte Carlo sample, let

W = 2, x = 3, µ = 1, σ = 0.5, ϵ = 2.

Using the reconstruction loss

1
Lrec = (x − W z)2 ,
2
compute the sampled latent z, the reconstruction loss, and the gradients
∂Lrec ∂Lrec
, .
∂µ ∂σ

Q16. For one latent dimension, let ℓ = log σ 2 . Show that the KL term can be written as
1 2 
DKL = µ + eℓ − ℓ − 1 .
2
Then derive
∂DKL ∂DKL
, ,
∂µ ∂ℓ
and evaluate them at µ = 1.5 and ℓ = −0.7.

3
 E. Autoencoders and Regularized Autoencoders

Q17. A linear undercomplete autoencoder with tied weights is trained on zero-mean data using
squared reconstruction loss:
X
h = W ⊤ x, x̂ = W h, min ∥xn − x̂n ∥22 .
W
n

State the precise connection between the optimal solution of this problem and PCA. What
subspace does the autoencoder recover?

Q18. A sparse autoencoder uses the penalty

m
X
Ωsparse = λ KL(ρ∥ρ̂j ),
j=1

where
ρ 1−ρ
KL(ρ∥ρ̂j ) = ρ log + (1 − ρ) log .
ρ̂j 1 − ρ̂j
Derive
∂Ωsparse
.
∂ ρ̂j
Then evaluate it for λ = 2, ρ = 0.1, and ρ̂j = 0.25.

Q19. In a contractive autoencoder with sigmoid hidden units,

hj = σ(wj⊤ x + bj ),

the regularizer is
Ωcontr = λ∥Jh (x)∥2F .
Show that this becomes X
2
Ωcontr = λ hj (1 − hj ) ∥wj ∥22 .
j

For a single hidden unit with w = (1, 2), activation h = 0.8, and λ = 0.5, compute the
regularization value contributed by this unit.

Q20. Let x be a zero-mean scalar random variable with variance σx2 , and let the corrupted obser-
vation be
x̃ = x + ε, ε ∼ N (0, σn2 ),
with ε independent of x. A linear denoising autoencoder reconstructs using

r(x̃) = ax̃.

Find the value of a that minimizes

E (x − ax̃)2 .
 

Interpret the result.

4
 F. Losses, Optimization, Training and Inference

Q21. A 3-class classifier produces logits

z = [2, 0, −1].
The true class is class 1 (that is, the first class), and label smoothing with ε = 0.1 is used,
with target
ε
y (ls) = (1 − ε)ey + 1.
K
Compute the smoothed target distribution, the softmax probabilities, and the gradient of the
cross-entropy loss with respect to the logits.

Q22. For binary classification with target y = 1, focal loss is

L = −α(1 − p)γ log p,

∂L
where p = σ(z). Derive ∂z for general α, γ. Then evaluate the loss and the gradient when

α = 0.25, γ = 2, p = 0.8.

Q23. Adam is run on a scalar parameter w with

w0 = 1, g1 = 0.2, g2 = −0.1,

β1 = 0.9, β2 = 0.999, η = 0.01,

and assume ε is negligible. Compute mt , vt , the bias-corrected moments m̂t , v̂t , and the
parameter values w1 and w2 .

Q24. Gradient descent with momentum uses

vt = µvt−1 + gt , wt = wt−1 − ηvt ,

with µ = 0.9, η = 0.1, and v0 = 0. Before the momentum update, raw gradients are clipped
to have norm at most 5. Starting from

w0 = (1, −1),

suppose the raw gradients are

g1 = (6, 8), g2 = (−3, 4).

Compute the clipped gradients, the velocities v1 , v2 , and the parameters w1 , w2 .

Q25. Consider one SGD step on a scalar parameter with learning rate η = 0.1, current value w = 2,
data gradient g = 0.5, and regularization strength λ = 0.01. Compute the update under:
(i) L2-regularized SGD, where the regularization is added to the gradient, and (ii) decoupled
weight decay. Explain briefly why these two are identical under plain SGD but not under
Adam-style adaptive methods.

Q26. A calibrated binary classifier outputs p(x) = P (y = 1|x). The cost of a false negative is
CFN = 5, and the cost of a false positive is CFP = 1. Derive the Bayes-optimal decision
threshold on p(x) for predicting the positive class.

5
Q27. A 3-class model produces logits
z = [2, 1, 0].
Compute the predicted probabilities before and after temperature scaling with T = 2, where
the scaled probabilities are obtained from softmax(z/T ). Comment on how temperature
scaling changes confidence without changing the class ranking.

Q28. Consider a two-layer network with one hidden ReLU layer:

h = ReLU(W1 x), ŷ = W2 h.

Let   
1 1 0  
x= , W1 = , W2 = 2 −1 ,
−1 −1 2
and the target be y = 1. Using the loss
1
L = (ŷ − y)2 ,
2
compute the forward pass, the loss, and the gradients with respect to W2 and W1 .

Q29. In an autoregressive sequence model, the first-step probabilities are

P (A) = 0.6, P (B) = 0.4.

At the second step,

P (EOS | A) = 0.1, P (A | A) = 0.9,
and
P (EOS | B) = 0.95, P (A | B) = 0.05.
Using sequence probability as the decoding score, compare greedy decoding and beam search
with beam width 2. Which complete sequence is optimal?

Q30. In inverted dropout, a scalar activation h is replaced during training by

m
h̃ = h, m ∼ Bernoulli(q),
q
where q is the keep probability. For h = 4 and q = 0.75, compute

E[h̃], Var(h̃).

Why does this formulation remove the need for rescaling at test time?