Deep Learning Practice Set

(All notations follow those used in the lecture slides)

*Learning rate is given as η

Section A: Multi Layer Perceptron (MLP)

1. For a neuron with input x = [1, 2], weights w = [0.5, −1], and bias b = 1, compute the output before
activation.

2. Using ReLU activation, compute the output for z = −3.

3. Using sigmoid activation, compute the output for z = 0.

4. Compute the output of a neuron with:

• Inputs: [2, −1, 3]

• Weights: [0.2, 0.5, −0.3]
• Bias: 0.1

5. Compute the derivative of sigmoid at z = 0.

6. If a layer has 5 neurons and each neuron receives 3 inputs, compute total number of weights (ignore
bias).

7. Compute total parameters (including bias) for a layer with:

• Input size = 4
• Output neurons = 3

8. Compute the output of tanh at z = 0.

9. Given:

• Input: x = [1, 2]
• Weights:  
0.1 0.2
W =
0.3 0.4
• Bias: b = [0.5, 0.5]

Compute:

• Linear output
• Output after sigmoid activation

1
10. Given:

• Input: x = [1, 0]

Layer 1:  
0.5 −0.5
W1 = , b1 = [0.1, 0.2]
0.3 0.8

Layer 2:  
W2 = 0.7 0.6 , b2 = [0.3]

Use ReLU in hidden layer and sigmoid in output. Compute final output.

11. An MLP has:

• Input layer: 10 neurons

• Hidden layer: 6 neurons
• Output layer: 2 neurons

Compute total number of parameters (including bias).

12. Given:

• Input: x = 2
• Weight: w = 0.5
• Bias: b = 0
• Target: y = 1

Activation: Linear.
Compute:

• Output
• Loss using Mean Squared Error
• Gradient of loss w.r.t weight

13. Given:
z = wx + b
Activation: Sigmoid, Loss:
1
L = (y − ŷ)2
2
Derive:
∂L
∂w
14. Given:

• Input vector size: 1 × 3

• Weight matrix size: 3 × 4

Compute:

• Output size

2
 • General expression for output
15. Given:
• Weight = 0.8
• Gradient = 0.2
• Learning rate = 0.1
Compute updated weight using gradient descent.
16. For the network:
x → z = wx → a = σ(z) → L

Derive:
∂L
∂w
17. Given:
σ ′ (z) = σ(z)(1 − σ(z))

Compute derivative at:

• z = 10
• z = −10
18. Network:
• Input = 8
• Hidden1 = 16
• Hidden2 = 8
• Output = 4
Compute total parameters (including bias).

Section B: Deep Learning Optimizers and Training Techniques

1. Given for SGD:
θ = [2, −1], g = [0.5, −0.2], η = 0.1
Compute updated weights using SGD.
2. Given for Adagrad:
θt−1 = [1.0, −2.0], gt = [0.5, −0.5], η = 0.1

Adagrad accumulator update:

rt = rt−1 + gt ⊙ gt

Weight update:
η
θt = θt−1 − √ ⊙ gt
δ + rt
If:
rt−1 = [0, 0], δ = 10−8

Compute:

3
 • rt
• Updated weights θt

3. Given Momentum update:

ut = 0.9ut−1 − 0.1gt
If:
ut−1 = [0.2, −0.1], gt = [0.5, 0.3]
Compute ut .

4. For RMSProp:
rt = 0.9rt−1 + 0.1gt ⊙ gt
Given:
rt−1 = [0.4, 0.2], gt = [0.5, −0.5]
Compute rt .

5. Compute standardization:
x−µ
z=
σ
for:
x = [10, 12], µ = [8, 10], σ = [2, 1]

6. Compute min-max normalization:

x − xmin
x′ =
xmax − xmin
for:
x = [6, 8], xmin = [2, 4], xmax = [10, 12]

7. Given learning rate scheduler after t-epochs as:

η0
ηt =
1 + kt
Compute ηt for:
η0 = 0.1, k = 0.01, t = 20

8. SGD (Vector Case – Multiple Steps)

Given:
θ0 = [1.0, −1.0], η = 0.1
Gradients:
g1 = [0.5, −0.3], g2 = [0.2, 0.4]

Compute:

• θ1
• θ2

4
9. Momentum (Vector Update)
Given:
θ0 = [1.0, 2.0], η = 0.1, α = 0.9
u0 = [0, 0]

Gradients:
g1 = [0.5, −0.2], g2 = [0.3, 0.1]

Compute:

• u1 , θ1
• u2 , θ2

10. Nesterov Momentum (Vector Case)

Given:
θ0 = [1.0, −1.0], η = 0.1, α = 0.9
u0 = [0, 0]

Gradient at lookahead point:

g1 = [0.6, −0.4]

Compute updated weight θ1 .

11. Adagrad
Given:
θ0 = [1.0, −1.0], η = 0.1
r0 = [0, 0], δ = 10−8

Gradients:
g1 = [0.5, −0.2], g2 = [0.4, 0.3]

Compute:

• r1 , θ1
• r2 , θ2

12. RMSProp (Vector Update)

Given:
θ0 = [2.0, 1.0], η = 0.01
ρ = 0.9, r0 = [0, 0]
g1 = [0.5, −0.2]

Compute:

• r1
• Updated θ1

5
13. Adam Optimizer (Vector Case)
Given:
θ0 = [1.0, −1.0], η = 0.01
ρ1 = 0.9, ρ2 = 0.999
r0 = [0, 0], s0 = [0, 0]
g1 = [0.5, −0.3]

Compute:

• r1 , s1
• Bias-corrected r̂1 , ŝ1
• Updated θ1

14. Learning Rate Scheduler (after t epochs)

Given:
ηt = η0 · γ t

η0 = 0.1, γ = 0.9

Weights:
θ0 = [1, 2], g1 = [0.5, 0.5]

Compute:

• Learning rate at t = 2
• Updated weights θ1

15. Xavier Initialization

A weight matrix connects 64 input neurons to 32 output neurons.
Compute:

• Variance using Xavier initialization

• Standard deviation

16. LeCun Initialization

Given:

• Input neurons = 128

• Weight matrix size = 128 × 64

Compute:

• Variance
• Range (±1 standard deviation)

6
17. Batch Normalization
Given:
x = [2, 4, 6, 8]

Compute:

• Mean
• Variance
• Normalized values

Then compute:
y = γxnorm + β
for γ = 2, β = 1.

18. Standardization vs Normalization

Given:
x = [5, 10, 15]

Compute:

• Standardized values
• Min-max normalized values

Section C: Convolutional Neural Networks (CNN)

1. Compute the output size for:

• Input = 280 × 250 × 3, Kernel = 3 × 3, Stride = 1, Padding = 0

• Compute the no. of parameters if the CNN needs to learn 8 such filters

2. Compute the output size for:

• Input = 320 × 240, Kernel = 5 × 5, Stride = 1, Padding = 2

• Compute the no. of parameters if the CNN needs to learn 16 such filters

3. After 2 × 2 max pooling (stride = 2), find the output volume of a 320 × 240 × 16 feature volume.

4. Compute the output size:

• Input = 64 × 64, Kernel = 7 × 7, Stride = 1, Padding = 3

• Compute the no. of parameters if the CNN needs to learn 16 such filters
• If the stride increases from 1 to 2, how do the output size and the number of parameters change?
Justify numerically.

5. An input image of size 64 × 64 passes through:

• Conv1: Kernel = 3 × 3, Stride = 1, Padding = 1

• Pool1: 2 × 2, Stride = 2
• Conv2: Kernel = 5 × 5, Stride = 1, Padding = 0

7
 Compute the final output size.

6. A CNN layer has:

• Input size = 32 × 32 × 3
• Filters = 20
• Kernel size = 5 × 5

Compute:

• Total number of parameters

• Output feature map size (Stride = 1, Padding = 0)

7. Perform convolution:
Input:  
2 1 0 2
1 3 1 0
 
0 2 2 1
1 0 1 3

Kernel:  
1 0
−1 1

Stride = 1, Padding = 0. Compute the output feature map.

8. For input size 32 × 32, compute output sizes:

• Case 1: Kernel = 3 × 3, Stride = 1, Padding = 0

• Case 2: Kernel = 3 × 3, Stride = 2, Padding = 1

9. Input has 3 channels. A convolution layer has:

• 12 filters
• Kernel size = 3 × 3

Compute total parameters (including bias).

10. Input = 28 × 28 passes through:

• Conv: 3 × 3, Stride = 1, Padding = 0

• Max Pool: 2 × 2, Stride = 2
• Conv: 3 × 3, Stride = 1, Padding = 0

Find final output size.

11. A CNN produces 16 feature maps of size 8 × 8.

• Find number of neurons after flattening

• If fully connected layer has 100 neurons, compute total parameters

12. A CNN has:

8
 • Layer 1: 3 × 3, stride 1
• Layer 2: 3 × 3, stride 1

Compute the receptive field.

13. Input: 128 × 128 × 3

Conv layer:

• 32 filters
• Kernel = 3 × 3
• Stride = 1
• Padding = 1

Find:

• Output volume
• Total parameters

14. Input: 32 × 32 × 3
Layers:

• Conv1: 3 × 3, stride 1, padding 1, filters = 8

• Pool1: 2 × 2, stride 2
• Conv2: 3 × 3, stride 1, padding 1, filters = 16
• Pool2: 2 × 2, stride 2

Compute output size after each layer and final flattened size.

15. Consider an input of size 50 × 50 × 3 passed through the following layers:

• Conv1: 5 × 5, stride = 1, padding = 2, filters = 6

• Pool1: 2 × 2, stride = 2
• Conv2: 3 × 3, stride = 1, padding = 1, filters = 16

Compute:

• Output size after each layer

• Total number of parameters in Conv1 and Conv2