IT549: Deep Learning

Lectures 07–08
Neural Networks with Keras

Comprehensive Study Notes with Deep Explanations

100+ Questions & Detailed Answers

Arpit Rana
21st January 2026
Based on lecture notes of Dr. Derek Bridge, UCC,
Ireland

These notes cover: Deep Learning Fundamentals · Keras API · Regression · Binary &
Multiclass Classification · Image Processing
IT549: Deep Learning Arpit Rana

0 1 Contents

1 Introduction to Deep Learning 2

1.1 Historical context . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2
1.2 Why depth matters . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2

2 Deep Learning Setup 2

2.1 Components of a deep learning model . . . . . . . . . . . . . . . . . . . . . 2
2.2 Forward pass . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
2.3 Training loop . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
2.4 Possible variations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3

3 Logistic Regression as a Neural Network 3

3.1 Binary logistic regression . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
3.2 Multinomial (multiclass) logistic regression . . . . . . . . . . . . . . . . . . 4

4 Neural Networks: Architecture 4

4.1 Single hidden layer network . . . . . . . . . . . . . . . . . . . . . . . . . . 4
4.2 Why hidden layers? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5
4.3 Number of hidden layers — practical guidelines . . . . . . . . . . . . . . . 5
4.4 Number of neurons per hidden layer — Heaton’s rules . . . . . . . . . . . . 5

5 Activation Functions 5
5.1 Sigmoid (logistic) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6
5.2 Rectified Linear Unit (ReLU) . . . . . . . . . . . . . . . . . . . . . . . . . 6
5.3 Softmax (output layer for multiclass) . . . . . . . . . . . . . . . . . . . . . 6
5.4 Linear (output layer for regression) . . . . . . . . . . . . . . . . . . . . . . 6
5.5 Summary table . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7

6 Loss Functions 7
6.1 Mean Squared Error (regression) . . . . . . . . . . . . . . . . . . . . . . . 7
6.2 Binary Cross-Entropy (binary classification) . . . . . . . . . . . . . . . . . 7
6.3 Categorical Cross-Entropy (multiclass) . . . . . . . . . . . . . . . . . . . . 7

7 Optimisation Algorithms 7
7.1 Gradient Descent variants . . . . . . . . . . . . . . . . . . . . . . . . . . . 7
7.2 Advanced optimisers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8
7.2.1 RMSprop . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8
7.2.2 Adam (Adaptive Moment Estimation) . . . . . . . . . . . . . . . . 8
7.2.3 Comparison table . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8

8 The Keras Library 8

8.1 Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9
8.2 Three API styles . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9
8.3 Build, compile, fit — the Keras workflow . . . . . . . . . . . . . . . . . . . 9
8.4 Important Keras layer types used in these lectures . . . . . . . . . . . . . . 10

1
IT549: Deep Learning Arpit Rana

9 Neural Network for Regression — House Rent Prediction 10

9.1 Architecture choice . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10
9.2 Why linear output activation for regression? . . . . . . . . . . . . . . . . . 10
9.3 Feature scaling for regression . . . . . . . . . . . . . . . . . . . . . . . . . . 10

10 Neural Network for Binary Classification — Class Performance 11

10.1 Architecture . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11
10.2 Why sigmoid for binary output? . . . . . . . . . . . . . . . . . . . . . . . . 11
10.3 Loss: binary cross-entropy . . . . . . . . . . . . . . . . . . . . . . . . . . . 11

11 Neural Network for Multiclass Classification — Iris Dataset 12

11.1 Architecture . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12
11.2 Two label encoding choices . . . . . . . . . . . . . . . . . . . . . . . . . . . 12
11.3 Softmax output interpretation . . . . . . . . . . . . . . . . . . . . . . . . . 12

12 Image Classification — Fashion MNIST 12

12.1 Dataset overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12
12.2 Preprocessing: reshape and rescale . . . . . . . . . . . . . . . . . . . . . . 13
12.3 Architecture . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13
12.4 Remarks on computer vision . . . . . . . . . . . . . . . . . . . . . . . . . . 13

13 Concluding Remarks on Hyperparameter Selection 14

14 100+ Questions and Detailed Answers 15

14.1 Part A: Deep Learning Fundamentals (Q1–Q15) . . . . . . . . . . . . . . . 15
14.2 Part B: Activation Functions (Q16–Q30) . . . . . . . . . . . . . . . . . . . 19
14.3 Part C: Loss Functions (Q31–Q45) . . . . . . . . . . . . . . . . . . . . . . 23
14.4 Part D: Optimisers (Q46–Q60) . . . . . . . . . . . . . . . . . . . . . . . . . 28
14.5 Part E: Keras Implementation (Q61–Q75) . . . . . . . . . . . . . . . . . . 33
14.6 Part F: Task-Specific Architecture (Q76–Q90) . . . . . . . . . . . . . . . . 38
14.7 Part G: Advanced Concepts & Analysis (Q91–Q110) . . . . . . . . . . . . 43

2
IT549: Deep Learning Arpit Rana

1 1 Introduc
to Deep Learning
What is “Deep” in Deep Learning?
The word “deep” in deep learning does NOT mean profound or philosophically
complex. It refers simply to the depth of the neural network — i.e. the number of
layers stacked between the input and the output. A “deep” network has tens or
even hundreds of layers.

1.1 Historical context

Shallow machine-learning models (logistic regression, SVMs, decision trees, etc.) typi-
cally have at most one hidden transformation layer. By contrast, modern deep-learning
architectures may contain:

ˆ Tens of convolutional blocks (e.g. ResNet-152 has 152 layers).

ˆ Hundreds of transformer blocks (e.g. GPT-4 is reported to have ∼100+ transformer

layers).
ˆ Specialised layer types: dense, convolutional, pooling, attention, normalization, dropout,
etc.

1.2 Why depth matters

Theorem 1.1 (Universal Approximation Theorem (informal)). A feed-forward neural
network with a single hidden layer containing a sufficient number of neurons can approxi-
mate any continuous function on a compact subset of Rn to arbitrary precision — but the
required number of neurons may be exponentially large. Greater depth allows the same
approximation with exponentially fewer parameters.
Intuitively, each layer learns a progressively more abstract representation:
edges shapes objects
pixels −−−→ features
| {z } −−−→ parts −−−−→ ŷ
| {z } | {z } |{z}
layer 0 layer 1 layer 2 output

2 1 Deep
Learning Setup
2.1 Components of a deep learning model
Formula: Model Decomposition

Model
| {z } = Architecture
| {z } + Parameters
| {z }
complete system number of layers, W1 ,b1 ,
neurons per layer, W2 ,b2 ,
layer types, ...
activation functions

3
IT549: Deep Learning Arpit Rana

2.2 Forward pass

Given input x(i) ∈ Rn , the forward pass through an L-layer network computes:

z[l] = W[l] a[l−1] + b[l] , l = 1, . . . , L (1)

a[l] = g [l] z[l]


(2)

where g [l] is the activation function of layer l, a[0] = x(i) , and ŷ = a[L] .

2.3 Training loop

1. Forward pass: compute ŷ via Eqs. (1)–(2).

2. Loss computation: L(ŷ, y).

3. Backward pass: compute ∇W[l] L and ∇b[l] L for all l via backpropagation.

4. Parameter update: W[l] ← W[l] − η ∇W[l] L.

5. Repeat for a fixed number of epochs.

2.4 Possible variations

Variation Details
Input / Output Raw pixels, normalised scalars, embeddings, one-
hot vectors, etc.
Architecture Dense, CNN, RNN, Transformer; depth and width
choices.
Activation function Sigmoid, ReLU, Tanh, Softmax (output layer),
Leaky ReLU, ELU, GELU.
Optimiser SGD, Momentum, RMSprop, Adam, Nadam,
Adagrad, AdamW.
Loss function MSE, MAE, Binary cross-entropy, Categorical
cross-entropy, Huber loss.

3 1 Logistic
Regression as a Neural Network
3.1 Binary logistic regression
From pixel to prediction
An image is flattened (“image2vector”) to a 1-D vector x(i) ∈ Rn , pixel values are

4
IT549: Deep Learning Arpit Rana

divided by 255 to scale into [0, 1], and then:

z = w⊤ x(i) + b (3)
1
ŷ = σ(z) = (4)
1 + e−z
If ŷ > 0.5 ⇒ class 1 (e.g. “Coat”); otherwise class 0.

This is simply a neural network with no hidden layers: one input layer and one
output neuron with sigmoid activation.

3.2 Multinomial (multiclass) logistic regression

For K classes, we have K output neurons, each computing:

zk = wk⊤ x(i) + bk , k = 1, . . . , K
Then softmax is applied:

Formula: Softmax Function

ezk
ŷk = softmax(zk ) = K
X
ezj
j=1
PK
Properties: ŷk ∈ (0, 1) and k=1 ŷk = 1.

The class with the highest ŷk is the predicted class.

4 1 Neural
Networks: Architecture
4.1 Single hidden layer network
Hidden Layer Notation
For a network with 1 hidden layer of h neurons:
n
X
[1] [1] [1] [1] [1]
Hidden layer: zj = wji xi + bj , aj = g(zj )
i=1
Xh
[2] [1]
Output layer: z [2] = wj aj + b[2] , ŷ = gout (z [2] )
j=1

The superscript [l] denotes the layer number.

5
IT549: Deep Learning Arpit Rana

4.2 Why hidden layers?

ˆ A network without hidden layers can only learn linearly separable decision boundaries.

ˆ Hidden layers introduce non-linearity, allowing the network to learn complex, curved
decision boundaries.

ˆ Each hidden neuron learns a feature of the input; successive layers learn increasingly
abstract features.

4.3 Number of hidden layers — practical guidelines

Architecture Rules of Thumb
1. 0 hidden layers: equivalent to a linear model (logistic/linear regression).

2. 1–2 hidden layers: sufficient for most non-linear problems on structured (tab-
ular) data.

3. ≥ 3 hidden layers: use for large datasets (images, text); risk of overfitting
increases.

4. Very deep networks: gradually increase depth until training-set overfitting is

observed, then apply regularisation (dropout, batch norm, etc.).

4.4 Number of neurons per hidden layer — Heaton’s rules

Let nin = number of input features, nout = number of output neurons.

2
nhidden ≈ nin + nout (Heaton Rule 1)
3
nin < nhidden < nout · nin (Heaton Rule 2)
nhidden < 2 nin (Heaton Rule 3)

5 1 Activatio
Functions
Definition: Activation Function
An activation function g : R → R introduces non-linearity into the network.
Without it, a stack of linear layers collapses to a single linear transformation:
WL · · · W1 x = Weff x.

6
IT549: Deep Learning Arpit Rana

5.1 Sigmoid (logistic)

Formula: Sigmoid
1 ′


σ(z) = , σ (z) = σ(z) 1 − σ(z)
1 + e−z
Range: (0, 1). Use in output layer for binary classification.

Disadvantages in hidden layers:

ˆ Vanishing gradient: σ ′ (z) → 0 for large |z|; gradients shrink as they propagate
through many layers.
ˆ Output not zero-centred: all activations positive, causing zig-zagging in gradient de-
scent.
ˆ Computationally expensive (exponential).

5.2 Rectified Linear Unit (ReLU)

Formula: ReLU
( (
z z>0 ′ 1 z>0
ReLU(z) = max(0, z) = ReLU (z) =
0 z≤0 0 z<0
Range: [0, ∞). Preferred for hidden layers in practice.

Advantages:
ˆ Does not suffer from vanishing gradient for z > 0.
ˆ Computationally very cheap (just a thresholding operation).
ˆ Sparse activation: roughly 50% of neurons output 0, promoting efficiency and implicit
regularisation.
Disadvantage – Dying ReLU: if a neuron’s pre-activation z is always ≤ 0, its
gradient is always 0 and the neuron never updates (“dies”).

5.3 Softmax (output layer for multiclass)

Formula: Softmax (repeated for reference)

K
ezk X
ŷk = PK , ŷk = 1
j=1 e zj k=1

5.4 Linear (output layer for regression)

g(z) = z ⇒ ŷ = z
No squashing — allows the output to take any real value, which is necessary for
predicting a continuous target.

7
IT549: Deep Learning Arpit Rana

5.5 Summary table

Function Layer type Task Range
Linear Output Regression R
Sigmoid Output Binary classification (0, 1)
Softmax Output Multiclass classif. (0, 1)K , sums to 1
Sigmoid Hidden (legacy) (0, 1)
ReLU Hidden Most tasks [0, ∞)

6 1 Loss
Functions
6.1 Mean Squared Error (regression)
Formula: MSE
m
1 X (i)
2
LMSE = ŷ − y (i)
m i=1

6.2 Binary Cross-Entropy (binary classification)

Formula: Binary Cross-Entropy
m
1 Xh (i) i
LBCE =− y log ŷ (i) + (1 − y (i) ) log(1 − ŷ (i) )
m i=1
Keras name: binary crossentropy

6.3 Categorical Cross-Entropy (multiclass)

Formula: Categorical Cross-Entropy
m K
1 X X (i) (i)
LCCE =− yk log ŷk
m i=1 k=1

ˆ sparse categorical crossentropy: labels are integers (0, 1, 2, . . . )

ˆ categorical crossentropy: labels are one-hot encoded

7 1 Optimisa
Algorithms
7.1 Gradient Descent variants

8
IT549: Deep Learning Arpit Rana

Variant Description
Batch GD Uses the entire training set to compute gradients per
update. Stable but very slow for large datasets.
Stochastic GD Uses one sample per update. Fast but very noisy
(SGD) gradients; may never converge to the exact minimum.
Mini-Batch GD Uses a batch of B samples. Best of both worlds.
Keras default (set via batch size).

7.2 Advanced optimisers

7.2.1 RMSprop
Maintains a running average of squared gradients E[g 2 ] and divides the learning rate by
p
E[g 2 ] + ϵ:
η
E[g 2 ]t = ρ E[g 2 ]t−1 + (1 − ρ) gt2 , θt+1 = θt − p gt
E[g 2 ]t + ϵ
Default learning rate in Keras: η = 0.001. Adaptive per-parameter step sizes help
navigate pathological curvature.

7.2.2 Adam (Adaptive Moment Estimation)

Combines momentum (first moment) and RMSprop (second moment):

mt = β1 mt−1 + (1 − β1 )gt (biased 1st moment) (5)

vt = β2 vt−1 + (1 − β2 )gt2 (biased 2nd moment) (6)
m̂t = mt /(1 − β1t ), v̂t = vt /(1 − β2t ) (bias-corrected) (7)
η
θt+1 = θt − √ m̂t (8)
v̂t + ϵ
Typical defaults: β1 = 0.9, β2 = 0.999, ϵ = 10−8 .

7.2.3 Comparison table

Optimiser Adaptive LR Momentum Notes

SGD No No Simple, baseline
Momentum No Yes Faster convergence
Adagrad Yes No LR shrinks monotonically
RMSprop Yes No Fixes Adagrad shrinkage
Adam Yes Yes Most popular default
Nadam Yes Nesterov Adam + Nesterov momentum

8 1 The
Keras Library

9
 IT549: Deep Learning Arpit Rana

8.1 Overview
Keras
Keras is a high-level deep-learning API for TensorFlow (and previously Theano /
CNTK). It was created by François Chollet at Google and first released in 2015. It
provides a simple, consistent interface for building, training, and evaluating neural
networks.

Trade-offs:

ˆ Pro: Very high-level; minimal boilerplate; rapid prototyping.

ˆ Con: Less fine-grained control than raw TensorFlow/PyTorch.

ˆ Solution: Mix Keras with TensorFlow ops when needed.

8.2 Three API styles

1. Sequential API: a plain stack of layers, each with one input and one output. Easiest;
limited to linear topologies.

2. Functional API: define layers as function calls and chain them; supports branching,
multiple inputs/outputs.

3. Model Subclassing: override init and call; full flexibility but most verbose.

8.3 Build, compile, fit — the Keras workflow

1 import tensorflow as tf
2 from tensorflow import keras
3 from tensorflow . keras import layers
4

5 # 1. Build
6 model = keras . Sequential ([
7 layers . Dense (64 , activation = ’ relu ’ , input_shape =( n_features ,)
),
8 layers . Dense (64 , activation = ’ relu ’) ,
9 layers . Dense ( n_output , activation = ’ softmax ’)
10 ])
11

12 # 2. Compile
13 model . compile (
14 optimizer = keras . optimizers . RMSprop ( learning_rate =0.001) ,
15 loss = ’ s p a r s e _ c a t e g o r i c a l _ c r o s s e n t r o p y ’ ,
16 metrics =[ ’ accuracy ’]
17 )
18

19 # 3. Fit
20 history = model . fit ( X_train , y_train ,
21 epochs =50 , batch_size =32 ,
22 validation_split =0.2)

10
 IT549: Deep Learning Arpit Rana

23

24 # 4. Evaluate
25 model . evaluate ( X_test , y_test )
Listing 1: Keras workflow skeleton

8.4 Important Keras layer types used in these lectures

Layer Description
Dense(units, Fully connected layer; every input neuron is
activation) connected to every output neuron.
[Link]() Computes mean and variance of training data;
normalises inputs at inference time.
[Link](1/255)Scales pixel values from [0, 255] to [0, 1].
[Link]() Reshapes a multi-dimensional input into 1-D.

9 1 Neural
Network for Regression — House Rent Prediction
9.1 Architecture choice
ˆ Input: 3 features — BHK (bedrooms), Size (sq ft), Bathrooms.

ˆ Hidden layers: 2 dense layers, 32 neurons each, ReLU activation.

ˆ Output: 1 neuron, linear activation (predict rent value).

9.2 Why linear output activation for regression?

ŷ = gout (z) = z
Rent can range from, say, $500 to $50,000 — any non-squashing behaviour. Sigmoid
would compress output to (0, 1); softmax would produce a probability distribution —
neither appropriate here.

9.3 Feature scaling for regression

Neural networks are sensitive to feature scale because:

1. Large-valued features dominate gradient updates.

2. Initial weights are typically small (∼ N (0, 0.1)); large inputs push pre-activations far
into saturation zones.

Keras solution: add a Normalization layer:

11
 IT549: Deep Learning Arpit Rana

1 normalizer = layers . Normalization ()

2 normalizer . adapt ( X_train ) # computes mean and variance
3

4 model = keras . Sequential ([

5 normalizer ,
6 layers . Dense (32 , activation = ’ relu ’) ,
7 layers . Dense (32 , activation = ’ relu ’) ,
8 layers . Dense (1) # linear activation ( default )
9 ])
10 model . compile ( optimizer = ’ rmsprop ’ , loss = ’ mse ’ , metrics =[ ’ mae ’ ])
Listing 2: Normalisation layer

10 1 Neural
Network for Binary Classification — Class Performance
10.1 Architecture
ˆ Input: 3 features — lecture attendance, lab score, CAO points.

ˆ Hidden layers: 2 dense layers, 64 neurons each, ReLU.

ˆ Output: 1 neuron, sigmoid activation.

10.2 Why sigmoid for binary output?

The sigmoid squashes z ∈ R to ŷ ∈ (0, 1), which is interpreted as P (y = 1 | x). The
decision boundary is:

ŷ ≥ 0.5 ⇒ class 1 ŷ < 0.5 ⇒ class 0

10.3 Loss: binary cross-entropy

 
L = − y log ŷ + (1 − y) log(1 − ŷ)
This is derived from the negative log-likelihood of a Bernoulli distribution and is the
canonical loss for binary classification.
1 model = keras . Sequential ([
2 layers . Normalization () , # feature scaling
3 layers . Dense (64 , activation = ’ relu ’) ,
4 layers . Dense (64 , activation = ’ relu ’) ,
5 layers . Dense (1 , activation = ’ sigmoid ’)
6 ])
7 model . compile ( optimizer = keras . optimizers . RMSprop (0.001) ,
8 loss = ’ bi n a ry _ c r os s e nt r o py ’ ,
9 metrics =[ ’ accuracy ’ ])
Listing 3: Binary classification network

12
IT549: Deep Learning Arpit Rana

11 1 Neural
Network for Multiclass Classification — Iris Dataset
11.1 Architecture
ˆ Input: 4 features — petal width, petal length, sepal width, sepal length.

ˆ Hidden layers: 2 dense layers, 64 neurons each, ReLU.

ˆ Output: 3 neurons (Setosa=0, Versicolor=1, Virginica=2), softmax.

11.2 Two label encoding choices

Integer labels vs One-Hot Encoding

Encoding Loss function Example label

Integer labels sparse categorical crossentropy 2
One-hot labels categorical crossentropy [0,0,1]
One-hot encoding with to categorical in Keras:
from tensorflow . keras . utils import to_categorical
y_train_oh = to_categorical ( y_train , num_classes =3)

11.3 Softmax output interpretation

For a test sample, the output is, e.g.:

ŷ = [0.02, 0.11, 0.87]

This means: P (Setosa) = 0.02, P (Versicolor) = 0.11, P (Virginica) = 0.87. Predicted
class: Virginica.

12 1 Image
Classification — Fashion MNIST
12.1 Dataset overview
ˆ 70,000 greyscale images of size 28 × 28 pixels.

ˆ 10 classes: T-shirt/top, Trouser, Pullover, Dress, Coat, Sandal, Shirt, Sneaker, Bag,
Ankle boot.

ˆ Split: 60,000 training, 10,000 test.

13
 IT549: Deep Learning Arpit Rana

12.2 Preprocessing: reshape and rescale

reshape ÷255
(60000, 28, 28) −−−−→ (60000, 784) −−−→ (60000, 784), values ∈ [0, 1]

1 X_train = X_train . reshape ( -1 , 28*28) . astype ( ’ float32 ’) / 255.0

2 X_test = X_test . reshape ( -1 , 28*28) . astype ( ’ float32 ’) / 255.0
Listing 4: Reshape and rescale

12.3 Architecture
Layer Type Units Activation
Input Dense (via reshape) 784 —
Hidden 1 Dense 300 ReLU
Hidden 2 Dense 100 ReLU
Output Dense 10 Softmax

1 model = keras . Sequential ([

2 layers . Rescaling (1./255 , input_shape =(784 ,) ) ,
3 layers . Dense (300 , activation = ’ relu ’) ,
4 layers . Dense (100 , activation = ’ relu ’) ,
5 layers . Dense (10 , activation = ’ softmax ’)
6 ])
7 model . compile ( optimizer = ’ rmsprop ’ ,
8 loss = ’ s p a r s e _ c a t e g o r i c a l _ c r o s s e n t r o p y ’ ,
9 metrics =[ ’ accuracy ’ ])
10 model . fit ( X_train , y_train , epochs =30 , batch_size =32 ,
11 validation_split =0.1)
Listing 5: Fashion MNIST model

12.4 Remarks on computer vision

Old vs New Computer Vision Pipeline
Old pipeline (1960s–1990s):
Raw image → hand-crafted feature extraction (SIFT, SURF, HOG, edge detectors)
→ ML classifier (SVM, logistic regression)
New pipeline (deep learning):
Raw pixel values → layers automatically discover features → output prediction
The key insight is that the layers of the neural network learn features automat-
ically from data, eliminating the need for domain-specific, hand-crafted feature
engineering.

14
IT549: Deep Learning Arpit Rana

13 1 Concludi
Remarks on Hyperparameter Selection
Exam Tip
Formulate every deep learning project in terms of five dimensions:
Data → Input → Output → Architecture → Loss function

Key challenges:

1. Constrained decisions: number of inputs; number of output neurons; output acti-

vation; loss function — these are determined by the task.

2. Free hyperparameters: number of hidden layers, neurons per layer, activation func-
tions in hidden layers, optimiser, learning rate, batch size, number of epochs.

3. Grid/random search: valid but very expensive — larger search space than classical
ML.

4. Overfitting risk: deep networks have millions of parameters; regularisation (dropout,

L2, early stopping, batch normalisation) is essential.

15
IT549: Deep Learning Arpit Rana

14 1 100+
Questions and Detailed Answers
How to use this section
Questions are grouped by topic. For each question, read it carefully, write your
own answer, then check against the detailed solution. Questions range from factual
recall (lower Bloom level) to application and analysis (higher Bloom levels).

14.1 Part A: Deep Learning Fundamentals (Q1–Q15)

Question 1
What does the word “deep” mean in deep learning?

Answer
“Deep” refers to the number of layers (depth) in the network, not to any philo-
sophical profundity. A deep network has tens or hundreds of layers between input
and output.

Question 2
State the Universal Approximation Theorem and explain its practical limitation.

Answer
The theorem states that a single-hidden-layer network can approximate any con-
tinuous function to arbitrary accuracy — but the required number of neurons may
be exponentially large. In practice, using more layers (depth) achieves the same
approximation power with exponentially fewer parameters, which is the main mo-
tivation for deep networks.

Question 3
Decompose a deep learning model into its constituent parts.

Answer
Model = Architecture + Parameters.

ˆ Architecture: the topology — number of layers, type of layers (dense, convolu-

tional, etc.), number of neurons per layer, activation functions.

ˆ Parameters: the learnable weights W[l] and biases b[l] for every layer l.

16
IT549: Deep Learning Arpit Rana

Question 4
Write out the equations for one full forward pass through a 2-hidden-layer dense
network for binary classification.

Answer
Let input x ∈ Rn :

z[1] = W[1] x + b[1]

a[1] = ReLU(z[1] )
z[2] = W[2] a[1] + b[2]
a[2] = ReLU(z[2] )
z [3] = w[3]⊤ a[2] + b[3]
ŷ = σ(z [3] )

Question 5
What happens if you remove all activation functions from a multi-layer network?

Answer
The entire network collapses to a single linear transformation:

W[L] · · · W[2] W[1] x + bias = Weff x + beff

No matter how many layers you stack, you can only represent a linear mapping,
which is no better than simple linear regression.

Question 6
What is the difference between supervised and unsupervised deep learning?

Answer
Supervised: the training set contains labelled pairs (x(i) , y (i) ). The model learns to
map inputs to labels by minimising a task-specific loss. All examples in the lecture
(regression, binary and multiclass classification) are supervised.
Unsupervised: no labels; the model discovers structure (clusters, generative fac-
tors) from raw data. Examples include autoencoders, GANs, variational autoen-
coders.

Question 7
Define an epoch and a batch in the context of training a neural network.

17
IT549: Deep Learning Arpit Rana

Answer
Epoch: one complete pass through the entire training dataset.
Batch (mini-batch): a subset of B training samples used to compute one gradient
update. After ⌈m/B⌉ batches, one epoch is complete.

Question 8
What is the role of the loss function in training a neural network?

Answer
The loss function L(ŷ, y) quantifies how far the model’s prediction ŷ is from
the true label y. It serves as the objective that gradient descent minimises. The
choice of loss function is determined by the task (regression vs classification) and it
must be differentiable for backpropagation.

Question 9
Why is it important to scale / normalise input features before training a neural
network?

Answer
1. Gradient symmetry: without scaling, features with large magnitude dominate
the gradient, causing elongated, ill-conditioned loss surfaces that are slow to
optimise.

2. Weight initialisation: weights are initialised near zero; large inputs push pre-
activations into saturation regions of sigmoid/tanh, causing vanishing gradients.

3. Learning rate sensitivity: unscaled features force very small learning rates to
prevent divergence.

Question 10
Describe the three main variants of gradient descent and state when each is pre-
ferred.

Answer
ˆ Batch GD: gradient computed on all m samples. Very stable, slow for large m.
Good for small datasets or convex problems.

ˆ Stochastic GD (SGD): one sample per update. Noisy, can escape local minima,
slow to converge. Rarely used in practice.

ˆ Mini-Batch GD: batch size B ∈ [32, 512]. Balances speed and stability. De-
fault in Keras (batch size argument).

18
IT549: Deep Learning Arpit Rana

Question 11
What is overfitting and why are deep networks particularly susceptible to it?

Answer
Overfitting occurs when a model learns the noise in the training data instead of
the underlying pattern, resulting in high training accuracy but poor test accuracy.
Deep networks are susceptible because they have millions of parameters — far
more than the number of training samples in many tasks — giving them the capacity
to memorise training data exactly.
Remedies: dropout, L2 regularisation, early stopping, data augmentation, batch
normalisation, using more data.

Question 12
State the three Heaton rules for choosing the number of hidden neurons.

Answer
Let nin = input features, nout = output neurons:
2
1. nhidden ≈ nin + nout
3
2. nin < nhidden < nin · nout

3. nhidden < 2 nin

These are heuristics, not hard rules. Cross-validation should guide the final choice.

Question 13
Why is scikit-learn insufficient for deep learning?

Answer
scikit-learn’s MLPClassifier/MLPRegressor has very limited support:

ˆ Only dense (MLP) layers; no CNN, RNN, attention, etc.

ˆ No GPU acceleration.

ˆ No access to custom loss functions, optimisers or callbacks.

ˆ Cannot handle raw tensors; limited to structured data.

TensorFlow/Keras and PyTorch provide tensor computation, auto-differentiation,

GPU support and a vast ecosystem of layer types.

Question 14
What is backpropagation and how does it relate to the chain rule?

19
IT549: Deep Learning Arpit Rana

Answer
Backpropagation is the algorithm that efficiently computes gradients ∂L/∂W[l] for
every layer by applying the chain rule of calculus in reverse through the network.
For a single output and one hidden layer:

∂L ∂L ∂ ŷ ∂z [2] ∂a[1] ∂z [1]

= · · · ·
∂W[1] ∂ ŷ ∂z [2] ∂a[1] ∂z [1] ∂W[1]
Intermediate values from the forward pass are cached and reused, making the com-
putation efficient.

Question 15
What is the vanishing gradient problem and which activation function helps avoid
it?

Answer
During backpropagation, gradients are multiplied by g ′ (z) at each layer. For sig-
moid, g ′ (z) ∈ (0, 0.25], so the gradient can shrink exponentially as it propagates
through many layers — the “vanishing gradient” problem. Early layers receive
near-zero gradients and learn very slowly.
ReLU has g ′ (z) = 1 for z > 0, so gradients are not compressed (no vanishing) in
the active region. This is the primary reason ReLU is preferred for hidden layers.

14.2 Part B: Activation Functions (Q16–Q30)

Question 16
Write the formula for the sigmoid function and derive its derivative.

Answer
1
σ(z) =
1 + e−z
Derivative (using quotient rule):

e−z 1 e−z
σ ′ (z) =


= · = σ(z) 1 − σ(z)
(1 + e−z )2 1 + e−z 1 + e−z

Maximum value: σ ′ (0) = 0.25. Approaches 0 as |z| → ∞.

Question 17
Why is ReLU preferred over sigmoid in hidden layers?

20
IT549: Deep Learning Arpit Rana

Answer
1. No vanishing gradient for z > 0: ReLU′ (z) = 1.

2. Computationally cheap: just max(0,z).

3. Sparse activation: roughly 50% of neurons output zero, acting as implicit

regularisation.

4. Faster convergence in practice observed empirically.

Sigmoid saturates for large |z|, causing near-zero gradients and slowing learning.

Question 18
Explain the Dying ReLU problem.

Answer
[l]
If a neuron’s pre-activation zj is always ≤ 0 (e.g. due to a large negative bias or
large negative weights), then ReLU(z) = 0 and ReLU′ (z) = 0 always. The gradient
is zero, so the neuron’s weights never update. The neuron is said to have “died.”
Solutions: Leaky ReLU (g(z) = max(αz, z) with small α > 0), ELU, or careful
weight initialisation (He/Kaiming initialisation).

Question 19
Write the softmax formula and show that its outputs sum to 1.

Answer
ezk
ŷk = PK
j=1 e zj
Sum:
K K PK zk
X X e zk k=1 e
ŷk = PK = PK z =1 ✓
k=1 k=1 j=1 ezj j=1 e
j

Question 20
For which tasks is each output activation function appropriate?

Answer
ˆ Linear (g(z) = z): regression — output must be any real value.

ˆ Sigmoid: binary classification — output is P (y = 1).

ˆ Softmax: multiclass classification — outputs are a probability distribution over

K classes.

21
IT549: Deep Learning Arpit Rana

Question 21
Why can sigmoid not be used as the output activation for regression?

Answer
Sigmoid squashes any input to (0, 1). If the target variable can be greater than 1 or
negative (e.g. house rent in dollars, temperature in Celsius), the network can never
predict the correct value — its range is fundamentally incompatible.

Question 22
Compare tanh and sigmoid activation functions.

Answer
ez − e−z
tanh(z) = , range: (−1, 1)
ez + e−z
ˆ Tanh is zero-centred: outputs in (−1, 1), so mean activation ≈ 0. This avoids
the all-positive gradient issue of sigmoid.

ˆ Both suffer vanishing gradients for large |z|.

ˆ Tanh is preferred over sigmoid in hidden layers for this reason.

ˆ ReLU is generally preferred over both.

Question 23
What is Leaky ReLU and how does it fix the dying ReLU problem?

Answer
(
z z>0
Leaky ReLU(z) = , α ≈ 0.01
αz z≤0
For z ≤ 0, the gradient is α ̸= 0, so the neuron still receives a small (nonzero)
gradient update even when inactive. The neuron cannot permanently “die.”

Question 24
Explain what softmax does numerically with an example.

Answer
Suppose z = [2.0, 1.0, 0.1] for 3 classes:

e2.0 = 7.389, e1.0 = 2.718, e0.1 = 1.105, S = 11.212

 
7.389 2.718 1.105
ŷ = , , = [0.659, 0.242, 0.099]
11.212 11.212 11.212

22
IT549: Deep Learning Arpit Rana

The model predicts class 0 with 65.9% confidence.

Question 25
What is the effect of temperature scaling on softmax outputs?

Answer
Temperature T modifies softmax as:

ezk /T
ŷk = P zj /T
je

ˆ T → 0: outputs approach a one-hot vector (argmax); very confident.

ˆ T = 1: standard softmax.

ˆ T → ∞: outputs approach uniform distribution; very uncertain.

Used in knowledge distillation and language model sampling.

Question 26
Why is ReLU not appropriate as an output activation for multiclass classification?

Answer
ReLU outputs values in [0, ∞) and does not produce a probability distribution.
Its outputs do not sum to 1, so they cannot be interpreted as class probabilities.
Softmax is specifically designed to produce a valid probability distribution over K
classes, which is required for cross-entropy loss computation.

Question 27
Name two activation functions that are zero-centred and explain why zero-
centredness matters.

Answer
Zero-centred: tanh (range (−1, 1)) and ELU (range (−1, ∞)).
Why it matters: if activations are all positive (like sigmoid), then the gradient
of the loss w.r.t. weights in a layer is either all positive or all negative (same sign
as δ [l] ). This forces weight updates to move only in the “positive quadrant” or
“negative quadrant,” causing zig-zagging and slow convergence.

Question 28
What is GELU and where is it used?

23
IT549: Deep Learning Arpit Rana

Answer
GELU (Gaussian Error Linear Unit):
 hp i
GELU(z) = z Φ(z) ≈ 0.5z 1 + tanh 2/π(z + 0.044715z 3 )

where Φ is the CDF of the standard normal. It smoothly gates the input by its
quantile. Used in transformer-based models (BERT, GPT) where it outperforms
ReLU in practice.

Question 29
Explain the concept of “saturating” activation functions.

Answer
A saturating function has regions where g ′ (z) ≈ 0 for large |z|. Sigmoid and tanh
are saturating: for z > 4 or z < −4, the gradient is essentially zero.
This is problematic because:

1. Neurons in saturation contribute almost nothing to the gradient.

2. Learning stalls (vanishing gradient).

ReLU is non-saturating for z > 0, avoiding this issue.

Question 30
For a binary classification problem with two output neurons (instead of one), what
activation function and loss function would you use?

Answer
Use softmax on the two output neurons, giving probabilities [ŷ0 , ŷ1 ] summing to
1. Loss: sparse categorical crossentropy (or categorical crossentropy with
one-hot labels).
Alternatively, one output neuron with sigmoid + binary crossentropy is equiva-
lent and more common. The two-neuron approach is valid but redundant.

14.3 Part C: Loss Functions (Q31–Q45)

Question 31
Derive the binary cross-entropy loss from the principle of maximum likelihood.

Answer
Assume y (i) ∈ {0, 1}. Model predicts ŷ (i) = P (y = 1|x(i) ).

24
IT549: Deep Learning Arpit Rana

Likelihood of the training set:

m
Y y (i)
1−y(i)
L= ŷ (i) 1 − ŷ (i)
i=1

Negative log-likelihood (to minimise):

m
X
y (i) log ŷ (i) + (1 − y (i) ) log(1 − ŷ (i) )
 
− log L = −
i=1

Dividing by m gives the binary cross-entropy.

Question 32
When do you use sparse categorical crossentropy vs
categorical crossentropy?

Answer
ˆ sparse categorical crossentropy: labels are integers (e.g. y ∈ {0, 1, 2}).
Keras internally one-hot encodes them.

ˆ categorical crossentropy: labels are already one-hot encoded (e.g. y =

[0, 0, 1]).

Both compute the same mathematical quantity; the difference is only in how labels
are represented.

Question 33
What is Mean Absolute Error (MAE) and how does it differ from MSE?

Answer
m m
1 X (i) 1 X (i)
MAE = |ŷ − y (i) |, MSE = (ŷ − y (i) )2
m i=1 m i=1

ˆ MSE is more sensitive to outliers (squares large errors).

ˆ MAE is more robust to outliers but not differentiable at 0.

ˆ In Keras, MAE is commonly used as a metric (to report), while MSE is the loss
(to optimise).

Question 34
Why is cross-entropy preferred over MSE for classification tasks?

25
IT549: Deep Learning Arpit Rana

Answer
1. Probabilistic grounding: cross-entropy is derived from maximum likelihood
estimation with a Bernoulli/categorical distribution.

2. Steeper gradients: when the model is very wrong, cross-entropy provides a

large gradient (− log(0+ ) → ∞), enabling faster correction. MSE with sigmoid
output produces very small gradients when predictions are near 0 or 1 (the
derivative of MSE w.r.t. z includes ŷ(1 − ŷ), which saturates).

3. Better convergence in practice for classification.

Question 35
Write the categorical cross-entropy for a single training example with K = 3 classes.

Answer
Let the true label be class 2 (one-hot: y = [0, 0, 1]) and predictions ŷ = [0.1, 0.2, 0.7]:
3
X
L=− yk log ŷk = −(0 · log 0.1 + 0 · log 0.2 + 1 · log 0.7) = − log(0.7) ≈ 0.357
k=1

Only the term for the true class contributes, which is why cross-entropy is efficient.

Question 36
What is Huber loss and when would you prefer it over MSE?

Answer
(
1
2
− y)2
(ŷ |ŷ − y| ≤ δ
Lδ (ŷ, y) =
δ|ŷ − y| − 12 δ 2 |ŷ − y| > δ
Quadratic for small errors (like MSE), linear for large errors (like MAE).
Use case: regression datasets with outliers. Huber loss is differentiable everywhere
(unlike MAE) and robust to outliers (unlike MSE).

Question 37
Why must the loss function be differentiable?

Answer
Gradient descent requires ∇θ L to update the parameters. If L is not differentiable
(e.g. accuracy = number of correct / total), the gradient is either undefined or zero
almost everywhere, making gradient descent impossible.
This is why accuracy is only used as a metric (for monitoring), never as the training
loss.

26
IT549: Deep Learning Arpit Rana

Question 38
How does the loss landscape differ between MSE and cross-entropy for a sigmoid
output?

Answer
With sigmoid + MSE, the gradient of the loss w.r.t. z includes ŷ(1 − ŷ) from the
chain rule. When ŷ is near 0 or 1 (model is confident but wrong), this term is near
0, causing slow learning.
With sigmoid + cross-entropy, the gradient of L w.r.t. z simplifies to (ŷ − y), which
does not include the saturation term. So when the model is confidently wrong, the
gradient is large (≈ ±1) and learning is fast.

Question 39
Explain label smoothing and why it helps.

Answer
Instead of hard labels yk ∈ {0, 1}, use soft labels:
ϵ
ỹk = yk (1 − ϵ) +
K
with small ϵ ≈ 0.1. This prevents the model from becoming overconfident (assign-
ing probability 1 to one class) and acts as a regulariser, improving generalisation
especially in multiclass settings.

Question 40
What is KL divergence and how does it relate to cross-entropy?

Answer

X P (k) X X
DKL (P ∥Q) = P (k) log =− P (k) log Q(k) + P (k) log P (k)
k
Q(k)
| k {z } |k {z }
cross-entropy H(P,Q) −H(P ), entropy of P

Since H(P ) (entropy of the true distribution) is constant w.r.t. the model parame-
ters, minimising cross-entropy H(P, Q) is equivalent to minimising DKL (P ∥Q), i.e.
making the predicted distribution Q as close to the true distribution P as possible.

Question 41
What is the role of metrics (e.g. accuracy, MAE) vs the loss function in Keras?

27
IT549: Deep Learning Arpit Rana

Answer
ˆ Loss function: used to compute gradients and update weights. Must be differ-
entiable.

ˆ Metrics: used only for monitoring training progress and evaluating the model.
They do not affect training. Can be non-differentiable (e.g. accuracy).

In Keras: [Link](loss=..., metrics=[...])

Question 42
For a multi-label classification problem (each sample may belong to multiple
classes), what loss function would you use?

Answer
Use binary cross-entropy on each output independently, with sigmoid (not soft-
max) on the output layer. Each output neuron independently predicts the proba-
bility of one class, and labels are binary vectors (e.g. [1, 0, 1, 1, 0]). Softmax would
force the probabilities to sum to 1, which is wrong here since multiple classes can
be simultaneously true.

Question 43
Why does increasing the number of epochs not always improve performance?

Answer
Beyond a certain number of epochs, the model begins to overfit: training loss
continues to decrease while validation loss starts to increase. The model memorises
training-set noise rather than generalising.
Solution: early stopping — monitor validation loss and stop training when it
stops improving for p consecutive epochs (patience p).

Question 44
What does the validation split argument in [Link]() do?

Answer
It reserves the specified fraction (e.g. 0.2 = 20%) of the training data as a validation
set used to monitor overfitting after each epoch. Keras evaluates the model on this
held-out portion and reports validation loss and metrics alongside training loss.
This data is not used for gradient updates.

Question 45
Explain the difference between loss and cost in deep learning.

28
IT549: Deep Learning Arpit Rana

Answer
Technically:

ˆ Loss L(i) : computed on a single training example.

ˆ Cost
Pm J: (i)the average (or sum) of losses over the entire training set: J =
1
m i=1 L .

In practice (and in Keras), both terms are often used interchangeably.

14.4 Part D: Optimisers (Q46–Q60)

Question 46
Write the parameter update rule for standard Stochastic Gradient Descent.

Answer

θ ← θ − η ∇θ L
where η is the learning rate and ∇θ L is computed on a single example (true SGD)
or a mini-batch (mini-batch GD).

Question 47
Explain the role of the learning rate η and the consequences of setting it too high
or too low.

Answer
ˆ Too high: gradient descent oscillates or diverges — parameters “overshoot” the
minimum and may blow up.

ˆ Too low: convergence is extremely slow; training takes too many epochs.

ˆ Just right: smooth convergence to a (local) minimum within a reasonable num-

ber of iterations.

Practical approach: start with η = 0.001 (Keras defaults) and tune with a learning-
rate schedule or grid search.

Question 48
Describe the RMSprop optimiser and state its default learning rate in Keras.

Answer
RMSprop maintains a running average of squared gradients:

E[g 2 ]t = ρ E[g 2 ]t−1 + (1 − ρ)gt2

29
IT549: Deep Learning Arpit Rana

Update:
η
θt+1 = θt − p gt
E[g 2 ]t + ϵ
This adapts the learning rate per parameter: parameters with large recent
gradients get a smaller effective step, and vice versa.
Default in Keras: η = 0.001, ρ = 0.9, ϵ = 10−7 .

Question 49
What is momentum in gradient descent and why does it help?

Answer
Momentum accumulates a “velocity” vector in the direction of persistent gradients:

vt = γvt−1 + η ∇θ L, θt+1 = θt − vt

Benefits:

1. Dampens oscillations in high-curvature directions.

2. Accelerates progress in low-curvature, consistent-gradient directions.

3. Can help escape shallow local minima.

Question 50
How does Adam combine momentum and RMSprop?

Answer
Adam maintains both:

ˆ mt (first moment = exponential moving average of gradients ∼ momentum).

ˆ vt (second moment = exponential moving average of squared gradients ∼ RM-

Sprop).

Bias-corrected estimates (m̂t , v̂t ) avoid zero-initialization bias in early steps. The
update is:
η
θt+1 = θt − √ m̂t
v̂t + ϵ
Adam is generally the most popular default optimiser due to robust, fast conver-
gence.

Question 51
What is Adagrad and what is its main disadvantage?

30
IT549: Deep Learning Arpit Rana

Answer
Adagrad accumulates the sum of squared gradients:
η
Gt = Gt−1 + gt2 , θt+1 = θt − √ gt
Gt + ϵ
Parameters with frequently large gradients get smaller updates. Good for sparse
data.
Disadvantage: Gt grows monotonically, so the effective learning rate shrinks to
near zero, halting learning prematurely. RMSprop fixes this by using an exponential
moving average instead of a cumulative sum.

Question 52
What is Nadam and how does it differ from Adam?

Answer
Nadam replaces the standard momentum term in Adam with Nesterov momen-
tum. Nesterov momentum “looks ahead”:

gtNesterov = ∇θ L(θt − γ m̂t−1 )

It evaluates the gradient at the lookahead position rather than the current position,
which can lead to better convergence, especially near minima.

Question 53
What is the batch size argument in [Link]() and how does it affect training?

Answer
batch size is the number of training samples used per gradient update.

ˆ Small batch (e.g. 1–32): noisy gradients; more updates per epoch; better gen-
eralisation (noise acts as regulariser); slower per epoch.

ˆ Large batch (e.g. 512–2048): stable gradients; fewer updates; faster per epoch;
may converge to sharp minima that generalise poorly.

ˆ Typical: 32–256.

Question 54
What is a learning rate schedule and give two common examples.

Answer
A learning rate schedule changes η during training:

1. Step decay: halve η every k epochs.

31
IT549: Deep Learning Arpit Rana

2. Exponential decay: ηt = η0 e−λt .

3. Cosine annealing: ηt = ηmin + 12 (ηmax − ηmin )(1 + cos(πt/T )).

4. Warm-up: start with small η, ramp up, then decay.

Helps avoid oscillation early in training and allows fine-grained convergence later.

Question 55
Explain the concept of a local minimum vs global minimum in the loss landscape.

Answer
ˆ Global minimum: the point θ∗ where L is absolutely lowest.

ˆ Local minimum: a point where L is lower than all nearby points but not globally
lowest.

In practice, deep networks have loss surfaces with many saddle points and shallow
local minima. Research suggests that in very high dimensions, most local minima
have similar loss values to the global minimum (Dauphin et al., 2014), so deep
networks are not as badly affected by local minima as previously feared.

Question 56
What is a saddle point and how does it affect gradient descent?

Answer
A saddle point is a critical point (∇θ L = 0) that is a minimum in some dimensions
and a maximum in others. Gradient descent can slow dramatically near saddle
points because gradients are near zero.
Optimisers with momentum (Adam, RMSprop) can escape saddle points more ef-
fectively than vanilla SGD.

Question 57
In the House Rent example, the lectures use RMSprop with default learning rate.
What would happen if you used a learning rate of 10?

Answer
A learning rate of 10 is extremely large. The parameter updates θ ← θ − 10 ∇θ L
would wildly overshoot the minimum, likely causing the loss to diverge (“explode”)
rather than decrease. In practice, the loss becomes NaN or inf. Gradient clipping
can partially mitigate this, but the root fix is to use a smaller η.

32
IT549: Deep Learning Arpit Rana

Question 58
What is gradient clipping and when is it used?

Answer
Gradient clipping caps the norm (or absolute value) of gradients before applying
the update:  
c
g ← g · min 1,
∥g∥
Used primarily in recurrent neural networks (RNNs) which are prone to ex-
ploding gradients over long sequences. In Keras:
keras . optimizers . RMSprop ( clipnorm =1.0)

Question 59
Explain why adaptive optimisers (Adam, RMSprop) are generally preferred over
vanilla SGD.

Answer
Adaptive optimisers automatically adjust the learning rate per parameter:

ˆ Parameters with sparse or small gradients receive larger updates.

ˆ Parameters with large, frequent gradients receive smaller updates.

This removes the need to hand-tune a single global learning rate and leads to faster,
more robust convergence on most problems. Vanilla SGD requires careful learning
rate tuning and scheduling.

Question 60
When might vanilla SGD outperform Adam?

Answer
Some research (Wilson et al., 2017) shows that SGD with momentum and a tuned
learning rate schedule can generalise better than Adam on image classification
benchmarks. Adam can converge to a sharp minimum (with large Hessian eigenval-
ues) that generalises poorly, while SGD tends to find flatter minima. In practice,
Adam is preferred for fast prototyping; SGD may be preferred for final fine-tuned
models.

33
IT549: Deep Learning Arpit Rana

14.5 Part E: Keras Implementation (Q61–Q75)

Question 61
Write Keras code for a regression network with: input size 5, two hidden layers of
[128, 64] neurons with ReLU, and one output neuron.

Answer
1 from tensorflow import keras
2 from tensorflow . keras import layers
3

4 model = keras . Sequential ([

5 layers . Dense (128 , activation = ’ relu ’ , input_shape =(5 ,) ) ,
6 layers . Dense (64 , activation = ’ relu ’) ,
7 layers . Dense (1) # linear activation ( default )
8 ])
9 model . compile ( optimizer = keras . optimizers . RMSprop (0.001) ,
10 loss = ’ mse ’ ,
11 metrics =[ ’ mae ’ ])
12 model . summary ()

Question 62
Write Keras code for a binary classification network with normalisation of input.

Answer
1 import tensorflow as tf
2 from tensorflow . keras import layers
3

4 norm = layers . Normalization ()

5 norm . adapt ( X_train ) # compute mean and variance
6

7 model = tf . keras . Sequential ([

8 norm ,
9 layers . Dense (64 , activation = ’ relu ’) ,
10 layers . Dense (64 , activation = ’ relu ’) ,
11 layers . Dense (1 , activation = ’ sigmoid ’)
12 ])
13 model . compile ( optimizer = ’ rmsprop ’ ,
14 loss = ’ b in a r y_ c r os s e nt r o py ’ ,
15 metrics =[ ’ accuracy ’ ])
16 history = model . fit ( X_train , y_train ,
17 epochs =100 , batch_size =32 ,
18 validation_split =0.2)

34
IT549: Deep Learning Arpit Rana

Question 63
Write Keras code for a multiclass network on the Iris dataset using one-hot encoding.

Answer
1 from tensorflow . keras . utils import to_categorical
2 y_train_oh = to_categorical ( y_train , num_classes =3)
3 y_test_oh = to_categorical ( y_test , num_classes =3)
4

5 model = keras . Sequential ([

6 layers . Dense (64 , activation = ’ relu ’ , input_shape =(4 ,) ) ,
7 layers . Dense (64 , activation = ’ relu ’) ,
8 layers . Dense (3 , activation = ’ softmax ’)
9 ])
10 model . compile ( optimizer = ’ rmsprop ’ ,
11 loss = ’ c a t e g o r i c a l _ c r o s s e n t r o p y ’ ,
12 metrics =[ ’ accuracy ’ ])
13 model . fit ( X_train , y_train_oh , epochs =200 , batch_size =16)
14 print ( model . evaluate ( X_test , y_test_oh ) )

Question 64
How do you access training history after calling [Link]()?

Answer
[Link]() returns a History object. Access metrics as:
1 history = model . fit (...)
2 print ( history . history . keys () )
3 # e . g . dict_keys ([ ’ loss ’, ’ mae ’, ’ val_loss ’, ’ val_mae ’])
4

5 import matplotlib . pyplot as plt

6 plt . plot ( history . history [ ’ loss ’] , label = ’ Train loss ’)
7 plt . plot ( history . history [ ’ val_loss ’] , label = ’ Val loss ’)
8 plt . legend () ; plt . show ()

Question 65
What is the purpose of the input shape argument in the first Dense layer?

Answer
input shape tells Keras the shape of one input sample (excluding the batch dimen-
sion), allowing it to:

1. Infer the weight matrix dimensions (nout × nin ).

2. Print a full model summary with parameter counts.

35
IT549: Deep Learning Arpit Rana

3. Perform shape validation before training.

Without it, Keras defers weight creation to the first call (lazy build).

Question 66
What does [Link]() display and why is it useful?

Answer
[Link]() prints:

ˆ Each layer’s name, type, output shape.

ˆ Number of trainable and non-trainable parameters per layer.

ˆ Total parameter count.

Useful for: verifying architecture; estimating memory requirements; checking that

layer shapes are compatible.

Question 67
How does Keras’s Normalization layer differ from scikit-learn’s StandardScaler?

Answer
ˆ Both standardise inputs to zero mean and unit variance.

ˆ Normalization is a Keras layer — it is part of the model and automatically

applied during inference.

ˆ StandardScaler is a separate preprocessing step outside the model; you must

manually apply it at inference time.

ˆ Normalization is call-specific to TensorFlow/Keras; it computes statistics via

.adapt(X train).

Question 68
What is the Functional API and when would you prefer it over the Sequential API?

Answer
1 inputs = keras . Input ( shape =( n ,) )
2 x = layers . Dense (64 , activation = ’ relu ’) ( inputs )
3 x = layers . Dense (64 , activation = ’ relu ’) ( x )
4 outputs = layers . Dense (1) ( x )
5 model = keras . Model ( inputs = inputs , outputs = outputs )

Prefer Functional API when:

ˆ The model has multiple inputs or outputs.

36
IT549: Deep Learning Arpit Rana

ˆ There are skip connections (ResNet-style).

ˆ There is shared layers (Siamese networks).

ˆ The topology is not a simple linear stack.

Question 69
How do you save and load a Keras model?

Answer
1 # Save entire model ( architecture + weights + compile info )
2 model . save ( ’ my_model . keras ’)
3

4 # Load
5 loaded_model = keras . models . load_model ( ’ my_model . keras ’)
6

7 # Save only weights

8 model . save_weights ( ’ weights . h5 ’)
9 model . load_weights ( ’ weights . h5 ’)

Question 70
What is a Keras callback and give two examples used in practice.

Answer
A callback is a function called at certain points during training (end of each epoch,
batch, etc.).
Examples:
1 callbacks = [
2 keras . callbacks . EarlyStopping (
3 monitor = ’ val_loss ’ , patience =10 , r e s t o r e _ b e s t _ w e i g h t s
= True ) ,
4 keras . callbacks . ModelCheckpoint (
5 filepath = ’ best_model . keras ’ , save_best_only = True ) ,
6 keras . callbacks . Reduc eLROnP lateau (
7 monitor = ’ val_loss ’ , factor =0.5 , patience =5)
8 ]
9 model . fit (... , callbacks = callbacks )

Question 71
What is dropout in Keras and how does it act as regularisation?

37
IT549: Deep Learning Arpit Rana

Answer
Dropout randomly sets a fraction p of neurons to 0 during each training step:
1 layers . Dropout ( rate =0.3) # drop 30% of neurons

Regularisation mechanism: forces the network to not rely on any single neuron,
learning redundant representations. At test time, all neurons are active but outputs
are scaled by (1 − p).

Question 72
What does [Link]() return and how does it differ from
[Link]()?

Answer
ˆ [Link](X): returns the model’s raw output (e.g. probabilities for clas-
sification, numeric values for regression). No labels needed.

ˆ [Link](X, y): computes the loss and metrics on labelled data. Re-
turns scalar values.

Question 73
How would you convert softmax probabilities to class predictions in NumPy?

Answer
1 import numpy as np
2 proba = model . predict ( X_test ) # shape : (m , K )
3 y_pred = np . argmax ( proba , axis =1) # shape : (m ,)

[Link] returns the index of the maximum probability, which corresponds to the
predicted class.

Question 74
What is the Rescaling layer in Keras?

Answer
[Link](scale, offset=0.0) multiplies inputs by scale and adds
offset. Common use case:
1 layers . Rescaling (1./255) # scales pixel [0 ,255] to [0 ,1]
2 layers . Rescaling (1./127.5 , offset = -1) # scales to [ -1 ,1]

Unlike Normalization, Rescaling uses a fixed scale (not computed from data), so
no .adapt() call is needed.

38
IT549: Deep Learning Arpit Rana

Question 75
Explain the Sequential API’s limitation with an example.

Answer
The Sequential API only supports linear, single-input single-output topologies.
Example of what it cannot do: A ResNet skip connection:

a[l+2] = g W[l+2] a[l+1] + b[l+2] + a[l]

The addition of a[l] (a shortcut) is not expressible in the Sequential API; the
Functional API or subclassing is required.

14.6 Part F: Task-Specific Architecture (Q76–Q90)

Question 76
Design the full architecture (input, hidden, output layers, activations, loss) for a
neural network that predicts house prices given 10 features.

Answer
ˆ Input: 10 neurons (one per feature); add Normalization layer.

ˆ Hidden 1: 32 neurons, ReLU.

ˆ Hidden 2: 16 neurons, ReLU.

ˆ Output: 1 neuron, linear (default).

ˆ Loss: MSE (mse).

ˆ Metric: MAE (mae).

ˆ Optimiser: RMSprop, η = 0.001.

Question 77
Design the architecture for classifying handwritten digits (0–9) from 28×28 pixel
greyscale images using only dense layers.

Answer
ˆ Preprocessing: flatten to 784-D; rescale by 1/255.
ˆ Input: 784 neurons.
ˆ Hidden 1: 300 neurons, ReLU.
ˆ Hidden 2: 100 neurons, ReLU.
ˆ Output: 10 neurons, softmax.

39
IT549: Deep Learning Arpit Rana

ˆ Loss: sparse categorical crossentropy.

ˆ Metric: accuracy.

ˆ Optimiser: Adam or RMSprop.

Question 78
Why do image classification tasks benefit from convolutional layers rather than
dense layers?

Answer
Dense layers treat each pixel independently and ignore spatial structure. A cat’s
ear at position (10,10) and at (200,200) are treated as completely different features.
Convolutional layers:

1. Are translation equivariant: the same filter detects a feature regardless of its
location.

2. Use weight sharing: one filter applied across the entire image — far fewer
parameters.

3. Exploit local connectivity: pixels near each other are more correlated; local
receptive fields capture spatial patterns.

Question 79
What is the old (pre-deep-learning) computer vision pipeline and how does deep
learning replace it?

Answer
Old pipeline:

Image → SIFT/SURF/HOG/Edge detectors → SVM/LR

| {z }
manual feature engineering

Deep learning pipeline:

Raw pixels → CNN layers → ŷ

| {z }
automatic feature
discovery

Deep networks learn features directly from data. This eliminates the need for
domain-expert-designed feature extractors, leading to better generalisation and
state-of-the-art performance on vision benchmarks.

40
IT549: Deep Learning Arpit Rana

Question 80
Why do we flatten the Fashion MNIST images before feeding them into a dense
network?

Answer
Dense (fully connected) layers expect a 1-D input vector. The Fashion MNIST
images are stored as 2-D arrays of shape (28, 28). Flattening reshapes each image
to a 784-D vector:
flatten
(28, 28) −−−→ (784, )
This discards spatial structure (which is why CNNs are preferred for images), but
allows dense layers to process all pixel values as independent features.

Question 81
Why is it “a bad idea to feed pixel values much larger than initial weights” into a
network?

Answer
Initial weights are typically drawn from N (0, 0.01) or using He/Glorot initialisation
(values near 0). Pre-activations:
X
z= w i xi + b
i

If xi ∈ [0, 255] and wi ≈ 0.01, then z can be very large (up to 784 × 0.01 × 255 ≈
2000). This:

ˆ Pushes sigmoid/tanh neurons into deep saturation.

ˆ Causes vanishing gradients immediately.

ˆ Makes training unstable or ineffective.

Dividing by 255 brings inputs to [0, 1], matching the scale of initial weights.

Question 82
In the Iris dataset example, the accuracy was reported as ≈ 0.90. Why might this
be considered “not great” for a 3-class problem?

Answer
ˆ Iris is a classic, small, very simple dataset.

ˆ Classical classifiers (k-NN, SVM, decision tree) routinely achieve 95–100% ac-
curacy on Iris.

ˆ 90% accuracy means ∼5 errors on 50 test samples, which is worse than a k-NN
classifier.

41
IT549: Deep Learning Arpit Rana

ˆ The lecture notes observe that deep learning is often not the best approach
for small, structured/tabular datasets. Classical ML methods (random
forests, gradient boosting) typically outperform neural networks on such data.

Question 83
What neural network architecture would you use for a binary classification problem
where the input is an RGB image of size 224 × 224?

Answer
1. Do not flatten: 224 × 224 × 3 = 150,528 inputs — too many for a dense layer.

2. Use a Convolutional Neural Network (CNN):

ˆ Multiple convolutional + pooling blocks.

ˆ Flatten or Global Average Pooling at the end.
ˆ Dense hidden layers.
ˆ 1 output neuron, sigmoid, binary cross-entropy.

3. Alternatively, use a pretrained model (VGG, ResNet, EfficientNet) via transfer

learning.

Question 84
How many parameters does a Dense layer with 64 input neurons and 32 output
neurons have?

Answer

Weights: 64 × 32 = 2048
Biases: 32 (one per output neuron)
Total: 2048 + 32 = 2080 trainable parameters

Question 85
Given the Fashion MNIST architecture (784 → 300 → 100 → 10), compute the
total number of trainable parameters.

Answer

Layer 1 (Dense 300) : 784 × 300 + 300 = 235,500

Layer 2 (Dense 100) : 300 × 100 + 100 = 30,100
Layer 3 (Dense 10) : 100 × 10 + 10 = 1,010
Total : 266,610 parameters

42
IT549: Deep Learning Arpit Rana

Question 86
What is the purpose of the output layer having exactly K neurons for K-class
classification?

Answer
Each of the K output neurons computes a score zk for one class. After softmax,
ŷk = P (class k|x).
If we used fewer neurons (e.g. K − 1 with threshold rules), the model cannot simul-
taneously represent the probability of all K classes, and training with cross-entropy
requires one output per class. Using exactly K neurons is both mathematically
correct and computationally convenient.

Question 87
What is meant by “compatible consecutive layers”?

Answer
Layer l produces output of shape (batch, dl ). Layer l +1 must accept input of shape
(batch, dl ). If the shapes don’t match, Keras raises a shape mismatch error.
In a Sequential model, each Dense layer takes the previous layer’s output size as its
input size automatically. The only shape you need to specify explicitly is the first
layer’s input shape.

Question 88
What is an “encoding” in the context of deep networks and why is it useful?

Answer
In deep networks, the hidden layers progressively encode the input into increasingly
abstract, compact representations. For example, in image classification:
Layer 1 encodes edges → Layer 2 encodes shapes → Layer 3 encodes parts → Output
layer classifies.
This is useful because:

1. Learned features are often more powerful than hand-crafted ones.

2. The final hidden layer’s activations can be reused for other tasks (transfer learn-
ing).

3. Autoencoders exploit this to compress data (unsupervised learning).

Question 89
For structured (tabular) data, why might a random forest outperform a neural
network?

43
IT549: Deep Learning Arpit Rana

Answer
1. Sample efficiency: random forests require less data. Neural networks need
large datasets to outperform classical methods.

2. No feature scaling required: trees are invariant to monotonic transformations

of features.

3. Handles categorical features natively: neural networks require encoding

(one-hot, embedding).

4. Fewer hyperparameters: random forests have fewer critical hyperparameters.

5. Interpretability: tree-based models offer feature importances.

Question 90
Describe the five key dimensions to frame a deep learning project (from the lecture).

Answer
1. Data: what is available? how much? is it labelled? what preprocessing?

2. Input: what format? image, tabular, text, audio? what shape?

3. Output: what to predict? regression, binary, multiclass? how many classes?

4. Architecture: how many layers, neurons, what types (dense, CNN, RNN)?
what activation functions?

5. Loss function: determined by the output type; MSE, BCE, CCE.

14.7 Part G: Advanced Concepts & Analysis (Q91–Q110)

Question 91
What is the bias-variance trade-off in the context of neural networks?

Answer
2
E[(ŷ − y)2 ] = Bias
| {z } + |Variance
{z } + σ2
|{z}
underfitting overfitting irreducible noise

ˆ High bias (underfitting): model too simple; cannot capture the true function.
Fix: increase depth/width, reduce regularisation.

ˆ High variance (overfitting): model too complex; memorises noise. Fix: reduce
depth/width, add regularisation, gather more data.

Deep networks have very low bias but high variance — regularisation is crucial.

44
IT549: Deep Learning Arpit Rana

Question 92
What is weight initialisation and why does it matter?

Answer
Weight initialisation sets the starting values of W[l] before training.
Bad initialisation:

ˆ All zeros: all neurons compute identical gradients; no symmetry breaking.

ˆ Too large: saturates activations from the start.

Good initialisations:
h q q i
ˆ Glorot/Xavier: U − nin +n
6
out
, 6
nin +nout
. For sigmoid/tanh.
 q 
ˆ He/Kaiming: N 0, n2in . For ReLU.

Keras uses Glorot uniform by default for Dense layers.

Question 93
What is batch normalisation and what problem does it solve?

Answer
Batch normalisation (BN) normalises the pre-activations z[l] to have zero mean and
unit variance within a mini-batch, then applies a learned scale and shift:
zj − µB
ẑj = p 2 , z̃j = γ ẑj + β
σB + ϵ

Benefits:

1. Reduces internal covariate shift (distribution of activations changes as earlier

layers update).

2. Allows higher learning rates.

3. Acts as a regulariser (reduces need for dropout).

4. Mitigates vanishing/exploding gradients.

Question 94
What is transfer learning and when is it beneficial?

Answer
Transfer learning uses the weights of a model pretrained on a large dataset (e.g.
ImageNet) as the starting point for a new task:

45
IT549: Deep Learning Arpit Rana

1 base = keras . applications . VGG16 ( include_top = False , weights = ’

imagenet ’)
2 base . trainable = False # freeze pretrained weights
3 x = layers . G l o b a l A v e r a g e P o o l i n g 2 D () ( base . output )
4 out = layers . Dense (10 , activation = ’ softmax ’) ( x )
5 model = keras . Model ( inputs = base . input , outputs = out )

When useful:

ˆ Small dataset (not enough to train from scratch).

ˆ Similar domain (e.g. medical images from a model trained on natural images).

Early layers learn general features (edges, textures); these transfer well.

Question 95
What is data augmentation and how does it help neural network training?

Answer
Data augmentation artificially expands the training set by applying label-
preserving transformations to existing samples (e.g. random flips, rotations,
crops, colour jitter for images).
Benefits:

1. Increases effective dataset size.

2. Forces the model to be invariant to the applied transformations.

3. Acts as a strong regulariser, reducing overfitting.

In Keras:
1 keras . Sequential ([
2 layers . RandomFlip ( " horizontal " ) ,
3 layers . RandomRotation (0.1) ,
4 layers . RandomZoom (0.2)
5 ])

Question 96
Explain the concept of a “hyperparameter” and distinguish it from a “parameter”.

Answer
ˆ Parameters (e.g. weights W, biases b): learned from data via gradient descent
during training.

ˆ Hyperparameters (e.g. learning rate, number of layers, neurons per layer, batch
size, dropout rate, activation function): set before training; not learned by gradi-

46
IT549: Deep Learning Arpit Rana

ent descent. Must be tuned via cross-validation, grid search, random search, or
Bayesian optimisation.

Question 97
What is the “no free lunch” theorem and its implication for deep learning?

Answer
The No Free Lunch (NFL) theorem states that no learning algorithm performs best
on all possible problems. An algorithm that outperforms others on some problems
must necessarily perform worse on others.
Implication for deep learning: deep networks are not universally superior. For
small, structured/tabular datasets, classical ML (gradient boosting, SVMs) often
outperforms neural networks (as observed in the Iris example). The choice of model
should be guided by the data characteristics, not fashion.

Question 98
What is the difference between underfitting and overfitting, and how can you detect
each from training curves?

Answer

Underfitting Overfitting
Train loss High Low
Val loss High High (but > train)
Gap Small Large
Fix More capacity Regularise / more data

In training curves: overfitting appears as the validation loss curve diverging upward
while the training loss continues to decrease.

Question 99
What is L2 regularisation (weight decay) and write its modified loss function.

Answer
L2 regularisation adds a penalty proportional to the squared magnitude of all
weights: X
Lreg = Loriginal + λ ∥W[l] ∥2F
l

where λ is the regularisation strength. In Keras:

1 layers . Dense (64 , activation = ’ relu ’ ,
2 ke rn el _r eg ul ar iz er = keras . regularizers . l2 (0.01) )

L2 encourages small weights, reducing model complexity and overfitting.

47
IT549: Deep Learning Arpit Rana

Question 100
What is L1 regularisation and how does it differ from L2?

Answer
X
LL1 = L + λ ∥W[l] ∥1
l

ˆ L1 penalty encourages sparsity: many weights go exactly to zero (feature selec-

tion effect).

ˆ L2 penalty encourages weights to be small but non-zero.

ˆ L1 is not smooth at zero (subgradient methods needed).

ˆ In practice, Elastic Net (L1+L2) combines both benefits.

Question 101
Explain how to use early stopping in Keras and describe the patience parameter.

Answer
1 es = keras . callbacks . EarlyStopping (
2 monitor = ’ val_loss ’ ,
3 patience =10 , # wait 10 epochs for
improvement
4 re s t o r e _ b e s t _ w e i g h t s = True # revert to best checkpoint
5 )
6 model . fit ( X_train , y_train , epochs =1000 , callbacks =[ es ])

patience=10: if the monitored metric does not improve for 10 consecutive epochs,
training stops. restore best weights=True restores the weights from the epoch
with the best validation loss, avoiding the model’s degraded state at the stopping
point.

Question 102
What is one-hot encoding and why is it used for categorical class labels?

Answer
One-hot encoding represents a class label k ∈ {0, 1, . . . , K − 1} as a binary vector
of length K with a 1 only at position k:

k = 2, K = 4 ⇒ [0, 0, 1, 0]

Why: integer labels (0, 1, 2, . . . ) imply an ordinal relationship that does not
exist between unordered classes. One-hot encoding treats all classes as equidistant.
Required when using categorical crossentropy loss.

48
IT549: Deep Learning Arpit Rana

Question 103
What is the Fashion MNIST classification task and what accuracy is typically
achievable with a dense network vs a CNN?

Answer
Task: classify 28×28 greyscale fashion images into 10 categories.

Model Approx. Test Accuracy

Dense (2 hidden layers) ∼87–89%
Simple CNN ∼92–94%
Deep CNN (ResNet) >95%

CNNs exploit spatial structure and are far more parameter-efficient, leading to
better performance on image data.

Question 104
What is the purpose of the [Link]() step in Keras?

Answer
[Link]() configures the model for training by specifying:

1. Optimiser: algorithm and hyperparameters for gradient-based updates.

2. Loss function: the objective to minimise.

3. Metrics: quantities to monitor (but not optimise).

Without compiling, [Link]() will raise an error.

Question 105
What are the class labels in the Fashion MNIST dataset?

Answer
The 10 classes (integer labels 0–9) are:

49
IT549: Deep Learning Arpit Rana

Label Class
0 T-shirt/top
1 Trouser
2 Pullover
3 Dress
4 Coat
5 Sandal
6 Shirt
7 Sneaker
8 Bag
9 Ankle boot

Question 106
Why does the Fashion MNIST dataset not need a Normalization layer?

Answer
All features (pixel values) are in the same range [0, 255]. Normalisation addresses
the problem of features on different scales. Since all pixels share the same scale,
feature-wise standardisation is not strictly necessary.
However, it is still a good idea to rescale to [0, 1] by dividing by 255 (using a
Rescaling layer or manual division), because values of 255 are much larger than
typical initial weights.

Question 107
Compare the Keras Sequential API, Functional API, and Model Subclassing.

Answer

Sequential Functional Subclassing

Topology Linear stack DAG Any
Ease Easiest Medium Hardest
Skip con- No Yes Yes
nections
Multiple No Yes Yes
I/O
Debugging Easy Medium Hardest
Use case Simple MLP ResNet, Siamese Research

Question 108
Describe three ways to prevent overfitting in a Keras deep learning model.

50
IT549: Deep Learning Arpit Rana

Answer
1. Dropout: randomly zero out neurons during training. [Link](0.3)

2. L2 regularisation: add weight penalty to loss.

kernel regularizer=[Link].l2(0.01)

3. Early stopping: stop when validation loss plateaus.

EarlyStopping(patience=10)

4. Data augmentation: artificially expand training data.

5. Reduce model capacity: fewer layers or neurons.

Question 109
What is the significance of the “holdout” strategy for Fashion MNIST?

Answer
Fashion MNIST has 70,000 samples — large enough that holdout validation
(splitting into fixed train/test sets) provides a reliable, low-variance estimate of
generalisation performance.
For small datasets (like Iris, 150 samples), holdout gives high-variance estimates;
k-fold cross-validation is preferred.
The Fashion MNIST dataset comes pre-partitioned into 60,000 training and 10,000
test images, so we use these canonical splits directly.

Question 110
Summarise the constrained vs free hyperparameters in designing a Keras neural
network.

Answer
Constrained (determined by the task):

ˆ Number of input neurons: = number of features.

ˆ Number of output neurons: = 1 (regression/binary), = K (multiclass).

ˆ Output activation: linear/sigmoid/softmax.

ˆ Loss function: MSE/BCE/CCE.

Free (hyperparameters to tune):

ˆ Number of hidden layers.

ˆ Neurons per hidden layer.

ˆ Hidden activation function (sigmoid, ReLU, tanh, . . . ).

ˆ Optimiser choice and learning rate.

51
IT549: Deep Learning Arpit Rana

ˆ Batch size.

ˆ Number of epochs / early stopping patience.

ˆ Regularisation strength and type.

ˆ Weight initialisation scheme.

52
IT549: Deep Learning Arpit Rana

14 1 Quick
Reference Cheat Sheet
Output Layer Quick Reference

Task Output neurons Activation Loss

Regression 1 Linear MSE

Binary classif. 1 Sigmoid binary crossentropy
Multiclass (int) K Softmax sparse categorical crossentropy
Multiclass (one-hot) K Softmax categorical crossentropy
Multi-label K Sigmoid binary crossentropy

Activation Function Reference

1 ezk
σ(z) = , ReLU(z) = max(0, z), softmaxk (z) = P zj
1 + e−z je

ez − e−z
tanh(z) = , LeakyReLU(z) = max(αz, z)
ez + e−z

Keras Compile Arguments

model . compile (
optimizer = ’ rmsprop ’ # or Adam , SGD , ...
| keras . optimizers . RMSprop ( learning_rate
=0.001) ,
loss = ’ mse ’
| ’ b i n ar y _ cr o s se n t ro p y ’
| ’ sparse_categorical_crossentropy ’
| ’ categorical_crossentropy ’,
metrics = [ ’ mae ’] # regression
| [ ’ accuracy ’] # classification
)

End of Notes
Arpit Rana · IT549 Deep Learning · Jan 2026

53