BAI701, DL & RL

MODULE 3
Training Supervised Deep Learning Networks

Supervised deep learning involves training a model using labelled data, where each input is
paired with a class label. The goal is for the model to learn parameters that enable it to
generalize and correctly classify new, unseen data. Among various architectures,
Convolutional Neural Networks (CNNs) are widely used for supervised learning tasks. This
chapter focuses on training CNNs.

3.1 Training Convolution Neural Networks

Training a supervised deep neural network (DNN) involves optimizing network weights to
minimize a loss function using labeled data. The most commonly used method is gradient
descent-based backpropagation, which consists of:
1. Forward Pass: Input is passed through the network to generate an output.
2. Error Calculation: Output is compared with the true label using a cost/loss function.
3. Backward Pass (Backpropagation): The error is propagated backward from the output
to input layer to compute gradients.
4. Weight Update: Gradients are used to adjust the weights, reducing the error iteratively.

Initially, weights are randomly initialized. Over time, through repeated training, the network
improves its predictions by minimizing the error. To understand the working of
backpropagation, a Convolutional Neural Network (CNN) model, the Table 1 outlines the
layer-wise operations of the CNN architecture, and this structure is visually illustrated in the
figure 1 below, which resembles the classic LeNet model. The figure helps reinforce the flow
of data through the convolutional, pooling, and fully connected layers, culminating in the final
softmax output used for classification.
Table 1: CNN Architecture Summary

Layer Type Operation

Input 32×32 Raw image input
C1 Convolution 6 filters of size 5×5 → 6 feature maps of size 28×28
P2 Max Pooling Downsample to 14×14
C3 Convolution 16 filters → 16 feature maps of size 10×10
P4 Max Pooling Downsample to 5×5
C5 Convolution 120 filters of size 5×5 → Output size 1×1
F6 Fully Connected 10 neurons (1 per class)
Softmax Output Layer Probabilities for 10 classes

DR. SHRUTHI U, AI&ML, RNSIT 1

 BAI701, DL & RL

Fig. 3.1 CNN architecture

Let us describe the CNN architecture in detail, including its mathematical operations.
Layer 1 (C1): Convolution Layer:
 Input Size: 32 × 32
 Number of Filters: 6 filters, each of size 5 × 5
 Output: 6 feature maps of size 28 × 28
Mathematical Operation:
The convolution operation for each feature map is defined as:

This operation performs the weighted sum of a local patch of the input image using the filter
weights, followed by a bias term bk, producing the linear output.
Activation Function – ReLU:
To introduce non-linearity, a Rectified Linear Unit (ReLU) is applied to the output of the
convolution:

Thus, the final output of the convolution layer with ReLU is:

Layer C1 in the CNN architecture performs convolution using six 5×5 filters on a 32×32 input image,
followed by ReLU activation, resulting in six 28×28 output feature maps.

DR. SHRUTHI U, AI&ML, RNSIT 2

 BAI701, DL & RL

Layer 2 (P2): Max-Pooling Layer

 Input: 6 feature maps from Convolution Layer C1
 Operation: Applies max-pooling to each feature map independently
Mathematical Operation:
For each output feature map P2k, the operation is:

This operation reduces the spatial dimensions by selecting the maximum value in a local
2×2 region (with a stride of 2), effectively downsampling the feature map.

Where i and j represent the row and column indices of a feature map, while k denotes the index
of the feature map (channel number)
 Max-pooling reduces each 28×28 feature map to a 14×14 feature map.
 It helps in downsampling, dimensionality reduction, and increasing translation
invariance.
Layer 3 (C3): Second Convolution Layer
 Takes input from P2 (16 channels), applies filters to generate 16 feature maps of size
10×10.
 Each output is computed as:

Layer 4 (P4): Max-Pooling Layer

 Performs max-pooling on the output from C3.
 Downsamples 16 feature maps of size 10×10 to 16 feature maps of size 5×5.
 For each map:

Layer 5 (C5): Third Convolution Layer

 Input from P4 is 16 channels of size 5×5.

DR. SHRUTHI U, AI&ML, RNSIT 3

 BAI701, DL & RL

 Applies 120 filters of size 5×5 (same as input size) to produce 120 output maps of
size 1×1.
 As input and filter size are same, output is scalar:

Layer 6 (F6): Fully Connected Layer

 Connects all 120 outputs from C5 to 10 output neurons (one per class).
 Uses Softmax function:

 Final output Zk gives the probability of input belonging to class k.

Backward Pass

In supervised deep learning, training aims to minimize the loss function by updating network
weights. During each iteration:
1. Input is fed forward through the CNN to generate output.
2. The output is compared with the true label to compute the loss/error.
3. The loss is used to compute gradients with respect to each weight.
4. Weights are updated using gradients to reduce error, repeated until convergence.
Loss Function
The Mean Squared Error (MSE) is one of the loss functions used with softmax:
Loss=E(Z,target)

 Zk: Output of softmax layer (probability of class k)

 targetk: Ground truth (one-hot encoded label)
The objective of training is to minimize this loss by updating the model weights through
gradient descent. The loss function E is composed of several nested functions
corresponding to different layers in the network, expressed as:

DR. SHRUTHI U, AI&ML, RNSIT 4

 BAI701, DL & RL

Since this composition is differentiable, we can apply the chain rule to compute gradients with
respect to any parameter. Specifically, the gradient of the loss with respect to a weight wk in
the last layer (F6) is given by:

 E: Total loss (cost function)

 Zk: Output of softmax
 F6k: Output of last layer F6
 𝑤𝑖𝑘 : ith weight of kth neuron of last layer.
To update the weight after finding the derivative, use the gradient descent rule:

where μ is the learning rate that controls the size of the weight update. Each component in the
chain rule is computed as follows.
First, the derivative of the loss function with respect to the softmax output is:

Second, the derivative of the softmax output with respect to its input is:

which is the standard derivative of the softmax activation function. The output of the fully
connected layer F6k is the weighted sum of the previous layer's activations:

and the derivative of this with respect to weight 𝑤𝑖𝑘 is:

From the Eq.

DR. SHRUTHI U, AI&ML, RNSIT 5

 BAI701, DL & RL

The derivative of above function is given as

Combining all the components, the final gradient of the loss function with respect to weight
𝑤𝑖𝑘 becomes:

To simplify further computations and prepare for backpropagation through earlier layers,
define a delta value for each neuron in layer F6 as:

This delta captures the error signal for each output neuron and helps in computing the gradient
compactly as:

Finally, each weight is updated by subtracting the scaled gradient:

This concludes the process of computing the backward pass for the output layer. These
gradients are then propagated backward through hidden layers to complete the training loop.

Hidden Layer(C5)

In the backward pass of a CNN, after updating the fully connected layer (F6), the weights of
Hidden Layer C5 are updated next. During the forward pass, each output feature map C5k in
C5 is computed using a weighted sum of inputs from the previous pooling layer (P4), followed
by a ReLU activation. The formula is:

To update the weights, the error ek is first backpropagated from the output layer F6 using:

DR. SHRUTHI U, AI&ML, RNSIT 6

 BAI701, DL & RL

This error is used to calculate the delta for each C5 neuron using the derivative of the ReLU
activation:

(𝑚,𝑛)
Finally, the gradient of the error with respect to each weight 𝑤𝑘,𝑑 in this layer is calculated
by:

This gradient is then used in gradient descent to update the weights in layer C5.

Hidden Layer (C3)

In the hidden layer C3, error gradients are backpropagated from layer C5 through the max-
pooling layer P4. The error at each position (i,j)) in C3 is calculated by summing over all 120
filters of C5:

This error is used to compute the delta values using the derivative of the ReLU function:

𝑘
where 𝑥𝑖,𝑗 is the pre-activation input to the neuron during the forward pass. Using these deltas,
the gradient of the loss with respect to any weight in C3 is:

Hidden Layer (C1)

In the hidden layer C1, errors are propagated from layer C3 through pooling layer P2 using a
180° flipped convolution of the delta matrix:

DR. SHRUTHI U, AI&ML, RNSIT 7

 BAI701, DL & RL

where, ek is the error matrix for each of the k filters.

This produces an error map for each of the six filters in C1, from which delta values are
computed as:

Finally, the gradient of the loss with respect to weights in C1 is calculated by convolving the
input image with the delta values:

𝑘
where, 𝑤𝑚,𝑛 is the weights of this layer, k is the filter number, (m, n) are the indices of the
weights and (i, j) are the indices of the delta matrix

3.2 Gradient Descent-Based Optimization Techniques

Gradient descent is an optimization method that updates network parameters by computing

gradients to minimize the cost function. Different variants of gradient descent determine how
these updates are calculated based on the computed gradients.
Gradient Descent Variants
There are three commonly used Gradient Descent (GD) variants, which differ based on the
number of training examples used to compute the gradient. These include batch gradient
descent, stochastic gradient descent, and mini-batch gradient descent.
1). Batch Gradient Descent (GD)
Batch Gradient Descent computes the gradient of the error with respect to the weight parameter
using the entire training set. The weight update rule is:

Here, ∇E(w) is the error gradient with respect to weight, and μ is the learning rate. A large
dataset can result in slow computation and large memory requirements.

DR. SHRUTHI U, AI&ML, RNSIT 8

 BAI701, DL & RL

2). Stochastic Gradient Descent (SGD)

Stochastic Gradient Descent computes the gradient for one training example at a time, making
it faster than batch GD. The update rule is:

Where ∇E(w; x(i); y(i)) is the gradient computed from a single data point (x(i), y(i)). It causes
frequent fluctuations in the loss function but allows faster updates.

3). Mini-batch Gradient Descent

Mini-batch Gradient Descent is a hybrid of batch and stochastic methods. It divides the training
set into smaller batches and updates weights using each batch. The update rule is:

Where a batch of size 'n' is used. Typical batch sizes range from 50 to 256. Large batches offer
accurate gradients but high memory use, while small batches provide a regularizing effect but
need smaller learning rates.

Improving Gradient Descent for Faster Convergence

The main objective of optimization is to minimize the cost/loss or objective function. There are
many methods available that help an optimization algorithm to converge faster. Some of the
commonly used methods are:
1). AdaGrad
AdaGrad addresses the issue of fixed learning rate in SGD by adapting the learning rate for
each parameter based on the sum of all previous squared gradients. This helps in adjusting
learning rate according to the frequency of parameter updates. The update rule is:

where Gi = Σ(∇τ)^2 from τ = 1 to t. This ensures parameters with high gradients receive smaller
updates and those with smaller gradients receive larger updates. However, the continually
increasing denominator can cause the learning rate to decay too much, slowing or halting
learning.

DR. SHRUTHI U, AI&ML, RNSIT 9

 BAI701, DL & RL

2). AdaDelta
AdaDelta improves AdaGrad by limiting the accumulation of past gradients to a fixed window
size, thereby preventing the learning rate from shrinking excessively. It maintains an
exponentially decaying average of past squared gradients for stability and calculates parameter
updates as:

Since the denominator is just the Root Mean Square (RMS) of the parameters

3). RMSProp
RMSProp is another variant of AdaGrad which avoids decaying learning rates by using an
exponentially weighted moving average of squared gradients. Instead of relying on gradient
magnitudes, it adapts learning rate using gradient signs. The weight update is adjusted as
follows:

(a) Set same magnitude of updates for all weights. Set maximum and minimum allowable
weight updates to ∆max and ∆min, respectively.
(b) At each iteration, if signs of current gradient and previous gradient are same, then
increase learning rate by a factor of 1.2, i.e. η= η + 1.2.
ij
Therefore, the update becomes

(c) If signs of current gradient and previous gradient are different, then decrease the learning
rate by a factor of 0.5, i.e., η = η − 0.5.

4). Adam
Adam is a widely used optimization algorithm in deep learning that combines the advantages
of AdaGrad (good for sparse gradients) and RMSProp (good for non-stationary objectives). It
uses adaptive learning rates and momentum to achieve fast and reliable convergence.

 First Moment Estimate (Mean of Gradients)

DR. SHRUTHI U, AI&ML, RNSIT 10
 BAI701, DL & RL

mₜ stores the exponentially decaying average of the past gradients (momentum). β₁ is typically
set to 0.9 and controls how much past gradients influence the update. This helps smooth out
noisy gradients.

 Second Moment Estimate (Uncentered Variance of Gradients)

vₜ is the exponentially decaying average of the squared gradients. β₂ is typically set to 0.999
and controls the influence of past squared gradients. This tracks the magnitude of the gradients
for each parameter.

 Adam updates exponential moving averages of the gradient and the squared gradient
where the hyperparameters β1, β2 ∈ [0, 1] control the decay rates of these moving
averages.

 Final Weight Update Rule

μ is the learning rate (typically around 0.001), and ε is a small constant (e.g., 10⁻⁸) to avoid
division by zero. The update is stable and adaptive for each parameter due to the use of both
the corrected mean and variance.

Adam converges faster and avoids issues like vanishing learning rate, high variance in updates,
and slow training.

3.3 Challenges in Training Deep Networks.

Training deep neural networks comes with several challenges that hinder efficient learning and
convergence. Below are the prominent issues faced during training:
 Vanishing Gradient

DR. SHRUTHI U, AI&ML, RNSIT 11

 BAI701, DL & RL

The vanishing gradient problem arises in deep neural networks when activation functions like
sigmoid or tanh are used. During backpropagation, the gradient values become progressively
smaller as they move backward through layers, making it hard to update weights in the initial
layers. The sigmoid function is defined as:

Its derivative:

This leads to negligible updates in the earlier layers, resulting in poor learning. For example,
the derivative of the sigmoid function never exceeds 0.25 as shown in Figure 3.2, so the errors
get squeezed and diminish at each layer.

ReLU (Rectified Linear Unit) overcomes this issue as it outputs 0 for negative inputs and
returns the input value for positive inputs, with a derivative of 1 for all positive inputs. That
is:

f (x)= x if x=0; f(x)=0 if x<= 0

f r(x) = 1 for x> 0.
ReLU does not suffer from vanishing gradients as it does not compress positive values.

 Training Data Size

Deep learning models require a vast number of parameters to capture complex patterns. Hence,
the size and quality of the training dataset significantly impact the model's performance. Large
datasets like ImageNet (1.2 million labeled images) are essential for training complex models

DR. SHRUTHI U, AI&ML, RNSIT 12

 BAI701, DL & RL

such as AlexNet or ResNet. However, in less complex tasks (e.g., medical image
classification), smaller datasets may suffice if the data quality is high. Noisy data with low
Signal-to-Noise Ratio (SNR) increases the demand for more data to ensure convergence.
 Overfitting and Underfitting
A well-trained model must generalize well to new, unseen data. Two common problems are:
 Overfitting: The model performs well on training data but poorly on unseen data.
That is, the model has low training error but is unable to achieve low test error
indicating that it has memorized instead of learned.
 Underfitting: The model fails to learn from the training data itself, resulting in
high training and test error.
Overfitting illustrated by low training error (blue) and high validation error (red) over iterations
is as shown in figure 3.3.

Overfitting is a common problem in deep networks, however, there are few techniques
available that can be used in deep learning models to limit overfitting:
 Techniques to reduce overfitting include:
(a) Increase training dataset
(b) Reduce network complexity
(c) Data augmentation (scaling, translation, etc.)
(d) Add regularization (L1/L2 penalties)
(e) Apply Dropout: temporarily disables neurons during training to prevent reliance on
specific paths, thus improving generalization.
Dropout refers to dropping out neurons in a neural network during training, meaning
temporarily disconnecting them along with all their inward and outward connections. These

DR. SHRUTHI U, AI&ML, RNSIT 13

 BAI701, DL & RL

neurons do not participate in forward or backward propagation. This randomness helps the
network learn more robust features and avoid overfitting. Dropout visual representation as
shown in figure 3.4a: Original network and figure 3.4b: Network after applying dropout (some
neurons disconnected.

 High-Performance Hardware
Training deep networks on large datasets requires powerful hardware with high processing
speed and memory. GPUs are preferred for their parallel processing capabilities. However,
such setups are expensive and energy-intensive, making deployment of deep learning solutions
costly in real-world applications.

3.4 Supervised Deep Learning Architectures: LetNet-5, AlexNet

Many supervised deep learning architectures have been developed in recent years, achieving
exceptional performance on various tasks, often surpassing human-level accuracy. These
architectures rely on training deep convolutional neural networks (CNNs) using large sets of
labeled data. Among the foundational CNN models are LeNet-5 and AlexNet, which marked
significant milestones in the evolution of deep learning:

LeNet-5:

LeNet-5 is composed of seven layers (excluding the input) and takes an input image of size 32
× 32 pixels. Input pixels are normalized for faster convergence. The architecture includes
convolutional and subsampling layers for feature extraction and fully connected layers for
classification. The architecture diagram of LeNet-5 is given in Figure. 4.1, and details of
various layers are given in Table 4.1.
Layer 1: Convolutional layer producing 6 feature maps of size 28×28 with 156 trainable
parameters and 122,304 connections.
Layer 2: Subsampling layer with 12 trainable parameters and 5,880 connections.
DR. SHRUTHI U, AI&ML, RNSIT 14
 BAI701, DL & RL

Layer 3: Convolutional layer producing 16 feature maps of size 10×10.

Layer 4: Subsampling layer with 32 trainable parameters and 2,000 connections.
Layer 5: Convolutional layer producing 120 feature maps of size 1×1 with 48,120
connections.
Layer 6: Fully connected layer with 84 outputs and 10,164 trainable parameters.
Output Layer: Radial Basis Function (Euclidean) for 10 output classes, each connected with
84 inputs.

Table 4.1 Details of various layers of LeNet-5

#
Input Filter Window # Output
Layer name Stride Padding Feature
size size size Filters size
maps
Conv 1 32 × 32 5×5 – 6 1 0 28 × 28 6
Subsampling
28 × 28 – 2×2 – 2 0 14 × 14 6
-1
Conv 2 14 × 14 5×5 – 16 1 0 10 × 10 16
Subsampling
10 × 10 2×2 16 2 0 5×5 16
-2
Conv 3 5×5 5×5 – 120 1 0 1×1 120
Fully
120 – – – – – 1×1 84
connected
Softmax 84 – – – – – 1×1 10
The convolutional and subsampling layers extract features from the input image. The fully
connected layer performs classification. Figure 4.2 shows the handwritten digit “7” as input
image. The target probability is 1 for the digit “7” and 0 for all the remaining nine digits.

DR. SHRUTHI U, AI&ML, RNSIT 15

 BAI701, DL & RL

Therefore, the target vector is: [0, 0, 0, 0, 0, 0, 1, 0, 0, 0]. Backpropagation is used to compute
the gradients of the loss function for all weights in all layers of the CNN network.

Training of LeNet-5
Training LeNet-5 involves a series of systematic steps:
Step 1. Weight Initialization: All filter parameters and weights are initialized with
random values.
Step 2. Forward Propagation: The input image is passed through all layers—
convolutional, subsampling, and fully connected—to compute output probabilities for
each class.
Step 3. Error Calculation: The total error between predicted probabilities and target
labels is calculated at the output layer.
Step 4. Backpropagation and Weight Update: Error gradients with respect to weights
are computed. Gradient Descent is used to update weights and filter parameters
proportionally based on their contribution to the total error. Only weights and filter
matrices are updated; hyperparameters like filter size and number remain fixed.
Step 5. Iteration Over Training Set: Steps 2 to 4 are repeated for all images in the
training dataset.
Layer-wise transformations include:
 Input (32×32) is convolved with 5×5 filters to generate 6 feature maps (28×28)
 Pooling reduces these to 14×14
 16 filters of size 5×5 produce 16 feature maps (10×10)
 Pooling reduces to 5×5
 Convolution with 5×5 filter generates 120 feature maps (1×1)
 Passed to a fully connected layer with 84 neurons
 Final classification through 10-neuron output layer

DR. SHRUTHI U, AI&ML, RNSIT 16

 BAI701, DL & RL

AlexNet:

AlexNet addressed the vanishing gradient problem using the ReLU activation function. It was
the first architecture to implement ReLU in a deep CNN. AlexNet was trained and evaluated
on a subset of the ImageNet database, using 1.2 million images across 1000 classes for training
and 150,000 images for testing. Figure 4.3 shows the architecture diagram of AlexNet, and the
details of various layers of the AlexNet are given in Table 4.2. The network architecture
includes convolutional, max-pooling, and fully connected layers:
Layer 1: Convolutional layer producing 96 feature maps of size 55×55
Layer 2: Max-pooling layer with output size 27×27×96 (3×3 window, stride 2)
Layer 3: Convolutional layer with 256 filters of size 5×5, stride 1, padding 2
Layer 4: Max-pooling layer outputting 13×13×256 (3×3 window, stride 2)
Layer 5: Convolutional layer with 384 filters of size 3×3, stride 1, padding 1
Layer 6: Convolutional layer with 384 filters of size 3×3, stride 1, padding 1
Layer 7: Convolutional layer with 256 filters of size 3×3, stride 1, padding 1
Layer 8: Max-pooling layer (3×3 window, stride 2)
Layer 9: Fully connected layer with 4096 neurons
Layer 10: Fully connected layer with 4096 neurons
Layer 11: Fully conected layer with 1000 neurons
Final Layer: Softmax classification layer
Table 4.2 Details of various layers of AlexNet
Layer name Input size Filter Window # Stride Padding Output #
size size Filters size Feature
maps
Conv 1 224 × 224 11 × 11 – 96 4 1 55 × 55 96
Max- pooling 1 55 × 55 – 3×3 – 2 0 27 × 27 96
Conv 2 27 × 27 5×5 – 256 1 2 27 × 27 256
Max- pooling 2 27 × 27 – 3×3 – 2 0 13 × 13 256
Conv 3 13 × 13 3×3 – 384 1 1 13 × 13 384
Conv 4 13 × 13 3×3 – 384 1 1 13 × 13 384
Conv 5 13 × 13 3×3 – 256 1 1 13 × 13 256
Max- pooling 3 13 × 13 – 3×3 – 2 0 6×6 256
Fully 4096 neurons
connected 1
Fully 4096 neurons
connected 2
Fully 1000 neurons
connected 3
Softmax 1000 Classes
DR. SHRUTHI U, AI&ML, RNSIT 17
 BAI701, DL & RL

AlexNet accepts input images of size 224×224. It utilizes five convolutional layers, three fully
connected layers, and ends with a softmax layer for classification. Training was conducted on
a GTX 580 GPU with 3 GB of memory. AlexNet's success with purely supervised learning
demonstrated the capability of deep CNNs to excel on large, complex datasets without
requiring unsupervised pretraining.

DR. SHRUTHI U, AI&ML, RNSIT 18