BAI701- Deep Learning and Reinforcement Learning

Module – 3 Notes (Training Supervised Deep Learning Networks)

Training Convolution Neural Networks:

Convolutional Neural Network (CNN):
The given CNN architecture is inspired by LeNet, but uses modern components such as
ReLU activation and max-pooling. It contains 3 convolution layers, 2 pooling layers, and 1
fully connected layer, ending with a Softmax classifier.
1. Input Layer
• Input image size: 32 × 32 (grayscale).
• This image is passed through a series of convolution and pooling operations to
extract features.

2. Convolution Layer 1 (C1)

• 6 filters of size 5 × 5 are used.
• This performs convolution on the 32 × 32 image.
• Output feature maps: 6 maps of size 28 × 28.
• Formula: 32 − 5 + 1 = 28
• Activation used: ReLU, which introduces non-linearity.
• Purpose: extract low-level features like edges and corners.

3. Pooling Layer 1 (P1)

• Operation: Max-pooling.
• Reduces the size of each feature map from 28 × 28 → 14 × 14.
• Total number of feature maps remains 6.
• Purpose: reduces dimension, computation, and controls overfitting.

4. Convolution Layer 2 (C2)

• Takes 6 pooled feature maps as input.
• Uses 16 filters of size 5 × 5.
• Output: 16 feature maps of size 10 × 10.
• Formula: 14 − 5 + 1 = 10.
• Purpose: extracts more complex patterns from previous features.

5. Pooling Layer 2 (P2)

• Max-pooling again.
• Reduces feature map size: 10 × 10 → 5 × 5.
• Number of feature maps stays 16.
• Purpose: further size reduction and retaining dominant features.

1
6. Convolution Layer 3 (C3)
• Input size is 5 × 5; filter size also 5 × 5.
• Since the filter covers the entire map, output is a single value per filter.
• This layer has 120 filters, each connected to all 16 feature maps.
• Output: 120 neurons.
• Purpose: convert spatial features into high-level abstract features.

7. Fully Connected Layer (FC)

• Takes the 120 units from C3.
• Fully connected to 10 output units, one for each class.
• Purpose: final classification decision based on extracted features.

8. Softmax Output Layer

• Produces probabilities for 10 classes.
• Error (loss) is computed using the Softmax cost function.
• The error is then used during backpropagation to update weights.

Fig. - CNN architecture

2
3
4
Backward Pass in CNN:
During training, the output of the network is compared with the target using a loss function
(MSE or cross-entropy). The goal is to minimize the loss by adjusting the network’s weights
using gradient descent.

5
Backward pass computes how much each weight contributed to the error.
Error is passed from output → F6 → C5 → C3 → C1 using chain rule.
Each layer’s delta is multiplied with its input to get gradients, and weights are updated using
gradient descent.

Gradient Descent-based optimization techniques & their various variants:

Gradient descent:
Gradient descent is an optimization method used to minimize the cost (loss) function by
computing gradients and updating the parameters of a neural network. Different variants of
gradient descent define how gradients are computed and how weights are updated.
Variants of Gradient Descent Algorithm
There are 3 types of gradient descent algorithms.
[Link] Gradient Descent (GD)
[Link] Gradient Descent (SGD)
[Link]-batch Gradient Descent
[Link] Gradient Descent (GD)

[Link] Gradient Descent (SGD)

6
[Link]-batch Gradient Descent

Improving Gradient Descent for Faster Convergence

AdaDelta:

RMSProp:

7
Adam:

Challenges in Training Deep Networks:

1. Vanishing Gradient
• Occurs in deep networks using sigmoid/tanh activations.
• During backpropagation, gradients get multiplied many times and become smaller at
each layer.
• Sigmoid derivative ≤ 0.25 → gradients shrink and almost become zero.
• Early layers stop learning (remain untrained).
• ReLU solves this because its derivative is 1 for positive inputs, so gradients don’t
vanish.
2. Training Data Size
• Deep networks have millions of parameters, so they need large datasets to learn
effectively.
• Large data improves accuracy and generalization.
• Successful models like AlexNet, VGG, ResNet were trained on ImageNet (1.2M
images).
• Required data size depends on:
o Complexity of the task
o Quality of the data (noisy data needs more samples)
• No fixed rule for “how much data is enough.”
[Link] & Underfitting:
Overfitting
• Overfitting occurs when a CNN learns the training data too well, including noise and
irrelevant patterns.
• The model shows low training error but high test/validation error.
• CNN memorizes data instead of learning general features, leading to poor
performance on new images.
• Occurs when training data is small compared to the large number of CNN
parameters.

8
 Underfitting
• Underfitting happens when a CNN fails to learn important patterns from the training
data.
• Model shows high training error and high test error.
• Occurs due to insufficient training, too simple architecture, or poor feature
learning.

4. High-Performance Hardware
• Deep models require huge computation and large memory.
• Training is faster with GPUs, multi-core processors.
• Hardware is expensive and consumes high energy → makes deep learning costly.

overfitting & underfitting with suitable examples.

Gradient descent:
Gradient descent is an optimization method used to minimize the cost (loss) function by
computing gradients and updating the parameters of a neural network. Different variants of
gradient descent define how gradients are computed and how weights are updated.
Variants of Gradient Descent Algorithm
There are 3 types of gradient descent algorithms.
[Link] Gradient Descent (GD)
[Link] Gradient Descent (SGD)
[Link]-batch Gradient Descent
[Link] Gradient Descent (GD)

[Link] Gradient Descent (SGD)

9
[Link]-batch Gradient Descent

Improving Gradient Descent for Faster Convergence

AdaDelta:

RMSProp:

10
Adam:

How does a neural network with dropout differ from a simple neural
network:
The image compares a normal neural network and the same network with dropout
applied.

Fig - A simple neural network, b neural network after dropout

(a) Normal Neural Network (Without Dropout)
• All neurons (white circles) are active.
• Every connection between layers is present.
• During training, all neurons participate in both forward pass and backpropagation.
• This can sometimes lead to overfitting, because the network becomes dependent on
specific neurons and connections.
(b) Neural Network with Dropout
• Some neurons are turned off temporarily (gray circles).
• These “dropped” neurons:
o Do not activate in forward pass
o Do not receive gradient in backpropagation
o Do not contribute any output
• Connections associated with dropped neurons are also disabled.

11
Purpose
• Prevents the network from relying too heavily on specific neurons.
• Forces the network to learn more robust and distributed features.
• Reduces overfitting significantly.
In One Line
(a): Full network active → risk of overfitting
(b): Dropout applied → some neurons removed → better generalization

LeNet-5 and AlexNet architectures and its features:

LeNet:
LeNet-5 is one of the earliest and most influential Convolutional Neural Networks (CNNs),
developed by Yann LeCun for handwritten digit recognition (MNIST dataset).
It processes an input image through a series of convolution, subsampling (pooling), and
fully connected layers to classify digits from 0 to 9.

Architecture Overview
LeNet-5 consists of 7 layers (excluding the input layer).
Each layer extracts deeper and more abstract features from the image.

Fig - Architecture diagram of LeNet-5

1. Input Layer
• Image size: 32 × 32
• Pixel values are normalized to speed up training.
2. Convolutional Layer 1 (C1)
• Filters: 6 filters of size 5×5

12
 • Output: 6 feature maps of size 28 × 28
• Purpose: Extract low-level features (edges, corners, textures)
3. Subsampling Layer 1 (S2) – Pooling
• 2×2 average pooling
• Output: 6 feature maps of size 14 × 14
• Purpose: Reduce spatial size, improve invariance to small shifts

4. Convolutional Layer 2 (C3)

• Filters: 16 filters of size 5×5
• Output: 16 feature maps of size 10 × 10
• Purpose: Learn more complex features (shapes, patterns)
5. Subsampling Layer 2 (S4)
• 2×2 pooling
• Output: 16 feature maps of size 5 × 5
6. Convolutional Layer 3 (C5)
• Filters: 120 filters of size 5×5
• Output: 120 feature maps of size 1 × 1
• Purpose: Combine all local features into a global representation
7. Fully Connected Layer (F6)
• Number of neurons: 84
• Works like a typical neural network layer.
8. Output Layer
• 10 neurons (for digits 0–9)
• Uses softmax activation for classification.

Key Features of LeNet-5

1. Combination of Convolution + Pooling

• First architecture to use alternating convolutional and pooling layers.
• Enables automatic feature extraction from images.
2. Local Receptive Fields
• Each neuron connects only to a small region of the previous layer.
• Helps capture spatial patterns and reduces number of weights.
3. Weight Sharing
• Same filter weights are applied across the entire image.
• Greatly reduces computation and improves generalization.
4. Subsampling (Pooling) for Dimensionality Reduction
• Reduces feature-map size.
• Controls overfitting and increases translation invariance.
5. Hierarchical Feature Learning
• Early layers learn simple features (edges)
• Later layers learn complex patterns (digits)
6. Fully Connected Layers for Classification

13
 • After feature extraction, output is passed to dense layers for final prediction.
7. Uses Tanh Activation
• LeNet originally used tanh, unlike modern networks that use ReLU.
8. Efficient and Lightweight
• Small number of parameters
• Can run on low computational power
• Ideal for simple image recognition tasks

AlexNet:
• AlexNet is a deep Convolutional Neural Network developed by Alex Krizhevsky and
his team.
It became famous after winning the ImageNet Large Scale Visual Recognition
Challenge (ILSVRC) 2012, reducing the top-5 error by a huge margin.
• AlexNet proved that deep learning + GPUs can achieve breakthrough accuracy in
large-scale image classification.
• Architecture Overview
• AlexNet takes an RGB input image of size 224 × 224 × 3 and passes it through:
• 5 Convolutional layers
• 3 Max-Pooling layers
• 3 Fully Connected layers
• Softmax classifier (1000 classes)
The network contains about 60 million parameters.

Fig- Architecture diagram of AlexNet

14
1. Input Layer
• Image size: 224 × 224 × 3
• Preprocessing: mean subtraction, normalization
2. Convolution Layer 1
• 96 filters of 11×11, stride 4
• Output: 55 × 55 × 96
• ReLU activation (faster than tanh/sigmoid)
• Followed by Local Response Normalization (LRN)
• Max-pooling (3×3, stride 2)
3. Convolution Layer 2
• 256 filters of size 5×5
• Padding 2
• ReLU + LRN + Max-pooling
• Output: 13 × 13 × 256
4. Convolution Layer 3
• 384 filters of 3×3
• ReLU activation
• No LRN or pooling
5. Convolution Layer 4
• 384 filters of 3×3
• ReLU activation
6. Convolution Layer 5
• 256 filters of 3×3
• ReLU + Max-pooling
• Output: 6 × 6 × 256
7. Fully Connected Layer 1
• 4096 neurons
• Uses ReLU + Dropout (0.5) to prevent overfitting
8. Fully Connected Layer 2
• 4096 neurons
• ReLU + Dropout
9. Fully Connected Layer 3 (Output Layer)
• 1000 neurons
• Corresponds to 1000 ImageNet classes
• Uses Softmax activation

Key Features of AlexNet

1. Use of ReLU Activation

• Faster training compared to tanh or sigmoid
• Helps avoid vanishing gradients
• One major reason for AlexNet’s success
2. Use of GPUs
• AlexNet was trained using two NVIDIA GPUs

15
 • Allowed training a very deep network on a massive dataset (1.2M images)
3. Data Augmentation
• Image transformations (crop, flip, color jitter)
• Helps avoid overfitting and improves accuracy
4. Dropout in Fully Connected Layers
• Randomly drops neurons
• Prevents co-adaptation of features
• Reduces overfitting significantly
5. Overlapping Max-Pooling
• Pooling windows overlap
• Reduces overfitting and captures more information

6. Local Response Normalization (LRN)

• Imitates lateral inhibition in biological neurons
• Helps improve generalization
• Used after early convolution layers
7. Deep and Wide Architecture
• 5 convolutional layers capture detailed visual patterns
• 3 fully connected layers perform high-level reasoning
• Higher capacity than older models (like LeNet)
8. Trained on Large-Scale Dataset (ImageNet)
• 1.2 million training images
• Demonstrated that deep CNNs work extremely well on big data

AlexNet is a deep CNN architecture with 5 convolution layers, 3 pooling layers, and 3 fully
connected layers.

It introduced innovations like ReLU, dropout, LRN, overlapping pooling, and GPU-based
training, making it the first widely successful deep learning model for large-scale image
classification.

16