Deep Learning (DL) architectures are specialized, multi-layered artificial neural networks

designed to automatically learn hierarchical representations and patterns from vast amounts
of data. They are structured to process complex, unstructured data like images, audio, and
text, mimicking the way human brains analyse information.
Convolutional Neural Networks (CNNs), also known as ConvNets, are neural network
architectures inspired by the human visual system and are widely used in computer vision
tasks. They are designed to process structured grid-like data, especially images by capturing
spatial relationships between pixels.
 Automatically learn hierarchical features through convolution operations, from
simple edges and textures to complex shapes and objects.
 Detect objects at different positions within an image, ensuring robustness to
spatial variations.
 Reduce computational complexity by processing local regions instead of the
entire image at once

Key Components

 Convolutional Layers: These layers apply convolutional

operations to input images using filters or kernels to
detect features such as edges, textures and more
complex patterns. Convolutional operations help
preserve the spatial relationships between pixels.
 Pooling Layers: They downsample the spatial
dimensions of the input, reducing the computational
complexity and the number of parameters in the
network. Max pooling is a common pooling operation
where we select a maximum value from a group of
neighbouring pixels.
 Activation Functions : They introduce non-linearity to
the model by allowing it to learn more complex
relationships in the data.
 Fully Connected Layers : These layers are responsible
for making predictions based on the high-level features
learned by the previous layers. They connect every
neuron in one layer to every neuron in the next layer.

Working of CNN
1. Input Image: CNN receives an input image which is pre-
processed to ensure uniformity in size and format.
2. Convolutional Layers: Filters are applied to the input
image to extract features like edges, textures and
shapes.
 3. Pooling Layers: The feature maps generated by the
convolutional layers are downsampled to reduce
dimensionality.
4. Fully Connected Layers: The downsampled feature
maps are passed through fully connected layers to
produce the final output, such as a classification label.
5. Output: The CNN outputs a prediction, such as the class
of the image

Training a Convolutional Neural Network

CNNs are mainly trained using a supervised learning approach.
This means that the CNN is given a set of labelled training
images. The CNN learns to map the input images to their correct
labels.
The training process for a CNN involves the following steps:
1. Data Preparation: The training images are pre-
processed to ensure that they are all in the same format
and size.
2. Loss Function: A loss function is used to measure how
well the CNN is performing on the training data. The loss
function is typically calculated by taking the difference
between the predicted labels and the actual labels of the
training images.
3. Optimizer: An optimizer is used to update the weights
of the CNN in order to minimize the loss function.
 4. Backpropagation: Backpropagation is a technique used
to calculate the gradients of the loss function with
respect to the weights of the CNN. The gradients are
then used to update the weights of the CNN using the
optimizer.
How to Evaluate CNN Models
Efficiency of CNN can be evaluated using a variety of criteria.
Among the most popular metrics are:
 Accuracy: Accuracy is the percentage of test images
that the CNN correctly classifies.
 Precision: Precision is the percentage of test images
that the CNN predicts as a particular class and that are
actually of that class.
 Recall: Recall is the percentage of test images that are
of a particular class and that the CNN predicts as that
class.
 F1 Score: The F1 Score is a harmonic mean of precision
and recall. It is a good metric for evaluating the
performance of a CNN on classes that are imbalanced.
LeNet-5
In the late 1990s, Yann LeCun, Léon Bottou, Yoshua Bengio, and
Patrick Haffner created a convolutional neural network (CNN) based
architecture called LeNet. The LeNet-5 architecture was developed
to recognize handwritten and machine-printed characters, a function
that showcased the potential of deep learning in practical
applications.

Introduction to LeNet-5
LeNet-5 is a convolutional neural network (CNN) architecture that
introduced several key features and innovations that have become
standard in modern deep learning. It demonstrated the effectiveness
of CNNs for image recognition tasks and introduced key concepts
such as convolution, pooling, and hierarchical feature extraction that
underpin modern deep learning models.
Originally designed for handwritten digit recognition, the principles
behind LeNet-5 have been extended to various applications,
including:
 Handwriting recognition in postal services and banking.
 Object and face recognition in images and videos.
 Autonomous driving systems for recognizing and interpreting
road signs.
The architecture of LeNet 5 contains 7 layers excluding the input
layer. Here is a detailed breakdown of the LeNet-5 architecture:

1. Input Layer

 Input Size: 32x32 pixels.

The input is larger than the largest character in the database, which
is at most 20x20 pixels, centered in a 28x28 field. The larger input
size ensures that distinctive features such as stroke endpoints or
corners can appear in the center of the receptive field of the highest-
level feature detectors.
 Normalization: Input pixel values are normalized such that the
background (white) corresponds to a value of 0, and the foreground
(black) corresponds to a value of 1. This normalization makes the
mean input roughly 0 and the variance roughly 1, which accelerates
the learning process.
2. Layer C1 (Convolutional Layer)
 Feature Maps: 6 feature maps.
 Connections: Each unit is connected to a 5x5 neighbourhood in
the input, producing 28x28 feature maps to prevent boundary effects.
 Parameters: 156 trainable parameters and 117,600 connections.

3. Layer S2 (Subsampling Layer)

 Feature Maps: 6 feature maps.
 Size: 14x14 (each unit connected to a 2x2 neighborhood in C1).
 Operation: Each unit adds four inputs, multiplies by a trainable
coefficient, adds a bias, and applies a sigmoid function.
 Parameters: 12 trainable parameters and 5,880 connections.

5. Layer S4 (Subsampling Layer) Layer C3 (Convolutional

Layer)
 Feature Maps: 16
 Connections: Partially connected to S2: 6 maps -> 3 S2 maps, 9
maps -> 4 S2 maps, 1 map -> all 6 S2 maps.
 Kernel Size: 5 x 5
 Parameters: 1,516 trainable parameters
 Connections: 151,600

Partial Connectivity: Reduces parameters and encourages each

map to learn complementary features
 Feature Maps: 16 feature maps.
 Size: 7x7 (each unit connected to a 2x2 neighborhood in C3).
 Parameters: 32 trainable parameters and 2,744 connections.

6. Layer C5 (Convolutional Layer)

 Feature Maps: 120 feature maps.
 Size: 1x1 (each unit connected to a 5x5 neighborhood on all 16 of
S4’s feature maps, effectively fully connected due to input size).
 Parameters: 48,000 trainable parameters and 48,000 connections.

6. Layer F6 (Fully Connected Layer)

 Units: 84 units.
 Connections: Each unit is fully connected to C5, resulting in
10,164 trainable parameters.
 Activation: Uses a scaled hyperbolic tangent
function f (a)= A tan ⁡(Sa), where A = 1.7159 and S = 2/3
8. Output Layer
In the output layer of LeNet, each class is represented by an
Euclidean Radial Basis Function (RBF) unit. The output of each
RBF unit y iis computed:

Details of Layers
 Convolutional Layers (Cx): These layers apply convolution
operations to the input, using multiple filters to extract different
features. The filters slide over the input image, computing the dot
product between the filter weights and the input pixels. This process
captures spatial hierarchies of features, such as edges and textures.
 Subsampling Layers (Sx): These layers perform pooling
operations (average pooling in the case of LeNet-5) to reduce the
spatial dimensions of the feature maps. This helps to control
overfitting, reduce the computational load, and make the
representation more compact.
  Fully Connected Layers (Fx): These layers are densely
connected, meaning each neuron in these layers is connected to
every neuron in the previous layer. This allows the network to
combine features learned in previous layers to make final
predictions.

2. AlexNet (2012)
AlexNet is a deep learning model that made a big impact in
image recognition. It became famous for its ability to classify
images accurately. It won the ImageNet Large Scale Visual
Recognition Challenge (ILSVRC) 2012 with a top-5 error rate
of 15.3% (beating the runner up which had a top-5 error rate
of 26.2%).
Most important features of the AlexNet are:
 Overfitting Prevention: Dropout (0.5) was applied to
the first two fully connected layers and data
augmentation dynamically expanded the dataset hence
both helping in reducing overfitting.
 Faster Training: ReLU activation was used instead of
tanh or sigmoid, leading to a 6× speedup in training by
avoiding activation saturation.

 AlexNet Architecture
 Its architecture includes:
 5 convolutional layers with Max-Pooling applied after
the 1st, 2nd and 5th layers to enhance feature
extraction.
 Overlapping Max-Pooling uses a 3×3 filter with stride
2 which improved performance by reducing top-1 error
by 0.4% and top-5 error by 0.3% compared to non-
overlapping pooling.
 Followed by 2 fully connected layers each using
dropout to prevent overfitting.
 Ends with a softmax layer for final classification.
Advantages of AlexNet
 Use of ReLU Activation: First major architecture to use
ReLU (Rectified Linear Unit) which enabled faster training
compared to traditional tanh/sigmoid functions.
 Dropout for Regularization: Introduced dropout layers
to reduce overfitting by randomly disabling neurons
during training.
 GPU Utilization: Split the network across two GPUs,
showing how deep learning can benefit from parallel
computing for faster training.
 Overlapping Max-Pooling: Used overlapping pooling
layers to improve generalization and reduce top-1 and
top-5 classification errors.
Disadvantages of AlexNet
 Large Model Size: Has around 60 million parameters
making it memory-intensive and slow for inference on
low-resource devices.
 High Computational Cost: Training is computationally
expensive even though it was optimized for GPUs.
 Not Optimal for Small Datasets: Tends to overfit on
smaller datasets like CIFAR-10 or MNIST without heavy
regularization.
Applications
1. Image Classification: Originally built for classifying
high-resolution images into 1000 object categories
(ImageNet dataset).
2. Feature Extraction: Intermediate layers are often used
as pretrained feature extractors for transfer learning
tasks.
3. Object Detection: Forms the backbone in early
detection systems like R-CNN when combined with region
proposal methods.
4. Medical Imaging: Applied to classify abnormalities in X-
rays, MRIs or retinal scans by fine-tuning on domain-
specific datasets.
 5. Facial Recognition and Emotion Detection: Can be
adapted for face verification, expression analysis or
identity recognition tasks.
6. Autonomous Vehicles: Used in early perception
modules for identifying road signs, pedestrians or
obstacles.
VGG-19 Architecture
VGG (Visual Geometry Group) is a widely used, deep convolutional neural
network (CNN) architecture designed by Oxford University in 2014, famous for
its simplicity and uniform structure. It uses stacked convolutional layers,
allowing for increased depth (16–19 layers) which significantly improved
performance on ImageNet.
VGG-19 is a deep convolutional neural network with 19 weight
layers, comprising 16 convolutional layers and 3 fully connected
layers. The architecture follows a straightforward and repetitive
pattern, making it easier to understand and implement.
The key components of the VGG-19 architecture are:
1. Convolutional Layers: 3x3 filters with a stride of 1 and
padding
of 1 to preserve spatial resolution.
2. Activation Function: ReLU (Rectified Linear Unit)
applied after each convolutional layer to introduce non-
linearity.
3. Pooling Layers: Max pooling with a 2x2 filter and a
stride of 2 to reduce the spatial dimensions.
4. Fully Connected Layers: Three fully connected layers
at the end of the network for classification.
5. Softmax Layer: Final layer for outputting class
probabilities.

Detailed Layer-by-Layer Architecture of

VGG-Net 19
The VGG-19 model consists of five blocks of convolutional layers,
followed by three fully connected layers.
Block 1
 Conv1_1: 64 filters, 3x3 kernel, ReLU activation
 Conv1_2: 64 filters, 3x3 kernel, ReLU activation
 Max Pooling: 2x2 filter, stride 2
Block 2
 Conv2_1: 128 filters, 3x3 kernel, ReLU activation
 Conv2_2: 128 filters, 3x3 kernel, ReLU activation
 Max Pooling: 2x2 filter, stride 2
Block 3
 Conv3_1: 256 filters, 3x3 kernel, ReLU activation
 Conv3_2: 256 filters, 3x3 kernel, ReLU activation
 Conv3_3: 256 filters, 3x3 kernel, ReLU activation
 Conv3_4: 256 filters, 3x3 kernel, ReLU activation
 Max Pooling: 2x2 filter, stride 2
Block 4
 Conv4_1: 512 filters, 3x3 kernel, ReLU activation
 Conv4_2: 512 filters, 3x3 kernel, ReLU activation
 Conv4_3: 512 filters, 3x3 kernel, ReLU activation
 Conv4_4: 512 filters, 3x3 kernel, ReLU activation
 Max Pooling: 2x2 filter, stride 2
Block 5
 Conv5_1: 512 filters, 3x3 kernel, ReLU activation
 Conv5_2: 512 filters, 3x3 kernel, ReLU activation
 Conv5_3: 512 filters, 3x3 kernel, ReLU activation
  Conv5_4: 512 filters, 3x3 kernel, ReLU activation
 Max Pooling: 2x2 filter, stride 2

Fully Connected Layers

 FC1: 4096 neurons, ReLU activation

 FC2: 4096 neurons, ReLU activation
 FC3: 1000 neurons, softmax activation (for 1000-class
classification)
Architectural Design Principles
The VGG-19 architecture follows several key design principles:
1. Uniform Convolution Filters: Consistently using 3x3
convolution filters simplifies the architecture and helps
maintain uniformity.
2. Deep Architecture: Increasing the depth of the network
enables learning more complex features.
3. ReLU Activation: Introducing non-linearity helps in
learning complex patterns.
4. Max Pooling: Reduces the spatial dimensions while
preserving important features.
5. Fully Connected Layers: Combines the learned
features for classification.
Impact of VGG-19

Influence on Subsequent Models

The simplicity and effectiveness of VGG-19 influenced the design

of subsequent deep learning models.
Use in Transfer Learning
VGG-19 has been extensively used in transfer learning due to its
robust feature extraction capabilities. Pre-trained VGG-19 models
on large datasets like ImageNet are often fine-tuned for various
computer vision tasks, including object detection, image
segmentation, and style transfer.

Research and Industry Applications

VGG-19 has found applications in numerous research and

industry projects.
Charcteristic VGGNet-19
1. Model Simplicity and Effectiveness: The VGG-19
architecture's simplicity, characterized by its uniform use
of 3x3 convolution filters and repetitive block structure,
makes it a highly effective and easy-to-implement model
for various computer vision tasks.
2. Computational Requirements: One of the key trade-
offs of the VGG-19 model is its computational demand.
Due to its depth and the use of small filters, it requires
significant memory and computational power, making it
more suited for environments with robust hardware
capabilities.
3. Robust Feature Extraction: The depth of the VGG-19
model allows it to capture intricate features in images,
making it an excellent feature extractor.
4. Data Augmentation: To enhance the performance and
generalization capability of VGG-19, data augmentation
techniques such as random cropping, horizontal flipping,
and color jittering are often employed during training.
5. Influence on Network Design: The principles
established by the VGG-19 architecture, such as the use
of small convolution filters and deep networks, have
influenced the design of subsequent state-of-the-art
models. Researchers have built upon these concepts to
develop more advanced architectures that continue to
push the boundaries of what is possible in computer
vision.
Feature LeNet AlexNet VGG
Year 1998 2012 2014
Layers 7 8 16 / 19
Activation Tanh ReLU ReLU
Pooling Avg Pool Max Pool Max Pool
Image Type Grayscale RGB RGB
Depth Shallow Medium Deep

GoogLeNet Model
GoogLeNet (Inception V1) is a deep convolutional neural network
architecture designed for efficient image classification. It
introduces the Inception module, which performs multiple
convolution operations (1x1, 3x3, 5x5) in parallel, along with
max pooling and concatenates their outputs. The architecture is
deep, yet optimized for speed and performance, which makes it
suitable for large-scale visual recognition tasks. It brought
forward innovative architectural choices such as 1×1
convolutions, global average pooling and the Inception module,
all aimed at improving depth and computational efficiency.
Key Features of GoogLeNet
It uses many different kinds of methods such as:
1. 1×1 Convolutions
One of the core techniques employed in GoogLeNet is the use of
1×1 convolutions, primarily for dimensionality reduction. These
layers help decrease the number of trainable parameters while
enabling deeper and more efficient architectures.
2. Global Average Pooling
In traditional architectures like AlexNet, fully connected layers at
the end introduce a large number of parameters. GoogLeNet
replaces these with Global Average Pooling, which computes the
average of each feature map (e.g. converting 7×7 maps to 1×1),
this significantly reduces the model’s parameter count and
solves overfitting.
Benefits:
 Zero additional trainable parameters
 Reduces overfitting
 Improves top-1 accuracy by approximately 0.6%

3. Inception Module

The Inception module is the architectural core of GoogLeNet. It

processes the input using multiple types of
operations in parallel, including 1×1, 3×3, 5×5 convolutions and
3×3 max pooling. The outputs from all paths are concatenated
depth-wise.
 Purpose: Enables the network to capture features
at multiple scales effectively.
 Advantage: Improves representational power without
dramatically increasing computation.

[Link] Classifiers

To address the vanishing gradient problem during training,

GoogLeNet introduces auxiliary classifiers(intermediate
branches that act as smaller classifiers). These are active only
during training and help regularize the network.
Structure of Each Auxiliary Classifier:
 Average pooling layer (5×5, stride 3)
 1×1 convolution (128 filters, ReLU)
 Fully connected layer (1024 units, ReLU)
 Dropout layer (dropout rate = 0.7)
 Fully connected softmax layer (1000 classes)
 The auxiliary losses are added to the main loss with a
weight of 0.3 to stabilize training.
5. Model Architecture
GoogLeNet is a 22-layer deep network (excluding pooling
layers) that emphasizes computational efficiency, making it
feasible to run even on hardware with limited [Link]
architecture also contains two auxiliary classifier layer connected
to the output of Inception (4a) and Inception (4d) layers.
Inception V1 architecture
Key highlights of the architecture:
 Input Layer: Accepts a 224×224 RGB image as input.
 Initial Convolutions and Pooling: Applies a series of
standard convolutional and max pooling layers to
downsample the input and extract low-level features.
 Local Response Normalization (LRN): Normalizes the
feature maps early in the network to improve
generalization.
 Inception Modules: Each module processes the input
through 1×1, 3×3, and 5×5 convolutions, as well as 3×3
max pooling, all in parallel. The outputs are
concatenated along the depth dimension, allowing the
network to capture both fine and coarse features.
 Auxiliary Classifiers: Appear as smaller branches
connected to intermediate layers of the network. Include
average pooling, 1×1 convolutions, fully connected
layers, and softmax outputs.
 Final Layers: Uses global average pooling (7×7) to
reduce each feature map to a single value. Followed by a
fully connected layer and a softmax activation to
produce the final classification output.
Performance and Results
 Winner of ILSVRC 2014 in both classification and
detection tasks
 Achieved a top-5 error rate of 6.67% in image
classification
  An ensemble of six GoogLeNet models achieved 43.9%
mAP (mean Average Precision) on the ImageNet
detection task.

Residual Networks
To overcome the challenges of training very deep neural
networks, Residual Networks (ResNet) was introduced, which
uses skip connections that allow the model to learn residual
mappings instead of direct transformations making deep neural
networks easier to train.
 It helps prevent vanishing gradient problems in very
deep models.
 Skip connections let information flow directly across
layers.
 ResNet enables building networks with hundreds or even
thousands of layers.
 It is widely used in computer vision tasks like image
classification and object detection.
A residual block lets the network skip layers by adding the
original input to the processed output, making deep networks
easier to train.
Challenges in Deep Neural Networks
Deep Neural Networks are useful models but they also come with
several training challenges, especially as the network depth
increases.
Two major issues are
1. Vanishing/Exploding Gradient Problem: As the number of
layers in a neural network increases, the gradients of the loss
function with respect to the weights can become extremely small
or excessively large during backpropagation.
2. Degradation Problem: The degradation problem occurs
when increasing the network depth does not improve
performance and may even worsen it. This problem has two
aspects:
 Performance Plateau: Training error saturates after a
certain depth meaning additional layers do not
significantly reduce the error.
 Accuracy Degradation: Beyond a certain depth
validation error increases and the model performs poorly
on unseen data.
 What Is Residual Learning?
 Normally, a few layers in a neural network try to learn
a direct mapping:
 H(x) = output we want
x = input to those layers
 So, the layers learn:
 H(x)
 ResNet changes this idea.
Instead of learning H(x) directly, it asks the layers to
learn the difference between the input and output:
 F(x) = H(x) — x
 Then it adds the input back at the end:
 H(x) = F(x) + x
 This addition is called a skip connection or shortcut
connection because the input “skips” the layers and
gets added to the output.

ResNet
ResNet is a deep learning architecture designed to train very
deep networks efficiently using residual connections. The key
features of ResNet:
 Residual Connections: Enable very deep networks by
allowing gradients to flow through identity shortcuts,
reducing the vanishing gradient problem.
 Identity Mapping: Simplifies training by learning
residual functions instead of full mappings.
 Depth: Supports extremely deep architectures for
improved image recognition performance.
 Fewer Parameters: Achieves high accuracy with fewer
parameters hence improving computational efficiency.
 Results: Delivers top performance on benchmark image
recognition tasks.
  Effective Approach: Residual connections provide a
reliable way to train deeper networks effectively,
enabling networks to learn more complex features.
Here, the graph compares the training and test error of a 20-
layered and 56-layered network across iterations showing how
deeper networks struggle without proper residual connections.

 Left graph (training error): The 56 layer network

reduces error slowly and shows strong fluctuations due
to vanishing gradients, whereas the 20-layer network
learns smoothly and reaches a much lower training error.
 Right graph (test error): The 56-layer network
maintains a higher test error (degradation problem),
while the 20-layer network generalizes better showing
why ResNet skip connections are essential for training
deep models.

ResNet-34
 ResNet-34 is a deep residual network built on a 34-layer
plain network inspired by VGG-19, with shortcut
connections forming 16 residual blocks.
Here are the different stages of the ResNet-34 architecture,
showing its structured arrangement of residual blocks.
 First set: 3 residual blocks each with 2 convolution
layers of 64 filters and identity skip connections.
 Second set: 4 residual blocks each with 2 convolution
layers of 128 filters uses zero-padding or 1x1 projections
for dimension changes.
 Third set: 6 residual blocks, each with 2 convolution
layers of 256 filters.
 Fourth set: 3 residual blocks with 2 convolution layers
of 512 filters each.
 Feature map: Passed through Global Average Pooling a
dense layer with 1000 neurons and softmax for
classification.

1. Residual Block: A Residual Block contains:

 One or more convolutional layers
 A skip connection that bypasses these layers
 Addition of input to convolution output
 This ensures uninterrupted flow of information and
gradients.
2. Skip (Shortcut) Connection
 Bypasses one or more layers
 Adds input directly to output
 Prevents vanishing gradients
 Improves parameter updates
3. Handling Dimension Mismatch: When input and output
dimensions differ
 Zero Padding: Adds extra zeros to the input to match
output dimensions in a residual block
 Linear Projection: Uses a learnable 1x1 convolution to
match input and output dimensions for the skip
connection.
3. Stacking Residual Blocks : Multiple residual blocks can be
stacked to create deep architectures. This allows networks to go
very deep without suffering from degradation.
4. Global Average Pooling (GAP): Before the final fully
connected layer ResNet uses GAP:
 Converts each feature map to a single value by
averaging
 Reduces parameters less overfitting
 Produces compact feature representation

Advantages
 Eases Training of Deep Networks: Skip connections
allow gradients to flow directly through the network,
reducing vanishing gradient problems.
 Enables Very Deep Architectures: ResNet can train
networks with 50, 100 or even 152+ layers effectively.
  Improves Accuracy: Residual learning helps the
network achieve higher performance on tasks like image
classification and object detection.
 Reduces Degradation: Adding more layers does not
increase training error unlike plain deep networks.
 Fewer Parameters for Better Efficiency: Deep
ResNets can have fewer parameters than traditional
deep networks but performing better.
Challenges
 High Computation: Requires high computational power
due to deep architecture.
 Dimension Mismatch: Dimension mismatch in skip
connections needs extra projection layers.
 Overfitting Risk: Risk of overfitting on small datasets
because of large model capacity.
 Training Instability: Training can become unstable
without proper batch normalization.
 Redundant Updates: Residual blocks may learn only
small or redundant updates.
 Deep Network Degradation: Gradient flow improves
but may still degrade in extremely deep networks.