Q1.

Describe the architecture of AlexNet (AlexNet-5) with detailed layer-wise explanation

and illustrate it using appropriate diagrams. Highlight its key innovations over traditional
neural networks.

Introduction

AlexNet, proposed by Alex Krizhevsky et al. in 2012, is a deep convolutional neural network
(CNN) that won the ImageNet Large Scale Visual Recognition Challenge (ILSVRC) with a top-
5 error rate of only 15.3%, outperforming all traditional models by a wide margin.
It was one of the first large-scale CNNs trained using GPUs and massive labeled datasets
(ImageNet).

Architecture Overview

AlexNet consists of 8 layers —

• 5 Convolutional layers for feature extraction

• 3 Fully Connected layers for classification

It accepts an image of size 227 × 227 × 3 (RGB) and outputs probabilities for 1000
classes using the Softmax layer.
Layer-wise Explanation

Filter/Kernel Output Stride /

Layer Type Remarks
Size Maps Pooling
Input - 227 × 227 × 3 - - Input RGB image
ReLU activation,
followed by LRN
96 Stride =
Conv1 Convolution 11 × 11 (Local Response
filters 4
Normalization) and
Max Pooling
Max Stride = Reduces dimension to
Pooling 3×3 -
Pool1 2 27 × 27 × 96
256 Stride = ReLU + LRN + Max
Conv2 Convolution 5×5
filters 1 Pooling
Max Stride =
Pooling 3×3 - Output: 13 × 13 × 256
Pool2 2
384 Stride =
Conv3 Convolution 3×3 ReLU activation
filters 1
384 Stride =
Conv4 Convolution 3×3 ReLU activation
filters 1
256 Stride =
Conv5 Convolution 3×3 ReLU + Max Pooling
filters 1
Max Stride =
Pooling 3×3 - Output: 6 × 6 × 256
Pool3 2
Fully 4096 ReLU + Dropout
FC1 - -
Connected neurons (50%)
Fully 4096
FC2 - - ReLU + Dropout
Connected neurons
Fully 1000
FC3 - - Softmax classifier
Connected neurons

Diagram: AlexNet Architecture

Key Innovations of AlexNet

1. Use of ReLU Activation Function:

ReLU (f(x) = max(0, x)) replaced sigmoid/tanh, resulting in faster training and less
vanishing gradient problem.

2. GPU-Based Training:
Used two NVIDIA GTX 580 GPUs to parallelize computation — one handled half of
the network layers.

3. Dropout Regularization:
Randomly dropped neurons during training to reduce overfitting in fully connected
layers.

4. Data Augmentation:
Applied image translations, flips, and color jittering to artificially expand training
data.

5. Local Response Normalization (LRN):

Encouraged competition among neuron outputs and improved generalization (later
replaced by BatchNorm in modern networks).
 6. Large Filter Sizes in Early Layers:
The first convolutional layer used 11×11 filters with stride 4 to capture low-level
visual features effectively.

7. Massive Dataset (ImageNet):

Trained on 1.2 million labeled images, enabling deep networks to generalize to
diverse object categories.

Advantages over Traditional Neural Networks

Traditional NN AlexNet Improvement

Shallow architecture Deep 8-layer CNN

Slow training GPU acceleration

Sigmoid/Tanh activation ReLU for faster convergence

Prone to overfitting Dropout + Data Augmentation

Poor image understanding Convolution + Pooling for spatial feature learning

Conclusion

AlexNet revolutionized deep learning by combining deep CNN architecture, GPU

computation, and effective regularization.
It laid the foundation for later models like VGGNet, GoogLeNet, and ResNet, marking the
start of the deep learning era in computer vision.

Q2. Using a labeled diagram, explain overfitting and underfitting in machine learning.
Show how they affect training and validation learning curves.

Introduction

In machine learning, the goal of a model is to learn patterns from training data and
generalize well to unseen data.
However, if the model is too simple or too complex, it may not perform effectively.
These two extremes are known as underfitting and overfitting.

1. Underfitting
 • Occurs when the model is too simple to capture the underlying structure of the
data.

• It performs poorly on both training and testing datasets.

• Example: Using a linear model to fit non-linear data patterns.

Characteristics:

• High training error

• High validation error

• Poor generalization capability

• Model fails to capture relationships in data

Causes:

• Model is not complex enough (few parameters)

• Insufficient training (too few epochs)

• Excessive regularization (too much constraint on weights)

Solutions:

• Increase model complexity (e.g., add more layers/features)

• Train for more epochs

• Reduce regularization strength

2. Overfitting

• Occurs when the model is too complex and learns not only the patterns but also
the noise in training data.

• It performs very well on training data but poorly on unseen test data.

Characteristics:

• Very low training error

• High validation/test error

• Model memorizes training examples instead of generalizing

Causes:

• Too many model parameters (very deep or wide network)

• Small dataset size

 • Lack of regularization or dropout

• Training for too many epochs

Solutions:

• Use regularization techniques (L1/L2, Dropout)

• Apply data augmentation

• Collect more training data

• Use early stopping

3. Optimal Fit

• The model generalizes well when it achieves a balance between bias and variance.

• Training and validation errors are both low and close to each other.

• This point represents the best generalization performance.

4. Labeled Diagram

Use the following labeled diagram in your answer:

Explanation of the diagram:

• X-axis: Model complexity or number of epochs

• Y-axis: Error or loss

• The blue curve shows training error

• The orange curve shows validation error

• Underfitting region: Both errors high and close together

• Overfitting region: Training error continues to decrease while validation error

increases

• Optimal point: The lowest validation error indicates the best trade-off between
bias and variance

5. Comparison Table

Aspect Underfitting Overfitting

Model Complexity Too Low Too High

Training Error High Low

Validation Error High High

Generalization Poor Poor

Cause Simple model, insufficient training Complex model, too much training

Solution Add complexity Regularize or simplify

Conclusion

Overfitting and underfitting are two key challenges in building robust machine learning
models.
An optimal model should balance bias (underfitting) and variance (overfitting) to
generalize well on unseen data.
Achieving this balance requires careful tuning of model complexity, regularization, and
sufficient training data.
Q3. Discuss the following concepts in deep learning with detailed explanations and
relevant examples:

(a) Vanishing Gradient Problem

(b) Significance of Training Dataset Size in Neural Network Performance

(a) Vanishing Gradient Problem

Definition

The Vanishing Gradient Problem occurs when gradients (used to update network weights
during backpropagation) become very small as they are propagated backward through the
layers of a deep neural network.
As a result, the early (input) layers of the network learn extremely slowly or stop learning
altogether.

Detailed Explanation

• During backpropagation, gradients are calculated using the chain rule:

𝛛𝑳 𝛛𝑳 𝛛𝒂𝒏 𝛛𝒂𝟏
= ⋅ ⋯
𝛛𝑾𝒊 𝛛𝒂𝒏 𝛛𝒂𝒏−𝟏 𝛛𝑾𝒊

where each derivative is often a small number (like < 1).

• When these small values are multiplied across many layers, the overall gradient
tends toward zero.

• This makes weight updates negligible for the early layers, effectively preventing
them from learning useful features.

Causes

1. Sigmoid / Tanh Activations:

These functions squash inputs into small ranges (Sigmoid: 0–1, Tanh: –1–1), where
derivatives are < 0.25.
This small derivative value propagates back and reduces exponentially.

2. Deep Networks:
Many layers compound the effect of small gradients.
 3. Poor Weight Initialization:
Improperly scaled initial weights can cause activations to saturate quickly.

4. Improper Network Design:

Absence of skip/residual connections in deep architectures.

Effects

• Slower or stalled training

• Network fails to converge

• Early layers learn almost nothing

• Model accuracy plateaus at low levels

Technique Explanation
ReLU has a gradient of 1 for positive inputs,
Use ReLU Activation
preventing gradients from vanishing.
Xavier / He Initializes weights to maintain variance of activations
Initialization and gradients across layers.
Keeps activations within an optimal range, stabilizing
Batch Normalization
gradients.
Residual Connections Allow gradients to flow directly across layers,
(ResNet) solving vanishing gradient in very deep networks.
Gradient Clipping Prevents extremely small or large gradient values.

Example

• A 10-layer Sigmoid-based network may show training stagnation due to vanishing

gradients.

• Replacing Sigmoid with ReLU or Leaky ReLU activations improves convergence

speed drastically.

(b) Significance of Training Dataset Size in Neural Network Performance

Definition

The training dataset size refers to the amount of labeled data available for model training.
In deep learning, the dataset size has a direct impact on model accuracy, generalization,
and overfitting.
Explanation

Deep neural networks contain millions of trainable parameters. To learn these parameters
effectively, a large and diverse dataset is essential.
The larger the dataset, the better the model can capture variations and generalize to
unseen data.

Dataset Size Impact on Model

Model tends to overfit, memorizing training examples instead of

Small dataset
learning patterns.

Model generalizes better, reducing variance and improving test

Large dataset
accuracy.

Imbalanced
Model becomes biased toward majority classes.
dataset

Example

• In ImageNet, AlexNet (60 million parameters) trained on 1.2 million images

achieved record-breaking accuracy.

• If the same model were trained on only 10,000 images, it would overfit severely
and perform poorly on test data.

Solutions to Handle Small Datasets

1. Data Augmentation: Generate new samples by rotating, flipping, or cropping

images.

2. Transfer Learning: Use pre-trained networks and fine-tune on small data.

3. Regularization: Dropout, L2 penalties to reduce overfitting.

4. Synthetic Data Generation: Use GANs or simulation-based methods to enlarge

datasets.

Graphical Understanding

If you plot accuracy vs dataset size, the accuracy curve increases sharply with more data,
then plateaus as it approaches maximum generalization capability.
Conclusion

• The Vanishing Gradient Problem limits deep network training efficiency and
convergence, while

• The Training Dataset Size determines how well a neural network generalizes to
unseen data.

Both must be handled carefully — using ReLU-based architectures, residual networks, and
large, well-augmented datasets — to achieve optimal deep learning performance.

Q4. Explain AlexNet-5 in detail with necessary diagrams.

Introduction

AlexNet-5 (or AlexNet) is a deep convolutional neural network (CNN) developed by Alex
Krizhevsky, Ilya Sutskever, and Geoffrey Hinton (2012).
It was the first CNN to achieve state-of-the-art performance on the ImageNet Large Scale
Visual Recognition Challenge (ILSVRC), reducing classification error from 26% to 15%.

AlexNet marked the beginning of the deep learning revolution in computer vision.

Architecture Overview

AlexNet has a total of 8 layers:

• 5 Convolutional Layers – used for feature extraction

• 3 Fully Connected Layers – used for classification

The final output layer uses Softmax activation for 1000 ImageNet categories.

Input Image: 227 × 227 × 3 (RGB image)

 Filter/Kernel Stride / Output
Layer Type Activation / Remark
Size Pool Feature Map

Input - 227×227×3 - - Input RGB Image

ReLU + Local Response

11×11, 96 Stride =
Layer 1 Conv1 55×55×96 Normalization (LRN) +
filters 4
Max Pooling

5×5, 256 Stride =

Layer 2 Conv2 27×27×256 ReLU + LRN + Max Pooling
filters 1

3×3, 384 Stride =

Layer 3 Conv3 13×13×384 ReLU activation
filters 1

3×3, 384 Stride =

Layer 4 Conv4 13×13×384 ReLU activation
filters 1

3×3, 256 Stride =

Layer 5 Conv5 13×13×256 ReLU + Max Pooling
filters 1

Flatten Converts 3D feature maps

- - - -
Layer to 1D vector

Fully
FC1 - - 4096 neurons ReLU + Dropout (50%)
Connected

Fully
FC2 - - 4096 neurons ReLU + Dropout (50%)
Connected

Fully
FC3 Connected - - 1000 neurons Softmax Classifier
(Output)
Key Features of AlexNet

1. ReLU Activation Function

o Faster convergence compared to sigmoid/tanh.

o Helps avoid vanishing gradient problems.

2. GPU-based Training

o Trained using two NVIDIA GTX 580 GPUs, each handling half the layers.

o Allowed for large-scale parallel computation.

3. Local Response Normalization (LRN)

o Normalizes neuron outputs to encourage competition among neurons,

improving generalization.

4. Dropout Regularization

o Randomly drops neurons during training (in fully connected layers).

o Prevents overfitting.

5. Data Augmentation
 o Techniques like random cropping, flipping, and color jittering used to
increase dataset diversity.

6. Overlapping Pooling

o Uses 3×3 pooling windows with stride 2 for better feature retention.

Advantages Over Traditional Neural Networks

Traditional NN AlexNet Improvement

Shallow architecture Deep 8-layer CNN

Sigmoid/Tanh activations ReLU activation

CPU training GPU acceleration

No regularization Dropout + Data Augmentation

Poor image understanding Convolution + Pooling capture spatial hierarchy

Training Details

• Dataset: ImageNet (1.2 million images, 1000 classes)

• Optimizer: Stochastic Gradient Descent (SGD)

• Learning Rate: 0.01 with momentum = 0.9

• Regularization: L2 weight decay

• Batch Size: 128

Results and Achievements

• Achieved top-5 error = 15.3%, outperforming the next best model by 10%.

• Triggered a wave of deep learning research, leading to architectures like VGGNet,

GoogLeNet, and ResNet.

Conclusion

AlexNet-5 demonstrated that deep convolutional architectures, trained with large datasets
and GPUs, can achieve exceptional accuracy in visual recognition tasks.
It introduced innovations like ReLU, Dropout, and GPU training, making it the foundation
for modern deep learning models in image processing.

Q5. Explain the complete process of training a Convolutional Neural Network (CNN) for an
image classification task.

Introduction

A Convolutional Neural Network (CNN) is a deep learning architecture primarily used for
image recognition and classification.
CNNs automatically learn spatial hierarchies of features from images through multiple
layers — convolution, pooling, and fully connected layers.
Training a CNN involves several key steps that allow the network to learn patterns from
labeled image data and classify new unseen images effectively.

Step-by-Step Process of Training a CNN

1. Data Collection

• Collect a large labeled dataset of images.

Example: Cats vs Dogs, MNIST digits, CIFAR-10, or ImageNet.

• Data should represent all classes evenly to avoid bias.

2. Data Preprocessing

Preprocessing ensures that all images are in a consistent format suitable for CNN input.

Common Preprocessing Steps:

• Resizing: Convert all images to a fixed dimension (e.g., 224×224).

• Normalization: Scale pixel values between 0–1 or –1 to +1 to stabilize training.

• Label Encoding: Convert class labels into numerical form (e.g., one-hot encoding).

3. Data Augmentation

• Used to artificially expand the dataset and reduce overfitting.

 • Techniques include:

o Random rotations

o Flips (horizontal/vertical)

o Shifts and zooms

o Color jittering or cropping

• Ensures the model generalizes to unseen variations.

4. Designing the CNN Architecture

A typical CNN consists of the following layers:

Layer Type Purpose

Convolution Layer Extracts feature maps using kernels/filters (e.g., 3×3).

Introduces non-linearity, making the model learn complex

Activation Layer (ReLU)
patterns.

Pooling Layer (Max Reduces dimensionality and computation by summarizing

Pooling) features.

Fully Connected Layer Performs classification based on learned features.

Softmax Layer Converts final outputs into class probabilities.

5. Forward Propagation

• The image passes through the CNN layers.

• Each convolution operation extracts spatial features (edges, textures, shapes).

• Pooling reduces feature map size, retaining important information.

• The final fully connected layer outputs a vector representing class probabilities.

6. Loss Function Calculation

• After forward pass, compare predicted output vs actual label using a loss function.

• Common loss functions:

 o Categorical Cross-Entropy (for multi-class problems)

o Binary Cross-Entropy (for two-class problems)

𝑵

𝑳 = −∑ ̂𝒊 )
𝒚𝒊 𝐥𝐨𝐠⁡(𝒚
𝒊=𝟏

Where:

• 𝒚𝒊 : True label

• ̂𝒊 : Predicted probability
𝒚

7. Backpropagation

• Calculates gradients of the loss with respect to weights using the chain rule.

• Propagates the error backward through the layers.

• Determines how much each weight contributed to the error.

8. Weight Update (Optimization)

• After computing gradients, update weights to minimize loss using an optimizer.

Common Optimizers:

• SGD (Stochastic Gradient Descent)

• Adam (Adaptive Moment Estimation)

• RMSProp

Weight Update Rule:

𝛛𝑳
𝑾𝒏𝒆𝒘 = 𝑾𝒐𝒍𝒅 − 𝜼
𝛛𝑾

Where 𝜼is the learning rate.

9. Model Evaluation (Validation Phase)

• Use a validation dataset to evaluate model performance after each epoch.

• Monitor loss and accuracy curves to detect overfitting or underfitting.

 • Techniques like early stopping can halt training when validation loss stops
improving.

10. Hyperparameter Tuning

Adjust parameters like:

• Learning rate

• Batch size

• Number of filters

• Dropout rate

• Number of epochs

These significantly affect convergence and performance.

11. Model Testing

• After training, test the CNN on a separate test dataset.

• Evaluate using metrics like:

o Accuracy

o Precision, Recall, F1-score

o Confusion Matrix

12. Deployment

• Save the trained model for real-world use.

• Deploy in applications like object detection, facial recognition, or medical

diagnosis.

Advantages of CNN Training

1. Automatic Feature Extraction: No manual feature engineering.

2. Translation Invariance: Robust to small changes in position and scale.

3. High Accuracy: Superior in image-based tasks.

4. Reusability: Pre-trained models can be fine-tuned (Transfer Learning).

Conclusion

Training a CNN involves systematic steps — from data preparation and architecture design
to forward propagation, backpropagation, and optimization.
A well-trained CNN can learn meaningful patterns, achieve high classification accuracy, and
generalize effectively to new images.

Q6. Gradient Descent (GD) is foundational but slow to converge. Explain three major
modifications to GD that significantly improve convergence speed and stability.

Introduction

Gradient Descent (GD) is an optimization algorithm used to minimize the loss function in
machine learning models by updating the model’s parameters in the opposite direction of
the gradient of the loss function.

The basic weight update rule is:

𝛛𝑳
𝑾𝒏𝒆𝒘 = 𝑾𝒐𝒍𝒅 − 𝜼
𝛛𝑾

where:

• 𝑾: Model weight

• 𝜼: Learning rate
𝛛𝑳
• : Gradient of the loss function
𝛛𝑾

While standard GD is conceptually simple, it has limitations like slow convergence,

oscillations, and local minima trapping. Hence, improved variants were introduced.

Limitations of Basic Gradient Descent

1. Slow convergence when learning rate is small.

2. Divergence when learning rate is too high.

3. Getting stuck in local minima or saddle points.

4. Oscillations in steep regions of the loss surface.

 5. Uniform learning rate for all parameters.

Three Major Modifications for Improved GD

1. Momentum-Based Gradient Descent

Concept:

• Momentum accelerates convergence by accumulating a velocity vector of past

gradients.

• It smooths out oscillations and helps the model continue moving in consistent
directions.

Mathematical Formulation:
𝛛𝑳
𝒗𝒕 = 𝜷𝒗𝒕−𝟏 + (𝟏 − 𝜷)
𝛛𝑾𝒕
𝑾𝒕+𝟏 = 𝑾𝒕 − 𝜼𝒗𝒕

where:

• 𝒗𝒕 : Velocity (moving average of gradients)

• 𝜷: Momentum coefficient (typically 0.9)

• 𝜼: Learning rate

Intuition:
Like rolling a ball down a hill — it gains momentum and avoids getting stuck in small pits.

Advantages:

• Faster convergence

• Reduces oscillations in steep areas

• Helps escape local minima

Example:
Used in networks like AlexNet and VGGNet for stable training.

2. RMSProp (Root Mean Square Propagation)

Concept:
 • RMSProp adapts the learning rate for each parameter individually based on the
magnitude of recent gradients.

• It divides the gradient by a running average of its recent magnitude.

Formulas:

𝑬[𝒈𝟐 ]𝒕 = 𝜷𝑬[𝒈𝟐 ]𝒕−𝟏 + (𝟏 − 𝜷)𝒈𝟐𝒕

𝜼
𝑾𝒕+𝟏 = 𝑾𝒕 − 𝒈𝒕
√𝑬[𝒈𝟐 ]𝒕 + 𝝐

where:

• 𝑬[𝒈𝟐 ]𝒕 : Running average of squared gradients

• 𝝐: Small constant to avoid division by zero

Advantages:

• Automatically scales the learning rate per parameter

• Suitable for non-stationary problems (e.g., RNNs)

• Handles oscillations in vertical directions

Used in:
Recurrent Neural Networks (RNNs) and reinforcement learning algorithms.

3. Adam (Adaptive Moment Estimation)

Concept:
Adam combines the ideas of Momentum and RMSProp by maintaining both first and
second moments of the gradients.

Formulas:

1. Compute first moment (mean of gradients):

𝒎𝒕 = 𝜷𝟏 𝒎𝒕−𝟏 + (𝟏 − 𝜷𝟏 )𝒈𝒕

2. Compute second moment (uncentered variance):

𝒗𝒕 = 𝜷𝟐 𝒗𝒕−𝟏 + (𝟏 − 𝜷𝟐 )𝒈𝟐𝒕

3. Bias correction:
 𝒎𝒕 𝒗𝒕
̂𝒕 =
𝒎 , ̂
𝒗 𝒕 =
𝟏 − 𝜷𝒕𝟏 𝟏 − 𝜷𝒕𝟐

4. Update rule:
̂𝒕
𝜼𝒎
𝑾𝒕+𝟏 = 𝑾𝒕 −
√𝒗̂𝒕 + 𝝐

Default hyperparameters:

𝜷𝟏 = 𝟎. 𝟗, 𝜷𝟐 = 𝟎. 𝟗𝟗𝟗, 𝜼 = 𝟎. 𝟎𝟎𝟏

Advantages:

• Combines benefits of Momentum + RMSProp

• Requires less tuning of learning rate

• Works efficiently on large datasets and parameters

• Fast and stable convergence

Used in:
Most modern architectures like ResNet, Transformer, BERT, etc.

5. Comparison Table
Learning
Speed of
Rate Momentu Stability
Algorithm Convergenc
Adaptatio m Used
e
n

Standard Moderat
Constant No Slow e
GD

Momentu Fast (less High

Constant Yes
m GD oscillation)
Adaptive Very
RMSProp per No Faster Stable
parameter
Adaptive Very
Adam per Yes Fastest Stable
parameter
 Advantages of Modified GD Algorithms
1. Faster convergence with adaptive learning rates
2. Better handling of noisy or sparse gradients
3. Reduces oscillations during training
4. Suitable for training deep networks
5. Requires minimal hyperparameter tuning

Conclusion
While traditional Gradient Descent provides the foundation for training neural
networks, its convergence can be inefficient.
The advanced optimizers — Momentum, RMSProp, and Adam — combine adaptive
learning rate and momentum strategies, making training faster, more stable, and
suitable for complex deep learning models.

Q7. Explain with a diagram how overfitting and underfitting impact the learning
curve of a machine learning model.

Introduction
In machine learning, the goal is to build a model that performs well on both
training data and unseen test (validation) data.
However, depending on model complexity and training, a model may suffer from
either underfitting or overfitting.
These can be identified by analyzing training and validation learning curves.

1. Understanding Learning Curves

A learning curve is a graphical representation showing how a model’s performance
(loss or error) changes with training time or model complexity.
• X-axis: Training iterations or model complexity
• Y-axis: Error or loss
Typically, two curves are plotted:
• Training Error Curve
• Validation Error Curve

2. Underfitting
Definition:
Underfitting occurs when the model is too simple to capture the underlying
patterns in the data.
It performs poorly on both training and validation sets.
Characteristics:
• High training error
• High validation error
• Model unable to learn complex relationships
Causes:
• Model is not complex enough (e.g., linear model on nonlinear data)
• Too few training epochs
• High regularization
Solutions:
• Increase model complexity
• Train for more epochs
• Reduce regularization

3. Overfitting
Definition:
Overfitting occurs when the model is too complex and learns not only the patterns
but also the noise in training data.
It performs well on training data but poorly on new unseen data.
Characteristics:
• Low training error
• High validation/test error
• Poor generalization
Causes:
• Too complex model (many parameters)
• Insufficient training data
• No regularization or dropout
Solutions:
• Use regularization (L1/L2, dropout)
• Apply data augmentation
• Use early stopping
• Simplify model architecture

4. Optimal Fit
Between underfitting and overfitting lies the optimal fit, where the model achieves
a balance between bias and variance.
• Training error and validation error are both low and close together.
• Model generalizes well to new data.

5. Labeled Diagram: Learning Curves

Use the following labeled diagram in your answer sheet:
 Explanation of Diagram:
• The blue curve represents training error
• The orange curve represents validation error
• Underfitting region: Both errors are high and close
• Overfitting region: Training error is low, but validation error increases
• Optimal fit: Point where validation error is minimum — best generalization
performance

6. Summary Table
Aspect Underfitting Overfitting Optimal Fit

Model Balanced
Too low Too high
Complexity

Training Error High Very Low Low

Validation Low
High High
Error

Generalization Poor Poor Good

Deep Neural Properly

Example Linear Regression
Network trained tuned CNN
Model on complex data
too long
 7. Key Observations
• Underfitting → High bias, low variance
• Overfitting → Low bias, high variance
• Optimal Fit → Balanced bias and variance
This relationship is often referred to as the Bias-Variance Trade-off.

Conclusion
Overfitting and underfitting can be identified through learning curves.
A well-tuned model achieves a balance between bias and variance — resulting in
low training and validation errors, representing the optimal fit for accurate and
generalized predictions.

Q8. Explain LeNet-5 in detail with necessary diagrams.

Introduction

LeNet-5 is one of the earliest and most influential Convolutional Neural Networks (CNNs),
developed by Yann LeCun et al. in 1998.
It was designed for handwritten digit recognition (on the MNIST dataset) and laid the
foundation for modern CNN architectures like AlexNet, VGG, and ResNet.

LeNet-5 demonstrated how convolutional layers, subsampling (pooling), and fully

connected layers can automatically learn hierarchical image features.

1. LeNet-5 Architecture Overview

LeNet-5 consists of 7 layers (excluding the input) —

3 Convolutional layers, 2 Subsampling layers, and 2 Fully Connected layers, followed by an
Output layer.
The input image size is 32 × 32 pixels (grayscale).

Input → C1 → S2 → C3 → S4 → C5 → F6 → Output

2. Layer-Wise Explanation
 Kernel / No. of
Output
Layer Type Filter Feature Activation Description
Size
Size Maps

1 (Gray
Input - - 32×32×1 - Input grayscale image
image)

Detects low-level
C1 Convolution 5×5 6 28×28×6 Tanh features like edges,
corners

Reduces
Subsampling Average
S2 2×2 6 14×14×6 dimensionality,
(Pooling) Pooling
retains essential info

Learns more complex

C3 Convolution 5×5 16 10×10×16 Tanh patterns
(shapes/textures)

Average Further reduces size

S4 Subsampling 2×2 16 5×5×16
Pooling for dense layers

Convolution
Acts as feature vector
C5 (Fully 5×5 120 1×1×120 Tanh
for classification
Connected)

Fully 84 Combines features for

F6 - - Tanh
Connected neurons classification

Outputs probabilities
Fully 10
Output - - Softmax for 10 digit classes (0–
Connected neurons
9)

4. Diagram of LeNet-5 Architecture

5. Training Details

Parameter Description

Dataset MNIST (70,000 grayscale handwritten digits)

Optimizer Gradient Descent with Backpropagation

Activation Function Tanh

Loss Function Mean Squared Error

Pooling Type Average Pooling

Total Parameters ~60,000

6. Advantages of LeNet-5

1. Automatic Feature Extraction: No need for manual feature engineering.

2. Reduced Parameters: Pooling layers reduce computation cost.

3. Translation Invariance: Pooling helps the network handle variations in position.

4. Foundation Model: Inspired all modern CNN architectures.

5. Efficient Training: Achieved excellent results on low-resource hardware.

7. Limitations

1. Shallow Architecture: Limited to small-scale images.

2. Low Computational Power (at the time): Constrained network depth.

 3. Activation Function: Sigmoid/tanh caused vanishing gradient issues in deeper
networks.

4. Not suitable for color or high-resolution images.

8. Comparison: LeNet-5 vs. AlexNet

Aspect LeNet-5 (1998) AlexNet (2012)

Depth 7 layers 8 layers

Input Image 32×32 grayscale 227×227 RGB

Activation Tanh/Sigmoid ReLU

Pooling Average pooling Max pooling

Dataset MNIST ImageNet

Computation CPU-based GPU-accelerated

Conclusion

LeNet-5 was the pioneering CNN architecture that proved the power of deep learning in
visual recognition.
Its combination of convolution, pooling, and fully connected layers established the
structural foundation for modern CNNs like AlexNet, VGG, and ResNet.
Although simple by today’s standards, LeNet-5 remains a cornerstone in the history of
deep learning.

Q10. Describe each stage involved in developing and training a CNN model for classifying
images into categories.

Introduction

A Convolutional Neural Network (CNN) is a deep learning model designed to process

image data efficiently by learning spatial hierarchies of features.
Developing and training a CNN involves several systematic stages, from data preparation
to model deployment.
Each stage contributes to ensuring that the model learns robust, generalizable features for
accurate image classification.

Stages in CNN Development and Training

1. Problem Definition and Dataset Selection

• Define the objective of classification (e.g., recognizing cats vs dogs, handwritten

digits, vehicles, etc.).

• Select or collect an appropriate labeled dataset (e.g., MNIST, CIFAR-10, ImageNet).

• Split the dataset into:

o Training set (70%)

o Validation set (15%)

o Test set (15%)

2. Data Preprocessing

Ensures the images are uniform and suitable for CNN input.

Steps:

• Resizing: Convert all images to a fixed dimension (e.g., 224×224).

• Normalization: Scale pixel values between 0 and 1 to stabilize gradient descent.

• Encoding: Convert labels to numerical form (e.g., one-hot encoding).

• Noise Removal: Clean corrupted or irrelevant images.

3. Data Augmentation

Used to increase dataset size artificially and improve generalization.

Techniques include:

• Rotation and flipping

• Random cropping and zooming

• Brightness and contrast adjustments

 • Color jittering

This helps reduce overfitting and improves model robustness.

4. Model Architecture Design

CNN models consist of multiple types of layers that extract and combine features
hierarchically.

Layer Type Function

Convolutional Layer Extracts features using kernels (e.g., 3×3 filters).

Activation Layer (ReLU) Introduces non-linearity, allowing learning of complex patterns.

Pooling Layer Reduces feature map size while retaining important information.

Fully Connected Layer Combines all extracted features for classification.

Softmax Layer Converts outputs into probability distributions for each class.

5. Forward Propagation

• Each image passes through convolution → activation → pooling layers.

• The convolutional layers learn low-level (edges) and high-level (objects) features.

• The fully connected layers interpret these features and assign probabilities to each
class.

̂) for given inputs.

The model computes predicted outputs (𝒚

6. Loss Function Calculation

The loss function measures how far predictions are from actual labels.

Commonly used losses:

• Cross-Entropy Loss (for classification):

̂𝒊 )
𝑳 = −∑𝒚𝒊 𝐥𝐨𝐠⁡(𝒚

̂𝒊 is the predicted probability.

where 𝒚𝒊 is the true label, 𝒚

• Lower loss means better model predictions.

7. Backpropagation

• Computes the gradient of loss with respect to every weight using the chain rule.

• Gradients are propagated backward through the network.

• Determines how much each neuron contributed to the error.

This process updates the weights to minimize loss in the next iteration.

8. Optimization (Weight Update)

Optimizers update network weights based on computed gradients.

Common Optimizers:

• SGD (Stochastic Gradient Descent)

• Momentum

• Adam (Adaptive Moment Estimation)

Update rule:
𝛛𝑳
𝑾𝒏𝒆𝒘 = 𝑾𝒐𝒍𝒅 − 𝜼
𝛛𝑾

where 𝜼= learning rate.

9. Model Evaluation (Validation)

• Evaluate model on validation data after each training epoch.

• Monitor:

o Training accuracy

o Validation accuracy

o Loss curves

• Detect overfitting if training accuracy rises but validation accuracy decreases.

• Use techniques like early stopping to prevent overfitting.

10. Testing and Performance Analysis

After final training:

• Test the model on unseen data (test set).

• Compute metrics:

o Accuracy

o Precision

o Recall

o F1-score

o Confusion matrix

These determine how well the model generalizes to new images.

11. Model Deployment

• Convert the trained model into a deployable format (e.g., TensorFlow SavedModel
or ONNX).

• Integrate into real-world applications such as:

o Face recognition systems

o Medical image diagnostics

o Traffic sign classification

12. Performance Optimization Techniques

Technique Purpose

Regularization (L1/L2) Reduces overfitting by penalizing large weights

Dropout Randomly drops neurons to prevent co-dependence

Batch Normalization Stabilizes learning by normalizing layer inputs

Learning Rate Scheduling Dynamically adjusts the learning rate

Transfer Learning Uses pre-trained models to reduce training time

Conclusion
Developing and training a CNN for image classification involves systematic steps — from
data preprocessing, model design, forward and backward propagation, to evaluation and
deployment.
Each stage is critical to ensure the model achieves high accuracy, robustness, and
generalization on unseen data.

Q11. What are Batch, Stochastic, and Mini-Batch Gradient Descent? Discuss how they
differ in terms of convergence speed, computational cost, and accuracy.

Introduction

Gradient Descent (GD) is an optimization algorithm used in machine learning and deep
learning to minimize the loss function by updating model parameters in the opposite
direction of the gradient.

The general weight update rule is:

𝛛𝑳
𝑾𝒏𝒆𝒘 = 𝑾𝒐𝒍𝒅 − 𝜼
𝛛𝑾

where:

• 𝜼: Learning rate
𝛛𝑳
• : Gradient of loss with respect to weights
𝛛𝑾

Depending on how much data is used to compute the gradient, GD can be categorized into
three types:
Batch GD, Stochastic GD, and Mini-Batch GD.

1. Batch Gradient Descent (BGD)

Concept

• Uses all training samples to compute the gradient before updating weights.

• Performs one update per epoch (after processing the entire dataset).

Formula
 𝑵
𝟏 𝛛𝑳(𝒙𝒊 , 𝒚𝒊 )
𝑾𝒏𝒆𝒘 = 𝑾𝒐𝒍𝒅 − 𝜼 ∑
𝑵 𝛛𝑾
𝒊=𝟏

where 𝑵= total number of samples.

Characteristics

• Gradient is accurate and stable since it uses the entire dataset.

• High computational cost and memory requirement for large datasets.

Advantages

Smooth convergence
Stable updates
Suitable for smaller datasets

Disadvantages

Very slow for large datasets

Difficult to parallelize

2. Stochastic Gradient Descent (SGD)

Concept

• Updates weights after each training sample.

• Performs one update per sample rather than per epoch.

Formula
𝛛𝑳(𝒙𝒊 , 𝒚𝒊 )
𝑾𝒏𝒆𝒘 = 𝑾𝒐𝒍𝒅 − 𝜼
𝛛𝑾

Characteristics

• Faster initial learning

• Noisy updates (fluctuating loss values)

• Can escape local minima due to randomness

Advantages
 Fast and memory-efficient
Can handle very large datasets
Useful in online learning scenarios

Disadvantages

High variance in updates

May oscillate around minima rather than converging smoothly

3. Mini-Batch Gradient Descent

Concept

• Divides dataset into small batches (e.g., 32, 64, 128 samples).

• Each batch is used to compute gradient and update weights.

Formula
𝒎
𝟏 𝛛𝑳(𝒙𝒊 , 𝒚𝒊 )
𝑾𝒏𝒆𝒘 = 𝑾𝒐𝒍𝒅 − 𝜼 ∑
𝒎 𝛛𝑾
𝒊=𝟏

where 𝒎= batch size (typically 32–256).

Characteristics

• Combines benefits of Batch and Stochastic GD.

• Reduces variance in updates while keeping computations efficient.

Advantages

Efficient on GPUs (vectorized operations)

Faster convergence
Better generalization than full batch

Disadvantages

Still requires tuning batch size and learning rate

4. Comparison Table

Aspect Batch GD Stochastic GD (SGD) Mini-Batch GD

Data used per update Full dataset One sample Small subset (batch)

Convergence speed Slow Fast (but noisy) Fast and stable

Memory requirement High Low Moderate

Computation per update High Low Balanced

Accuracy High (stable) Medium (fluctuates) High (best trade-off)

Suitability Small datasets Very large datasets Deep learning tasks

Noise level Low High Medium

Parallelization Hard Easy Very easy

5. Learning Behavior (Graphical Explanation)

Diagram Description (for your answer sheet):

Use a simple labeled plot with:

• X-axis: Number of iterations (epochs)

• Y-axis: Loss (error)

Three curves showing:

• Batch GD: Smooth downward curve

• SGD: Zigzag path (noisy descent)

• Mini-Batch GD: Smooth but slightly oscillating curve

6. Example (Practical View)

Scenario Best Approach

Small datasets (few MBs) Batch GD

Streaming or real-time data SGD

Large image datasets (CIFAR, ImageNet) Mini-Batch GD

7. Summary

Criteria Best Performing Variant

Convergence Speed Mini-Batch GD

Stability Batch GD

Scalability Mini-Batch GD

Generalization Mini-Batch GD

Computational Efficiency Mini-Batch GD

8. Conclusion

All three gradient descent variants have their own strengths:

• Batch GD ensures stable convergence but is computationally heavy.

• SGD is fast and works well on large datasets but has noisy updates.

• Mini-Batch GD provides the best trade-off between convergence speed,

computational cost, and generalization.

Thus, Mini-Batch Gradient Descent is the most widely used optimization method for
training deep CNNs today.

Q12. Provide a comprehensive explanation of the CNN training workflow for an image
classification problem, covering data preparation, network architecture design, forward
propagation, backpropagation, and performance optimization.

Introduction

A Convolutional Neural Network (CNN) is a deep learning model designed to automatically

learn spatial hierarchies of features from images.
The training workflow of a CNN for image classification involves several systematic stages —
starting from data preparation, through model design and optimization, to evaluation.
This process ensures that the model generalizes effectively to classify unseen images
accurately.

STAGES OF CNN TRAINING WORKFLOW

1. Data Preparation

Data preparation ensures that the input to the CNN is consistent and meaningful.

Steps:

• Data Collection: Gather a labeled dataset (e.g., CIFAR-10, ImageNet).

• Resizing: Convert all images to a uniform size (e.g., 224×224 pixels).

• Normalization: Scale pixel values between 0 and 1 to stabilize gradients.

• Label Encoding: Convert class labels into one-hot encoded vectors.

• Data Augmentation: Artificially enlarge dataset using:

o Rotations, flips, zooms

o Cropping or brightness variation

o Prevents overfitting and improves generalization

2. Network Architecture Design

The architecture defines how the CNN processes and learns from the data.

Typical Layers in a CNN:

Layer Purpose

Convolution Layer Extracts features by applying filters/kernels (e.g., 3×3).

Activation Layer (ReLU) Introduces non-linearity.

Pooling Layer (Max or Average) Reduces feature map dimensions, retaining key features.

Flatten Layer Converts 2D feature maps into a 1D vector.

Fully Connected (Dense) Layer Combines learned features for classification.

Softmax Layer Outputs class probabilities.

Example:

Input → Conv → ReLU → Pool → Conv → ReLU → Pool → Flatten → FC → Softmax

3. Forward Propagation

In the forward pass, data flows through the network to generate predictions.

Steps:

1. Convolution:
Applies filters to extract spatial features from images.

𝑧 =𝑤∗𝑥+𝑏

2. Activation (ReLU):
Replaces negative values with zero → 𝑓(𝑧) = max⁡(0, 𝑧)

3. Pooling:
Reduces dimensions (e.g., 2×2 Max Pool selects the largest value).

4. Fully Connected Layer:

Combines features to form high-level representations.

5. Output Layer (Softmax):

Produces probabilities for each class:
𝑒 𝑧𝑖
𝑃(𝑦𝑖 ) =
∑𝑗 𝑒 𝑧 𝑗

4. Loss Function Computation

After the forward pass, the loss function quantifies prediction error.

Common choice:

• Categorical Cross-Entropy Loss:

𝐶

𝐿 = −∑ 𝑦𝑖 log⁡(𝑦̂𝑖 )
𝑖=1

where
𝑦𝑖 = true label (one-hot),
𝑦̂𝑖 = predicted probability.

Lower loss indicates that predictions are closer to true labels.

5. Backpropagation

Backpropagation updates model weights by propagating the error backward.

Steps:

1. Compute gradient of loss with respect to output (∂L/∂output).

2. Propagate gradients layer by layer using the chain rule:

∂𝐿 ∂𝐿 ∂𝑎 ∂𝑧
= ⋅ ⋅
∂𝑊 ∂𝑎 ∂𝑧 ∂𝑊

3. Adjust each weight 𝑊to minimize error.

Goal: Reduce the loss function over time through repeated updates.

6. Optimization

Optimizers control how weights are updated after computing gradients.

Common Optimizers:

• SGD (Stochastic Gradient Descent)

• Momentum

• RMSProp

• Adam (Adaptive Moment Estimation)

Update rule:
∂𝐿
𝑊𝑛𝑒𝑤 = 𝑊𝑜𝑙𝑑 − 𝜂
∂𝑊

where 𝜂is the learning rate.

7. Model Evaluation and Validation

After each epoch, the model is evaluated on validation data to check performance.

Metrics:

• Training Accuracy

• Validation Accuracy
 • Loss Curves

Observations:

• If validation loss increases while training loss decreases → Overfitting.

• Use early stopping to halt training when validation accuracy stops improving.

8. Performance Optimization Techniques

Technique Purpose

Dropout Randomly deactivates neurons to prevent overfitting

Batch Normalization Stabilizes learning and accelerates convergence

Learning Rate Scheduling Gradually decreases learning rate for fine-tuning

Transfer Learning Uses pre-trained models (e.g., VGG, ResNet) to save time

Data Augmentation Improves generalization by increasing data variety

9. Model Testing

After final training:

• Evaluate the model on a test dataset (unseen data).

• Compute:

o Accuracy

o Precision

o Recall

o F1-score

o Confusion Matrix

These metrics confirm model robustness and reliability.

10. Deployment
Once validated, the model can be exported and deployed for real-world applications such as:

• Object detection

• Facial recognition

• Medical image analysis

• Autonomous driving systems