DEEP LEARNING &

CONVOLUTIONAL NEURAL
NETWORKS
Comprehensive Exam Notes
Graduation Level | [Link] / [Link] / MCA / [Link]

TOPICS COVERED
Unit 1 – What is Deep Learning?
Unit 2 – Deep vs Shallow Networks
Unit 3 – CNN: Convolution Layers
Unit 4 – CNN: Pooling, Flattening, Fully-Connected Layers
Unit 5 – Softmax & Cross-Entropy Loss
Unit 6 – Building a CNN in Python
Unit 7 – Fully Connected CNN
Unit 8 – CNN Architectures: LeNet to DenseNet
Unit 9 – Training ConvNets: Init, BatchNorm, Hyperparameter Opt.
UNIT 1: What is Deep Learning?

◆ 1.1 Definition
Deep Learning is a branch of machine learning that uses artificial neural networks with multiple
layers to learn patterns from large amounts of data automatically.
The word 'deep' refers to the number of hidden layers in the network.
Deep learning models can learn complex relationships in data and are widely used in fields like image
recognition, speech recognition, and natural language processing.

Feature Shallow Network Deep Network

Number of Hidden Layers One or very few hidden Multiple hidden layers
layers
Complexity Simple models Complex models
Feature Learning Limited feature extraction Automatically learns complex features
Performance Suitable for simple tasks Suitable for complex tasks
Training Data Works with smaller datasets Requires large datasets

◆ 1.2 AI → ML → Deep Learning Hierarchy

AI / ML / DL RELATIONSHIP
┌──────────────────────────────────────────────────────────┐
│ ARTIFICIAL INTELLIGENCE │
│ (Rule-based systems, Expert systems, Search, etc.) │
│ ┌────────────────────────────────────────────────┐ │
│ │ MACHINE LEARNING │ │
│ │ (SVM, Decision Trees, Random Forest, etc.) │ │
│ │ ┌──────────────────────────────────────┐ │ │
│ │ │ DEEP LEARNING │ │ │
│ │ │ (CNN, RNN, Transformer, GAN, etc.) │ │ │
│ │ └──────────────────────────────────────┘ │ │
│ └────────────────────────────────────────────────┘ │
└──────────────────────────────────────────────────────────┘

◆ 1.3 Why Deep Learning? Key Advantages

• Automatic Feature Extraction: No need for hand-crafted features (unlike traditional ML).
• Scalability: Performance keeps improving with more data and compute.
• State-of-the-art Results: Dominates image, speech, text, and game-playing tasks.
• Transfer Learning: Pre-trained models can be fine-tuned for new tasks.
• End-to-End Learning: Raw input → prediction in one pipeline.
◆ 1.4 Applications of Deep Learning
Domain Application Architecture Used
Computer Vision Image classification, Object CNN
detection
NLP Machine translation, Sentiment RNN, Transformer
analysis
Speech Speech recognition, Text-to- RNN, CNN
speech
Healthcare Disease diagnosis, Drug CNN, GAN
discovery
Autonomous Vehicles Self-driving, LiDAR processing CNN, RNN
Generative AI Image synthesis, ChatGPT GAN, Transformer

UNIT 3: Convolutional Neural Networks (CNN) –

Convolution Layers
Convolutional Neural Network (CNN)
Convolutional Neural Network (CNN) is a type of Deep Neural Network designed to automatically detect and
learn important features from images and visual data.

CNNs are widely used in computer vision tasks because they can automatically extract features such as edges,
shapes, textures, and objects from images.

CNN uses a mathematical operation called convolution, where a filter (kernel) slides over the image to extract
important features.

Applications of CNN
CNNs are widely used in many real-world applications:
 Image recognition
 Video analysis
 Image classification
 Medical image analysis
 Computer vision

CNN Architecture
A Convolutional Neural Network consists of multiple layers arranged sequentially.
Main layers of CNN:
1. Input Layer
2. Convolution Layer
3. Pooling Layer
4. Fully Connected (Dense) Layer
5. Output Layer

Additional parameters include:

 Dropout Layer
  Filters (Kernel)
 Activation Functions

Simple CNN Architecture Flow

Input Image → Convolution Layer → Activation → Pooling → Flatten → Fully Connected Layer →
Output

1. Input Layer
The Input Layer is the first layer of the CNN where the raw image data is provided to the network.
 Images are represented as matrices of pixel values.
 Example: An RGB image of size 32 × 32 × 3
o 32 = width
o 32 = height
o 3 = color channels (Red, Green, Blue)
This layer only passes the image data to the next layer without any computation.

2. Convolution Layer
The Convolution Layer is the main layer responsible for feature extraction.
 Performs convolution operation between the input image and filter (kernel) of size M × M.
 These filters slide across the image and perform a dot product operation.
 The result is called a Feature Map.
 Feature maps help detect edges, corners, textures, and patterns in images
This layer helps the network learn spatial relationships between pixels.

3. Activation Layer
After convolution, an Activation Function is applied to introduce non-linearity.
 Help the network learn complex relationships between variables.
 Decide whether a neuron should be activated or not.
 Determine which information should move forward in the network.

Common activation functions:

 ReLU (Rectified Linear Unit) – most commonly used
 Sigmoid
 TanH
Example of ReLU:
f (x)=max(0 , x )
This helps the network learn complex patterns in data.

4. Pooling Layer
The Pooling Layer reduces the size of the feature maps.
 Helps reduce computational cost and memory usage.
 Decreases the number of connections between layers.
 Works independently on each feature map.
 Helps the model generalize features and recognize them independently.
 Acts as a bridge between convolution layers and fully connected layers.
Types of Pooling:
 Max Pooling: Selects the maximum value from a region of the feature map.
 Average Pooling: Calculates the average value of elements in a region.
 Sum Pooling: Calculates the sum of values in a region.
Example:
If input size = 32 × 32
After 2×2 pooling
Output size = 16 × 16

5. Flatten Layer
After convolution and pooling, the feature maps are converted into a one-dimensional vector.
Example:
16 × 16 × 12 → 3072 vector
This vector is used as input for the fully connected layer.

6. Fully Connected Layer (Dense Layer)

In this layer, all neurons are connected to every neuron in the next layer.
 The output from previous layers is flattened into a one-dimensional vector.
 This vector is fed into the fully connected layers.
 Performs mathematical operations for classification.
 Helps determine the final class or prediction.
The flattened vector is passed through one or more dense layers.

7. Output Layer
The Output Layer produces the final prediction.
Common activation functions used:
Sigmoid
 Used for binary classification
Softmax
 Used for multi-class classification

UNIT 5: Softmax & Cross-Entropy Loss

1. Softmax Function
The Softmax function is an activation function used in the output layer of neural networks for multi-
class classification problems.
Definition
Softmax converts the output values (logits) of a neural network into probability values between 0 and
1, where the sum of all probabilities equals 1.

Formula
xi
e
Softmax(x i)= n

∑ ex j

j=1
Where:
 x i= input value (logit)
 e = exponential function
  n = number of classes

Key Points
 Converts raw output scores into probabilities.
 Used in multi-class classification problems.
 The class with the highest probability is selected as the final prediction.
 Ensures that all output probabilities sum to 1.

Example
If a model predicts for 3 classes:
Class Output Score Softmax Probability
Cat 2.0 0.65
Dog 1.0 0.24
Bird 0.5 0.11
The predicted class will be Cat because it has the highest probability.

2. Cross-Entropy Loss
Cross-Entropy is a loss function used to measure the difference between predicted probabilities
and actual class labels.
It is commonly used together with Softmax in classification problems.
The loss is calculated based on the probability assigned to the correct class.

Formula
n
Loss=−∑ y i log ⁡(p i)
i=1
Where:
 y i= actual label
 pi= predicted probability
 n = number of classes

Key Points
 Measures how close the predicted probability is to the actual label.
 Lower cross-entropy value means better model performance.
 Used mainly for classification tasks.

UNIT 8: CNN Architectures

◆ 8.1 Evolution of CNN Architectures

1. LeNet
 LeNet is one of the earliest Convolutional Neural Network architectures developed by Yann
LeCun in 1998.
 Designed mainly for handwritten digit recognition.
 Used for recognizing digits in bank cheques and postal codes.
Features
 Consists of 7 layers (excluding input layer).
 Architecture includes Convolution → Pooling → Fully Connected layers.
 Uses tanh activation function.
  Works well for small grayscale images (32×32).

2. AlexNet
 AlexNet was developed by Alex Krizhevsky in 2012.
 It won the ImageNet Large Scale Visual Recognition Challenge (ILSVRC 2012).
Features
 Contains 8 layers (5 convolution layers + 3 fully connected layers).
 Uses ReLU activation function.
 Introduced Dropout to reduce overfitting.
 Uses GPU for faster training.
 Much deeper and more accurate than LeNet.

3. ZFNet (Zeiler & Fergus Network)

 ZFNet was introduced by Matthew Zeiler and Rob Fergus in 2013.
 It is an improvement of AlexNet.
Features
 Uses smaller convolution filters.
 Improves visualization of feature maps.
 Helps understand how CNN learns features.
 Won ILSVRC 2013 competition.

4. GoogLeNet (Inception Network)

 GoogLeNet was developed by Google in 2014.
 Also called Inception Network.
Features
 Contains 22 layers deep architecture.
 Uses Inception modules which apply multiple filters (1×1, 3×3, 5×5) in parallel.
 Reduces computation using 1×1 convolutions.
 Uses global average pooling instead of fully connected layers.
 More efficient than previous networks.

5. VGGNet
 VGGNet was developed by Visual Geometry Group (Oxford) in 2014.
Features
 Uses very small convolution filters (3×3).
 Has very deep architecture.
Two common versions:
 VGG16 (16 layers)
 VGG19 (19 layers)
Advantages:
 Simple and uniform architecture
 Good performance in image recognition
Disadvantages:
 Very large number of parameters
 Requires high memory and computation

6. ResNet (Residual Network)

 ResNet was introduced by Microsoft Research in 2015.
 It won ILSVRC 2015 competition.
Features
 Introduces Residual Connections (Skip Connections).
 Helps solve Vanishing Gradient Problem.
 Enables very deep networks (50, 101, 152 layers).
Example architectures:
 ResNet50
 ResNet101
 ResNet152
Residual connections allow the network to skip layers and learn identity mappings.

7. DenseNet (Dense Convolutional Network)

 DenseNet was introduced in 2017.
Features
 Each layer is connected to every other layer in the network.
 Feature maps are reused throughout the network.
Advantages:
 Improves information flow
 Reduces vanishing gradient problem
 Requires fewer parameters compared to ResNet
Training a ConvNet
Training a Convolutional Neural Network (ConvNet) involves adjusting the network
parameters so that the model can learn patterns from data and make accurate predictions.
Important aspects of training include weight initialization, batch normalization, and
hyperparameter optimization.

1. Weight Initialization
Weight initialization refers to the process of assigning initial values to the weights of the neural
network before training begins.
Importance
 Proper initialization helps the model converge faster during training.
 Prevents problems such as vanishing gradients or exploding gradients.
 Poor initialization can make training slow or unstable.
Common Methods
 Random Initialization: Weights are initialized with small random values.
 Xavier Initialization: Designed to keep the variance of inputs and outputs similar across layers.
 He Initialization: Commonly used with ReLU activation functions for better performance.
Key Idea
Weights should not be initialized to zero, because all neurons would learn the same features.

2. Batch Normalization
Batch Normalization is a technique used to normalize the inputs of each layer during training.
Purpose
 Improves training speed and stability.
 Reduces the problem of internal covariate shift (change in data distribution during training).
Working
1. Calculate mean and variance of the batch.
2. Normalize the data.
3. Apply scaling and shifting parameters.
Advantages
 Faster training
 Higher learning rates can be used
 Reduces overfitting
 Improves model accuracy
Batch normalization is usually applied after the convolution layer and before the activation
function.

3. Hyperparameter Optimization
Hyperparameter optimization is the process of selecting the best hyperparameters for a neural
network model.
Hyperparameters are parameters set before training and are not learned automatically.
Examples of Hyperparameters
 Learning rate, Batch size, Number of epochs
  Number of layers
 Number of filters
 Dropout rate
Optimization Methods
 Grid Search: Tests all possible combinations of hyperparameters.
 Random Search: Randomly selects combinations to test.
 Bayesian Optimization: Uses probability models to find optimal parameters.
Goal
To find the best combination of hyperparameters that improves model performance and
accuracy.