Introduction to Deep Learning

Deep learning is an advanced stage in the evolution of artificial intelligence and

machine learning. To understand deep learning, it is important to first understand how it
developed from earlier approaches in artificial intelligence. Artificial intelligence is the
broad field that aims to create machines capable of performing tasks that normally
require human intelligence, such as reasoning, decision-making, and problem-solving.
Machine learning is a subset of artificial intelligence that focuses on enabling machines
to learn patterns from data instead of being explicitly programmed. Deep learning is a
further subset of machine learning that uses multi-layered neural networks to
automatically learn complex patterns from large amounts of data.

In the early days of artificial intelligence, systems were mainly rule-based. These
systems were called symbolic or classical AI systems. They relied on human experts to
write rules that the computer would follow. For example, an expert system for medical
diagnosis would use a set of rules like “If the patient has a fever and cough, then it may
be flu.” While these systems worked well in limited domains, they had several problems.
They required a large number of manually written rules, were difficult to scale, and could
not handle uncertainty or complex real-world data.

To overcome these limitations, researchers developed machine learning techniques.

Instead of writing rules manually, machine learning allows computers to learn patterns
from data. In machine learning, a model is trained using a dataset, and the model learns
the relationship between inputs and outputs. For example, a machine learning model
can be trained on emails labeled as spam or not spam. The model learns patterns in the
data and can then classify new emails. Traditional machine learning algorithms include
decision trees, support vector machines, k-nearest neighbors, and logistic regression.

However, traditional machine learning has some limitations. One of the main challenges
is feature engineering. In machine learning, humans must manually select and design
features that help the model learn. For example, in image classification, engineers must
design features such as edges, textures, or shapes. This process requires domain
expertise and is time-consuming. If the chosen features are not good, the model’s
performance will be poor.

Deep learning was developed to overcome this limitation. Instead of manually designing
features, deep learning models automatically learn features from raw data. Deep
learning uses artificial neural networks with multiple hidden layers. Each layer learns a
different level of representation. For example, in image recognition, the first layer may
detect edges, the next layer may detect shapes, and deeper layers may detect objects.
This hierarchical learning of features is the main strength of deep learning.

The evolution from machine learning to deep learning was made possible by several
technological advancements. One major factor was the availability of large datasets,
often referred to as big data. Modern applications generate massive amounts of data
from the internet, social media, sensors, and mobile devices. Deep learning models
require large datasets to perform well, and the availability of such data has enabled their
success.

Another important factor was the increase in computational power. Training deep neural
networks requires a large number of calculations. The development of graphics
processing units (GPUs) and tensor processing units (TPUs) made it possible to train
deep networks efficiently. GPUs can perform many calculations in parallel, significantly
speeding up the training process.

Advancements in algorithms also played a key role in the evolution of deep learning.
The backpropagation algorithm made it possible to train multi-layer neural networks.
Activation functions such as ReLU helped solve the vanishing gradient problem and
improved training speed. Techniques like dropout and batch normalization improved
model performance and stability.

A major milestone in deep learning occurred in 2012, when a deep convolutional neural
network called AlexNet won the ImageNet competition. It achieved significantly higher
accuracy than traditional machine learning methods. This success demonstrated the
power of deep learning and led to widespread adoption in many fields, including
computer vision, speech recognition, and natural language processing.

Today, deep learning is used in many real-world applications. It powers voice

assistants, facial recognition systems, recommendation engines, self-driving cars, and
medical diagnosis tools. Deep learning models are also used in natural language
processing tasks such as machine translation and chatbots.

Biological Neuron and Artificial Neural Network

Biological Neuron

A biological neuron is a basic cell in the human brain and nervous system. It is
responsible for receiving, processing, and sending information. The brain contains
billions of these neurons, and they work together to perform tasks such as thinking,
learning, speaking, and seeing.

A biological neuron has three main parts. The first part is dendrites, which receive
signals from other neurons. The second part is the cell body (soma), which processes
these signals. If the signal is strong enough, the neuron becomes active. The third part
is the axon, which sends the signal to other neurons. The connection between neurons
is called a synapse, where chemical signals are passed from one neuron to another.

Learning in the brain happens when the connections between neurons become stronger
or weaker over time. This process allows humans to remember information and adapt to
new situations.
Artificial Neuron

An artificial neuron is a simple mathematical model inspired by the biological neuron. It

is used in computers to process information. Artificial neurons are the basic units of
artificial neural networks.

An artificial neuron receives several input values. Each input is multiplied by a weight
that shows its importance. All the weighted inputs are added together, and a bias is
included. The result is passed through an activation function to produce the output.

In simple terms:

❖ Inputs come into the neuron.

❖ The neuron processes the inputs.

❖ The neuron produces an output.

During training, the weights are adjusted so the neuron can make better predictions.

Main Difference Between Biological Neuron and Artificial Neuron

Biological Neuron Artificial Neuron

A real cell found in the human brain A mathematical model used in computers

Part of the nervous system Part of artificial neural networks

Receives signals through dendrites Receives numerical inputs

Processes signals in the cell body Processes inputs using weighted sum and
(soma) bias

Sends signals through the axon Produces output through activation

function

Connections called synapses Connections called weights

Learning occurs by changing synaptic Learning occurs by adjusting weights

strength during training

Uses electrical and chemical signals Uses mathematical calculations

Very complex and energy-efficient Simpler and depends on computer

hardware
Works naturally in parallel with billions Works using artificial networks in software
of neurons or hardware

McCulloch Pitts neuron

1. Background

❖ Proposed in 1943 by Warren McCulloch (a neurophysiologist) and Walter Pitts

(a logician).

❖ It’s considered the first computational model of a biological neuron — the

starting point of Artificial Neural Networks (ANNs).

❖ Their paper “A Logical Calculus of the Ideas Immanent in Nervous Activity”

described how networks of simple binary neurons could compute any logical
function.

2. Concept

A McCulloch Pitts neuron (M-P neuron) is a simplified model of a biological neuron

that:

1. Receives multiple binary inputs (0 or 1).

2. Applies weights (often assumed to be 1 for simplicity).

3. Calculates the weighted sum of inputs.

4. Compare the sum to a threshold value.

5. Produces a binary output:

○ Output = 1 → if the sum ≥ threshold

○ Output = 0 → otherwise
Property Description
Inputs/Outputs Binary (0 or 1)

Weights Usually 1 (or small integers)

Threshold Determines firing condition

Output Binary (fires = 1, silent = 0)

Learning No learning – weights fixed manually

Use Implements logic gates (AND, OR,

NOT)
Limitations

Only works with binary inputs/outputs.

No learning – weights and thresholds are fixed manually.

Cannot solve non-linearly separable problems (like XOR).

Doesn’t model real neuron timing, inhibitory/excitatory balance in detail.

Significance

Despite simplicity, the McCulloch–Pitts neuron:

● Laid the foundation for later models (Perceptron, ANN, Deep Learning).

● Demonstrated that logical computation can emerge from networks of simple

neurons.

● Inspired decades of research into computational neuroscience and AI.

Perceptron

The perceptron is the simplest form of an artificial neural network, designed for binary
classification problems. It was introduced by Frank Rosenblatt in the 1950s as a
basic learning model inspired by the working of a biological neuron. The perceptron acts
like a single artificial neuron that receives inputs, processes them, and produces an
output of either 0 or 1, representing two different classes.

The main goal of the perceptron is to separate data into two groups by learning a
linear decision boundary, such as a straight line in 2D space or a plane in higher
dimensions.

Working of the Perceptron

 7. Weight Update (Learning Step)

If the prediction is incorrect, the weights and bias are adjusted to reduce the
error.
Limitation of Single-Layer Perceptron

The main limitation is:

Cannot Solve Non-Linear Problems

A single perceptron can only create a linear decision boundary.

For example:

● It can solve AND, OR problems.

● It cannot solve XOR because XOR is not linearly separable.

Multilayer Perceptron (MLP)

Definition

A Multilayer Perceptron is a type of feedforward neural network that contains:

● One input layer

● One or more hidden layers
● One output layer

Each neuron in one layer is usually connected to all neurons in the next layer.

Structure of an MLP
How MLP Overcomes Perceptron Limitations

1. Use of Hidden Layers

● Hidden layers allow the network to learn intermediate features.

● These features help create complex decision boundaries.

2. Nonlinear Activation Functions

MLP uses activation functions such as:

● Sigmoid
● Tanh
● ReLU

These functions introduce nonlinearity, allowing the network to solve problems like
XOR.

3. Ability to Learn Complex Patterns

MLP can:

● Recognize images
● Process speech
● Predict prices
● Solve non-linear classification problems

Working of MLP (Basic Steps)

1. Inputs are fed into the input layer.

2. Each neuron multiplies inputs with weights.
3. Weighted sums pass through activation functions.
4. Hidden layers process intermediate results.
5. Final output is produced at the output layer.
6. Error is calculated.
7. Backpropagation updates weights.
8. Process repeats until error is minimized.

Key Advantages of MLP over Perceptron

Feature Single-Layer Multilayer

 Perceptron Perceptron

Hidden layers No Yes

Decision Linear only Linear and nonlinear

boundary

Can solve XOR No Yes

Complexity Simple problems Complex problems

handled

Learning method Perceptron rule Backpropagation

Gradient Descent

Gradient Descent is an optimization algorithm used to minimize the loss (error)

function in machine learning and deep learning models.
It works by iteratively adjusting the model parameters (weights and bias) in the
direction that reduces the error.

In simple terms, gradient descent helps the model learn the best values of weights so
that the predictions become more accurate.

Intuition

Imagine you are standing on a hill and want to reach the lowest point in the valley.
You take steps downward in the steepest direction until you reach the bottom.
Gradient descent works in the same way:

● The hill represents the loss function.

● The lowest point represents the minimum error.
● The steps represent weight updates.

Visualization of Gradient Descent

Working of Gradient Descent (Step-by-Step)

1. Initialize weights randomly.

2. Feed input data into the model.
3. Compute predicted output.
4. Calculate loss (difference between predicted and actual output).
5. Compute gradient of the loss with respect to weights.
6. Update weights using gradient descent formula.
7. Repeat the process until the loss becomes very small or stops improving.

This repeated process is called training.

Learning Rate (𝛼α)

The learning rate controls how big a step we take during each update.

Effects of Learning Rate

Learning Effect
Rate

Too small Very slow learning

Too large May overshoot minimum or

diverge

Optimal value Fast and stable convergence

Types of Gradient Descent

1. Batch Gradient Descent (BGD)

Batch Gradient Descent computes the gradient using the entire dataset before taking
each update step.

If the dataset has mmm samples:

● Look at every training example

● Compute loss & gradient for each
● Average them
● Update parameters once.

Intuition

Imagine climbing down a mountain and stopping after each large observation of the
whole landscape.
You take very stable but very slow steps.

Computational Cost

For each step:

● Cost = O(m)O(m)O(m)

● If m=1,000,000m = 1,000,000m=1,000,000, every update requires computing

loss for all 1M samples.

This is extremely slow for deep learning.

Advantages

❖ Very stable
❖ Gradient direction is exact
❖ Converges smoothly to a minimum
❖ Works well for small datasets or convex problems
Limitations

❖ Very slow for large datasets

❖ Requires loading full dataset into memory
❖ Every update is expensive

In deep learning, datasets are huge → BGD becomes nearly impossible.

Where It Is Used

● Linear regression on small datasets

● Logistic regression with small feature sets

● When exact gradient is required (theoretical work)

2. Stochastic Gradient Descent (SGD)

Instead of using the whole dataset, SGD uses ONE training example per update.

Update:

Where i is randomly selected for each step.

Intuition

Imagine climbing down the mountain but using only a tiny piece of information each
time.
You move fast, but the path is shaky.

Key Characteristics

● No waiting for full dataset

● Updates happen immediately

● Gradient is extremely noisy

Advantages

❖ Very fast
❖ Works with streaming data
❖ Helps escape local minima due to noise
❖ Great for deep learning on massive datasets

Disadvantages

❖ Very noisy updates

❖ Loss fluctuates heavily
❖ Hard to converge exactly
❖ Requires good learning rate scheduling

Mini-Batch Gradient Descent (Most Popular in Deep Learning)

This method balances:

● Stability (like Batch GD)

● Speed (like SGD)

You take steps with moderate certainty: not too noisy, not too slow.

Why Mini-Batch Works Best

● Strong statistical estimate of gradient

● Efficient GPU utilization

 ● Batches fit in GPU memory

● Vectorized operations are much faster

● Smooth-enough updates for convergence

This is why nearly all modern deep learning frameworks (PyTorch, TensorFlow) use
mini-batches by default.

Advantages

Fast
More stable than SGD
More scalable than Batch GD
Works perfectly with GPUs
Allows parallelism
The best trade-off

Disadvantages

Adds a hyperparameter: batch size

Too small batch → noisy
Too large batch → memory issues + slower generalization

Optimal Batch Sizes in Practice

General rule of thumb:

● 32 or 64 for small networks

● 128 or 256 for medium networks

● 512+ for large datasets and GPUs with high memory

Comparison of GD, SGD, and Mini-Batch GD

Feature Batch GD SGD Mini-Batch

GD
Data used per Entire One sample Small batch
update dataset

Speed Slow Fast Moderate

Stability Very stable Noisy Balanced

Memory usage High Low Moderate

Common use Small Online Deep learning

datasets learning

Backpropagation Algorithm
Backpropagation (short for backward propagation of errors) is the main algorithm
used to train neural networks. It works by calculating the error at the output, then
propagating this error backward through the network to update the weights and
biases.

The goal of backpropagation is to minimize the loss function by adjusting weights

using gradient descent.

Basic Idea

● The network makes a prediction.

● The prediction is compared with the actual target.
● The error is calculated.
● This error is sent backward through the network.
● Each weight is updated to reduce the error.

This process is repeated many times during training.

Flow of Backpropagation
Main Stages of Backpropagation

Backpropagation consists of two main phases:

1. Forward Pass

● Input data is passed through the network.

● Each neuron performs:

● Activation function produces output.

● Final prediction is obtained.
● Loss is calculated.

2. Backward Pass

● Error is calculated at the output layer.

● Error is propagated backward through hidden layers.
● Gradients of loss with respect to each weight are computed.
● Weights are updated using gradient descent.
Step-by-Step Working of Backpropagation

1. Initialize weights and biases randomly.

2. Provide input data to the network.
3. Perform forward propagation to get output.
4. Compute the loss (difference between predicted and actual output).
5. Calculate the gradient of loss at the output layer.
6. Propagate the error backward through hidden layers.
7. Compute gradients for all weights and biases.
8. Update weights using gradient descent:

9. Repeat for many epochs until loss is minimized.

Relationship Between Backpropagation and Gradient Descent

Backpropagation Gradient Descent

Computes gradients of loss Uses gradients to update weights

Works backward through Updates parameters to minimize

network loss

Uses chain rule Uses learning rate for updates

Backpropagation provides the gradients, and gradient descent uses them to adjust
weights.

Advantages

● Efficient training of deep neural networks.

● Automatically computes gradients for all weights.
● Works for complex, multi-layer architectures.

Limitations

● Can suffer from vanishing or exploding gradients.

● Requires differentiable activation functions.

● Training may be slow for very deep networks.

Activation functions

Activation functions introduce non-linearity into neural networks, allowing them to learn
complex patterns. They determine how neurons fire and control the information flow
through the network. Common activation functions include Step, Sigmoid, Tanh, ReLU,
Leaky ReLU, and Softmax. Each has unique properties, advantages, and limitations
depending on the task.

1. Step Function

Used in early perceptrons.

 ● Hard threshold

● Output is binary

● Not differentiable → not suitable for deep learning

● Suitable for logic gates

2. Sigmoid Function

A smooth "S-shaped" curve.

Properties:

● Output range: (0, 1)

● Good for probabilities

● Used in binary classification (output layer)

Disadvantages:

● Causes vanishing gradient

● Slow convergence
3. Tanh (Hyperbolic Tangent)

Similar to sigmoid but range is (-1, 1)

Characteristics:

● Zero-centered (better than sigmoid)

● Still suffers from vanishing gradients

4. ReLU (Rectified Linear Unit)

Most widely used activation function today.

Advantages:

● Simple

● Very fast

● Does not saturate for large values

● Reduces vanishing gradient problems

Disadvantages:

● "Dying ReLU" problem (neurons stuck at zero)

5. Leaky ReLU
Fixes ReLU's dying problem.

More stable than ReLU.

6. Softmax Function

Used in multi-class classification (output layer).

Characteristics:

● Turns raw scores into probabilities

● Sum of all outputs = 1

● Used in CNNs & Transformers for classification

How Activation Functions Affect Learning

1. Gradient Flow

Smooth functions like ReLU allow gradients to flow easily → faster learning.

2. Range of Outputs

Some functions bound the output (sigmoid, tanh), which helps in normalization.

3. Computational Cost

ReLU is computationally cheap → used in deep networks.

Loss Functions in Deep Learning

A loss function (also called a cost function or error function) measures how far the
model’s predictions are from the actual target values.
It tells the model how bad its prediction is.

● If the prediction is correct → loss is small.

● If the prediction is wrong → loss is large.

The goal of training is to minimize the loss using algorithms like gradient descent.

Why Loss Functions Are Important

Loss functions:

● Guide the learning process.

● Measure model performance.

● Help update weights during backpropagation.

● Determine how the model improves over time.

● Basic Idea
1. Model makes a prediction.

2. Loss function compares prediction with actual value.

3. Loss is calculated.

4. Backpropagation uses this loss to update weights.

5. Process repeats until loss becomes small.

Types of Loss Functions

Loss functions depend on the type of problem:

Problem Type Common Loss Functions

Regression Mean Squared Error (MSE), Mean Absolute Error
(MAE)

Binary Classification Binary Cross-Entropy

Multi-class Categorical Cross-Entropy

Classification

1. Mean Squared Error (MSE)

2. Mean Absolute Error (MAE)

3. Binary Cross-Entropy (Log Loss)

4. Categorical Cross-Entropy
Comparison of Common Loss Functions

Loss Function Used For Key Idea

MSE Regression Squares error

MAE Regression Absolute error

Binary Cross-Entropy Binary classification Probability error

Categorical Cross- Multi-class Multi-class probability

Entropy classification error

Bias and Variance

In machine learning, prediction error can be divided into two main components: bias
and variance. Understanding these helps explain why overfitting occurs.
Bias

Bias is the difference between the predicted output and the true output.
It shows how much the model’s predictions deviate from the actual values.

● High bias means the model is too simple.

● It cannot capture the underlying pattern of the data.
● Leads to underfitting.

Example:
Using a straight line to fit highly curved data.

Variance

Variance measures how much the model’s performance changes with different
training data.

● High variance means the model is too sensitive to the training data.
● It learns noise and unnecessary details.
● Leads to overfitting.

Bias–Variance Trade-off

Model Type Bias Varianc Result

e

Very simple model High Low Underfittin

g

Balanced model Moderat Moderat Good fit

e e

Very complex Low High Overfitting

model

Goal:
Find a model with balanced bias and variance.
2. Overfitting

Definition

Overfitting occurs when a model:

● Performs very well on training data.

● Performs poorly on new or test data.

This happens because the model learns noise, fluctuations, and unnecessary
details in the training data.

Visualization of Overfitting
4

Characteristics of Overfitting

● Very low training error.

● High test or validation error.
● Model memorizes data instead of learning patterns.

Causes of Overfitting

1. Small training dataset.

2. Noisy or unclean data.
3. Too many parameters.
4. Very deep or complex model.
 5. Training for too many epochs.
regularization

3. Underfitting

Underfitting occurs when:

● The model is too simple.

● It cannot capture patterns in training data.

Characteristics

● High training error.

● High test error.
● Poor performance overall.

4. Regularization

Regularization is a technique used to:

● Reduce overfitting.
● Control model complexity.
● Improve generalization.
regularization

It works by:

● Adding constraints to the model.

● Preventing weights from becoming too large.

5. Types of Regularization Techniques

5.1 L1 Regularization (Lasso)

● Adds penalty equal to sum of absolute values of weights.

regularization
Effects

● Reduces weight magnitudes.

● Keeps all features.
● Produces smoother models.

L1 vs L2 Regularization

Feature L1 (Lasso) L2 (Ridge)

Penalty Absolute weights Squared

weights

Weight effect Some become All become

zero small
Feature Yes No
selection

Model type Sparse Smooth

5.3 Dropout

Dropout randomly removes some neurons during training.

regularization

Working

● Each neuron has a probability 𝑝p of being kept.

● Remaining neurons train normally.
● Different neurons are dropped in each iteration.

Benefits

● Prevents co-dependency of neurons.

● Reduces overfitting.
● Acts like an ensemble of many networks.

5.4 Early Stopping

Stop training when validation error starts increasing.

regularization

Steps

1. Train the model.

2. Monitor validation loss.
3. Stop when validation loss increases.

Benefit

● Prevents the model from memorizing training data.

5.5 Adding Noise to Input

● Small random noise is added to training data.

regularization
● Forces the model to learn general patterns.
● Improves robustness.

5.6 Ensemble Methods

Combine predictions from multiple models.

regularization

Idea

● Different models make different errors.

● Combining them reduces overall error.

Examples

● Bagging
● Boosting
● Random Forest

6. How Regularization Reduces Overfitting

Without With
Regularization Regularization

Large weights Smaller weights

Complex model Simpler model

Memorizes noise Learns patterns

High variance Balanced variance

Deep Feedforward Networks

A Deep Feedforward Network is a type of artificial neural network where information

flows in one direction only, from the input layer to the output layer, through multiple
hidden layers.
It is called deep because it contains more than one hidden layer.

These networks are also known as:

● Deep Neural Networks (DNNs)

● Multilayer Feedforward Networks

Basic Structure

A deep feedforward network consists of three main parts:

1. Input Layer
○ Receives the input data.
○ Each neuron represents one feature.
2. Hidden Layers
○ One or more layers between input and output.
○ Perform intermediate computations.
○ Learn complex patterns in the data.
3. Output Layer
○ Produces the final prediction.

Structure of a Deep Feedforward Network

4
Why It Is Called “Feedforward”

● Data flows only forward:

𝐼𝑛𝑝𝑢𝑡→𝐻𝑖𝑑𝑑𝑒𝑛 𝐿𝑎𝑦𝑒𝑟𝑠→𝑂𝑢𝑡𝑝𝑢𝑡Input→Hidden Layers→Output

● There are no loops or cycles.

● Unlike recurrent networks, outputs are not fed back into the network.

Working of a Deep Feedforward Network

Step-by-Step Process

1. Input data enters the input layer.

2. Inputs are multiplied by weights.
3. Bias is added.
4. Activation function is applied.
5. Output becomes input to the next layer.
6. Process continues through all hidden layers.
7. Final output is produced.
8. Loss is calculated.
9. Backpropagation updates weights.
10. Process repeats during training.
Activation Functions Used

Common activation functions:

Functio Use
n

ReLU Most common in hidden

layers

Sigmoid Binary output

Tanh Zero-centered activation

Softmax Multi-class output

Key Characteristics

● Multiple hidden layers.

● Fully connected neurons.
● Uses nonlinear activation functions.
● Trained using backpropagation and gradient descent.
● Can approximate complex functions.

Advantages

● Can learn complex, nonlinear relationships.

● High accuracy for many tasks.
● Foundation of deep learning.
Limitations

● Requires large datasets.

● High computational cost.
● Can suffer from vanishing gradients.

Difference: Shallow vs Deep Network

Feature Shallow Network Deep Feedforward

Network

Hidden layers One Two or more

Complexity Simple patterns Complex patterns

Learning Limited Powerful

ability

Example Single hidden layer Deep neural network

MLP

Applications

Deep feedforward networks are used in:

● Image classification
● Speech recognition
● Text classification
● Medical diagnosis
● Financial prediction

Convolutional Neural Network (CNN)

A Convolutional Neural Network (CNN) is a deep learning model designed to process
image and spatial data. It automatically learns features such as edges, shapes,
textures, and objects from images.

CNNs are widely used in:

● Image classification
● Face recognition
● Object detection
● Medical imaging

2. Why CNN is Needed

Problem with Traditional Neural Networks (MLP)

Traditional models flatten images into vectors.

Example:

28×28 image→784 values

This causes:

1. Loss of shape information

2. Loss of local patterns
3. Loss of spatial relationships
4. Huge number of parameters
5. Need to relearn objects at different positions

CNN solves these problems using:

● Local connections
● Shared weights
● Feature extraction layers

3. Basic CNN Architecture

A typical CNN contains:

1. Convolutional layer
2. Activation function (ReLU)
3. Pooling layer
4. Fully connected layer
5. Output layer
CNN Architecture Overview
4

4. Main Components of CNN

4.1 Convolutional Layer

The convolutional layer is the core building block of a CNN.

It uses filters (kernels) that slide over the input image.

Working

1. A small filter moves across the image.

2. Element-wise multiplication is performed.
3. Results are summed.
4. A feature map is produced.

Purpose

Detects:

● Edges
● Corners
● Textures
● Patterns
4.2 Activation Function (ReLU)

After convolution, a nonlinear function is applied.

Purpose

● Removes negative values.

● Introduces nonlinearity.
● Helps learn complex patterns.

4.3 Pooling Layer

Pooling reduces the size of feature maps.

Common Types

● Max pooling
● Average pooling

Example

Input:

4×4

After 2×2 max pooling:

2×2

Benefits

● Reduces computation
● Reduces overfitting
● Provides translation invariance

4.4 Fully Connected Layer

● Similar to traditional neural networks.

● Receives features from convolution layers.
 ● Performs final classification.

4.5 Softmax Output Layer

● Converts outputs into probabilities.

● The class with highest probability is selected.

5. Working of CNN

1. Input image is provided.

2. Convolution layer extracts features.
3. ReLU activation is applied.
4. Pooling reduces size.
5. Steps repeat in deeper layers.
6. Feature maps are flattened.
7. Fully connected layer processes features.
8. Softmax layer produces final prediction.

6. Hierarchical Feature Learning

CNN learns features in stages.

Early Layers

● Edges
● Lines
● Simple textures

Middle Layers

● Object parts
● Shapes (eyes, wheels)

Deep Layers

● Complete objects
● Faces, cars, digits

7. Feature Map
A feature map is the output of a convolution layer.

It represents:

● What features were detected

● Where they were detected
●

8. Flattening in CNN

Purpose

● Connects convolution layers to fully connected layers.

● Used for final classification.
Important:

● Flattening does not learn anything.

● It only reshapes data.
What Problem Do CNNs Solve

9. Key Properties of CNN

1. Local connectivity
2. Weight sharing
3. Translation invariance
4. Automatic feature extraction

10. Advantages of CNN

● High accuracy in image tasks.

● Fewer parameters than fully connected networks.
● Learns features automatically.
● Works well with large datasets.

11. Limitations of CNN

● Requires large training data.

● High computational cost.
● Hard to interpret internal features.

12. Applications of CNN

● Image classification
● Face detection
● Object detection
● Medical image analysis
● Autonomous driving
● Handwritten digit recognition

13. CNN vs Traditional Neural Network

Feature Traditional CNN
NN

Input Flattened 2D image structure

vector

Connections Fully Local connections

connected

Parameters Very large Reduced (weight

sharing)

Feature Manual Automatic

extraction