Backpropagation in Neural Network

Backpropagation, short for Backwards Propagation of Errors, is a key algorithm

used to train neural networks by minimising the difference between predicted
and actual outputs. It works by propagating errors backwards through the
network, using the chain rule of calculus to compute gradients and then
iteratively updating the weights and biases.

Back Propagation plays a critical role in how neural networks improve over
time. Here's why:
1. Efficient Weight Update: It computes the gradient of the loss function
with respect to each weight using the chain rule, making it possible to
update weights efficiently.
2. Scalability: The Back Propagation algorithm scales well to networks
with multiple layers and complex architectures, making deep learning
feasible.
3. Automated Learning: With Back Propagation, the learning process
becomes automated, and the model can adjust itself to optimise its
performance.
 Working of the Back Propagation Algorithm
The Back Propagation algorithm involves two main steps: the Forward Pass
and the Backwards Pass.
1. Forward Pass Work
In the forward pass, the input data is fed into the input layer. These inputs,
combined with their respective weights, are passed to the hidden layers. For
example, in a network with two hidden layers (h1 and h2), the output from h1
serves as the input to h2. Before applying an activation function, a bias is
added to the weighted inputs.
Each hidden layer computes the weighted sum (`a`) of the inputs, then applies
an activation function to obtain the output (`o`). The output is passed to the
next layer, where an activation function converts the weighted outputs into
probabilities for classification.

2. Backward Pass
In the backward pass, the error (the difference between the predicted and actual
output) is propagated back through the network to adjust the weights and biases.
One common method for error calculation is the Mean Squared Error
(MSE) given by:
 MSE=¿
Once the error is calculated, the network adjusts weights using gradients, which
are computed with the chain rule. These gradients indicate how much each
weight and bias should be adjusted to minimise the error in the next iteration.
The backward pass continues layer by layer, ensuring that the network learns
and improves its performance. The activation function, through its derivative,
plays a crucial role in computing these gradients during Back Propagation.

Example of Back Propagation in Machine Learning

Let’s walk through an example of Back Propagation in machine learning.
Assume the neurons use the sigmoid activation function for the forward and
backward pass. The target output is 0.5, and the learning rate is 1.

Forward Propagation
1. Initial Calculation
The weighted sum at each node is calculated using:
aj=∑(wi , j∗xi)
Were,
 a j is the weighted sum of all the inputs and weights at each node

 w i , j represents the weights between the i thinput and the j th neuron

 x i represents the value of the i th input

O (output): After applying the activation function to a, we get the output of the
neuron:
o j = activation function(a j)

2. Sigmoid Function
The sigmoid function returns a value between 0 and 1, introducing non-linearity
into the model.
1
y j= −a
1+ e j

3. Computing Outputs
At h1 node
a 1 ¿(w1 ,1 x1 )+( w2 ,1 x 2)
¿ ¿ 0.21

Once we calculated the a1 value, we can now proceed to find the y3 value:
1
y j=F( a j)= −a
1+e 1
 1
y 3=F (0.21)= −0.21
1+e
y 3=0.56

Similarly find the values of y4 at h2 and y5 at O3

1
y 4 =F(0.315)= −0.315
1+ e
a 3=(w1 ,3∗y 3)+(w 2 ,3∗y 4)=(0.3∗0.57)+(0.9∗0.59)=0.702
1
y 5=F (0.702)= −0.702
=0.67
1+e

4. Error Calculation
Our actual output is 0.5, but we obtained 0.67. To calculate the error, we can
use the following formula:
Erro r j= y target − y 5

=> 0.5−0.67=−0.17

Using this error value, we will be backpropagating.

Back Propagation
1. Calculating Gradients
The change in each weight is calculated as:
Δ wij =η × δ j × O j

Where:
 δ j is the error term for each unit,

 η is the learning rate.

2. Output Unit Error

δ j=¿Oj (1-Oj) ( y target −O j )

For O3:
δ 5= y 5 (1− y 5)( y target − y 5 )

¿ 0.67 (1−0.67)(−0.17)=−0.0376

3. Hidden Unit Error

For h1:
δ 3= y 3 (1− y 3)(w1 ,3 ×δ 5 )

¿ 0.56 (1−0.56)(0.3 ×−0.0376)=−0.0027

For h2:
δ 4 = y 4 (1− y 4 )(w 2, 3 × δ 5 )

¿ 0.59(1−0.59)(0.9 ×−0.0376)=−0.0819

4. Weight Updates
For the weights from the hidden to the output layer:
Δ wij =η × δ j × O j

Δ w2 ,3 =1×(−0.0376)× 0.59=−0.022184

New weight: Change weight + Actual weight

w 2 ,3 ( new)=−0.022184+ 0.9=0.877816

For weights from input to hidden layer:

Δ w1 ,1 =1×(−0.0027)×0.35=0.000945
New weight:
w 1 ,1 (new)=0.000945+0.2=0.200945

Similarly, other weights are updated:

 w 1 ,2 (new)=0.273225
 w 1 ,3 (new )=0.086615
 w 2 ,1 (new)=0.269445
 w 2 ,2 (new)=0.18534

The updated weights are illustrated below

After updating the weights, the forward pass is repeated, hence giving:
 y 3=0.57

 y 4 =0.56

 y 5=0.61

Since y 5=0.61 is still not the target output. The process of calculating the error
and backpropagating continues until the desired output is reached.

This process demonstrates how Back Propagation iteratively updates weights by

minimising errors until the network accurately predicts the output.
Error= y target − y 5
 ¿ 0.5−0.61=−0.11

This process is said to be continued until the actual output is gained by the
neural network.

Backpropagation Applications
 Computer Vision: Backpropagation is fundamental to training
Convolutional Neural Networks (CNNs) used in applications such as
image recognition, object detection, facial recognition, and medical
image analysis (e.g., diagnosing diseases from X-rays and MRIs).
 Natural Language Processing (NLP): Neural networks trained with
backpropagation are used for tasks like sentiment analysis, language
translation (e.g., Google Translate), text generation (e.g., GPT models),
and speech recognition systems (e.g., voice assistants).
 Autonomous Systems: It is used to train the systems that enable self-
driving cars to sense obstacles, recognize traffic signs, and make real-
time decisions on the road.
 Recommendation Systems: Platforms like Netflix and Amazon use
neural networks trained with backpropagation to analyze user behavior
and provide personalized product or content recommendations,
continuously improving their accuracy based on user feedback.
 Medical Diagnosis: Neural networks are trained with backpropagation
on medical data to help detect diseases or other medical conditions,
assisting in diagnosis.
 Finance: Backpropagation is applied in financial systems for fraud
detection, credit scoring, and algorithmic trading models.
 Robotics: It is used in training robots to learn and adapt to their
environment.
 Bioinformatics: Backpropagation learning has been applied to analyze
gene expression and sequence annotation.
Key Variants by Approach
 Data Processing (Gradient Descent Types):
1. Batch Gradient Descent: Uses the entire dataset for stable, but slow,
updates.
Batch Gradient Descent (BGD) is an optimisation algorithm used in machine
learning to find the optimal parameters (weights and biases) of a model by
minimising the cost or loss function. It calculates the gradient using the entire
training dataset for each single parameter update, in contrast to variants like
Stochastic Gradient Descent (SGD) or Mini-Batch Gradient Descent.

How It Works
The process is iterative and typically follows these steps:
1. Initialisation: Model parameters (weights and biases) are initialised with
random values.
2. Gradient Calculation: The gradient of the loss function is computed for
all examples in the entire training dataset. The average error is used to
ensure a stable direction for the update.

4. Iteration: Steps 2 and 3 are repeated for a specified number of epochs

(passes through the entire dataset) until the loss function is minimized
and convergence is achieved.

2. Stochastic Gradient Descent (SGD): Updates after each example; fast but
noisy. Stochastic Gradient Descent (SGD) is a core machine learning
optimisation algorithm that trains models by iteratively updating parameters
using small batches (or individual samples) of data, making it much faster and
more memory-efficient than standard Gradient Descent (GD) for large datasets.
Instead of using the entire dataset for each update (like GD), SGD estimates the
gradient from random subsets, leading to faster, though noisier, convergence,
and it's crucial for training deep neural networks.

How it Works
1. Start: Begin with random model parameters (weights/biases).
2. Sample Data: Pick a small random subset (mini-batch) or single data
point.
3. Calculate Gradient: Compute the loss and its gradient using only that
small subset.
4. Update Parameters: Adjust model weights in the opposite direction of
the gradient (downhill).
5. Repeat: Continue iterating until the model converges.
Key Benefits
 Efficiency: Much faster for large datasets because it doesn't process
everything at once.
 Scalability: Handles massive datasets that wouldn't fit in memory.
 Online Learning: Adapts to streaming data in real-time.

3. Mini-Batch Gradient Descent: A balanced mix, using small data subsets

(mini-batches) for efficiency and stability. Mini-batch Gradient
Descent divides the dataset into smaller mini-batches, allowing for
parallel computation of gradients on modern hardware like GPUs. This
speeds up training by processing multiple data points simultaneously.

How Mini-Batch Gradient Descent Works

1. Splitting the Data: The training dataset is divided into smaller mini-
batches. Each mini-batch contains a subset of data points. For example, if
the dataset has 10,000 examples, we might split it into 100 mini-batches,
each containing 100 data points.
2. Computing the Gradient: For each mini-batch, the gradient of the loss
function is computed and used to update the model’s parameters. The loss
is averaged over the mini-batch, which helps in reducing the noise
compared to the SGD approach.
3. Updating the Parameters: Once the gradient is computed for a mini-
batch, the model’s parameters are updated using the learning rate and the
gradient. This step is repeated for each mini-batch, and the process
continues until all mini-batches have been processed.
4. Epochs: An epoch refers to one complete pass through the entire dataset.
After each epoch, the mini-batches are typically reshuffled to ensure that
the model does not overfit to the specific order of the data.

 Network Architecture:
o Static Backpropagation: For Feedforward Networks (FFNNs),
one-directional data flow for static problems (e.g., image
classification).
 o Recurrent Backpropagation / BPTT: For Recurrent Neural
Networks (RNNs), handles data sequences with feedback loops
(e.g., language).
 Optimisation Enhancements:
o Momentum: Adds "inertia" to speed up learning and escape
shallow local minima.
o Adaptive Learning Rates (AdaGrad, RMSprop,
Adam): Adjusts learning rates per parameter.
o Regularisation (Dropout, Weight Decay): Prevents overfitting.
o Batch Normalisation: Stabilizes training by normalising layer
inputs.