Module 2:

Training deep models

Introduction, setup and initialization- Kaiming, Xavier weight intializations, Vanishing and
exploding gradient problems, Optimization techniques - Gradient Descent (GD), Stochastic GD,
GD with momentum, GD with Nesterov momentum, AdaGrad, RMSProp, Adam.,
Regularization Techniques - L1 and L2 regularization, Early stopping, Dataset augmentation,
Parameter tying and sharing, Ensemble methods, Dropout, Batch normalization.
Introduction to Training Deep Models

Training deep models refers to the process of adjusting the internal weights of a neural network
so that it can learn patterns from data and make accurate predictions.

Challenges in training deep models include:

 Vanishing gradients
 Exploding gradients
 Slow convergence
 Overfitting or underfitting
 High computational cost

Goal: Train a model that learns effectively from data, generalizes to new data, and converges
efficiently.

Vanishing Gradient

The vanishing gradient problem is a challenge that emerges during backpropagation when the
derivatives or slopes of the activation functions become progressively smaller as we move
backward through the layers of a neural network. This phenomenon is particularly prominent in
deep networks with many layers, hindering the effective training of the model. The weight
updates becomes extremely tiny, or even exponentially small, it can significantly prolong the
training time, and in the worst-case scenario, it can halt the training process altogether.
During backpropagation, the gradients propagate back through the layers of the network, they
decrease significantly. This means that as they leave the output layer and return to the input
layer, the gradients become progressively smaller. As a result, the weights associated with the
initial levels, which accommodate these small gradients, are updated little or not at each iteration
of the optimization process.
The vanishing gradient problem is particularly associated with the sigmoid and hyperbolic
tangent (tanh) activation functions because their derivatives fall within the range of 0 to 0.25 and
0 to 1, respectively. Consequently, extreme weights becomes very small, causing the updated
weights to closely resemble the original ones. This persistence of small updates contributes to the
vanishing gradient issue.
The sigmoid and tanh functions limit the input values to the ranges [0,1] and [-1,1], so that they
saturate at 0 or 1 for sigmoid and -1 or 1 for Tanh. The derivatives at points becomes zero as
they are moving. In these regions, especially when inputs are very small or large, the gradients
are very close to zero. While this may not be a major concern in shallow networks with a few
layers, it is a more pronounced issue in deep networks. When the inputs fall in saturated regions,
the gradients approach zero, resulting in little update to the weights of the previous layer. In
simple networks this does not pose much of a problem, but as more layers are added, these small
gradients, which multiply between layers, decay significantly and consequently the first layer
tears very slowly , and hinders overall model performance and can lead to convergence failure.

Exploding Gradient

The exploding gradient problem is a challenge encountered during training deep neural
networks. It occurs when the gradients of the network's loss function with respect to the
weights (parameters) become excessively large.
The issue of exploding gradients arises when, during backpropagation, the derivatives or
slopes of the neural network's layers grow progressively larger as we move backward. This is
essentially the opposite of the vanishing gradient problem.
The root cause of this problem lies in the weights of the network, rather than the choice of
activation function. High weight values lead to correspondingly high derivatives, causing
significant deviations in new weight values from the previous ones. As a result, the gradient
fails to converge and can lead to the network oscillating around local minima, making it
challenging to reach the global minimum point.
In summary, exploding gradients occur when weight values lead to excessively large
derivatives, making convergence difficult and potentially preventing the neural network from
effectively learning and optimizing its parameters.

The update for the weights during back propagation in a neural network is given by:

The exploding gradient problem occurs when the gradients become very large during back
propagation. This is often the result of gradients greater than 1, leading to a rapid increase in
values as you propagate them backward through the layers.
Slow Convergence

In deep learning, training a model involves minimizing a loss function by adjusting the weights
through optimization algorithms such as gradient descent. Slow convergence refers to a situation
where this optimization process progresses very gradually.

Following are the results of the slow convergence:

 the model takes a large number of epochs (iterations over the dataset) to achieve a
desirable performance.
 The loss function decreases very slowly over time.
 The model shows minimal improvement in accuracy or performance metrics.
 Training takes a long time to reach an acceptable level of performance.

Causes of Slow Convergence

1. Very Small Learning Rate:

 A small learning rate means that the weight updates are tiny.
 The model requires many more epochs to make significant progress.

2. Poor Weight Initialization:

 If weights are not initialized properly, the model may start in regions where gradients are
too small or ineffective.
 This leads to inefficient learning in the early phases of training.

3. Vanishing Gradients:

 In deep networks, gradients can become very small as they are backpropagated through
many layers.
 This reduces the magnitude of weight updates, especially in earlier layers.

4. Improper Activation Functions:

 Use of activation functions like sigmoid or tanh in deep networks can saturate and lead to
slow learning.

5. Unoptimized Network Architecture:

 Excessively deep or wide networks without proper regularization can learn slowly.
 Overfitting

When a model performs very well for training data but has poor performance with test
data (new data), it is known as overfitting. In this case, the machine learning model learns the
details and noise in the training data such that it negatively affects the performance of the model
on test data. Overfitting can happen due to low bias and high variance.

Reasons for Overfitting

 Data used for training is not cleaned and contains noise (garbage values) in it
 The model has a high variance
 The size of the training dataset used is not enough
 The model is too complex
Ways to Tackle Overfitting

 Using K-fold cross-validation

 Using Regularization techniques such as Lasso and Ridge
 Training model with sufficient data
 Adopting ensembling techniques
 Underfitting
When a model has not learned the patterns in the training data well and is unable to generalize
well on the new data, it is known as underfitting. An underfit model has poor performance on the
training data and will result in unreliable predictions. Underfitting occurs due to high bias and
low variance.

Reasons for Underfitting

 Data used for training is not cleaned and contains noise (garbage values) in it
 The model has a high bias
 The size of the training dataset used is not enough
 The model is too simple
Ways to Tackle Underfitting

 Increase the number of features in the dataset

 Increase model complexity
 Reduce noise in the data
 Increase the duration of training the data
Setup and Weight Initialization
Setup refers to the preparation phase before training a neural network. This includes configuring
the data pipeline, choosing the right model architecture, loss function, optimizer, and training
strategy.

Typical components include:

 Data Pipeline: Load, preprocess, and normalize data

 Model Architecture: Define layers, activations, etc.
 Loss Function: Measures prediction error
 Optimizer: Algorithm to update weights (e.g., SGD, Adam)
 Metrics: Used to monitor training performance
 Batching & Epochs: Decide batch size and number of epochs

Weight Initialization

Initialization refers to assigning initial values to the weights of the neural network before training
begins. Proper initialization helps prevent vanishing or exploding gradients and speeds up
convergence. While building and training neural networks, it is crucial to initialize the weights
appropriately to ensure a model with high accuracy. If the weights are not correctly initialized, it
may give rise to the Vanishing Gradient problem or the Exploding Gradient problem. Hence,
selecting an appropriate weight initialization strategy is critical when training DL models

Importance of Weight Initialization

1. Vanishing and Exploding Gradient: Problem of vanishing gradients occurs when gradients
during training become extremely small causing the network to learn very slowly or not at all
particularly in deep networks. On the other hand the problem of exploding gradients happens
when gradients become extremely large, leading to unstable and ineffective training often
causing the model to diverge. Both issues can hinder the successful training of deep neural
networks.
2. Problem of Overfitting: Neural networks particularly deep ones have a high capacity to learn
complex patterns from data. However this capacity also makes them prone to overfitting.
Weight initialization indirectly helps tackle overfitting by ensuring that the neural network
starts training with well-scaled weights which prevents issues like vanishing gradients and
neuron saturation.
3. Saturation: Saturation of activation functions refers to a situation where the output of an
activation function becomes extremely close to its minimum or maximum value for a wide
range of inputs. In this state activation function becomes insensitive to changes in its input
 and its gradient approaches zero. By setting the initial weights appropriately weight
initialization helps keep activations in a balanced range preventing saturation and associated
gradient problems.

Common initialization methods:

 - Xavier (Glorot) Initialization: Best for sigmoid/tanh activations
 - He (Kaiming) Initialization: Best for ReLU activations

Xavier (Glorot) Initialization

Xavier initialization is a technique used to initialize the weights of neural network which solves
the problem of vanishing and exploding gradients which can hinder the training of deep neural
networks. The technique involves initializing the weights of neural networks by selecting
random values from a uniform distribution that ranges between -1 and 1. After this random
initialization, the weights are then scaled down by a factor of 1 divided by the square root of the
number of input units (denoted as 'n').
It aims to address the issue of maintaining variance in the forward and backward passes of a
neural network, specifically when using certain activation functions like the hyperbolic tangent
(tanh) and the logistic sigmoid. Regardless of how many input connections a neuron in a layer
has, the variance of its output should be roughly the same. This property helps to prevent the
vanishing or exploding gradient problem which can occur if the variances change drastically
between layers. Similarly the variance of the gradients during backpropagation should also be
roughly constant regardless of the number of neurons in the subsequent layer. This helps in
maintaining stable training dynamics.

Uniform Xavier Initialization:

We can initialize the weights by drawing them from a random uniform distribution within a
specific range which is determined by the formula:

 x is calculated using above formula.

 ninputs : Number of Input in the input layer
 noutput : Number of Output in the Output layer

For each weight in network we draw a random value w from a uniform distribution in the range
[-x, x]

.
Normal Xavier Initialization:

This initialization sets the initial weights by drawing them from a gaussian distribution with a
mean of 0 and a specific standard deviation, which is determined by the formula:

 σ is calculated using the provided formula

 ninputs : Number of Input in the input layer
 noutput: Number of Output in the Output layer

For each weight in the network, draw a random value w from a normal distribution with mean 0
and standard deviation . Assign this random value as the initial weight for that connection. By
setting the standard deviation based on the number of inputs and outputs, it adjusts the scale of
the weights in a way that keeps the network's activations within a reasonable range, regardless of
the layer size
Choice between Normal Xavier initialization and Uniform Xavier initialization may depend on
the specific neural network architecture and the activation functions used.

Kaiming Initialization

Kaiming Initialization, also known as Kaiming He initialization or He normal initialization, is a

weight initialization technique in deep learning that adjusts the initial weights of neural network
layers to facilitate efficient training by addressing the vanishing or exploding gradient problem.
In deep networks, especially those using rectified linear unit (ReLU) activation functions,
traditional weight initialization methods such as random normal or Xavier initialization may lead
to gradients that vanish or explode as they are propagated through the layers during
backpropagation.

Kaiming Normal Initialization

The Kaiming initialization method is calculated as a random number with a Gaussian probability
distribution (G) with a mean of 0.0 and a standard deviation of sqrt(2/n), where is the number
of inputs to the node. The factor of 2 in the variance is specific to the ReLU activation function

Kaiming Uniform Initialization

Optimization Techniques

Optimization techniques in Deep Learning are methods used to update the parameters (weights
and biases) of a neural network in order to minimize the loss function and improve model
performance. Since training deep networks involves a very high-dimensional and non-convex
optimization problem, choosing the right optimization method is crucial. In machine learning,
optimizers and loss functions are two fundamental components that help improve a model’s
performance.
 A loss function evaluates a model's effectiveness by computing the difference between
expected and actual outputs. Common loss functions include log loss, hinge loss and mean
square loss.
 An optimizer improves the model by adjusting its parameters (weights and biases) to
minimize the loss function value. Examples include RMSProp, ADAM and SGD (Stochastic
Gradient Descent).
Gradient Descent
Gradient Descent is a popular optimization method for training machine learning models. It
works by iteratively adjusting the model parameters in the direction that minimizes the loss
function.

Gradient Descent

Key Steps in Gradient Descent

1. Initialize parameters: Randomly initialize the model parameters.
2. Compute the gradient: Calculate the gradient (derivative) of the loss function with respect
to the parameters.
3. Update parameters: Adjust the parameters by moving in the opposite direction of the
gradient, scaled by the learning rate.

Formula :
Variants of Gradient Descent

1. Stochastic Gradient Descent (SGD)

Stochastic Gradient Descent (SGD) updates the model parameters after each training
example, making it more efficient for large datasets compared to traditional Gradient Descent,
which uses the entire dataset for each update.

Stochastic Gradient Descent

Steps:
1. Select a training example.
2. Compute the gradient of the loss function.
3. Update the model parameters.
Advantages: Requires less memory and may find new minima.
Disadvantages: Noisier, requiring more iterations to converge.

2. Mini Batch Gradient Descent

Mini-batch gradient descent consists of a predetermined number of training examples, smaller

than the full dataset. This approach combines the advantages of the previously mentioned
variants.

Mini-Batch Gradient Descent

In one epoch, following the creation of fixed-size mini-batches, we execute the following
steps:
1. Select a mini-batch.
2. Compute the mean gradient of the mini-batch.
3. Apply the mean gradient obtained in step 2 to update the model's weights.
4. Repeat steps 1 to 2 for all the mini-batches that have been created.

Advantages: Requires medium amount of memory and less time required to converge when
compared to SGD

Disadvantage: May get stuck at local minima

3. SGD with Momentum

Momentum-Based Gradient Descent

Momentum helps accelerate convergence by smoothing out the noisy gradients of SGD, thus
reducing fluctuations and improving the speed of convergence.

Then, the model parameters are updated using:

Advantages: Mitigates oscillations, reduces variance and faster convergence.

Disadvantages: Requires tuning the momentum coefficient β.

Advanced Optimizers

1. AdaGrad (Adaptive Gradient)

AdaGrad adapts the learning rate for each parameter based on the historical gradient
information. The learning rate decreases over time, making AdaGrad effective for sparse
features.

Advantages: Adapts the learning rate, improving training efficiency.

Disadvantages: Learning rate decays too quickly, causing slow convergence.

2. RMSProp (Root Mean Square Propogation)

RMSProp improves upon AdaGrad by introducing a decay factor to prevent the learning rate
from decreasing too rapidly.
Advantages: Prevents excessive decay of learning rates.

Disadvantages: Computationally expensive due to the additional parameter.

3. Adam (Adaptive Moment Estimation)

Adam combines the advantages of Momentum and RMSProp. It uses both the first moment
(mean) and second moment (variance) of gradients to adapt the learning rate for each
parameter.

Advantages: Fast convergence.

Disadvantages: Requires significant memory due to the need to store first and second moment
estimates.
Comparison of Optimizers

Each optimizer has its own strengths and weaknesses. The choice of optimizer depends on the
specific problem, dataset characteristics and the computational resources available. Adam is
often the default choice due to its robust performance, but each situation may call for a
different optimizer to achieve optimal results

Hyperparameter tuning
Hyperparameter tuning is the practice of identifying and selecting the optimal hyperparameters
for use in training a machine learning model. When performed correctly, hyperparameter tuning
minimizes the loss function of a machine learning model, which means that the model
performance is trained to be as accurate as possible.
Hyperparameter tuning is an experimental practice, with each iteration testing different
hyperparameter values until the best ones are identified. This process is critical to the
performance of the model as hyperparameters govern its learning process. The amount of
neurons in a neural network, a generative AI model’s learning rate and a support vector
machine’s kernel size are all examples of hyperparameters.
Good hyperparameter tuning means a stronger performance overall from the machine learning
model according to the metrics for its intended task. This is why hyperparameter tuning is also
known as hyperparameter optimization.

Neural network hyperparameters

Neural networks take inspiration from the human brain and are composed of interconnected
nodes that send signals to one another. In general, here are some of the most common
hyperparameters for neural network model training:

Learning rate
Learning rate sets the speed at which a model adjusts its parameters in each iteration. These
adjustments are known as steps. A high learning rate means that a model will adjust more
quickly, but at the risk of unstable performance and data drift. Meanwhile, while a low learning
rate is more time-consuming and requires more data, it also makes it more likely that data
scientists will pinpoint a model’s minimum loss. Gradient descent optimization is an example of
a training metric requiring a set learning rate.

Learning rate decay

Learning rate decay sets the rate at which the learning rate of a network drops over time,
allowing the model to learn more quickly. An algorithm's training progression from its initial
activation to ideal performance is known as convergence.

Batch size
Batch size sets the amount of samples the model will compute before updating its parameters. It
has a significant effect on both compute efficiency and accuracy of the training process. On its
own, a higher batch size weakens overall performance, but adjusting the learning rate along with
batch size can mitigate this loss.
Number of hidden layers
The number of hidden layers in a neural network determines its depth, which affects its
complexity and learning ability. Fewer layers make for a simpler and faster model, but more
layers—such as with deep learning networks—lead to better classification of input data.
Identifying the optimal hyperparameter value here from all the possible combinations is all about
a tradeoff between speed with accuracy.
Number of nodes or neurons per layer
The number of nodes or neurons per layer sets the width of the model. The more nodes or
neurons per layer, the greater the breadth of the model and the better able it is to depict complex
relationships between data points.

Momentum
Momentum is the degree to which models update parameters in the same direction as previous
iterations, rather than reversing course. Most data scientists begin with a lower hyperparameter
value for momentum and then tweak upwards as needed to keep the model on course as it takes
in training data.

Epochs
Epochs is a hyperparameter that sets the amount of times that a model is exposed to its entire
training dataset during the training process. Greater exposure can lead to improved performance
but runs the risk of overfitting.

Activation function
Activation function introduces nonlinearity into a model, allowing it to handle more complex
datasets. Nonlinear models can generalize and adapt to a greater variety of data.

Regularization in Deep Learning

Regularization in deep learning refers to a set of techniques used to prevent overfitting and
improve a model’s ability to generalize to new, unseen data. Deep neural networks have millions
of parameters, which makes them prone to memorizing the training data rather than learning
general patterns. Regularization methods help control this by constraining the model, modifying
the training process, or increasing the diversity of training examples.

L1 regularization adds a penalty equal to the sum of the absolute values of the weights to the
loss function. This encourages sparsity by making some weights exactly zero, which effectively
performs feature selection. When applied to linear regression, it is called Lasso Regression.

L2 regularization, also called Ridge Regression or weight decay, adds a penalty equal to the
sum of the squares of the weights. It shrinks weights toward zero without setting them exactly to
zero, keeping all features but reducing their influence. This makes the model smoother and more
stable.
Dropout is a technique where random neurons are temporarily ignored during training. This
forces the network to distribute learned features across many neurons, improving robustness and
reducing reliance on any single neuron.

Early stopping monitors the validation error during training and stops the process once the
validation error starts increasing, preventing the network from overfitting the training data.

Dataset augmentation artificially increases the size and diversity of the training set by
modifying existing examples, such as rotating or flipping images, adding noise to audio, or
paraphrasing text. This exposes the network to more variations, improving generalization.

Parameter tying and sharing reduce model complexity by forcing certain weights to be reused
or equal, as in convolutional layers where the same filter is applied across an image, or in
recurrent networks where the same weights are applied at each time step.

Batch normalization normalizes the inputs to each layer across a mini-batch to have zero mean
and unit variance, stabilizing training, allowing higher learning rates, and providing a mild
regularizing effect.

Ensemble methods combine multiple models to make predictions, which reduces variance and
improves accuracy. Examples include bagging, boosting, and stacking, where the collective
decision of multiple models is more reliable than a single model.

L1 Regularization (Lasso)
L1 regularization is a technique used to prevent overfitting by adding a penalty equal to the sum
of the absolute values of the model’s weights to the loss function. The modified loss function
becomes:

Here, λ is the regularization parameter that controls the strength of the penalty. Because the
penalty depends on the absolute values, the optimization process tends to push some weights
completely to zero. This makes the model sparse, meaning that it automatically selects the most
important features and ignores less relevant ones.

For example, in a regression problem with ten features, L1 regularization may shrink the
coefficients of less useful features to exactly zero, leaving only three or four active features in
the final model. This property makes L1 regularization useful for feature selection in high-
dimensional data.

In linear regression, when L1 regularization is applied, the method is known as Lasso

Regression (Least Absolute Shrinkage and Selection Operator).
L2 Regularization (Ridge / Weight Decay)
L2 regularization is a technique used in machine learning and deep learning to reduce overfitting
by adding a penalty equal to the sum of the squares of the model’s weights to the loss function.
The modified loss function becomes:

Here, λ is the regularization parameter that decides how strong the penalty is. Because the
penalty depends on the squares of the weights, the optimization process discourages large weight
values and pushes them to be small. However, unlike L1 regularization, L2 does not make
weights exactly zero, so all features are retained, but their influence is reduced.

For example, if a model has weights w=[4,−3,0.5], applying L2 regularization with λ>0 will
shrink them toward zero, such as [3.6,−2.7,0.4], but none of them will vanish completely. This
helps the model become smoother and more stable.

When L2 regularization is applied to linear regression, the method is called Ridge Regression.
In deep learning, the same idea is commonly known as weight decay, because the weights
“decay” a little at every update step.

Early Stopping

Early stopping is a regularization technique used in deep learning to prevent overfitting by

monitoring the model’s performance on a validation dataset during training. As training
progresses, both training error and validation error are recorded. Initially, both errors decrease,
but after a certain point, the validation error starts to increase even though the training error
continues to decrease. This indicates that the model has begun to memorize the training data
instead of learning general patterns.

With early stopping, training is stopped at the point where the validation error is the lowest. This
ensures that the model has learned enough from the data to generalize well but has not overfitted.
For example, if a neural network is trained for 100 epochs, the training error might keep
decreasing, but the validation error may start rising after 40 epochs. In this case, training would
be stopped at 40 epochs instead of running all 100.

Dataset Augmentation

Dataset augmentation is a regularization technique in deep learning where the training dataset is
artificially expanded by creating modified versions of existing data. The idea is to expose the
model to a wider variety of examples so that it learns general patterns instead of memorizing
specific details, thereby reducing overfitting.

In computer vision, dataset augmentation is very common. For example, images can be rotated,
flipped, cropped, zoomed, blurred, or have brightness and contrast adjusted. These modifications
produce new training samples that are still realistic but slightly different from the original.
Similarly, in natural language processing, text data can be augmented by replacing words with
synonyms, rephrasing sentences, or introducing small noise. In speech processing, audio data can
be augmented by changing pitch, speed, or adding background noise.

For example, if we have only 1,000 images of cats, we can create many more training samples by
flipping them horizontally, rotating them by 15°, or slightly changing the brightness. The model
then “sees” different versions of the same cat images, which improves its ability to recognize
cats in real-world conditions.

Parameter Tying and Sharing

Parameter tying and sharing are regularization techniques in deep learning that help reduce the
number of trainable parameters in a model, making it simpler and less prone to overfitting.

Parameter sharing means that the same set of parameters (weights) is reused across different
parts of the model instead of learning separate weights everywhere. A common example is in
Convolutional Neural Networks (CNNs), where the same filter (kernel) is applied across the
entire image. Instead of learning a separate weight for each pixel, the model learns one small
filter and slides it over the image, reusing the same weights at every location. This not only
reduces the number of parameters drastically but also makes the model invariant to translations
(shifts) in the input.

Parameter tying is when different parameters are explicitly constrained to be equal. For
example, in certain probabilistic models or in Recurrent Neural Networks (RNNs), the same
weights are used at every time step. This means that instead of having a new set of weights for
each time step, the model ties them together, ensuring that the same transformation is applied
repeatedly.

Ensemble Methods

Ensemble methods are regularization techniques in machine learning and deep learning where
multiple models are combined together to make predictions, instead of relying on a single model.
The idea is that by combining several “weak learners,” we can create a much stronger and more
accurate model. This reduces overfitting and improves generalization.

The most common ensemble techniques are:

  Bagging (Bootstrap Aggregating): Multiple models are trained on different random
subsets of the training data, and their predictions are averaged (for regression) or voted
(for classification). Example: Random Forests.
 Boosting: Models are trained sequentially, where each new model focuses more on the
mistakes of the previous ones. Example: AdaBoost, Gradient Boosting, XGBoost.
 Stacking: Multiple different models (e.g., decision trees, logistic regression, neural
networks) are trained, and another model (called a meta-learner) combines their outputs
for the final prediction.

For example, if three classifiers predict whether an email is spam and two say “spam” while one
says “not spam,” then using ensemble voting, the final decision will be “spam.”

Dropout

Dropout is a regularization technique used in deep learning to prevent overfitting by randomly

“dropping out” or ignoring a subset of neurons during each training iteration. This means that
during training, certain neurons and their connections are temporarily removed from the network,
forcing the model to not rely too heavily on any single neuron.

For example, if a layer has 100 neurons and the dropout rate is 0.5, then in each training step,
approximately 50 neurons are randomly ignored. The remaining neurons must learn to represent
the data in a more distributed and robust way. During testing or inference, all neurons are used,
but their outputs are scaled appropriately to account for dropout during training.

Dropout helps the network generalize better because it reduces inter-dependencies between
neurons and prevents the model from memorizing the training data. In simple terms, dropout is
like training multiple slightly different versions of the same network simultaneously, which
improves overall performance and reduces overfitting.

Batch Normalization

Batch normalization is a technique used in deep learning to stabilize and accelerate training by
normalizing the inputs to each layer of a neural network. During training, the inputs of each layer
are adjusted to have zero mean and unit variance across each mini-batch, which reduces
internal covariate shift—the change in the distribution of inputs to a layer as the network trains.

After normalization, the layer applies learnable scaling and shifting parameters so that the
network can still represent complex functions. Batch normalization not only allows the use of
higher learning rates but also acts as a regularizer, reducing overfitting in many cases.

For example, in an image classification network, batch normalization ensures that the activations
of each layer remain stable even as the weights change during training, making the optimization
faster and more reliable. In simple terms, batch normalization is like keeping the input to each
layer in a consistent range, so the network learns more efficiently and generalizes better.