MODULE 2

REGULARIZATION FOR
DEEP LEARNING

-
 Definition
Regularization in deep learning refers to techniques used to
prevent overfitting and improve the generalization of a
model, especially when training deep neural networks.
Overfitting occurs when a model performs well on training
data but poorly on unseen (test) data, meaning it has
learned the details and noise in the training set rather than
the general patterns. Regularization helps the model focus
on learning useful patterns.
 PARAMETER NORM PENALTIES
Regularization has been used for decades, even before deep
learning, in models like linear regression and logistic regression.
Many regularization methods limit model capacity by adding a
parameter norm penalty to the objective function J

The regularized objective function is where α controls the

strength of regularization.
α = 0 means no regularization; larger α means stronger regularization.
Minimizing J reduces both the training error and the size of the
model parameters.
Different choices of norms (e.g., L1, L2) result in different
preferred solutions.
In neural networks, parameter norm penalties are usually applied only
to weights, not biases.
Biases need less data to fit and regularizing them can cause
underfitting.
w denotes the weights affected by regularization; θ denotes all
parameters, including biases.
It’s sometimes desirable to use different α values for each layer, but
using the same α (weight decay) for all layers reduces hyperparameter
search complexity.
Their are three diffrent kinds of Techniques under Regularization
L1 Regularization
L2 Regularization
Dropout Regularization
 L2 REGULARIZATION (WEIGHT DECAY)

Adds a penalty to the loss function based on the

squared values of the model's weights. This
encourages the model to keep the weights small,
reducing the model’s complexity and making it
less likely to overfit.
L2 parameter norm penalty, also known as weight decay, drives the
weights closer to the origin by adding a term
the objective function.
In other fields, L2 regularization is also called ridge regression or
Tikhonov regularization.
Weight decay modifies the learning rule to shrink the weight vector by
a constant factor on each step before the usual gradient update.
Over the entire training, L2 regularization reduces the weights along
directions that do not contribute much to reducing the objective
function.
The effect can be analyzed using a quadratic approximation of the
objective function around the minimum of the unregularized training
cost.
Regularization changes the solution from

Adding α to the diagonal of XᵀX makes the matrix invertible and

stabilizes the solution.
Important directions that significantly reduce the objective are
preserved, while unimportant directions are decayed.
 L1 REGULARIZATION
Similar to L2, but instead of squaring the weights, it adds a
penalty based on the absolute value of the weights.
It encourages sparsity, meaning some weights become
exactly zero, simplifying the model.
In this type of regularization, the absolute value of the
magnitude of coefficients or weights multiplied with a
regularizer term is added to the loss or cost function. It can
be represented with the following equation.
Another way to penalize model parameters by adding the sum of
absolute values of weights to the loss function.
Regularized objective function:

L1 gradient (sub-gradient) is constant with sign of each weight, unlike

L2 which scales linearly.
L1 tends to produce sparse solutions—some weights become exactly
zero.
For a diagonal Hessian in linear regression, the optimal weight under L1
regularization is:
L1 regularization results in sparse solutions, meaning many weights
become zero.
Sparsity allows feature selection, as zero-weighted features can be
discarded.
 Dataset Augmentation
Data Augmentation is a technique used in deep learning to
artificially increase the size and diversity of a training dataset by
applying controlled modifications to existing data.
This helps models generalize better to unseen data and reduces
overfitting, especially when the original dataset is small or lacks
variation.
Augmentation is easiest for classification tasks, where the model
needs to ignore transformations and still identify the correct class.
New training examples (x,y) can be created by transforming the
input x while keeping the label y the same.
Dataset augmentation has been very effective in object recognition,
where images have many variations that can be simulated.
Operations like translation, rotation, and scaling improve
generalization even if models already have some invariance
Augmentation is also useful for speech recognition tasks, not just
images.
Adding random noise to inputs or hidden units acts like
augmentation, improving robustness and enabling algorithms like
denoising autoencoders.
Dropout adds random noise by turning off some hidden units,
which creates slightly different inputs each time. This works like
data augmentation and helps the model learn better.
When comparing different algorithms, we must control
augmentation, because sometimes the improvement comes from
the extra data tricks, not from the algorithm itself.
 Noise Robustness
Noise robustness refers to the ability of a deep learning model to
perform well even when small random variations (noise) are added
to its inputs, hidden units, or weights. Adding noise acts as a form
of regularization, helping the model generalize better and avoid
overfitting.
Adding small noise to inputs can act like dataset augmentation,
improving generalization.
For some models, input noise with tiny variance is mathematically
equivalent to imposing a penalty on the weight norms.
Adding noise to hidden units is more powerful than just shrinking
parameters and is a major technique in regularization.
Dropout is a special case of noise injection applied to hidden units,
creating robust models.
 Adding noise to the weights, especially in recurrent networks,
reflects uncertainty and makes learning more flexible.
Weight noise makes the model less sensitive to small changes in
weights.
Training with weight noise adds a regularization term to the loss
function.
Noise pushes the model to find flat regions in the loss surface,
giving more stable solutions.
Noise robustness ensures that machine learning models remain stable
and perform reliably even when inputs, hidden units, or weights are
subject to small perturbations. By injecting noise strategically—through
input noise, hidden unit noise, dropout, or weight noise—models
achieve better generalization, avoid overfitting, and converge to stable
solutions that are resilient to variability in the data or parameters.
Regression cost function without noise:

With weight noise the perturbed model is

Objective function with noisy weights:

For small η, this is equivalent to:

This encourages parameters to move to regions where small

weight perturbations have little influence on the output.

In linear regression the regularization term collapses:

into which is a constant and does not affect optimization.
 Injecting Noise at the Output Targets
The noise injection is the process of adding random noise to output data
during the training process. Noise injection can be considered a form of
regularization, similar to techniques like dropout or L2 regularization.
Instead of modifying the network structure or weights directly, noise injection
introduces randomness into the input data or hidden layers.
Injecting noise at the output target means we don’t force the model to predict
the labels as 100% certain. Instead, we give a little uncertainty to account for
possible mistakes in the data.
The formal purpose of injecting noise at the output targets is to regularize the
learning process by accounting for label uncertainty, thereby preventing the
model from overfitting to potentially incorrect labels, stabilizing training, and
improving generalization performance on unseen data.
Injecting noise at the output targets makes the model robust to mistakes
in real-world labels. Instead of treating labels as perfectly correct, soft
targets introduce slight uncertainty. This prevents overconfidence,
stabilizes training, and avoids extreme predictions. As a result, the model
generalizes better to unseen data and performs more reliably.
 Semi-Supervised Learning
Semi-supervised learning is paradigm where both labeled and
unlabeled data are used to train a model.
In deep learning, SSL often focuses on learning a representation of the
data such that examples from the same class have similar
representations, making it easier for a classifier to separate classes.
SSL uses labeled examples from P(x,y) and unlabeled examples from
∣
P(x) to estimate P(y x)or predict y.
The goal is for examples from the same class to have similar
representations.
Unsupervised learning provides clues for grouping similar examples in
the representation space.
Unsupervised learning in SSL helps the model learn the structure of the
data from unlabeled examples, improving generalization and
representation quality.
 Models can share parameters between a generative model modeling
P(x) or P(x,y) and a discriminative model P(y ∣x).
The trade-off between supervised and generative objectives can
improve performance compared to using just one type of training.
Using unlabeled examples signif icantly improves the model’s
ability to estimate P(y ∣x) and generalize to new data.
Semi-supervised learning is a type of machine learning that uses both labeled
and unlabeled data. Labeled data tells the model the correct answers, while
unlabeled data helps the model understand the structure and patterns in the
data.
By combining both, the model can learn better, group similar examples
together, and make more accurate predictions on new, unseen data. It’s like
using a small amount of guidance (labeled data) along with a lot of clues
(unlabeled data) to learn more efficiently.
 Multi task learning
In deep learning, MTL refers to training a neural network to perform multiple tasks by
sharing some of the network's layers and parameters across tasks.
In MTL, the goal is to improve the generalization performance of the model by
leveraging the information shared across tasks.
There are different ways to implement MTL in deep learning, but the most common
approach is to use a shared feature extractor and multiple task-specific heads
The network first learns common features from all tasks, then each task has its own
separate output.
Another approach is to use a shared decision-making layer, where the decision-
making layer is shared across tasks, and the task-specific layers are connected to the
shared decision-making layer.
Each task processes its own information, then the network shares the final decision
layer
MTL can be useful in many applications such as natural language processing,
computer vision, and healthcare, where multiple tasks are related or have some
commonalities.
 Early Stopping
Early stopping is a regularization technique that stops model training when
overfitting signs appear. It prevents the model from performing well on the
training set but underperforming on unseen data i.e validation set
In large models, training error keeps decreasing but validation error eventually
starts to rise due to overfitting, and this behavior is very reliable
The best model is obtained at the time when the validation error is lowest, not at
the end of training
Training stops when no improvement in validation error is seen for a fixed number
of iterations, called patience.
Early stopping is the most commonly used regularization in deep learning because
it is simple and effective.
Validation error against training steps often forms a U-shaped curve, and early
stopping selects the point at the bottom.
It does not disturb the training process, loss function, or parameter values, making
it unobtrusive compared to other methods like weight decay.
Since a validation set is required, some data is held out, but after early stopping,
training can be done again on all the data for the optimal number of steps.
 Parameter Tying and Parameter Sharing
Parameter tying and sharing are techniques in neural networks that
introduce structured relationships between parameters
Parameter tying encourages parameters to be similar, whereas
parameter sharing enforces them to be identical and reused across the
network.
L2 regularization penalizes parameters for deviating from a reference
value, usually zero.
Dependencies between parameters can exist even when exact values
are unknown.
Norm-based penalties, such as the L2 norm, can enforce parameter
closeness.
Parameter tying has been applied between supervised and
unsupervised models to maintain similarity.
Parameter sharing enforces exact equality among parameters
instead of just similarity.
Sharing reduces memory requirements by maintaining only a
unique set of parameters.
Convolutional Neural Networks (CNNs) use parameter sharing by
applying the same filter across all spatial locations.
Parameter sharing allows larger networks without requiring
proportional increases in training data, effectively leveraging
domain knowledge.
 Sparse representation
Sparse representation is a method of representing data or neural network
activations such that most elements are zero or near-zero, and only a few carry
meaningful information.
It allows the model to focus on the most important features while ignoring
irrelevant or redundant data, effectively acting as an automatic feature
selector.
Sparse representation can occur in model parameters (parameter sparsity),
where many weights are zero, or in activations/hidden layers (representational
sparsity), where most units remain inactive for a given input.
Representational sparsity is typically achieved by adding a penalty to the loss
function that discourages non-zero activations. The regularized loss can be
written as:
Common penalties for inducing sparsity include L1 norm, KL divergence, or penalties
based on Student-t priors, all of which encourage most activations to become zero.
Some methods aim to regularize the average activation of units across multiple
examples, keeping it near a small target value, such as 0.01 per unit.
Hard constraints can also enforce sparsity, e.g., Orthogonal Matching Pursuit (OMP-k),
which limits the number of non-zero elements in the representation while minimizing
reconstruction error:

Sparse representations are effective feature extractors in deep architectures and can be
applied to any model with hidden units.
They reduce memory usage and computation, improve generalization, and make the
model more interpretable by highlighting essential features.
Sparse representations are widely used in images, text, audio, and neural networks,
especially when dealing with high-dimensional data, as they efficiently capture key
information while discarding irrelevant details.
 CHAPTER 2

OPTIMIZATION FOR TRAINING DEEP

MODELS
 What is Optimization?
Optimization is the process of finding the best solution to a problem from a
set of possible options, usually by maximizing or minimizing a specific
objective. In mathematics and machine learning, optimization typically
involves finding values for variables (parameters) that minimize or maximize
a function, known as the objective or cost function.
Optimization is used to adjust a model’s parameters to minimize the error
between its predictions and the actual outcomes. This process is critical for
training models, as it enables them to make more accurate predictions on
new data by minimizing the cost function.
Common optimization methods include gradient descent, where the model
adjusts its parameters step-by-step in the direction that reduces the cost
function, and advanced versions like stochastic gradient descent (SGD),
Adam, and RMSprop, which are particularly useful for training deep learning
models.
 How Learning Differs from Pure Optimization
Optimization algorithms used for training deep models differ from traditional
optimization algorithms in several ways.
In machine learning, we usually care about a performance measure P, defined
with respect to the test set.
P is often intractable, so we optimize it only indirectly.
Instead, we minimize a cost function J(θ) with the hope that doing so will
improve P.
This differs from pure optimization, where minimizing J itself is the main goal.
Optimization algorithms for deep models are specialized for the structure of
machine learning objective functions.
Typically, the cost function can be expressed as an average over the training
set.
In supervised learning, the objective is based on the per-example loss
L(f(x;θ),y)
 Empirical Risk Minimization
Empirical Risk Minimization is a fundamental principle in machine
learning where we choose a model that minimizes the average
error (or “risk”) on the training data
It addresses the fact that the true data distribution is unknown, so
optimization is performed on the available training samples.
The model parameters are adjusted to reduce the discrepancy
between predictions and actual outputs.
The central assumption is that minimizing training error will
generalize well to unseen data.
ERM underpins most optimization and learning techniques in deep
learning.
A key limitation is the risk of overfitting, where performance
degrades on new data despite low training error.
 Surrogate Loss Functions and Early Stopping
A surrogate loss is an alternative loss function used when the true loss (such
as 0-1 classification error) is difficult or intractable to optimize.
Minimizing expected 0-1 loss is typically intractable, even for simple
classifiers.
Therefore, surrogate loss functions act as proxies that are easier to optimize
but still guide the model toward reducing classification error.
The negative log-likelihood (cross-entropy loss) is a common surrogate for 0-
1 loss.
Surrogate loss allows the model to estimate conditional probabilities of
classes, which helps in indirectly minimizing classification error.
In many cases, surrogate loss functions can help the model learn more than
direct 0-1 loss minimization.
Even after training set 0-1 loss becomes zero, test set error can still decrease
when optimizing a surrogate loss.
This is because surrogate loss encourages robustness by pushing classes
further apart, making the classifier more confident.
Unlike pure optimization, training algorithms in machine learning do not
usually stop at a local minimum.
When the training set is very large, it is computationally expensive to
compute the sum over the entire set.
A simple and effective approach is to approximate the gradient using
only a subset of the data.
 Basic Algorithms
Stochastic Gradient Descent:
Stochastic gradient descent (SGD) and its variants are probably the most used
optimization algorithms for machine learning in general and for deep learning
in particular.

In Gradient Descent, we calculate the gradient (slope) of the loss function

using all the training data, then move the parameters a little in the opposite
direction of the slope. This can be slow when datasets are large.
In Stochastic Gradient Descent, instead of using the whole dataset, we
update the parameters using just one training example (or a small batch) at
a time. This makes the process much faster and allows the model to start
learning immediately, though the updates are noisier (less stable).
This algorithm outlines the steps for Stochastic Gradient Descent
(SGD), a popular optimization technique in machine learning. SGD
works by updating the model parameters in the direction of the
negative gradient of a subset of data, allowing for faster and more
efficient optimization, especially on large datasets.
 Momentum
Momentum is a method used in training models to make learning faster and
smoother. In normal gradient descent, the updates can jump around too much
and slow things down. With momentum, the model remembers part of its
previous steps and combines that with the current step
The method of momentum (Polyak, 1964) is designed to accelerate learning,
especially in the face of high curvature, small but consistent gradients, or noisy
gradients.
The momentum algorithm accumulates an exponentially decaying moving
average of past gradients and continues to move in their direction.
Formally, the momentum algorithm introduces a variable v that plays the role of
velocity, which is the direction and speed at which the parameters move
through parameter space.
The size of the step is largest when many successive gradients point in the same
direction, allowing momentum to accelerate learning.
Common values of α used in practice include 0.5, 0.9, and 0.99, with 0.9 meaning
steps are about 10× larger than plain gradient descent.
To ensure convergence, we add another force proportional to −v(t), called viscous
drag, which prevents oscillations and allows the particle to come to rest at a local
minimum.
Figure illustrates the behavior of momentum in optimization, especially when the loss surface
has poor conditioning (such as a narrow valley). The contour plot shows a quadratic loss
function, where the momentum method can help avoid oscillatory back-and-forth movements.
The red path in the figure demonstrates how momentum traverses the canyon-like valley
efficiently, while standard gradient descent would waste time oscillating across the narrow axis.
 Nesterov Momentum
Nesterov Momentum, also known as Nesterov Accelerated Gradient
(NAG), is an advanced optimization technique that refines classical
momentum by introducing a look-ahead mechanism. Instead of
computing the gradient at the current parameters, it evaluates the
gradient at the anticipated future position (θ + αv). This adjustment
reduces the risk of overshooting, enhances stability, and accelerates
convergence. NAG is particularly effective in scenarios with high
curvature, small but consistent gradients, or noisy updates. Typical
values of the momentum parameter α are 0.9 or 0.99, combined with an
appropriately tuned learning rate.
8.4 Parameter Initialization Strategies
 Algorithms with Adaptive Learning Rates
Learning rate difficulty:
The learning rate (how big or small the steps are when adjusting the model's
parameters) is critical to the performance of neural networks.
However, choosing the right learning rate is challenging because: Some parts
of the model's parameters are very sensitive (small adjustments work
better).Other parts are less sensitive (larger adjustments are okay).
Momentum helps, but adds complexity:
The momentum algorithm can reduce some of these challenges by helping the
optimization move faster in the right direction, but it introduces another
hyperparameter to tune, making things more complicated.
 AdaGrad
AdaGrad adjusts the learning rate for each parameter of the model
individually. It does this by taking into account how often and by
how much each parameter has been updated in the past.
Specifically:
Parameters that have been updated a lot (with large gradients) get a
smaller learning rate. This slows down the updates for these
parameters.
Parameters that have been updated less (with small gradients) keep
a larger learning rate. This allows these parameters to be updated
more significantly.