Regularization for

Deep Learning:
Module 3
 Introduction
• A central problem in machine learning is how to make an algorithm that will
perform well not just on the training data, but also on new inputs. Many strategies
used in machine learning are explicitly designed to reduce the test error, possibly at
the expense of increased training error. These strategies are known collectively as
regularization.
• Regularization - “any modification we make to a learning algorithm that is
intended to reduce its generalization error but not its training error.”
• Regularization = techniques we use to make sure our model studies properly — not
memorizing, not being lazy.
 Introduction
• Regularization strategies –
• Constraints on ML models(adding restrictions on the parameter values)
• the objective function that can be thought of as corresponding to a soft constraint on
the parameter values.

• If chosen carefully, these extra constraints and penalties can lead to improved
performance on the test set.
 Introduction
• Bias -is the difference between the expected (or average) prediction of a model and
the true value we are trying to predict. High Bias – Underfitting – use regularization
techniques - reducing the strength of regularization or removing it altogether can
help to improve its performance.
• Variance – measures how much the predictions of a model fluctuate for different
training datasets. It reflects the model’s sensitivity to the specific data it is trained
on. High Variance – Overfitting – regularization.
• Regularization: Helps balance both.
 Regularization Strategies
•Weight penalties (L1, L2).
•Model: restricts parameter values, avoids “too large” weights.
•Dropout
•Model: randomly drops neurons so it doesn’t rely on a single path → learns more robustly.
•Data Augmentation
•Model: sees more variety, generalizes better.
•Early Stopping
•Model: stop training before it memorizes noise.
•Ensemble Methods
•Model: combines multiple learners → reduces error.
 Deep Learning

•Deep models often have millions of parameters. If left unchecked, they easily
memorize data.
•Regularization helps us train big models safely so they generalize well.
•Key idea: We don’t shrink the model size; we control it with discipline
(regularization).
Demo
 Parameter Norm Penalties
• Goal of regularization: reduce generalization error (test error) even if that sometimes increases
training error.
• One common way: add a penalty on the parameter vector to the original objective. If the original
loss is J(θ;X,y), the regularized objective is

• where Ω(θ) is a penalty (often a norm of parameters) and α≥0 controls the penalty strength.
• This is a soft constraint that discourages large parameter values; we tune α using validation data.
• where α ∈ [0 , ∞] α is a hyperparameter that weights the relative contribution of the norm penalty
term Ω, , relative to the standard objective function results J. Setting α to 0 in no regularization.
Larger values of α correspond to more regularization.
 Parameter Norm Penalties
•
 Parameter Norm Penalties
• Each weight specifies how two variables interact.
• Fitting the weight well requires observing both variables in a variety of conditions.
• Each bias controls only a single variable means that we do not induce too much
variance by leaving the biases unregularized.
• Also, regularizing the bias parameters can introduce a significant amount of
underfitting.
• We therefore use the vector w to indicate all of the weights that should be affected
by a norm penalty, while the vector θ denotes all of the parameters, including both
parameters. w and the unregularized parameters.
 2
L Parameter Regularization
 2
L Parameter Regularization
 2
L Parameter Regularization

• Substitute Ω(w)
 2
L Parameter Regularization
 2
L Parameter Regularization
 2
L Parameter Regularization
 2
L Parameter Regularization
• The real J(w)- the neural network’s training loss) is super complicated.
• But near the minimum, we don’t need the full messy shape.
• We just need a local picture, like zooming in with a magnifying glass.
• That local picture can be well-described by a quadratic (bowl-shaped) function.
• Because J(w) could be very nonlinear and messy, we don’t want the full function. We just
want a local, simple approximation near w⋆.
• Optimal weights without regularization w∗:
• w∗=argminwJ(w) means:
• Find the set of weights w that minimize the training loss (without any regularization).
 2
L Parameter Regularization
• A complicated loss function J(w)- training loss of a neural net.
• Pick a particular point w⋆ that is a local minimizer (i.e. the training algorithm
found it).
• A simple analytic description of J(w) near that point. The Taylor expansion
gives exactly that: a local approximation of the function by a polynomial in
w−w⋆.
 2
L Parameter Regularization
 2
L Parameter Regularization
 Why Dataset Augmentation?
•Problem: Models generalize better if they see more training data.
•Reality: We usually have limited data.
•Solution: Create fake but realistic data → add it to the training set. This is called dataset augmentation.
• In classification, we only care about predicting the correct label y.
• The input x (like an image, sound, or text) can change in many ways, but the label often stays the same.
• Example: A cat photo is still a cat even if you rotate it slightly or shift it a few pixels.
• So we can easily generate new (x,y) pairs by transforming x.
• For tasks like density estimation (modeling the probability distribution of data), it’s very hard to
generate new fake data unless you already know the distribution.
• So augmentation is mostly useful for classification.
 Data Augmentation
• Images are high-dimensional → they vary a lot (position, size, orientation, lighting, etc.).
• Many of these variations can be simulated:
• Translation: shift the image slightly.
• Rotation: tilt the image a few degrees.
• Scaling: zoom in or out.
• Even though CNNs (convolution + pooling) already give some invariance, augmentation still helps a lot.
• But be careful:
• Some transformations change the label.
• Example: In OCR (character recognition), flipping “b” horizontally makes it a “d”. Rotating “6” by 180°
makes it a “9”. These would confuse the model.
 Data Augmentation
• Other Domains (Speech & Noise Injection)
• Speech recognition: augmentation is also effective (e.g., time-shifting audio, adding
noise).
• Noise injection: adding random noise to inputs is like augmentation.
• Helps the model learn robustness.
• Example: denoising autoencoders are trained with noisy inputs but must predict the clean version.
• Noise can also be applied to hidden units inside the network, not just inputs → this acts like
augmentation at deeper levels.
• Dropout is a form of noise injection (randomly dropping neurons during training).
 Data Augmentation
• When comparing ML algorithms:
• It’s unfair to compare Algorithm A without augmentation vs. Algorithm B with heavy
augmentation.
• The improved performance may come from augmentation, not from the algorithm itself.
• General guideline:
• Generic operations (e.g., adding Gaussian noise) → considered part of the algorithm.
• Domain-specific operations (e.g., cropping random parts of an image) → considered
pre-processing, separate from the algorithm.
 OPTIMIZATION FOR
TRAINING DEEP MODELS
 Challenges in Neural Network Optimization

• Optimization in general is an extremely difficult task.

• Machine learning has avoided the difficulty of general optimization by
carefully designing the objective function and constraints to ensure that the
optimization problem is convex.
• When training neural networks, we must confront the general non-convex
case.
 Convex Optimization Problem
• A problem is convex if:
• The objective (loss) function is convex means its graph looks like a bowl (curves upwards).
• The feasible region (constraints) is also convex.
• Key property:
Any local minimum is also a global minimum.
So if gradient descent finds a low point, it’s guaranteed to be the best possible.
• Example:
• Function: f(x)=x2
• Graph: U-shaped parabola
• Global minimum at x=0.
• This is easy to optimize we just keep moving downhill, and we’ll always reach the bottom.
 Non-Convex Optimization Problem
• A problem is non-convex if the function has multiple valleys, peaks, or saddle points.
• The loss surface is “rugged” — not just one bowl but many.
Key property:
• Local minima may not be global minima.
• Optimization algorithm (like SGD) can get stuck in:
• Local minima (small valleys).
• Saddle points (flat areas where gradient = 0 but not minimum).
Example:
• Function: f(x)=x4−3x2+2
• Graph: W-shaped curve
• Multiple minima (not all equally good).
• This is much harder to optimize.
 Convex and Non-Convex Optimization
• Convex optimization →
easy, one global minimum
(linear regression, logistic
regression).
 Convex and Non-Convex Optimization
• Non-convex optimization
→ hard, many local minima
and saddle points (neural
networks).
 Challenges in Neural NetworkOptimization

• The most prominent challenges involved in optimization for

training in deep learning models are:
• Ill Conditioning
• Local Minima
• Plateaus, Saddle Points and Other Flat Regions
 Ill Conditioning Problem
• Some challenges arise even when optimizing convex functions.
• The most prominent is ill-conditioning of the Hessian matrix H . This is a
very general problem in most numerical optimization, convex or otherwise.
• The ill-conditioning problem is generally believed to be present in neural
network training problems. Ill-conditioning can manifest by causing SGD to
get “stuck” in the sense that even very small steps increase the cost function.
 Ill Conditioning Problem
• Usual gradient-descent update
• θnew=θ−εg with g=∇J(θ) θ – Weight, ε- Learning Rate,
• A second-order Taylor series approximation to the function f (x ) around the
current point :

• Where g is the gradient and H is the Hessian at x(0) . If we use a learning rate of ε ,
then the new point x will be given by x(0) - ε g. Substituting this into the above
equation
 Ill Conditioning Problem
• a second-order Taylor series expansion of the cost function predicts that a gradient
descent step of - ε g will add to the cost.

• Ill-conditioning of the gradient becomes a problem when exceeds

• To determine whether ill-conditioning is detrimental to a neural network training
task, one can monitor the squared gradient norm and the
 Ill Conditioning Problem
• In many cases, the gradient norm does not shrink significantly throughout learning,
but the term grows by more than an order of magnitude.
• The result is that learning becomes very slow despite the presence of a strong
gradient because the learning rate must be shrunk to compensate for even stronger
curvature.
• Newton’s method is an excellent tool for minimizing convex functions with
poorly conditioned Hessian matrices, but requires significant modification before it
can be applied to neural networks.
 Local Minima
• One of the most prominent features of a convex optimization problem is
that it can be reduced to the problem of finding a local minimum.
• Any local minimum is guaranteed to be a global minimum.
• Some convex functions have a flat region at the bottom rather than a single
global minimum point, but any point within such a flat region is an
acceptable solution.
• When optimizing a convex function, we know that we have reached a good
solution if we find a critical point of any kind.
 Local Minima
• A model is said to be identifiable if a sufficiently large training set can rule out all but one
setting of the model’s parameters.
• Models with latent variables are often not identifiable because we can obtain equivalent
models by exchanging latent variables with each other.
• For example, we could take a neural network and modify layer 1 by swapping the incoming
weight vector for unit I with the incoming weight vector for unit j’ then doing the same for
the outgoing weight vectors.
• For m layers with n units each, then there are n ! m ways of arranging the hidden units. This
kind of non-identifiability is known as weight space symmetry.
 Local Minima
• With non-convex functions, such as neural nets, it is possible to have many local minima.
• Neural networks and any models with multiple equivalently parametrized latent variables all
have multiple local minima because of the model identifiability problem.
• Latent variables are hidden or unobserved variables in a model — they are not directly
measurable from the data, but they influence the observed outcomes.
• Observed variables - inputs and outputs you can measure (like images, labels, etc.)
Latent variables -internal “hidden” features or representations learned by the model.
• In a neural network: The hidden layer neurons and their weights act as latent variables.
• You don’t directly observe what each hidden neuron represents it could correspond to
patterns like “edges,” “textures,” or abstract features but the network discovers these
automatically during training.
 Local Minima

•Identifiability means: given enough data, the model’s parameters are uniquely
determined (only one parameter setting explains the data).
•Neural nets are often not identifiable because many different parameter
configurations implement the same input→output function. These equivalent
parameter settings give the same loss value, so each is a (possibly different) local
minimum in parameter space. Two common causes:
•Permutation symmetry (weight-space symmetry)
•Scaling symmetry (for ReLU / maxout etc.)
 Local Minima
•Permutation symmetry (weight-space symmetry)

•Hidden units in the same layer are often interchangeable.

•Example: if layer 1 has three neurons, you can permute (swap) their entire incoming weight
vectors and their outgoing vectors together — the network computes exactly the same function.
•With n identical units in a layer and m layers, there are roughly (n!)m equivalent parameter
reorderings. Each permutation corresponds to a different point in weight space, creating many
equivalent minima.
 Local Minima
• Scaling symmetry (for ReLU / maxout etc.)
•For a unit with linear+ReLU style behavior: scale incoming weights and bias by
factor α and scale outgoing weights by 1/α. The unit’s input→output mapping is
unchanged.
•If the loss depends only on model outputs (not directly on raw weight
magnitudes), then this scaling produces a continuum of equivalent parameter
settings. Geometrically, minima lie along curves or hyperbolas in parameter space
(not isolated points).
•Consequence: the parameter space contains many points (possibly uncountably
many) that represent the same function and the same loss value.
 Is Local Minima problematic?

•No — not the ones created by symmetry.

The equivalent minima arising from permutations or scalings all have the same
loss. They are different points in parameter space but equivalent in terms of the
network’s behavior. These are not the kind of bad minima we fear.
•The real problem would be many bad local minima — minima with
significantly higher loss than the global optimum. If an optimizer gets stuck in
such a high-loss basin, performance suffers.
 Tests to Check Local minima a problem
•Plot the norm of the gradient over time:
•If the gradient norm does not shrink to near zero, you are not stuck at a critical point (so local
minima/saddle/flatness aren’t the cause).
•If it does shrink to nearly zero but loss is still high, you may be at a saddle, plateau, or bad local
minimum.
•Plot training vs validation loss curves: different shapes point to overfitting, underfitting, or
optimization stall.
•Try multiple random restarts: if many random initializations yield similar low loss, bad local
minima are unlikely to be the cause.
•Visualize projected loss surface (if small model) or monitor Hessian information (trace, top
eigenvalues) to see curvature behavior.
Plateaus, Saddle Points and Other Flat Regions

• a saddle point is a point where the gradient is zero but the point is neither a pure
minimum nor a pure maximum.
• Some points around a saddle point have greater cost than the saddle point, while others
have a lower cost.
• At a saddle point, the Hessian matrix has both positive and negative eigenvalues(curvature
along that direction).
• Points lying along eigenvectors(direction) associated with positive eigenvalues have greater
cost than the saddle point, while points lying along negative eigenvalues have lower value.
• a saddle point as being a local minimum along one cross-section of the cost function and a
local maximum along another cross-section.
Plateaus, Saddle Points and Other Flat Regions

• in low dimensional spaces, local minima are common. In higher dimensional

spaces, local minima are rare and saddle points are more common.
• Math view: at a point x0 with gradient ∇f(x0)=0, examine the Hessian H=∇2f(x0).
• If all eigenvalues λi>0 → local minimum.
• If all eigenvalues λi<0 → local maximum.
• If some eigenvalues >0 and some <0 → saddle point.
Direction along eigenvector qi: second derivative ≈λi, so sign of λi says whether that
direction is up or down.
Plateaus, Saddle Points and Other Flat Regions

• saddle points matter more in high dimensions

• In 1D, a critical point with zero derivative is either min, max, or inflection — minima are
common.
• In high dimensions n, to be a local minimum you need all nnn curvature directions to
bend upward (all eigenvalues positive). If you think of each eigenvalue’s sign like a coin
flip, the probability that all are positive shrinks exponentially with n. So:
• Saddle points become exponentially more numerous than true minima as n grows.
• Consequence: for neural nets (huge nnn), most critical points at moderate/high loss are
saddle points, not local minima.
Plateaus, Saddle Points and Other Flat Regions

• low/high cost critical points occur

• Empirical/theoretical work shows low-loss critical points are more likely to be minima
(their Hessians tend to have positive eigenvalues).

• High-loss critical points are more often saddles or maxima.

• So: if you’re at a high-loss critical point, it’s probably a saddle (or max); low-loss critical points
are likelier to be useful minima.
 Plateaus, Saddle Points and Other Flat Regions
• Practical effect on optimization algorithms
• Gradient descent (first-order)
• GD and SGD follow −∇f. Near a saddle point the gradient can be very small → updates become tiny → training
slows.
• But stochastic noise (minibatch SGD) helps: the noise perturbs parameters and helps escape many saddles.
Empirically SGD often escapes saddles reasonably well.
• Newton’s method / second-order
• Newton tries to solve ∇f=0 by using Hessian inverse. If Hessian has negative eigenvalues at a saddle, Newton’s step
can move toward the saddle or make the algorithm unstable — Newton is attracted to any critical point if you
naively set the update to −H−1∇f That is why unmodified second-order methods can mistakenly converge to
saddles; they need modifications (e.g., saddle-free Newton) that avoid using negative curvature directions naively.
 Plateaus, Saddle Points and Other Flat Regions
• Other flat regions: plateaus and degenerate points
• Plateau / flat region: gradient ≈ 0 and Hessian ≈ 0 (or very small). No useful curvature. Optimizers get almost no
signal and stall.
• In convex problems, a flat region at bottom means many equivalent minima (all good). In nonconvex cases, a wide
flat region might be high-loss and useless.
• Degenerate points (zero gradient & Hessian) are worst — higher-order derivatives matter and numerical methods
struggle.
Plateaus, Saddle Points and Other Flat Regions

• Few regions are flat with constant value.

• In these locations, the gradient and also the Hessian are all zero.
• Such degenerate locations pose major problems for all numerical
optimization algorithms.
• In a convex problem, a wide, flat region must consist entirely of global
minima, but in a general optimization problem, such a region could
correspond to a high value of the objective function.