MODULE-2

Regularization for Deep Learning: Parameter Norm Penalties, Norm Penalties as Constrained
Optimization, Regularization and Under-Constrained Problems, Dataset Augmentation, Noise
Robustness, Semi- Supervised Learning, Multi-Task Learning, Early Stopping, Parameter Tying
and Parameter Sharing, Sparse Representations.

Optimization for Training Deep Models: How Learning Differs from Pure Optimization, Basic
Algorithms. Parameter Initialization Strategies, Algorithms with Adaptive Learning Rates.
REGULARIZATION FOR DEEP LEARNING
• It’s a method to prevent overfitting — when your model memorizes the training
data too well but fails to generalize to new data.
• Main idea is to limit the complexity of the model.
PARAMETER NORM PENALTIES
It is a technique used to prevent neural networks from becoming too complex or
overfitting the training data.
PARAMETERS: These are weights and biases associated with each connection
between neurons.
A model’s complexity refers to how “flexible” it is in fitting data.
• A high complexity model can wiggle a lot to fit even noisy training data (risk of
overfitting).
• A low complexity model is more rigid and captures only the main trend (risk of
underfitting).
How do weights relate to complexity?
• Large weights (w) → the model output changes a lot for small changes in input.
• Example: In linear regression y=w1x, if w1=100 even tiny noise in x makes
predictions jump drastically.
• Small weights → smoother predictions, less sensitivity to noise.
• Big weights = more complex model, small weights = simpler model.
 • Before regularization: J focuses only on fitting the data well.
• After regularization: J = fit the data well + keep parameters small to avoid
overfitting.

• Penalty term means how large the model’s weights (parameters)are and penalize
big values in the loss function.
Why measure the size of parameters?
• Large weights mean the model is very sensitive to small changes in input → it
can memorize noise in training data (overfitting).
• If we keep weights smaller, the model becomes simpler and more likely to
generalize well.
It’s like telling the model:
“Not only should you fit the data well, but also, don’t use huge numbers for your
weights. If you do, I’ll increase your loss.”
Why this helps:
• Large weights make models more complex and prone to overfitting.
• Adding the penalty discourages large weights → simpler models → better
generalization.
L² Parameter Regularization

• Objective function /Loss Function

• J(W)=it measures how well the model fits the data and also minimizes by
adjusting the weights W
Gradient=Slope
• If I change this weight a little, how much will the loss (error) go up or down?
Imagine climbing a hill
• You’re standing on the side of a hill.
• If you look around, you can walk in many directions.
• Some directions are almost flat → you won’t climb much.
• But one direction is the steepest uphill → that’s the direction where the hill rises
the fastest.
That direction = gradient.
In ML terms
• The “hill” = loss function (error).
• The steepest uphill = direction where the error increases fastest.
• But since we want to reduce error, we walk opposite to that direction (downhill).
Objective Function (Loss Function)
• In ML/DL, we have a loss function J(w).
• w = weights (the numbers our model learns).
• J(w) = how wrong the model is for those weights.
Goal: find weights that make J(w) as small as possible.
 • w∗ is the special weight value where the loss J(w) is the lowest possible.

Neighborhood Around w∗
• Near this minimum point, the curve looks like a U-shape (parabola).
• So instead of studying the full complicated curve, we can say:
“Close to the minimum, the loss function behaves like a simple quadratic
(squared) function.”
That’s why we do a quadratic approximation.
Why Quadratic Approximation?
• Because quadratic functions are easy to work with (they’re just like ax2+bx+c).
• Around w*, this approximation is good enough to analyze training.
Example:
If your real loss curve is wiggly, near the bottom it still looks like a smooth U-
shape.
simple terms:
Instead of dealing with a messy loss curve everywhere, we zoom in near the best
weights w*. There, the loss function looks like a parabola (quadratic). We use
that simpler quadratic shape to analyze and understand training.
Here’s the loss function plot:
• The blue curve is the loss J(w).
• The red dot at w∗=2 is the minimum point.
• This means: when the weight w is 2, the loss is lowest possible (best fit for data).
The phrase “quadratic approximation near w*” means:
Close to this minimum, the curve looks like a parabola (U-shape). So, instead of
studying the full complicated function, we approximate it with this nice parabola
around w*.
If the objective function is truly quadratic, as in the case of fitting a linear regression
model with mean squared error, then the approximation is perfect.
The approximation Ĵ is given by,
 Saying “H is positive semidefinite” is just a fancy way of saying:
Around the best weights w*, the loss function curves like a bowl (upward), so we’re
really at a minimum, not at a maximum or weird flat point.
• There is no first-order term in this quadratic approximation, because w∗ is defined
to be a minimum, where the gradient vanishes.
Whole derivation is just:
• Start with quadratic approx.
• Add weight decay penalty.
• Take derivative.
• Set gradient = 0.
• Rearrange → gives the formula.
Solid ellipses (the oval shapes)
• Think of these as the “hills” or “valleys” of your original loss function (without
regularization).
• The smallest ellipse (centered at 𝑤∗ (is the best place the model found if we don’t
use regularization.
• Moving away from 𝑤∗makes the loss larger, but not equally in all directions:
• Along 𝑤1)horizontal), the ellipses are stretched → the loss doesn’t change
much (flat).
• Along 𝑤2)vertical), the ellipses are narrow → the loss changes quickly
(steep).
Dotted circles (the round shapes)
• These represent the L² penalty (weight decay).
• The penalty cares only about the distance from the origin (0,0), not about the
shape of the ellipses.
• That’s why they are circles — all directions are treated equally by the regularizer.
• The “goal” of the regularizer is to pull weights toward zero (the origin).

Weight decay shrinks parameters more in directions where the loss function is flat
(low curvature), and shrinks less in directions where the loss is steep (high
curvature).
L1 REGULARIZATION

While L2 weight decay is the most common form of weight decay, there are other
ways to penalize the size of the model parameters. Another option is to use L1
regularization.
L1and L2
L2 Regularization (Weight Decay)
• Think of it like a teacher saying:
“Don’t give too much importance to any one factor. Spread out the importance
fairly.”
• The model will shrink all factors a bit, even the unimportant ones.
• So, every feature keeps some role, but weaker.
Effect: All features contribute, but with smaller influence.
L1 Regularization (Sparsity)
 • Think of it like a strict teacher saying:
“If a factor is not useful, completely ignore it!”
• The model will eliminate some features entirely (e.g., shirt color, pencils).
• Only the important ones (hours studied, sleep) stay.
Effect: The model automatically chooses the most useful features.

• L2 = soft shrinkage (everyone contributes, but less).

• L1 = hard selection (some features completely cut off).
•
Norm Penalties as Constrained Optimization
STEP2:Thinking as a constrained problem:
• Instead of just adding the penalty, we can think of it like this:
"Minimize the cost 𝐽, but only allow parameters 迿ÿwhose penalty Ω(迿ÿ)is less than
some constant 𝑘."
That’s a constraint.

Suppose you are picking a car.

• Your goal: minimize cost (J).
• But you also add a rule: the car’s fuel consumption (Ω) must be less than 10
liters/100km (k).
So now you’re not free to pick any car—you only pick cars that stay within the
constraint.
• Regularization can be viewed not just as “adding punishment” but also as “forcing
parameters to stay inside a safe boundary.”
STEP3:Lagrange function trick
Imagine you’re solving a problem like this:
• You want to minimize your cost (make the model error small).
• But you also have a rule/constraint: don’t let the parameters get too big (stay
inside the red circle we discussed).
Problem: Handling “rules” directly in optimization is hard.
The Lagrangian trick makes it easier:
Instead of keeping the rule separate, we add the rule into the cost function itself with
a penalty.
• If the model stays within the rule → small penalty (or none).
• If the model breaks the rule → big penalty is added.
• So Instead of solving two problems (minimize cost + check rule), we solve one
problem:
a new cost function = original cost + penalty.
• Example:
1. Your Goal (Cost Function):
You want to eat as much tasty food as possible in the hostel mess (this is like
minimizing error in ML).
2. Constraint:
The warden says: “You cannot spend more than ₹2000 per month.”
• Two Ways to Handle It:
• Constraint directly: You must plan meals so your monthly bill ≤ ₹2000.
• Lagrange Trick: Instead of forcing the rule, you change your goal to:
• Enjoyment of food +fine for breaking the rule
• So, if you spend more than ₹2000, the warden gives you a fine.
The fine increases as you spend more.
3. Why is this useful?
Now, you don’t need to check every time if you cross ₹2000.
The new “goal + fine” system automatically pushes you toward the best balance:
• Eat enough to be happy,
• But not too much, or else fines make it worse.
Dataset Augmentation
• The best way to make a machine learning model generalize better is to train it on
more data. Of course, in practice, the amount of data we have is limited. One way
to get around this problem is to create fake data and add it to the training set. For
some machine learning tasks, it is reasonably straightforward to create new fake
data.
• This approach is easiest for classification. A classifier needs to take a compli
cated, high dimensional input x and summarize it with a single category identity
y. This means that the main task facing a classifier is to be invariant to a wide
variety of transformations. We can generate new (x,y) pairs easily just by
transforming x the inputs in our training set
• This approach is not as readily applicable to many other tasks. For example, it is
difficult to generate new fake data for a density estimation task unless we have
already solved the density estimate on problem.
When we train a machine learning model, we usually want more data to make it learn
better. But often, we don’t have enough real data. So,
the trick is: create new fake data from the existing one — this is called dataset
augmentation.

Not every transformation is safe.

For example, in text recognition (like reading letters or numbers):
• If you flip ‘b’, it may look like ‘d’.
• If you flip ‘6’, it may look like ‘9’.
That would confuse the model instead of helping it.
NOISE ROBUSTNESS
Noise and regularization (controlling weights)
• If you add very small random noise to the inputs, it turns out to be mathematically
similar to adding a penalty on the weights of the model.
• This penalty prevents the weights from becoming too large, which is a form of
regularization (helps avoid overfitting).
• This idea was discussed by Bishop (1995).
Noise injection is more powerful
• Noise injection is not just about shrinking weights.
• Adding noise inside the hidden layers (not just inputs) can be even more
effective.
• It forces the network to be more robust and generalize better.
What is Dropout?
• Dropout is a regularization technique used in neural networks.
• During training, random neurons are “dropped out” (ignored/turned off) with
some probability (say 50%).
• This forces the network to not depend too heavily on any one neuron.

Dropout (special case of noise injection)

• Adding noise to hidden units is so important that it deserves its own discussion.
• Dropout is the main technique based on this idea.
• In dropout, random hidden units are "dropped" (set to zero) during training.
• This is like adding noise, and it prevents the network from relying too much
on specific neurons.
Noise added to weights
• Another way to regularize a model (prevent overfitting) is by adding noise
directly to the weights instead of inputs or hidden units.
• This was used in recurrent neural networks (RNNs).
This can be interpreted as a stochastic implementation of Bayesian inference over the
weights. The Bayesian treatment of learning would consider the model weights to
be uncertain and representable via a probability distribution that reflects this
uncertainty.

• Imagine you’re guessing a student’s math exam score.

• Based on practice tests, you think the student usually scores 80 marks.
1. Traditional (non-Bayesian) view
• You say: “Okay, the score is exactly 80.”
• Just one fixed number — no uncertainty.
2. Bayesian view
• You say: “I’m not 100% sure. Maybe the score will be between 75 and 85.”
• So instead of one number, you describe the score with a probability distribution:
 • Most likely around 80
• Some chance of 78, 82, etc.
Adding noise (practical way)
• Instead of always using “80,” you sometimes try 79, 82, 77, 83…
• Each time you add a little random noise to your guess.
This way, you’re not overconfident, and you’re capturing the uncertainty.
Injecting Noise at the Output Targets
The problem
• In real datasets, labels (y) are not always correct.
• If the model always tries to make predictions with 100% confidence in those
labels, it can overfit to mistakes.
• Example: A picture of a cat wrongly labeled as a dog → the model will learn
wrong things.
The solution → Add noise to labels
• Instead of saying “this label is 100% correct,” we say:
• With high probability (1 − ε) → the label is correct.
• With small probability (ε) → it could be one of the other labels.

Label smoothing = making labels a little uncertain (soft, not hard 0/1), so the model
doesn’t become overconfident and generalizes better.
Semi Supervised Learning
Representation ℎ = 𝑓(𝑥)
• In deep learning, instead of directly mapping input 𝑥 to output 𝑦, we often first
transform 𝑥 into a representation ℎ(a vector that captures the important
features).
• Example: An image of a cat → deep network learns ℎ (like "has fur", "has
whiskers", "has pointed ears").
GOAL: Learn [Same-class examples should have similar representations]
• If two inputs belong to the same class (say, two cat pictures), then their learned
representations ℎ should be close together.
• If one is a cat and another is a dog, their representations should be far apart.
Linear classifier on top of representation
• Once a good representation ℎ is learned, even a simple classifier (like
logistic regression or a linear SVM) can separate the classes effectively.
• Why?
• Because the hard work (grouping, clustering) has already been done in the
representation space.
 • P(x) = “what inputs look like” (unlabeled data).
• P(x,y) = “what inputs and labels look like together” (labeled data).

MULTI-TASK LEARNING
• Multi-task learning (introduced by Caruana, 1993) is a method where a single
model is trained to do multiple related tasks at the same time.
• This helps the model generalize better (perform well on new unseen data).
• Why? Because when tasks are related, sharing information between them acts like
a constraint that prevents the model from overfitting to just one task.
• Multi-task learning = one model, multiple tasks, better generalization.
Train one model for multiple related tasks.
When we build an MTL model, it usually has two types of parameters (parts):
Architecture of Multi-Task Learning (MTL) in a neural network.
1. Input (x):
• The same input is given for all tasks.
• Example: an image of a face.
2. Shared Layers (h(shared)):
• The first few layer's extract generic features (edges, colors, shapes, embeddings).
• These are common for all tasks.
• Example: features like eyes, nose, hair texture.
3. Task-Specific Layers:
• After the shared part, the network branches out into different “heads,” each
solving a specific task.
• Example:
• Task 1 head predicts age.
• Task 2 head predicts gender.
• Task 3 head predicts emotion.
4. Outputs (y¹, y², …):
• Each branch produces its own result.
• Even though tasks are different, they all benefit from the shared lower features.
EARLY STOPPING
• Early stopping is a regularization technique used to prevent overfitting while
training neural networks.
• The idea:
 • During training, the model’s training loss keeps decreasing (it memorizes the
training data).
• But the validation loss (performance on unseen data) first decreases, then starts
increasing (overfitting starts).
• Early stopping means: stop training when validation loss starts getting worse,
even if training loss is still improving.

Best Parameters Tracking

• During training, we don’t just stop and take the model from the last iteration.
• Instead, we keep a copy of the parameters (weights) whenever the validation error
improves.
That way, we always know which set of weights gave the lowest validation error.
Returning the Best Model
• Once training ends (because of early stopping), we return the best recorded
parameters, not the final parameters.
• This ensures we get the model with the best generalization performance.
• Patience (Pre-specified number of iterations)
• The text also explains that we don’t stop immediately when the validation error
fails to improve once.
• Instead, we wait for a pre-specified number of iterations/epochs (called patience).
• If there is no improvement in that patience window, training stops.
Early stopping isn’t just about when to stop; it’s also about which model
parameters to keep. We save the weights that performed best on the validation set
to ensure the final model generalizes well.
STEPT1
1.θ ← θ₀
• θ₀ are the initial parameters (weights) of the model before training begins.
• θ will keep changing during training as the model learns.
2.i ← 0
• i is the counter for the number of training steps (or epochs).
• Since we haven’t started training, it’s 0.
3.j ← 0
• j is the patience counter.
• It counts how many times the validation error got worse in a row.
• Starts at 0 because no bad epochs have happened yet.
4.v ← ∞
• v is the “best validation error so far.”
• At the start, we haven’t seen any error yet, so we set it to infinity (∞), meaning
anything we see will be better.
5. θ* ← θ
• θ* stores the best model parameters found so far.
• Initially, this is just the starting parameters.
6. i* ← i
• i* is the training step where the best parameters were found.
• At the start, that’s just 0.
Step 2: Training Loop (while j < p do …)
This step is the core of early stopping.
We keep training the model and checking validation error until the patience limit is
exceeded.
1. Update θ by running the training algorithm for n steps.
• We don’t check validation after every tiny update (that would be inefficient).
• Instead, we run training for n steps/epochs before checking again.
• After this update, θ changes (the model has learned more).
2. i ← i + n
• Increase the training step counter.
• If we trained for n=5 steps, and i was 0, now i = 5.

• Stopping Condition:
 • The loop continues as long as j < p.
• That means:
• If validation keeps improving, training continues.
• If validation stops improving for p times in a row, training stops.

Step 3: Output (after loop ends)

• When the patience counter j reaches p, the loop stops.
At this point:
• θ* = the parameters that gave the lowest validation error.
• i* = the step/epoch at which that lowest error occurred.
• These are returned as the final model, instead of the parameters from the very last
iteration.
Early stopping watches validation performance during training and keeps the best
model parameters seen so far (call that the checkpoint).
• To do that you must store a copy of those best parameters somewhere (memory
or disk). That storage requirement is the “additional cost” .
• Cost is usually negligible because you can save the best parameters in slower
storage (like host RAM or disk) instead of the fast-but-limited GPU memory.
• Early stopping needs a validation set — tradeoff.
• We must hold out some data (validation) to monitor when to stop. That means the
model trains on slightly less data during the early-stopping run.
• To make use of that held-out data (so nothing is wasted), we use an extra training
step after you finish early stopping, using all available labeled data.
The two ways to do that extra training:
 • Strategy A — Reinitialize and retrain on the full dataset
• Strategy B — Restore the best parameters and continue / fine-
tune on the full dataset.
STRATEGY A
• Split your labelled training data into subtrain and validation.
• Run early stopping on the subtrain (use the validation only to decide when to
stop). This gives you the best number of training steps i∗.
• Reinitialize the model weights (start fresh).
• Train the model on the entire training set (subtrain + validation) for i* steps.
Save that final model.
Strategy B — Continue training from θ* using all data
• After early stopping, keep the learned θ* (don’t reinitialize).
• Continue training but now include the whole training set.
• Stop when validation loss falls below the target value that was achieved at the end
of the early-stopping run (or use some other rule).
Pros
• Saves time (no full retrain from scratch).
 • Uses the good initialization already found.
Cons
• No guarantee it will ever reach the target value — may not terminate or may
diverge.
• Less principled; behavior depends on learning rate / optimization dynamics.

•
Start with your full training set and split it into subtrain and validation (just
like before).
• Run early stopping on the subtrain (monitoring validation) until it stops. Keep
the final parameters it produced; call them θ.
• Compute ε = the training objective (loss) evaluated on the subtrain at θ. (This is
the loss value where early stopping stopped.)
• Continue training the model from θ, but now train on the entire training set
(subtrain + validation).
• After each chunk of training (say every n steps), evaluate the .
• Stop when the validation loss falls to or below ε
• . In words: keep training until the validation error becomes as small (or smaller)
than the training loss you observed at the end of the first pass.
Sparse Representations:

• Sparse representations aim to express a signal, image, or dataset using a minimal

number of non-zero elements in a chosen basis or dictionary.
• Instead of representing data with all possible features, sparsity assumes that only
a few are necessary to capture the essence of the data.
• This is often achieved by transforming the data into a domain (e.g., frequency or
wavelet domain) where its representation is naturally sparse.
Definition: A representation is sparse if most of its coefficients are zero, and only a
few are non-zero.
• In sparse representation, we want to express a signal (like an image, sound, or
data vector) as a combination of a few “building blocks”.
These building blocks are called basis vectors, and the whole collection is called
a dictionary.
• Think of a dictionary in language: it has many words.
• Similarly, in math, a dictionary has many basis vectors.
• Any signal can be written as a “sentence” using a small number of these
vectors.
• The dictionary can be predefined (Fourier basis, wavelets) or learned from data
(like in dictionary learning, K-SVD).
Sparse in the context of neural networks
• In the context of neural networks, this idea extends to the activations of units in
a layer (i.e., the output of neurons, often denoted as h) rather than just the model
parameters (weights, denoted as θ). The goal is to make the representation (the
activation vector h) sparse,
The two types of sparsity:
• Parameter Sparsity: Achieved by penalizing the model parameters (e.g.,
weights) directly, often using an L1-norm penalty to drive many parameters to
zero.
• Representational Sparsity: Achieved by penalizing the activations of units in a
neural network, encouraging the representation h to have few non-zero elements.
Regularization for Representational Sparsity:
Representational sparsity is achieved by adding a penalty term to the loss function that
encourages the activation vector h to be sparse. The regularized loss function is:

• where: J(θ;X,y) is the standard loss (e.g., mean squared error for regression or
cross-entropy for classification).
• Ω(h) is a penalty on the representation h.
• α≥0 controls the strength of the regularization (larger α means more sparsity).
Common Penalties for Sparsity
L1-Norm Penalty

Hard Constraints for Sparsity:

Instead of a penalty, sparsity can be enforced with a hard constraint on the number of
non-zero entries in h h h. For example, Orthogonal Matching Pursuit (OMP)
solves:

• where: ∥h∥0 is the number of non-zero entries in h.

• k is the maximum number of non-zero entries allowed.
• W is an orthogonal dictionary (basis).
Why Use Representational Sparsity
Efficiency: Sparse activations reduce the number of active neurons, lowering
computational and memory requirements.
Robustness: Sparse representations focus on the most important features, making
models less sensitive to noise or irrelevant inputs.
Interpretability: Sparse activations highlight which units are most relevant for a
given input, aiding in understanding the network’s behavior.
Generalization: By preventing overfitting (similar to weight decay), sparsity
encourages simpler, more generalizable models.
Connection to Weight Decay
• The excerpt contrasts representational sparsity with weight decay, which directly
penalizes model parameters (e.g., weights θ) using an L2-norm (or sometimes L1

-norm) penalty:
• Weight decay shrinks parameters toward zero but doesn’t necessarily make them
exactly zero (unless using L1).
• Representational sparsity, by penalizing activations h , indirectly constrains the
parameters because the activations depend on the weights and inputs h = f(Wx +
b). This creates a more complex, data-dependent regularization effect compared
to weight decay.
Chapter8: Optimization for Training Deep Models
How Learning Differs from Pure Optimization, Basic Algorithms. Parameter
Initialization Strategies, Algorithms with Adaptive Learning Rates.
Chapter 8 (8.1,8.3,8.4,8.5)

What is Optimization in Deep Learning?

• Optimization refers to the process of adjusting the parameters (weights and
biases) of a neural network so that it can minimize a given objective function,
usually the loss function and maximize the accuracy.
Loss function measures how far the model’s predictions are from the true outputs.
Optimizer is the algorithm that updates the weights to reduce the loss.
Common optimization algorithms:
• Gradient Descent (GD) → basic method.
• Stochastic Gradient Descent (SGD) → updates with small batches for
efficiency.
• Momentum → remembers past gradients for faster convergence.
• Adam → combines momentum + adaptive learning rates.
• RMSProp → scales learning rate with moving average of squared gradients.

How Learning Differs from Pure Optimization

Traditional optimization:
• Optimization is about directly minimizing or maximizing a function J.
• Example: minimize cost of production, maximize profit.
Machine learning optimization:
• In ML, goal is good performance on new (test) data.
 • That performance measure is usually called P (like accuracy, F1-score, AUROC).
• But the problem: we cannot directly optimize P, because it depends on unseen
test data.
What we actually do:
• Instead, we optimize a proxy function (a cost/loss function J(θ) on training data.
• Example: cross-entropy loss, mean squared error.
• The hope is that by minimizing J(θ), the model’s performance P on test data will
also improve.

8.1.1 Empirical Risk Minimization

Goal of machine learning:

We want to reduce the expected generalization error → this means making the
model perform well on new/unseen data.
 • This expected error is called the risk.

If we knew the true data distribution 𝑝𝑑𝑎𝑡𝑎 (𝑥, 𝑦):

• Then minimizing risk would be a normal optimization problem (easy to define
mathematically).
• Example: If we had all possible exam questions, we could just directly train for
them.
But reality:
• We don’t know the true distribution of all data in the world.
• We only have a finite training dataset (a small sample).
• That’s why machine learning is tricky → we’re solving an optimization problem
with only limited data.

The training process based on minimizing this average training error is known
as empirical risk minimization.
Surrogate Loss Functions and Early Stopping
• Sometimes, the loss function we actually care about (say classification error) is
not one that can be optimized efficiently. For example, exactly minimizing
expected 0-1 loss is typically intractable (exponential in the input dimension),
even for a linear classifier
• In such situations, one typically optimizes a surrogate loss function instead, which
acts as a proxy but has advantages.
• For example, the negative log-likelihood of the correct class is typically used as
a surrogate for the 0-1 loss. The negative log-likelihood allows the model to
estimate the conditional probability of the classes, given the input, and if the
model can do that well, then it can pick the classes that yield the least
classification error in expectation
Batch
• Gradient Descent in Deep Learning
When training a neural network, we update weights using gradient descent.
But the way we compute the gradient depends on how much data we use at once.
• Optimization algorithms that use the entire training set are called batch or
deterministic gradient methods, because they process all of the training examples
simultaneously in a large batch.
• Example: If you have 1 million images, you calculate the loss and gradient for
all 1 million before updating weights.
Mini-Batch Gradient Descent
 • Splits the dataset into small batches (e.g., 32, 64, 128 samples).
• For each mini-batch: compute gradient → update weights.
• Most commonly used in deep learning.
• Pros :
• Faster training than full batch.
• More stable than using single samples.
• Works well with GPUs (parallel processing).
• Cons : Still approximate, so updates can be noisy.
• Example: With 1 million images and batch size = 64 → you update weights after
every 64 images.
Stochastic Gradient Descent (SGD)
• Special case of mini-batch, where batch size = 1.
• Update weights after every single training example.

8.3 BASIC ALGORITHMS

8.3.1 Stochastic Gradient Descent
• Gradient descent = method to train models by adjusting weights step by step to
reduce loss.
• Stochastic gradient descent (SGD) is a version where:
• Instead of using the whole dataset to compute the update (which is slow),
• We use just a few samples (a mini-batch) to estimate the gradient and
update weights.
• Stochastic Gradient Descent (SGD) trains a model by updating weights using
small random subsets of data (mini-batches) instead of the whole dataset, making
training much faster and efficient.
Inputs needed:
• Learning rate (𝜖𝑘 → (how big a step we take in each update.
• Initial parameters (迿ÿ) → the starting weights of the model.
Loop until stopping condition is met:
• “Stopping condition” could be:
• number of iterations reached,
• loss becomes very small,
• or validation accuracy stops improving.
Repeat until the model has trained enough.
• ^g is the average of the gradients computed for each example in the chosen
mini-batch.
What it means
• 迿ÿ= the parameters (weights) of the model.
• 𝜖= learning rate (step size).
• 𝑔̂ =the gradient estimate (direction of steepest increase of the loss, computed from
the mini-batch).
• The update rule says:
Take the current parameter values, and subtract a small step in the direction
of the gradient.
Why subtract?
• The gradient points in the direction of increasing the loss.
• We want to decrease the loss, so we go in the opposite direction.
Learning rate is crucial in SGD
 • The learning rate (𝜖) controls how big a step the algorithm takes when updating
weights.
• If it’s too large → the algorithm may overshoot and never converge.
• If it’s too small → learning becomes very slow.
• Before, we assumed 𝜖was constant (same step size at every iteration).
• It’s usually better to start with a larger learning rate (so the model learns
quickly in the beginning),
• Then gradually decrease it as training progresses (to fine-tune and converge
smoothly).
• That’s why the learning rate at iteration 𝑘is written as:
• instead of a fixed 𝜖.
8.3.2 Momentum

• Imagine you're trying to roll a ball down a hill to reach the very bottom, where
there's a prize. The hill isn't smooth—it has bumps, dips, and flat spots that can
slow you down or even trap the ball.
• If you just push the ball gently each time, it might get stuck in a small dip or keep
bouncing back and forth without going far.
• Momentum is like giving the ball a stronger, steadier push that keeps it moving
in one direction, even if it hits a bump. It helps the ball ignore the small dips and
wiggles, so it can roll faster and smoother toward the prize at the bottom
• In deep learning, this "ball" is like the computer trying to find the best solution,
and momentum helps it move past obstacles and get to the goal faster!
Nesterov Momentum
Nesterov Momentum, also known as Nesterov Accelerated Gradient (NAG), is an
advanced optimization technique that improves upon traditional momentum in
gradient descent by incorporating a "look-ahead" step to better anticipate future
parameter updates, leading to faster and more stable convergence.
How Nesterov Momentum Works
• Traditional momentum updates parameters by adding a fraction of the past update
(velocity) to the current gradient step, which helps accelerate convergence and
smooth out oscillations.
• Nesterov Momentum modifies this by first making a look-ahead step using the
current velocity (momentum), then calculating the gradient at this look-ahead
position.
• This lets the optimizer "peek" where it is going and adjust the update based on
this future position, effectively acting with more foresight.
Basic Idea: Moving Down a Hill
• Imagine you’re rolling a ball down a curvy hill to find the bottom (the best
solution for your model). In basic gradient descent, you push the ball based on
the slope where it is right now. But if the hill has flat spots or small bumps, the
ball might slow down or wiggle a lot.
• Momentum adds a "push" from the direction the ball was already moving, helping
it keep going even on flat areas.
• Nesterov Momentum takes this further by being smarter about where it looks
before pushing.
 • Instead of just checking the slope at the current spot, it peeks ahead to where
the ball is about to go, then decides how to push based on that future spot. This
makes the movement more accurate and efficient.
•

This anticipatory update allows the optimizer to better adjust its course and reduce
overshooting or slowdowns.
Key Benefits
• Faster convergence: Especially effective in loss surfaces with flat or curved
regions.
• Greater stability: By correcting the step before the full update, it smooths
oscillations more effectively than classical momentum.
• Better responsiveness: The look-ahead gradient allows the method to avoid poor
local minima with more agility.
Monentun vs Nesterov nonentun

Parameter Initialization Strategies

• Deep learning models are trained with iterative algorithms (like gradient
descent).
• These algorithms start from an initial point (initial weights & biases).
• The choice of initialization affects:
• Whether training converges at all.
• How fast it converges.
• Whether it finds a good solution (low cost, good generalization).
• Bad initialization can cause:
• Numerical instability → algorithm crashes.
• Slow convergence → learning drags.
• Poor generalization → model works well on training but fails on new data.
Why Initialization Matters
• Deep learning training algorithms are iterative and sensitive to the initial
parameter values.
• The initial point influences:
• Convergence: Poor initialization may cause slow or no convergence.
• Numerical stability: Bad initial points can cause exploding or vanishing
gradients.
• Final solution quality: Different initializations can lead to different local
minima with varying generalization error.
• A key requirement is to break symmetry: units with the same activation receiving
the same inputs must start with different parameters to learn distinct features.
Common Initialization Strategies
• Random Initialization
• Most weights are initialized randomly from Gaussian or uniform
distributions.
• Often biases are initialized to zero or a small constant.
• The scale of the distribution is crucial; too large causes exploding values,
too small causes vanishing gradients.
• Heuristic Scales
• Xavier (Glorot) Initialization: Scales weights depending on the number of
input and output connections, suitable for sigmoid/tanh activations.
• He Initialization: Specifically designed for ReLU activations; weights are
drawn with variance scaled by 2 divided by the number of inputs.
• Both approaches aim to maintain activation variance and gradient variance
through layers.
• Orthogonal Initialization
• Use random orthogonal matrices for weight initialization.
• Ensures different units start with independent directions.
• Sparse Initialization
• Only a few weights per neuron are non-zero at start.
• Helps with diversity but can slow early learning if wrong weights are chosen.
• Data-Based Initialization
• Set biases/vances using training data statistics.
• Example: in autoencoders, bias = mean of input distribution.
• Pretraining
• Initialize weights from another model:
 • Unsupervised pretraining (e.g., autoencoder).
• Supervised pretraining on related or even unrelated task (transfer learning).
Algorithms with Adaptive Learning Rates
• Neural network researchers have long realized that the learning rate was reliably
one of the hyperparameters that is the most difficult to set because it has a
significant impact on model performance.
• Learning rate difficulty: The learning rate is a crucial hyperparameter, difficult
to set because it strongly impacts training performance.
• Sensitivity in parameter space: Some directions in parameter space affect the loss
a lot (high sensitivity), while others do not.
• Momentum helps but adds complexity: Momentum smooths training but
introduces another hyperparameter.
• Separate learning rate per parameter: If sensitivities are axis-aligned, it's logical
to use different learning rates for each parameter.
Delta-bar-delta algorithm:
• An early heuristic from Jacobs (1988) that adjusts individual learning rates
dynamically.
• If the gradient's sign for a parameter remains consistent, the learning rate for that
parameter increases—to make faster progress.
• If the gradient's sign flips, it indicates overshooting or oscillation, so the learning
rate decreases.
Full batch only: This algorithm generally applies to full batch gradient descent (using
the full dataset at each update).

AdaGrad (Adaptive Gradient) algorithm

• The AdaGrad algorithm is an adaptive optimization technique that adjusts the
learning rate for each individual parameter based on past gradients.
 • Individually adapts the learning rates for each model parameter by scaling
them inversely proportional to the square root of the sum of all historical
squared gradients for that parameter.

How AdaGrad Works

• For every parameter, AdaGrad keeps a running sum of all previous squared
gradients.
• When updating, the learning rate for a parameter is divided by the square root of
its accumulated squared gradient.
• If a parameter has a large gradient often, its learning rate drops quickly, slowing
updates in steep directions.
• If a parameter has small gradient (partial derivatives), its learning rate decreases
more slowly, allowing bigger steps in flat regions of the loss landscape
Effects
• Rapid decrease for large gradients: Parameters with large gradients quickly get a
much smaller learning rate, preventing overshooting.
• Gentle progress in flat areas: Parameters with small gradients keep relatively
larger learning rates, encouraging faster movement through flat directions.
• This makes AdaGrad effective for problems with sparse data, or where different
features are important at different times (e.g., NLP or recommendation systems)
Limitations in Deep Learning
• constant accumulation of squared gradients causes the learning rate to shrink too
much over time—sometimes becoming so small that learning nearly stops before
reaching a good solution.
• This means AdaGrad may stall prematurely and not always work well for deep
models.
 • AdaGrad shrinks the learning rate for parameters that have large gradients
frequently, but keeps it higher for those with smaller gradients.
• This means the algorithm moves faster in flatter directions and slower in steeper
directions, helping especially in sparse data scenarios.
• Over time, the learning rate can become very small for some parameters, causing
the effective updates to stop for those directions

RMSProp(Root Mean Square Propagation) algorithm

The RMSProp algorithm is an adaptive learning rate optimization method designed to
improve deep learning training, especially on non-convex problems.
RMSProp vs. AdaGrad
• AdaGrad adapts the learning rate for each parameter by dividing it by the square
root of the running sum of all its past squared gradients.
• Problem: In deep or non-convex models, this running sum grows too large,
causing the learning rate to shrink excessively. Training may slow down or
even stop before finding a good solution.
RMSProp solves this by using an exponentially weighted moving average (EWMA)
instead of a simple sum:
• Only recent squared gradients matter most, so old gradient information
fades away.
• This keeps the learning rate from decaying too quickly and allows the
optimizer to adapt to new regions in parameter space.
RMSprop

AdaGrad

Why is this important?

• RMSProp helps maintain a balanced learning rate as optimization progresses,
enabling effective learning in deep, non-convex settings where AdaGrad often
fails due to learning rate collapse.
• By focusing on recent gradients, RMSProp can navigate complex, changing loss
landscapes better than AdaGrad, which only works best for convex functions or
sparse features.
summary:
RMSProp is designed for deep learning by using a decaying average of squared
gradients, preserving effective learning rates and ensuring the optimizer remains
adaptable throughout the training process.
Adam (Adaptive Moment Estimation) Algorithm

Adam is an adaptive learning rate optimization algorithm widely used in deep learning.
It combines the benefits of two other algorithms:RMSProp and Momentum
Key Concepts in Adam
• Momentum (First-order moment):
• Adam keeps track of an exponentially weighted moving average of past
gradients, smoothing the gradient updates.
• This helps accelerate convergence by reducing oscillations.
 • RMSProp (Second-order moment):
• Adam also tracks an exponentially weighted moving average of the squared
gradients.
• This rescales the learning rate individually per parameter, adjusting for
gradient magnitudes.
• Bias correction:
• Because both first and second moment estimates start at zero, Adam
includes bias-correction terms to counteract initial bias, improving early
training stability.

Why Adam is Effective

• By combining momentum and adaptive learning rates, Adam allows faster
convergence and more stable training, especially for complex, high-dimensional
neural networks.
• Bias correction ensures accurate moment estimates early on, making Adam robust
to hyperparameter choices.
• This makes Adam a popular default optimizer for many deep learning tasks.
Comparison with RMSProp
• RMSProp also adapts learning rates based on squared gradients but lacks
momentum incorporation and bias correction.
• Adam explicitly models both first and second moments with corrections,
providing a theoretically stronger foundation and often better performance.