Regularization

Regularization for Deep Learning - Dataset Augmentation

Regularization is a set of techniques that can prevent overfitting in neural networks
and thus improve the accuracy of a Deep Learning model when facing completely
new data from the problem domain.

Regularization is a technique used in machine learning and deep learning to

prevent overfitting and improve the generalization performance of a model. It
involves adding a penalty term to the loss function during training.
Parameter Norm Penalties
Parameter Norm Penalties are regularization methods that apply a penalty to the
norm of parameters in the objective function of a neural network.

This approach limits the capacity of the model by adding the penalty Ω(θ) to the
objective function resulting in:

α ∈[0,∞) is a hyperparameter that weights the relative contribution of the norm

penalty to the value of the objective function.
When the optimization procedure tries to minimize the objective function, it will
also decrease some measure of size of the parameters θ.

Note: The bias terms in the affine transformations of deep models usually require
less data to be fit and are usually left unregularized.

Without loss of generality, we will assume we will be regularizing only the weights
w
L² Parameter Regularization

L2 regularization, also known as Ridge regularization or weight decay, is a technique used to prevent
overfitting by adding a penalty to the loss function proportional to the sum of the squares of the model’s
weights. Unlike L1 regularization, which promotes sparsity, L2 regularization encourages the weights to be
small but does not necessarily push them to zero.
L2 regularization adds a term to the loss function that is proportional to the sum of the squares of the
weights. The regularized loss function can be expressed as:
L2 Norm Parameter Regularization
● Weight Shrinkage: L2 regularization shrinks the weights towards zero but does not force them to
be exactly zero. This results in smaller weights, which can reduce model complexity and prevent
overfitting.
● Smoothness: It tends to produce models with more evenly distributed weights, avoiding scenarios
where a few weights are excessively large.
● Computational Stability: L2 regularization can improve the numerical stability of the optimization
process, especially in the presence of multicollinearity or when features are highly correlated.
● Interpretability: While L2 regularization does not produce sparse models, it can still contribute to
improved model performance by reducing the influence of less important features.
L1 Regularization (Lasso Regularization)
L1 regularization, also known as Lasso regularization, adds a penalty equal to the absolute value of
the magnitude of the coefficients to the loss function. In simpler terms, L1 regularization
discourages the model from learning overly complex patterns by shrinking less important feature
weights to zero, effectively performing feature selection.
where:

● λ is the regularization strength, a hyperparameter that controls the amount of regularization.

● wi are the weights of the model.

Lasso regression automatically performs feature selection by eliminating the least important features.

Ignoring the least important features helps emphasize the model’s essential features.
L1 Norm Parameter Regularization
L1 norm is another option that can be used to penalize the size of model
parameters.
L1 regularization on the model parameters w is
#creating sequential model model=Sequential()
[Link](Conv2D(filters=16,kernel_size=2,padding="same",activation="relu",input_shape =(50,50,3)))
[Link](MaxPooling2D(pool_size=2))
[Link](Conv2D(filters=32,kernel_size=2,padding="same",activation="relu"))
[Link](MaxPooling2D(pool_size=2))
[Link](Conv2D(filters=64,kernel_size=2,padding="same",activation="relu"))

[Link](MaxPooling2D(pool_size=2)) [Link](Flatten()) #l2 regularizer

[Link](Dense(500,kernel_regularizer=regularizers.l2(0.01),activation="relu"))

[Link](Dense(2,activation="softmax"))#2 represent output layer neurons

On the other hand, the L1 norm penalty provides solutions that are sparse.

This sparsity property can be thought of as a feature selection mechanism

Dataset Augmentation
● We have seen that for consistent estimators, the best way to get better
generalization is to train on more data.
● The problem is that under most circumstances, data is limited. Furthermore,
labelling is an extremely tedious task.
● Dataset Augmentation provides a cheap and easy way to increase the
amount of your training data.
● Certain tasks such as steering angle regression require dataset augmentation
to perform well
Data augmentation involves artificially increasing the size of the training dataset by creating
modified versions of images, text, or data points. For images, this could mean flipping, rotating,
zooming, or changing the brightness of the images. This helps the model generalize better by
learning from slightly altered versions of the original data.
Pros:

● Helps reduce overfitting by providing more diverse data to learn from.

● Helps improve the generalization and robustness of the model.

● Provides an economical way to simulate a more extensive dataset without the need for

additional data collection.

Cons:

● Can significantly increase training time.

● Requires careful implementation to avoid introducing unrealistic artifacts.

● Augmented data may not perfectly represent real-world variations, and using it exclusively

can lead to a lack of trust in the model’s performance.

from [Link] import ImageDataGenerator

# Create an ImageDataGenerator with augmentation

datagen = ImageDataGenerator(
rotation_range=20,
width_shift_range=0.2,
height_shift_range=0.2,
shear_range=0.2,
zoom_range=0.2,
horizontal_flip=True,
fill_mode='nearest'
)

# Example of applying the augmentation to an image

# Assuming 'images' is a numpy array of images
augmented_images = [Link](images, batch_size=32)

# Use the augmented data for training

[Link](augmented_images, epochs=10)
Small datasets can make learning challenging for neural nets and the examples
can be memorized.
Adding noise during training can make the training process more robust and
reduce generalization error.
Noise is traditionally added to the inputs, but can also be added to weights,
gradients, and even activation functions.
Regularization Strategies: Noise Robustness
Noise with infinitesimal variance imposes a penalty on the norm of the
Weights.
Noise added to hidden units is very important and is discussed later
in Dropout. Noise can even be added to the weights.
This has several interpretations. One of them is that adding noise to weights is a
stochastic implementation of Bayesian inference over the weights, where the
weights are considered to be uncertain, with the uncertainty being modelled by a
probability distribution. It is also interpreted as a more traditional form of
regularization by ensuring stability in learning.
This form of regularization encourages the parameters to go to regions of
parameter space where small perturbations of the weights have a relatively small
influence on the output.

In other words, it pushes the model into regions where the model is relatively
insensitive to small variations in the weights, finding points that are not merely
minima, but minima surrounded by flat regions (Hochreiter and
Schmidhuber, 1995).
Add noise to activations, i.e. the outputs of each layer.
Add noise to weights, i.e. an alternative to the inputs.
Add noise to the gradients, i.e. the direction to update weights.
Add noise to the outputs, i.e. the labels or target variables.

The addition of noise to the layer activations allows noise to be used at any point in the
network. This can be beneficial for very deep networks. Noise can be added to the layer
outputs themselves, but this is more likely achieved via the use of a noisy activation
function.
Most datasets have some amount of mistakes in the y labels. Minimizing our cost
function on wrong labels can be extremely harmful.

One way to remedy this is to explicitly model the noise on labels. This is done
through setting a probability for which we think the labels are correct.

This probability is easily incorporated into the cross entropy cost function
analytically. An example is label smoothing.
The addition of noise to gradients focuses more on improving the robustness of the
optimization process itself rather than the structure of the input domain. The amount of
noise can start high at the beginning of training and decrease over time, much like a
decaying learning rate. This approach has proven to be an effective method for very deep
networks and for a variety of different network types

Adding noise to the activations, weights, or gradients all provide a more generic approach to
adding noise that is invariant to the types of input variables provided to the model.

If the problem domain is believed or expected to have mislabeled examples, then the
addition of noise to the class label can improve the model’s robustness to this type of error.
Although, it can be easy to derail the learning process.

Adding noise to a continuous target variable in the case of regression or time series
forecasting is much like the addition of noise to the input variables and may be a better use
case.
Label Smoothing
Usually, we have output vectors provided to us as
ylabel = [1, 0, 0, 0...0].

Softmax output is usually of the form

yout = [0.87, 0.001, 0.04, 0.1, ....0.03].

Maximum likelihood learning with a softmax classifier and hard targets may
actually never converge, the softmax can never predict a probability of exactly 0 or
exactly 1, so it will continue to learn larger and larger weights, making more
extreme predictions. forever
Semi-Supervised Learning
Semi-supervised learning is a type of machine learning that falls in between supervised
and unsupervised learning. It is a method that uses a small amount of labeled data and a
large amount of unlabeled data to train a model. The goal of semi-supervised learning is
to learn a function that can accurately predict the output variable based on the input
variables, similar to supervised learning.

However, unlike supervised learning, the algorithm is trained on a dataset that contains
both labeled and unlabeled data.

Semi-supervised learning is particularly useful when there is a large amount of unlabeled

data available, but it’s too expensive or difficult to label all of it.
Instead of having separate unsupervised and supervised components in the model, one
can construct models in which a generative model of either P (x) or P(x,y) shares
parameters with a discriminative model of P(y | x).

One can then trade off the supervised criterion −log P(y | x) with the unsupervised or
generative one (such as −log P(x) or −log P(x,y)).

The generative criterion then expresses a particular form of prior belief about the solution
to the supervised learning problem (Lasserre et al., 2006), namely that the structure of
P(x) is connected to the structure of P(y | x) in a way that is captured by the shared
parametrization.

By controlling how much of the generative criterion is included

in the total criterion, one can find a better trade-off than with a purely generative or a
purely discriminative training criterion (Lasserre et al., 2006; Larochelle and Bengio,
2008). Salakhutdinov and Hinton (2008) describe a method for learning the kernel
Examples of Semi-Supervised Learning

Text classification: In text classification, the goal is to classify a given text into one or
more predefined categories. Semi-supervised learning can be used to train a text
classification model using a small amount of labeled data and a large amount of
unlabeled text data.
Image classification: In image classification, the goal is to classify a given image into
one or more predefined categories. Semi-supervised learning can be used to train an
image classification model using a small amount of labeled data and a large amount of
unlabeled image data.
Anomaly detection: In anomaly detection, the goal is to detect patterns or
observations that are unusual or different from the norm
Applications of Semi-Supervised Learning
1. Speech Analysis: Since labeling audio files is a very intensive task,
Semi-Supervised
learning is a very natural approach to solve this problem.
2. Internet Content Classification: Labeling each webpage is an impractical and
unfeasible process and thus uses Semi-Supervised learning algorithms. Even the
Google search algorithm uses a variant of Semi-Supervised learning to rank the
relevance of a webpage for a given query.
3. Protein Sequence Classification: Since DNA strands are typically very large in
size, the
rise of Semi-Supervised learning has been imminent in this field.
Multi-Task Learning
Multi-Task Learning (MTL) is a type of machine learning technique where a model is
trained to perform multiple tasks simultaneously. In deep learning, MTL refers to training
a neural network to perform multiple tasks by sharing some of the network’s layers and
parameters across tasks.
In MTL, the goal is to improve the generalization performance of the model by leveraging
the information shared across tasks. By sharing some of the network’s parameters, the
model can learn a more efficient and compact representation of the data, which can be
beneficial when the tasks are related or have some commonalities.
Multitask Learning is a way to improve generalization by pooling the examples
arising out of several tasks.
Usually, the most common form of multitask learning is performed through an
architecture which is divided to two parts:

Task-specific parameters (which only benefit from the examples of their task to
achieve good generalization).
Generic parameters, shared across all the tasks (which benefit from the pooled
data of all the tasks).

Multitask learning is a form of parameter sharing.

Hard Parameter Sharing – A common hidden layer is used for all tasks but several
task specific layers are kept intact towards the end of the model. This technique is
very useful as by learning a representation for various tasks by a common hidden
layer, we reduce the risk of overfitting.
Soft Parameter Sharing – Each model has their own sets of weights and biases
and the distance between these parameters in different models is regularized so
that the parameters become similar and can represent all the tasks.
Assumptions and Considerations –

Using MTL to share knowledge among tasks are very useful only when the tasks are very
similar, but when this assumption is violated, the performance will significantly decline.

Applications: MTL techniques have found various uses, some of the major applications
are-
Object detection and Facial recognition
Self Driving Cars: Pedestrians, stop signs and other obstacles can be detected
together
Multi-domain collaborative filtering for web applications
Stock Prediction
Language Modelling and other NLP applications
1. Task relatedness: MTL is most effective when the tasks are related or have some
commonalities, such as natural language processing, computer vision, and healthcare.

2. Data limitation: MTL can be useful when the data is limited, as it allows the model to
leverage the information shared across tasks to improve the generalization performance.

3. Shared feature extractor: A common approach in MTL is to use a shared feature

extractor, which is a part of the network that is shared across tasks and is used to
extract features from the input data.

4. Task-specific heads: Task-specific heads are used to make predictions for each task
and are typically connected to the shared feature extractor.

5. Shared decision-making layer: another approach is to use a shared decision-making

layer, where the decision-making layer is shared across tasks, and the task-specific
layers are connected to the shared decision-making layer.
Multi-task learning like a gym coach training a group of athletes
The warm-up and strength training (shared lower layers) are the same for
everyone — building core muscles, stamina, and agility that benefit all sports.

Then, each athlete spends time on sport-specific drills (task-specific upper

layers) — e.g., basketball players practice shooting, swimmers practice strokes.

Because the early training is shared, athletes improve in general fitness faster than
if they each trained completely separately.

This works best if the sports are somewhat related — say, soccer and basketball
— so the shared training really transfers. If you mix ballet with powerlifting, the
benefits might be smaller (or even negative).

…………….
Shared layers = common training that builds general representations.

Task-specific layers = fine-tuning for each goal.

The shared part learns better because it sees more combined examples, like a coach having more
opportunities to teach good form.

…………….
Facial Analysis System
We want a deep network to process face images and do three related tasks:
1. Identify the person (classification)

2. Predict the person’s age group (classification)

3. Detect facial expression (classification)

MTL setup:
○ The shared CNN sees more total examples
● Shared layers: A CNN feature extractor learns edges, textures,
skin patterns, and facial shapes — these are useful for all three
across tasks → its filters (edges, shapes,
tasks. patterns) are tuned to general features of faces,
not just those needed for one task.
● Task-specific layers:

○ Head 1: Fully connected layers for identity recognition.

○ If we trained each task separately, the feature
extractor might overfit to quirks of that dataset.
○ Head 2: Fully connected layers for age group classification.

○ Head 3: Fully connected layers for expression detection.

○ By training jointly, the model is forced to find
features useful for all tasks, acting as a
regularizer.
…………….
Early Stopping
When training models with sufficient representational capacity to overfit the task,
we often observe that training error decreases steadily over time, while the error
on the validation set begins to rise again.

The occurrence of this behaviour in the scope of our applications is almost certain.
This means we can obtain a model with better validation set error (and
thus,hopefully better test set error) by returning to the parameter setting at the
point in time with the lowest validation set error.

This is termed Early Stopping.

Early Stopping is probably one of the most used regularization strategies in deep
learning.

Early stopping can be thought of as a hyperparameter selection method, where

training time is the hyperparameter to be chosen.

However, a portion of data should be reserved for validation.

Early stopping is an unobtrusive form of regularization, in that it requires almost no
change in the underlying training procedure, the objective function, or the set of
allowable parameter values. This means that it is easy to use early
stopping without damaging the learning dynamics.

This is in contrast to weight decay, where one must be careful not to use too much
weight decay and trap the network in a bad local minimum corresponding to a
solution with pathologically small weights.

Early stopping may be used either alone or in conjunction with other regularization
strategies. Even when using regularization strategies that modify the objective
function to encourage better generalization, it is rare for the best generalization to
occur at a local minimum of the training objective.

Early stopping requires a validation set, which means some training data is not fed
to the model. To best exploit this extra data, one can perform extra training
after the initial training with early stopping has completed.
 One strategy (algorithm 7.2) is to initialize the model again and retrain on all the data. In this
second training pass, we train for the same number of steps as the early stopping procedure
determined was optimal in the first pass.

There are some subtleties associated with this procedure. For example, there is not a good way
of knowing whether to retrain for the same number of parameter updates or the same number of
passes through the dataset. On the second round of training, each pass through the dataset will
require more parameter updates because the
training set is bigger. Another strategy for using all the data is to keep the parameters obtained
from the first round of training and then continue training, but now using all the data. At this
stage, we now no longer have a guide for when to stop in terms of a number of steps. Instead,
we can monitor the average loss function on the validation set and continue training until it falls
below the value of the training set objective at which the early stopping procedure halted. This
strategy avoids the high cost
 Early stopping
Early stopping is a regularization technique used in machine learning to
prevent overfitting by halting training before the model starts to memorize
noise in the training data.
○ Split the data into:
■ Training set – used to fit the model
■ Validation set – used to monitor generalization performance
○ During training, track the validation loss after each epoch.
At first:
■ Training loss ↓
■ Validation loss ↓
○ After some point (overfitting starts):
■ Training loss ↓ continues
■ Validation loss ↑ starts increasing
Stop training at the point where validation loss is minimal (or when it hasn’t
improved for a fixed number of epochs – the patience parameter).
…………….
Example:

We train a small neural network to classify handwritten digits (MNIST).

We track training loss and validation loss for each epoch:
● The best validation performance was at epoch 3 (val loss = 0.35).
● Even though the training loss kept decreasing after that, the validation loss worsened, showing overfitting.
● Early stopping halts training after epoch 5 (with patience = 2) and restores model weights from epoch 3.

Epoch Training Validation Action

Loss Loss
1 0.50 0.48 Keep training
2 0.40 0.38 Keep training
3 0.32 0.35 Keep training
4 0.28 0.36 Validation loss increased – patience counter = 1

5 0.25 0.38 Validation loss still up – patience counter = 2

…………….
Why does the validation loss keep increasing when the training loss is reducing?
1. Early in training
○ The model learns general patterns that apply to both training and validation data.
○ Training loss ↓, validation loss ↓ — both improve.

1. Later in training
○ The model starts fitting details specific to the training set: random fluctuations, mislabeled examples,
rare combinations.
○ Training loss keeps decreasing (because it’s memorizing),
but these memorized quirks don’t exist in the validation set.

1. Effect on validation loss

○ Since validation data is different, the memorized noise hurts performance there.
○ Result: Training loss ↓, validation loss ↑.
○ This is the textbook signal of overfitting.
…………….
Why it’s a form of regularization
● It prevents the model from learning overly complex patterns that fit training noise.
● By stopping earlier, the model parameters remain simpler and less tuned to random
fluctuations.
● Unlike L1/L2 penalties, it doesn’t explicitly change the objective function — instead, it limits the

effective capacity by controlling training duration.

Analogy:
○ Think of it like baking cookies:
○ You want them fully baked, but not burnt.
○ If you keep them in the oven too long (keep training), they’ll burn (overfit).
○ Early stopping is pulling them out just when they’re perfect.

…………….
● Large models can fit training data perfectly (low training error).
● But as training continues, validation error starts to go up — the model is memorizing noise → overfitting.
● Idea: Stop training when validation error is at its minimum.
Steps:
1. Initialization:
○ Split training data into:
■ Subtrain set (for learning parameters)
■ Validation set (for monitoring)
○ Pick patience p (how many worsening steps to tolerate).
○ Pick evaluation interval n (steps between validation checks).
2. Training loop:
○ Train n steps.
○ Measure validation loss.
○ If validation loss improves, save the current parameters and reset patience counter.
○ If validation loss worsens, increment patience counter.
○ Stop when patience counter exceeds p.
3. Return the parameters with the best validation performance, not the final parameters.

3. Two Strategies to Use All Data

● Retraining: Train on the full dataset for exactly the number of steps i* found in early stopping.
● Continue training: Resume from the early-stopped parameters, but now train on all data until loss is below the previous
best level. …………….
Parameter Tying and Parameter Sharing
Parameter tying and parameter sharing are techniques to reduce the model’s
effective capacity by limiting how many independent parameters it can learn —
which in turn helps prevent overfitting.
Parameter Tying
You explicitly force two or more parameters in your model to be equal to each other during
training.
How it works:
Instead of learning separate values for each parameter, you impose a constraint such that they
are mathematically the same.
Example:
Suppose in a model you have two weights w1 and w2, but you require w1=w2 at all times.
You can tie them so that updating one also updates the other.

Purpose: Reduces the number of free parameters, improving generalization and reducing
overfitting.
Parameter Sharing
A design choice where the same parameter is used in multiple parts of the model, instead of being
duplicated and stored separately.
How it works:
You don’t just constrain parameters to be equal — you literally reuse the same memory location for them.
Example:
In Convolutional Neural Networks (CNNs), the same filter (kernel) weights are used across different spatial
positions in the image — this is parameter sharing.
In RNNs, the same weight matrix is used at each time step.
Purpose:
Reduces memory usage.
Encourages translation or temporal invariance.
Improves data efficiency.
Parameter Tying — Matching Outfits Rule
Imagine two friends going to a party:
They each have their own closets (separate storage).
The rule is: they must always wear identical outfits.
They can shop separately, but whatever one chooses, the other has to match exactly.
That’s parameter tying: different storage, but a constraint forces equality.

Parameter Sharing — Same Closet Rule

Now imagine those same two friends share one single closet:
Every time they get dressed, they pick clothes from the same shared space.
It’s literally the same set of clothes, not two copies.
That’s parameter sharing: one set of parameters, used in multiple places
💡 In short:
Tying = different storage, forced to be equal.
Sharing = same storage, naturally equal because it’s reused.
Sparse Representations
A sparse representation is one where most of the elements in a vector (or matrix)
are zero (or very close to zero).

Only a few features or neurons are “active” for any given input.

Most weights or activations are zero, so the model uses a small subset of possible
features.
Two kinds of sparsity:
Parameter sparsity → Many model parameters (weights) are zero. Achieved by L1 penalty on parameters.

Representational sparsity → Many elements of the activations (representations) are zero for a given input. Achieved
by penalizing the activations rather than the parameters.

Almost any hidden-unit model can be made sparse by adding penalties or constraints on the activations, improving
generalization and interpretability.

In linear regression, sparse parameters mean the weight vector is mostly zero.
Sparse representation means the transformed input vector h=f(x) is mostly zero, even if parameters aren’t sparse.
A sparse representation is one where most of the elements in a vector (or matrix) are zero (or very close to
zero).
In machine learning, this usually means:

Only a few features or neurons are “active” for any given input.
Most weights or activations are zero, so the model uses a small subset of possible features.
📚 Big library
Parameter sparsity → The library only owns a small number of books. Most shelves are empty.

Representation sparsity → The library has lots of books, but on any given day, only a few are taken off the
shelves and opened.

In both cases, you’re dealing with “less stuff in use,” but:

Parameter sparsity = you don’t have much stuff to begin with.
Representation sparsity = you have a lot, but you rarely use most of it at the same time.
for eg.
Parameter sparsity = Having few wires in the circuit (parameter sparsity), vs.
Representation sparsity= Having many wires, but only a few switched on at a time (representation
sparsity).
Regularize by representational sparsity:
Add a norm penalty Ω(h) to the loss: ,where α controls regularization
strength.

L1 penalty on activations encourages many activation values to be zero.

Other penalties:
Student-t prior on activations
KL divergence to target average activation (e.g., 0.01 per unit for binary-like activations)
Example: Average activation regularization ensures each neuron is active only rarely.
Hard sparsity constraints:
Orthogonal Matching Pursuit (OMP-k): Find h with at most k non-zero entries that reconstructs the input
via Wh. Efficient when W is orthogonal. Often used for feature extraction.
 Student-t prior
what’s a prior?
In probability/statistics (especially Bayesian methods), a prior is just a way of saying
“Before seeing the data, I believe my values should look like this distribution.”
When we say prior on activations, we mean:
“We expect the neuron outputs (activations) to follow a certain statistical shape.”
Student-t distribution?
● It looks like a bell curve (like the normal/Gaussian), but with heavier tails.
● Heavier tails mean it expects most values to be near zero, but it also allows some large values without
punishing them as harshly as a Gaussian would.
If we assume activations follow a Student-t distribution, that assumption naturally encourages sparsity:
● Most activations → close to zero.
● A few → can be large (important features firing strongly).
● This is good for sparse representation learning, where only a few neurons should be active at a time.
Party guest list:
● Gaussian prior = expects everyone’s energy levels to be similar and moderate.
● Student-t prior = expects most people to be quiet, but allows a…………….
few to be very loud without kicking them
A Student-t prior on activations is just a statistical assumption that most neuron outputs will be near
zero, but a few can be big.
This naturally pushes the network toward sparse activations — most “off,” a few strongly “on.”

…………….
 KL divergence
KL divergence (Kullback–Leibler divergence) is a number that measures how different one
probability distribution is from another.
It’s not symmetric (so and it’s always ≥ 0.

● If P and Q are the same → KL divergence = 0 (perfect match).

● Bigger KL → the two distributions are more different.
We want the average neuron activation to match some target distribution.
Example:
● For binary-like neurons, maybe we want them active only 1% of the time → target average
activation = 0.01.
● We measure the KL divergence between:
○ P = actual distribution of activations
○ Q = desired target distribution
● The KL penalty pushes the activations to match the target → most units stay “off,” a few turn
“on.”
…………….
 It’s like comparing two recipes:
● P = the real cake you baked.
● Q= the recipe you wanted to follow.

● KL divergence = “how far did you stray from the intended recipe?”

● The smaller the KL, the closer you stuck to your intended taste.

…………….
 Hard sparsity
Hard sparsity constraint — the idea
● A hard sparsity constraint means we don’t just encourage sparsity (like with L1 penalties) — we
force it.
● In math terms:

This means the representation vector h can have at most k non-zero entries.

Example: If k=3, only 3 features can be “on” for any input — everything else must be exactly zero.

…………….
 Orthogonal Matching Pursuit (OMP-k)

Orthogonal Matching Pursuit (OMP-k)

● Purpose: Given an input x, find a sparse representation h using at most k non-zero coefficients such
that:
where W is a set of “basis vectors” (like feature templates).

● How it works :
1. Start with all coefficients in h = 0.
2. Pick the feature (column of W) most correlated with the part of x we haven’t explained yet.
3. Add it to the active set and update h to best fit x using these active features.
4. Repeat until k features are selected (or the error is small enough).

● “orthogonal”:
At each step, it makes sure the new feature adds information that isn’t already explained by the
chosen ones (keeps them independent in effect).

…………….
Imagine you’re trying to recreate a song 🎵 with a big instrument library:
● Soft sparsity = You prefer to use few instruments, but if needed, you might use more.

● Hard sparsity (OMP-k) = You’re only allowed to use exactly k instruments.

● OMP process = Pick the instrument that best matches the most obvious part of the song, then pick
the next one that adds the most to what’s missing, and so on, until you’ve used your allowed number.

…………….
Bagging and Other
Ensemble Methods
● Bagging and other ensemble methods regularize models
by averaging predictions from multiple diverse learners,
which reduces variance and overfitting.
● Imagine asking several photographers to take a picture of the same
object from slightly different angles. One photo may have glare or
blurriness, but if you average them into a composite image, you get a
cleaner, more reliable picture.

● Bagging (bootstrap aggregating) reduces generalization error by training

multiple models on different bootstrap-resampled datasets and
averaging their predictions, which cancels out uncorrelated errors and
lowers variance.

…………….
Definition & Purpose
● Bagging (Bootstrap Aggregating), introduced by Breiman (1994), is a method to reduce
generalization error by training multiple models separately and combining their predictions
(averaging for regression, voting for classification).

● It is part of the broader strategy in machine learning called model averaging, used in many
ensemble methods.
● How Bagging Works
○ Construct k different training datasets by sampling with replacement from the original dataset
(bootstrap samples).
○ Each dataset has the same size as the original but contains duplicates and omits some examples
(~⅔ unique samples on average).
○ Train one model per dataset; differences in data lead to diversity in learned models.
Example in the text: digit “8” classifier learning different partial rules in different resamples; combined
rules give robust prediction.
● …………….
 "8" example
Original dataset
● Suppose your dataset has handwritten digits,
including an 8, a 6, and a 9.
Creating bootstrap datasets
● Dataset 1: Sampled with replacement, it omits the
“9” and repeats the “8.”
○ The model trained on this data learns:
“If there’s a loop on top of the digit, it’s an 8.”

● Dataset 2: Sampled differently, it omits the “6” and

● Problem with individual models
repeats the “9.”
Each of these “rules” is brittle—if only one loop is present (say, top○ The model trained here learns:
loop only), that model might confidently call it an 8, even if it’s “If there’s a loop on the bottom, it’s an 8.”
actually a different digit.
● Combining the models (bagging)
When we average their predictions (or vote), an image gets
classified as an 8 only if both rules agree—meaning both loops are
present.
● This makes the combined decision more accurate and less overfit …………….
to quirks of one particular training set.
The "8" example shows that each bagged model captures a partial, noisy piece of
the truth, but when combined, these pieces form a stronger, more general rule that
reduces overfitting.
Dropout

Reduce overfitting by
randomly “dropping out”
units (along with their
connections) during training
 Dropout aims to approximate this process, but with an exponentially large number of neural networks.
Specifically, to train with dropout, we use a minibatch-based learning algorithm that makes small steps,
such as stochastic gradient descent.
● Bagging involves training multiple models,
and evaluating multiple models on each test
example.
● This seems impractical when each model is a
large neural network, since training and
evaluating such networks is costly in terms of
runtime and memory.
● Dropout provides an inexpensive
approximation to training and evaluating a
bagged ensemble of exponentially many
neural networks.
● dropout trains the ensemble consisting of all
sub-networks that can be formed by removing
non-output units from an underlying base
network.
●
●
Pg no 258 Ian Goodfellow
● Imagine a basketball team where, during practice, you randomly tell some players to sit out.
● The rest must learn to play effectively without always relying on the same star players.
● When the actual match comes (inference time), the full team plays — now stronger because
everyone learned to contribute.

…………….
…………….
…………….
…………….
One of the key insights of dropout is that training a network with stochastic behavior and making
predictions by averaging over multiple stochastic decisions implements a form of bagging with
parameter sharing.

Dropout trains not just a bagged ensemble of models, but an ensemble of models that share
hidden units. This means each hidden unit must be able to perform well regardless of which
other hidden units are in the model.

. Dropout thus regularizes each hidden unit to be not merely a good feature but a
feature that is good in many contexts. Warde
Farley et al. ( 2014 ) compared dropout training to training of large ensembles and
concluded that dropout offers additional improvements to generalization error
beyond those obtained by ensembles of independent models.
 It is important to understand that a large portion of the power of dropout arises from the fact that the masking
noise is applied to the hidden units. This can be seen as a form of highly intelligent, adaptive destruction of the
information content of the input rather than destruction of the raw values of the input.

Another important aspect of dropout is that the noise is multiplicative. If the noise were additive with
fixed scale, then a rectified linear hidden unit added noise could simply learn to have become very
large in order to make the added noise insignificant by comparison.

Multiplicative noise does not allow such a pathological solution to the noise robustness problem.

Another deep learning algorithm, batch normalization, reparametrize the model in a way that introduces
both additive and multiplicative noise on the hidden units at training time. The primary purpose of batch
normalization is to improve optimization, but the noise can have a regularizing effect, and sometimes
makes dropout unnecessary