UNIT-II

Feed forward Networks- Multilayer Perceptron, Gradient

Descent, Backpropagation, Empirical Risk Minimization,
regularization, auto encoders.
Deep Neural Networks: Difficulty of training deep neural
networks, Greedy layer wise training.

FEED FORWARD NETWORKS

A Feedforward Neural Network (FNN) is a type of artificial neural network

where information moves only in one direction, from the input layer
through any hidden layers and finally to the output layer. This is the
simplest kind of artificial neural network. There are no cycles or loops in
the network. The term “feedforward” refers to the fact that the connections
between the units do not form a cycle, unlike in more complex types of
networks (like RNN).

1. Inputs go into the first layer (the input layer).

2. They're transformed by weights and biases and passed through an

activation function.

3. This continues through one or more hidden layers.

4. Finally, the output layer makes a prediction.

 Feed-Forward Neural Networks are the same as MLPs.

 Feed-Forward Neural Network (FFNN): The general category of any

neural network where data flows in one direction (input → hidden →
output), no loops.
 Multi-Layer Perceptron (MLP): A type of FFN that specifically has at least
one hidden layer of perceptron (neurons with weights, bias, and activation).

Architecture of a Feed-Forward Neural Network

Input layer

 This is the entry point of the network.

 Each neuron here represents one feature from the dataset.

Importantly, the input layer does not perform any computation itself
— it simply passes raw numbers forward.

 In our example, since there are only two neurons in the Input Layer,
this means that our dataset contains two features

Hidden layers

These are where the real computation happens. We have two Hidden
layers, each containing four neurons.

Each neuron in a hidden layer does these three things:

1. Weighted sum: Each input is multiplied by a weight and then all of them
are summed.

2. Bias: A constant bias is added to give flexibility in shifting the function.

3. Activation function: A non-linear function like ReLU, tanh, or sigmoid is

applied. This step is what allows the network to learn complex, non-linear
patterns.

Output layer

The output layer produces the final prediction, and its design depends on
the problem we are trying to solve: Regression, Binary Classification and
Multi Class Classification.

Multilayer Perceptron (MLP)

 A Multilayer Perceptron (MLP) consists of an input layer, one or

more hidden layers, and an output layer.

 The MLP is fully connected, in the sense that each neuron in one layer

connects (with a specific weight) to every neuron in the following

layer.

 The number of hidden layers and the number of neurons in each

layer are hyperparameters that influence the model’s capacity and

complexity.
  Each neuron in an MLP performs a non-linear activation function,

often a sigmoid or ReLU function, to introduce non-linearity into the

network.

 This allows MLPs to learn complex relationships — data that is not

linearly separable — between input and output variables.

 Every connection in the diagram is a representation of the fully

connected nature of an MLP.

 This means that every node in one layer connects to every node in

the next layer. As the data moves through the network each layer

transforms it until the final output is generated in the output layer.

Working of Multi-Layer Perceptron

The key mechanisms such as forward propagation, loss function,
backpropagation and optimization.
1. Forward Propagation
In forward propagation the data flows from the input layer to the output
layer, passing through any hidden layers. Each neuron in the hidden
layers processes the input as follows:
1. Weighted Sum: The neuron computes the weighted sum of the inputs.
2. Activation Function: The weighted sum z is passed through an
activation function to introduce non-linearity. Common activation
functions include Sigmoid, Tanh, ReLU etc.
2. Loss Function
Once the network generates an output the next step is to calculate the loss
using a loss function. In supervised learning this compares the predicted
output to the actual label.
For a classification problem the commonly used binary cross-entropy loss
function is used.
For regression problems the mean squared error (MSE) is often used.
3. Backpropagation
The goal of training an MLP is to minimize the loss function by adjusting
the network's weights and biases. This is achieved
through backpropagation:
1. Gradient Calculation: The gradients of the loss function with respect to
each weight and bias are calculated using the chain rule of calculus.
2. Error Propagation: The error is propagated back through the
network, layer by layer.
3. Gradient Descent: The network updates the weights and biases by
moving in the opposite direction of the gradient to reduce the loss.
4. Optimization
MLPs rely on optimization algorithms to iteratively refine the weights and
biases during training. Popular optimization methods include Adam,
Stochastic Gradient descent etc.
Gradient Descent

 Gradient descent is one of the most important algorithms in all

of machine learning and deep learning.

 Gradient descent is an optimization algorithm. It is used to find the

minimum value of a function more quickly.

 It is an algorithm to find the minimum of a convex function.

 Gradient descent is also called “the deepest downward slope

algorithm”. It is very important in machine learning, where it is used

to minimize a cost function. The more the cost is minimized, the more

the machine will be able to make good predictions.

The best way to define the local minimum or local maximum of a function

using gradient descent is as follows:

 o If we move towards a negative gradient or away from the gradient of

the function at the current point, it will give the local minimum of

that function.

o Whenever we move towards a positive gradient or towards the

gradient of the function at the current point, we will get the local

maximum of that function.

A loss function refers to the error of one training example, while a cost

function calculates the average error across an entire training set.

Working of Gradient Descent

The main objective of gradient descent is to minimize the cost function or

the error between expected and actual. To minimize the cost function, two

components are required:

 Direction

 Learning Rate

It is defined as the step size taken to reach the minimum or lowest point.

This is typically a small value that is evaluated and updated based on

the behaviour of the cost function. If the learning rate is high, it results

in larger steps but also leads to risks of overshooting the minimum. At

the same time, a low learning rate shows the small step sizes, which

compromises overall efficiency but gives the advantage of more

precision.
1. Slope: the local direction of change

The slope is the derivative of a function.

 Direction: whether the function is increasing (+) or decreasing

(−)
 Magnitude: how steep the increase or decrease is

where L(w) is the loss and w is a model parameter.

Deep learning models have millions of parameters, so we generalize slope to
the gradient.

The gradient is a vector of slopes

Each component is the slope of the loss with respect to one parameter

2. Gradient: the direction of steepest increase

Key property:

 The gradient points in the direction of steepest ascent (fastest

increase of the loss)

This means:

 To minimize the loss, we move in the opposite direction of the

gradient

Gradient descent: using slope to learn

Where:

 θt: current parameters

 ∇L(θt): gradient (slopes)
 η: learning rate (step size)

If the slope is large, the update is large

If the slope is small, the update is small

If the slope is zero, you are at a minimum (or saddle point)

 The loss function measures how wrong the model’s predictions are
compared to the true labels.
where θ represents all model parameters (weights and biases).

3. Forward propagation: computing predictions and loss

Step 1: Input to model

Given input x, weights w, and bias b:

where f is an activation function (ReLU, sigmoid, softmax, etc.).

Step 2: Compute loss

Compare prediction y^ with true label y:

Gradient descent does NOT compute loss

 Loss computation happens in the forward pass

 Gradient descent uses the loss to compute gradients and update
parameters
4. Backward propagation: computing gradients of the loss
Compute gradients (slopes)

We compute:

How much the loss changes if a parameter changes slightly

This process is called backpropagation.

5. Gradient descent update rule

Update parameters

Where:

 η = learning rate
 Gradient direction points toward increasing loss
 Subtracting moves us toward decreasing loss

6. Iterative process

Training repeats:

1. Forward propagation → compute loss

2. Backward propagation → compute gradients
3. Gradient descent → update parameters
4. Repeat until convergence
This loop is called an epoch.

Types of Gradient Descent

Based on the error in various training models, the Gradient Descent

learning algorithm can be divided into Batch gradient descent, stochastic

gradient descent, and mini-batch gradient descent.

1. Batch Gradient Descent:

Batch gradient descent (BGD) is used to find the error for each point in the

training set and update the model after evaluating all training examples.

This procedure is known as the training epoch. In simple words, it is a

greedy approach where we have to sum over all examples for each update.

Advantages of Batch gradient descent:

o It produces less noise in comparison to other gradient descent.

o It produces stable gradient descent convergence.

o It is Computationally efficient as all resources are used for all

training samples.

2. Stochastic gradient descent

Stochastic gradient descent (SGD) is a type of gradient descent that runs

one training example per iteration. Or in other words, it processes a

training epoch for each example within a dataset and updates each training

example's parameters one at a time. As it requires only one training

example at a time, hence it is easier to store in allocated memory. However,

it shows some computational efficiency losses in comparison to batch

gradient systems as it shows frequent updates that require more detail and

speed. Further, due to frequent updates, it is also treated as a noisy

gradient. However, sometimes it can be helpful in finding the global

minimum and also escaping the local minimum.

Advantages of Stochastic gradient descent:

In Stochastic gradient descent (SGD), learning happens on every example,

and it consists of a few advantages over other gradient descent.

o It is easier to allocate in desired memory.

o It is relatively fast to compute than batch gradient descent.

o It is more efficient for large datasets.

3. Mini Batch Gradient Descent:

Mini Batch gradient descent is the combination of both batch gradient

descent and stochastic gradient descent. It divides the training datasets into

small batch sizes then performs the updates on those batches separately.

Splitting training datasets into smaller batches make a balance to maintain

the computational efficiency of batch gradient descent and speed of

stochastic gradient descent. Hence, we can achieve a special type of

gradient descent with higher computational efficiency and less noisy

gradient descent.

Advantages of Mini Batch gradient descent:

o It is easier to fit in allocated memory.

o It is computationally efficient.

o It produces stable gradient descent convergence.

Backpropagation

 Backpropagation, short for “backward propagation of errors,” is a

critical component in training and refining neural network models.

Its role is to adjust the weights, in order to minimize the “error” it

makes during the learning process.

 The term “back-propagating error correction” was introduced in

1962 by Frank Rosenblatt. But the current method, which includes

the use of gradient descent or variants such as stochastic gradient

descent, was popularized by David E. Rumelhart.

Backpropagation process relies on two key concepts:

1. Loss function: This function measures the discrepancy between the

network’s output and the desired outcome. It essentially quantifies how

“wrong” the network is in its predictions.

2. Gradient of the loss function: This represents the direction and

magnitude of change required to minimize the error. It tells us how much

and in what direction we need to adjust the weights to improve the

network’s performance.

 The gradient of the loss function, denoted as ∇L, is a vector

containing the partial derivatives of the loss function (L) with respect

to each of the parameters (weights and biases) in the neural network.

It acts as a guide for optimizing neural networks.

 It points in the direction in which the loss function increases the most

(or the steepest increase). By adjusting the network parameters in the

opposite direction (negative gradient), the loss function can be

decreased, leading to improved model performance.

 Each element of the gradient represents the rate of change of the loss

function with respect to the corresponding parameter. In simpler

terms, it tells us how much the loss will increase or decrease if we

slightly change the value of that specific parameter. For higher values
 of error (loss), we will have larger values of the gradient (and larger

change to the associated parameter).

 Backpropagation utilizes the chain rule of calculus to efficiently

calculate this gradient. This rule allows us to propagate the error

backward through the network, layer by layer, ultimately

determining how each weight contributes to the overall error.

The chain rule plays a crucial role in making backpropagation a practical

and efficient algorithm for training neural networks. It works as follows:

1. The loss function is viewed as a chain of nested functions, where each

function represents the output of a layer in the network.

2. The chain rule provides a formula to calculate the derivative of a

composite function by differentiating each inner function and multiplying

the results.

3. Backpropagation uses the chain rule recursively, starting from the output

layer and working backwards towards the input layer. At each layer, it

calculates the derivative of the loss function with respect to the outputs of

that layer, and then propagates this derivative back to the inputs of that

layer, using the weights and activation functions.

Empirical Risk Minimization

 It is a principle in statistical learning theory which defines a family of

learning algorithms and is used to give theoretical bounds on their
performance. The idea is that we don’t know exactly how well an
algorithm will work in practice (the true "risk") because we don't
know the true distribution of data that the algorithm will work on
but as an alternative we can measure its performance on a known set
of training data.
 If we compute the loss using the data points in our dataset, it’s called
empirical risk. It is “empirical” and not “true” because we are using
a dataset that’s a subset of the whole population.
 When our learning model is built, we have to pick a function that
minimizes the empirical risk that is the delta between predicted
output and actual output for data points in the dataset. This process
of finding this function is called empirical risk minimization (ERM).
 We want to minimize the true risk.

For Example. We would want to build a model that can differentiate

between a male and a female based on specific features. If we select 150
random people where women are really short, and men are really tall, then
the model might incorrectly assume that height is the differentiating
feature. For building a truly accurate model, we have to gather all the
women and men in the world to extract differentiating features.
Unfortunately, that is not possible! So we select a small number of people
and hope that this sample is representative of the whole population.

True Risk (Expected Risk)

The average error over the entire data distribution. Mathematically, it is
expressed as:

Here, 𝑃 represents the true data distribution and 𝐿 is the loss function.

Empirical Risk
Since the true data distribution 𝑃 is usually unknown, we approximate it
using the finite training dataset. The empirical risk is defined as:

where 𝑛 is the number of training samples.

Hypothesis Space and ERM

The success of ERM depends on the hypothesis space 𝐻, which defines
the set of all possible models. If 𝐻 is too simple, the model may underfit
the data. Conversely, if 𝐻 is too complex, the model may overfit.
Loss Functions in ERM
Loss functions play a critical role in ERM by quantifying the error
between predicted and actual values. Common loss functions include:
 Mean Squared Error (MSE) (for regression):

 Cross-Entropy Loss (for classification):

Steps to Implement ERM

 Choose a hypothesis space 𝐻 (e.g., linear models, decision trees).
 Define a suitable loss function.
 Use optimization algorithms (e.g., gradient descent) to minimize the
empirical risk.
Applications of ERM

1. Regression Tasks: Linear and polynomial regression directly apply

ERM to minimize squared errors.
2. Classification Models: Logistic regression and SVMs use ERM with
appropriate loss functions.
3. Deep Learning: Neural networks rely on ERM to optimize their loss
functions using backpropagation.
Limitations of ERM

1. Overfitting: Minimizing empirical risk alone may lead to overfitting,

especially when the hypothesis space is complex.
2. No Guarantee of Generalization: ERM focuses on minimizing training
error, which may not always translate to better performance on unseen
data.
3. Sensitive to Noise: Outliers in the training data can heavily influence
the empirical risk.

Regularization

Regularization is a technique used to reduce errors by fitting the function

appropriately on the given training set and avoiding overfitting.

 Neural networks can learn to represent complex relationships

between network inputs and outputs. This representational power
helps them perform better than traditional machine learning
algorithms in computer vision and natural language processing tasks.
 However, one of the challenges associated with training neural
networks is overfitting.
 When a neural network overfits on the training dataset, it learns
an overly complex representation that models the training
dataset too well. As a result, it performs exceptionally well on the
training dataset but generalizes poorly to unseen test data.
 Regularization techniques help improve a neural network’s
generalization ability by reducing overfitting. They do this by
minimizing needless complexity and exposing the network to more
diverse data.

common regularization techniques:

 Early stopping
 L1 and L2 regularization
 Data augmentation
 Addition of noise
 Dropout

 It involves stopping the training of the neural network at

an earlier epoch hence the name early stopping. Early stopping
reduces overfitting by stopping the training error from becoming
too low consistently.
 Early stopping is a form of regularization used to prevent
overfitting in machine learning and deep learning models. It
involves stopping the training process before the model starts to
overfit.
 If the training error becomes too low and reaches arbitrarily close
to zero, then the network is sure to overfit on the training dataset.
Such a neural network is a high variance model that performs
badly on test data that it has never seen before despite its near-
perfect performance on the training samples.
 if we can prevent the training loss from becoming arbitrarily low,
the model is less likely to overfit on the training dataset, and will
generalize better.
 This can be done using 2 ways:

 The change in metrics such as validation error and validation

accuracy.
 The change in the weight vector.
Data Augmentation

 Data augmentation is a regularization technique that helps a neural

network generalize better by exposing it to a more diverse set of
training examples. As deep neural networks require a large training
dataset, data augmentation is also helpful when we have insufficient
data to train a neural network.

 Data augmentation is a technique to artificially increase the size and

diversity of a training dataset by creating modified copies of existing
data (like flipping images, changing text words, or adding noise) to
make models more robust, reduce overfitting, and improve
performance, especially when real data is scarce.

Applies transformations: Creates new, synthetic data by altering original

samples with techniques like:
o Images: Rotating, flipping, cropping, changing brightness/contrast, adding
noise.

o Text: Replacing words, shuffling sentences, adding/deleting words.

o Audio: Changing pitch, adding background noise, speeding up/slowing

down.

Suppose we have a dataset with N training examples across C classes. We

can apply certain transformations to these N images to construct a larger
dataset.

Any operation that does not alter the original label is a valid
transformation.

L1 and L2 Regularization
When modelling the data, a low bias and high variance scenario is referred
to as overfitting. To handle this, regularization techniques trade more bias
for less variance.

1. Lasso Regularization – L1 Regularization

2. Ridge Regularization – L2 Regularization
3. Elastic Net Regularization-L1 & L2 Regularization

A regression model which uses the L1 Regularization technique is

called LASSO (Least Absolute Shrinkage and Selection
Operator) regression. Lasso Regression adds the “absolute value of
magnitude” of the coefficient as a penalty term to the loss function(L).
Lasso regression also helps us achieve feature selection by penalizing the
weights to approximately equal to zero if that feature does not serve any
purpose in the model.

where,

 m – Number of Features
 n – Number of Examples
 y_i – Actual Target Value
 y_i(hat) – Predicted Target Value

A regression model that uses the L2 regularization technique is

called Ridge regression. Ridge regression adds the “squared magnitude” of
the coefficient as a penalty term to the loss function(L).

Elastic Net Regularization – L1 and L2 Regularization:

This model is a combination of L1 as well as L2 regularization. That

implies that we add the absolute norm of the weights as well as the
squared measure of the weights. With the help of an
extra hyperparameter that controls the ratio of the L1 and L2
regularization.

[Link] of Noise

Another regularization approach is the addition of noise. We can add noise

to the input, the output labels, or the gradients of the neural network.
  Noise addition is a form of regularization that increases model
robustness and generalization by introducing variability, making the
model less sensitive to minor input changes and less prone to
overfitting.

4. Drop out

 Usually, when all the features are connected to the FC layer, it can
cause overfitting in the training dataset. To overcome this problem, a
dropout layer is utilized wherein a few neurons are dropped from the
neural network during training process resulting in reduced size of
the model.
 Usually 30% of the nodes are dropped out randomly from the neural
network. Dropout results in improving the performance of a Neural
Network model as it prevents overfitting by making the network
simpler. It drops neurons from the neural networks during training.

Auto Encoders

 Autoencoders in deep learning are unsupervised neural networks

that learn efficient data representations (encodings) by compressing
 input into a lower-dimensional "bottleneck" (latent space) and then
reconstructing it back to the original form.
 They consist of an encoder (compresses data) and
a decoder (reconstructs data), useful for dimensionality reduction,
feature extraction, anomaly detection, denoising, and even generating
new data.
 Auto encoders are a type of deep learning algorithm that are
designed to receive an input and transform it into a different
representation. They play an important part in image construction.
 Auto encoders are used to reduce the size of our inputs into a smaller
representation. If anyone needs the original data, they can
reconstruct it from the compressed data.

 An auto encoder provides a representation of each layer as the

output.
 It can make use of pre-trained layers from another model to apply
transfer learning to enhance the encoder/decoder.

Architecture of Autoencoders

An Autoencoder consist of three layers:

 1. Encoder

2. Code

3. Decoder

 Encoder: This part of the network compresses the input into a latent space
representation. The encoder layer encodes the input image as a
compressed representation in a reduced dimension. The compressed image
is the distorted version of the original image.
Code: This part of the network represents the compressed input which is
fed to the decoder.
Decoder: This layer decodes the encoded image back to the original
dimension. The decoded image is a lossy reconstruction of the original
image and it is reconstructed from the latent space representation.
  The layer between the encode and decoder, ie. the code is also
known as Bottleneck. This is a well-designed approach to
decide which aspects of observed data are relevant
information and what aspects can be discarded.

 An autoencoder consists of two parts: an encoder network and a

decoder network. The encoder network compresses the input data,
while the decoder network reconstructs the compressed data back
into its original form. The compressed data, also known as the
bottleneck layer, is typically much smaller than the input data.

 The encoder network takes the input data and maps it to a lower-
dimensional representation. This lower-dimensional representation
is the compressed data. The decoder network takes this compressed
data and maps it back to the original input data. The decoder
network is essentially the inverse of the encoder network.

 The bottleneck layer is the layer in the middle of the autoencoder

that contains the compressed data. This layer is much smaller than
the input data, that allows for compression. The size of the
bottleneck layer determines the amount of compression that can be
achieved.
Applications of Autoencoders

Image Coloring
Autoencoders are used for converting any black and white picture into a
colored image. Depending on what is in the picture, it is possible to tell what
the color should be.

Feature variation
It extracts only the required features of an image and generates
the output by removing any noise or unnecessary interruption.

Dimensionality Reduction
The reconstructed image is the same as our input but with reduced
dimensions. It helps in providing the similar image with a reduced pixel value.

Denoising Image
The input seen by the autoencoder is not the raw input but a stochastically
corrupted version. A denoising autoencoder is thus trained to reconstruct the
original input from the noisy version.

Watermark Removal

It is also used for removing watermarks from images or to remove any

object while filming a video or a movie.

Deep Neural Networks: Difficulty of training deep neural

networks

DEEP NEURAL NETWORKS

 Artificial Neural Networks usually consist of an input layer, one to

two hidden layers, and an output layer which it is possible to solve
 easy mathematical questions, and computer problems, including
basic gate structures with their respective truth tables.
 But it is tough for these ANNs to solve complicated image processing,
computer vision, and natural language processing tasks.
 For these problems, we utilize deep neural networks, which often
have a complex hidden layer structure with a wide variety of
different layers, such as a convolutional layer, max-pooling layer,
dense layer, and other unique layers.
 These additional layers help the model to understand problems
better and provide optimal solutions to complex projects.
 A deep neural network has more layers (more depth) than ANN and
each layer adds complexity to the model while enabling the model to
process the inputs concisely for outputting the ideal solution.
 Deep Neural Networks are effective not only because they are large,
but also because they have many hidden layers.
 This depth allows them to learn hierarchical features: early layers
capture simple patterns, which are gradually composed into more
complex structures in later layers. This is known as compositional
representation.
 Two primarily utilized deep learning models are convolutional
neural networks (CNN) and Recurrent Neural Networks (RNN).
 It enables the network to have a more scalable approach producing
higher efficiency and accurate results. In image classification and
object detection tasks, there is a lot of data and images for the model
to compute.

 These convolutional neural networks help to combat these issues

successfully.
 One of the primary requirements for deep learning is data. Data is
the most critical component in constructing a highly accurate model.
In several cases, deep neural networks typically require large
amounts of data in order to prevent overfitting and perform well.
The data requirements for object detection tasks might require more
data for a model to detect different objects with high accuracy.
 Apart from a large amount of data, one must also consider the high
computational cost of computing the deep neural network. The
compilation and training of models for complex tasks will require a
resourceful GPU. Models can often be trained more efficiently on
GPUs or TPUs rather than CPUs. For extremely complex tasks, the
system requirements range higher, requiring more resources for a
particular task.
 During training, the model might also encounter issues such as
underfitting or overfitting. Underfitting usually occurs due to a lack
of data, while overfitting is a more prominent issue that occurs due to
training data consistently improving while the test data remains
constant. Hence, the training accuracy is high, but the validation
accuracy is low, leading to a highly unstable model that does not yield
the best results.
DIFFICULTY IN TRAINING DEEP NEURAL NETWORKS

Vanishing and Exploding Gradients

During training, errors are sent backward through many layers.

In very deep neural networks:

 Vanishing gradients become extremely small → earlier layers learn

very slowly.
 Exploding gradients become very large → training becomes unstable.

 This makes it hard for the network to learn meaningful patterns.

Overfitting

Deep networks have millions of parameters.

If the dataset is small or not diverse, the model may:

 Memorize training data

 Perform poorly on new, unseen data

High Computational Cost

Deep models require:

 Large amounts of data

 Powerful hardware (GPUs/TPUs)
 Long training times
 This can be expensive and slow.

Difficulty in Choosing Hyperparameters

Various factors such as
  Learning rate
 Number of layers
 Number of neurons per layer
 Batch size

 Poor choices can lead to slow learning or failure to converge.

Internal Covariate(Feature) Shift

 As parameters update, the input distribution to deeper layers keeps

changing.
 This slows training and makes optimization harder.
 Techniques like Batch Normalization help reduce this issue.

Initialization Problems

If weights are initialized poorly:

 Training may be very slow

 Neurons may become inactive (“dead neurons”)
 Proper initialization methods (e.g., Xavier, He initialization) are
important.

Optimization Becomes Harder

Deep networks have complex loss surfaces with:

 Many local minima

 Saddle points
 This makes finding a good solution difficult.

Lack of Interpretability
  Deep models are often considered black boxes.
 It is hard to understand why they make certain predictions.

Greedy layer-wise Training

 Traditionally, training deep neural networks with many layers was

challenging.

 As the number of hidden layers is increased, the amount of error

information propagated back to earlier layers is dramatically

reduced. This means that weights in hidden layers close to the output

layer are updated normally, whereas weights in hidden layers close to

the input layer are updated minimally or not at all. Generally, this

problem prevented the training of very deep neural networks and

was referred to as the vanishing gradient problem.

 Pre training a neural network refers to first training a model on one

task or dataset and then using the learned parameters or model from

this training to train another model on a different task or dataset.

 The key benefits of pre-training are:

 Simplified training process.

 Facilitates the development of deeper networks.

 Useful as a weight initialization scheme.

 Perhaps lower generalization error.

 Training deep neural networks was traditionally challenging due to

the vanishing gradient problem. So better optimization and better

regularization is required.

 In this case the weights in the layers close to the input layer were not

updated effectively in response to errors calculated on the training

dataset.

 To overcome this problem, greedy layer wise pre-training is used.

 As the number of hidden layers’ increases, the error information

propagated back to earlier layers is significantly reduced.

 Consequently, the weights in the hidden layer near to the output

layer are updated normally, while those closer to the input layer

receive minimal or no updates.

 Pre-training involves successively adding a new hidden layer to the

model and refitting it.

 It allows the newly added layer to learn from the outputs of the

existing hidden layers.

 The weights of the existing hidden layers remain fixed during

training.

 This technique is called layer-wise because the model is trained one

layer at a time.
 It is referred to as greedy due to its piece wise approach to training

deep neural networks.

 The technique is referred to as “greedy” because the piecewise or

layer-wise approach to solving the harder problem of training a deep

network. As an optimization process, dividing the training process

into a succession of layer-wise training processes is seen as a greedy

shortcut that likely leads to an aggregate of locally optimal solutions,

a shortcut to a good enough global solution.

 greedy, layer-wise procedure

 Train one layer at a time with unsupervised criterion

 Fix the parameters of previous hidden layers

 Previous layers viewed as feature extraction

 By dividing the training process into multiple layer wise steps, the

model follows a greedy optimization process.

 This approach serves as a shortcut that leads to an aggregate of

locally optimal solutions instead of a single globally optimal solution.

 There are 2 main approaches to pre-training

 Supervised greedy layer-wise pre-training

 Unsupervised greedy layer-wise pre-training

 In supervised pre-training hidden layers are added to the model

trained on a supervised learning task.

 In unsupervised pre-training, it uses the greedy layer wise process to

build up an unsupervised autoencoder model. Later to this a

supervised output layer is added.

 Once all the layers are pre-trained it is fine tuned.

 Once all layers are pre-trained

 Add output layer

 Train the whole network using supervised learning

 Supervised learning is performed as in a regular network.

 forward propagation, backpropagation and update.