UNIT-II:

Feed forward Networks-Multilayer Perceptron, Gradient Descent, Backpropagation,

Empirical Risk Minimization, regularization, auto encoders.
Deep Neural Networks: Difficulty of training deep neural networks, Greedy layer wise
training.

Multilayer Perceptron:
A Multilayer Perceptron (MLP) is a foundational type of feedforward artificial neural
network within deep learning. It is characterized by having one or more hidden layers
between the input and output layers, allowing it to model complex, non-linear relationships in
data, unlike a single-layer perceptron which is limited to linear functions.
Architecture
An MLP is composed of several layers of interconnected nodes, or "neurons":
●​ Input Layer: Receives the raw input features of the dataset. No computation is
performed here; data is simply passed to the next layer.
●​ Hidden Layer(s): One or more intermediate layers where the primary computation
occurs. Each neuron in a hidden layer calculates a weighted sum of its inputs from the
previous layer, adds a bias, and then applies a non-linear activation function (e.g.,
ReLU, sigmoid, or tanh). This non-linearity is what gives MLPs their power to solve
complex problems.
●​ Output Layer: The final layer that produces the network's prediction or classification
result. The activation function here depends on the task (e.g., softmax for multi-class
classification, linear for regression).
Every connection between neurons in adjacent layers has an associated weight, which
determines the strength of that connection, and each neuron has a bias term.
How it Works
The functioning of an MLP involves two main processes:
1.​ Forward Propagation: Input data moves in a single, forward direction through the
network, from the input layer, through the hidden layers, and to the output layer, to
generate a prediction.
2.​ Backpropagation: This is the learning algorithm used to train the network. The error
(difference between the predicted and actual output) is calculated using a loss function
and then propagated backward through the network. This process, which uses gradient
descent and the chain rule of calculus, iteratively adjusts the weights and biases to
minimize the error, allowing the network to learn.
Applications
Due to their ability to learn complex patterns, MLPs are widely used in various applications,
including:
 ●​ Image recognition and optical character recognition
●​ Speech recognition
●​ Natural language processing (e.g., sentiment analysis)
●​ Financial forecasting
●​ Medical diagnosis
●​ Fraud detection
Limitations
While powerful, MLPs have some limitations:
●​ Computationally Expensive: Training large MLPs can require significant processing
power and time.
●​ Overfitting Risks: MLPs can easily overfit to training data if not properly regularized
or if the dataset is small.
●​ Sensitivity to Data Scaling: They require input data to be properly scaled or
normalized for optimal performance.
●​ Cannot Capture Sequence/Spatial Patterns Well: More specialized architectures
like Convolutional Neural Networks (CNNs) for images or Recurrent Neural
Networks (RNNs) for sequential data are generally more effective for those specific
tasks.
Multi Layer Perceptron Algorithm :

1.​ Initialize the weights (Wi) & Bias (B0) to small random values near Zero
2.​ Set learning rate η or α in the range of “0” to “1”
3.​ Check for stop condition. If stop condition is false do steps 3 to 7
4.​ For each Training pairs do step 4 to 7
5.​ Set activations of Output units: xi = si for i=1 to N
6.​ Calculate the output Response yin = b0 + Σ xiwi
7.​ Activation function used is Bipolar sigmoidal or Bipolar Step functions
For Multi Layer networks, based on the number of layers steps 6 & 7 are
repeated
8.​ If the Targets is (not equal to) = to the actual output (Y), then update weights and bias
based on Perceptron Learning Law
Wi (new) = Wi (old) + Change in weight vector
​ ​ Change in weight vector = ηtixi
 Where η =
Learning
Rate
ti = Target
output of ith
unit
xi = ith Input
vector
b0(new) = b0 (old) + Change in Bias
Change in
Bias = ηti Else
Wi (new) = Wi (old)

b0(new) = b0 (old)
9.​ Test for Stop condition

Gradient Descent:

Gradient Descent is an iterative optimization algorithm used to minimize a

cost function by adjusting model parameters in the direction of the steepest descent
of the function’s gradient. In simple terms, it finds the optimal values of weights and
biases by gradually reducing the error between predicted and actual outputs.
It measures the degree of change of a variable in response to the changes of
another variable. Mathematically, Gradient Descent is a convex function whose output is
the partial derivative of a set of parameters of its inputs. The greater the gradient, the
steeper the slope. Starting from an initial value, Gradient Descent is run iteratively to find
the optimal values of the parameters to find the minimum possible value of the given cost
function.
 3.​ Take a Step Down: Move in the direction where the slope is steepest (this is adjusting
the model's parameters). The bigger the slope, the bigger the step you take.
4.​ Repeat: You keep repeating the process — feeling the slope and moving downhill —
until you reach the bottom of the valley (this is when the model has learned and
minimized the error).
The key idea is that, just like walking down a hill, Gradient Descent moves towards the
"bottom" or minimum of the loss function, which represents the error in predictions.

3. Stop when change in loss is very small (convergence).

4. Return final w and b.

Types of Gradient Descent:

Typically, there are three types of Gradient Descent:
1.​ Batch Gradient Descent
2.​ Stochastic Gradient Descent
3.​ Mini-batch Gradient Descent
Stochastic Gradient Descent (SGD):
Stochastic Gradient Descent (SGD) is a variant of the gradient descent algorithm where the
model parameters are updated using the gradient of the loss function with respect to a single
training example at each iteration. Unlike batch gradient descent which uses the entire
dataset SGD updates the parameters more frequently, leading to faster convergence.
Mini-Batch Gradient Descent
Mini-Batch Gradient Descent is a compromise between Batch Gradient Descent and
Stochastic Gradient Descent. Instead of using the entire dataset or a single training example
Mini-Batch Gradient Descent updates the model parameters using a small, random subset of
the training data called a mini-batch.
Batch Gradient Descent
Batch Gradient Descent is a variant of the gradient descent algorithm where the entire dataset
is used to compute the gradient of the loss function with respect to the parameters. In each
iteration the algorithm calculates the average gradient of the loss function for all the training
examples and updates the model parameters accordingly.

Backpropagation:
Backpropagation (Backward Propagation of Errors) is the core algorithm for training deep
neural networks, efficiently calculating how much each weight and bias contributed to the
network's prediction error and adjusting them to improve accuracy, using calculus (chain
rule) to propagate errors backward from output to input layers. It works with gradient
descent to iteratively minimize the loss function, enabling models to automatically learn
complex patterns by reducing prediction mistakes over time.
How Backpropagation Works
1.​ Forward Pass: Input data travels forward through the network, layer by layer,
producing an output prediction.
2.​ Calculate Error: The predicted output is compared to the actual target using a loss
function (e.g., Mean Squared Error) to quantify the error.
3.​ Backward Pass (Error Propagation):
o​ The error is propagated backward, from the output layer through hidden
layers to the input.
o​ The chain rule of calculus is used to determine the gradient (partial
derivative) of the loss with respect to each weight and bias, indicating how
much each parameter influenced the error.
2.​ Weight Update:
o​ These gradients inform how to adjust weights and biases to reduce the error.
o​ Gradient Descent optimization (using a learning rate) moves weights in the
direction that most rapidly decreases the loss.
 3.​ Iteration: This entire process repeats for many training examples and epochs (full
passes through the data) until the network's performance is optimized.
Example of Back Propagation in Machine Learning

Forward Propagation

1. Initial Calculation

The weighted sum at each node is calculated using: aj​=∑(wi​,j∗xi​)

Where,
●​ Aj is the weighted sum of all the inputs and weights at each node
●​ wi,j represents the weights between the ith input and the jth neuron
●​ xi represents the value of the ithith input
O (output): After applying the activation function to a, we get the output of the neuron:
oj = activation function(aj)

2. Sigmoid Function

The sigmoid function returns a value between 0 and 1, introducing non-linearity into the
model.
Empirical Risk Minimization (ERM):
Empirical Risk Minimization (ERM) is a core principle in deep learning and machine
learning where the learning algorithm minimizes the average loss (error) on a given training
dataset to find an optimal model. It is a practical approach to approximate the "true risk"
(expected error on all possible data, which is unknown) by using the available data as a
representative sample.
Example:
Step 1: Training Data
Suppose we have 3 training samples:
x y (actual)

1 2

2 4

3 5
Step 2: Our Model
Assume model: f(x)=wx
Let current weight be: w=1.5
Step 3: Prediction
Adjust weights :
Adjust weight w so that average loss becomes minimum.
Example:
●​ If we change www to 1.8, loss becomes smaller.
●​ If we change www to 2, loss becomes even smaller.
Finally, best w≈ 1.9 or 2.
That process = Empirical Risk Minimization (ERM).
Empirical risk = what we want to reduce
Backpropagation = how we calculate how to change weights
So:
Term Meaning

Empirical Risk Average training error

ERM Minimizing that average error

Backpropagation Finding gradients to reduce it

Regularization:
Regularization in machine learning is a technique used to reduce overfitting and improve a model’s
generalization ability.

Regularization is a technique used in machine learning and deep learning to prevent

overfitting and improve a model’s generalization performance. It involves adding a penalty
term to the loss function during training.

How does Regularization help Reduce Overfitting?Let’s consider a neural network that is
overfitting on the training data as shown in the image below:
Dropout
This is one of the most interesting types of regularization techniques. It also produces very
good results and is consequently the most frequently used regularization technique in the
field of deep learning.
To understand dropout, let’s say our neural network structure is akin to the one shown below:
At every iteration, it randomly selects some nodes and removes them along with all of their
incoming and outgoing connections.
Early Stopping
Early stopping is a cross-validation strategy in which we keep one part of the training set as
the validation set. When we see that the performance on the validation set is getting worse,
we immediately stop the training on the model.

Types of regularization:
L1 regularization or Lasso regression
L2 regularization or ridge regression
Dropout
Early stopping
By applying regularization for deep learning, models become more robust and better at
making accurate predictions on unseen data.
Auto encoders:
Autoencoders are a special type of neural networks that learn to compress data into a compact
form and then reconstruct it to closely match the original input. They consist of an:
●​ Encoder that captures important features by reducing dimensionality.
●​ Decoder that rebuilds the data from this compressed representation.
The model trains by minimizing reconstruction error using loss functions like Mean Squared
Error or Binary Cross-Entropy. These are applied in tasks such as noise removal, error
detection and feature extraction where capturing efficient data representations is important.

Architecture of Autoencoder
An autoencoder’s architecture consists of three main components that work together to
compress and then reconstruct data which are as follows:

1. Encoder
It compress the input data into a smaller, more manageable form by reducing its
dimensionality while preserving important information. It has three layers which are:
●​ Input Layer: This is where the original data enters the network. It can be images, text
features or any other structured data.
 ●​ Hidden Layers: These layers perform a series of transformations on the input data.
Each hidden layer applies weights and activation functions to capture important
patterns, progressively reducing the data's size and complexity.
●​ Output(Latent Space): The encoder outputs a compressed vector known as the latent
representation or encoding. This vector captures the important features of the input
data in a condensed form helps in filtering out noise and redundancies.
2. Bottleneck (Latent Space)
It is the smallest layer of the network which represents the most compressed version of the
input data. It serves as the information bottleneck which force the network to prioritize the
most significant features. This compact representation helps the model learn the underlying
structure and key patterns of the input helps in enabling better generalization and efficient
data encoding.
3. Decoder
It is responsible for taking the compressed representation from the latent space and
reconstructing it back into the original data form.
●​ Hidden Layers: These layers progressively expand the latent vector back into a
higher-dimensional space. Through successive transformations decoder attempts to
restore the original data shape and details
●​ Output Layer: The final layer produces the reconstructed output which aims to
closely resemble the original input. The quality of reconstruction depends on how
well the encoder-decoder pair can minimize the difference between the input and
output during training.
Loss Function in Autoencoder Training
During training an autoencoder’s goal is to minimize the reconstruction loss which measures
how different the reconstructed output is from the original input. The choice of loss function
depends on the type of data being processed:
●​ Mean Squared Error (MSE): This is commonly used for continuous data. It
measures the average squared differences between the input and the reconstructed
data.
●​ Binary Cross-Entropy: Used for binary data (0 or 1 values). It calculates the
difference in probability between the original and reconstructed output.
During training the network updates its weights using backpropagation to minimize this
reconstruction loss. By doing this it learns to extract and retain the most important features of
the input data which are encoded in the latent space.
Types of Autoencoders
Lets see different types of Autoencoders which are designed for specific tasks with unique
features:
1. Denoising Autoencoder
Denoising Autoencoder is trained to handle corrupted or noisy inputs, it learns to remove
noise and helps in reconstructing clean data. It prevent the network from simply memorizing
the input and encourages learning the core features.
2. Sparse Autoencoder
Sparse Autoencoder contains more hidden units than input features but only allows a few
neurons to be active simultaneously. This sparsity is controlled by zeroing some hidden units,
adjusting activation functions or adding a sparsity penalty to the loss function.
3. Variational Autoencoder
Variational autoencoder (VAE) makes assumptions about the probability distribution of the
data and tries to learn a better approximation of it. It uses stochastic gradient descent to
optimize and learn the distribution of latent variables. They used for generating new data
such as creating realistic images or text.
It assumes that the data is generated by a Directed Graphical Model and tries to learn an
approximation to qϕ(z∣x) to the conditional property qθ(z∣x) where ϕ ϕ and θ θ are the
parameters of the encoder and the decoder respectively.
4. Convolutional Autoencoder
Convolutional autoencoder uses convolutional neural networks (CNNs) which are designed
for processing images. The encoder extracts features using convolutional layers and the
decoder reconstructs the image through deconvolution also called as upsampling.

Difficulty of training deep neural networks:

●​ Vanishing & Exploding Gradients: Gradients become extremely small (vanish) or

large (explode) during backpropagation, hindering learning in early layers or causing
instability.
●​ Overfitting & Underfitting: Models can memorize training data (overfit) or fail to
capture patterns (underfit), requiring careful balancing.
●​ Computational Intensity: Deep models require immense processing power
(GPUs/TPUs) and time, making training costly and slow.
 ●​ Hyperparameter Tuning: Finding the right learning rate, batch size, and architecture is
complex and crucial for performance.
●​ Data Dependency: Large amounts of high-quality, labeled data are needed, which can
be hard to acquire.
●​ Interpretability ("Black Box"): Understanding why a deep network makes a decision
is challenging, which is a concern in critical fields.
●​ Local Minima: The optimization landscape has many "dips" (local minima) where the
model can get stuck, failing to find the best solution.
●​ Hardware Limitations: Access to specialized hardware remains a bottleneck.
Common Solutions:
●​ Activation Functions: ReLU, Leaky ReLU, and ELU help prevent vanishing
gradients.
●​ Weight Initialization: Techniques like He initialization improve gradient flow.
●​ Optimizers: Adam, RMSprop adapt learning rates dynamically.
●​ Regularization: Dropout, Batch Normalization combat overfitting.
●​ Hardware/Software: GPUs, TPUs, cloud platforms, and efficient libraries accelerate
training.

Greedy layer wise training:

Greedy layer-wise training is a training strategy used mainly in deep neural networks,
especially in the early days of deep learning, to overcome difficulties in training very deep
models.
why do we need Greedy Layer-Wise Training?
When training deep networks end-to-end using backpropagation, we often face problems
like:
1.​ Vanishing gradients
2.​ Poor local minima
3.​ Slow convergence
4.​ Overfitting when data is limited
Greedy layer-wise training was introduced to make training deep networks easier and
more stable.
The Algorithm
Step 1: Train First Layer
●​ Treat the first hidden layer as a shallow network
●​ Train it to learn good representations of the input data
●​ Often uses unsupervised learning (autoencoders, RBMs)
Step 2: Freeze and Add Next Layer
●​ Freeze the weights of the trained layer
●​ Add the next layer on top
●​ Train only this new layer using outputs from the previous layer as input
Step 3: Repeat
●​ Continue adding and training layers one at a time
●​ Each layer learns progressively more abstract features
Step 4: Fine-tuning (Optional)
●​ After all layers are pre-trained, unfreeze all weights
●​ Perform supervised training on the entire network
●​ This "fine-tunes" the pre-trained features for the specific task
Advantages
●​ Better initialization: Provides a good starting point in parameter space
●​ Reduced vanishing gradients: Each layer is trained with shorter backpropagation paths
●​ Feature learning: Each layer learns meaningful representations independently
●​ Regularization effect: Pre-training can act as a form of regularization, reducing overfitting
Methods for Layer-Wise Training
1. Stacked Autoencoders
●​ Each layer is trained as an autoencoder (input reconstruction)
●​ Forces the layer to capture essential information
●​ Unsupervised: doesn't require labels
2. Restricted Boltzmann Machines (RBMs)
●​ Probabilistic generative models
 ●​ Learn to model the probability distribution of input data
●​ Stack multiple RBMs to create Deep Belief Networks (DBNs)
3. Supervised Layer-Wise Training
●​ Train each layer with auxiliary supervised tasks
●​ Less common but can be effective for specific problems

Modern Perspective
Why Greedy Layer-Wise Training is Less Common Now
While revolutionary in the 2000s, greedy layer-wise pre-training is now rarely used because:
1.​ Better initialization methods: Xavier/He initialization solve many initialization problems
2.​ Better activation functions: ReLU and variants reduce vanishing gradients
3.​ Batch normalization: Normalizes layer inputs, stabilizing training
4.​ Residual connections: Skip connections in ResNets allow gradients to flow directly
5.​ Better optimizers: Adam, RMSprop adapt learning rates automatically