MODULE-2
Multilayer Perceptrons: Introduction, Batch Learning
and Online Learning, The Back-Propagation
Algorithm, XOR Problem, Heuristics for Making the
Back- Propagation Algorithm Perform Better, Back
Propagation and Differentiation.
INTRODUCTION
In Chapter 1,we studied Rosenblatt’s perceptron,which is basically a single-layer neural
[Link],we showed that this network is limited to the classification of linearly
separable patterns
Algorithm is also based on a single linear neuron with adjustable weights, which limits the
computing power of the algorithm.
To overcome the practical limitations of the perceptron and the LMS algorithm, we look to a
neural network structure known as the multilayer perceptron.
● Key features of MLPs:
1. Neurons use nonlinear differential activation functions.
2. Networks include one or more hidden layers.
3. High connectivity via synaptic weights.
Challenges:
● Nonlinearity + high connectivity → hard to analyze theoretically.
● Hidden neurons make the learning process less interpretable.
A popular method for the training of multilayer perceptrons is the back-propagation
algorithm,which includes the LMS algorithm as a special [Link] training proceeds in
two phases:
● In the forward phase, the synaptic weights of the network are fixed and the input
signal is propagated through the network,layer by layer,until it reaches the output.
● In the backward phase, an error signal is produced by comparing the output of
the network with a desired [Link] resulting error signal is propagated
through the network,again layer by layer,but this time the propagation is
performed in the backward direction.
The development of the back-propagation algorithm in the mid-1980s represented a
landmark in neural networks in that it provided a computationally efficient method for
the training of multilayer perceptrons
SOME PRELIMINARIES
Figure 4.1 shows the architectural graph of a multilayer perceptron with two hidden layers
and an output layer.
To set the stage for a description of the multilayer perceptron in its general form, the
network shown here is fully connected.
This means that a neuron in any layer of the network is connected to all the neurons
(nodes) in the previous layer. Signal flow through the network progresses in a forward
direction,from left to right and on a layer-by-layer basis.
Figure 4.2 depicts a portion of the multilayer [Link] kinds of signals are
identified in this network:
1. Function Signals.
2. Error Signals.
1. Function Signals (Forward Pass)
● Start at the input layer (sensory units).
● Propagate forward neuron by neuron through hidden layers.
● Reach the output layer as the final output signal.
● Called “function signals” because:
○ They perform a useful function at the output.
○ At each neuron, the signal is computed as a function of inputs +
synaptic weights + activation function.
● Also referred to as the input signals of the network.
2. Error Signals (Backward Pass)
● Originate at the output neurons (difference between actual vs desired output).
● Propagate backward layer by layer through the network.
● Called “error signals” because:
○ Each neuron computes them using an error-dependent function.
● Drive the adjustment of synaptic weights during learning (backpropagation).
Network Structure
● Input layer: sensory/source nodes.
● Hidden layers: neurons not visible to input/output, responsible for feature
transformations.
● Output layer: produces the final predicted outputs.
In short:
● Function signals = forward propagation of activations.
● Error signals = backward propagation of errors for weight updates.
Functions Hidden Neurons in MLPs
● Feature Detectors:
Hidden neurons act as feature detectors that extract important patterns from input
data.
● Learning Salient Features:
As training progresses, they discover the most relevant features that help represent
the data effectively.
● Nonlinear Transformation:
They perform nonlinear mappings of input data into a new feature space.
● Improved Class Separation:
In this feature space, different classes (in classification tasks) become
more easily separable than in the raw input space.
● Key Difference from Rosenblatt’s Perceptron:
Unlike Rosenblatt’s single-layer perceptron (limited to linearly separable
problems), the MLP with hidden neurons can solve nonlinear
classification tasks because of its learned feature space.
Credit-Assignment Problem in MLPs
● In distributed learning systems like the Multilayer Perceptron (MLP), the
credit-assignment problem refers to:
How to assign “credit” (or blame) for the overall network output to the
individual hidden neurons that contributed to it.
● For Output Neurons:
○ Easy to assign credit.
○ Desired response (target output) is known → weights can be adjusted
directly using an error-correction algorithm.
● For Hidden Neurons:
○ Much harder.
○ Hidden neurons do not have direct target outputs.
○ Yet, their internal decisions influence the final output.
○ The challenge = figuring out how much each hidden neuron
contributed to the final error.
Why It Matters:
● Effective learning requires assigning responsibility correctly across all
layers.
● Without solving this, the network cannot learn meaningful hidden
representations.
Solution:
● The Backpropagation algorithm solves the credit-assignment problem.
● It uses the chain rule of calculus to compute error gradients for hidden neurons,
thereby determining how each hidden weight contributed to the final error.
● This allows systematic weight updates for both hidden and output layers.
The credit-assignment problem asks: “Which hidden neurons are responsible for the
error, and by how much?”
Backpropagation provides an elegant solution by propagating error signals backward
and distributing credit (or blame) to all neurons in proportion to their contribution.
Key Features
● Requires differentiable activation functions (sigmoid, tanh, ReLU).
● Iteratively reduces error by adjusting weights.
● Solves the credit-assignment problem (assigns blame/credit to hidden neurons).
● Includes LMS algorithm as a special case (when there are no hidden layers).
●
The backpropagation (BP) algorithm is a supervised learning method used to train MLPs by minimizing the
error between predicted and desired outputs.
Backpropagation trains MLPs by combining forward computation of outputs with backward propagation of
errors, systematically updating weights to minimize the error.
The LMS algorithm (Widrow & Hoff, 1960) is a stochastic gradient descent method used to adaptively adjust
the weights of a linear neuron in order to minimize the Mean Squared Error (MSE) between the desired output
and the actual output.
BATCH LEARNING AND ONLINE LEARNING
THE BACK-PROPAGATION ALGORITHM
Back-propagation (BP) is a supervised learning algorithm used to train multilayer perceptrons
(MLPs).
It combines:
● Forward pass → compute outputs layer by layer.
● Backward pass → compute error gradients and update weights.
It’s based on gradient descent, minimizing the mean squared error (MSE).
Forward Pass
Backward Pass (Error Propagation)
Weight Update Rule
Stopping Criteria
Training stops when:
● Gradient norm is very small.
● Change in error per epoch < threshold.
● Generalization performance stops improving.
Step-by-Step Algorithm
1. Initialize weights (small random values, zero mean).
2. For each epoch:
a. Present each training pair (x(n),d(n))
b. Forward pass → compute activations.
c. Backward pass → compute local gradients.
d. Update weights using delta rule.
3. Repeat until stopping criterion is satisfied.
Fig. 4.3, which depicts neuron j being fed by a set of function signals produced by a layer of neurons to its
left.
The local gradient is the “error signal” that each neuron uses to adjust its incoming
weights during back-propagation.
Why Important?
Mathematical derivation of back-propagation weight updates
Step 1: Induced Local Field
Step 2: Neuron Output
Step 3: Gradient (Chain Rule)
Step 4: Compute Each Term
Step 5: Combine Results
continues the derivation and introduces the delta rule and the local gradient concept
Weight Correction (Delta Rule)
Substitution of Gradient
Two Cases
It formalizes weight updates with the delta rule.
It introduces local gradients as the core quantity for back-propagation.
Local Gradient for Hidden Neuron
Consider the situation in Fig. 4.4, which depicts neuron j as a hidden node of the network
This diagram illustrates how:
1. Hidden neurons (like j) don’t have their own target output.
2. Their influence on the final error is indirect, through the neurons in the next layer (like k).
3. To compute the local gradient for hidden neuron j, we must propagate back the error from kkk.
Weight Update Rule (Delta Rule)
Local Gradient
Output neurons: Local gradient = direct error × slope.
Hidden neurons: Local gradient = slope × (weighted sum of downstream gradients).
Together, they allow error signals to flow backward and adjust weights properly.
The Two Passes of Computation
Back-propagation learning consists of two distinct phases for each training example:
Activation Function
Rate of Learning
Effect of Learning Rate
To improve stability and speed, introduces momentum:
Key Observations
Stopping Criteria
Since back-propagation cannot be guaranteed to converge, there are no strict rules for stopping. Instead,
we rely on reasonable practical criteria, each with advantages and drawbacks.
Summary of the Back-Propagation Algorithm
We train a multilayer perceptron (MLP) using supervised learning with back-propagation.
The process alternates between a forward pass (computing activations) and a backward pass (updating
weights).
Forward pass → compute outputs & errors.
Backward pass → propagate errors backward, compute local gradients, and update weights.
Learning continues until a stopping rule is satisfied.
Back-propagation with nonlinear differential activations (sigmoid, tanh, etc.) enables the MLP
to approximate complex nonlinear mappings.
The corresponding signal-flow graph for back-propagation learning,incorporating both the forward and backward
phases of the computations involved in the learning process, is presented in Fig. 4.7
XOR PROBLEM
Single-Layer Perceptron Limitation
● Rosenblatt’s perceptron has only one layer of weights, no hidden neurons.
● It implements a linear decision boundary (a line in 2D, a plane in 3D, a
hyperplane in nD).
● This works fine for problems like AND, OR, NOT, which are linearly separable.
● But it fails for patterns that are non linearly separable.
XOR as the Example
● The exclusive-OR (XOR) is the simplest nonlinearly separable problem.
● Inputs: two bits (x₁, x₂).
● Output: 1 if inputs are different, 0 if they are the same.
Generalization (Unit Hypercube)
● XOR is a special case of a larger class of problems:
○ Classifying points inside an n-dimensional unit hypercube (each coordinate is 0 or 1).
○ Some classification rules require nonlinear separation (like XOR).
So:
● Linear perceptron → fails.
● Multilayer perceptron (with hidden neurons) → succeeds, because it can form nonlinear decision
boundaries.
[Link]: Two Hidden Neurons (Second Layer)
4. Final Output Layer
○ These two hidden neurons’ outputs are then combined by an output neuron to
form the nonlinear boundary needed for XOR.
○ Effectively, the network partitions the unit square into regions that correspond to XOR
outputs.
Structure of the Network
● Input layer: 2 neurons (x1,x2x_1, x_2x1,x2)
● Hidden layer: 2 neurons (Neuron 1, Neuron 2)
● Output layer: 1 neuron (Neuron 3)
Hidden Layer Neurons
Output Neuron
Step 1: Define the Network
Step 2: Compute XOR Outputs
Step 3: Result (Truth Table)
By adding 2 hidden neurons, the network creates two linear decision boundaries.
Their combination at the output neuron produces the nonlinear separation required
for XOR.
This two-layer perceptron successfully
models XOR, which a single-layer
perceptron could never do. That’s why
XOR became the historical proof
that hidden layers are necessary for
solving nonlinear problems
HEURISTICS FOR MAKING THE BACK-PROPAGATION ALGORITHM PERFORM
BETTER
It is often said that the design of a neural network using the backpropagation
algorithm is more of an art than a science, in the sense that many of the factors
involved in the design are the results of one’s own personal experience.
Nevertheless, there are methods that will significantly improve the backpropagation
algorithm performance, as described here:
1)Stochastic versus batch update.
Stochastic (Online) Update
● In stochastic mode, the weights are updated after each individual training sample.
● Gradient is computed for one sample at a time.
● Advantages:
○ Faster (computationally cheaper per update).
○ Noisy updates help the algorithm escape local minima.
○ Typically converges faster in practice for large datasets.
● Disadvantages:
○ More fluctuations in the error curve (not smooth like batch).
Batch Update
● In batch mode, the weight updates are computed after processing all training
samples in the dataset.
● Gradient is averaged over the entire dataset before applying updates.
● Advantages:
○ Produces a stable and accurate estimate of the gradient.
● Disadvantages:
○ Computationally expensive (especially for large datasets).
○ Convergence is usually slower.
○ Can get stuck in local minima because updates are very smooth.
Stochastic back-propagation learning (pattern-by-pattern updates) is computationally faster and helps
explore the error surface better than batch mode, especially when the training dataset is large and contains
redundancy.
[Link] Information Content
○ Choose training examples with maximum error or those very different from previous
ones.
○ Shuffle examples each epoch to avoid bias from class ordering.
When training a neural network with back-propagation, not all training examples are equally useful.
The idea is: present examples that give the network the most “new” information.
Two practical ways:
1. Largest Error Examples
o Pick examples where the network currently makes the biggest mistakes.
o These force the network to correct itself more effectively.
2. Radically Different Examples
o Choose examples that are very different from those already used.
o This expands the exploration of the weight space and prevents the network from “over-focusing”
on similar patterns.
3. Activation function
Why activation functions matter
● They introduce nonlinearity, which allows networks to approximate complex decision boundaries
(e.g., XOR).
● They must be differentiable, since back-propagation relies on gradients.
● The choice of function affects learning speed, gradient flow, and final performance
● Reduces redundancy:
○ If an activation is always positive (like sigmoid), then all weight updates shift in the same
direction → inefficient learning.
○ Odd functions allow weights to adjust in both positive and negative directions.
What “zero-centered” means
● When an activation function is odd (φ(−v)=−φ(v)), the outputs are symmetric around zero.
● That means for positive inputs you get positive outputs, and for negative inputs you get negative outputs of the
same size.
● On average (over many random inputs), the neuron’s output will be close to 0 instead of being shifted up (all
positive).
The Function
Why This Function is Preferred
● Balances positive and
negative outputs.
● Provides good gradients near
the origin.
● Avoids problems of sigmoid
(always positive, slower
convergence).
4. Target values.
If we choose target values exactly equal to the extreme limits of the activation function , then during
training:
● The network tries to push the outputs all the way to those extremes.
● This forces hidden neurons into saturation (flat regions of tanh), where derivatives ≈ 0.
● When derivatives vanish → gradients vanish → learning slows down drastically.
● Sometimes, weights even diverge (go toward infinity).
5. Normalizing the inputs.
Make training faster and more stable by preprocessing inputs before feeding them into
the network.
Steps shown in the figure:
1. Original Data (x₁, x₂)
o Raw input features may have nonzero means, correlations, and very different scales.
o This causes inefficient weight updates (zigzagging across error surface).
2. Mean Removal
o Subtract the mean of each input variable.
o Ensures that input features are centered around zero.
Helps neurons receive balanced positive/negative inputs
3. Decorrelation
o Remove linear dependencies between input features (using PCA).
o Transforms input variables so they are uncorrelated.
o Ensures learning in one direction doesn’t interfere with another.
4. Covariance Equalization
o Scale the decorrelated inputs so that variances are equal.
o Each input contributes equally → all weights learn at a similar speed.
Why this helps
· Prevents weights from being updated in a skewed, zigzag fashion.
· Ensures smoother gradient descent.
· When combined with the scaled tanh activation, the neural outputs tend to have variance ≈ 1 →
balanced and efficient learning.
6. Initialization.
When training a neural network with back-propagation, the starting values of the weights and biases strongly
influence:
● Speed of convergence (how fast error decreases).
● Avoiding poor local minima.
● Avoiding saturation (flat regions where gradients vanish).
If weights are chosen poorly (e.g., all zeros or very large values), learning can become extremely slow or fail
altogether
Small random weights
● Weights are usually initialized with small random numbers close to zero.
● This prevents large activations that push neurons into saturation (where derivatives ≈ 0).
● Randomness ensures neurons don’t all learn the same thing (symmetry breaking).
Zero biases (or small constants)
● Biases can often start at 0 (or small positive values for ReLU to avoid dead neurons).
Avoid symmetry
● If all weights are initialized to the same value (e.g., all zeros), all neurons in a layer compute the same output →
they learn the same thing. Randomness prevents this.
7. Learning from hints.
Normally, back-propagation learns only from examples (input–output pairs).
● We don’t know the true mapping
● The network just “guesses” by adjusting weights based on training samples.
Learning from hints means:
● We use additional prior knowledge about , beyond just examples.
● This prior knowledge is injected into the learning process, so the network learns
faster and better.
Why is This Useful?
● Accelerates learning: fewer examples are needed, since the network already has
guidance.
● Improves generalization: the final function learned is more accurate and consistent.
● Reduces search space: instead of blindly exploring all possible mappings, the network
is guided toward better solutions.
“Learning from hints” = incorporating prior knowledge (invariance, symmetry, rules,
constraints) into the learning process. This reduces training time, improves accuracy,
and helps the network approximate f(⋅)f(\cdot)f(⋅) more effectively than learning only
from raw examples.
8. Learning rates.
Problems with Learning Rate
Choice
Heuristic Guidelines
Back-Propagation and Differentiation
Back-propagation is used to compute gradients of the error function with respect to
network weights.
Gradients are needed to adjust weights and minimize error.
5. Differentiability Requirement
● Back-propagation requires activation functions to be differentiable.
● That’s why functions like sigmoid, tanh, ReLU (piecewise differentiable) are
used.
● Non-differentiable functions (like a step function) cannot be used directly
because their derivative is undefined or zero almost everywhere.
In summary:
● Differentiation is at the heart of back-propagation.
● Using the chain rule, we compute gradients layer by layer.
● These gradients guide weight updates that minimize error.