Multilayer Perceptrons

Basic features of multilayer perceptrons:

•Nonlinear activation functions
•Each neuron doesn’t just pass information straight through; it applies a
nonlinear mathematical function (like a squashing or curve).
•This nonlinearity is what allows the network to solve complex
problems that a simple perceptron cannot.
•Hidden layers
•Between input and output, there are hidden neurons that transform the
data.
•These hidden layers make the network more powerful, because they
can extract features that aren’t obvious in the raw input.
•High connectivity
•Every neuron is usually connected to many others, with the strength of each
connection determined by synaptic weights.
•This dense web of connections gives the network flexibility to learn, but also
makes it more complicated to understand.
 Challenges Caused by These Same
Features
•Difficult theoretical analysis
•Because of the nonlinear functions and many connections, it’s
hard to write down simple equations that fully describe how the
whole network behaves.
•Hard to visualize learning
•With hidden neurons, we can’t directly “see” what features the
network is learning. It’s like trying to understand the mind of a
student just by looking at test scores.
•Huge search space
•The network has to figure out which features should be
represented by hidden neurons. This means searching through a
very large number of possible solutions, which makes learning
slower and harder.
Training of multilayer perceptrons
 The training proceeds in two phases:
1. In the forward phase, the synaptic weights of the network are fixed
and the input signal is propagated through the network layer by layer,
until it reaches the output. Thus, in this phase, changes are confined to
the activation potentials and outputs of the neurons in the network.
2. In the backward phase, an error signal is produced by comparing
the output of the network with a desired response. The resulting error
signal is propagated through the network, again layer by layer, but this
time the propagation is performed in the backward direction. In this
second phase, successive adjustments are made to the synaptic
weights of the network. Calculation of the adjustments for the output
layer is straightforward, but it is much more challenging for the hidden
layers.
SOME PRELIMINARIES
SOME PRELIMINARIES
 Two kinds of signals are identified in this network:
 1. Function Signals. A function signal is an input
signal (stimulus) that comes in at the input end
of the network, propagates forward (neuron by
neuron) through the network, and emerges at
the output end of the network as an output
signal.
 “function signal” –
 First, it is presumed to perform a useful function at
the output of the network.
 Second,at each neuron of the network through
which a function signal passes, the signal is
calculated as a function of the inputs and
associated weights applied to that [Link]
function signal is also referred to as the input
signal.
2. Error Signals. An error signal originates at an
output neuron of the network and propagates
backward (layer by layer) through the network.
We refer to it as an “error signal”because its
computation by every neuron of the network
involves an error-dependent function in one form
or another.
Preliminaries
 The output neurons constitute the output layer of the network.
 The remaining neurons constitute hidden layers of the network.
 Thus, the hidden units are not part of the output or input of the network—hence their
designation as “hidden.”
 The first hidden layer is fed from the input layer made up of sensory units (source
nodes); the resulting outputs of the first hidden layer are in turn applied to the next
hidden layer; and so on for the rest of the network.
 Each hidden or output neuron of a multilayer perceptron is designed to perform two
computations:
 1. the computation of the function signal appearing at the output of each neuron, which is
expressed as a continuous nonlinear function of the input signal and synaptic weights
associated with that neuron;
 2. the computation of an estimate of the gradient vector (i.e., the gradients of the error
surface with respect to the weights connected to the inputs of a neuron), which is needed for
the backward pass through the network.
Function of the Hidden Neurons

 The hidden neurons act as feature detectors;

 they play a critical role in the operation of a multilayer [Link] the
learning process progresses across the multilayer perceptron, the hidden
neurons begin to gradually “discover” the salient features that characterize
the training data.
 They do so by performing a nonlinear transformation on the input data into
a new space called the feature space.
 In this new space, the classes of interest in a pattern-classification task, for
example, may be more easily separated from each other than could be
the case in the original input data space.
 it is the formation of this feature space through supervised learning that
distinguishes the multilayer perceptron from Rosenblatt’s perceptron.
Credit-Assignment Problem
 is the problem of assigning credit or blame for overall outcomes to each of
the internal decisions made by the hidden computational units of the
distributed learning system, recognizing that those decisions are responsible
for the overall outcomes in the first place.
 The credit-assignment problem is about figuring out which parts of a network
(hidden neurons) are responsible for success or failure.
 Why it’s a problem in MLPs:
 For output neurons, it’s easy: we know what the correct answer should be, so
we adjust their weights directly.
 For hidden neurons, we can’t see their “desired output” — we only see the
final result. So we need a way to figure out how much each hidden neuron
helped or hurt the final outcome.
 The solution:
The backpropagation algorithm solves this by working backwards from the
output error, distributing responsibility (“credit” or “blame”) to each hidden
neuron in proportion to how much it contributed to the error.
Batch Learning and Online Learning
 Consider a multilayer perceptron with an input layer of source nodes, one
or more hidden layers, and an output layer consisting of one or more
neurons;

 Let yj(n) denote the function signal produced at the output of neuron j in
the output layer by the stimulus x(n) applied to the input layer.
 the error signal produced at the output of neuron j is defined by:

 the instantaneous error energy of neuron j is defined by

Batch Learning and Online Learning
Batch Learning
 adjustments to the synaptic weights of the multilayer perceptron are performed after the
presentation of all the N examples in the training sample t that constitute one epoch of training.
 the cost function for batch learning is defined by the average error energy eav.
 Adjustments to the synaptic weights of the multilayer perceptron are made on an epoch-by-epoch
basis.
 the learning curve is obtained by plotting eav versus the number of epochs, where, for each epoch
of training, the examples in the training sample t are randomly shuffled.
 The learning curve is then computed by ensemble averaging a large enough number of such
realizations, where each realization is performed for a different set of initial conditions chosen at
random.
 the advantages of batch learning include the following:
 accurate estimation of the gradient vector (i.e., the derivative of the cost function eav with
respect to the weight vector w), thereby guaranteeing, under simple conditions, convergence
of the method of steepest descent to a local minimum;
 parallelization of the learning process.
 However, from a practical perspective, batch learning is rather demanding in terms of storage
requirements.
 In a statistical context, batch learning may be viewed as a form of statistical inference.
 It is therefore well suited for solving nonlinear regression problems
 Online Learning
 adjustments to the synaptic weights of the multilayer perceptron are performed on an
example-by-example basis.
 The cost function to be minimized is therefore the total instantaneous error energy e(n).
 on-line learning makes the search in the multidimensional weight space stochastic in
nature; it is for this reason that the method of on-line learning is sometimes referred to
as a stochastic method.
 This stochasticity has the desirable effect of making it less likely for the learning process
to be trapped in a local minimum, which is a definite advantage of on-line learning
over batch learning.
 Another advantage of on-line learning is the fact that it requires much less storage
than batch learning.
 Another useful property of on-line learning is its ability to track small changes in the
training data, particularly when the environment responsible for generating the data is
nonstationary.
 it is highly popular for solving pattern-classification problems for two important
practical reasons:
 On-line learning is simple to implement.
 It provides effective solutions to large-scale and difficult pattern-classification problems
Backpropogation Algorithm
 Backpropogation Algorithm
 neuron j being fed by a set of function signals produced by a layer of neurons to its left.
The induced local field vj(n) produced at the input of the activation function
associated with neuron j is therefore:

 the function signal yj(n) appearing at the output of neuron j at iteration n is

 the back propagation algorithm applies a correction wji(n) to the synaptic weight
wji(n),which is proportional to the partial derivative 0e(n) 0wji(n) calculus, we may
express this gradient as
Backpropogation Algorithm
Backpropogation Algorithm
Backpropogation Algorithm
Backpropogation Algorithm
Case 1 Neuron j Is an Output Node

When neuron j is located in the output layer of the network,it is supplied with a desired
response of its own.

Case 2 Neuron j Is a Hidden Node

When neuron j is located in a hidden layer of the network, there is no specified desired
response for that neuron. Accordingly, the error signal for a hidden neuron would
have to be determined recursively and working backwards in terms of the error signals
of all the neurons to which that hidden neuron is directly connected; this is where the
development of the back-propagation algorithm gets complicated.
Backpropogation Algorithm
Backpropogation Algorithm
Backpropogation Algorithm
Backpropogation Algorithm
Backpropogation Algorithm
Backpropogation Algorithm
Activation Function
 The computation of the for each neuron of the multilayer
perceptron requires knowledge of the derivative of the
activation function ϕ(·).
 For this derivative to exist, we require the function ϕ(·) to
be continuous.
 An example of a continuously differentiable nonlinear
activation function commonly used in multilayer
perceptrons is sigmoidal nonlinearity, two forms of which
are described here:
Logistic Function.
Hyperbolic tangent function
Activation Function – Logistic Function
 Logistic Function: This form of sigmoidal nonlinearity, in its
general form, is defined by
Activation Function – Logistic Function
Activation Function – Logistic Function
Activation Function – Logistic Function
Activation Function – Logistic Function
Activation Function – Logistic Function
Activation Function – Logistic Function
Activation Function – Logistic Function
Activation Function - Hyperbolic tangent function.
Another commonly used form of sigmoidal non linearity is the hyperbolic tangent
function, which, in its most general form, is defined by
Activation Function - Hyperbolic tangent function.
Activation Function - Hyperbolic tangent function.
Activation Function - Hyperbolic tangent function.
 Rate of Learning
 The learning rate (η) is a crucial hyperparameter in the back-propagation
algorithm. It determines the size of the steps taken to adjust the synaptic weights.
 The Problem with a Fixed Learning Rate
 Using a constant, fixed learning rate can lead to trade-offs between speed and
stability.
 Small Learning Rate:
 Leads to a smooth trajectory in the weight space and ensures the network
reaches a good solution.
 The learning process is very slow, requiring many iterations to converge.
 Large Learning Rate:
 Initially, the learning process is fast.
 The network can become unstable and "oscillatory," meaning the weight
adjustments are large and cause the network to jump around the solution,
rather than settling down.
 Introducing Momentum to the Back-propagation Algorithm

 To get the best of both worlds—fast learning and stability—we introduce a "momentum
constant" (α).

 The role of α:
 α is a positive number, typically between 0 and 1.
 It controls the "feedback loop" of the weight correction. A higher α means the momentum
term has a stronger influence.
The Role of the Momentum Constant
(α)
 The momentum constant α helps the algorithm "remember" its previous
direction of descent.
 Steady Descent: If the partial derivatives (∂E/∂wji) on consecutive iterations
have the same sign (meaning the algorithm is consistently moving
downhill), the momentum term grows exponentially. This accelerates the
descent in the right direction, speeding up convergence.
 Oscillations (Bouncing): If the partial derivatives on consecutive iterations
have opposite signs (meaning the algorithm is oscillating or "bouncing"
around), the momentum term shrinks. This has a stabilizing effect and
dampens the oscillations, allowing the algorithm to settle more smoothly
into the minimum.
 Benefits of Momentum
•Averaging the Past. The momentum term effectively
averages the weight changes over an exponentially
weighted time series. It's like a smoother, more stable
update.
•Accelerated Descent. Momentum helps the back-
propagation algorithm escape from shallow local minima
and descend faster in areas with a consistent slope. It
prevents the algorithm from getting "stuck."
•Stabilizing Effect. Momentum dampens oscillations, which
often occur with high learning rates. This allows us to use a
larger learning rate, achieving faster convergence without
sacrificing stability.
 Rate of Learning

• The learning rate (η) controls the step size, while

the momentum constant (α) controls the influence
of past steps.
•By combining a learning rate with momentum, the
back-propagation algorithm becomes more
efficient and robust. It can navigate the error
surface more effectively, leading to faster training
and better final models.
Stopping Criteria

 traditional algorithms with a clear end, the back-propagation algorithm

does not have a single, well-defined point of convergence.
 When training a neural network with backpropagation, we adjust weights
until the network "learns.“ when should we stop training?
 If we stop too early → underfitting.
 If we train too long → overfitting or wasted computation.
 Three stopping Criteria
 Stopping Criterion 1: Small Gradient Threshold
 Stopping Criterion 2: Small Change in Error
 Stopping Criterion 3: Generalization Performance
Stopping Criterion 1: Small Gradient
Threshold
 A theoretically sound criterion is to stop when the network is at a point
where the error gradient is effectively zero.
 he Theory: A necessary condition for a minimum is that the gradient vector,
g(w) (which contains the first-order partial derivatives of the error with
respect to each weight), must be zero.
 The back-propagation algorithm is considered to have converged when
the Euclidean norm of the gradient vector reaches a sufficiently small
gradient threshold.
 Drawbacks:
 Computational Cost: Calculating the full gradient vector after every iteration can
be computationally expensive.
 Long Training Times: For a successful trial, training might continue for a very long
time before the gradient becomes small enough.
 Stopping Criterion 2: Small Change in Error
•A more practical approach is to stop when the network's performance is no
longer improving significantly.
•At a minimum point, the value of the cost function (e.g., the average
squared error) is stationary. This means it is no longer changing much from one
epoch to the next.
•The back-propagation algorithm is considered to have converged when the
absolute rate of change in the average squared error per epoch is sufficiently
small.
•How it Works:
•We check the change in the average error per epoch.
•The change is considered small if it falls within a certain percentage range,
typically 0.1% to 1% per epoch.
•Drawback:
•Premature Termination: This criterion can sometimes cause the learning
process to stop too early if the error plateaus for a short period before continuing
to decrease.
 Stopping Criterion 3: Generalization
Performance
•The most robust and widely used criterion is to monitor the network's performance on unseen
data.
•The real goal is not to memorize the training data but to perform well on new, unseen data.
This is called generalization.
•The Criterion:
"After each learning iteration, the network is tested for its generalization performance. The
learning process is stopped when the generalization performance is adequate or when it is
apparent that the generalization performance has peaked."
•How it Works (Early Stopping):
•We split our data into a training set and a validation set (also called a generalization set).
•We train the network and monitor its error on both the training set and the validation set
after each epoch.
•The training error will almost always continue to decrease. However, the validation error will
initially decrease, but eventually start to increase as the network begins to overfit the training
data.
•We stop training at the point where the validation error is at its minimum. This is known as
Early Stopping.
Summary of the Back-Propagation Algorithm
Summary of the Back-Propagation Algorithm
 the algorithm cycles through the training sample {(x(n), d(n))}N n =1 as follows:
 1. Initialization. Assuming that no prior information is available, pick the synaptic
weights and thresholds from a uniform distribution whose mean is zero and whose
variance is chosen to make the standard deviation of the induced local fields of the
neurons lie at the transition between the linear and standards parts of the sigmoid
activation function.
 2. Presentations of Training Examples. Present the network an epoch of training
examples. For each example in the sample, ordered in some fashion, perform the
sequence of forward and backward computations described under points 3 and 4,
respectively.
 3. Forward Computation. Let a training example in the epoch be denoted by (x(n),
d(n)), with the input vector x(n) applied to the input layer of sensory nodes and the
desired response vector d(n) presented to the output layer of computation nodes.
Compute the induced local fields and function signals of the network by proceeding
forward through the network, layer by layer. The induced local field v(l) j (n) is:
 XOR Problem
 A single-layer perceptron (one neuron) can only separate inputs with a
straight line (linear separation).
 The XOR problem has input patterns that cannot be separated by a single
straight line.
 Therefore, to solve XOR, we must introduce a hidden layer → this is the birth
of the multilayer perceptron (MLP).
 XOR outputs 1 only if the inputs are different.
 In the 2D plane, (0,0) and (1,1) are class 0, while (0,1) and (1,0) are class 1.
 These cannot be separated with one line → hence, not linearly separable.
 The Solution: Using a Hidden Layer
 The Main Idea: The key to solving this problem is to transform the input data
into a new representation where it is linearly separable. We do this by adding
a hidden layer.
 XOR Problem
 To solve the XOR problem by using a single hidden layer with two neurons, as
in Fig a. The signal-flow graph of the network is shown in Fig. b.
 The following assumptions are made here:
 Each neuron is represented by a McCulloch–Pitts model, which uses a
threshold function for its activation function.
 Bits 0 and 1 are represented by the levels 0 and 1, respectively
 XOR Problem
A. The Hidden Layer: Creating the Decision Boundaries
•Neuron 1 (Top Hidden Neuron):
•Function: This neuron acts like a "gate" that detects if both inputs are "large."
• The neuron will output a 1 only if its weighted sum is greater than its threshold. With
weights of w11=+1, w12=+1 and a bias of b1=−3/2, it turns on when:
(1⋅x1+1⋅x2)≥3/2

•This condition is only met by the input pattern (1,1). So, this neuron effectively creates a
decision boundary that isolates the (1,1) corner.
•Output: Neuron 1 outputs 1 for (1,1) and 0 for all other inputs.
 XOR Problem
•Neuron 1 (Top Hidden Neuron):
The goal of this neuron is to only fire when the input is (1,1). For all other inputs ((0,0), (0,1), (1,0)),
the output should be 0.
•Weights: w11=+1 and w12=+1.
•Let's test the weighted sum for each of the four possible inputs:
•Input (0,0): Weighted sum = (1⋅0)+(1⋅0)=0
•Input (0,1): Weighted sum = (1⋅0)+(1⋅1)=1
•Input (1,0): Weighted sum = (1⋅1)+(1⋅0)=1
•Input (1,1): Weighted sum = (1⋅1)+(1⋅1)=2
•a bias that makes the neuron fire only for the weighted sum of 2.
•If we choose a bias b1=−3/2, the threshold is 1.5 (since −b=−(−3/2)=1.5).
•The neuron's condition becomes: weighted_sum >= 1.5.
•As you can see from our test sums, only the 2 from the (1,1) input is greater than or equal to
1.5. All other sums (0 and 1) are less than 1.5.
 XOR Problem
2. Neuron 2 (Bottom Hidden Neuron)
•Goal: The goal of this neuron is to fire for (0,1), (1,0), and (1,1). It should not fire for (0,0).
•Weights: w21=+1 and w22=+1.
•How the bias was chosen: weighted sums from before:
•Input (0,0): Weighted sum = 0
•Input (0,1): Weighted sum = 1
•Input (1,0): Weighted sum = 1
•Input (1,1): Weighted sum = 2
•We need a bias that makes the neuron fire for weighted sums of 1 and 2, but not for 0.
•If we choose a bias b2=−1/2, the threshold is 0.5 (since −b=−(−1/2)=0.5).
•The neuron's condition becomes: weighted_sum >= 0.5.
•From our test sums, the 0 from the (0,0) input is less than 0.5, so the neuron remains
off. The sums of 1 and 2 are greater than 0.5, so the neuron fires.
•Conclusion: The bias of −1/2 works perfectly to distinguish the (0,0) pattern from the
other three.
XOR Problem
XOR Problem
 HEURISTICS FOR MAKING THE BACK-PROPAGATION
ALGORITHM PERFORM BETTER
 When training a neural network, sometimes it learns too slowly, gets stuck, or
doesn’t generalize well. These heuristics (rules of thumb) are tricks to make
backpropagation training better.
 1. Stochastic vs Batch Update: This is about how often you update the network's
weights.
 Stochastic (Sequential) Update: The network updates its weights after processing
each individual training example
 Batch Update: The network waits to process the entire training dataset (or a large
batch of it) before calculating the average error and updating the weights.
 Stochastic updates are computationally faster and are less likely to get stuck in
"flat" areas of the error surface because the frequent, small updates prevent
the network from settling too quickly. They introduce a bit of noise that helps
the algorithm explore the error landscape more effectively.
 HEURISTICS FOR MAKING THE BACK-
PROPAGATION ALGORITHM PERFORM BETTER
2. Maximizing Information Content
 Not all data points are equally valuable for learning. The best data points are those
that teach the network the most.
 Present the network with examples that result in the largest training error. This
forces the network to focus on the examples it's getting wrong, making it learn
more efficiently.
 Present an example that is radically different from the ones it has just seen. This
prevents the network from getting "stuck" in a local pattern and encourages it to
explore more of the problem space.
 A simple and effective way to do this is to randomize (shuffle) the order of the training
data from one training "epoch" to the next. This ensures that the network is constantly
challenged with new, varied information and doesn't get biased by the order of the
data.
HEURISTICS FOR MAKING THE BACK-
PROPAGATION ALGORITHM PERFORM BETTER
3. Activation Function
 The activation function decides whether a neuron "fires" or not. The choice of
function can significantly affect learning speed.
 Use a hyperbolic tangent function instead of a logistic (sigmoid) function.
 The hyperbolic tangent function is symmetrical around the origin (it's an
"odd" function). This is beneficial because its output range is from -1 to 1,
with a mean of 0. This helps the inputs to subsequent layers also have a
mean of zero, which is a key to faster learning (as explained in point 5).
 Its slope at the origin is close to 1, which means the initial learning steps are
not too slow.
 HEURISTICS FOR MAKING THE BACK-PROPAGATION
ALGORITHM PERFORM BETTER
4. Target Values
 These are the "correct answers" that we give the network during training (e.g., a
target of 1 for a "dog" image).
 Choose target values that are within the active range of the activation function, not
at the extreme limits.
 If you set a target value at the absolute limit (e.g., a target of 1.0 for a sigmoid
function), the network's output neuron will have to get very close to that limit.
 To do this, the neuron's weights must become extremely large. This drives the
neuron into "saturation," where the slope of the activation function becomes very
flat (close to zero).
 Since back-propagation relies on the gradient (the slope) to adjust weights, a zero
gradient means no learning. Setting targets slightly inside the range (e.g., 0.9 for a
sigmoid that maxes out at 1.0) prevents this "saturation" and keeps the learning
process active.
HEURISTICS FOR MAKING THE BACK-PROPAGATION
ALGORITHM PERFORM BETTER
5. Normalizing the Inputs
 This involves preprocessing the input data to make it more suitable for the algorithm.
 Normalize the input variables so that:
 Their mean is close to zero.
 They are uncorrelated with each other.
 They have approximately equal standard deviations (or variances).
 If input variables are all positive, the weight updates in the first hidden layer will all be
the same sign (all positive or all negative). This forces the network to "zigzag" its way
towards a solution, which is very slow. Normalizing to a mean of zero allows the
weights to change in any direction, making the learning path more direct.
 Uncorrelated inputs with equal variances ensure that all the synaptic weights learn at
roughly the same speed. Without this, some weights would get updated very slowly
while others change too quickly, leading to an inefficient training process.
 HEURISTICS FOR MAKING THE BACK-PROPAGATION
ALGORITHM PERFORM BETTER
6. Initialization
 This refers to the initial values given to the synaptic weights before training begins.
 Avoid both very large and very small initial weights.
 Large Weights: If the weights are too large, the neuron's output will likely be
driven to the extreme ends of the activation function (saturation), where the
gradient is near zero. This leads to extremely slow learning.
 Small Weights: If the weights are too small, the network will operate in a very flat
area of the error surface, which can also be slow. It can also cause all neurons in
a hidden layer to learn the same features, making them redundant.
 The text suggests a specific formula (LeCun's rule) for initializing weights. This formula
ensures that the initial standard deviation of the neuron's output is in the "sweet
spot"—the steep, non-linear part of the activation function, where learning can
happen effectively.
HEURISTICS FOR MAKING THE BACK-PROPAGATION
ALGORITHM PERFORM BETTER
7. Learning from Hints
 This is an advanced idea that suggests we can use prior knowledge about
the problem to improve learning, not just the training data.
 Incorporate known properties of the function you're trying to learn (e.g.,
symmetry, invariance) into the learning process.
 This accelerates the training and, more importantly, improves the quality of
the final solution. The initialization heuristic (point 6) is a simple example of
this—we used our knowledge of the activation function to choose a better
starting point for the weights.
HEURISTICS FOR MAKING THE BACK-PROPAGATION
ALGORITHM PERFORM BETTER
8. Learning Rates
 The learning rate determines the size of the steps taken during weight
updates.
 Learning rates should be adapted based on the neuron's position in the
network and the number of its inputs.
 Layer-by-Layer: Later layers (closer to the output) often have larger local
gradients than earlier layers. Therefore, they should be assigned a smaller
learning rate to prevent large, destabilizing updates.
 Number of Inputs: Neurons with many inputs should have a smaller
learning rate to avoid large changes in the total weighted sum. The text
suggests making the learning rate inversely proportional to the square root
of the number of connections.