2.

1 Introduction

• The multilayer perceptron, which stands for a neural network with one or more hidden layers.
• In previous studies, we studied Rosenblatt’s perceptron, which is basically a single-layer neural
network, and this network is limited to the classification of linearly separable patterns.

• The LMS algorithm is also based on a single linear neuron with adjustable weights, which limits
the computing power of the algorithm.

• To overcome the practical limitations of the perceptron and the LMS algorithm, we look to a
neural network structure known as the multilayer perceptron.
2.1.1 The basic features of a multilayer perceptron:

• . The model of each neuron in the network includes a differentiable nonlinear activation function.
• The network contains one or more layers that are hidden from both the input and output nodes.

• The network exhibits a high degree of connectivity, the extent of which is determined by the the
synaptic weights of the network.
The training of multilayer perceptron

• The training of multilayer perceptron is the back-propagation algorithm, which includes the
LMS algorithm.

• The training proceeds in two phases:

1. Forward phase: Input - Output (prediction)

2. Backward pass: Output error - Gradient - Weight update

1. Forward phase:
• In the forward phase, the synaptic weights of the network are fixed

• The input signal is propagated through the network, layer by layer, until it reaches the output.

• Thus, in this phase, changes are confined to the activation potentials and outputs of the
neurons in the network.

• Computes outputs layer by layer until final output.

2. Backward phase
 • In the backward phase, an error signal is produced by comparing the output of the network with
a desired response.

• The resulting error signal is propagated through the network, again layer by layer, but this time
the propagation is performed in the backward direction.

• In this second phase, successive adjustments are made to the synaptic weights of the network.

• In this phase, the error is propagated backward through the network to update the weights.

• The update is done using Gradient Descent.

• Compute errors at the output layer, propagate them back, and update weights using LMS rule.

• Usage of the term “back propagation” appears to have evolved after 1985, when the term was
popularized through the publication of the seminal book entitled Parallel Distributed Processing
(Rumelhart and McClelland, 1986).

• The development of the back-propagation algorithm in the mid-1980s represented a landmark

in neural networks in that it provided a computationally efficient method for the training of
multilayer perceptrons, putting to rest the pessimism about learning in multilayer perceptrons
that may have been inferred from the book by Minsky and Papert (1969).

2.1.2 Multiplayer perceptron

• Multiplayer perceptron with two hidden layers and an output layer.

• Fully connected.
• This means that a neuron in any layer of the network is connected to all the neurons (nodes) in
the previous layer.

• Signal flow through the network progresses in a forward direction, from left to right and on a
layer-by-layer basis.
Two kinds of signals are identified in this network:
1. Function Signals
2. Error Signals.

Function Signals:

• A function signal is an input signal (stimulus) that comes in at the input end of the network,
propagates forward (neuron by neuron) through the network, and emerges at the output end of
the network as an output signal.

• “Function signal” for two reasons.

• First, it is presumed to perform a useful function at the output of the network.
• Second, at each neuron of the network through which a function signal passes, the signal
is calculated as a function of the inputs and associated weights applied to that neuron.

• The function signal is also referred to as the input signal.

Error Signals:

• An error signal originates at an output neuron of the network and propagates backward (layer by
layer) through the network.

• We refer to it as an “error signal” because its computation by every neuron of the network
involves an error-dependent function in one form or another.

• The output neurons constitute the output layer of the network.

• The remaining neurons constitute hidden layers of the network.

• Thus, the hidden units are not part of the output or input of the network—hence their designation
as “hidden.”

• The first hidden layer is fed from the input layer made up of sensory units (source nodes)
• The resulting outputs of the first hidden layer are in turn applied to the next hidden layer and so
on for the rest of the network.
Each hidden or output neuron of a multilayer perceptron is designed to perform two
computations:
1. The computation of the function signal appearing at the output of each neuron, which is
expressed as a continuous nonlinear function of the input signal and synaptic weights associated
with that neuron
2. The computation of an estimate of the gradient vector (i.e., the gradients of the error surface
concerning the weights connected to the inputs of a neuron), which is needed for the backward
pass through the network.

2.2 Batch Learning vs. Online Learning

1. Batch Learning
Definition: The learning algorithm updates the weights after processing the entire training dataset
(one epoch).
• Feed all training samples through the network.

• Compute the total error (e.g., Mean Squared Error).

• Update the weights once per epoch.
2. Online Learning (Stochastic Learning)
Definition: The weights are updated immediately after each training sample is presented.

• For each training example:

• Feed forward.
• Compute error.
• Update weights instantly.

BATCH LEARNING AND ONLINE LEARNING

• Consider a multilayer perceptron with an input layer of source nodes, one or more hidden layers,
and an output layer consisting of one or more neurons
 • Both instantaneous error E(n) and average error depend on the weights of the MLP.
• That’s why we apply backpropagation + gradient descent:
• Compute gradient of error w.r.t. weights.
• Adjust weights to reduce error.

Batch Learning
In batch learning, the weights of the neural network are not updated immediately after each training
sample.
Instead:

. The algorithm waits until all N training samples (one epoch) are processed.
. It calculates the average error energy (eav) across the whole dataset.
· Then performs one weight update step (using gradient descent).
This is why the cost function for batch learning is defined as:

where e(n) is the error for the n-th training sample.

Example:

• Training a CNN for medical image classification:

 • Hospital dataset of X-ray images (fixed, not changing).

• Whole dataset is used → update weights after each epoch.

• Goal: precise convergence to minimize diagnosis errors.

Online Learning (Stochastic Learning)

In online learning, the network updates its weights immediately after each training example, instead of
waiting for the whole dataset (epoch).
. Cost function minimized: instantaneous error energy

where d(n) is desired output and y(n) is network output for input x(n).

Example:

• Spam email filtering:

• New emails arrive continuously.

• Update model weights immediately after each email is classified.

• Goal: adapt quickly to new spam techniques (nonstationary environment).
Difference between Batch Learning and On-line Learning
.
2 .3 THE BACK-PROPAGATION ALGORITHM
• Backpropagation is the learning algorithm used to train multilayer perceptrons (MLPs).
• It updates the weights of the network using gradient descent by propagating the error backwards
from the output layer to the hidden layers.
Partially differentiating each term in Eqn(2.13)
Two cases of Local Gradient
FIGURE 4.5 Signal-flow graph of a part of the adjoint system pertaining to backpropagation of error
signals.

The Two Passes of Computation

Activation Function
An activation function is a mathematical function applied to the weighted sum of a neuron’s
inputs in a neural network to determine its output signal.
It introduces non-linearity into the model so the network can learn complex patterns rather than just
simple linear relationships.
Each neuron in a multilayer perceptron (MLP) requires the derivative of its activation function
ϕ(⋅) to compute weight updates.
XOR PROBLEM

Single-layer perceptron limitation

• Rosenblatt’s perceptron has no hidden layer.

• Its decision boundary is a straight line in 2-D input space (or a hyperplane in higher dimensions).
• Therefore, it can only classify linearly separable patterns.

XOR problem

• Inputs: four corners of the unit square — (0,0), (0,1), (1,0), (1,1).
• Desired outputs:

• Class 0 → (0,0) and (1,1).

• Class 1 → (0,1) and (1,0).
• These points cannot be separated by a single straight line → not linearly separable.

• A single neuron produces output 1 on one side of its linear boundary and 0 on the [Link] matter
how the line is placed, it cannot group (0,0) with (1,1) separately from (0,1) and (1,0).

• Introduce one hidden layer with two neurons (Touretzky & Pomerleau, 1989). Each hidden
neuron implements its own linear boundary. The combination of their outputs allows the network
to create a non-linear (piecewise linear) decision region that solves XOR.
Hidden neurons are McCulloch–Pitts units using a threshold activation. Logical bits:
0 → level 0, 1 → level +1.
One hidden neuron creates a boundary of slope −1; the second neuron forms a
complementary boundary. Proper weights and biases (e.g., b2=-1/2) position these boundaries
so their outputs can be combined in the output neuron to implement the XOR function.
•

The top neuron, labeled as “Neuron 1” in the hidden layer, is characterized as

The slope of the decision boundary constructed by this hidden neuron is equal to -1
and positioned as in fig(a).The bottom neuron, labeled as “Neuron 2” in the hidden
layer, is characterized as
 The output neuron, labeled as “Neuron 3” in Fig. 4.8a, is characterized as

The function of the output neuron is to construct a linear combination of the decision
boundaries formed by the two hidden [Link] result of this computation is shown
in Fig. [Link] bottom hidden neuron has an excitatory (positive) connection to the output
neuron, whereas the top hidden neuron has an inhibitory (negative) connection to
the output [Link] both hidden neurons are off, which occurs when the input pattern
is (0,0), the output neuron remains [Link] both hidden neurons are on, which
occurs when the input pattern is (1,1), the output neuron is switched off again because
the inhibitory effect of the larger negative weight connected to the top hidden neuron
overpowers the excitatory effect of the positive weight connected to the bottom hidden
[Link] the top hidden neuron is off and the bottom hidden neuron is on, which
occurs when the input pattern is (0,1) or (1,0), the output neuron is switched on because
of the excitatory effect of the positive weight connected to the bottom hidden neuron.
Thus, the network of Fig. 4.8a does indeed solve the XOR problem.

So:
• Hidden Neuron 1 = First line (roughly selects one diagonal).

• Hidden Neuron 2 = Second line (roughly selects the other diagonal).

 • Output Neuron = Combines these two decisions (one excitatory weight, one inhibitory weight)
to realize XOR.

Thus, we need two lines because XOR is not linearly separable. Each hidden neuron contributes one
line (one linear cut), and their combination allows the network to create a nonlinear decision boundary
that solves XOR.

Problem:

• Figure P4.1 shows a neural network involving a single hidden neuron for solving the XOR
problem; this network may be viewed as an alternative to that considered in Section 4.5. Show
that the network of Fig. P4.1 solves the XOR problem by constructing
(a) decision regions and

(b) a truth table for the network.

Heuristics for making the back-propagation algorithm

perform better

1. Stochastic versus batch update

2. Maximizing information content- the largest possible for the task

3. Activation function
4. Target values.

5. Normalizing the inputs

6. Initialization.

7. Learning from hints

8. Learning rates

1. Stochastic versus batch update:

Stochastic (sequential) mode: Updates network weights after every training pattern.

Batch mode: Accumulates gradients over the entire training set and updates once per epoch.

2. Maximizing information content.

Every training example presented to the back-propagation algorithm should be chosen on the
basis that its information content is the largest possible for the task at hand (LeCun, 1993).
Two ways of realizing this choice are as follows:
• Use an example that results in the largest training error.
• Use an example that is radically different from all those previously used.
 These two heuristics are motivated by a desire to search more of the weight space. In pattern-
classification tasks using sequential back-propagation learning, a simple and commonly used
technique is to randomize (i.e., shuffle) the order in which the examples are presented to the
multilayer perceptron from one epoch to the next. Ideally, the randomization ensure that
successive examples in an epoch presented to the network rarely belong to the same class.

3. Activation function.
Insofar as the speed of learning is concerned, the preferred choice is to use a sigmoid activation
function that is an odd function of its argument, as shown by

as shown in Fig. 4.10, but not the logistic function. Suitable values for the
constraints a and b in the formula for _(v) are as follows (LeCun, 1993):

The hyperbolic tangent function _(v) of Fig. 4.10 has the following useful properties:
 4. Target values. It is important that the target values (desired response) be chosen within the range
of the sigmoid activation function. More specifically, the desired response dj for neuron j in the
output layer of the multilayer perceptron should be offset by some amount _ away from the
limiting value of the sigmoid activation function, depending on whether the limiting value is
positive or negative. Otherwise, the backpropagation algorithm tends to drive the free parameters
of the network to infinity and thereby slow down the learning process by driving the hidden
neurons into saturation. To be specific, consider the hyperbolic tangent function of Fig. 4.10. For
the limiting value _a, we set

5. Normalizing the inputs. Each input variable should be preprocessed so that its mean value, averaged
over the entire training sample, is close to zero, or else it will be small compared to its standard deviation
(LeCun, 1993).To appreciate the practical significance of this rule, consider the extreme case where the
input variables are consistently positive. In this situation, the synaptic weights of a neuron in the first
hidden layer can only increase together or decrease together. Accordingly, if the weight vector of that
neuron is to change direction, it can do so only by zigzagging its way through the error

surface, which is typically slow and should therefore be avoided.

In order to accelerate the back-propagation learning process, the normalization of

the inputs should also include two other measures (LeCun, 1993):

• The input variables contained in the training set should be uncorrelated; this can be done by using
principal-components analysis.

• The decorrelated input variables should be scaled so that their covariances are approximately equal,
thereby ensuring that the different synaptic weights in the network learn at approximately the same
speed.
 Figure 4.11 illustrates the results of three normalization steps: mean removal, decorrelation,

and covariance equalization, applied in that order.

6. Initialization. A good choice for the initial values of the synaptic weights and thresholds of the
network can be of tremendous help in a successful network design.

When the synaptic weights are assigned large initial values, it is highly likely that the neurons in
the network will be driven into saturation. If this happens, the local gradients in the backpropagation
algorithm assume small values, which in turn will cause the learning process to slow down. However, if
the synaptic weights are assigned small initial values, the back-propagation algorithm may operate on a
very flat area around the origin of the error surface; this is particularly true in the case of sigmoid

functions such as the hyperbolic tangent function. Unfortunately, the origin is a saddle point, which
refers to a stationary point where the curvature of the error surface across the saddle is negative and the
curvature along the saddle is positive. For these reasons, the use of both large and small values for
initializing the synaptic weights should be avoided. The proper choice of initialization lies somewhere
between these two extreme cases.
To be specific, consider a multilayer perceptron using the hyperbolic tangent function for its activation
functions. Let the bias applied to each neuron in the network be set to zero.
The induced local field of neuron j as

The mean and variance of the induced local field vj as

7. Learning from hints.
Learning from a sample of training examples deals with an unknown input–output mapping
function f(·). In effect, the learning process exploits the information contained in the examples about the
function f(·) to infer an approximate implementation of it. The process of learning from examples may
be generalized to include learning from hints, which is achieved by allowing prior information that we
may have about the function f(·) to be included in the learning process. Such information may include
invariance properties, symmetries, or any other knowledge about the function f(·) that may be used to
accelerate the search for its approximate realization.

[Link] rates.

All neurons in the multilayer perceptron should ideally learn at the same rate. The last layers
usually have larger local gradients than the layers at the front end of the network. Hence, the learning-
rate parameter _ should be assigned a smaller value in the last layers than in the front layers of the
multilayer perceptron. smaller value in the last layers than in the front layers of the multilayer
perceptron.

Neurons with many inputs should have a smaller learning-rate parameter than neurons with few
inputs so as to maintain a similar learning time for all neurons in the network. In LeCun (1993), it is
suggested that for a given neuron, the learning rate should be inversely proportional to the square root
of synaptic connections made to that neuron.

4.7 BACK PROPAGATION AND DIFFERENTIATION

• Back propagation is a specific technique for implementing gradient descent in weight space for
a multilayer perceptron.

• The basic idea is to efficiently compute partial derivatives of an approximating function F(w, x)
realized by the network with respect to all the elements of the adjustable weight vector w for a
given value of input vector x.
• Back-propagation is used because it gives an exact analytic gradient of the network output (or
loss) with respect to all weights using one forward pass + one backward pass, exploiting the
chain rule and reuse of intermediate results — making gradient-based learning feasible for
multilayer networks with many parameters.

• Back-propagation is the efficient way to compute every partial derivative

• The gradient of the network output or of the loss for a given input x.

• These partials are exactly the information gradient-descent (or second-order) optimizers need to
change the weights w so the network output moves toward the desired target.

Figure: Multilayer perceptron with two hidden layers and one output neuron.