Artificial Neural Networks

(CSA4002)
Learning Processes:
Learning Processes: Introduction, Error correction learning,
Memory-based learning, Hebbian learning, Competitive learning,
Boltzmann learning, credit assignment problem, Learning with and
without teacher, learning tasks, Memory and Adaptation.
Learning Processes:

The significance property of ANN

-is the ability of the network to learn from its environment, and
-to improve its performance through learning.

A neural network learns by

-adjustments applied to its synaptic weights and bias levels.
-becomes more knowledgeable after each iteration of the learning process.

Neural Networks-Comprehensive Foundation by Simon Haykin

Learning Processes:

The significance property of ANN

-is the ability of the network to learn from its environment, and
-to improve its performance through learning.

A neural network learns by

-adjustments applied to its synaptic weights and bias levels.
M
-becomes more knowledgeable after each iteration of the learning process.
oreov
more er, the p
d r
viewe ifficult to ocess of
[Link] le
a psy agree on arning is
cholo a
gist is precise a matter
quite de of
differ finition of viewpoin
ent fr t,
om le the term. which m
arnin F a
g in a or examp kes it all
classr le th
oom , learning e
sense as
.
Neural Networks-Comprehensive Foundation by Simon Haykin
Learning Processes:

A definition of learning that is adapted from Mendel and McClaren (1970).

“Learning is a process by which the free parameters of a neural network

are adapted through a process of stimulation by the environment in
which the network is embedded. The type of learning is determined by
the manner in which the parameter changes take place.”

Neural Networks-Comprehensive Foundation by Simon Haykin

Learning Processes:

A definition of learning that is adapted from Mendel and McClaren (1970).

“Learning is a process by which the free parameters of a neural network

are adapted through a process of stimulation by the environment in
which the network is embedded. The type of learning is determined by
the manner in which the parameter changes take place.”

Neural Networks-Comprehensive Foundation by Simon Haykin

Learning Processes:

Learning process implies the following sequence of events:

1. The neural network is stimulated by an environment.
2. The neural network undergoes changes in its free parameters as a
result of this stimulation.
3. The neural network responds in a new way to the environment
because of the changes that have occurred in its internal structure.

A prescribed set of well-defined rules for the solution of a learning

problem is called a learning algorithm.

Neural Networks-Comprehensive Foundation by Simon Haykin

Learning Processes:

A prescribed set of well-defined rules for the solution of a learning

problem is called a learning algorithm.
learn
in
the a g algorith
djust
ment ms differ
to a s fr
ynap om each o
tic w
eight ther in th
of a n e
euron way in w
is for hic
mula h
ted.

Another factor to be considered is the manner in which a

neural network (learning machine), made up of a set of
interconnected neurons, relates to its environment.

Neural Networks-Comprehensive Foundation by Simon Haykin

ERROR-CORRECTION LEARNING
To illustrate our first learning rule,
consider the simple case of a neuron k constituting the only computational node in the output
layer of a feedforward neural network, as depicted in Fig..
Neuron k is driven by a signal vector x(n) produced by one or more layers of hidden neurons,
which are themselves driven by an input vector (stimulus) applied to the source nodes (i.e.,
input layer) of the neural network.

Neural Networks-Comprehensive Foundation by Simon Haykin

ERROR-CORRECTION LEARNING
To illustrate our first learning rule,

The argument n denotes discrete time, or more precisely, the time step of an iterative process
involved in adjusting the synaptic weights of neuron k.
The output signal of neuron k is denoted by yk(n). This output signal, representing the only
output of the neural network, is compared to a desired response or target output, denoted by
dk(n). Consequently, an error signal, denoted by ek(n),is produced.

By definition, we thus have

Neural Networks-Comprehensive Foundation by Simon Haykin

ERROR-CORRECTION LEARNING

The error signal ek(n) actuates a control mechanism, the purpose of which is to apply
a sequence of corrective adjustments to the synaptic weights of neuron k. The
corrective adjustments are designed to make the output signal y k(n) come closer to
the desired response dk(n) in a step-by-step manner. This objective is achieved by
minimizing a cost function or index of performance, , defined in terms of the
error signal ek(n) as:

That is, is the instantaneous value of the error energy. The step-by-step
adjustments to the synaptic weights of neuron k are continued until the
system reaches a steady state (i.e.,the synaptic weights are essentially
stabilized). At that point the learning process is terminated.

Neural Networks-Comprehensive Foundation by Simon Haykin

ERROR-CORRECTION LEARNING
The learning process described herein is obviously referred to as error-correction learning. In
particular, minimization of the cost function leads to a learning rule commonly referred
to as the delta rule or Widrow-Hoff rule, named in honor of its originators (Widrow and Hoff,
1960).
Let wkj(n) denote the value of synaptic weight wkj of neuron k excited by element xj (n) of the
signal vector x(n) at time step n. According to the delta rule, the adjustment Δw kj (n) applied
to the synaptic weight wkj at time step n is defined by

where η is a positive constant that determines the rate of learning as we proceed

from one step in the learning process to another. It is therefore natural that we refer
to η as the learning-rate parameter.

In other words, the delta rule may be stated as:

The adjustment made to a synaptic weight of a neuron is proportional to the
product of the error signal and the input signal of the synapse in question.

Neural Networks-Comprehensive Foundation by Simon Haykin

ERROR-CORRECTION LEARNING
Having computed the synaptic adjustment Δwkj(n), the updated
value of synaptic weight wkj, is determined by

In effect, wkj (n) and wkj (n +1) may be viewed as the old and
new values of synaptic weight w respectively.

Figure shows a signal-flow graph

representation of the error-correction
learning process, focusing on the
activity surrounding neuron k.

Neural Networks-Comprehensive Foundation by Simon Haykin

MEMORY-BASED LEARNING
In memory-based learning, all (or most) of the past experiences are explicitly
stored in a large memory of correctly classified input-output examples:

where x denotes an input vector and d, denotes the corresponding desired

response.

Without loss of generality, we have restricted the desired response to be a scalar.

For example, in a binary pattern classification problem there are two

classes/hypotheses, denoted by 1 and €2, to be considered. In this example, the
desired response d takes the value 0 (or -1) for class , and the value 1 for class .
When classification of a test vector xtest (not seen before) is required, the algorithm
responds by retrieving and analyzing the training data in a "local neighborhood" of
xtest

Neural Networks-Comprehensive Foundation by Simon Haykin

MEMORY-BASED LEARNING

All memory-based learning algorithms involve two essential ingredients:

• Criterion used for defining the local neighborhood of the test vector
xtest
• Learning rule applied to the training examples in the local
neighborhood of Xtest

Neural Networks-Comprehensive Foundation by Simon Haykin

MEMORY-BASED LEARNING

In a simple yet effective type of memory-based learning known as the

nearest neighbor rule, the local neighborhood is defined as the training
example that lies in the immediate neighborhood of the test vector x test. In
particular, the vector

is said to be the nearest neighbor of xtest if

where d(xi, xtest) is the Euclidean distance between the vectors xi, and x test.
The class associated with the minimum distance, that is, vector x, is reported
as the classification of xtest. This rule is independent of the underlying
distribution responsible for generating the training examples.

Neural Networks-Comprehensive Foundation by Simon Haykin

MEMORY-BASED LEARNING
Cover and Hart (1967) have formally studied the nearest neighbor rule as a tool for
pattern classification. The analysis presented therein is based on two assumptions:
The classified examples (xi, di) are independently and identically distributed (iid),
according to the joint probability distribution of the example (x, d).
The sample size N is infinitely large.

Neural Networks-Comprehensive Foundation by Simon Haykin

MEMORY-BASED LEARNING
Cover and Hart (1967) have formally studied the nearest neighbor rule as a tool for
pattern classification. The analysis presented therein is based on two assumptions:
The classified examples (xi, di) are independently and identically distributed (iid),
according to the joint probability distribution of the example (x, d).
The sample size N is infinitely large.

A variant of the nearest neighbor classifier is the k-nearest

neighbor classifier, which proceeds as follows:
• Identify the k classified patterns that lie nearest to the test
vector xtest for some integer k.
• Assign Xtest to the class (hypothesis) that is most frequently
represented in the k nearest neighbors to xtest (i.e., use a
majority vote to make the classification).

FIGURE : The area lying inside the dashed circle includes two
points pertaining to class 1 and an outlier from class 0. The
point d corresponds to the test vector Xtest.
With k = 3, the k-nearest neighbor classifier assigns class 1 to
point d even though it lies closest to the outlier.

Neural Networks-Comprehensive Foundation by Simon Haykin

COMPETITIVE LEARNING

In competitive learning, as the name implies, the output

neurons of a neural network compete among themselves to
become active (fired).

Whereas in a neural network based on Hebbian learning several

output neurons may be active simultaneously, in competitive
learning only a single output neuron is active at any one time.

It is this feature that makes competitive learning highly suited

to discover statistically salient features that may be used to
classify a set of input patterns.

Neural Networks-Comprehensive Foundation by Simon Haykin

COMPETITIVE LEARNING
There are three basic elements to a competitive learning rule
(Rumelhart and Zipser, 1985):

A set of neurons that are all the same except for some
randomly distributed synaptic weights, and which therefore
respond differently to a given set of input patterns.
A limit imposed on the "strength" of each neuron.
A mechanism that permits the neurons to compete for the
right to respond to a given subset of inputs, such that only
one output neuron, or only one neuron per group, is active FIGURE: Architectural
(i.e., "on") at a time. The neuron that wins the competition graph of a simple
is called a winner-takes-all neuron. competitive learning
network with feedforward
Accordingly the individual neurons of the network learn to (excitatory) connections
from the source nodes to
specialize on ensembles of similar patterns; in so doing they
the neurons, and lateral
become feature detectors for different classes of input (inhibitory) connections
patterns. among the neurons; the
lateral connections are
signified by open arrows.

Neural Networks-Comprehensive Foundation by Simon Haykin

COMPETITIVE LEARNING
In the simplest form of competitive learning, the neural
network has a single layer of output neurons, each of
which is fully connected to the input nodes.
The network may include feedback connections among
the neurons, as indicated in Fig..

In the network architecture described herein, the

feedback connections perform lateral inhibition, with
each neuron tending to inhibit the neuron to which it is FIGURE: Architectural
laterally connected. In contrast, the feedforward graph of a simple
synaptic connections in the network of Fig. are all competitive learning
network with feedforward
excitatory. (excitatory) connections
from the source nodes to
the neurons, and lateral
(inhibitory) connections
among the neurons; the
lateral connections are
signified by open arrows.

Neural Networks-Comprehensive Foundation by Simon Haykin

COMPETITIVE LEARNING
For a neuron k to be the winning neuron, its induced
local field vk for a specified input pattern x must be the
largest among all the neurons in the network. The output
signal yk of winning neuron k is set equal to one; the
output signals of all the neurons that lose the
competition are set equal to zero. We thus write

FIGURE: Architectural
graph of a simple
competitive learning
network with feedforward
(excitatory) connections
from the source nodes to
where the induced local field vk represents the the neurons, and lateral
combined action of all the forward and feedback inputs (inhibitory) connections
to neuron k. among the neurons; the
lateral connections are
signified by open arrows.

Neural Networks-Comprehensive Foundation by Simon Haykin

COMPETITIVE LEARNING
Let wkj denote the synaptic weight connecting input
node j to neuron k. Suppose that each neuron is allotted
a fixed amount of synaptic weight (i.e., all synaptic
weights are positive), which is distributed among its
input nodes; that is,

FIGURE: Architectural
graph of a simple
competitive learning
network with feedforward
(excitatory) connections
from the source nodes to
the neurons, and lateral
(inhibitory) connections
among the neurons; the
lateral connections are
signified by open arrows.

Neural Networks-Comprehensive Foundation by Simon Haykin

COMPETITIVE LEARNING
A neuron then learns by shifting synaptic weights from
its inactive to active input nodes. If a neuron does not
respond to a particular input pattern, no learning takes
place in that neuron. If a particular neuron wins the
competition, each input node of that neuron relinquishes
some proportion of its synaptic weight, and the weight
relinquished is then distributed equally among the active
input nodes. According to the standard competitive
learning rule, the change Δwkj applied to synaptic FIGURE: Architectural
graph of a simple
weight wkj is defined by competitive learning
network with feedforward
(excitatory) connections
from the source nodes to
the neurons, and lateral
where η is the learning-rate parameter. This rule has the (inhibitory) connections
overall effect of moving the synaptic weight vector wk of among the neurons; the
lateral connections are
winning neuron k toward the input pattern x. signified by open arrows.

Neural Networks-Comprehensive Foundation by Simon Haykin

COMPETITIVE LEARNING
It is assumed that each input pattern (vector) x has some constant Euclidean length
so that we may view it as a point on an N-dimensional unit sphere where N is the
number of input nodes. N also represents the dimension of each synaptic weight
vector w.
It is further assumed that all neurons in the network are constrained to have the
same Euclidean length (norm), as shown by

When the synaptic weights are properly scaled they form a set of vectors that fall on
the same N-dimensional unit sphere.

FIGURE: Geometric interpretation of the

competitive learning process. The dots
represent the input vectors, and the crosses
represent the synaptic weight vectors of three
output neurons. (a) Initial state of the network.
(b) Final state of the network.

Neural Networks-Comprehensive Foundation by Simon Haykin

COMPETITIVE LEARNING
In Fig. a, it show three natural groupings (clusters) of the stimulus patterns
represented by dots. This figure also includes a possible initial state of the network
(represented by crosses) that may exist before learning. Figure b shows a typical
final state of the network that results from the use of competitive learning.

In particular, each output neuron has discovered a cluster of input patterns by

moving its synaptic weight vector to the center of gravity of the discovered cluster
(Rumelhart and Zipser, 1985; Hertz et al., 1991).

FIGURE: Geometric interpretation of the

competitive learning process. The dots
represent the input vectors, and the crosses
represent the synaptic weight vectors of three
output neurons. (a) Initial state of the network.
(b) Final state of the network.

Neural Networks-Comprehensive Foundation by Simon Haykin

COMPETITIVE LEARNING
This figure illustrates the ability of a neural network to perform clustering through
competitive learning. However, for this function to be performed in a "stable"
fashion the input patterns must fall into sufficiently distinct groupings to begin with.
Otherwise the network may be unstable because it will no longer respond to a given
input pattern with the same output neuron.

FIGURE: Geometric interpretation of the

competitive learning process. The dots
represent the input vectors, and the crosses
represent the synaptic weight vectors of three
output neurons. (a) Initial state of the network.
(b) Final state of the network.

Neural Networks-Comprehensive Foundation by Simon Haykin

BOLTZMANN LEARNING
The Boltzmann learning rule, named in honor of Ludwig Boltzmann, is a stochastic
learning algorithm derived from ideas rooted in statistical mechanics.

A neural network designed on the basis of the Boltzmann learning rule is called a
Boltzmann machine (Ackley et al., 1985; Hinton and Sejnowski, 1986).

Neural Networks-Comprehensive Foundation by Simon Haykin

BOLTZMANN LEARNING
The Boltzmann learning rule, named in honor of Ludwig Boltzmann, is a stochastic
learning algorithm derived from ideas rooted in statistical mechanics.

A neural network designed on the basis of the Boltzmann learning rule is called a
Boltzmann machine (Ackley et al., 1985; Hinton and Sejnowski, 1986).

In a Boltzmann machine the neurons constitute a recurrent structure, and they

operate in a binary manner since, for example, they are either
• in an "on" state denoted by +1 or
• in an "off" state denoted by -1.
The machine is characterized by an energy function, E, the value of which is
determined by the particular states occupied by the individual neurons of the
machine, as shown by

where x , is the state of neuron j, and wkj is the synaptic weight connecting
j
neuron j to neuron k. The fact that j ≠ k means simply that none of the
neurons in the machine has self-feedback.
Neural Networks-Comprehensive Foundation by Simon Haykin
BOLTZMANN LEARNING

The machine operates by choosing a neuron at random-for example, neuron

k-at some step of the learning process, then flipping the state of neuron k
from state xk to state -xk at some temperature T with probability

where ΔEk is the energy change (i.e.,the change in the energy function of the
machine) resulting from such a flip. Notice that T is not a physical
temperature, but rather a pseudotemperature. If this rule is applied
repeatedly, the machine will reach thermal equilibrium.

that is used to control the noise level and therefore the

uncertainty in firing.

Neural Networks-Comprehensive Foundation by Simon Haykin

BOLTZMANN LEARNING
The neurons of a Boltzmann machine partition into two functional groups:
• The visible neurons provide an interface between the network and the
environment in which it operates,
• The hidden neurons always operate freely.

There are two modes of operation to be considered:

• Clamped condition, in which the visible neurons are all clamped onto
specific states determined by the environment.
• Free-running condition, in which all the neurons (visible and hidden) are
allowed to operate freely.

Neural Networks-Comprehensive Foundation by Simon Haykin

BOLTZMANN LEARNING

Neural Networks-Comprehensive Foundation by Simon Haykin

CREDIT-ASSIGNMENT PROBLEM
When studying learning algorithms for distributed systems, it is useful to consider
the notion of credit assignment (Minsky, 1961). Basically, the credit-assignment
problem is the problem of assigning credit or blame for overall outcomes to each of
the internal decisions made by a learning machine and which contributed to those
outcomes.
(The credit assignment problem is also referred to as the loading problem, the
problem of "loading" a given set of training data into the free parameters of the
network.)

Neural Networks-Comprehensive Foundation by Simon Haykin

CREDIT-ASSIGNMENT PROBLEM
In many cases the dependence of outcomes on internal decisions is mediated by a
sequence of actions taken by the learning machine. In other words, internal
decisions affect which particular actions are taken, and then the actions, not the
internal decisions, directly influence overall outcomes.
In these situations, we may decompose the credit-assignment problem into two
subproblems (Sutton, 1984):
1. The assignment of credit for outcomes to actions. This is called the temporal
credit-assignment problem in that it involves the instants of time when the
actions that deserve credit were actually taken.
2. The assignment of credit for actions to internal decisions. This is called the
structural credit-assignment problem in that it involves assigning credit to the
internal structures of actions generated by the system.

Neural Networks-Comprehensive Foundation by Simon Haykin

CREDIT-ASSIGNMENT PROBLEM
The structural credit-assignment problem is relevant in the context of a
multicomponent learning machine when we must determine precisely which
particular component of the system should have its behavior altered and by how
much in order to improve overall system performance.

On the other hand, the temporal credit-assignment problem is relevant when there
are many actions taken by a learning machine that result in certain outcomes, and
we must determine which of these actions were responsible for the outcomes.

The combined temporal and structural credit-assignment problem faces any

distributed learning machine that attempts to improve its performance in situations
involving temporally extended behavior (Williams, 1988).

Neural Networks-Comprehensive Foundation by Simon Haykin

CREDIT-ASSIGNMENT PROBLEM
The credit-assignment problem, for example, arises when error-correction learning is applied
to a multilayer feedforward neural network. The operation of each hidden neuron, as well as
that of each output neuron in such a network is important to its correct overall operation on a
learning task of interest. That is, in order to solve the prescribed task the network must assign
certain forms of behavior to all of its neurons through the specification of error-correction
learning.
With this background in mind, consider the situation
described in Fig. Since the output neuron k is visible to the
outside world, it is possible to supply a desired response to
this neuron. As far as the output neuron is concerned, it is
a straightforward matter to adjust the synaptic weights of
the output neuron in accordance with error-correction
learning.
But how do we assign credit or blame for the action of the
hidden neurons when the error-correction learning
process is used to adjust the respective synaptic weights of
these neurons?

Neural Networks-Comprehensive Foundation by Simon Haykin

CREDIT-ASSIGNMENT PROBLEM

Neural Networks-Comprehensive Foundation by Simon Haykin

LEARNING WITH A TEACHER
Learning with a teacher, also referred to as supervised learning.
In conceptual terms, we may think of the teacher as having knowledge of the environment, with that
knowledge being represented by a set of input-output examples. The environment is, however, unknown
to the neural network of interest.
Suppose now that the teacher and the neural network are both exposed to a training vector (i.e., example)
drawn from the environment. By virtue of built-in knowledge, the teacher is able to provide the neural
network with a desired response for that training vector.
Indeed, the desired response represents the optimum action to be performed by the neural network. The
network parameters are adjusted under the combined influence of the training vector and the error signal.

FIGURE: Block diagram of learning with a

teacher.

Neural Networks-Comprehensive Foundation by Simon Haykin

LEARNING WITH A TEACHER
The error signal is defined as the difference between the desired response and the actual response of
the network. This adjustment is carried out iteratively in a step-by-step fashion with the aim of
eventually making the neural network emulate the teacher; the emulation is presumed to be
optimum in some statistical sense.
In this way knowledge of the environment available to the teacher is transferred to the neural
network through training as fully as possible.
When this condition is reached, we may then dispense with the teacher and let the neural network
deal with the environment completely by itself.

FIGURE: Block diagram of learning with a

teacher.

Neural Networks-Comprehensive Foundation by Simon Haykin

LEARNING WITH A TEACHER
It is a closed-loop feedback system, but the unknown environment is not in the loop. As a performance
measure for the system we may think in terms of the mean-square error or the sum of squared errors over
the training sample, defined as a function of the free parameters of the system.
This function may be visualized as a multidimensional error-performance surface or simply error surface,
with the free parameters as coordinates. The true error surface is averaged over all possible input-output
examples. Any given operation of the system under the teacher's supervision is represented as a point on
the error surface.

FIGURE: Block diagram of learning with a

teacher.

Neural Networks-Comprehensive Foundation by Simon Haykin

LEARNING WITH A TEACHER
For the system to improve performance over time and therefore learn from the
teacher, the operating point has to move down successively toward a minimum point of
the error surface; the minimum point may be a local minimum or a global minimum.
A supervised learning system is able to do this with the useful information it has about
the gradient of the error surface corresponding to the current behavior of the system.
The gradient of an error surface at any point is a vector that points in the direction of
steepest descent.
In fact, in the case of supervised learning from examples, the system may use an
instantaneous estimate of the gradient vector, with the example indices presumed to
be those of time. The use of such an estimate results in a motion of the operating point
on the error surface that is typically in the form of a. "random walk.“
Nevertheless, given an algorithm designed to minimize the cost function, an adequate
set of input-output examples, and enough time permitted to do the training, a
supervised learning system is usually able to perform such tasks as pattern classification
and function approximation.

Neural Networks-Comprehensive Foundation by Simon Haykin

LEARNING WITHOUT A TEACHER
learning without a teacher, as the name implies, there is no teacher to oversee the
learning process

there are no labeled examples of the function to be learned by the network.

two subdivisions are identified:

1. Reinforcement learning / Neurodynamic programming
2. Unsupervised Learning

Neural Networks-Comprehensive Foundation by Simon Haykin

LEARNING WITHOUT A TEACHER
1. Reinforcement learning / Neurodynamic programming

In reinforcement learning, the learning of an input-output mapping is performed

through continued interaction with the environment in order to minimize a scalar
index of performance.
Figure shows the block diagram of one form of a reinforcement learning system built
around a critic that converts a primary reinforcement signal received from the
environment into a higher quality reinforcement signal called the heuristic
reinforcement signal, both of which are scalar inputs (Barto et al., 1983).

FIGURE Block diagram of reinforcement

learning.

Neural Networks-Comprehensive Foundation by Simon Haykin

LEARNING WITHOUT A TEACHER
1. Reinforcement learning / Neurodynamic programming

The system is designed to learn under delayed reinforcement, which means that the
system observes a temporal sequence of stimuli (i.e., state vectors) also received
from the environment, which eventually result in the generation of the heuristic
reinforcement signal.
The goal of learning is to minimize a cost-to-go function, defined as the expectation
of the cumulative cost of actions taken over a sequence of steps instead of simply
the immediate cost. It may turn out that certain actions taken earlier in that
sequence of time steps are in fact the best determinants of overall system behavior.

The function of the learning

FIGURE Block diagram of reinforcement
machine, which constitutes the learning.
second component of the
system, is to discover these
actions and to feed them back
to the environment.

Neural Networks-Comprehensive Foundation by Simon Haykin

LEARNING WITHOUT A TEACHER
1. Reinforcement learning / Neurodynamic programming

Delayed-reinforcement learning is difficult to perform for two basic reasons:

• There is no teacher to provide a desired response at each step of the learning
process.
• The delay incurred in the generation of the primary reinforcement signal implies
that the learning machine must solve a temporal credit assignment problem. By
this we mean that the learning machine must be able to assign credit and blame
individually to each action in the sequence of time steps that led to the final
outcome, while the primary reinforcement may only evaluate the outcome.

Notwithstanding these difficulties, delayed-reinforcement learning is very appealing.

It provides the basis for the system to interact with its environment, thereby
developing the ability to learn to perform a prescribed task solely on the basis of the
outcomes of its experience that result from the interaction.

Neural Networks-Comprehensive Foundation by Simon Haykin

LEARNING WITHOUT A TEACHER
1. Reinforcement learning / Neurodynamic programming

Reinforcement learning is closely related to dynamic programming, which was

developed by Bellman (1957) in the context of optimal control theory.
Dynamic programming provides the mathematical formalism for sequential decision
making. By casting reinforcement learning within the framework of dynamic
programming, the subject matter becomes all the richer for it, as demonstrated in
Bertsekas and Tsitsiklis (1996).

Neural Networks-Comprehensive Foundation by Simon Haykin

LEARNING WITHOUT A TEACHER
2. Unsupervised Learning
In unsupervised or self-organized learning there is no external teacher or
critic to oversee the learning process, as indicated in Fig.
Rather, provision is made for a task-independent measure of the quality of
representation that the network is required to learn, and the free
parameters of the network are optimized with respect to that measure.
Once the network has become tuned to the statistical regularities of the
input data, it develops the ability to form internal representations for
encoding features of the input and thereby to create new classes
automatically (Becker, 1991)

FIGURE: Block diagram of unsupervised

learning.

Neural Networks-Comprehensive Foundation by Simon Haykin

LEARNING WITHOUT A TEACHER
2. Unsupervised Learning
To perform unsupervised learning we may use a competitive learning rule.
For example, we may use a neural network that consists of two layers-an
input layer and a competitive layer.
• The input layer receives the available data.
• The competitive layer consists of neurons that compete with each other
(in accordance with a learning rule) for the "opportunity" to respond to
features contained in the input data.
In its simplest form, the network operates in accordance with a
"winner-takes-all" strategy. In such a strategy the neuron with the greatest
total input "wins" the competition and turns on; all the other neurons then
switch off.

FIGURE: Block diagram of unsupervised

learning.

Neural Networks-Comprehensive Foundation by Simon Haykin

LEARNING TASKS
There are six learning tasks that apply to the use of neural networks in one
form or another.
Pattern Association
Pattern Recognition
Function Approximation
Control
Filtering
Beamforming

Neural Networks-Comprehensive Foundation by Simon Haykin

LEARNING TASKS
There are six learning tasks that apply to the use of neural networks in one
form or another.
Pattern Association: An associative memory is a brainlike distributed
memory that learns by association. Association has been known to be a
prominent feature of human memory since Aristotle, and all models of
cognition use association in one form or another as the basic operation
(Anderson, 1995).
Pattern Recognition
Function Approximation
Control
Filtering
Beamforming

Neural Networks-Comprehensive Foundation by Simon Haykin

LEARNING TASKS
Pattern Association:
Association takes one of two forms:
1. autoassociation or
2. heteroassociation.

In autoassociation, a neural network is required to store a set of patterns (vectors) by repeatedly

presenting them to the network. The network is subsequently presented a partial description or
distorted (noisy) version of an original pattern stored in it, and the task is to retrieve (recall) that
particular pattern.

Heteroassociation differs from autoassociation in that an arbitrary set of input patterns is paired
with another arbitrary set of output patterns.

Autoassociation involves the use of unsupervised learning, whereas the type of learning involved
in heteroassociation is supervised.

Neural Networks-Comprehensive Foundation by Simon Haykin

LEARNING TASKS
Pattern Association:

Let x denote a key pattern (vector) applied to an associative memory and yk denote a
memorized pattern (vector). The pattern association performed by the network is
described by

where q is the number of patterns stored in the network. The key pattern xk acts as a
stimulus that not only determines the storage location of memorized pattern yk, but
also holds the key for its retrieval.

Neural Networks-Comprehensive Foundation by Simon Haykin

LEARNING TASKS
Pattern Association:

In an autoassociative memory, yk = xk, so the input and output (data) spaces of the
network have the same dimensionality.

In a heteroassociative memory, Уk ≠ xk hence, the dimensionality of the output space

in this second case may or may not equal the dimensionality of the input space.

Neural Networks-Comprehensive Foundation by Simon Haykin

LEARNING TASKS
Pattern Association:

There are two phases involved in the operation of an associative memory:

1. Storage phase, which refers to the training of the network in accordance with Eq.

1. Recall phase, which involves the retrieval of a memorized pattern in response to the
presentation of a noisy or distorted version of a key pattern to the network.

Neural Networks-Comprehensive Foundation by Simon Haykin

LEARNING TASKS
Pattern Association:

Let the stimulus (input) x represent a noisy or distorted version of a key pattern xj. This
stimulus produces a response (output) y, as indicated in Fig.. For perfect recall, we should
find that y = yj, where yj is the memorized pattern associated with the key pattern xj. When
y ≠ yj for x = xj the associative memory is said to have made an error in recall.

Neural Networks-Comprehensive Foundation by Simon Haykin

LEARNING TASKS
Pattern Association:
The number q of patterns stored in an associative memory provides a direct measure of the storage
capacity of the network. In designing an associative memory, the challenge is to make the storage
capacity q (expressed as a percentage of the total number N of neurons used to construct the network)
as large as possible and yet insist that a large fraction of the memorized patterns is recalled correctly.

Neural Networks-Comprehensive Foundation by Simon Haykin

LEARNING TASKS
There are six learning tasks that apply to the use of neural networks in one
form or another.
Pattern Association
Pattern Recognition: Pattern recognition is formally defined as the
process whereby a received pattern/signal is assigned to one of a prescribed
number of classes (categories). A neural network performs pattern
recognition by first undergoing a training session. Later, a new pattern is
presented to the network. The network is able to identify the class of that
particular pattern.
Function Approximation
Control
Filtering
Beamforming

Neural Networks-Comprehensive Foundation by Simon Haykin

LEARNING TASKS
Pattern Recognition:
Pattern recognition performed by a neural network is statistical in nature, with the patterns
being represented by points in a multidimensional decision space. The decision space is
divided into regions, each one of which is associated with a class. The decision boundaries are
determined by the training process. The construction of these boundaries is made statistical
by the inherent variability that exists within and between classes.

In generic terms, pattern-recognition machines using neural networks may take one of two
forms:

Neural Networks-Comprehensive Foundation by Simon Haykin

LEARNING TASKS
Pattern Recognition:
In conceptual terms, a pattern is represented by a set of m observables, which may be viewed as a
point x in an m-dimensional. observation (data) space. Feature extraction is described by a
transformation that maps the point x into an intermediate point y in a q-dimensional feature space
with q <m, as indicated in Fig. right.
This transformation may be viewed as one of dimensionality reduction (i.e., data compression), the
use of which is justified on the grounds that it simplifies the task of classification. The classification
itself described as a transformation that maps the intermediate point y into one of the classes in an
r-dimensional decision space, where r is the number of classes to be distinguished.

Neural Networks-Comprehensive Foundation by Simon Haykin

LEARNING TASKS
There are six learning tasks that apply to the use of neural networks in one
form or another.
Pattern Association
Pattern Recognition
Function Approximation: The third learning task of interest is that of
function approximation. Consider a nonlinear input-output mapping
described by the functional relationship d=f(x), where d is output and x is
input vector. The requirement is to design a neural network that
approximates the unknown function f(.) such that the function F(.) describing
the input-output mapping actually realized by the network is close enough to
f() in a Euclidean sense over all inputs.
Control
Filtering
Beamforming

Neural Networks-Comprehensive Foundation by Simon Haykin

LEARNING TASKS
There are six learning tasks that apply to the use of neural networks in one
form or another.
Pattern Association
Pattern Recognition
Function Approximation: The third learning task of interest is that of
function approximation. Consider a nonlinear input-output mapping
described by the functional relationship d=f(x), where d is output and x is
input vector. The requirement is to design a neural network that
approximates the unknown function f(.) such that the function F(.) describing
the input-output mapping actually realized by the network is close enough to
f() in a Euclidean sense over all inputs.
Control
Filtering
Beamforming

Neural Networks-Comprehensive Foundation by Simon Haykin

LEARNING TASKS
There are six learning tasks that apply to the use of neural networks in one
form or another.
Pattern Association
Pattern Recognition
Function Approximation
Control: The control of a plant is another learning task that can be done
by a neural network; by a "plant" we mean a process or critical part of a
system that is to be maintained in a controlled condition. The relevance of
learning to control should not be surprising because, after all, the human
brain is a computer (i.e., information processor), the outputs of which as a
whole system are actions.
Filtering
Beamforming

Neural Networks-Comprehensive Foundation by Simon Haykin

LEARNING TASKS
There are six learning tasks that apply to the use of neural networks in one
form or another.
Pattern Association
Pattern Recognition
Function Approximation
Control
Filtering: The term filter often refers to a device or algorithm used to
extract information about a prescribed quantity of interest from a set of
noisy data. The noise may arise from a variety of sources. For example, the
data may have been measured by means of noisy sensors or may represent
an information-bearing signal that has been corrupted by transmission
through a communication channel.
Beamforming

Neural Networks-Comprehensive Foundation by Simon Haykin

LEARNING TASKS
There are six learning tasks that apply to the use of neural networks in one
form or another.
Pattern Association
Pattern Recognition
Function Approximation
Control
Filtering
Beamforming: Beamforming is a spatial form of filtering and is used to
distinguish between the spatial properties of a target signal and background
noise. The device used to do the beamforming is called a beamformer.

Neural Networks-Comprehensive Foundation by Simon Haykin

LEARNING TASKS

Neural Networks-Comprehensive Foundation by Simon Haykin

MEMORY
In a neurobiological context, memory refers to the relatively enduring neural
alterations induced by the interaction of an organism with its environment (Teyler,
1986). Without such a change there can be no memory.

Furthermore, for the memory to be useful it must be accessible to the nervous

system in order to influence future behavior. However, an activity pattern must
initially be stored in memory through a learning process.

Memory and learning are intricately connected. When a particular activity pattern is
learned, it is stored in the brain where it can be, recalled later when required.

Memory may be divided into "short-term" and "long-term" memory, depending on

the retention time (Arbib, 1989). Short-term memory refers to a compilation of
knowledge representing the "current" state of the environment. Any discrepancies
between knowledge stored in short-term memory and a "new" state are used to
update the short-term memory. Long-term memory, on the other hand, refers to
knowledge stored for a long time or permanently.

Neural Networks-Comprehensive Foundation by Simon Haykin

MEMORY
An associative memory offers the following characteristics:
• The memory is distributed.
• Both the stimulus (key) pattern and the response (stored) pattern of an
associative memory consist of data vectors.
• Information is stored in memory by setting up a spatial pattern of neural activities
across a large number of neurons.
• Information contained in a stimulus not only determines its storage location in
memory but also an address for its retrieval.
• Although neurons do not represent reliable and low-noise computing cells, the
memory exhibits a high degree of resistance to noise and damage of a diffusive
kind.
• There may be interactions between individual patterns stored in memory.
(Otherwise the memory would have to be exceptionally large for it to
accommodate the storage of a large number of patterns in perfect isolation from
each other.) There is therefore the distinct possibility for the memory to make
errors during the recall process.

Neural Networks-Comprehensive Foundation by Simon Haykin

MEMORY

(a) Associative memory model (b) Associative memory model

component of a nervous system using artificial neurons

In a distributed memory, the basic issue of interest is the simultaneous or near-simultaneous

activities of many different neurons, which are the result of external or internal stimuli. The
neural activities form a spatial pattern inside the memory that contains information about the
stimuli. The memory is therefore said to perform a distributed mapping that transforms an
activity pattern in the input space into another activity pattern in the output space. We may
illustrate some important properties of a distributed memory mapping by considering an
idealized neural network that consists of two layers of neurons.

Neural Networks-Comprehensive Foundation by Simon Haykin

MEMORY

(a) Associative memory model (b) Associative memory model

component of a nervous system using artificial neurons

Each neuron in the input layer is connected to every one of the neurons in the output layer.
The actual synaptic connections between the neurons are complex and redundant. In the
model of Fig.(a), a single ideal junction is used to represent the integrated effect of all the
synaptic contacts between the dendrites of a neuron in the input layer and the axon branches
of a neuron in the output layer. The level of activity of a neuron in the input layer may affect
the level of activity of every other neuron in the output layer.

Neural Networks-Comprehensive Foundation by Simon Haykin

MEMORY

(a) Associative memory model (b) Associative memory model

component of a nervous system using artificial neurons

The corresponding situation for an artificial neural network is depicted in Fig.(b). Here we have
an input layer of source nodes and an output layer of neurons acting as computation nodes. In
this case, the synaptic weights of the network are included as integral parts of the neurons in
the output layer. The connecting links between the two layers of the network are simply wires.

Neural Networks-Comprehensive Foundation by Simon Haykin

MEMORY
Suppose that an activity pattern x occurs in the input layer of the
network and that an activity pattern y occurs simultaneously in
the output layer.

The issue we wish to consider here is that of learning from the

association between the patterns xk and yk. The patterns xk and
yk are represented by vectors, written in their expanded forms Signal-flow graph model of
as: a linear neuron labeled i

and

For convenience of presentation: assumed that

the input space dimensionality == output space dimensionality == equal to m.

network dimensionality or simply dimensionality.

Note that m equals the number of source nodes in the input layer or neurons in the
output layer.

Neural Networks-Comprehensive Foundation by Simon Haykin

MEMORY
Both xk and yk can assume positive and negative values

Networks in fig is assumed to be linear, the association of key

vector xk with memorized vector yk may be described in matrix
form as:

where W(k) is a weight matrix determined solely by the

input-output pair (xk,yk)

Neural Networks-Comprehensive Foundation by Simon Haykin

MEMORY
The output Yki of neuron i due to the combined action of the
elements of the key pattern xk applied as stimulus to the input
layer is given by

where the wij(k), j = 1, 2, ..., m, are the synaptic weights of

neuron i corresponding to the kth pair of associated patterns.
Using matrix notation, we may express yki in the equivalent
form

Neural Networks-Comprehensive Foundation by Simon Haykin

MEMORY

The column vector on the right-hand side of Eq. is recognized

as the key vector xk. By substituting Eq. in the definition of the
m-by-1 stored vector yk we get

In particular, the m-by-m weight matrix W(k) is defined by

Neural Networks-Comprehensive Foundation by Simon Haykin

MEMORY
The individual presentations of the q pairs of associated
patterns xk yk, k = 1, 2,...,q, produce corresponding values of
the individual matrix, namely, W(1), W(2), ..., W(q). Recognizing
that this pattern association is represented by the weight
matrix W(k), we may define an m-by–m memory matrix that
describes the summation of the weight matrices for the entire
set of pattern associations as follows:

The memory matrix M defines the overall connectivity between the input and output layers of the
associative memory. In effect, it represents the total experience gained by the memory as a result of
the presentations of q input-output patterns. Stated in another way, the memory matrix M contains
a piece of every input-output pair of activity patterns presented to the memory.
It may be restructured in the form of a recursion as shown by

where the initial value M0 is zero, and

the final value Mq, is identically equal to M

Neural Networks-Comprehensive Foundation by Simon Haykin

ADAPTATION
In performing a task of interest, we often find that-
space is one fundamental dimension of the learning process;
time is the other.

The spatiotemporal nature of learning is exemplified by many of the learning tasks (e.g.,
control, beamforming). Species ranging from insects to humans have an inherent capacity to
represent the temporal structure of experience. Such a representation makes it possible for
an animal to adapt its behavior to the temporal structure of an event in its behavioral space
(Gallistel, 1990).

Neural Networks-Comprehensive Foundation by Simon Haykin

ADAPTATION
When a neural network operates in a stationary environment (i.e., an
environment whose statistical characteristics do not change with time), the
essential statistics of the environment can, in theory, be learned by the
network under the supervision of a teacher.

In particular, the synaptic weights of the network can be computed by having

the network undergo a training session with a set of data that is
representative of the environment. Once the training process has completed,
the synaptic weights of the network should capture the underlying statistical
structure of the environment, which would justify "freezing" their values
thereafter. Thus a learning system relies on memory, in one form or another,
to recall and exploit past experiences.

Neural Networks-Comprehensive Foundation by Simon Haykin

ADAPTATION
However, the environment of interest is nonstationary, which means that
the statistical parameters of the information-bearing signals generated by
the environment vary with time.

In situations of this kind, the traditional methods of supervised learning may

prove to be inadequate because the network is not equipped with the
necessary means to track the statistical variations of the environment in
which it operates.

To overcome this shortcoming, it is desirable for a neural network to

continually adapt its free parameters to variations in the incoming signals in a
real-time fashion. Thus an adaptive system responds to every distinct input
as a novel one. In other words the learning process encountered in an
adaptive system never stops, with learning going on while signal processing is
being performed by the system. This form of learning is called continuous
learning or learning-on-the-fly.

Neural Networks-Comprehensive Foundation by Simon Haykin

ADAPTATION
Linear adaptive filters, built around a linear combiner (i.e., a single neuron
operating in its linear mode), are designed to perform continuous learning.
Despite their simple structure (and perhaps because of it), they are widely
used in such diverse applications as radar, sonar, communications,
seismology, and biomedical signal processing. The theory of linear adaptive
filters has reached a highly mature stage of development (Haykin, 1996;
Widrow and Stearns, 1985). However, the same cannot be said about
nonlinear adaptive filters.

Neural Networks-Comprehensive Foundation by Simon Haykin

ADAPTATION
With continuous learning as the property of interest and a neural network as
the vehicle for its implementation, the question we need to address is: How
can a neural network adapt its behavior to the varying temporal structure of
the incoming signals in its behavioral space?

Neural Networks-Comprehensive Foundation by Simon Haykin

ADAPTATION
With continuous learning as the property of interest and a neural network as
the vehicle for its implementation, the question we need to address is: How
can a neural network adapt its behavior to the varying temporal structure of
the incoming signals in its behavioral space?

One way of addressing this fundamental issue is to recognize that statistical

characteristics of a nonstationary process usually change slowly enough for
the process to be considered pseudostationary over a window of short
enough duration.

Examples include:
• The mechanism responsible for the production of a speech signal may be
considered essentially stationary over a period of 10 to 30 milliseconds.
• Radar returns from an ocean surface remain essentially stationary over a
period of several seconds.
• With long-range weather forecasting in mind, weather related data may
be viewed as essentially stationary over a period of minutes.

Neural Networks-Comprehensive Foundation by Simon Haykin

ADAPTATION
We may thus exploit the pseudostationary property of a stochastic process to
extend the utility of a neural network by retraining it at some regular
intervals to account for statistical fluctuations of the incoming data. Such an
approach may, for example, be suitable for processing stock market data.

For a more refined dynamic approach to learning, we may proceed as

follows:
• Select a window short enough for the input data to be considered
pseudostationary, and use the data to train the network.
• When a new data sample is received, update the window by dropping the
oldest data sample and shifting the remaining data samples back by one
time unit to make room for the new sample.
• Use the updated data window to retrain the network. Repeat the
procedure on a continuing basis.

Neural Networks-Comprehensive Foundation by Simon Haykin

ADAPTATION
We may thus build temporal structure into the design of a neural network by
having the network undergo continual training with time-ordered examples.

According to this dynamic approach, a neural network is viewed as a

nonlinear adaptive filter that represents a generalization of linear adaptive
filters. However, for this dynamic approach to nonlinear adaptive filters to be
feasible, the resources available must be fast enough to complete all the
described computations in one sampling period. Only then can the filter keep
up with changes in the input.

Neural Networks-Comprehensive Foundation by Simon Haykin

Thank-you