Supervised Hebbian Learning
Who? UBC4006-01
From? Underwood International College, Yonsei University
When? Spring Semester 2026
Theory and Example
Linear Associator
Hebb Rule
If two neurons on either side of a synapse are activated
simultaneously, the strength of the synapse will increase.
Hebb’s postulate implies that if a positive pj produces a
positive ai , then wij should increase.
where piq is the jth element of the qth input vector pq ;
aiq is the ith element of the network output when the
qth input vector is presented to the network. α is a
positive constant, called the learning rate.
This equation says that the change in the weight wij is
proportional to a product of functions of the activities
on either side of the synapse.
Unsupervised Hebb Rule
For this chapter, we use the following simplified form
wijnew = wijold + αaiq pjq
This formulation includes the increase of the weight
when they are both negative. Also, it will decrease the
weight whenever pj and ai have opposite sign.
This formulation does not require any information
concerning the target output.
Supervised Hebb Rule
For the supervised Hebb rule, we substitute the target
output for the actual output.
wijnew = wijold + αtiq pjq ,
where tiq is the ith element of the qth target vector tq .
This algorithm simply tells what the network should do,
rather than what it is currently doing.
This can be written in vector notation such as
Wnew = Wold + tq pT
q
Batch Operation
Performance Analysis
The magnitude of the error depends on the amount of
correlation between the prototype input patterns.
Example
The outputs are close, but not the exact ones.
Pseudoinverse Rule
When the prototype input patterns are not orthogonal,
the Hebb rule produces some error. However, there are
several procedures that can be used to reduce these
error.
Consider the task of the linear associator to produce an
output of tq for an input of pq such as
Wpq = tq , 1≤q≤Q
If we cannot find a matrix to satisfy these equations,
then we want them to be approximately satisfied. One
approach is to minimize the following performance index
Q
X
F (W) = ∥tq − Wpq ∥2
q=1
If pq are orthonormal, then F (W) will be zero. For
other cases, the weight matrix that minimizes F (W) is
obtained by using the pseudoinverse matrix.
Example
Relationship to the Hebb Rule
Example
Application
In an autoassociative memory, the desired output vector
is equal to the input vector such as tq = pq .
We use an autoassociative memory to store a set of
patterns and to recall these patterns.
They represent the digits {0, 1, 2} displayed in a 6x5
grid. We first convert these digits to vectors. Each white
square represent ’-1’ and each dark square represent ’1’.
Then, the weight matrix is computed as
W = p1 pT T T
1 + p2 p2 + p3 p3
To take only ’-1’ and ’1’ for the output, we replace the
linear transfer function with a symmetrical hard limit
transfer function.
Autoassociative memory example
Test
Also try nnd7sh
Variations of Hebbian Learning (I)
The basic rule is given as
Wnew = Wold + tq pT
q
A positive parameter α, called the learning rate, can be
used to limit the amount of increase in the weight
matrix elements.
Wnew = Wold + αtq pT
q
We can add a decay term to behave like a smoothing
filter, remembering the most recent inputs more clearly.
For 0 ≤ γ ≤ 1
Wnew = (1 − γ)Wold + αtq pT
q
Variations of Hebbian Learning (II)
We can replace the desired output with the difference
between the desired output and the actual output such
as
Wnew = Wold + α(tq − aq )pT q
which is known as the delta rule, or the Widrow-Hoff
algorithm. This rule adjusts the weights so as to
minimize the mean square error. Thus, it produces the
same results as the pseudoinverse rule.
One advantage of the delta rule is that it can update
the weights after each new input pattern is presented.
In the unsupervised Hebb rule, the actual network
output is used instead of the desired network output as
Wnew = Wold + αaq pT
q