Dr.

Lafta Al-Qaisy

Chapter two
An Illustrative Example
Objectives
Theory and Examples
Problem Statement
Perceptron
Two-Input Case
Pattern Recognition Example
Hamming Network
Feedforward Layer
Recurrent Layer
Hopfield Network
Exercises

18
Dr. Lafta Al-Qaisy

Objectives

Think of this chapter as a preview of coming attractions. We will take a simple

pattern recognition problem and show how it can be solved using three different
neural network architectures. It will be an opportunity to see how the architectures
described in the previous chapter can be used to solve a practical (although
extremely oversimplified) problem. Do not expect to completely understand these
three networks after reading this chapter. We present them simply to give you a
taste of what can be done with neural networks, and to demonstrate that there are
many different types of networks that can be used to solve a given problem. The
three networks presented in this chapter are representative of the types of networks
discussed in the remaining chapters: feedforward networks (represented here by
the perceptron), competitive networks (represented here by the Hamming network)
and recurrent associative memory networks (represented here by the Hopfield
network).

Theory and Examples

Problem Statement
A produce dealer has a warehouse that stores a variety of fruits and vegetables.
When fruit is brought to the warehouse, various types of fruit may be mixed
together. The dealer wants a machine that will sort the fruit according to type.
There is a conveyer belt on which the fruit is loaded. This conveyer passes through
a set of sensors, which measure three properties of the fruit: shape, texture and
weight. These sensors are somewhat primitive.
The shape sensor will output a 1 if the fruit is approximately round and a -1 if it is
more elliptical. The texture sensor will output a 1 if the surface of the fruit is
smooth and a -1 if it is rough. The weight sensor will output a 1 if the fruit is more
than one pound and a -1 if it is less than one pound.
The three sensor outputs will then be input to a neural network. The purpose of the
network is to decide which kind of fruit is on the conveyor, so that the fruit can be
directed to the correct storage bin. To make the problem even simpler, let’s assume
that there are only two kinds of fruit on the conveyor: apples and oranges.

19
Dr. Lafta Al-Qaisy

As each fruit passes through the sensors it can be represented by a three

dimensional vector. The first element of the vector will represent shape, the second
element will represent texture and the third element will represent weight:

[ ] (3.1)

Therefore, a prototype orange would be represented by

[ ] (3.2)

and a prototype apple would be represented by

[ ] (3.3)

The neural network will receive one three-dimensional input vector for each fruit
on the conveyer and must make a decision as to whether the fruit is an orange(
or an apple ( .
Now that we have defined this simple (trivial?) pattern recognition problem, let’s
look briefly at three different neural networks that could be used to solve it. The
simplicity of our problem will facilitate our understanding of the operation of the
networks.

20
Dr. Lafta Al-Qaisy

Perceptron

The first network we will discuss is the perceptron. Figure 3.1 illustrates a single-
layer perceptron with a symmetric hard limit transfer function hardlims.

Figure 3.1 Single – Layer Perceptron

Two-Input Case

Before we use the perceptron to solve the orange and apple recognition problem
(which will require a three-input perceptron, i.e., R=3 ), it is useful to investigate
the capabilities of a two-input/single-neuron perceptron ( R=2 ), which can be
easily analyzed graphically. The two-input perceptron is shown in Figure 3.2.

Figure 3.2 Two- Input/Single – Neuron Perceptron

Single-neuron perceptrons can classify input vectors into two categories. For
example, for a two-input perceptron, if and then
( ([ ] (3.4)
Therefore, if the inner product of the weight matrix (a single row vector in this
case) with the input vector is greater than or equal to -b , the output will be 1. If the
inner product of the weight vector and the input is less than -b , the output will be
-1. This divides the input space into two parts. Figure 3.3 illustrates this for the
case where b = -1 . The blue line in the figure represents all points for which the
net input n is equal to 0:
[ ] (3.5)

21
Dr. Lafta Al-Qaisy

Notice that this decision boundary will always be orthogonal to the weight matrix,
and the position of the boundary can be shifted by changing . (In the general case,
W is a matrix consisting of a number of row vectors, each of which will be used in
an equation like Eq. (3.5). There will be one boundary for each row of W . See
Chapter 4 for more on this topic.) The shaded region contains all input vectors for
which the output of the network will be 1. The output will be -1 for all other input
vectors.

Figure 3.3 Perceptron Decision Boundary

The key property of the single-neuron perceptron, therefore, is that it can separate
input vectors into two categories. The decision boundary between the categories is
determined by the equation
(3.6)

Pattern Recognition Example

Now consider the apple and orange pattern recognition problem. Because there are
only two categories, we can use a single-neuron perceptron. The vector inputs are
three-dimensional ( R=3 ), therefore the perceptron equation will be

[[ ][ ] ] (3.7)

We want to choose the bias b and the elements of the weight matrix so that the
perceptron will be able to distinguish between apples and oranges. For example,
we may want the output of the perceptron to be 1 when an apple is input and -1
when an orange is input. Using the concept illustrated in Figure 3.3, let’s find a
linear boundary that can separate oranges and apples. The two prototype vectors
(recall Eq. (3.2) and Eq. (3.3)) are shown in Figure 3.4. From this figure we can

22
Dr. Lafta Al-Qaisy

see that the linear boundary that divides these two vectors symmetrically is the
plane.

Figure 3.4 Prototype Vectors

The plane, which will be our decision boundary, can be described by the
equation
(3.8)

Or

[ ][ ] (3.9)

Therefore the weight matrix and bias will be

[ ] (3.10)

The weight matrix is orthogonal to the decision boundary and points toward the
region that contains the prototype pattern (apple) for which we want the
perceptron to produce an output of 1. The bias is 0 because the decision boundary
passes through the origin.
Now let’s test the operation of our perceptron pattern classifier. It classifies perfect
apples and oranges correctly since

Orange:

[[ ][ ] ] ( (3.11)

23
Dr. Lafta Al-Qaisy

Apple:

[[ ][ ] ] ( (3.12)

The input vector would then be

[ ] (3.13)

The response of the network would be

[[ ][ ] ] ( (3.14)

In fact, any input vector that is closer to the orange prototype vector than to the
apple prototype vector (in Euclidean distance) will be classified as an orange (and
vice versa).

Hamming Network

The next network we will consider is the Hamming network [Lipp87]. It was
designed explicitly to solve binary pattern recognition problems (where each
element of the input vector has only two possible values — in our example 1 or -1
). This is an interesting network, because it uses both feedforward and recurrent
(feedback) layers, which were both described in Chapter 2. Figure 3.5 shows the
standard Hamming network. Note that the number of neurons in the first layer is
the same as the number of neurons in the second layer.
The objective of the Hamming network is to decide which prototype vector is
closest to the input vector. This decision is indicated by the output of the recurrent
layer. There is one neuron in the recurrent layer for each prototype pattern. When
the recurrent layer converges, there will be only one neuron with nonzero output.
This neuron indicates the prototype pattern that is closest to the input vector. Now
let’s investigate the two layers of the Hamming network in detail.

24
Dr. Lafta Al-Qaisy

Figure 3.5 Hamming Network

Feedforward Layer

The feedforward layer performs a correlation, or inner product, between each of

the prototype patterns and the input pattern (as we will see in Eq. (3.17)). In order
for the feedforward layer to perform this correlation, the rows of the weight matrix
in the feedforward layer, represented by the connection matrix , are set to the
prototype patterns. For our apple and orange example this would mean
0 1 * + (3.15)

The feedforward layer uses a linear transfer function, and each element of the bias
vector is equal to , where R is the number of elements in the input vector. For our
example the bias vector would be
* + (3.16)

With these choices for the weight matrix and bias vector, the output of the
feedforward layer is
0 1 * + 0 1 (3.17)

Recurrent Layer

The recurrent layer of the Hamming network is what is known as a ―competitive‖

layer. The neurons in this layer are initialized with the outputs of the feedforward
layer, which indicate the correlation between the prototype patterns and the input
vector. Then the neurons compete with each other to determine a winner. After the
competition, only one neuron will have a nonzero output. The winning neuron

25
Dr. Lafta Al-Qaisy

indicates which category of input was presented to the network (for our example
the two categories are
apples and oranges). The equations that describe the competition are:

( ( (3.18)

and

( ( ( (3.19)

(Don’t forget that the superscripts here indicate the layer number, not a power of
2.) The poslin transfer function is linear for positive values and zero for negative
values. The weight matrix W2 has the form
* + (3.20)
Where is some number less than , and is the number of neurons in the
recurrent layer. (Can you show why must be less than ?) An iteration of the
recurrent layer proceeds as follows:
( (
( 0* + ( 1 .0 1/ (3.21)
( (

Each element is reduced by the same fraction of the other. The larger element will
be reduced by less, and the smaller element will be reduced by more, therefore the
difference between large and small will be increased.
The effect of the recurrent layer is to zero out all neuron outputs, except the one
with the largest initial value (which corresponds to the prototype pattern that is
closest in Hamming distance to the input). To illustrate the operation of the
Hamming network, consider again the oblong orange that we used to test the
perceptron:

[ ] (3.22)

The output of the feedforward layer will be

(
* +[ ] * + [ ] * + (3.23)
(

which will then become the initial condition for the recurrent layer.
The weight matrix for the recurrent layer will be given by Eq. (3.20) with
26
Dr. Lafta Al-Qaisy

(any number less than 1 would work). The first iteration of the recurrent layer
produces
(* + * +)
( ( ( ) { (3.24)
(* +) * +

The second iteration produces

(* + * +)
( ( ( ) { (3.25)
(* +) * +

Since the outputs of successive iterations produce the same result, the network has
converged. Prototype pattern number one, the orange, is chosen as the correct
match, since neuron number one has the only nonzero output. (Recall that the first
element of was ( .) This is the correct choice, since the Hamming
distance from the orange prototype to the input pattern is 1, and the Hamming
distance from the apple prototype to the input pattern is 2.

Hopfield Network

The final network we will discuss in this brief preview is the Hopfield network.
This is a recurrent network that is similar in some respects to the recurrent layer of
the Hamming network, but which can effectively perform the operations of both
layers of the Hamming network. A diagram of the Hopfield network is shown in
Figure 3.6. (This figure is actually a slight variation of the standard Hopfield
network. We use this variation because it is somewhat simpler to describe and yet
demonstrates the basic concepts.)
The neurons in this network are initialized with the input vector, then the network
iterates until the output converges. When the network is operating correctly, the
resulting output should be one of the prototype vectors. Therefore, whereas in the
Hamming network the nonzero neuron indicates which prototype pattern is chosen,
the Hopfield network actually produces the selected prototype pattern at its output.

27
Dr. Lafta Al-Qaisy

Figure 3.6 Hopfield Network

The equations that describe the network operation are

( (3.26)

and

( ( ( (3.27)

where satlins is the transfer function that is linear in the range [-1, 1] and saturates
at 1 for inputs greater than 1 and at -1 for inputs less than -1. The design of the
weight matrix and the bias vector for the Hopfield network is a more complex
procedure than it is for the Hamming network, where the weights in the
feedforward layer are the prototype patterns.
To illustrate the operation of the network, we have determined a weight
matrix and a bias vector that can solve our orange and apple pattern recognition
problem. They are given in Eq. (3.28).

[ ] [ ] (3.28)

Although the procedure for computing the weights and biases for the Hopfield
network is beyond the scope of this chapter, we can say a few things about why the
parameters in Eq. (3.28) work for the apple and orange example. We want the
network output to converge to either the orange pattern, ,or the apple pattern,
. In both patterns, the first element is -1 , and the third element is . The
difference between the patterns occurs in the second element. Therefore, no matter
what pattern is input to the network, we want the first element of the output pattern
to converge to , the third element to converge to -1 , and the second element to go
to either 1 or -1, whichever is closer to the second element of the input vector.

28
Dr. Lafta Al-Qaisy

The equations of operation of the Hopfield network, using the parameters given in
Eq. (3.28), are

( ( (

( ( ( )

( ( ( (3.29)

Regardless of the initial values, ( , the first element will be increased until it
saturates at , and the third element will be decreased until it saturates at -1 . The
second element is multiplied by a number larger than [Link], if it is initially
negative, it will eventually saturate at -1 ; if it is initially positive it will saturate at
1.
(It should be noted that this is not the only ( pair that could be used. You
might want to try some others. See if you can discover what makes these work.)
Let’s again take our oblong orange to test the Hopfield network. The outputs of the
Hopfield network for the first three iterations would be

( [ ] ( [ ] ( [ ] ( [ ] (3.30)

The network has converged to the orange pattern, as did both the Hamming
network and the perceptron, although each network operated in a different way.
The perceptron had a single output, which could take on values of -1 (orange) or 1
(apple). In the Hamming network the single nonzero neuron indicated which
prototype pattern had the closest match. If the first neuron was nonzero, that
indicated orange, and if the second neuron was nonzero, that indicated apple. In
the Hopfield network the prototype pattern itself appears at the output of the
network.

29
Dr. Lafta Al-Qaisy

Homework
E3.1 In this chapter we have designed three different neural networks to is tinguish
between apples and oranges, based on three sensor measurements (shape, texture
and weight). Suppose that we want to distinguish between bananas and pineapples:

[ ](

[ ](

i. Design a perceptron to recognize these patterns.

ii. Design a Hamming network to recognize these patterns.
iii. Design a Hopfield network to recognize these patterns.
iv. Test the operation of your networks by applying several different input patterns.
Discuss the advantages and disadvantages of each network.

E3.2 Consider the following prototype patterns.

* + * +

i. Find and sketch a decision boundary for a perceptron network that will recognize
these two vectors.
ii. Find weights and bias which will produce the decision boundary you found in
part i, and sketch the network diagram.
iii. Calculate the network output for the following input. Is the network response
(decision) reasonable? Explain.
* +

iv. Design a Hamming network to recognize the two prototype vectors above.
v. Calculate the network output for the Hamming network with the input vector
given in part iii, showing all steps. Does the Hamming network produce the same
decision as the perceptron? Explain why or why not. Which network is better
suited to this problem? Explain.

30
Dr. Lafta Al-Qaisy

E3.3 Consider a Hopfield network, with the following weight and bias.

* + * +

i. The following input (initial condition) is applied to the network.

Find the network response (show the network output at each iteration until the
network converges).
* +

ii. Draw a sketch indicating what region of the input space will converge to the
same final output that you found in part i. (In other words, for what other p vectors
will the network converge to the same final output?) Explain how you obtained
your answer.
iii. What other prototypes will this network converge to, and what regions of the
input space correspond to each prototype (sketch the regions). Explain how you
obtained your answer.

E3.4 Consider the following perceptron network.

With * + * +

i. How many different classes can this network classify?

ii. Draw a diagram illustrating the regions corresponding to each class. Label each
region with the corresponding network output.
iii. Calculate the network output for the following input.

* +

31
Dr. Lafta Al-Qaisy

iv. Plot the input from part iii in your diagram from part ii, and verify that it falls in
the correctly labeled region.
E3.5 We want to design a perceptron network to output a 1 when either of these
two vectors are input to the network:
,* + * +-

and to output a -1 when either of the following vectors are input to the network:

,* + * +-

i. Find and sketch a decision boundary for a network that will solve this problem.
ii. Find weights and biases that will produce the decision boundary you found in
part i. Show all work.
iii. Draw the network diagram using abbreviated notation.
iv. For each of the four vectors given above, calculate the net input, n, and the
network output, a, for the network you have designed. Verify that your network
solves the problem.
v. Are there other weights and biases that would solve the problem? If so, would
you consider your weights best? Explain.
E3.6 We have the folowing two prototype vectors:

,* + * +-

i. Find and sketch a decision boundary for a perceptron network that will recognize
these two vectors.
ii. Find weights and bias that will produce the decision boundary you found in part
i.
iii. Draw the network diagram using abbreviated notation.

iv. For the vector given below, calculate the net input, n, and the network output, a,
for the network you have designed. Does the network produce a good output?
Explain.
* +

v. Design a Hamming network to recognize the two vectors used in part i.

vi. Calculate the network output for the Hamming network for the input vector
given in part iv. Does the network produce a good output? Explain.
vii. Design a Hopfield network to recognize the two vectors used in part i.

32
Dr. Lafta Al-Qaisy

viii. Calculate the network output for the Hopfield network for the input vector
given in part iv. Does the network produce a good output? Explain.

E3.7 We want to design a Hamming network to recognize the following prototype

vectors:
,* + * + * +-

i. Find the weight matrices and bias vectors for the Hamming network.
ii. Draw the network diagram.
iii. Apply the following input vector and calculate the total network response
(iterating the second layer to convergence). Explain the meaning of the final
network output.
* +

iv. Sketch the decision boundaries for this network. Explain how you determined
the boundaries.

33