UNIT II

[Type here] [Type here] [Type here]

ASSOCIATIVE MEMORY AND UNSUPERVISED LEARNING NETWORKS

Training Algorithms for Pattern Association-Autoassociative Memory Network-

Heteroassociative Memory Network-Bidirectional Associative Memory (BAM)-Hopfield
Networks-Iterative Autoassociative Memory Networks-Temporal Associative Memory
Network-Fixed Weight Competitive Nets-Kohonen Self-Organizing Feature Maps-Learning
Vector Quantization-Counter propagation Networks-Adaptive Resonance Theory Network.

Training Algorithms for Pattern Association

These kinds of neural networks work on the basis of pattern association, which means they
can store different patterns and at the time of giving an output they can produce one of the
stored patterns by matching them with the given input pattern. These types of memories are
also called Content-Addressable Memory. Associative memory makes a parallel search with
the stored patterns as data files.
Following are the two types of associative memories we can observe –
Auto Associative Memory
Hetero Associative memory
Auto Associative Memory
This is a single layer neural network in which the input training vector and the output target
vectors are the same. The weights are determined so that the network stores a set of patterns.
Architecture
As shown in the following figure, the architecture of Auto Associative memory network
has ‘n’ number of input training vectors and similar ‘n’ number of output target vectors.
or

Auto-associative Neural Networks

Auto associative Neural networks are the types of neural networks whose input and output
vectors are identical. These are special kinds of neural networks that are used to simulate and
explore the associative process. Association in this architecture comes from the instruction of
a set of simple processing elements called units which are connected through weighted
connections.
In these networks, we performed training to store the vector either bipolar or binary. A stored
vector can be retrieved from a distorted or noisy vector if the input is similar to it.
Architecture
AANN contains five-layer perceptron feed-forward network, that can be divided into two
neural networks of 3 layers each connected in series (similar to autoencoder architecture).
The network consists of an input layer followed by a hidden layer and bottleneck layer. This
bottleneck layer is common between both the network and a key component of the network.
It provides data compression to the input and topology with powerful feature extraction
capabilities. The bottleneck layer is followed by a second non-linear hidden layer and the
output layer of the second network.

Auto-Associative NN architecture
The first network compresses the information of the n-dimensional vector to smaller
dimension vectors that contain a smaller number of characteristic variables and represent the
whole process. The second network works opposite to the first and uses compressed
information to regenerate the original n redundant measures.
Algorithm
We will be using Hebb Rule in the algorithm for setting weights because input and output
vectors are perfectly correlated since the input and output both have the same number of
output units and input units.
Hebb Rule:
when A and B are positively correlated, then increase the strength of the connection between
them.
when A and B are negatively correlated, then decrease the strength of the connection between
them.
In practice, we use following formula to set the weights:

where, W = weighted matrix

T= Learning Rate
S(p) : p-distinct n-dimensional prototype patterns
Training Algorithm
Initialize all weights for i= 1,2,3 …n and j= 1,2,3 …n such that: wij=0.

For each vector to be stored repeat the following steps:

Set activation for each input unit i= 1 to n: xi = si.
Set activation for each output unit j= 1 to n: yj = sj.
Update the weights for i= 1,2,3 …n and j= 1,2,3 …n such that : wij (new) = wij (old) + xiyj
Testing / Inference Algorithm:
For testing whether the input is ‘known’ and ‘unknown’ to the model, we need to perform the
following steps:
Take the weights that were generated during the training phase using Hebb’s rule.
For each input vector perform the following steps:
Set activation in the input units equal to input vectors.

Set activation in output units for j= 1,2,3 …n:

Apply activation function for j= 1, 2, 3 … n:

AANN recognizes the input vector to be known if the output unit after activation generated
same pattern as one stored in it.
Storage Capacity
One of the important features of AANN is the number of patterns that can be stored before
the network begins to forget.
The number of vectors that can be stored in the network is called the capacity of the network.
The capacity of the vectors depends upon the relationship between them i.e more vectors can be
stored if they are mutually orthogonal. Generally, n-1 mutually orthogonal vectors with n
components can be stored.
Applications
Auto-associative Neural Networks can be used in many fields:
Pattern Recognition
Bio-informatics
Voice Recognition
Signal Validation etc.

Hetero Associative memory

Similar to Auto Associative Memory network, this is also a single layer neural network.
However, in this network the input training vector and the output target vectors are not the
same. The weights are determined so that the network stores a set of patterns. Hetero
associative network is static in nature, hence, there would be no non-linear and delay
operations.
Architecture
As shown in the following figure, the architecture of Hetero Associative Memory network
has ‘n’ number of input training vectors and ‘m’ number of output target vectors.
ANN – Bidirectional Associative Memory (BAM)

Bidirectional Associative Memory (BAM) is a supervised learning model in Artificial

Neural Network. This is hetero-associative memory, for an input pattern, it returns another
pattern which is potentially of a different size. This phenomenon is very similar to the human
brain. Human memory is necessarily associative. It uses a chain of mental associations to
recover a lost memory like associations of faces with names, in exam questions with answers,
etc. In such memory associations for one type of object with another, a Recurrent Neural
Network (RNN) is needed to receive a pattern of one set of neurons as an input and generate
a related, but different, output pattern of another set of neurons. Why BAM is required? The
main objective to introduce such a network model is to store hetero-associative pattern pairs.
This is used to retrieve a pattern given a noisy or incomplete pattern. BAM
Architecture: When BAM accepts an input of n-dimensional vector X from set A then the
model recalls m-dimensional vector Y from set B. Similarly when Y is treated as input, the
BAM recalls X.

Algorithm:
Storage (Learning): In this learning step of BAM, weight matrix is calculated between M
pairs of patterns (fundamental memories) are stored in the synaptic weights of the network

following the equation

Testing: We have to check that the BAM recalls perfectly for corresponding and
recalls for corresponding . Using,

All pairs should be recalled accordingly.

Retrieval: For an unknown vector X (a corrupted or incomplete version of a pattern from
set A or B) to the BAM and retrieve a previously stored association:
Repeat the iteration until convergence, when input and output remain unchanged.
Limitations of BAM:
Storage capacity of the BAM: In the BAM, stored number of associations should not be
exceeded the number of neurons in the smaller layer.
Incorrect convergence: Always the closest association may not be produced by BAM.

Hopfield Neural Network

The Hopfield Neural Networks, invented by Dr John J. Hopfield consists of one layer
of ‘n’ fully connected recurrent neurons. It is generally used in performing auto-association
and optimization tasks. It is calculated using a converging interactive process and it generates
a different response than our normal neural nets.
Discrete Hopfield Network
It is a fully interconnected neural network where each unit is connected to every other unit. It
behaves in a discrete manner, i.e. it gives finite distinct output, generally of two types:
Binary (0/1)
Bipolar (-1/1)
The weights associated with this network are symmetric in nature and have the following
properties.
Structure & Architecture of Hopfield Network
Each neuron has an inverting and a non-inverting output.
Being fully connected, the output of each neuron is an input to all other neurons but not the
self.
The below figure shows a sample representation of a Discrete Hopfield Neural Network
architecture having the following elements.

Discrete Hopfield Network Architecture

[ x1 , x2 , ... , xn ] -> Input to the n given neurons.

[ y1 , y2 , ... , yn ] -> Output obtained from the n given neurons

Wij -> weight associated with the connection between the ith and the jth neuron.

Steps Involved in the training of a Hopfield Network are as mapped below:

Initialize weights (wij) to store patterns (using training algorithm).
For each input vector yi, perform steps 3-7.

Make the initial activators of the network equal to the external input vector x.

or each vector yi, perform steps 5-7.

Calculate the total input of the network yin using the equation given below.

Apply activation over the total input to calculate the output as per the equation given
below:
[Type here] [Type here] [Type here]

(where θi (threshold) and is normally taken as 0)

Now feedback the obtained output yi to all other units. Thus, the activation vectors are
updated.
Test the network for convergence.
Consider the following problem. We are required to create a Discrete Hopfield Network
with the bipolar representation of the input vector as [1 1 1 -1] or [1 1 1 0] (in case of
binary representation) is stored in the network. Test the Hopfield network with missing
entries in the first and second components of the stored vector (i.e. [0 0 1 0]).
Given the input vector, x = [1 1 1 -1] (bipolar) and we initialize the weight matrix (wij) as:

Now for next unit, we will take updated value via feedback. (i.e. y = [1 0 1 0])
[Type here] [Type here] [Type here]

Now for next unit, we will take updated value via feedback. (i.e. y = [1 0 1 0])

Now for next unit, we will take updated value via feedback. (i.e. y = [1 0 1 0])
[Type here] [Type here] [Type here]

Continuous Hopfield Network

Unlike the discrete Hopfield networks, here the time parameter is treated as a continuous
variable. So, instead of getting binary/bipolar outputs, we can obtain values that lie between
0 and 1. It can be used to solve constrained optimization and associative memory problems.
The output is defined as:

where,
vi = output from the continuous hopfield network
ui = internal activity of a node in continuous hopfield network.
Energy Function
The Hopfield networks have an energy function associated with them. It either diminishes
or remains unchanged on update (feedback) after every iteration. The energy function for a
continuous Hopfield network is defined as:
[Type here] [Type here] [Type here]
[Type here] [Type here] [Type here]
[Type here] [Type here] [Type here]

Kohonen Self-Organizing Feature Maps

Suppose we have some pattern of arbitrary dimensions, however, we need them in one
dimension or two dimensions. Then the process of feature mapping would be very useful to
convert the wide pattern space into a typical feature space. Now, the question arises why do
we require self-organizing feature map? The reason is, along with the capability to convert
the arbitrary dimensions into 1-D or 2-D, it must also have the ability to preserve the
neighbor topology.
Self Organizing Map (or Kohonen Map or SOM) is a type of Artificial Neural Network
which is also inspired by biological models of neural systems from the 1970s. It follows an
unsupervised learning approach and trained its network through a competitive learning
algorithm. SOM is used for clustering and mapping (or dimensionality reduction)
techniques to map multidimensional data onto lower-dimensional which allows people to
reduce complex problems for easy interpretation. SOM has two layers, one is the Input
layer and the other one is the Output layer.
The architecture of the Self Organizing Map with two clusters and n input features of any
sample is given below:
[Type here] [Type here] [Type here]

How do SOM works?

Let’s say an input data of size (m, n) where m is the number of training examples and n is the
number of features in each example. First, it initializes the weights of size (n, C) where C is
the number of clusters. Then iterating over the input data, for each training example, it
updates the winning vector (weight vector with the shortest distance (e.g Euclidean distance)
from training example). Weight updation rule is given by :
wij = wij(old) + alpha(t) * (xik - wij(old))
where alpha is a learning rate at time t, j denotes the winning vector, i denotes the ith feature
of training example and k denotes the kth training example from the input data. After
training the SOM network, trained weights are used for clustering new examples. A new
example falls in the cluster of winning vectors.
Algorithm
Training:
Step 1: Initialize the weights wij random value may be assumed. Initialize the learning rate
α.
Step 2: Calculate squared Euclidean distance.
D(j) = Σ (wij – xi)^2 where i=1 to n and j=1 to m
Step 3: Find index J, when D(j) is minimum that will be considered as winning index.
Step 4: For each j within a specific neighborhood of j and for all i, calculate the new
weight.
wij(new)=wij(old) + α[xi – wij(old)]
[Type here] [Type here] [Type here]

Step 5: Update the learning rule by using :

α(t+1) = 0.5 * t
Step 6: Test the Stopping Condition.

Neighbor Topologies in Kohonen SOM

There can be various topologies, however the following two topologies are used the most −
Rectangular Grid Topology
This topology has 24 nodes in the distance-2 grid, 16 nodes in the distance-1 grid, and 8
nodes in the distance-0 grid, which means the difference between each rectangular grid is 8
nodes. The winning unit is indicated by #.

Hexagonal Grid Topology

This topology has 18 nodes in the distance-2 grid, 12 nodes in the distance-1 grid, and 6
nodes in the distance-0 grid, which means the difference between each rectangular grid is 6
nodes. The winning unit is indicated by #.
[Type here] [Type here] [Type here]

Architecture
The architecture of KSOM is similar to that of the competitive network. With the help of
neighborhood schemes, discussed earlier, the training can take place over the extended region
of the network.

Algorithm for training

Step 1 − Initialize the weights, the learning rate α and the neighborhood topological scheme.
Step 2 − Continue step 3-9, when the stopping condition is not true.
Step 3 − Continue step 4-6 for every input vector x.
Step 4 − Calculate Square of Euclidean Distance for j = 1 to m
[Type here] [Type here] [Type here]

Learning Vector Quantization

Learning Vector Quantization LVQ, different from Vector quantization VQ and Kohonen
Self-Organizing Maps KSOM, basically is a competitive network which uses supervised
learning. We may define it as a process of classifying the patterns where each output unit
represents a class. As it uses supervised learning, the network will be given a set of training
patterns with known classification along with an initial distribution of the output class. After
completing the training process, LVQ will classify an input vector by assigning it to the same
class as that of the output unit.
Architecture
Following figure shows the architecture of LVQ which is quite similar to the architecture of
KSOM. As we can see, there are “n” number of input units and “m” number of output units.
The layers are fully interconnected with having weights on them.

Parameters Used
Following are the parameters used in LVQ training process as well as in the flowchart
x = training vector (x1,...,xi,...,xn)
[Type here] [Type here] [Type here]

T = class for training vector x

wj = weight vector for jth output unit
Cj = class associated with the jth output unit
Training Algorithm
Step 1 − Initialize reference vectors, which can be done as follows −
Step 1a − From the given set of training vectors, take the first “m” numberofclusters training
vectors and use them as weight vectors. The remaining vectors can be used for training.
Step 1b − Assign the initial weight and classification randomly.
Step 1c− Apply K-means clustering method.
Step 2 − Initialize reference vector α
Step 3 − Continue with steps 4-9, if the condition for stopping this algorithm is not met.
Step 4 − Follow steps 5-6 for every training input vector x.
Step 5 − Calculate Square of Euclidean Distance for j = 1 to m and i = 1 to n
 Variants
Three other variants namely LVQ2, LVQ2.1 and LVQ3 have been developed by Kohonen.
Complexity in all these three variants, due to the concept that the winner as well as the
runner-up unit will learn, is more than in LVQ.
LVQ2
As discussed, the concept of other variants of LVQ above, the condition of LVQ2 is formed
by window. This window will be based on the following parameters −
x − the current input vector
yc − the reference vector closest to x
yr − the other reference vector, which is next closest to x
dc − the distance from x to yc
dr − the distance from x to yr
The input vector x falls in the window, if

Here, θ is the number of training samples.

Updating can be done with the following formula –

LVQ2.1
In LVQ2.1, we will take the two closest vectors namely yc1 and yc2 and the condition for
window is as follows
Adaptive Resonance Theory Network.
[Type here] [Type here] [Type here]