Module-2

Associative Memory
In soft computing, Associative Memory (AM) is a neural network model that
stores patterns and retrieves them based on similarity to a given input, acting
like human memory by associating keys with related outputs, perfect for pattern
recognition and noise handling through models like Hopfield
Networks and Bidirectional Associative Memories (BAMs). These networks
store data by mapping input patterns to outputs, enabling efficient recall from
partial or noisy inputs, crucial for fuzzy logic and neural computation.

1. Autocorrelation
Autocorrelation (also known as serial correlation in discrete time series)
quantifies the linear relationship between a variable's current value and its past
values at various time lags.
 Key Concept: It helps identify repeating patterns, seasonality, or trends
within a single dataset, even if obscured by noise.
 Interpretation:
o A high value at a specific lag suggests a strong dependence on the
value at that past time point.
 o A value near zero indicates little to no linear dependence.
 Applications:
o Signal Processing: Pitch detection in audio, identifying
periodicities in biological signals (ECG/EEG), and correcting
propagation delays in GPS systems.
o Statistics/Econometrics: Used in time series analysis for model
validation, determining the order of ARIMA models, and financial
market trend analysis.

Calculation of Autocorrelation
Mathematically, the autocorrelation coefficient is denoted by the symbol ρ
(rho) and is expressed as ρ(k), where 'k' represents the time lag or the number
of intervals between the observations. The autocorrelation coefficient is
computed using Pearson correlation or covariance.

For a time series dataset, the autocorrelation at lag 'k' (ρ(k)) is determined by
comparing the values of the variable at time 't' with the values at time 't-k'.
Cov( X t , X t −k )
ρ(k )=
σ (X t )⋅σ (X t −k )

Here,
 Cov is the covariance
 σ is the standard deviation

 Xt represents the variable at time 't'

Hebb Rule:
 When A and B are positively correlated, then increase the strength of the
connection between them.
 When A and B are negatively correlated, then decrease the strength of the
connection between them.
 In practice, we use the following formula to set the weights:
 p
W =∑ S T ( p)S ( p)
p=1

 where, W = weighted matrix

 T= Learning Rate
 S(p) : p-distinct n-dimensional prototype patterns
Training Algorithm
1. Initialize all weights for i= 1,2,3 ...n and j= 1,2,3 ...n such that: wij=0.
2. For each vector to be stored, repeat the following steps:
3. Set activation for each input unit i= 1 to n: xi = si.
4. Set activation for each output unit j= 1 to n: yj = sj.
5. Update the weights for i= 1,2,3 ...n and j= 1,2,3 ...n such that : wij (new)
= wij (old) + xiyj
Testing / Inference Algorithm:
For testing whether the input is 'known' and 'unknown' to the model, we need to
perform the following steps:
1. Take the weights that were generated during the training phase using
Hebb's rule.
2. For each input vector, perform the following steps:
3. Set activation in the input units equal to input vectors.

4. Set activation in output units for j= 1,2,3 ...n: y−i n j=∑

i
x i wij

5. Apply activation function for j= 1, 2, 3 ... n:

y j=f ( y−i n j )=
{1 ify −in j >0
−1 ify−i n j ≤ 0 }
NN recognises the input vector to be known if the output unit, after activation,
generated the same pattern as one stored in it.
2. Heterocorrelation
Heterocorrelation (more commonly referred to as cross-correlation in signal
processing and statistics) describes the statistical relationship between
two different variables or signals.
  Key Concept: Unlike autocorrelation, which compares a signal with
itself, heterocorrelation compares two distinct signals to see how they
relate to each other at different time shifts or points in space.
 Interpretation: It reveals how strongly one variable's changes are
associated with another variable's changes, and at what delay that
relationship is strongest.
Applications:
o NMR Spectroscopy: Heteronuclear correlation (HETCOR, HSQC,
HMQC) experiments are fundamental in organic chemistry and
structural biology for determining molecular structures by
identifying which atoms (e.g., hydrogen and carbon) .
o Signal Processing: Used to measure the similarity between two
different signals, such as a transmitted signal and a received one, in
radar.
Training Algorithms for Pattern Association: Outer Products Rule
 Exponential Bidirectional Associative Memory (BAM)
A Bidirectional Associative Memory (BAM) is a type of recurrent neural
network that stores and retrieves associated patterns (e.g., pairs of images or
data sets). The term "Exponential BAM" refers to advanced models within this
class that use specific mathematical functions (often exponential functions in
their stability analysis) to ensure efficient and stable recall, particularly when
dealing with complex dynamics, time delays, or high-order interactions.

Types of BAM
BAM is a neural network model that stores associations between two sets of
patterns.
Example:
 Pattern X ↔ Pattern Y
If you present X, BAM retrieves Y, and if you present Y, it retrieves X.
In the discrete BAM, all neuron values are binary or bipolar:
 Binary: {0, 1}
 Bipolar: {–1, +1}
Architecture
BAM contains two layers:
1. X-Layer (input layer):
Contains n neurons
2. Y-Layer (output layer):
Contains m neurons
Layers are fully interconnected with no intra-layer connections.
Training (Learning Rule)
Discrete BAM uses a Hebbian correlation rule, accumulating outer products.
If you want to store pattern pairs:
(X₁, Y₁), (X₂, Y₂), ..., (Xₚ, Yₚ)
Then the weight matrix W is computed as:
p
W =∑ X Tk Y k
k=1

Recall (Bidirectional Operation)

Forward recall
Y =sign(XW ) Backward recall
T
X =sign(Y W )
This process continues iteratively until the network reaches a stable state
(fixed point).

Key Features
 ✔ Hetero-associative – stores pairs
✔ Bidirectional – recall works both ways
✔ Convergent – steps move toward stable states
✔ clean noisy inputs – acts as error-correcting memory

Example (Simple)
X = [1, -1]
Y = [1, 1]
Weight matrix:

[ ]
W =X Y = 1 [ 1 1 ] =
T

−1 [1 1
−1 −1 ]
Now, if you give noisy X, BAM retrieves

clean Y.

Applications
 Pattern recognition
 Signal matching
 Error correction
 Memory recall systems
 Bio-inspired learning systems

What is Continuous BAM?

Continuous BAM is an extension of Discrete BAM where neuron activations
are continuous-valued instead of binary/bipolar.

Continuous BAM allows any real value, typically in the interval [–1, +1].
It uses continuous activation functions and differential equations to reach
stable states.
Architecture
It has two layers:
X-Layer (n neurons)
Y-Layer (m neurons)
Connections:
 Fully connected between layers
 No intra-layer connections
 Weight matrix W is real-valued (n × m)
Training (Learning Rule)
Same Hebbian correlation rule as discrete BAM, but weights become
continuous:
p
W =∑ X Tk Y k
k=1

Where patterns Xₖ and Yₖ are continuous-valued vectors.

CBAM Updates State)

CBAM updates neuron states using differential equations:
Forward dynamics (X → Y)
dy T
=− y + f (W x )
dt
Backward dynamics (Y → X)
dx
=−x+ f (Wy )Where:
dt
f (⋅)is a continuous activation function such as

o Sigmoid
o tanh
Activation Functions
Common uses,
(a) Hyperbolic Tangent (Most Common)
f (v )=tanh ⁡(v )
Maps values to (–1, +1) smoothly.
(b) Sigmoid
1
f (v )= −v Maps values to (0, 1).
1+e

Recall Process in Continuous BAM

Start with an initial input pattern (could be noisy):
 1. Compute Y using forward dynamics
2. Compute X using backward dynamics
3. Repeat until continuous convergence
4. Network settles into a stable point
The stable point corresponds to one of the stored pattern pairs.

Energy Function
Kosko proved that continuous BAM has a Lyapunov energy function:
−1 T −1 −1
E(x , y)= x Wy + ∫ f (x ) dx+ ∫ f ( y)dy This guarantees:
2
✔ Convergence
✔ No oscillation (if f is monotonic)
✔ Stable recall
Comparison: Discrete BAM vs Continuous BAM
Feature Discrete BAM Continuous BAM

Values Binary/Bipolar Real-valued

Synchronous Differential
Dynamics
asynchronous equations

Activation Sign function Tanh/sigmoid

Continuous
Stability Fixed points
attractors

Noise
Good Excellent
tolerance

Use cases Simple patterns Real-valued signals

Applications
 Signal processing
 Speech pattern mapping
 Continuous control systems
 Audio ↔ Text mapping
  Time-series pattern association
 Adaptive filters
Simple Example
X = [0.2, –0.4]
Y = [0.5, 0.1]
Weight matrix:

[ ]
W =X Y = 0.2 [ 0.5 0.1 ]=
T

−0.4 [0.10
−0.20
0.02
−0.04 ]
Recall evolves continuously using

tanh(Wx) and tanh(Wᵀy) until reaching a stable match.