Hopfield Network – Continuous and

Discrete Models
1. Introduction
Hopfield Network was proposed by John Hopfield in 1982. It is a recurrent neural network used
for associative memory.
Unlike feedforward networks, Hopfield networks have feedback connections where neuron
outputs are fed back as inputs.
The network evolves dynamically until it reaches a stable equilibrium state.

2. Architecture of Hopfield Network

Key properties:
• Single layer network
• Fully connected neurons
• Symmetric weights
𝑊𝑖𝑗 = 𝑊𝑗𝑖

• No self-connection
𝑊𝑖𝑖 = 0

Neuron Variables
Each neuron 𝑗 has two variables:
• Internal state (membrane potential)
𝑢𝑗 (𝑡)

• Output of neuron
𝑣𝑗 (𝑡)

Relationship Between Internal State and Output

The output of the neuron is obtained by applying an activation function to the internal state:
𝑣𝑗 (𝑡) = 𝑔(𝑢𝑗 (𝑡))

where
• 𝑔(⋅)= activation function
• Common choices: sign function, sigmoid, or tanh
Meaning of Indices
• 𝑖→ index of input neuron
• 𝑗→ index of current neuron
• 𝑊𝑖𝑗 → weight from neuron 𝑖to neuron 𝑗
1. Each neuron collects signals from other neurons.
2. These signals produce the internal state 𝑢𝑗 .
 3. The activation function converts 𝑢𝑗 into output 𝑣𝑗 .

3. Continuous Hopfield Network Model

The dynamics of neuron 𝑗in a continuous Hopfield network are described by the differential
equation
𝑛
𝑑𝑢𝑗 (𝑡)
= −𝜂 𝑢𝑗 (𝑡) + ∑ 𝑊𝑗𝑖 𝑣𝑖 (𝑡) + 𝑖𝑗𝑏
𝑑𝑡
𝑖=1

Where
• 𝑢𝑗 (𝑡): internal state (membrane potential) of neuron 𝑗at time 𝑡
• 𝑣𝑖 (𝑡): output of neuron 𝑖
• 𝑊𝑗𝑖 : synaptic weight from neuron 𝑖to neuron 𝑗
• 𝜂: decay (leakage) constant ( Decay (or leakage) is the gradual reduction of the
neuron’s internal state over time in the absence of inputs, represented by the term
−𝜂𝑢𝑗 (𝑡)in the Hopfield network equation.)

• 𝑖𝑗𝑏 : external bias input to neuron 𝑗

• 𝑛: number of neurons in the network

Since solving the differential equation analytically is difficult, numerical simulation is used.
Using Euler's method, the neuron state is updated as
𝑑𝑢𝑗 (𝑡)
𝑢𝑗 (𝑡 + Δ𝑡) = 𝑢𝑗 (𝑡) + Δ𝑡
𝑑𝑡

Where
• Δ𝑡: small time step used for numerical integration

1. The term −𝜂𝑢𝑗 (𝑡)represents decay (leakage) of neuron activity.

2. The term ∑𝑊𝑗𝑖 𝑣𝑖 (𝑡)represents weighted inputs from other neurons.
3. The term 𝑖𝑗𝑏 represents external bias input.
The neuron state evolves over time until the network reaches a stable equilibrium state.
4. Discrete Hopfield Network – Learning Rule
In a discrete Hopfield network, patterns are stored using the Hebbian learning rule.
The weight matrix is computed as:
𝑷

𝑾 = ∑ 𝒙(𝒑) (𝒙(𝒑) )𝑻
𝒑=𝟏

where:
• 𝒙(𝒑) : 𝒑𝒕𝒉 stored pattern
• 𝑷: total number of stored patterns
• (𝒙(𝒑) )𝑻 : transpose of pattern vector
After computing the weight matrix, the diagonal elements are set to zero:
𝑾𝒊𝒊 = 𝟎

This ensures that neurons do not have self-connections.

Weight Calculation Example

Consider storing the following bipolar patterns:
𝑷𝟏 = [𝟏 − 𝟏 𝟏]
𝑷𝟐 = [−𝟏 𝟏 − 𝟏]

Step 1: Outer Product of 𝑷𝟏

𝟏
𝑷𝟏 (𝑷𝟏 )𝑻 = [−𝟏] [𝟏 − 𝟏 𝟏]
𝟏
𝟏 −𝟏 𝟏
= [−𝟏 𝟏 −𝟏]
𝟏 −𝟏 𝟏

Step 2: Outer Product of 𝑷𝟐

−𝟏
𝑷𝟐 (𝑷𝟐 )𝑻 = [ 𝟏 ] [−𝟏 𝟏 − 𝟏]
−𝟏
𝟏 −𝟏 𝟏
= [−𝟏 𝟏 −𝟏]
𝟏 −𝟏 𝟏

Step 3: Add the Matrices

𝟏 −𝟏 𝟏 𝟏 −𝟏 𝟏
𝑾 = [−𝟏 𝟏 −𝟏] + [−𝟏 𝟏 −𝟏]
𝟏 −𝟏 𝟏 𝟏 −𝟏 𝟏
 𝟐 −𝟐 𝟐
= [−𝟐 𝟐 −𝟐]
𝟐 −𝟐 𝟐

Step 4: Set Diagonal Elements to Zero

𝟎 −𝟐 𝟐
𝑾 = [−𝟐 𝟎 −𝟐]
𝟐 −𝟐 𝟎

This is the final weight matrix used by the Hopfield network.

Hopfield Recall Algorithm

Once the weights are computed, the network recalls stored patterns using the following
algorithm:
Step 1: Initialize the input pattern
𝒗(𝟎)

Step 2: Compute the neuron input

𝒏

𝒖𝒋 = ∑ 𝑾𝒋𝒊 𝒗𝒊
𝒊=𝟏

Step 3: Apply the activation function

𝒗𝒋 = sign(𝒖𝒋 )

+𝟏 if 𝒖𝒋 > 𝟎
𝒗𝒋 = { −𝟏 if 𝒖𝒋 < 𝟎
𝒗𝒋 (previous) if 𝒖𝒋 = 𝟎

Step 4: Update neuron outputs.

Step 5: Repeat steps 2–4 until the output vector becomes stable.

Example: Discrete Hopfield Network

A Hopfield network stores the following pattern:
𝑃 = [1 − 1 1]
A corrupted pattern is given as input:
𝑋 = [1 − 1 − 1]
Show how the network recalls the stored pattern.

Step 1: Compute Weight Matrix (Learning Phase)

The weight matrix is computed using the Hebbian rule:
𝑊 = 𝑥𝑥 𝑇
where
𝑥 = [1 − 1 1]
Outer Product
1
𝑥 𝑇 𝑥 = [−1] [1 − 1 1]
1
1 −1 1
= [−1 1 −1]
1 −1 1

Set Diagonal Elements to Zero

0 −1 1
𝑊 = [−1 0 −1]
1 −1 0
This matrix stores the pattern.
Step 2: Input Corrupted Pattern
𝑣(0) = [1 − 1 − 1]
Step 3: Compute Neuron Inputs
Formula:
𝑛

𝑢𝑗 = ∑ 𝑊𝑗𝑖 𝑣𝑖
𝑖=1
Neuron 1
𝑢1 = 0(1) + (−1)(−1) + (1)(−1)
𝑢1 = 1 − 1 = 0
𝑣1 = 𝑠𝑖𝑔𝑛(0) = 1
Neuron 2
𝑢2 = (−1)(1) + 0(−1) + (−1)(−1)
𝑢2 = −1 + 1 = 0
𝑣2 = −1
Neuron 3
𝑢3 = (1)(1) + (−1)(−1) + 0(−1)
𝑢3 = 1 + 1 = 2
𝑣3 = 1
Step 4: Updated Output
𝑣(1) = [1 − 1 1]
Step 5: Check Convergence
Initial input
[1 −1 −1]
New output
[1 −1 1]
This matches the stored pattern.
Therefore, the Hopfield network successfully recalls the correct pattern.
5. Stability Conditions of Hopfield Network
In a Hopfield Network, convergence to a stable equilibrium state is guaranteed using an
energy function (also called a Lyapunov function). This function measures how “stable”
the network state is. During the update of neurons, the energy always decreases or stays
the same, and finally the network settles at a minimum energy state, which corresponds
to a stored memory pattern.

1. Energy (Lyapunov) Function

The energy of the Hopfield network at time 𝑡is
1
𝐸(𝑡) = − 𝑣(𝑡)𝑇 𝑊𝑣(𝑡) − 𝑣(𝑡)𝑇 𝑖 𝑏
2

2. Meaning of Each Term

𝒗(𝒕)– State Vector
• 𝑣(𝑡)represents the output/state of all neurons at time 𝑡.
• Each neuron state is typically +1 or −1 in a discrete Hopfield network.
Example:
𝑣(𝑡) = [1, −1,1]

𝑾– Weight Matrix
• 𝑊contains the synaptic weights between neurons.
• For convergence, it must satisfy:
1. Symmetric weights
𝑤𝑖𝑗 = 𝑤𝑗𝑖

2. No self-connections
𝑤𝑖𝑖 = 0

These conditions ensure the energy function behaves properly.

𝒗(𝒕)𝑻 𝑾𝒗(𝒕)
• Represents the interaction energy between neurons.
• If neuron states agree with stored patterns, this term reduces energy.
The factor 1/2is used to avoid double counting connections, since 𝑤𝑖𝑗 and 𝑤𝑗𝑖 represent
the same connection.

𝒊𝒃 – Bias Vector
• 𝑖 𝑏 represents the bias or threshold input to neurons.
• It influences whether a neuron switches to +1 or −1.
The term
𝑣(𝑡)𝑇 𝑖 𝑏
represents external influence on neuron states.

3. Why This Function Ensures Convergence

When a neuron updates its state:
• If the new state reduces the energy, it is accepted.
• If it increases the energy, the update will not occur.
Thus,
𝐸(𝑡 + 1) ≤ 𝐸(𝑡)

Meaning the energy never increases.

Because the energy is bounded below, the network eventually reaches a minimum energy
state, which is a stable equilibrium (attractor state).

4. Interpretation
• High energy → unstable network state
• Low energy → stable pattern (stored memory)
The Hopfield network behaves like a ball rolling down a hill:
• The ball always moves downward (energy decreases).
• It eventually stops at the lowest valley (minimum energy).

5. Conditions for Convergence

For guaranteed convergence:
1. Weight matrix must be symmetric
2. No self connections 𝑤𝑖𝑖 = 0
3. Neurons updated asynchronously (one at a time)
4. Energy function must decrease after each update

6. Associative Memories
Associative memory is a memory system capable of retrieving stored information from
partial or noisy inputs.

Hopfield networks act as content-addressable memories where patterns are retrieved

based on similarity.

Types:
1. Auto-associative memory – retrieves the same stored pattern.
2. Hetero-associative memory – retrieves a different associated pattern.

Working:
1. Patterns are stored in weight matrix W.
2. A noisy version is presented as input.
3. The network iteratively updates neurons.
4. The system converges to the closest stored pattern.
Applications:
• Image reconstruction
• Pattern recognition
• Error correction
• Optimization problems

7. Outer Product Method

The Outer Product Method is a Hebbian learning rule used to compute the weight matrix.

Weight Formula:
1 𝑝
𝑊= ∑ 𝑧 (𝑘) (𝑧 (𝑘) )𝑇
𝑛 𝑘=1

Where:
n = number of neurons
z(k) = kth stored pattern

Steps:
1. Convert patterns into bipolar form (+1, -1).
2. Compute outer product of each pattern vector.
3. Sum all outer products.
4. Divide by number of neurons.
5. Set diagonal elements to zero.

Advantages:
• Simple to compute
• Direct implementation of Hebbian learning

Limitations:
• Limited storage capacity
• Pattern interference may occur

8. Pseudoinverse Matrix Method

This method computes the weight matrix by minimizing the error between stored patterns
and recalled patterns.

Let:
Z = matrix of stored patterns
R = matrix of noisy patterns

Network output:
V=WR

Error:
E= ||Z - V||

To minimize error, optimal weight matrix is:

𝑊 = 𝑍(𝑅 𝑇 𝑅)−1 𝑅𝑇

The term (𝑅 𝑇 𝑅)−1 𝑅𝑇 is known as the Moore–Penrose pseudoinverse.

Advantages:
• Reduces interference between patterns
• Better recall accuracy

Disadvantages:
• Higher computational cost
• Requires matrix inversion

9. Storage Capacity of Memories

Storage Capacity of Hopfield Network
One important question:
How many patterns can a Hopfield network store?
This is called Storage Capacity.

Hopfield Storage Capacity

Hopfield (1982) experimentally found that the number of patterns that can be stored is
approximately
𝐶𝐻𝑜𝑝𝑓 = 0.15𝑛

Where
• 𝑛= number of neurons
• 𝐶𝐻𝑜𝑝𝑓 = number of stored patterns

Example
If
𝑛 = 100

Then
𝐶𝐻𝑜𝑝𝑓 = 0.15 × 100 = 15

So the network can store about 15 patterns.

Maximum Storage Capacity

Further research gave a more accurate estimate:
𝑛
𝐶 𝑀𝑎𝑥 =
2ln (𝑛)
Where
• ln (𝑛)= natural logarithm
This gives the maximum patterns with very small error.

Perfect Recovery Capacity

If we require almost 100% correct recall, then
𝑛
𝐶 100% =
4ln (𝑛)

This means fewer patterns must be stored to guarantee perfect recall.

12. Example Storage Capacity Table

Neurons (n) Hopfield Capacity Maximum Capacity Perfect Recall
20 3 3.3 1.7
50 7.5 6.4 3.2
100 15 10.9 5.5
500 75 40.2 20.1
1000 150 72.4 36.2
Observation:
• As neurons increase, storage capacity increases.
• But perfect recall requires fewer patterns.

Limitation of Hopfield Network

One major weakness is low storage capacity.
It can store only about
0.15𝑛

patterns reliably.

Spurious States
If too many patterns are stored:
• Network creates spurious states.
These are:
• Stable states
• But not equal to stored patterns
This happens because of interference between patterns.

1Why Spurious States Occur

When storage exceeds capacity:
• Weight matrix becomes degenerate
• Eigenvalues overlap
• Network converges to false patterns