2.

1 ERROR-CORRECTION LEARNING (WITH HAYKIN DIAGRAM)

1. Definition

Error-correction learning is a supervised learning mechanism in which the synaptic weights are adjusted to minimize the
error between the desired output and the actual output.

2. Error Signal

e(n) = d(n) − y(n) Where:

d(n) = desired output y(n) = actual output e(n) = error signal

3. Linear Combiner Model

y(n) = Σ wᵢ(n) xᵢ(n) (or) y(n) = wᵀ(n) x(n)

4. Cost Function

J(n) = ½ e²(n) Objective: Minimize J(n)

5. Delta Rule (Weight Update)

Δw(n) = η e(n) x(n) w(n+1) = w(n) + η e(n) x(n)

6. HAYKIN DIAGRAM (VERY IMPORTANT – DRAW IN EXAM)

x1(n) ──►(w1)──┐
x2(n) ──►(w2)──┤
x3(n) ──►(w3)──┤
├──► Σ ───► y(n)
+ b ───┘
│
▼
(Comparator)
/ \
d(n) y(n)
\ /
▼
e(n) = d(n) - y(n)
│
▼
Weight Adjustment
Δw = η e(n) x(n)

7. Explanation of Diagram

• Inputs x₁, x₂, x₃ are applied to the neuron

• Each input is multiplied by weight (w₁, w₂, w₃)
• Summation block (Σ) produces output y(n)
• Output is compared with desired response d(n)
• Error signal e(n) is generated
• Error is used to adjust weights

👉 This forms a closed learning loop (adaptive system)

8. Learning Algorithm

Step 1: Apply input vector x(n)

Step 2: Compute output y(n)
Step 3: Compute error e(n)
Step 4: Update weights using delta rule
Step 5: Repeat until error is minimized

9. Error Surface Concept

• Error is a function of weights

• Learning minimizes error
• Surface is quadratic for linear systems
• Has single global minimum

10. Types of Learning

Batch Learning

• Uses entire dataset • Stable but slow

Sequential Learning

• Updates per sample • Faster but less stable

11. Convergence Condition

0 < η < 2 / λmax

12. Key Features

• Based on error minimization • Uses gradient descent

• Requires target output • Forms basis for backpropagation

13. Conclusion

Error-correction learning adjusts the weights of a neural network using the error signal to minimize the difference between
actual and desired output. The adaptive linear combiner with feedback loop is the fundamental structure used in this learning
method.

2.2 MEMORY-BASED LEARNING

1. Definition

Memory-based learning is a learning mechanism in which all the training data are stored in memory, and the output for a new
input is determined using stored examples.

2. Key Idea (Haykin Concept)

• No explicit training phase • Learning = storing input-output pairs

• Computation happens during recall

👉 Also called instance-based learning

3. Working Principle

Step 1: Store training samples (x, d)

Step 2: When new input arrives, compare with stored data
Step 3: Find closest matching samples
Step 4: Produce output based on similarity

4. Mathematical Representation

For input x, output is: y(x) = function of stored samples

Often based on distance measure: || x − xᵢ ||

5. Nearest Neighbor Rule

Output is determined by closest stored sample:

y(x) = d(k)

Where k = index of nearest neighbor

6. Diagram (Draw in Exam)

Training Data Storage

(x1, d1)
(x2, d2)
(x3, d3)
│
▼
MEMORY
│
▼
New Input x(n)
│
▼
Similarity Check
│
▼
Nearest Match
│
▼
Output y(n)
7. Characteristics

• Stores all data • No weight update

• Simple concept • High memory requirement

8. Advantages

• Easy to implement • No training time • Works well for small datasets

9. Limitations

• Large memory required • Slow during testing • Sensitive to noise

10. Applications

• Pattern recognition • Classification problems • Recommendation systems

11. Conclusion

Memory-based learning stores all training data and produces output by comparing new inputs with stored examples. It is
simple but requires large memory and computation during recall.

2.3 HEBBIAN LEARNING

1. Definition

Hebbian learning is an unsupervised learning rule in which the synaptic weight between two neurons is increased if both
neurons are activated simultaneously.
👉 Famous rule:
“Neurons that fire together, wire together.”

2. Basic Idea (Haykin Concept)

• No target output required • Learning depends only on input and output

• Strengthens correlation between neurons

3. Mathematical Representation

Weight update rule:

Δwᵢ = η xᵢ y

Where:
η = learning rate xᵢ = input y = output

4. Vector Form Δw = η x y

5. Working Principle

Step 1: Input x is applied Step 2: Output y is produced

Step 3: If x and y are active → weight increases Step 4: Repeat for all inputs

6. Hebbian Learning Diagram (Important)

x1 ──►(w1)──┐
x2 ──►(w2)──┤
x3 ──►(w3)──┤──► Σ ──► y
│

Weight Update:
Δw = η x y
7. Interpretation

• If xᵢ and y have same sign → weight increases • If opposite → weight decreases

8. Characteristics

• Unsupervised learning • Based on correlation • No error signal used

9. Advantages

• Simple rule • Biologically inspired • Useful for feature detection

10. Limitations

• No error correction • Weights may grow indefinitely• Not suitable for complex problems

11. Applications

• Associative memory • Pattern recognition • Feature extraction

12. Conclusion

Hebbian learning is an unsupervised learning rule based on correlation between input and output neurons. It strengthens
connections when neurons are activated together and forms the basis for many neural network models.

⭐ IMPORTANT DIFFERENCE (VERY EXPECTED QUESTION)

Feature Error-Correction Hebbian Memory-Based

Type Supervised Unsupervised Instance-based

Uses Error Yes No No

Weight Update Yes Yes No

Data Storage No No Yes

Example Backpropagation Correlation learning k-NN

2.4 COMPETITIVE LEARNING

1. Definition

Competitive learning is an unsupervised learning mechanism in which neurons compete among themselves to become active,
and only one neuron (winner) is allowed to respond to a given input.

2. Basic Idea (Haykin Concept)

• Neurons compete for activation • Only one neuron wins (Winner-Takes-All)

• Winning neuron updates its weights

3. Working Principle

Step 1: Input vector x is applied

Step 2: Each neuron computes its response
Step 3: Neuron with highest response wins
Step 4: Only winner updates weights
Step 5: Repeat for all inputs

4. Mathematical Representation

Winning neuron k satisfies:

|| x − wₖ || = minimum

👉 Closest weight vector wins

5. Weight Update Rule

For winning neuron: Δwₖ = η (x − wₖ)

For other neurons: Δw = 0

6. Diagram (Important for Exam)

x(n)
│
▼
┌───────────────┐
│ Competitive │
│ Layer │
└───────────────┘
│ │ │
▼ ▼ ▼
N1 N2 N3
│ │ │
└────┴────┘
│
Winner
│
▼
Weight Update (Only Winner)
7. Key Features

• Unsupervised learning • Winner-takes-all strategy

• Self-organizing behavior • Clustering of input data

8. Applications

• Clustering • Pattern recognition • Vector quantization

9. Advantages

• Simple and efficient • Performs automatic classification • No target output required

10. Limitations

• Only one neuron learns at a time • May ignore useful information

• Sensitive to initial weights

11. Conclusion

Competitive learning is an unsupervised learning rule where neurons compete, and only the winning neuron updates its
weights. It is widely used in clustering and pattern recognition tasks.

2.5 BOLTZMANN LEARNING

1. Definition

Boltzmann learning is a stochastic learning algorithm used in Boltzmann machines, where learning is based on probabilistic
principles and energy minimization.

2. Basic Idea (Haykin Concept)

• Network consists of stochastic neurons • Learning is based on probability

• System reaches minimum energy state 👉 Inspired by statistical mechanics

3. Energy Function

The energy of the network is defined as: E = − ½ Σ Σ wᵢⱼ sᵢ sⱼ

Where: wᵢⱼ = weight between neurons sᵢ, sⱼ = neuron states

4. Working Principle

Step 1: Initialize network randomly

Step 2: Neurons change state probabilistically
Step 3: System reaches equilibrium
Step 4: Compute probability distributions
Step 5: Update weights to reduce energy

5. Weight Update Rule

Δwᵢⱼ = η ( ⟨sᵢ sⱼ⟩data − ⟨sᵢ sⱼ⟩model )

Where: ⟨ ⟩ = expected value

6. Boltzmann Machine Diagram (Exam Diagram)

●──────●
│\ /│
│ \/ │
│ /\ │
│/ \│
●──────●

(All neurons connected) (Stochastic connections)👉 Fully connected network (no direction)

7. Key Features

• Stochastic (random behavior) • Uses probability distribution

• Based on energy minimization • Can escape local minima

8. Applications

• Optimization problems • Pattern recognition • Combinatorial problems

9. Advantages

• Finds global optimum • Avoids local minima

• Strong theoretical foundation

10. Limitations

• Very slow learning • Computationally expensive • Complex implementation

11. Conclusion

Boltzmann learning is a probabilistic learning method based on energy minimization. It is powerful for solving complex
optimization problems but requires high computational cost.

⭐ IMPORTANT COMPARISON (VERY EXPECTED)

Feature Competitive Boltzmann

Type Unsupervised Stochastic

Learning Winner neuron only All neurons

Concept Competition Energy minimization

Output Deterministic Probabilistic

Speed Fast Slow

2.6 CREDIT ASSIGNMENT PROBLEM

1. Definition

The credit assignment problem refers to the problem of determining how to assign responsibility (credit or blame) to
individual neurons or synaptic weights for the overall error in a neural network.

2. Basic Idea (Haykin Concept)

• In a neural network, many neurons contribute to the output

• When an error occurs, it is difficult to identify which neuron caused the error
• The problem is to distribute the error properly among all neurons

👉 This is called the credit assignment problem

3. Types of Credit Assignment

3.1 Structural Credit Assignment

• Assigns responsibility to neurons in different layers

• Determines how each neuron contributes to the final output

3.2 Temporal Credit Assignment

• Assigns credit over time • Important in sequential and time-dependent problems

4. Explanation

In multilayer networks:

• Output depends on many hidden neurons • Hidden neurons do not have direct target output • Error must be
propagated backward Hence, assigning credit becomes difficult

5. Diagram (Draw in Exam)

Input Layer → Hidden Layer → Output Layer

│ │ │
│ │ ▼
│ │ Error (e)
│ │ │
│ ◄─────────────┘
│ (Credit Assignment)
▼
Adjust Weights

6. Solution to Credit Assignment Problem

The main solution is: 👉 Backpropagation Algorithm

• Error is propagated backward • Each neuron receives a portion of error

• Weights are updated accordingly

7. Importance
• Essential for training multilayer networks • Helps in proper weight adjustment
• Enables deep learning

8. Applications

• Deep neural networks • Speech recognition

• Image processing • Time-series prediction

9. Advantages

• Enables learning in complex networks • Improves accuracy

• Supports multilayer architectures

10. Limitations

• Complex computation • Requires backpropagation

• May suffer from vanishing gradient problem

11. Key Points (For Revision)

• Problem of assigning error to neurons

• Occurs in multilayer networks
• Solved using backpropagation
• Two types: Structural and Temporal

12. Conclusion

The credit assignment problem is a fundamental issue in neural networks where the contribution of each neuron to the overall
error must be determined. It is effectively solved using backpropagation, enabling efficient training of multilayer neural
networks.

2.7 MEMORY

1. Definition

Memory in a neural network refers to the ability of the system to store and recall information learned from past experiences.

2. Basic Idea (Haykin Concept)

• Learning = storing knowledge in weights

• Memory is distributed across the network
• Not stored in a single location 👉 Called distributed memory

3. Types of Memory

3.1 Short-Term Memory

• Temporary storage • Depends on current input

3.2 Long-Term Memory

• Permanent storage • Stored in synaptic weights

4. Memory Representation

Memory is stored in: • Synaptic weights • Network structure

5. Associative Memory (Important)

• Stores patterns• Retrieves output using partial input 👉 Example: Pattern completion

6. Diagram (Exam Diagram)

Input Pattern → Neural Network → Stored Weights

│
▼
Memory Storage
│
▼
Output Recall
7. Characteristics

• Distributed storage • Fault tolerant • Parallel processing

8. Applications

• Pattern recognition • Image recall • Speech recognition

9. Conclusion

Memory in neural networks is distributed and stored in synaptic weights, allowing the system to recall and recognize patterns
effectively.

2.8 ADAPTATION

1. Definition: Adaptation is the ability of a neural network to adjust its parameters (weights) in response to changes in input
data or environment.

2. Basic Idea (Haykin Concept)

• Neural networks learn continuously • Weights are updated dynamically

• System adapts to new data

3. Mechanism of Adaptation

Adaptation is achieved by: • Learning rules • Weight updates • Error minimization

4. Mathematical Representation

w(n+1) = w(n) + Δw(n) Where: Δw(n) = change in weight

5. Types of Adaptation

5.1 Supervised Adaptation • Uses error signal

5.2 Unsupervised Adaptation • Based on input patterns

6. Diagram (Exam Diagram)

Input → Neural Network → Output
│
▼
Error / Signal
│
▼
Weight Adjustment
│
▼
Adapted System
7. Characteristics

• Dynamic learning • Improves performance • Responds to environment

8. Applications

• Adaptive filters • Control systems • Prediction systems

9. Conclusion

Adaptation enables neural networks to learn continuously by adjusting weights, making them flexible and capable of handling
changing environments.

2.9 STATISTICAL NATURE OF THE LEARNING PROCESS

1. Definition: The learning process in neural networks is statistical in nature because it involves estimation, probability, and
averaging over data.

2. Basic Idea (Haykin Concept) • Learning depends on data distribution • Uses statistical methods
• Involves randomness and uncertainty

3. Key Concepts

3.1 Expectation (Mean) E[x] = average value

3.2 Correlation Measures relationship between variables

3.3 Probability Distribution Describes likelihood of data

4. Cost Function (Statistical Form) J = E[e²(n)] 👉 Expected value of error

5. Importance

• Handles noisy data • Improves generalization • Ensures reliable learning

6. Diagram (Exam Diagram)

Input Data → Neural Network → Output

│
▼
Probability / Statistics
│
▼
Error Estimation
│
▼
Weight Update
7. Characteristics

• Based on probability • Uses averages (expectation) • Works with large datasets

8. Applications

• Machine learning models • Data analysis • Pattern recognition

9. Conclusion

The learning process in neural networks is statistical because it relies on probability, expectation, and data distribution to
update weights and improve performance.