Unit 1 – Module 1

Fundamental Concepts of Neural Networks

1. Introduction to Neural Networks

A Neural Network is a computational model inspired by the working of the human brain.
It consists of interconnected processing elements called neurons that work together to solve problems like
classification, prediction, pattern recognition, and decision-making.

Neural networks are widely used in:

• Image recognition
• Speech recognition
• Medical diagnosis
• Robotics
• Forecasting

2. Biological Neural Network

The human brain contains billions of nerve cells called neurons.

Structure of Biological Neuron

Components

1. Dendrites

• Receive signals from other neurons.

2. Cell Body (Soma)

• Processes received information.

3. Axon

• Carries signals away from the neuron.

4. Synapse

• Junction between two neurons where information is transmitted.

3. Artificial Neural Network (ANN)

An Artificial Neural Network is a simplified mathematical model of the biological brain.

Basic Structure of ANN

1. Input Layer

• Receives input data.

2. Hidden Layer

• Performs computations and feature extraction.

3. Output Layer

• Produces final result.

4. Comparison Between Biological and Artificial Neural Networks

Biological Neural Network Artificial Neural Network

Made of biological neurons Made of artificial neurons
Signals are electrochemical Signals are numerical
Very complex and adaptive Simplified mathematical model
Learns naturally Learns using algorithms

5. Neuron Model (Artificial Neuron)

The artificial neuron receives inputs, processes them, and produces an output.

Components of Artificial Neuron

Inputs (x₁, x₂, x₃...)

Input signals given to the neuron.

Weights (w₁, w₂, w₃...)

Importance assigned to each input.

Bias (b)

Helps in shifting activation function.

Summation Function

Calculates weighted sum.

Mathematical Representation

6. Activation Function

Activation function decides whether the neuron should activate or not.

Types of Activation Functions

1. Step Function

Produces binary output (0 or 1).

2. Sigmoid Function

Output ranges between 0 and 1.

3. Tanh Function

Output ranges from −1 to +1.

4. ReLU Function

Returns positive values and converts negative values to 0.

7. Basic Learning Rules

Learning means adjusting weights to improve performance.

1. Hebbian Learning Rule

• “Neurons that fire together wire together.”

• Weights increase if both neurons activate simultaneously.

2. Perceptron Learning Rule

• Used in classification problems.

• Adjusts weights based on error.

3. Delta Learning Rule

• Minimizes error using gradient descent.

4. Competitive Learning

• Neurons compete with each other.

• Winner neuron gets activated.

8. History of Neural Networks

Year Development
1943 McCulloch and Pitts developed first neuron model
1958 Rosenblatt introduced Perceptron
1969 Minsky and Papert highlighted perceptron limitations
1986 Backpropagation algorithm became popular
Present Deep Learning and AI applications

9. Artificial Intelligence and Neural Networks

Artificial Intelligence (AI) is the broader field of making machines intelligent.

Neural Networks are a subset of AI used for:

• Learning from data

• Pattern recognition
• Prediction
• Decision-making

Relationship:

• AI → Machine Learning → Deep Learning → Neural Networks

Unit 1 – Module 2

Learning of a Single Neuron and Single Layer Neural Networks

1. Learning of a Single Neuron

A single neuron learns by adjusting its weights and bias based on input data and error.

The neuron:

1. Receives inputs
2. Multiplies inputs with weights
3. Calculates weighted sum
4. Applies activation function
5. Produces output
6. Updates weights during learning

2. Learning Process of a Single Neuron

Step 1: Initialize Weights

Weights are assigned small random values.

Step 2: Apply Inputs

Input values are given to the neuron.

Step 3: Calculate Net Input

Step 4: Generate Output

Activation function produces output.

Step 5: Compute Error

Step 6: Update Weights

Weights are modified to reduce error.

3. Single Layer Neural Network

A Single Layer Neural Network contains:

• One input layer

• One output layer
• No hidden layer

It is mainly used for simple classification problems.

4. Characteristics of Single Layer Neural Network

• Simple structure
• Easy to implement
• Fast learning
• Used for linear classification problems
• Cannot solve complex non-linear problems

5. Perceptron Model

The perceptron is the simplest single-layer neural network developed by Rosenblatt.

Working of Perceptron

1. Inputs are applied

2. Weighted sum is calculated
3. Activation function is applied
4. Output is generated
5. Weights are updated if output is incorrect

6. Perceptron Learning Rule

The perceptron learning rule adjusts weights to minimize classification error.

Weight Update Equation

7. Advantages of Single Layer Neural Networks

• Simple and efficient

• Requires less computation
• Suitable for basic pattern recognition

8. Limitations of Single Layer Neural Networks

• Cannot solve XOR problem

• Works only for linearly separable data
• Limited learning capability
Unit 2 – Module 1

Neurons and Neural Networks

1. Biological Neurons

A biological neuron is the basic functional unit of the nervous system responsible for receiving, processing, and
transmitting information.

Parts of Biological Neuron

1. Dendrites

• Receive signals from other neurons.

2. Cell Body (Soma)

 • Processes incoming information.

3. Axon

• Carries signals away from the neuron.

4. Synapse

• Junction through which neurons communicate.

2. Models of Single Neurons

A single artificial neuron is a mathematical model inspired by the biological neuron.

Components of Single Neuron Model

• Inputs
• Weights
• Bias
• Summation unit
• Activation function
• Output

3. Different Neural Network Models

Neural network models are classified based on architecture and learning process.

1. Single Layer Feed Forward Network

• Contains one input layer and one output layer.

• No hidden layer.
• Data flows in one direction.
2. Multilayer Feed Forward Network

• Contains one or more hidden layers.

• Used for complex problems.
• Most commonly used ANN model.

3. Recurrent Neural Network (RNN)

• Output is fed back as input.

• Has memory capability.
• Used in sequence prediction and speech recognition.

4. Multilayer Neural Networks

A multilayer neural network contains:

• Input layer
• One or more hidden layers
• Output layer

Features

• Can solve non-linear problems

• High learning capability
• Used in deep learning applications

Advantages

• Better accuracy
• Handles complex patterns
• Learns complicated relationships

Limitations
 • Requires more training time
• Computationally expensive

5. Backpropagation

Backpropagation is a supervised learning algorithm used for training multilayer neural networks.

It minimizes error by adjusting weights using gradient descent.

6. Steps in Backpropagation Algorithm

Step 1: Initialize Weights

Assign small random weights.

Step 2: Forward Propagation

Inputs move from input layer to output layer.

Step 3: Compute Error

Step 4: Backward Propagation

Error is propagated backward through the network.

Step 5: Update Weights

Step 6: Repeat

Training continues until minimum error is achieved.

7. Advantages of Backpropagation

• High accuracy
• Learns complex patterns
• Suitable for multilayer networks

8. Limitations of Backpropagation

• Slow training process

• Requires large data
• Can get trapped in local minima

Unit 2 – Module 2

Associative Memory, Single Layer Perceptrons, Least Mean Square Algorithm, Learning
Curves, Learning Rates, Perceptron

1. Associative Memory

Associative memory is a type of neural network memory that stores and recalls patterns based on association.

It retrieves stored information when a related input is provided.

Types of Associative Memory

1. Auto Associative Memory

• Input pattern and output pattern are the same.

• Used for pattern completion.

2. Hetero Associative Memory

• Input and output patterns are different.

• Used for pattern mapping.

2. Single Layer Perceptrons

A Single Layer Perceptron is the simplest neural network having:

• Input layer
• Output layer
• No hidden layer
It is mainly used for binary classification.

Working of Single Layer Perceptron

1. Inputs are applied.

2. Weighted sum is calculated.
3. Activation function is applied.
4. Output is generated.
5. Weights are updated if error exists.

Perceptron Equation

3. Least Mean Square (LMS) Algorithm

The Least Mean Square algorithm is a supervised learning algorithm used to minimize mean square error.

It is also called the Widrow-Hoff learning rule.

Objective

To adjust weights such that the error becomes minimum.

4. Learning Curves

A learning curve represents the performance improvement of a neural network during training.

It shows relation between:

• Number of training iterations

• Error or accuracy

Characteristics

• Error decreases with training.

• Accuracy increases gradually.
• Helps evaluate learning performance.

5. Learning Rates

Learning rate determines how much weights are updated during training.

It is represented by η\etaη.

Effects of Learning Rate

Small Learning Rate

• Slow learning
• More stable

Large Learning Rate

• Faster learning
• May become unstable
6. Perceptron

A perceptron is a linear classifier introduced by Rosenblatt.

It classifies input data into different categories.

Components

• Inputs
• Weights
• Bias
• Summation unit
• Activation function

Perceptron Learning Rule

Advantages

• Simple implementation
• Fast training
• Suitable for linear problems

Limitations

• Cannot solve non-linear problems

• Fails for XOR problem
Unit 3 – Module 1

Multilayer Perceptrons: XOR Problem, Backpropagation Algorithm, Heuristics for

Improving Backpropagation Algorithm, Examples

1. Multilayer Perceptrons (MLP)

A Multilayer Perceptron is a feedforward neural network containing:

• Input layer
• One or more hidden layers
• Output layer

It uses the backpropagation algorithm for learning.

Features of MLP

• Solves complex problems

• Handles non-linear data
• Uses supervised learning
• High prediction accuracy

2. XOR Problem

The XOR (Exclusive OR) problem is a classic problem in neural networks used to show the limitation of a
single layer perceptron.

XOR Logic
The output is:

• 1 when inputs are different

• 0 when inputs are the same

Truth Table

X1 X2 Output
0 0 0
0 1 1
1 0 1
1 1 0

Why Single Layer Perceptron Fails

A single layer perceptron can solve only linearly separable problems.

In the XOR problem, data points cannot be separated using a single straight line, so the perceptron fails.

Solution

A Multilayer Perceptron (MLP) with one or more hidden layers can solve the XOR problem because it can learn
non-linear decision boundaries.

Importance of XOR Problem

• Demonstrated limitation of single layer networks

• Led to development of multilayer neural networks
• Important application of backpropagation algorithm

3. Backpropagation Algorithm

Backpropagation is a supervised learning algorithm used for training multilayer neural networks.

It adjusts weights by propagating error backward through the network.

4. Steps of Backpropagation Algorithm

Step 1: Initialize Weights

Assign small random values to weights and bias.

Step 2: Forward Propagation

Inputs pass from input layer to output layer.

Net Input Equation

Step 3: Calculate Output Error

Step 4: Backward Propagation

Error is propagated backward from output layer to hidden layer.

Step 5: Update Weights

Step 6: Repeat Process

Training continues until minimum error is achieved.

5. Heuristics for Improving Backpropagation Algorithm

Heuristics are techniques used to improve training performance and convergence speed.

1. Proper Learning Rate Selection

• Small learning rate gives stable learning.

• Large learning rate gives faster learning but may become unstable.

2. Momentum Term

Helps avoid oscillations and speeds up convergence.

Momentum Equation
3. Weight Initialization

• Weights should be initialized with small random values.

4. Normalization of Inputs

• Input values should be scaled to improve training.

5. Use of Hidden Layers

• Appropriate number of hidden neurons improves performance.

6. Stopping Criteria

Training stops when:

• Error becomes very small

• Maximum epochs are reached

6. Examples of Backpropagation Applications

• Handwriting recognition
• Speech recognition
• Image classification
• Medical diagnosis
• Stock market prediction

Advantages of Multilayer Perceptron

• Solves non-linear problems

• High accuracy
• Learns complex patterns

Limitations of Multilayer Perceptron

• Requires more computation

• Slow training process
• Needs large training data
Unit 3 – Module 2

Self-Organizing Neural Networks, Learning of Self-Organizing Network, Self-Organizing

Feature Map

1. Self-Organizing Neural Networks (SONN)

Self-Organizing Neural Networks are unsupervised learning networks that learn patterns automatically from
input data without target outputs.

They organize neurons based on similarities in input patterns.

Features

• Uses unsupervised learning

• Groups similar data together
• Performs clustering and pattern recognition
• Learns automatically from input data

2. Learning of Self-Organizing Network

The network learns by competition among neurons.

The neuron whose weight vector is closest to the input vector becomes the winner neuron.

Steps in Learning Process

Step 1: Initialize Weights

Weights are assigned small random values.

Step 2: Apply Input Vector

Input pattern is presented to the network.

Step 3: Compute Distance

Distance between input vector and weight vector is calculated.

Step 4: Select Winner Neuron

Neuron with minimum distance is selected.

This process is called competitive learning.

Step 5: Update Weights

Weights of winner neuron are adjusted toward input vector.

Step 6: Repeat Process

Training continues until stable organization is achieved.

3. Self-Organizing Feature Map (SOFM)

Self-Organizing Feature Map, also called Kohonen Map, is a neural network developed by Teuvo Kohonen.

It converts high-dimensional input data into a low-dimensional map while preserving data relationships.

Architecture of SOFM

SOFM contains:

• Input layer
• Competitive layer (output layer)

Each neuron competes to become the winner.

Working of SOFM

1. Input vector is applied.

2. Distance between input and neurons is computed.
3. Winner neuron is selected.
4. Winner neuron and neighboring neurons update weights.
5. Feature map is formed after repeated learning.

Characteristics of SOFM

• Unsupervised learning
• Topological mapping
• Competitive learning
• Preserves neighborhood relationships

Applications of SOFM

• Pattern recognition
• Image processing
• Data clustering
• Speech recognition
• Data visualization

Advantages of SOFM

• Reduces data dimensions

• Finds hidden patterns
• Organizes similar data together

Limitations of SOFM

• Training can be slow

• Difficult parameter selection
• Sensitive to initialization
Unit 4 – Module 1

Radial Basis Function Networks: Interpolation, Regularization, Learning Strategies

1. Radial Basis Function (RBF) Network

A Radial Basis Function Network is a type of artificial neural network used for:

• Classification
• Function approximation
• Pattern recognition

It is a feedforward neural network with:

• Input layer
• Hidden layer with radial basis functions
• Output layer

2. Structure of RBF Network

1. Input Layer

Receives input data.

2. Hidden Layer

Uses radial basis activation functions.

The most common activation function is the Gaussian function.

Gaussian Function
3. Output Layer

Produces final output using weighted sum.

3. Working of RBF Network

1. Input vector is applied.

2. Distance between input and center is calculated.
3. Radial basis function generates activation.
4. Output layer computes final output.

4. Interpolation

Interpolation is the process of estimating unknown output values using known data points.

In RBF networks:

• Hidden neurons act as interpolation functions.

• Network approximates smooth functions between data points.

Features of Interpolation

• Accurate function approximation

• Smooth output generation
• Useful in prediction problems

5. Regularization

Regularization is a technique used to reduce overfitting and improve generalization.

It controls complexity of the network.

Purpose

• Prevent memorization of training data

• Improve performance on unseen data
6. Learning Strategies in RBF Networks

Learning strategy refers to methods used to train the network.

1. Selection of Centers

Centers of radial basis functions are chosen using:

• Random selection
• Clustering methods like K-means

2. Selection of Width (Spread)

Determines area influenced by each neuron.

• Small width → narrow response

• Large width → smooth response

3. Weight Adjustment

Weights between hidden layer and output layer are updated to minimize error.

7. Advantages of RBF Networks

• Fast learning
• Simple structure
• Good approximation capability
• Handles non-linear problems

8. Limitations of RBF Networks

• Requires proper center selection

• Large networks may need many neurons
• Performance depends on spread parameter

Applications of RBF Networks

• Pattern classification
• Time series prediction
• Signal processing
• Function approximation
• Image recognition

Unit 4 – Module 2
Neural Network Trees, Neural Network Trees Based Learning Case Study

1. Neural Network Trees

Neural Network Trees combine the concepts of:

• Decision Trees
• Neural Networks

They use a tree-like structure where each node may contain a neural network or decision-making unit.

These models improve classification and decision-making performance.

2. Structure of Neural Network Trees

A Neural Network Tree consists of:

1. Root Node

• Starting point of the tree

• Receives input data

2. Internal Nodes

• Perform processing or classification

• May contain small neural networks

3. Leaf Nodes

• Produce final output or decision

3. Working of Neural Network Trees

1. Input data is applied to root node.

2. Data passes through internal nodes.
3. Each node performs classification or decision-making.
4. Final output is generated at leaf nodes.

4. Features of Neural Network Trees

• Hierarchical structure
• Combines learning and decision-making
• Handles complex classification problems
• Improves prediction accuracy

5. Advantages of Neural Network Trees

• Better interpretability than deep networks

• Handles large datasets
• Good classification performance
• Faster decision-making

6. Limitations of Neural Network Trees

• Complex training process

• Requires large memory
• Tree structure may become complicated

7. Applications of Neural Network Trees

• Medical diagnosis
• Pattern recognition
• Data mining
• Financial prediction
• Image classification

8. Neural Network Trees Based Learning Case Study

A case study explains practical implementation of Neural Network Trees in solving real-world problems.
Example Case Study: Medical Disease Classification

Objective

To classify patients as diseased or healthy using medical data.

Steps Involved

Step 1: Data Collection

Patient information such as:

• Age
• Blood pressure
• Sugar level
• Symptoms

is collected.

Step 2: Preprocessing

• Remove missing data

• Normalize input values

Step 3: Tree Construction

A neural network tree is created with:

• Root node
• Internal classification nodes
• Output leaf nodes

Step 4: Training

The network learns patterns from training data.

Weights are adjusted to minimize classification error.

Step 5: Testing

New patient data is given to evaluate accuracy.

Result

The model classifies diseases with improved accuracy and faster decision-making.

Importance of Neural Network Trees

• Combines strengths of neural networks and trees

• Useful for intelligent decision systems
• Supports complex real-world applications
Unit 5 – Module 1

Kohonen Self-Organising Maps: Self-Organising Map, SOM Algorithm, Learning Vector

Quantization

1. Kohonen Self-Organising Map (SOM)

Kohonen Self-Organising Map (SOM) is an unsupervised neural network developed by Teuvo Kohonen.

It converts high-dimensional input data into a low-dimensional map while preserving data relationships.

SOM is mainly used for:

• Clustering
• Pattern recognition
• Data visualization

2. Self-Organising Map (SOM)

A Self-Organising Map consists of:

• Input layer
• Competitive output layer

Neurons compete with each other to become the winner neuron.

Features of SOM

• Unsupervised learning
• Competitive learning
• Topological mapping
• Groups similar patterns together

Working of SOM

1. Input vector is applied.

2. Distance between input and neurons is calculated.
3. Winner neuron is selected.
4. Winner neuron updates its weights.
5. Neighboring neurons also update weights.
6. Map becomes organized after repeated training.

3. SOM Algorithm
The SOM algorithm trains the network through competitive learning.

Steps of SOM Algorithm

Step 1: Initialize Weights

Assign small random values to weights.

Step 2: Apply Input Vector

Input data is presented to the network.

Step 3: Calculate Distance

Usually Euclidean distance is used.

Step 4: Select Winner Neuron

Neuron with minimum distance becomes the winner.

This is called the Best Matching Unit (BMU).

Step 5: Update Weights

Weights are adjusted toward the input vector.

Step 6: Repeat Training

Process continues until stable mapping is achieved.

4. Learning Vector Quantization (LVQ)

Learning Vector Quantization is a supervised learning extension of SOM.

It classifies input vectors into predefined classes.

Working of LVQ

1. Input vector is applied.

2. Closest prototype vector is identified.
3. If classification is correct:
o Weights move closer to input.
4. If classification is wrong:
o Weights move away from input.

Weight Update Rule in LVQ

Correct Classification

Incorrect Classification

Applications of SOM and LVQ

• Pattern recognition
• Image processing
• Speech recognition
• Data clustering
• Medical diagnosis

Advantages of SOM

• Reduces dimensionality
• Finds hidden patterns
• Visualizes complex data

Limitations of SOM

• Training can be slow

• Sensitive to parameter selection
• Difficult for very large datasets
Unit 5 – Module 2

RBF Neural Networks and Support Vector Machines

1. RBF Neural Networks

Radial Basis Function (RBF) Neural Network is a feedforward neural network used for:

• Classification
• Pattern recognition
• Function approximation

It contains:

• Input layer
• Hidden layer with radial basis functions
• Output layer

2. Structure of RBF Network

1. Input Layer

Receives input data.

2. Hidden Layer

Uses radial basis activation functions.

The most common function is the Gaussian function.

3. Output Layer

Produces final output using weighted sum.

3. Working of RBF Network

1. Input vector is applied.

2. Distance from center is calculated.
3. Hidden neurons generate activations.
4. Output layer computes final result.

4. Features of RBF Networks

• Fast learning
• Simple architecture
• Good approximation capability
• Handles non-linear problems

5. Advantages of RBF Networks

• Faster training than multilayer perceptrons

• Better interpolation capability
• Good classification accuracy

6. Limitations of RBF Networks

• Requires proper center selection

• Performance depends on spread value
• Large datasets may require many neurons

7. Support Vector Machines (SVM)

Support Vector Machine is a supervised learning model used for:

• Classification
• Regression
• Pattern recognition

It separates data using an optimal hyperplane.

8. Hyperplane in SVM

A hyperplane is a decision boundary that separates different classes of data.

SVM selects the hyperplane with maximum margin.

9. Support Vectors

Support vectors are the data points closest to the hyperplane.

They are important because they determine the position of the hyperplane.

10. Working of SVM

1. Training data is provided.

2. SVM identifies support vectors.
3. Optimal hyperplane is created.
4. New data is classified based on hyperplane position.

11. Types of SVM

1. Linear SVM

Used for linearly separable data.

2. Non-Linear SVM

Uses kernel functions for non-linear data.

Common kernels:

• Polynomial kernel
• Radial basis kernel
• Sigmoid kernel

12. Advantages of SVM

• High classification accuracy

• Effective in high-dimensional data
• Works well with small datasets

13. Limitations of SVM

• Slow for large datasets

• Kernel selection is difficult
 • Requires high computation

14. Applications of SVM

• Image classification
• Face recognition
• Text classification
• Bioinformatics
• Handwriting recognition

Difference Between RBF Network and SVM

RBF Network SVM

Neural network model Machine learning classifier
Uses hidden neurons Uses support vectors
Faster training Better generalization
Used for approximation and classification Mainly used for classification