LECTURE NOTES

ON

Artificial Intelligence and

Machine Learning (BDS602)
2024 – 2025

MODULE 5:
ARTIFICIAL NEURAL NETWORKS
B. E VI Semester

Prof. Lavanya S
Assistant Professor
Department of CSE (DATA SCIENCE)
1. INTRODUCTION
• Biological Basis of ANNs(Artificial Neural Networks):
Artificial Neural Networks are inspired by the structure and functioning of the
human brain and nervous system, particularly the way neurons communicate
and process information.
• Neuron Analogy:
• Biological Neuron: Basic unit of the nervous system; receives signals
through dendrites, processes in the cell body, and transmits via the axon.
• Artificial Neuron (Perceptron): Receives inputs, processes them using
weights and an activation function, and produces output.
• Signal Transmission:
• In the Brain: Electrical impulses are transmitted between neurons across
synapses.
• In ANN: Numerical inputs (signals) are multiplied by weights and summed
up; passed through an activation function to simulate neuron firing.
• Network Formation:
• Human Brain: Made of billions of interconnected neurons forming a complex
network.
• ANN: Composed of layers (input, hidden, output) where artificial neurons
are interconnected to process and transmit data.
• Learning Mechanism:
• Human Brain: Learns through experience, strengthening or weakening
synaptic connections based on feedback.
• ANN: Learns through training using algorithms like backpropagation,
adjusting weights based on error signals.
• Response to Stimuli:
• Nervous System: Reacts to sensory inputs with appropriate motor outputs.
• ANN: Processes input data (like images or text) and provides appropriate
output (like classification or prediction).
• Adaptability:
• Both systems adapt and learn from experience.
• ANNs improve performance over time with more data, just as the human
brain becomes better with practice.
2. Biological neurons
• Definition: A biological neuron is the basic building block of the nervous
system, responsible for receiving, processing, and transmitting information
through electrical and chemical signals.
• Key Parts of a Biological Neuron:
1. Dendrites: Tree-like extensions that receive incoming signals (impulses)
from other neurons.
2. Cell Body (Soma): The control center of the neuron; processes incoming
signals and contains the nucleus.
3. Axon: A long fiber that carries signals away from the cell body to other
neurons or muscles.
4. Myelin Sheath: A fatty layer covering the axon, which increases the speed
of signal transmission.
5. Axon Terminals (Synaptic Endings): Endpoints of the axon where signals
are transmitted to the next neuron across a synapse.
6. Synapse: The small gap between neurons where chemical
neurotransmitters carry the signal to the next neuron.

A Biological Neuron
• Signal Flow:
Input → Dendrites → Soma → Axon → Axon Terminals → Synapse → Next
Neuron
• Working:
• A biological neuron works by receiving signals through its dendrites from
other neurons or sensory organs.
• These signals, often chemical in nature, are converted into small electrical
impulses and transmitted to the cell body (soma), where the signals are
integrated.
• If the combined signal strength crosses a specific threshold, the neuron
generates an action potential—a rapid electrical impulse—which travels
down the axon.
• The axon, often covered with a myelin sheath, speeds up this transmission.
 • When the impulse reaches the axon terminals, it triggers the release of
neurotransmitters into the synapse (the gap between neurons), which then
pass the signal to the next neuron’s dendrites, continuing the communication
process within the nervous system.

3. ARTIFICIAL NEURONS
• Definition: An artificial neuron (also called a perceptron) is a mathematical
model inspired by the biological neuron. It is the basic unit of an Artificial
Neural Network (ANN) and performs simple computations to process and
transfer data.
• Working of an Artificial Neuron:
• Inputs (x1,x2,...,xn ):
The neuron receives multiple numerical inputs, similar to signals received
by dendrites.
• Weights (w1,w2,...,wn ):
Each input is multiplied by a corresponding weight, representing the
importance of the input.
• Summation Function:
The weighted inputs are summed:

• Activation Function (f(z)f(z)f(z)):

The summed value is passed through an activation function (like Sigmoid,
ReLU, or Tanh) to determine the neuron's output.

• Output:
The final output is passed to the next layer or as final output depending on
the architecture.

An Artificial Neuron
• Key Notes:
• Weights and bias control how the neuron responds to input.
• Activation function introduces non-linearity into the model.
• It is the building block of deep learning models.

• SIMPLE MODEL OF AN ARTIFICIAL NEURON (McCulloh-

Pitts:MCP Model):
• The McCulloch-Pitts model, proposed in 1943 by Warren McCulloch and
Walter Pitts, is the earliest and most fundamental model of an artificial
neuron. It simplifies the working of a biological neuron into a binary logic
system.

• Key Concepts of the Model:

• Inputs (x1,x2,...,xn ):
The neuron receives multiple binary inputs (either 0 or 1).
• Weights (w1,w2,...,wn ):
Each input is assigned a weight, determining its importance.
• Summation Function:
Calculates the weighted sum of inputs:

• Threshold (θ):
A fixed value that the weighted sum is compared to.
• Activation (Step) Function:
The output y is determined using a step function:

• Mathematical Representation:

• Mathematical model of McCulloch and Pitts neuron

McCulloch and Pitts neuron mathematical model

• Working Method of McCulloch and Pitts Neuron:

• Input Reception: The neuron receives multiple binary inputs
(x1,x2,...,xn), each either 0 or 1.
• Weighted Summation: Each input is multiplied by a corresponding
weight (w1,w2,...,wn). The neuron computes the weighted sum:
 • Threshold Comparison: A threshold value θ is set. If the weighted
sum zzz is greater than or equal to θ, the neuron fires (outputs 1);
otherwise, it outputs 0.
• Output Generation: The neuron uses a step activation function:

• Advantages of MCP Neuron:

• Simple and easy to implement: Uses basic arithmetic and logic.
• Foundation for neural networks: Laid the groundwork for perceptrons and
deep learning.
• Good for logic operations: Can model simple logic gates (AND, OR,
NOT).
• Fast computation: Due to binary inputs and step function.
• Disadvantages of MCP Neuron:
• Binary limitation: Only handles binary input and output (0 or 1).
• No learning ability: Weights and threshold are fixed; no training or
learning mechanism.
• Cannot solve non-linear problems: Fails for XOR or complex decision
boundaries.
• No handling of real-world data: Cannot process continuous or noisy data.

• ARTIFICIAL NEURAL NETWORK STRUCTURE:

• An Artificial Neural Network is made up of layers of interconnected
artificial neurons (also called nodes or units) that mimic the way the
human brain processes information. The typical structure of an ANN
includes the following three main layers:

Artificial Neural Network Structure

 1. Input Layer:
• Receives raw data (features) from the external environment.
• Each neuron in this layer represents a single input feature (e.g., height,
weight, pixel value).
• It does not perform any computation, just forwards data to the next
layer.
2. Hidden Layer(s):
• One or more layers between input and output layers.
• Performs computations by applying weights, biases, and activation
functions.
• Responsible for feature extraction and pattern recognition.
• Each hidden layer neuron processes a weighted sum of inputs and
passes the result through an activation function (e.g., ReLU, sigmoid).
3. Output Layer:
• Produces the final result of the network.
• The number of neurons in this layer depends on the task:
o 1 neuron for binary classification
o Multiple neurons for multi-class classification or regression tasks.

• ACTIVATION FUNCTION:
• Definition: An activation function is a mathematical function applied to
the output of each neuron that introduces non-linearity into the model.
It decides whether a neuron should be activated or not, helping the
network learn complex patterns.
• Need for Activation Function:
• Introduces non-linearity to model complex data distributions.
• Controls the flow of information in the network.
• Enables learning of non-linear decision boundaries.
• Helps in gradient-based optimization during backpropagation.
• Advantages:
• Makes neural networks powerful and able to solve real-world
problems.
• Allows modeling of complex relationships.
• Helps networks converge faster during training (e.g., ReLU).
• Provides differentiability needed for training via gradient descent.
• Disadvantages:
• Some functions (like sigmoid, tanh) can cause vanishing gradients.
• Improper choice can lead to slow learning or dead neurons (especially
with ReLU).
• Different problems require different activation functions — no
universal best.
• Different Activation functions are as shown in the following table
4. Perceptron and learning theory

• Perceptron is the simplest type of artificial neural network model introduced by

Frank Rosenblatt in 1958.
• It is a binary classification algorithm that decides whether an input, represented by a
vector of numbers, belongs to one class or another.
• A perceptron consists of input features, weights, a bias term, a summation function,
and an activation function (typically the step function).

Perceptron Model

• Structure of Perceptron:
• Inputs (x₁, x₂, ..., xₙ) – the features of data.
• Weights (w₁, w₂, ..., wₙ) – each input has an associated weight.
• Summation Function – computes the weighted sum:

• Activation Function (usually Step Function) – determines the output:

• Working:
• Take inputs and multiply with corresponding weights.
• Add the bias term.
• Pass the result to an activation function.
• Output is either 0 or 1 depending on the threshold.

• The Perceptron algorithm is as follows

• LEARNING THEORY
• Learning Theory (specifically, computational or statistical learning theory) is
a framework that explains how machines can learn from data and make
predictions.

• It provides theoretical foundations to understand:

o How well a model generalizes to unseen data.
o How much data is needed to learn accurately.
o What types of functions can be learned (hypothesis space).
o The trade-offs between bias and variance.
•
• Key Concepts:
• Hypothesis Space (H): Set of all functions the learning algorithm can
choose from.
• True Error vs. Empirical Error:
o True Error (Generalization Error): Error on all possible data
(unknown).
o Empirical Error (Training Error): Error on the training data.
• PAC Learning (Probably Approximately Correct):A framework that
defines when a concept can be learned:
 With high probability (1 – δ), the hypothesis will have low error (ε).
• VC Dimension (Vapnik-Chervonenkis): A measure of model
complexity: the higher the VC dimension, the more complex the model.
• Overfitting & Underfitting:
o Overfitting: Model fits training data too well, poor generalization.
o Underfitting: Model too simple to capture underlying pattern.

• Relation between Perceptron and Learning Theory

o The Perceptron Convergence Theorem proves that if data is linearly separable,
the algorithm converges in finite steps.
o It’s an early example of online learning, updating model after each example.
o Perceptron fits into the broader framework of learning theory, helping us
understand its capabilities and limitations.

• Solved Example
• THE XOR PROBLEM:

• The XOR problem is a classic example in neural networks and machine learning
that demonstrates the limitation of a single-layer perceptron. It played a major
role in the development of more advanced neural network architectures
• XOR (Exclusive OR) is a binary function defined as:
Input A Input B A XOR B •
0 0 0
0 1 1
1 0 1
1 1 0
Output is 1 if inputs are different, 0 if inputs are the same.
• In 1969, Minsky and Papert showed that perceptrons can’t solve XOR, which
stalled neural network research for a decade.
• In the 1980s, the introduction of backpropagation revived interest by enabling
training of multi-layer networks that can solve XOR.

• A single-layer perceptron can only solve linearly separable problems, where

data can be separated by a straight line (in 2D).
• However, the XOR function is not linearly separable — there's no straight line
that can divide the 1s and 0s in the output.
• Visual Explanation:
• If we plot the XOR inputs:
o (0,0) → 0
o (0,1) → 1
o (1,0) → 1
o (1,1) → 0
• Its observed that the points labelled ‘1’ are diagonal, and cannot be separated
from the ‘0’ points using a single straight line.
• Solution to XOR Problem:
o The XOR problem led to the development of multi-layer neural
networks (e.g., Multilayer Perceptron - MLP).
o A Multi-Layer Perceptron (MLP) with one hidden layer and a nonlinear
activation function like sigmoid or ReLU can solve the XOR problem.
• Typical MLP for XOR:
o 2 input neurons
o 1 hidden layer with 2 neurons
o 1 output neuron
• With proper weights and nonlinear activation, it can learn XOR.
• DELTA LEARNING RULE AND GRADIENT DESCENT

• Comparison: Delta Rule vs Gradient Descent

Feature Delta Rule Gradient Descent
Type Specific case General optimization
algorithm
Use case Single-layer perceptrons Multi-layer networks (deep
learning)
Loss Function Mean Squared Error Any differentiable function
Update formula Based on linear error Based on gradient of loss
Activation Function Usually linear/step Usually nonlinear (e.g.,
sigmoid)

• Types of Gradient Descent:

1. Batch Gradient Descent – Uses the whole dataset to compute gradients.
2. Stochastic Gradient Descent (SGD) – Uses one training example at a time.
3. Mini-batch Gradient Descent – Uses small batches of data; balances speed and
accuracy.
5. TYPES OF ARTIFICIAL NEURAL NETWORK

5.1 FEEDFORWARD NEURAL NETWORK

• A Feedforward Neural Network is the simplest and most commonly used type
of artificial neural network. It is called "feedforward" because the flow of
information is strictly in one direction — from input to output — without any
cycles or loops.

Model of Feedforward Neural Network

• Architecture:
• Input Layer:
▪ Accepts input features (e.g., pixels of an image or numerical data).
▪ Each node represents one feature.
• Hidden Layer(s):
▪ Perform computations using weights, biases, and activation
functions.
▪ Can have one or more hidden layers (deep networks).
• Output Layer:
▪ Produces final prediction (e.g., class label or regression value).
• Information Flow:
Input→Hidden Layer(s)→Output

• Working Mechanism:
o Inputs are multiplied by weights and passed through an activation
function (e.g., sigmoid, ReLU).
o Each neuron's output becomes the input for the next layer.
o Final output is compared to the target to compute error.
o During training, the weights are updated using algorithms like
gradient descent via backpropagation.

• Advantages:
o Simple to implement.
o Works well for problems where data is linearly or non-linearly
separable.
 o Foundation for deep learning.
• Disadvantages:
o Cannot handle sequential or time-series data.
o Performance may degrade without enough hidden units or training
data.

5.2 FULLY CONNECTED NEURAL NETWORK

• A Fully Connected Neural Network (FCNN) is a type of Feedforward Neural
Network where each neuron in one layer is connected to every neuron in the
next layer. It is also commonly known as a Dense Neural Network.

A Model of Fully Connected Neural Network

• Structure:
o Input Layer:
▪ Receives raw input data (e.g., pixel values, features).
o Hidden Layers:
▪ Each neuron receives input from every neuron in the previous layer.
▪ Applies a weighted sum + bias, then passes through an activation function
(e.g., ReLU, sigmoid).
o Output Layer:
▪ Produces the final output (e.g., probabilities for classification or a value for
regression).

• Working:
 • Advantages:
o Simple and general-purpose.
o Can approximate any continuous function (Universal Approximation
Theorem).
o Suitable for structured data like tabular datasets.

• Disadvantages:
o High number of parameters → prone to overfitting.
o Not efficient for high-dimensional inputs like images (better handled by
CNNs).
o Does not capture spatial or temporal dependencies.

5.3 MULTI-LAYER PERCEPTRON(MLP)

• A Multi-Layer Perceptron (MLP) is a class of feedforward artificial

neural networks that consists of at least three layers of nodes: an input
layer, one or more hidden layers, and an output layer. It is the foundational
architecture for most deep learning models.

A Model of Multi-Layer Perceptron

• Structure:
1. Input Layer:
• Takes the features from the dataset.
2. Hidden Layer(s):
 • One or more layers where each neuron is connected to all neurons in
the previous and next layer (fully connected).
• Non-linear activation functions (e.g., ReLU, sigmoid, tanh) are
applied.
3. Output Layer:
• Produces final predictions.
• Activation depends on the task (e.g., sigmoid for binary classification,
softmax for multi-class).

• Working Mechanism:

1. Perform forward propagation:

2. Compute the loss/error between predicted and actual output.
3. Apply backpropagation to compute gradients.
4. Update weights using gradient descent or its variants.

• Advantages:

o Can model non-linear and complex relationships.

o Works well for classification and regression.
o Flexible and widely used in deep learning.

• Disadvantages:
o Requires large training data and computational power.
o Sensitive to hyperparameters (like learning rate, number of layers).
o May overfit if not regularized.

5.4FEEDBACK NEURAL NETWORK

• A Feedback Neural Network (also known as a Recurrent Neural Network

(RNN)) is a type of artificial neural network where the output of some neurons
is fed back into the same or previous layer. This creates loops or cycles in the
network, enabling it to maintain memory of previous inputs — making it ideal
for sequential or time-dependent data.
 A Model of Feedback Neural Network

• Structure:
• Input Layer: Receives input sequence at each time step.
• Hidden Layer(s): Maintains internal memory by feeding previous outputs
back into the current input.
• Output Layer: Produces output for each step (or at the final step, depending
on the application).
• Working:
• At each time step t:

• Advantages:
o Can learn from time-series, sequential, or context-based data.
o Remembers past information due to feedback loops.

• Disadvantages:
o Training is complex due to vanishing/exploding gradients.
o Hard to capture long-term dependencies (solved by LSTM, GRU).
6. LEARNING IN MULTILAYER PERCEPTRON
• Learning in a Multi-Layer Perceptron (MLP) is the process of adjusting
weights and biases to minimize the error between the predicted and actual
outputs. This is done through supervised learning using a combination of
forward propagation, backpropagation, and gradient descent.

• Key Concepts Involved:

o Activation Functions: Non-linear functions like ReLU, Sigmoid, or Tanh
used after each layer.
o Epochs: One full pass over the entire dataset.
o Batch Size: Number of samples processed before weights are updated.
o Optimizer: Algorithm like SGD, Adam used for updating weights.

• Advantages:
o Can Model Non-Linear Relationships: MLPs can learn complex
functions using non-linear activation functions.
o Universal Function Approximation: With enough hidden neurons and
data, MLPs can approximate any continuous function.
o Flexible Architecture: Number of layers and neurons can be tuned to fit a
variety of problems.
o Widely Applicable: Suitable for classification, regression, and pattern
recognition tasks.
o Supports Supervised Learning: Can learn from labeled data efficiently
using backpropagation and gradient descent.
•
• The algorithm is as follows:
• Disadvantages:

o Requires Large Training Data: MLPs typically need a lot of data to

generalize well, especially for deeper networks.
 o Prone to Overfitting: Without proper regularization (like dropout or weight
decay), MLPs can memorize the training data.
o Computationally Expensive: Training can be slow and resource-intensive,
especially with many layers and nodes.
o Not Ideal for Sequential Data: MLPs do not handle time-series or sequence
data well (RNNs or LSTMs are preferred).
o Hyperparameter Sensitivity: Performance depends on tuning many
parameters (learning rate, epochs, layer size, etc.).
• Solved Example:
7. RADIAL BASIS FUNCTION NEURAL NETWORK
• A Radial Basis Function Neural Network (RBFNN) is a type of artificial neural
network that uses radial basis functions (usually Gaussian) as activation functions
in the hidden layer. It is primarily used for function approximation, pattern
recognition, and interpolation tasks.

• Need for RBFNN:

• Traditional feedforward networks (like MLPs) may struggle with local
feature learning or nonlinear decision boundaries.
• RBFNNs are useful when:
o Fast training is needed.
o The problem is non-linear and localized.
o You require high accuracy in function interpolation or classification with
fewer training examples.
 Architecture of RBFNN

• Structure:
RBFNN typically has three layers:

• Input Layer:
o Passes features directly to the hidden layer.
o No computation is performed here.
• Hidden Layer:
o Applies Radial Basis Functions (commonly Gaussian).
o Each neuron has a center and spread (σ).
o Computes the similarity between the input and its center.

• Output Layer:
• Produces the final result via a linear combination of hidden neuron
outputs.

• The Algorithm is as follows:

• Working of the model:
o Select centers for RBFs (usually via k-means or random samples).
o Compute the activation of each hidden neuron using the RBF formula.
o Train output weights using a supervised learning method (e.g., least squares).
o During inference, for a given input, the hidden layer computes RBF values, and
the output layer calculates the weighted sum.
• Advantages:
o Fast training speed (especially compared to deep networks).
o Good at local approximation due to localized activation functions.
o Simple architecture and easy to implement.
o Interpretable: Centers can represent prototypes of classes.

• Disadvantages:
o Poor scalability to high-dimensional data or large datasets.
o Requires careful selection of RBF centers and spreads.
o Sensitive to the choice of parameters (σ and number of centers).
o Overfitting risk if too many centers are used.

• Solved Example:
8. SELF-ORGANIZING FEATURE MAP
• A Self-Organizing Feature Map (SOFM), also known as a Self-Organizing Map
(SOM), is an unsupervised learning neural network model introduced by Teuvo
Kohonen. It is used to reduce the dimensions of data while preserving the topological
relationships between input features.
• Need for SOM:
o Helps visualize and understand high-dimensional data in 2D or 3D.
o Useful in clustering, pattern discovery, and feature extraction.
o Unlike supervised models, SOMs do not require labeled data.

• Structure:
o Input Layer:Takes high-dimensional data (e.g., x1,x2,...,xn ).
o Output Layer (Map/Grid):
o A grid of neurons (e.g., 2D lattice) where each neuron has a weight
vector of the same dimension as the input.
o Each neuron competes to represent the input.
 Network architecture of SOFM

• Working:

• Initialization:
o Each neuron’s weight vector is randomly initialized.
• Competition:
o For a given input vector, compute the Euclidean distance to all
neurons.
o The neuron with the minimum distance is the Best Matching Unit
(BMU).
• Cooperation:
o A neighborhood of the BMU is selected (adjacent neurons on the
grid).
• Adaptation:
o BMU and its neighbors update their weights to become more similar
to the input:

• Repeat for multiple epochs, gradually reducing learning rate and

neighborhood size.

• The Algorithm is as follows:

 • Advantages:
o Visualizes complex data in low dimensions.
o Preserves topological and metric relationships.
o Good at clustering and feature extraction.
o Works without labeled data (unsupervised learning).

• Disadvantages:
o Training time increases with large data and grid size.
o Requires careful tuning of learning rate and neighborhood size.
o Difficult to interpret for very large maps.
o Less effective for non-topological tasks (like direct classification).
• Solved Example:
9. APPLICATIONS OF ANN
10. PROS AND CONS OF ANN

11. CHALLENGES OF ANN