CSE 3175: ARTIFICIAL NEURAL NETWORKS

[Module 3]

3 September 2025 1
Syllabus

3 September 2025 2
 Lecture-16_17
Introduction
Cover’s Theorem (Separability of Patterns)
Radial Basis Functions
 Introduction

Supervised learning in a multilayer perceptron (MLP):

– Recursive technique known as stochastic approximation,
e.g., backpropagation algorithm.
Learn from errors

Solve the problem of classifying nonlinearly separable

patterns in a Introducing
hybridArtificial
manner, involving
Neural Networks Using thetwo stages
7 Ws and 1H

1. Transforms a given set of nonlinearly separable patterns

into a new set for which, under certain conditions, the
likelihood of the transformed patterns becoming linearly
separable is high.
2. The solution to the prescribed classification problem by
using least-squares estimation.
3 September 2025 4
 Radial Basis Function (RBF) in brief
• Implementation of a hybrid approach to pattern classification
by using a radial-basis function (RBF) network.
• Consists of 3 layers.
• The input layer is made up of source nodes (sensory units)
that connect the network to its environment.
• The second layer, consisting of hidden units, applies a
nonlinearIntroducing
transformation from the input space to the
Artificial Neural Networks Using the 7 Ws and 1 H

hidden (feature) space.

• The output layer is linear, designed to supply the response of
the network to the activation pattern applied to the input
layer.
• Design of neural network as a curve-fitting (approximation)
problem, e.g., RBF.
3 September 2025 5
 Cover’s Theorem on the Separability of Patterns

➢ While performing a complex pattern classification task using RBF, first

the problem is transformed into a high-dimensional space in a nonlinear
manner, and then the classes in the output layer are separated.
(Justification as per Cover’s theorem, 1965)
Introducing Artificial Neural Networks Using the 7 Ws and 1 H

“A complex pattern-classification problem, cast in a high-dimensional

space nonlinearly, is more likely to be linearly separable than in a low-
dimensional space, provided that the space is not densely populated.”

➢ i.e., linearly separable patterns are easy to classify

3 September 2025 6
This picture explains the concept of Radial Basis Function (RBF) in machine learning,
particularly in the context of classification using RBF networks.
Here’s the breakdown:
Top Image (Input Space):
You see two types of data points (red circles and yellow squares) scattered in a 2D
input space.
The two classes are not linearly separable in this space, meaning you can’t draw a
simple straight line (or plane) to separate them.
Bottom Left (Red Points Mapped):
The red circular class points are mapped into a new feature space using the RBF
kernel.
Each point’s position is transformed based on its distance from a center
(Gaussian-like transformation).
Bottom Right (Yellow Points Mapped):
Similarly, the yellow square class points are mapped into another region of the
new feature space.
Bottom Center (New Feature Space with Separation):
After transformation using the RBF kernel, the two classes are now more
separable in the higher-dimensional space.
This makes it easier for a classifier (like an RBF Network with an RBF kernel) to draw
a decision boundary between them.
 Introducing Artificial Neural Networks Using the 7 Ws and 1 H

3 September 2025 9
 Introducing Artificial Neural Networks Using the 7 Ws and 1 H

3 September 2025 10
 Introducing Artificial Neural Networks Using the 7 Ws and 1 H

3 September 2025 11
 Introducing Artificial Neural Networks Using the 7 Ws and 1 H

3 September 2025 12
 Introducing Artificial Neural Networks Using the 7 Ws and 1 H

3 September 2025 13
 Introducing Artificial Neural Networks Using the 7 Ws and 1 H

3 September 2025 14
 Introducing Artificial Neural Networks Using the 7 Ws and 1 H

3 September 2025 15
 Radial Basis Function Network

• Type of Multilayer Perceptron having one input layer, one output

layer, and with strictly one hidden layer.

• Hidden layer uses non nonlinear radial basis function as the

activation function,
• converting the input parameters into high high-dimensional
Introducing Artificial Neural Networks Using the 7 Ws and 1 H
space
• which is then fed to the network to linearly separate the
problem.

• Useful for interpolation, function approximation, time series

prediction, classification etc.

3 September 2025 16
 Radial Basis Function Network

The idea of Radial Basis Function (RBF) Networks derives from the theory of
function approximation. We have already seen how Multi-Layer Perceptron (MLP)
networks with a hidden layer of sigmoidal units can learn to approximate
functions. RBF Networks take a slightly different approach. Their main features
are:
Introducing Artificial Neural Networks Using the 7 Ws and 1 H
Basic Form of RBF

Input layer: Source node connected to the environment.

Hidden layer: Provide a set of functions that form a base for mapping into the
hidden layer.

Output Layer: Supplies Response.

3 September 2025 17
 Transformation of non-linear data to linear data

• Can be done in two ways

1. Increase the dimensionality (Eg, 2D to 3D)
2. Expand the direction (Horizontally), Compress the
direction (Vertically)
Introducing Artificial Neural Networks Using the 7 Ws and 1 H

Non linear data

Linear data

3 September 2025 18
 How RBF transforms the data?

• Consider one center c randomly.

• Draw multiple concentric circles.
• Data points might fall on any of the concentric circles;
get the radius r.
Introducing Artificial Neural Networks Using the 7 Ws and 1 H

3 September 2025 19
phi Radial Basis Function Neural Network

Introducing Artificial Neural Networks Using the 7 Ws and 1 H

3 September 2025 20
 Radial Basis Function Neural Network

Introducing Artificial Neural Networks Using the 7 Ws and 1 H

P is the dimensionality of the input feature space, and M is the

dimensionality of the transformed feature space where we have
imposed our RBF.
3 September 2025 21
 Introducing Artificial Neural Networks Using the 7 Ws and 1 H

3 September 2025 22
 Introducing Artificial Neural Networks Using the 7 Ws and 1 H

3 September 2025 23
 Introducing Artificial Neural Networks Using the 7 Ws and 1 H

3 September 2025 24
 Introducing Artificial Neural Networks Using the 7 Ws and 1 H

3 September 2025 25
 Introducing Artificial Neural Networks Using the 7 Ws and 1 H

3 September 2025 26
 Introducing Artificial Neural Networks Using the 7 Ws and 1 H

3 September 2025 27
 RBF Network Training

Introducing Artificial Neural Networks Using the 7 Ws and 1 H

3 September 2025 28
 Introducing Artificial Neural Networks Using the 7 Ws and 1 H

3 September 2025 29
Calculation of receptors:
Let’s look at the above example where we have M=3, so we need to determine
three t’s. So initially, we divide our feature vector space into three arbitrary clusters
and take their means as the initial receptors, then we need to iterate for every
sample feature vector and perform below steps:
➢ a) From the Introducing
selected Artificial
input feature vectorUsing
Neural Networks x, determine
the 7 Ws and 1distances
H of means
(t1,t2,t3) of three different clusters. Whichever mean distance is minimum, the
sample x will get assigned to that cluster.
➢ b) After x got assigned to different cluster all the means (t1,t2,t3) gets
recomputed.
➢ c) Perform step 1 and step 2 for all sample points.
➢ Once the iteration finishes, we will get the optimal t1,t2, and t3.
3 September 2025 30
 Introducing Artificial Neural Networks Using the 7 Ws and 1 H

3 September 2025 31
 Lecture-18
Radial-Basis-Function
Networks
 Introducing Artificial Neural Networks Using the 7 Ws and 1 H

3 September 2025 33
 How do RBF networks work?

• Input Vector: The network receives an n-dimensional input vector that

needs classification or regression.

• RBF Neurons: Each neuron in the hidden layer represents a prototype

vector from the training set. The network computes the Euclidean
distance between the input vector and each neuron’s center.

Introducing Artificial Neural Networks Using the 7 Ws and 1 H

• Activation Function: The Euclidean distance is transformed using a
Radial Basis Function (typically a Gaussian function) to compute the
neuron’s activation value. This value decreases exponentially as the
distance increases.

• Output Nodes: Each output node calculates a score based on a weighted

sum of the activation values from all RBF neurons. For classification, the
category with the highest score is chosen.

3 September 2025 34
 Characteristics of RBF

• Radial Basis Functions: These are real-valued

functions dependent solely on the distance from a
central point. The Gaussian function is the most
commonly used type.

• Dimensionality: The network’s dimensions correspond

Introducing Artificial Neural Networks Using the 7 Ws and 1 H
to the number of predictor variables.

• Center and Radius: Each RBF neuron has a center and

a radius (spread). The radius affects how broadly each
neuron influences the input space.

3 September 2025 35
 Architecture of RBF networks

• Input Layer

• Function: After receiving the input features, the

input layerIntroducing
sends them straight to the hidden layer.
Artificial Neural Networks Using the 7 Ws and 1 H

• Components: It is made up of the same number

of neurons as the characteristics in the input data.
One feature of the input vector corresponds to
each neuron in the input layer.

3 September 2025 36
 Architecture of RBF networks

• Hidden Layer

• Function: This layer uses radial basis functions (RBFs) to conduct

the non-linear transformation of the input data.

• Components:Introducing
Neurons in Neural
Artificial the Networks
buriedUsing
layer
the 7 apply the RBF to the
Ws and 1 H
incoming data. The Gaussian function is the RBF that is most
frequently utilized.

• RBF Neurons: Every neuron in the hidden layer has a spread

parameter (σ) and a center, which are also referred to as prototype
vectors. The spread parameter modulates the distance between the
center of an RBF neuron and the input vector, which in turn
determines the neuron’s output.
3 September 2025 37
 Architecture of RBF networks

• Output Layer

• Function: The output layer uses weighted sums to integrate the

hidden layer neurons’ outputs to create the network’s final output.

Introducing Artificial Neural Networks Using the 7 Ws and 1 H

• Components: It is made up of neurons that combine the outputs of
the hidden layer in a linear fashion. To reduce the error between
the network’s predictions and the actual target values, the weights
of these combinations are changed during training.

3 September 2025 38
 Radial Basis Function Neural Network

Introducing Artificial Neural Networks Using the 7 Ws and 1 H

3 September 2025 39
 Training Process of RBF networks

• An RBF neural network must be trained in three stages: choosing

the centers, figuring out the spread parameters, and training the
output weights.

• Step 1: Selecting the Centers

Introducing Artificial Neural Networks Using the 7 Ws and 1 H

• Techniques for Centre Selection: Centres can be picked at random
from the training set of data or by applying techniques such as k-
means clustering.

• K-Means Clustering: The centers of these clusters are employed as

the centers for the RBF neurons in this widely used center selection
technique, which groups the input data into k groups.

3 September 2025 40
 Training Process of RBF networks

• Step 2: Determining the Spread Parameters

• The spread parameter (σ) governs each RBF neuron’s area

of effect and establishes the width of the RBF.

Introducing Artificial Neural Networks Using the 7 Ws and 1 H

• Calculation:
• The spread parameter can be manually adjusted for each
neuron or set as a constant for all neurons.
• Setting σ based on the separation between the centers is a
popular method, frequently accomplished with the help of a
heuristic like dividing the greatest distance between centers
by the square root of twice the number of centers.
3 September 2025 41
 Training Process of RBF networks

• Step 3: Training the Output Weights

• Linear Regression: The objective of linear regression techniques,

which are commonly used to estimate the output layer weights, is to
minimize the error between the anticipated output and the actual
target [Link] Artificial Neural Networks Using the 7 Ws and 1 H

• Pseudo-Inverse Method: One popular technique for figuring out

the weights is to utilize the pseudo-inverse of the hidden layer
outputs matrix.

3 September 2025 42
 Types of RBF

• Gaussian Radial Basis Function

• It has a bell-shaped curve, monotonically decreasing with distance from the
center.
• Multiquadric Radial Basis Function
• monotonically increases with distance from the center.
• Inverse Multiquadric Radial Basis Function
• decreasingIntroducing
nature provides a smooth
Artificial Neural transition
Networks Using the from the1 H
7 Ws and centre outwards,
which can be beneficial for smoothing and regularisation.
• Thin Plate Spline Radial Basis Function
• often used in applications involving thin-plate splines, which are used for
surface interpolation and deformation.
• Cubic Radial Basis Function
• It has cubic polynomial behavior and is sometimes used in interpolation

3 September 2025 43
• c is the center and r is the radius

3 September 2025 44
 Interpretation of RBF network example

• [Link]
types-advantages-and-use-cases

3 September 2025 45
 RBF Algorithm

3 September 2025 46
 RBF Algorithm

3 September 2025 47
 Summary of RBF

• Initialize centers, widths, and weights.

• Transform input data using radial basis functions in the

hidden layer.

• Compute the output using a linear combination of hidden

layer outputs.

• Train the weights using a supervised learning approach.

• Use the trained model for prediction.

3 September 2025 48
 RBF PROS and CONS

PROS
• strong resistance to input noise
• easy to solve the problems that exist in datasets that have complex
non-linear distributions
• hidden patterns in the distribution can be generalized in a better
way
• interpret the exact meaning of each node present in the hidden layer
CONS
• performance can degrade with high-dimensional or large-scale
datasets requiring extensive computational resources for centroid
determination and training
• Classification is slow
3 September 2025 49
The choice between lower-dimensional and higher-dimensional
representations in machine learning, such as using
dimensionality reduction techniques like PCA (Principal
Component Analysis) to reduce dimensions or using methods
like Radial Basis Function (RBF) networks to increase
dimensions, depends on the specific problem you are trying to
solve and the characteristics of your data. Each approach has its
own purpose and use cases:

3 September 2025 50
 Dimensionality Reduction (Lower Dimension):

➢Purpose: Dimensionality reduction techniques like PCA are used

to reduce the number of features (dimensions) in your data while
preserving as much information as possible.
➢Use Cases:
Data Visualization: Lower-dimensional representations can be
helpful for visualizing high-dimensional data in a more
manageable form.

3 September 2025 51
 Dimensionality Reduction (Lower Dimension):

➢ Noise Reduction: By reducing dimensionality, you can remove

noise or redundant information from the data, which can
improve the performance of some machine learning algorithms.

➢ Simplification: In cases where high dimensionality leads to

computational or model complexity issues, dimensionality
reduction can make the problem more tractable.

3 September 2025 52
 Lecture-19
Radial-Basis-Function
Networks
Solved examples
Example 1
Example 2
Example 3 – Using K means clustering algorithm
Iteration 1:
Iteration 2:
 Lecture-20
K-Means Clustering
Example
Find New Centroids
 Lecture-21-22
Hybrid Learning Procedure for RBF
Networks
Recursive Least-Squares (RLS) algorithm
▪ The Recursive Least-Squares (RLS) algorithm is a powerful
technique for online learning and updating the weights of
neural networks, especially in adaptive systems like Radial
Basis Function (RBF) networks and other feedforward neural
networks.
▪ It is used in adaptive filtering and neural networks to minimize
the error between the predicted output and the actual output by
recursively updating the weight vector. The data is processed
sequentially, and the model parameters are updated in real-time.
▪ RLS is an extension of the least squares method that is
designed to update weight estimates recursively, making it
more efficient for systems that need to adapt in real-time to
changing data.
Overview of RLS Algorithm

• The Recursive Least-Squares algorithm aims to

minimize the sum of squared errors between the actual
output and the desired output in a recursive manner,
updating weights with each new sample.
• Unlike the simple least squares method that requires
batch processing of all available data, RLS updates the
weights iteratively with every new data point.
• This makes it useful for online learning, where data
becomes available over time.

[Link]
Key Idea

• At each step, RLS adjusts the weights of the neural

network to minimize the error on the new data point
while considering the previously learned information.
• It does so by maintaining an updated estimate of the
weights and a matrix that reflects the uncertainty or
sensitivity of the current model parameters to the new
data.
• This results in faster convergence compared to gradient-
based methods like standard backpropagation, especially
in the context of time-varying or noisy data.
[Link]
How RLS works in a simple single-layer neural network
(linear regression problem)
Advantages of RLS in Neural Networks

1. Faster Convergence: RLS generally converges much

faster than gradient-based algorithms, especially in
systems with rapidly changing data or non-stationary
environments.
2. Real-time Adaptation: RLS is well-suited for online
learning since it updates the weights with each new
data point, making it useful for applications like time-
series prediction and adaptive control.
3. Reduced Computational Complexity: By updating
the weights recursively, RLS avoids the need to
reprocess all past data, reducing the computational
load compared to batch least squares.
Applications of RLS in Neural Networks

1. Radial Basis Function (RBF) Networks: RLS is

commonly used in the output layer of RBF networks to
update the output weights after the RBF centers and
widths have been fixed (via unsupervised learning or
other methods).
2. Adaptive Filtering: RLS is widely used in adaptive
signal processing, where systems need to adjust to
changing conditions in real-time, such as noise
cancellation and echo suppression.
3. Time-Series Prediction: RLS is effective in
environments where data is time-dependent and needs
to adapt quickly to new trends or changes.
Hybrid learning procedure for RBF networks

➢The K-means algorithm for training the hidden layer is

applied first; it is then followed by the RLS algorithm
for training the output layer.
➢Hereafter, we refer to this hybrid learning procedure as
the “K-means, RLS” algorithm, aimed at training an
RBF network with the following composition:

Input layer. The size of the input layer is determined by

the dimensionality of the input vector x, which is denoted
by m0.
➢An attractive feature of the “K-means, RLS”
algorithm is its computational efficiency, which
follows from the fact that the K-means and RLS
algorithms are both computationally efficient in
their own individual ways.
➢The only questionable feature of the algorithm is the
absence of an overall optimality criterion that
combines the training of the hidden and output
layers, assuring the whole system of optimality in
some statistical sense.
Basically, the procedure operates in two stages:

Stage 1: applies the K-means clustering algorithm to train

the hidden layer in an unsupervised manner. Typically, the
number of clusters, and therefore the number of
computational units in the hidden layer, is significantly
smaller than the size of the training sample.

Stage 2: applies the recursive least-squares (RLS)

algorithm to compute the weight vector of the linear
output layer.

This two-stage design procedure has a couple of desirable

features: computational simplicity and accelerated
convergence.
 Thank You !!

3 September 2025 130