UNIT IV NEURO FUZZY MODELING 6

ANFIS architecture – hybrid learning – ANFIS as universal approximator – Coactive Neuro fuzzy modeling
– Framework – Neuron functions for adaptive networks – Neuro fuzzy spectrum - Analysis of Adaptive
Learning Capability

ANFIS:

An adaptive neuro-fuzzy inference system or adaptive network-based fuzzy inference

system (ANFIS) is a kind of artificial neural network that is based on Takagi–Sugeno
fuzzy inference system. Since it integrates both neural networks and fuzzy logic
principles, it has potential to capture the benefits of both in a single framework. Its
inference system corresponds to a set of fuzzy IF–THEN rules that have learning
capability to approximate nonlinear functions. Hence, ANFIS is considered to be a
universal estimator. For using the ANFIS in a more efficient and optimal way, one can
use the best parameters obtained by genetic algorithm.

ANFIS Architecture:

1. Representing Takagi-Sugeno Fuzzy Model

For simplicity, we assume that the fuzzy inference sytem under consideration has two
inputs x and y and one output z. For a first-order Takagi-Sugeno fuzzy model, a common
rule set with two fuzzy if-then rules is the following:

Rule 1: If x is A1 and y is B1, then f1=p1x+q1y+r1;

Rule 2: If x is A2 and y is B2, then f2=p2x+q2y+r2;

Figure 1: (a) A two inputs first order Takagi-Sugeno fuzzy model with two rules; (b)
The equivalent ANFIS architecture.

Figure 1(a) illustrates the reasoning mechanism for this Takagi-Sugeno model; the
corresponding equivalent ANFIS architecture is as shown in Figure 1(b), where nodes of
the same layer have similar functions, as described next. (Here we denote the output of
the ith node in layer l as Ol,i )

Layer 1 Every node i in this layer is an adaptive node with a node function

where x (or y) is the input to node i and Ai (or Bi-2) is a linguistic label (such as "small"
or "large") associated with this node. In other words, O1,i is the membership grade of a
fuzzy set A ( =A1 , A2 , B1 or B2 ) and it specifies the degree to which the given input x
(or y) satifies the quantifier A.
where {ai, bi, ci} is the parameter set. As the values of these parameters change, the bell-
shaped function varies accordingly, thus exhibiting various forms of membership
function for fuzzy set A. Parameters in this layer are referred to as premise parameters.

Layer 2 Every node in this layer is a fixed node labeled anfis, whose output is the
product of all the incoming signals:

Each node output represents the firing strength of a rule. In general, any other T-norm
operators that perform fuzzy AND can be used as the node function in this layer.

Layer 3 Every node in this layer is a fixed node labeled N. The ith node calculates the
ratio of the ith rule's firing strength to the sum of all rules' firing strenghts:

For convenience, outputs of this layer are called normalized firing strengthes.

Layer 4 Every node i in this layer is an adaptive node with a node function:

where anfis is a normalized firing strength from layer 3 and {pi, qi, ri} is the parameter
set of this node. Parameters in this layer are referred to as consequent parameters.

Layer 5 The single node in this layer is a fixed node labeled anfis, which computes the
overall output as the summation of all incoming singals:

Thus we have constructed an adaptive network that is functionally equivalent to a Sugeno

fuzzy model.
2. Representing Tsukamoto Fuzzy Models:

Figure 2(a)

Figure 2: (b)

Figure 2: (a) A two-rule Tsukamoto fuzzy model; (b) The equivalent ANFIS
architecture

The extension from TS ANFIS to Tsukamoto ANFIS is straightforward, as show in

Figure 2, where the output of each rule (fi, i=1, 2) is induced jointly by a consequent
membership function and a firing strength.

3. Representing Mamdani Fuzzy Model

For the Mamdani fuzzy inference system with max-min composition, a corresponding
ANFIS can be constructed if discrete approximations are used to replace the integrals in
the centroid defuzzification scheme introduced in here. However, the resulting ANFIS is
much more complicated than either TS ANFIS or Tsukamoto ANFIS. The extra
complexity in structure and computation of Mamdani ANFIS with max-min composition
does not necessarily imply better learning capability or approximation power. If we adopt
sum-product composition and centroid defuzzification for a Mamdani fuzzy model, a
corresponding ANFIS can be constructed easily based on Theorem directly without using
any approximation at all.

ANFIS architecture and hybrid learning algorithm

At the core of the ANFIS architecture is the adaptive network that uses the supervised
learning algorithm. Let's understand this with a simple example. Consider that there are
two inputs, x and y, and an output, z. We can consider the use of two simple rules in the
method of if-then as follows:

Rule 1: If x is A1 and y is B1 then z1 = p1x + q1x + r1

Rule 2: If x is A2 and y is B2 then z2 = p2y + q2y + r2

A1, A2 and B1, B2 are the membership functions of each input x and y. p1, q1, r1 and p2,
q2, r2 are linear parameters of the fuzzy inference model.

Let's illustrate this with a diagram:

The ANFIS architecture in this case can be considered a five-layer neural network. The
first and fourth layers contain an adaptive node and the other layers contain fixed nodes,
as we have already seen in the previous chapters on ANNs. The network is illustrated in
the following diagram:
The ANFIS architecture in this case can be considered a five-layer neural network. The
first and fourth layers contain an adaptive node and the other layers contain fixed nodes,
as we have already seen in the previous chapters on ANNs. The network is illustrated in
the following diagram:

● Layer 1: This layer consists of two adaptive nodes that adapt to a function
parameter based on the input values (x and y). The output from each of these nodes
denotes the degree of membership corresponding to the input value (refer the
ANFIS premise in the previous diagram). The membership function, as we have
seen in the previous sections, can take any form (Gaussian, bell function, and so
on). The parameters in this layer are termed premise parameters:
o z1 = p1x + q1y + r1
o z2 = p2x + q2y + r2
● Layer 2: The nodes in this layer are fixed nodes that are non-adaptive in nature
and resemble a hidden layer node in a neural network. The output from these
nodes is obtained by multiplying the signal coming from the adaptive nodes and
delivered to the next layer nodes. The nodes in this layer represent the firing
strength of each of the rules that are inherited by the adaptive nodes in the
previous layer.

● Layer 3: The nodes in this layer are also fixed nodes. Each node is a calculated
value of the ratio between the nth rule's firing strength and the sum of all the rules'
firing strength. The overall result represents the normalized firing strength.
 ● Layer 4: The nodes in this layer are the adaptive nodes. In this layer, the
normalized firing strength from the previous layer nodes is multiplied with the
output from the rule functions ( p1x + q1x + r1 and p2y + q2y + r2). The output
parameters from this layer are called consequent parameters.
● Layer 5: This is the output layer and has one fixed output node resembling the
ANN. This node performs the summation on the signals from the previous layer.
This is the overall output of the ANFIS network. This represents the quantitative
actionable outcome from the fuzzy system. This output can be utilized in the
control loop and back-propagated for training and optimization, eventually
minimizing the error.

With this network topology in place, we can apply a hybrid learning algorithm in order to
optimize the output and reduce the error. The hybrid algorithm also ensures that we are
able to converge quicker and avoid local minima. The hybrid algorithm is a two-step
process that essentially tweaks the parameters for the first and fourth adaptive layers
based on the rule set.

During the forward pass, the parameters for the first layer (premise parameters) are kept
constant and the parameters for the fourth layer (consequent parameters) are adjusted
based on the recursive least square estimator (RLSE) method.

Note that the consequent layer parameters are linear and we can accelerate the
convergence rate in the learning process. Once the consequent parameters values are
obtained, the data is passed through the input space and the aggregated membership
functions, and the output is generated. The output is then compared with the actual
output.

When the backward pass is executed, the consequent parameters obtained from the first
step are kept constant and the premise parameters are tweaked with the learning method
of gradient descent or backward propagation. The output is once again generated with the
changed values for the premise parameters and compared with actual output for further
tuning and optimization. Use of this hybrid algorithm, which combines RLSE and
gradient descent, ensures faster convergence.

ANFIS as Universal Approximator

Coactive neuro-fuzzy modeling

Coactive neuro-fuzzy modeling Coactive neuro-fuzzy modeling is a hybrid approach that

combines elements of neural networks, fuzzy logic, and cooperative coevolutionary
algorithms to build predictive models or controllers for complex systems.

Coactive neuro-fuzzy modeling combines these three elements in the following way:

● Neural networks are used to capture the underlying relationships within the data or
the system being modeled. They provide a basis for function approximation and
learning from data.
● Fuzzy logic is integrated to handle uncertainty and linguistic variables. It helps in
interpreting and representing the data in a way that is more understandable and
interpretable, especially when dealing with real-world problems.
● Cooperative coevolutionary algorithms are employed to optimize the parameters
of the neuro-fuzzy model. These algorithms can efficiently search for the best
combination of fuzzy sets, membership functions, and neural network
architectures to create a model that accurately represents the system or data.

The coactive neuro-fuzzy modeling approach is particularly useful when dealing with
complex systems where traditional modeling techniques may struggle due to the presence
of uncertain or imprecise data. It allows for the development of models that can capture
complex relationships while maintaining transparency and interpretability through the use
of fuzzy logic.

FRAMEWORK:

Towards multiple inputs/outputs systems

 Multiple output anfis with nonlinear fuzzy rules

• Generalized anfis is called as CANFIS
• In CANFIS both NN and FIS play an active role in a effort to reach a specific goal

• Canfis has extended the notion of single- output system of ANFIS to produce multiple
outputs.
• One way to accomplish is to place as many ANFIS models side by side as the number
of required outputs.
 In CANFIS the antecedents are the same, but the consequents are different according
the number of outputs required.
• Fuzzy rules are constructed with shared membership values to express correlations
between outputs.
Multiple ANFIS :

 • In MANFIS no modifiable parameters are shared by the near ANFIS models.

• Each ANFIS has an independent set of fuzzy rules, which makes it difficult to realize
possible correlations between outputs.

• Also the adjustable parameters increase with the increase in the number of outputs.

Coactive Neuro-Fuzzy Modeling (CANFIS) is a hybrid modeling approach that combines neuro-
fuzzy systems with cooperative coevolutionary algorithms to build predictive models for
complex systems. This framework is designed to address problems that involve uncertainty,
imprecision, and complex relationships. The framework for coactive neuro-fuzzy modeling are:

1. Problem Definition:
o Begin by clearly defining the problem you want to solve or the system you want
to model. Identify the inputs, outputs, and the nature of the relationships between
them.
2. Problem Decomposition:
o Decompose the complex problem into smaller, more manageable subproblems or
subsystems. Each subsystem should be amenable to neuro-fuzzy modeling.
3. Cooperative Coevolutionary Algorithm:
o Apply a cooperative coevolutionary algorithm to optimize the parameters and
structures of the neuro-fuzzy subsystems. These algorithms simultaneously
optimize multiple subsystems, allowing them to interact and influence each
other's evolution.
o Cooperative coevolution helps in finding the best combination of fuzzy sets,
membership functions, and neural network architectures for each subsystem.
4. Neuro-Fuzzy Subsystems:
o Within each subsystem, implement a neuro-fuzzy system. This typically involves
the following components:
▪ Input Layer: Accepts input variables.
▪ Fuzzification Layer: Converts crisp input values into fuzzy sets using
membership functions.
▪ Rule-Based Inference Layer: Applies fuzzy logic rules to make
inferences and generate fuzzy outputs.
▪ Defuzzification Layer: Converts fuzzy outputs into crisp values.
5. Cooperative Interaction:
o The results or solutions obtained from each neuro-fuzzy subsystem interact
cooperatively. They can share information, such as optimized parameters, fuzzy
rules, or membership functions, to collectively enhance the modeling accuracy
and effectiveness across subsystems.
6. Integration:
o Integrate the individual neuro-fuzzy subsystems to create the final CANFIS
model. This integration may involve combining the outputs or rules from different
subsystems to form a cohesive model.
7. Evaluation and Refinement:
o Evaluate the overall performance of the CANFIS model using appropriate metrics
and validation techniques.
o Refine the CANFIS model based on the evaluation results. This may involve
further optimization, fine-tuning, or adjustments to improve model accuracy and
generalization.
8. Application:
o Once the CANFIS model is trained and refined, it can be applied to various tasks,
such as prediction, classification, control, or decision support, depending on the
specific problem domain.
Coactive neuro-fuzzy modeling offers a powerful approach for addressing complex and
uncertain systems by leveraging the adaptive capabilities of neuro-fuzzy systems and the
cooperative interactions facilitated by coevolutionary algorithms. This framework has found
applications in diverse fields, including control systems, prediction, optimization, and decision
support systems.

NEURON FUNCTIONS FOR ADAPTIVE NETWORKS

In adaptive neural networks, the functions of neurons (also referred to as nodes or units) play a
crucial role in processing information, learning from data, and adapting to different tasks. The
specific functions of neurons can vary depending on the type of neural network architecture
being used.

Here are some common functions of neurons in adaptive networks:

1. Linear Combination of Inputs:

o Neurons often compute a weighted sum of their input signals. Each input is
multiplied by a weight, and the weighted inputs are summed together. This
operation is expressed as:

o Here, "z" is the weighted sum, "w_i" are the weights associated with each input
"x_i," and "n" is the total number of inputs.
2. Activation Function:
o After computing the weighted sum, neurons typically pass the result through an
activation function, which introduces non-linearity into the network. Common
activation functions include:
▪ Sigmoid Function: Scales the output to a range between 0 and 1, often
used in traditional feedforward neural networks.
▪ Hyperbolic Tangent (tanh) Function: Scales the output to a range
between -1 and 1, similar to the sigmoid but centered at zero.
▪ Rectified Linear Unit (ReLU): Outputs the input if it's positive, or zero
otherwise. It's widely used in deep learning due to its efficiency and
effectiveness.
▪ Leaky ReLU: Similar to ReLU but allows a small, non-zero gradient for
negative inputs to prevent "dying" neurons.
▪ Softmax Function: Used in the output layer of classification networks to
normalize and produce class probabilities.
3. Learning Rules:
o Neurons in adaptive networks use learning rules to adjust their weights and
parameters based on the input data and learning objectives. Common learning
rules include:
▪ Gradient Descent: Adjusts weights in the direction of the negative
gradient of a loss or error function.
▪ Backpropagation: A specific form of gradient descent used in
feedforward neural networks with supervised learning.
▪ Hebbian Learning: Strengthening connections between neurons that fire
together, often used in unsupervised learning.
▪ Reinforcement Learning Rules: Used in reinforcement learning tasks to
update weights based on rewards and penalties.
4. Memory and Adaptation:
o Some neurons in recurrent neural networks (RNNs) and memory-augmented
networks have functions related to maintaining memory and adapting their
internal states over time.
o These neurons may have recurrent connections that allow them to store and
retrieve information over multiple time steps.
5. Gate Functions:
o In networks like Long Short-Term Memory (LSTM) and Gated Recurrent Unit
(GRU), gate functions control the flow of information and help with learning
long-range dependencies.
o Gate functions include input gates, forget gates, and output gates, which
determine what information is passed through the neuron.
6. Adaptive Functions:
o In some networks, neurons may have adaptive mechanisms, such as adaptive
learning rates or parameters that change dynamically during training.
 7. Custom Functions:
o Depending on the architecture and problem, custom functions can be designed for
neurons to perform specific computations or transformations tailored to the task at
hand.

The specific function of a neuron in an adaptive network depends on the architecture and the role
it plays within the network. Different types of networks, such as feedforward, recurrent,
convolutional, and memory-augmented networks, may employ neurons with different functions
to suit the requirements of the task they are designed for.

Neuro Fuzzy Spectrum.

When we apply advanced optimization techniques, the dilemma becomes more

Conspicuous.
Dealing with neuro-fuzzy systems, the problem known as dilemma between
interpretability and precision comes out and becomes more conspicuous.

It can be described by the neuro-fuzzy spectrum that emphasizes the trade-off between
input-output mapping precision of NNs and MF interpretability of FIS. Adaptive neuro-
fuzzy models like ANFIS and CANFIS stand between a completely understandable FIS
and a black box NN. As shown in Fig. 2, the neural learning should follow the vertical
route in such a way that the mapping precision is being improved while the
interpretability maintained. In practise, however, it often takes the diagonal route of
improving precision at the expense of interpretability. When efficient and advanced
learning algorithms are applied, the dilemma increases and the original interpretability
may be lost. Similar observations can be found when there is a high number of fuzzy

rules.

The first approach to alleviating the dilemma is the reduction of fuzzy rules, starting from
the conflicting rules, namely rules which have the same IF part but a different THEN
part. Conflicting rules are useful to define the system uncertainties but they deteriorate
the system interpretability at the same time. A more mapping precision with the same
computational effort and interpretability maintained can be achieved by setting up
nonlinear rules or neural rules with more learning power compared to linear rules.

This extension emphasizes characteristics of a more fused neuro-fuzzy system called

CANFIS (Co-Active Neuro-Fuzzy Inference System).
ANALYSIS OF ADAPTIVE LEARNING CAPABILITY:

Two examples are used to clarify CANFIS’s learning capabilities. The overall
performance of the steepest descent method in contrast to the hybrid learning algorithm is
discussed. These problems provide us an insight into the power of CANFIS.

Convergence based on the Steepest Descent Method Alone:

The first example is simple: fitting an N-shaped letter that has corners, a pointed top left-
hand corner and a rounded right hand corner. The target letter is shown as a dashed line
in Figure 13.10(a):

To see how adaptive capability depends on architecture itself, five CANFIS models were
trained on the basis of conventional steepest descent algorithm with a fixed momentum of
0.8 and a small fixed learning rate. The models are : CANFIS with linear rules (i.e.,
ANFIS), CANFIS with sigmoidal rules, CANFIS with neural rules, each neural rule has
two hidden layers, CANFIS have identity functions, CANFIS has sigmoidal function at
the fuzzy association layer to generate final outputs. Figure 13.10 shows comparison of
results from CANFIS and simple backpropagation MLPs. CANFIS with two linear rules
is able to capture the peculiarity of N-shape in Figure 13.10 (b), fitting well both pointed
and round corners of the N-shape.

Due to the discontinuity of the left hand corner of N-shape, MLPs with small number of
hidden neurons (three or four), possessing sigmoidal neuron functions, do not evolve to
fit the pointed corner as shown in Figure 13.10(c). These results reinforce the strength of
CANFIS with generalized bell MFs in the convergence capability.
Interpretability Spectrum:

CANFIS with two rules can accomplish the task well, but CANFIS with three
rules fits the N-shape better, obtaining higher precision within a fixed amount of
computation time.

When we split three fuzzy rules in to two combinations – “Rule 1 and Rule 2” and
“Rule 1 and Rule 3” – neither output fits the N-shape, obviously, the three rules help each
other in improving overall performance. Adding more fuzzy rules result in lack of
interpretability or ill-defined fuzzy rules.

Evolution of Antecedents (MFs):

CANFIS based on the steepest descent method alone unexpectedly converged faster than
CANFIS with the hybrid learning procedure.

Moreover, CANFIS based on the hybrid leaning procedure did not fit the N-shape very
well, while CANFIS with the steepest descent method alone recognized the features of
the N-shape well. The hybrid learning procedure predominated where intuitively
positioned MFs do not need to evolve very much.

Initially LSE (Least Square estimation) may specialize rules consequents to a great
extent, which may prevent MFs from evolving. LSE can find certain rules consequent
values that have minimal errors with the current MF setup, but after updating
coefficients, it may end up losing its way to a better fitting level.

Evolution of Consequents (Rules) :

There must be some optimal combinations of “shapes” of MFs and “forms” of rules
consequents. The outputs of adapted rules consequents shown in Figure 13.15 end up
being different and far from the desired N-shape.

We have trained both antecedent and consequent parts simultaneously. Thus, each rules
output does not have to fit the desired output, N-shape; the final combined outputs fit it.
Evolving Partitions:

The data set consisted of 80 patterns in two-dimensional patter space as illustrated in

Figure 13.16. These patterns classified in to four categories: (ON,ON), (ON,OFF),
(OFF,ON) and (OFF,OFF). The outputs Oi (i=1,2), can be regarded as ON (0.9) and OFF
(0.1).
if (Oi >=0.8) Oi is classified to ON (0.9),

if (Oi <=0.2) Oi is classified to OFF (0.1),

otherwise Oi is “undecided”

When two CANFIS models with three MFs per input tested using nine linear rules per
output, and two simple back propagation MLPs with 14 hidden units. All models have
same number of adjustable parameters, 72. The “Stopped epoch” shoes when the models
classified all 80 patterns correctly according to the preceding criterion. Training was
performed using the steepest descent method alone up to the preset iteration limit, 10,000.

The better results from CANFIStrc and NNmod confirm that truncation filter function and
modified sigmoidal function help in learning given mappings.