LECTURE NOTES

ON
PRINCIPLES OF SOFT COMPUTING

PREPARED BY

DR. PRASHANTA KUMAR PATRA

COLLEGE OF ENGINEERING AND

TECHNOLOGY, BHUBANESWAR

1
Introduction
Basics of Soft Computing

What is Soft Computing?

• The idea of soft computing was initiated in 1981 when Lotfi A. Zadeh published his first
paper on soft data analysis “What is Soft Computing”, Soft Computing. Springer-Verlag
Germany/USA 1997.]

• Zadeh, defined Soft Computing into one multidisciplinary system as the fusion of the
fields of Fuzzy Logic, Neuro-Computing, Evolutionary and Genetic Computing, and
Probabilistic Computing.

• Soft Computing is the fusion of methodologies designed to model and enable solutions to
real world problems, which are not modeled or too difficult to model mathematically.

• The aim of Soft Computing is to exploit the tolerance for imprecision, uncertainty,
approximate reasoning, and partial truth in order to achieve close resemblance with
human like decision making.
• The Soft Computing – development history

SC = EC + NN + FL

Soft Evolutionary Neural Fuzzy

Computing Computing Network Logic

Zadeh Rechenberg McCulloch Zadeh

1981 1960 1943 1965

EC = GP + ES + EP + GA

Evolutionary Genetic Evolution Evolutionary Genetic

Computing Programming Strategies Programming Algorithms

Rechenberg Koza Rechenberg Fogel Holland

1960 1992 1965 1962 1970

2
Definitions of Soft Computing (SC)

Lotfi A. Zadeh, 1992 : “Soft Computing is an emerging approach to computing which parallel
the remarkable ability of the human mind to reason and learn in a environment of uncertainty
and imprecision”.

The Soft Computing consists of several computing paradigms mainly :

Fuzzy Systems, Neural Networks, and Genetic Algorithms.

• Fuzzy set : for knowledge representation via fuzzy If – Then rules.

• Neural Networks : for learning and adaptation

• Genetic Algorithms : for evolutionary computation

These methodologies form the core of SC.

Hybridization of these three creates a successful synergic effect; that is, hybridization creates a
situation where different entities cooperate advantageously for a final outcome.

Soft Computing is still growing and developing.

Hence, a clear definite agreement on what comprises Soft Computing has not yet been reached.
More new sciences are still merging into Soft Computing.

Goals of Soft Computing

Soft Computing is a new multidisciplinary field, to construct new generation of
Artificial Intelligence, known as Computational Intelligence.

• The main goal of Soft Computing is to develop intelligent machines to provide solutions
to real world problems, which are not modeled, or too difficult to model mathematically.

• Its aim is to exploit the tolerance for Approximation, Uncertainty, Imprecision, and
Partial Truth in order to achieve close resemblance with human like decision making.

Approximation : here the model features are similar to the real ones, but not the same.

3
 Uncertainty : here we are not sure that the features of the model are the same as that of the
entity (belief).

Imprecision : here the model features (quantities) are not the same as that of the real ones,
but close to them.

Importance of Soft Computing

Soft computing differs from hard (conventional) computing. Unlike

hard computing, the soft computing is tolerant of imprecision, uncertainty,

partial truth, and approximation. The guiding principle of soft computing is to

exploit these tolerance to achieve tractability, robustness and low solution cost. In
effect, the role model for soft computing is the human mind.

The four fields that constitute Soft Computing (SC) are : Fuzzy Computing (FC),
Evolutionary Computing (EC), Neural computing (NC), and Probabilistic
Computing (PC), with the latter subsuming belief networks, chaos theory and parts
of learning theory.

Soft computing is not a concoction, mixture, or combination, rather,

Soft computing is a partnership in which each of the partners contributes

a distinct methodology for addressing problems in its domain. In principal

the constituent methodologies in Soft computing are complementary rather

than competitive.

Soft computing may be viewed as a foundation component for the emerging

field of Conceptual Intelligence.

4
Fuzzy Computing

In the real world there exists much fuzzy knowledge, that is, knowledge which
is vague, imprecise, uncertain, ambiguous, inexact, or probabilistic in nature.

Human can use such information because the human thinking and reasoning
frequently involve fuzzy information, possibly originating from inherently
inexact human concepts and matching of similar rather then identical
experiences.

The computing systems, based upon classical set theory and two-valued logic,
can not answer to some questions, as human does, because they do not have
completely true answers.

We want, the computing systems should not only give human like answers but
also describe their reality levels. These levels need to be calculated using
imprecision and the uncertainty of facts and rules that were applied.

Fuzzy Sets

Introduced by Lotfi Zadeh in 1965, the fuzzy set theory is an extension of

classical set theory where elements have degrees of membership.

• Classical Set Theory

– Sets are defined by a simple statement describing whether an

element having a certain property belongs to a particular set.

– When set A is contained in an universal space X,

5
 then we can state explicitly whether each element x of space X "is or
is not" an element of A.

– Set A is well described by a function called characteristic function A.

This function, defined on the universal space X, assumes :

value 1 for those elements x that belong to set A, and

value 0 for those elements x that do not belong to set A.

The notations used to express these mathematically are

Α : Χ → [0, 1]

A(x) = 1 , x is a member of A Eq.(1)

A(x) = 0 , x is not a member of A

Alternatively, the set A can be represented for all elements x ∈ X

by its characteristic function A (x) defined as

1 if x∈ X

A (x) = Eq.(2)
0 otherwise

– Thus, in classical set theory A (x) has only the values 0 ('false') and 1
('true''). Such sets are called crisp sets.

6
• Crisp and Non-crisp Set

– As said before, in classical set theory, the characteristic function A(x)

of Eq.(2) has only values 0 ('false') and 1 ('true'').

Such sets are crisp sets.

– For Non-crisp sets the characteristic function A(x) can be defined.

ƒ The characteristic function A(x) of Eq. (2) for the crisp set is
generalized for the Non-crisp sets.

ƒ This generalized characteristic function A(x) of Eq.(2) is called

membership function.

Such Non-crisp sets are called Fuzzy Sets.

– Crisp set theory is not capable of representing descriptions and

classifications in many cases; In fact, Crisp set does not provide
adequate representation for most cases.

– The proposition of Fuzzy Sets are motivated by the need to capture and
represent real world data with uncertainty due to imprecise
measurement.

– The uncertainties are also caused by vagueness in the language.

7
 • Example 1 : Heap Paradox

This example represents a situation where vagueness and uncertainty are inevitable.

- If we remove one grain from a heap of grains, we will still have a heap.
- However, if we keep removing one-by-one grain from a heap of grains, there will be
a time when we do not have a heap anymore.
- The question is, at what time does the heap turn into a countable collection of grains
that do not form a heap? There is no one correct answer to this question.

• Example 2 : Classify Students for a basketball team

This example explains the grade of truth value.
- tall students qualify and not tall students do not qualify
- if students 1.8 m tall are to be qualified, then

should we exclude a student who is 1/10" less? or should we

exclude a student who is 1" shorter?
■
Non-Crisp Representation to represent the notion of a tall person.

A student of height 1.79m would belong to both tall and not tall sets with a particular degree of
[Link] the height increases the membership grade within the tall set would increase whilst
the membership grade within the not-tall set would decrease.

8
• Capturing Uncertainty

Instead of avoiding or ignoring uncertainty, Lotfi Zadeh introduced Fuzzy

Set theory that captures uncertainty.

■ In the case of Crisp Sets the members of a set are :

either out of the set, with membership of degree " 0
", or in the set, with membership of degree " 1 ",

Therefore, Crisp Sets ⊆ Fuzzy Sets In other words, Crisp Sets are
Special cases of Fuzzy Sets.

9
Example 2: Set of SMALL ( as non-crisp set) Example 1: Set of prime
numbers ( a crisp set)

If we consider space X consisting of natural numbers ≤ 12

ie X = {1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12}

Then, the set of prime numbers could be described as follows.

PRIME = {x contained in X | x is a prime number} = {2, 3, 5, 6, 7, 11}

A Set X that consists of SMALL cannot be described;

for example 1 is a member of SMALL and 12 is not a member of SMALL.

Set A, as SMALL, has un-sharp boundaries, can be characterized by a

function that assigns a real number from the closed interval from 0 to
1 to each element x in the set X.

10
• Definition of Fuzzy Set

A fuzzy set A defined in the universal space X is a function defined in X

which assumes values in the range [0, 1].

A fuzzy set A is written as a set of pairs {x, A(x)} as

A = {{x , A(x)}} , x in the set X

where x is an element of the universal space X, and

A(x) is the value of the function A for this element.

The value A(x) is the membership grade of the element x in a fuzzy set A.

Example : Set SMALL in set X consisting of natural numbers to 12.

Assume:
SMALL(1) = 1, SMALL(2) = 1, SMALL(3) = 0.9, SMALL(4) = 0.6,

11
 SMALL(5) = 0.4, SMALL(6) = 0.3, SMALL(7) = 0.2, SMALL(8) = 0.1,
SMALL(u) = 0 for u >= 9.

Then, following the notations described in the definition above :

Set SMALL = {{1, 1 }, {2, 1 }, {3, 0.9}, {4, 0.6}, {5, 0.4}, {6, 0.3}, {7, 0.2},

{8, 0.1}, {9, 0 }, {10, 0 }, {11, 0}, {12, 0}}

Note that a fuzzy set can be defined precisely by associating with each x ,
its grade of membership in SMALL.

• Definition of Universal Space

Originally the universal space for fuzzy sets in fuzzy logic was defined only
on the integers. Now, the universal space for fuzzy sets and fuzzy
relations is defined with three numbers. The first two numbers specify the
start and end of the universal space, and the third argument specifies the
increment between elements. This gives the user more flexibility in
choosing the universal space.

Example : The fuzzy set of numbers, defined in the universal space

X = { xi } = {1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12} is presented as

SetOption [FuzzySet, UniversalSpace → {1, 12, 1}]

• Graphic Interpretation of Fuzzy Sets SMALL

The fuzzy set SMALL of small numbers, defined in the universal space

12
X = { xi } = {1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12} is presented as

SetOption [FuzzySet, UniversalSpace → {1, 12, 1}]

The Set SMALL in set X is :

SMALL = FuzzySet {{1, 1 }, {2, 1 }, {3, 0.9}, {4, 0.6}, {5, 0.4}, {6, 0.3},

{7, 0.2}, {8, 0.1}, {9, 0 }, {10, 0 }, {11, 0}, {12, 0}}

Therefore SetSmall is represented as

SetSmall = FuzzySet [{{1,1},{2,1}, {3,0.9}, {4,0.6}, {5,0.4},{6,0.3}, {7,0.2},

{8, 0.1}, {9, 0}, {10, 0}, {11, 0}, {12, 0}} , UniversalSpace → {1, 12, 1}]

FuzzyPlot [ SMALL, AxesLable → {"X", "SMALL"}]

13
 • Graphic Interpretation of Fuzzy Sets PRIME Numbers

The fuzzy set PRIME numbers, defined in the universal space

X = { xi } = {1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12} is presented as

SetOption [FuzzySet, UniversalSpace → {1, 12, 1}]

The Set PRIME in set X is :

PRIME = FuzzySet {{1, 0}, {2, 1}, {3, 1}, {4, 0}, {5, 1}, {6, 0}, {7, 1}, {8, 0}, {9, 0}, {10, 0}, {11, 1},

{12, 0}}

Therefore SetPrime is represented as

SetPrime = FuzzySet [{{1,0},{2,1}, {3,1}, {4,0}, {5,1},{6,0}, {7,1},

{8, 0}, {9, 0}, {10, 0}, {11, 1}, {12, 0}} , UniversalSpace → {1, 12, 1}]

FuzzyPlot [ PRIME, AxesLable → {"X", "PRIME"}]

14
• Graphic Interpretation of Fuzzy Sets UNIVERSALSPACE

In any application of sets or fuzzy sets theory, all sets are subsets of

a fixed set called universal space or universe of discourse denoted by X.

Universal space X as a fuzzy set is a function equal to 1 for all elements.

The fuzzy set UNIVERSALSPACE numbers, defined in the universal

space X = { xi } = {1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12} is presented as

SetOption [FuzzySet, UniversalSpace

→ {1, 12, 1}]

The Set UNIVERSALSPACE in set X is :

UNIVERSALSPACE = FuzzySet {{1, 1}, {2, 1}, {3, 1}, {4, 1}, {5, 1}, {6, 1},

{7, 1}, {8, 1}, {9, 1}, {10, 1}, {11, 1}, {12, 1}}

Therefore SetUniversal is represented as

SetUniversal = FuzzySet [{{1,1},{2,1}, {3,1}, {4,1}, {5,1},{6,1}, {7,1},

{10, 1}, {11, 1}, {12, 1}} , UniversalSpace

{8, 1}, {9, 1}, → {1, 12, 1}]

FuzzyPlot [ UNIVERSALSPACE, AxesLable → {"X", " UNIVERSAL SPACE "}]

15
Finite and Infinite Universal Space

Universal sets can be finite or infinite.

Any universal set is finite if it consists of a specific number of different

elements, that is, if in counting the different elements of the set, the
counting can come to an end, else the set is infinite.

Examples:

1. Let N be the universal space of the days of the week.

N = {Mo, Tu, We, Th, Fr, Sa, Su}. N is finite.

2. Let M = {1, 3, 5, 7, 9, ...}. M is infinite.

3. Let L = {u | u is a lake in a city }. L is finite.

(Although it may be difficult to count the number of lakes in a

city, but L is still a finite universal set.)

16
• Graphic Interpretation of Fuzzy Sets EMPTY

An empty set is a set that contains only elements with a grade of

membership equal to 0.

Example: Let EMPTY be a set of people, in Minnesota, older than

120. The Empty set is also called the Null set.

The fuzzy set EMPTY , defined in the universal space

X = { xi } = {1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12} is presented as

SetOption [FuzzySet, UniversalSpace → {1, 12, 1}]

The Set EMPTY in set X is :

EMPTY = FuzzySet {{1, 0}, {2, 0}, {3, 0}, {4, 0}, {5, 0}, {6, 0}, {7, 0}, {8,

0}, {9, 0}, {10, 0}, {11, 0}, {12, 0}}

Therefore SetEmpty is represented as

SetEmpty = FuzzySet [{{1,0},{2,0}, {3,0}, {4,0}, {5,0},{6,0}, {7,0},

{8, 0}, {9, 0}, {10, 0}, {11, 0}, {12, 0}} , UniversalSpace → {1, 12, 1}]

FuzzyPlot [ EMPTY, AxesLable → {"X", " UNIVERSAL SPACE "}]

17
Fuzzy Operations

A fuzzy set operations are the operations on fuzzy sets. The fuzzy set
operations are generalization of crisp set operations. Zadeh [1965]
formulated the fuzzy set theory in the terms of standard operations:
Complement, Union, Intersection, and Difference.

In this section, the graphical interpretation of the following standard fuzzy

set terms and the Fuzzy Logic operations are illustrated:

Inclusion :
FuzzyInclude [VERYSMALL, SMALL]

Equality
: FuzzyEQUALITY [SMALL, STILLSMALL]

Complement
: FuzzyNOTSMALL = FuzzyCompliment [Small]

Union :
∪
FuzzyUNION = [SMALL MEDIUM]

Intersection :
∩
FUZZYINTERSECTON = [SMALL MEDIUM]

18
• Inclusion

Let A and B be fuzzy sets defined in the same universal space X.

The fuzzy set A is included in the fuzzy set B if and only if for every x in

the set X we have A(x) ≤ B(x)

Example :

The fuzzy set UNIVERSALSPACE numbers, defined in the universal

space X = { xi } = {1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12} is presented as

SetOption [FuzzySet, UniversalSpace → {1, 12, 1}]

The fuzzy set B SMALL

The Set SMALL in set X is :

SMALL = FuzzySet {{1, 1 }, {2, 1 }, {3, 0.9}, {4, 0.6}, {5, 0.4}, {6, 0.3},

{7, 0.2}, {8, 0.1}, {9, 0 }, {10, 0 }, {11, 0}, {12, 0}}

Therefore SetSmall is represented as

SetSmall = FuzzySet [{{1,1},{2,1}, {3,0.9}, {4,0.6}, {5,0.4},{6,0.3}, {7,0.2},

{8, 0.1}, {9, 0}, {10, 0}, {11, 0}, {12, 0}} , UniversalSpace → {1, 12, 1}]

The fuzzy set A VERYSMALL

The Set VERYSMALL in set X is :

VERYSMALL = FuzzySet {{1, 1

{6, 0.1}, {7, 0 },

}, {2, 0.8 }, {3, 0.7}, {4, 0.4}, {5, 0.2},

{8, 0 }, {9, 0 }, {10, 0 }, {11, 0}, {12, 0}}

Therefore SetVerySmall is represented as

19
 SetVerySmall = FuzzySet [{{1,1},{2,0.8}, {3,0.7}, {4,0.4}, {5,0.2},{6,0.1},
{7,0}, {8, 0}, {9, 0}, {10, 0}, {11, 0}, {12, 0}} , UniversalSpace → {1, 12, 1}]

The Fuzzy Operation :

nclusion

Include

[VERYSMALL,
SMALL]

20
• Comparability

Two fuzzy sets A and B are comparable

if the condition A ⊂ B or B ⊂ A holds, ie,

if one of the fuzzy sets is a subset of the other set, they are comparable.

Two fuzzy sets A and B are incomparable

if the condition A ⊄ B or B ⊄ A holds.

Example 1:

Let A = {{a, 1}, {b, 1}, {c, 0}} and

B = {{a, 1}, {b, 1}, {c, 1}}.

Then A is comparable to B, since A is a subset of B.

Example 2 :

Let C = {{a, 1}, {b, 1}, {c, 0.5}} and

D = {{a, 1}, {b, 0.9}, {c, 0.6}}.

Then C and D are not comparable since

C is not a subset of D and

D is not a subset of C.

Property Related to Inclusion :

for all x in the set X, if A(x) ⊂ B(x) ⊂ C(x), then accordingly A ⊂ C.

21
• Equality

Let A and B
Then A and B if be fuzzy sets defined in the same space X.
and only if
are equal, which is denoted X = Y

for all x in the set X, A(x) = B(x).

Example.

The fuzzy set B SMALL

SMALL = FuzzySet {{1, 1 }, {2, 1 }, {3, 0.9}, {4, 0.6}, {5, 0.4}, {6, 0.3},

{7, 0.2}, {8, 0.1}, {9, 0 }, {10, 0 }, {11, 0}, {12, 0}}

The fuzzy set A STILLSMALL

STILLSMALL = FuzzySet {{1, 1 }, {2, 1 }, {3, 0.9}, {4, 0.6}, {5, 0.4},

{6, 0.3}, {7, 0.2}, {8, 0.1}, {9, 0 }, {10, 0 }, {11, 0}, {12, 0}}

The Fuzzy Operation : Equality

Equality [SMALL, STILLSMALL]

the set X, then we say that A is not equal to B.

22
• Complement

Let A be a fuzzy set defined in the space X.

Then the fuzzy set B is a complement of the fuzzy set A, if and only if,

for all x in the set X, B(x) = 1 - A(x).

The complement of the fuzzy set A is often denoted by A' or Ac or A

Fuzzy Complement : Ac(x) = 1 – A(x)

Example 1.

The fuzzy set A SMALL

SMALL = FuzzySet {{1, 1 }, {2, 1 }, {3, 0.9}, {4, 0.6}, {5, 0.4}, {6, 0.3},

{7, 0.2}, {8, 0.1}, {9, 0 }, {10, 0 }, {11, 0}, {12, 0}}

The fuzzy set Ac NOTSMALL

NOTSMALL = FuzzySet {{1, 0 }, {2, 0 }, {3, 0.1}, {4, 0.4}, {5, 0.6}, {6, 0.7},

{7, 0.8}, {8, 0.9}, {9, 1 }, {10, 1 }, {11, 1}, {12, 1}}

The Fuzzy Operation : Compliment

NOTSMALL = Compliment [SMALL]

23
 SC – Fuzzy Computing

Example 2.

The empty set Φ and the universal set X, as fuzzy sets, are

complements of one another.

Φ'= X' =
X Φ
,

The fuzzy set B EMPTY

Empty = FuzzySet {{1, 0 }, {2, 0 }, {3, 0}, {4, 0}, {5, 0}, {6, 0},

{7, 0}, {8, 0}, {9, 0 }, {10, 0 }, {11, 0}, {12, 0}}

The fuzzy set A UNIVERSAL

Universal = FuzzySet {{1, 1 }, {2, 1 }, {3, 1}, {4, 1}, {5, 1}, {6, 1},

{7, 1}, {8, 1}, {9, 1 }, {10, 1 }, {11, 1}, {12, 1}}

The fuzzy operation : Compliment

EMPTY = Compliment [UNIVERSALSPACE]

24
• Union

Let A and B be fuzzy sets defined in the space X.

The union is defined as the smallest fuzzy set that contains both A and
B. The union of A and B is denoted by A ∪ B.

The following relation must be satisfied for the union operation

: for all x in the set X, (A ∪ B)(x) = Max (A(x), B(x)).

Fuzzy Union : (A ∪ B)(x) = max [A(x), B(x)] for all x ∈ X

Example 1 : Union of Fuzzy A and B

A(x) = 0.6 and B(x) = 0.4 ∴ (A ∪ B)(x) = max [0.6, 0.4] = 0.6

Example 2 : Union of SMALL and MEDIUM

The fuzzy set A SMALL

SMALL = FuzzySet {{1, 1 }, {2, 1 }, {3, 0.9}, {4, 0.6}, {5, 0.4}, {6, 0.3},

{7, 0.2}, {8, 0.1}, {9, 0 }, {10, 0 }, {11, 0}, {12, 0}}

The fuzzy set B MEDIUM

MEDIUM = FuzzySet {{1, 0 }, {2, 0 }, {3, 0}, {4, 0.2}, {5, 0.5}, {6, 0.8},

{7, 1}, {8, 1}, {9, 0.7 }, {10, 0.4 }, {11, 0.1}, {12, 0}}

The fuzzy operation : Union

FUZZYUNION = [SMALL
∪ MEDIUM]

SetSmallUNIONMedium = FuzzySet [{{1,1},{2,1}, {3,0.9}, {4,0.6}, {5,0.5},

{6,0.8}, {7,1}, {8, 1}, {9, 0.7}, {10, 0.4}, {11, 0.1}, {12, 0}} , UniversalSpace →
{1, 12, 1}]

The notion of the union is closely related to that of the connective "or".

Let A is a class of "Young" men, B is a class of "Bald" men.

25
 If "David is Young" or "David is Bald," then David is associated with the
union of A and B. Implies David is a member of A ∪ B.

• Properties Related to Union

The properties related to union are :

Identity, Idempotence, Commutativity and Associativity.

■ Identity:

A∪
Φ =A

input
= Equality [SMALL ∪ EMPTY , SMALL]

output = True

A
∪ X=X

input
= Equality [SMALL ∪ UnivrsalSpace , UnivrsalSpace]

output = True

■ Idempotence :
A∪A=A

input = Equality [SMALL ∪ SMALL , SMALL]

output = True

■ Commutativity :

A ∪ B =B ∪ A

26
 input = Equality [SMALL ∪ MEDIUM, MEDIUM ∪ SMALL]

output = True

■ Associativity:

A
∪ (B
C) = B) C
∪
(A∪ ∪

input = Equality [SMALL

∪ (MEDIUM BIG) , (SMALL MEDIUM)
∪
BIG]
∪ ∪

output = True

SMALL = FuzzySet {{1, 1 },

{7, 0.2}, {2, 1 }, {3, 0.9}, {4, 0.6}, {5, 0.4}, {6, 0.3}, {8,
0.1}, {9, 0.7 }, {10, 0.4 }, {11, 0}, {12, 0}}

MEDIUM = FuzzySet {{1, 0 }, {2, 0 }, {3, 0}, {4, 0.2}, {5, 0.5}, {6, 0.8},

{7, 1}, {8, 1}, {9, 0 }, {10, 0 }, {11, 0.1}, {12, 0}}

BIG
= FuzzySet [{{1,0}, {2,0}, {3,0}, {4,0}, {5,0}, {6,0.1}, {7,0.2},
{8,0.4}, {9,0.6}, {10,0.8}, {11,1}, {12,1}}]

Medium ∪ BIG = FuzzySet [{1,0},{2,0}, {3,0}, {4,0.2}, {5,0.5}, {6,0.8},

{7,1}, {8, 1}, {9, 0.6}, {10, 0.8}, {11, 1}, {12, 1}]

Small ∪ Medium = FuzzySet [{1,1},{2,1}, {3,0.9}, {4,0.6}, {5,0.5},

{6,0.8}, {7,1}, {8, 1}, {9, 0.7}, {10, 0.4}, {11, 0.1}, {12, 0}]

27

27
 SC – Fuzzy Computing

• Intersection

Let A and B be fuzzy sets defined in the space X.

The intersection is defined as the greatest fuzzy set included both A and
B. The intersection of A and B is denoted by A ∩ B.

The following relation must be satisfied for the union operation :

for all x in the set X, (A ∩ B)(x) = Min (A(x), B(x)).

Fuzzy Intersection : (A ∩ B)(x) = min [A(x), B(x)]

∈
for all x X

Example 1 : Intersection of Fuzzy A and B

A(x) = 0.6 and B(x) = 0.4 ∴ (A ∩ B)(x) = min [0.6, 0.4] = 0.4

Example 2 : Union of SMALL and MEDIUM

The fuzzy set A SMALL

SMALL = FuzzySet {{1, 1 }, {2, 1 }, {3, 0.9}, {4, 0.6}, {5, 0.4}, {6, 0.3},

{7, 0.2}, {8, 0.1}, {9, 0 }, {10, 0 }, {11, 0}, {12, 0}}

The fuzzy set B MEDIUM

MEDIUM = FuzzySet {{1, 0 }, {2, 0 }, {3, 0}, {4, 0.2}, {5, 0.5}, {6, 0.8},

{7, 1}, {8, 1}, {9, 0.7 }, {10, 0.4 }, {11, 0.1}, {12, 0}}

The fuzzy operation : Intersection FUZZYINTERSECTION = min

[SMALL ∩ MEDIUM] SetSmallINTERSECTIONMedium = FuzzySet [{{1,0},
{2,0}, {3,0}, {4,0.2},

{5,0.4}, {6,0.3}, {7,0.2}, {8, 0.1}, {9, 0},{10, 0}, {11, 0}, {12, 0}} , UniversalSpace →
{1, 12, 1}]

28
Neural Computing

Neural Computers mimic certain processing capabilities of the human brain.

- Neural Computing is an information processing paradigm, inspired by

biological system, composed of a large number of highly interconnected
processing elements (neurons) working in unison to solve specific problems.

- A neural net is an artificial representation of the human brain that tries to

simulate its learning process. The term "artificial" means that neural nets
are implemented in computer programs that are able to handle the large
number of necessary calculations during the learning process.

- Artificial Neural Networks (ANNs), like people, learn by example.

- An ANN is configured for a specific application, such as pattern recognition

or data classification, through a learning process.

- Learning in biological systems involves adjustments to the synaptic

connections that exist between the neurons. This is true for ANNs as well.

29
Biological Model:

The human brain consists of a large number (more than a billion) of neural
cells that process information. Each cell works like a simple processor. The
massive interaction between all cells and their parallel processing, makes
the brain's abilities possible. The
structure of neuron is shown below.

Dendrites are the branching fibers

extending from the cell body or soma.
Soma or cell body of a neuron
contains the nucleus and other
structures, support chemical processing
and production of neurotransmitters.

Axon is a singular fiber carries information

away from the soma to the synaptic sites
of other neurons (dendrites and somas),
muscles, or glands.
Fig. Structure of Neuron

Axon hillock is the site of summation

for incoming information. At any
moment, the collective influence of all
neurons, that conduct as impulses to a
given neuron, will determine whether or
not an action potential will be initiated at

the axon hillock and propagated along the axon.

Myelin Sheath consists of fat-containing cells that insulate the axon from electrical
activity. This insulation acts to increase the rate of transmission of signals. A gap
exists between each myelin sheath cell along the axon. Since fat inhibits the
propagation of electricity, the signals jump from one gap to the next.

30
Nodes of Ranvier are the gaps (about 1 m) between myelin sheath cells long
axons. Since fat serves as a good insulator, the myelin sheaths speed the rate of
transmission of an electrical impulse along the axon.

Synapse is the point of connection between two neurons or a neuron and a muscle or
a gland. Electrochemical communication between neurons takes place at these
junctions.

Terminal Buttons of a neuron are the small knobs at the end of an axon that
release chemicals called neurotransmitters.

31
• Information flow in a Neural Cell

The input /output and the propagation of information are shown below.

Fig. Structure of a neural cell in the human brain

■ Dendrites receive activation from other neurons.

■ Soma processes the incoming activations and converts them into

output activations.

■ Axons act as transmission lines to send activation to other neurons.

■ Synapses the junctions allow signal transmission between the axons

and dendrites.

■ The process of transmission is by diffusion of chemicals called neuro-

transmitters.

McCulloch-Pitts introduced a simplified model of this real neurons.

32
Artificial Neuron

• The McCulloch-Pitts Neuron

This is a simplified model of real neurons, known as a Threshold Logic Unit.

■ A set of synapses (ie connections) brings in activations from other

neurons.

■ A processing unit sums the inputs, and then applies a non-linear

activation function (i.e. transfer / threshold function).

■ An output line transmits the result to other neurons.

In other words, the input to a neuron arrives in the form of signals.

The signals build up in the cell. Finally the cell fires (discharges)

through the output. The cell can start building up signals again.

33
• Functions :

The function y = f(x) describes a relationship, an input-output mapping,

from x to y.

■ Threshold or Sign function sgn(x) : defined as

■ Threshold or Sign function sigmoid (x) : defined as a smoothed

(differentiable) form of the threshold function

34
 • McCulloch-Pitts (M-P) Neuron Equation

Fig below is the same previously shown simplified model of a real

neuron, as a threshold Logic Unit.

Note : The McCulloch-Pitts neuron is an extremely simplified model of real

biological neurons. Some of its missing features include: non-binary input
and output, non-linear summation, smooth thresholding, stochastic (non-
deterministic), and temporal information processing.

34

35
• Basic Elements of an Artificial Neuron

It consists of three basic components - weights, thresholds, and a single

activation function.

Weighting Factors

The values W1 , W2 , . . . Wn are weights to determine the strength of input

T
row vector X = [x1 , x2 , . . . , xn] . Each input is multiplied by the
associated weight of the neuron connection XT W. The +ve weight excites
and the -ve weight inhibits the node output.

Threshold

The node’s internal threshold Φ is the magnitude offset. It affects the

activation of the node output y as:

y= Σn Xi W i - Φ k

i=1

36
Activation Function

An activation function performs a mathematical operation on the signal

output. The most common activation functions are, Linear

Function, Threshold Function, Piecewise Linear Function, Sigmoidal (S

shaped) function, Tangent hyperbolic function and are chose depending
upon the type of problem to be solved by the network.

37
• Example :

With a binary activation function, the outputs of the neuron is:

y (threshold) = 1

38
• Single and Multi - Layer Perceptrons

A perceptron is a name for simulated neuron in the computer program.

The usually way to represent a neuron model is described below.

The neurons are shown as circles in the diagram. It has several inputs and
a single output. The neurons have gone under various names.

- Each individual cell is called either a node or a perceptron.

- A neural network consisting of a layer of nodes or perceptrons between

the input and the output is called a single layer perceptron.

- A network consisting of several layers of single layer perceptron

stacked on top of other, between input and output , is called a

multi-layer perceptron
Output

Output Output

Input InputInput

Fig Single and Multi - Layer Perceptrons

Multi-layer perceptrons are more powerful than single-layer perceptrons.

39
• Perceptron

Any number of McCulloch-Pitts neurons can be connected together in any

way.

Definition : An arrangement of one input layer of McCulloch-Pitts neurons,

that is feeding forward to one output layer of McCulloch-Pitts neurons is
known as a Perceptron.

A Perceptron is a powerful computational device.

40
Genetic Algorithms
Genetic Algorithms (GAs) were invented by John Holland in early 1970's to mimic
some of the processes observed in natural evolution.

Later in 1992 John Koza used GAs to evolve programs to perform certain tasks.
He called his method "Genetic Programming" (GP).

GAs simulate natural evolution, a combination of selection, recombination and

mutation to evolve a solution to a problem.

GAs simulate the survival of the fittest , among individuals over consecutive
generation for solving a problem. Each generation consists of a population of
character strings that are analogous to the chromosome in our DNA
(Deoxyribonucleic acid). DNA contains the genetic instructions used in the
development and functioning of all known living organisms.

What are Genetic Algorithms

■ Genetic Algorithms (GAs) are adaptive heuristic search algorithm

based on the evolutionary ideas of natural selection and
genetics.

■ Genetic algorithms (GAs) are a part of evolutionary computing, a

rapidly growing area of artificial intelligence. GAs are inspired by
Darwin's theory about evolution - "survival of the fittest".

■ GAs represent an intelligent exploitation of a random search used to

solve optimization problems.
■ GAs, although randomized, exploit historical information to direct the

search into the region of better performance within the search space.
■ In nature, competition among individuals for scanty resources results
in the fittest individuals dominating over the weaker ones.

41
• Why Genetic Algorithms

"Genetic Algorithms are good at taking large, potentially huge search

spaces and navigating them, looking for optimal combinations of things,
solutions you might not otherwise find in a lifetime.” - Salvatore Mangano
Computer Design, May 1995.

- GA is better than conventional AI, in that it is more robust.

- Unlike older AI systems, GAs do not break easily even if the

inputs changed slightly, or in the presence of reasonable noise.

- In searching a large state-space, multi-modal state-space, or n-

dimensional surface, a GA may offer significant benefits over more
typical search of optimization techniques, like - linear programming,
heuristic, depth-first, breath-first.

• Mechanics of Biological Evolution

Genetic Algorithms are a way of solving problems by mimicking processes

the nature uses - Selection, Crosses over, Mutation and Accepting to
evolve a solution to a problem.

■ Every organism has a set of rules, describing how that organism is

built, and encoded in the genes of an organism.

■ The genes are connected together into long strings called

chromosomes.

42
■ Each gene represents a specific trait (feature) of the organism and has
several different settings, e.g. setting for a hair color gene may be
black or brown.

■ The genes and their settings are referred as an organism's genotype.

■ When two organisms mate they share their genes. The resultant
offspring may end up having half the genes from one parent and half
from the other parent. This process is called crossover
(recombination).

■ The newly created offspring can then be mutated. A gene may be

mutated and expressed in the organism as a completely new trait.
Mutation means, that the elements of DNA are a bit changed. This
change is mainly caused by errors in copying genes from parents.

■ The fitness of an organism is measured by success of the organism in

its life.

43
Artificial Evolution and Search Optimization

The problem of finding solutions to problems is itself a problem with no

general solution. Solving problems usually mean looking for solutions,
which will be the best among others.

■ In engineering and mathematics finding the solution to a problem is

often thought as a process of optimization.

■ Here the process is : first formulate the problems as mathematical

models expressed in terms of functions; then to find a solution,

discover the parameters that optimize the model or the function
components that provide optimal system performance.

The well-established search / optimization techniques are usually classified

in to three broad categories : Enumerative, Calculus-based, and Guided
random search techniques. A taxonomy of Evolution & Search
Optimization classes is illustrated in the next slide.

44
• Taxonomy of Evolution & Search Optimization Classes

■ Enumerative Methods

These are the traditional search and control strategies. They search for
a solution in a problem space within the domain of artificial
intelligence. There are many control structures for search. The depth-
first search and breadth-first search are the two most

common search strategies. Here the search goes through every

point related to the function's domain space (finite or discretized),

one point at a time. They are very simple to implement but

usually require significant computation. These techniques are

not suitable for applications with large domain spaces.

In the field of AI, enumerative methods are subdivide into two

45
categories : uninformed and informed methods.

◊ Uninformed or blind methods : Such as mini-max algorithm

searches all points in the space in a predefined order; this is

used in game playing;

◊ Informed methods : Such as Alpha-Beta and A*, does more

sophisticated search using domain specific knowledge in the form
of a cost function or heuristic in order to reduce the cost of the
search.

46
■ Calculus based techniques

Here a set of necessary and sufficient conditions to be satisfied by the

solutions of an optimization problem. They subdivide into direct and
indirect methods.

◊ Direct or Numerical methods, such as Newton or Fibonacci,

seek extremes by "hopping" around the search space and

assessing the gradient of the new point, which guides

the search. This is simply the notion of "hill climbing", which

finds the best local point by climbing the steepest permissible
gradient. These techniques can be used only on a restricted set
of "well behaved" functions.

◊ Indirect methods search for local extremes by solving the usually

non-linear set of equations resulting from setting the

gradient of the objective function to zero. The search for

possible solutions (function peaks) starts by restricting itself to
points with zero slope in all directions.

47
■ Guided Random Search techniques

These are based on enumerative techniques but they use additional

information to guide the search. Two major subclasses

are simulated annealing and evolutionary algorithms. Both are

evolutionary processes.

◊ Simulated annealing uses a thermodynamic evolution process

to search minimum energy states.

◊ Evolutionary algorithms (EAs) use natural selection principles.

This form of search evolves throughout generations, improving the
features of potential solutions by means of biological inspired
operations. Genetic Algorithms (GAs) are a good example of this
technique.

Our main concern is, how does an Evolutionary algorithm :

- implement and carry out search,

- describes the process of search,

- what are the elements required to carry out search, and

- what are the different search strategies.

48
Evolutionary Algorithms (EAs)

Evolutionary algorithms are search methods. They take inspirations

from natural selection and survival of the fittest in the biological

world, and therefore differ from traditional search optimization

techniques. EAs involve search from a "population" of solutions,

and not from a single point. Each iteration of an EA involves a

competitive selection that weeds out poor solutions. The solutions

with high "fitness" are "recombined" with other solutions by

swapping parts of a solution with another. Solutions are also "mutated" by

making a small change to a single element of the

solution. Recombination and mutation are used to generate new

solutions that are biased towards regions of the space for which

good solutions have already been seen.

Evolutionary search algorithm (issues related to search) :

In the search space, each point represent one feasible solution.

Each feasible solution is marked by its value or fitness for the problem.

The issues related to search are :

- Search for a solution point, means finding which one point (or more)
among many feasible solution points in the search space is the solution.
This requires looking for some extremes, minimum or maximum.

- Search space can be whole known, but usually we know only a few
points and we are generating other points as the process of finding
solution continues.

49
 - Search can be very complicated. One does not know where to look

for the solution and where to start.

- What we find is some suitable solution, not necessarily the best solution.
The solution found is often considered as a good solution, because it is not
often possible to prove what is the real optimum solution.
Associative Memory

An associative memory is a content-addressable structure that maps a set of

input patterns to a set of output patterns. The associative memory are of two
types : auto-associative and hetero-associative.

ƒ An auto-associative memory retrieves a previously stored pattern that most

closely resembles the current pattern.

ƒ In a hetero-associative memory, the retrieved pattern is, in general, different

from the input pattern not only in content but possibly also in type and
format.

• Example : Associative Memory

The figure below shows a memory containing names of several people.

If the given memory is content-addressable,

Then using the erroneous string "Crhistpher Columbos" as key is

sufficient to retrieve the correct name "Christopher Colombus."

In this sense, this type of memory is robust and fault-tolerant, because

this type of memory exhibits some form of error-correction capability.

50
Description of Associative Memory

An associative memory is a content-addressable structure that maps

specific input representations to specific output representations.

■ A content-addressable memory is a type of memory that allows, the

recall of data based on the degree of similarity between the input
pattern and the patterns stored in memory.

■ It refers to a memory organization in which the memory is accessed by

its content and not or opposed to an explicit address in the traditional
computer memory system.

■
This type of memory allows the recall of information based on partial
knowledge of its contents.

51
■ It is a system that “associates” two patterns (X, Y) such that

when one is encountered, the other can be recalled.

- Let X and Y be two vectors of length m and n respectively.

- Typically, XÎ {-1, +1}m, Y Î {-1, +1}n

- The components of the vectors can be thought of as pixels

when the two patterns are considered as bitmap images.

■ There are two classes of associative memory:

- auto-associative and

- hetero-associative.

An auto-associative memory is used to retrieve a previously stored

pattern that most closely resembles the current pattern.

In a hetero-associative memory, the retrieved pattern is, in general,

different from the input pattern not only in content but possibly also
different in type and format.

■ Artificial neural networks can be used as associative memories.

The simplest artificial neural associative memory is the linear

associater. The other popular ANN models used as associative

memories are Hopfield model and Bidirectional

Associative Memory (BAM) models.

52
Adaptive Resonance Theory (ART)

ART stands for "Adaptive Resonance Theory", invented by Stephen Grossberg

in 1976. ART encompasses a wide variety of neural networks, based explicitly
on neurophysiology. The word "Resonance" is a concept, just a matter of being
within a certain threshold of a second similarity measure.

The basic ART system is an unsupervised learning model, similar to many

iterative clustering algorithm where each case is processed by finding the
"nearest" cluster seed that resonate with the case and update the cluster seed
to be "closer" to the case. If no seed resonate with the case then a new cluster
is created.

Note : The terms nearest and closer are defined in many ways in clustering
algorithm. In ART, these two terms are defined in slightly different way by
introducing the concept of "resonance".

53
• Definitions of ART and other types of Learning

ART is a neural network topology whose dynamics are based on Adaptive

Resonance Theory (ART). Grossberg developed ART as a theory of human
cognitive information processing. The emphasis of ART neural networks
lies at unsupervised learning and self-organization to mimic biological
behavior. Self-organization means that the system must be able to build
stable recognition categories in real-time.

The unsupervised learning means that the network learns the significant
patterns on the basis of the inputs only. There is no feedback. There is no
external teacher that instructs the network or tells to which category a
certain input belongs. Learning in biological systems always starts as
unsupervised learning; Example : For the newly born, hardly any pre-
existing categories exist.

The other two types of learning are reinforcement learning and

supervised learning. In reinforcement learning the net receives only

limited feedback, like "on this input you performed well" or "on this

input you have made an error". In supervised mode of learning a net

receives for each input the correct response.

Note: A system that can learn in unsupervised mode can always

be adjusted to learn in the other modes, like reinforcement mode

or supervised mode. But a system specifically designed to learn in

supervised mode can never perform in unsupervised mode.

54
• Description of Adaptive Resonance Theory

The basic ART system is an unsupervised learning model.

The model typically consists of :

– a comparison field and a recognition field composed of neurons,

– a vigilance parameter, and

– a reset module.

The functions of each of these constituents are explained below.

■ Comparison field and Recognition field

- The Comparison field takes an input vector (a 1-D array of values)

and transfers it to its best match in the Recognition field; the

best match is, the single neuron whose set of weights (weight
vector) matches most closely the input vector.

- Each Recognition Field neuron outputs a negative signal

(proportional to that neuron’s quality of match to the input vector)
to each of the other Recognition field neurons and inhibits their
output accordingly.

- Recognition field thus exhibits lateral inhibition, allowing each

neuron in it to represent a category to which input vectors are
classified.

■ Vigilance parameter

It has considerable influence on the system memories:

- higher vigilance produces highly detailed memories,

- lower vigilance results in more general memories

55
 ■ Reset module

After the input vector is classified, the Reset module compares the
strength of the recognition match with the vigilance parameter.

- If the vigilance threshold is met, Then training commences.

- Else, the firing recognition neuron is inhibited until a new input

vector is applied;

• Training ART-based Neural Networks

Training commences only upon completion of a search procedure.

What happens in this search procedure :

- The Recognition neurons are disabled one by one by the reset function
until the vigilance parameter is satisfied by a recognition match.

- If no committed recognition neuron’s match meets the vigilance

threshold, then an uncommitted neuron is committed and adjusted
towards matching the input vector.

Methods of training ART-based Neural Networks:

There are two basic methods, the slow and fast learning.

- Slow learning method : here the degree of training of the recognition

neuron’s weights towards the input vector is calculated using
differential equations and is thus dependent on the length of time the
input vector is presented.

- Fast learning method : here the algebraic equations are used to calculate
degree of weight adjustments to be made, and binary values are used.

56
 Note : While fast learning is effective and efficient for a variety of tasks,
the slow learning method is more biologically plausible and can be used
with continuous-time networks (i.e. when the input vector can vary
continuously).

• Types of ART Systems :

The ART Systems have many variations :

ART 1, ART 2, Fuzzy ART, ARTMAP

■ ART 1: The simplest variety of ART networks, accept only binary inputs.

■ ART 2 : It extends network capabilities to support continuous inputs.

■ Fuzzy ART : It Implements fuzzy logic into ART’s pattern recognition, thus
enhances generalizing ability. One very useful feature of fuzzy ART is
complement coding, a means of incorporating the absence of features into
pattern classifications, which goes a long way towards preventing
inefficient and unnecessary category proliferation.

■ ARTMAP : Also known as Predictive ART, combines two slightly

modified ARTs , may be two ART-1 or two ART-2 units into a

supervised learning structure where the first unit takes the input data
and the second unit takes the correct output data, then used to make
the minimum possible adjustment of the vigilance parameter in the
first unit in order to make the correct classification.

57
Applications of Soft Computing

The applications of Soft Computing have proved two main advantages.

- First, in solving nonlinear problems, where mathematical models are not

available, or not possible.

- Second, introducing the human knowledge such as cognition, recognition,

understanding, learning, and others into the fields of computing.

This resulted in the possibility of constructing intelligent systems such as

autonomous self-tuning systems, and automated designed systems.

The relevance of soft computing for pattern recognition and image processing
is already established during the last few years. The subject has recently
gained importance because of its potential applications in problems like :

- Remotely Sensed Data Analysis,

- Data Mining, Web Mining,

- Global Positioning Systems,

- Medical Imaging,

- Forensic Applications,

- Optical Character Recognition,

- Signature Verification,

- Multimedia,

- Target Recognition,

- Face Recognition and

- Man Machine Communication.

58
 Fundamentals of Neural Networks

What is Neural Net ?

• A neural net is an artificial representation of the human brain that tries to

simulate its learning process. An artificial neural network (ANN) is often
called a "Neural Network" or simply Neural Net (NN).

• Traditionally, the word neural network is referred to a network of biological

neurons in the nervous system that process and transmit information.

• Artificial neural network is an interconnected group of artificial neurons

that uses a mathematical model or computational model for information
processing based on a connectionist approach to computation.

• The artificial neural networks are made of interconnecting artificial

neurons which may share some properties of biological neural networks.

• Artificial Neural network is a network of simple processing elements

(neurons) which can exhibit complex global behavior, determined by the
connections between the processing elements and element parameters.

59
Introduction

Neural Computers mimic certain processing capabilities of the human brain.

- Neural Computing is an information processing paradigm, inspired by

biological system, composed of a large number of highly interconnected
processing elements (neurons) working in unison to solve specific problems.

- Artificial Neural Networks (ANNs), like people, learn by example.

- An ANN is configured for a specific application, such as pattern recognition

or data classification, through a learning process.

- Learning in biological systems involves adjustments to the synaptic

connections that exist between the neurons. This is true of ANNs as well.

60
Why Neural Network

Neural Networks follow a different paradigm for computing.

■ The conventional computers are good for - fast arithmetic and does
what programmer programs, ask them to do.

■ The conventional computers are not so good for - interacting with

noisy data or data from the environment, massive parallelism, fault
tolerance, and adapting to circumstances.

■ The neural network systems help where we can not formulate an

algorithmic solution or where we can get lots of examples of the
behavior we require.

■ Neural Networks follow different paradigm for computing.

The von Neumann machines are based on the processing/memory

abstraction of human information processing.

The neural networks are based on the parallel architecture of biological

brains.

■ Neural networks are a form of multiprocessor computer system, with

- simple processing elements ,

- a high degree of interconnection,

- simple scalar messages, and

- adaptive interaction between elements.

61
Research History

The history is relevant because for nearly two decades the future of Neural
network remained uncertain.

McCulloch and Pitts (1943) are generally recognized as the designers of the
first neural network. They combined many simple processing units together
that could lead to an overall increase in computational power. They suggested
many ideas like : a neuron has a threshold level and once that level is
reached the neuron fires. It is still the fundamental way in which ANNs
operate. The McCulloch and Pitts's network had a fixed set of weights.

Hebb (1949) developed the first learning rule,

active at the same time then the strength that is if two neurons are
increased. between them should be

In the 1950 and 60's, many researchers (Block, Minsky, Papert, and
Rosenblatt worked on perceptron. The neural network model could be proved
to converge to the correct weights, that will solve the problem. The weight
adjustment (learning algorithm) used in the perceptron was found more
powerful than the learning rules used by Hebb. The perceptron caused great
excitement. It was thought to produce programs that could think.

Minsky & Papert (1969) showed that perceptron could not learn those
functions which are not linearly separable.

The neural networks research declined throughout the 1970 and until mid
80's because the perceptron could not learn certain important functions.

62
Neural network regained importance in 1985-86. The researchers, Parker
and LeCun discovered a learning algorithm for multi-layer networks called
back propagation that could solve problems that were not linearly
separable.

63
Biological Neuron Model

The human brain consists of a large number, more than a billion of neural
cells that process information. Each cell works like a simple processor. The
massive interaction between all cells and their parallel processing only
makes the brain's abilities possible.

Dendrites are branching fibers that

extend from the cell body or soma.
Soma or cell body of a neuron
contains the nucleus and other
structures, support chemical processing
and production of neurotransmitters.

Axon is a singular fiber carries information

away from the soma to the synaptic sites
of other neurons (dendrites and somas),
muscles, or glands.

Axon hillock is the site of summation for

incoming information. At any moment, the
collective influence of all neurons that
conduct impulses to a given neuron will
determine whether or not an

Fig. Structure of Neuron

action potential will be initiated at the

axon hillock and propagated along the axon.

Myelin Sheath consists of fat-containing cells that insulate the axon from electrical
activity. This insulation acts to increase the rate of transmission of signals. A gap
exists between each myelin sheath cell along the axon. Since fat inhibits the
propagation of electricity, the signals jump from one gap to the next.

64
 Nodes of Ranvier are the gaps (about 1 m) between myelin sheath cells long
axons are Since fat serves as a good insulator, the myelin sheaths speed the rate of
transmission of an electrical impulse along the axon.

Synapse is the point of connection between two neurons or a neuron and a muscle or
a gland. Electrochemical communication between neurons takes place at these
junctions.

Terminal Buttons of a neuron are the small knobs at the end of an axon that
release chemicals called neurotransmitters.

• Information flow in a Neural Cell

The input /output and the propagation of information are shown below.

Fig. Structure of a neural cell in the human brain

■ Dendrites receive activation from other neurons.

■ Soma processes the incoming activations and converts them into

output activations.

65
 ■ Axons act as transmission lines to send activation to other neurons.

■ Synapses the junctions allow signal transmission between the axons

and dendrites.

■ The process of transmission is by diffusion of chemicals called neuro-

transmitters.

McCulloch-Pitts introduced a simplified model of this real neurons.

66
 SC - Neural Network – Introduction

1.4 Artificial Neuron Model

An artificial neuron is a mathematical function conceived as a simple

model of a real (biological) neuron.

• The McCulloch-Pitts Neuron

This is a simplified model of real neurons, known as a Threshold Logic Unit.

■ A set of input connections brings in activations from other neurons.

■ A processing unit sums the inputs, and then applies a non-linear

activation function (i.e. squashing / transfer / threshold function).

■ An output line transmits the result to other neurons.

In other words ,

- The input to a neuron arrives in the form of signals.

- The signals build up in the cell.

- Finally the cell discharges (cell fires) through the output .

- The cell can start building up signals again.

67
Single Layer Feed-forward Network

The Single Layer Feed-forward Network consists of a single layer of

weights , where the inputs are directly connected to the outputs, via a
series of weights. The synaptic links carrying weights connect every input
to every output , but not other way. This way it is considered a network of
feed-forward type. The sum of the products of the weights and the inputs is
calculated in each neuron node, and if the value is above some threshold
(typically 0) the neuron fires and takes the activated value (typically 1);
otherwise it takes the deactivated value (typically -1).

input xi output yj
weights wij

w11
x1 y1
w21

w12

w22
x2 y2
w2m

w1m
wn1

wn2

xn ym
wnm

Single layer

Neurons

Fig. Single Layer Feed-forward Network

68
Multi Layer Feed-forward Network

The name suggests, it consists of multiple layers. The architecture of this

class of network, besides having the input and the output layers, also have
one or more intermediary layers called hidden layers . The computational
units of the hidden layer are known as hidden neurons.

Fig.
Multilayer feed-forward network in (ℓ – m – n) configuration.

■ The hidden layer does intermediate computation before directing

the input to output layer.

■ The input layer neurons are linked to the hidden layer neurons; the
weights on these links are referred to as input-hidden layer weights.

■ The hidden layer neurons and the corresponding weights are referred to
as output-hidden layer weights.

69
 ■A multi-layer feed-forward network with ℓ input neurons, m1 neurons in
the first hidden layers, m2 neurons in the second hidden layers, and n
output neurons in the output layers is written as (ℓ - m1 - m2 – n ).

The Fig. above illustrates a multilayer feed-forward network with a

configuration (ℓ - m – n).

Recurrent Networks

The Recurrent Networks differ from feed-forward architecture. A Recurrent

network has at least one feed back loop.

Example :

There could be neurons with self-feedback links; that is the output of a

neuron is fed back into it self as input.

70
Learning Methods in Neural Networks

The learning methods in neural networks are classified into three basic types :

• Supervised Learning,
• Reinforced Learning

These three types are classified based on :

• presence or absence of teacher and

• the information provided for the system to learn.

These are further categorized, based on the rules used, as

• Hebbian,

• Gradient descent,

• Competitive and

• Stochastic learning.

71
• Classification of Learning Algorithms

Fig. below indicate the hierarchical representation of the algorithms

mentioned in the previous slide. These algorithms are explained in
subsequent slides.

72
 b Supervised Learning

A teacher is present during learning process and presents expected

output.

Every input pattern is used to train the network.

Learning process is based on comparison, between network's computed

output and the correct expected output, generating "error".

The "error" generated is used to change network parameters that result

improved performance.

c Unsupervised Learning

No teacher is present.

The expected or desired output is not presented to the network.

The system learns of it own by discovering and adapting to the

structural features in the input patterns.

d Reinforced learning

A teacher is present but does not present the expected or desired

output but only indicated if the computed output is correct or incorrect.

The information provided helps the network in its learning process.

A reward is given for correct answer computed and a penalty for a

wrong answer.

Note : The Supervised and Unsupervised learning methods are most popular
forms of learning compared to Reinforced learning.

73
• Hebbian Learning

Hebb proposed a rule based on correlative weight adjustment.

In this rule, the input-output pattern pairs (Xi , Yi) are associated by the
weight matrix W, known as correlation matrix computed as

T
W= Σn Xi Yi

i=1

T
where Yi is the transpose of the associated output vector Yi

There are many variations of this rule proposed by the other researchers
(Kosko, Anderson, Lippman) .

74
 • Gradient descent Learning

This is based on the minimization of errors E defined in terms of weights

and the activation function of the network.

- Here, the activation function

differentiable, because the of the network is required to be
updates of weight is dependent on

the gradient of the error E.

• If ∆ Wij is the weight update of the link connecting the i th and the j th

neuron of the two neighboring layers, then ∆ Wij is defined as

Wij = ( E/ Wij )
η ∂ ∂

where
η is the learning rate parameters and E/ Wij ) is error
(∂ ∂
gradient
with reference to the weight Wij .

Note : The Hoffs Delta rule and Back-propagation

learning rule are

the examples of Gradient descent learning.

75
• Competitive Learning

In this method, those neurons which respond strongly to the input

stimuli have their weights updated.

When an input pattern is presented, all neurons in the layer compete,

and the winning neuron undergoes weight adjustment .

This strategy is called "winner-takes-all".

• Stochastic Learning

In this method the weights are adjusted in a probabilistic fashion.

- Example : Simulated annealing which is a learning mechanism

employed by Boltzmann and Cauchy machines.

76
• Taxonomy Of Neural Network Systems

In the previous sections, the Neural Network Architectures and the Learning
methods have been discussed. Here the popular neural network

systems are listed. The grouping of these systems in terms of architectures and
the learning methods are presented in the next slide.

• Neural Network Systems

– ADALINE (Adaptive Linear Neural Element)

– ART (Adaptive Resonance Theory)

– AM (Associative Memory)

– BAM (Bidirectional Associative Memory)

– Boltzmann machines

– BSB ( Brain-State-in-a-Box)

– Cauchy machines

– Hopfield Network

– LVQ (Learning Vector Quantization)

– Neoconition

77
 – Perceptron

– RBF ( Radial Basis Function)

– RNN (Recurrent Neural Network)

– SOFM (Self-organizing Feature Map)

30

78
 • Classification of Neural Network

A taxonomy of neural network systems based on Architectural types and

the Learning methods is illustrated below.

Learning Methods

Gradient Hebbian Competitive Stochastic

descent

Single-layer ADALINE, AM, LVQ, -

feed-forward Hopfield, Hopfield, SOFM

Percepton,

Multi-layer CCM, Neocognition

feed- forward MLFF,

RBF

Recurrent RNN BAM, ART Boltzmann and

Networks BSB, Cauchy

Hopfield, machines

Table : Classification of Neural Network Systems with respect to

learning methods and Architecture types

31

79
■ Single-Layer NN Systems

Here, a simple Perceptron Model and an ADALINE Network Model is presented.

6.1 Single layer Perceptron

Definition : An arrangement of one input layer of neurons feed forward to

one output layer of neurons is known as Single Layer Perceptron.

input output yj
xi
weights wij

w11
y1
x1
w21
w12

w22
x2 y2
w2m

w1m
wn1

wn2

xn ym
wnm

Single layer

Perceptron

Fig. Simple Perceptron Model

where net j =

y j = f (net j) = 1 if net j ≥ 0 Σn xi wij

0 if net j  0 i=1

80
■ Learning Algorithm : Training Perceptron

The training of Perceptron is a supervised learning algorithm where

weights are adjusted to minimize error when ever the output does not
match the desired output.

– If the output is correct then no adjustment of weights is done.

K+1 K
i.e. W = W
ij ij

– If the output is 1 but should have been 0 then the weights are

decreased on the active input link

K+1 K

W – α.
i.e. ij = W i j xi

– If the output is 0 but should have been 1 then the weights are
increased on the active input link

K+1 K
i.e. W ij + . xi
α
=W ij

Where

K+1 is the new adjusted weight,

K
W ij is the old weight

Wi j

xi is the input and α is the learning rate parameter.

81
82
 αsmall leads to slow and α large leads to fast learning.

• Perceptron and Linearly Separable Task

Perceptron can not handle tasks which are not separable.

■ Definition : Sets of points in 2-D space are linearly separable if the sets
can be separated by a straight line.

■ Generalizing, a set of points in n-dimensional space are linearly

separable if there is a hyper plane of (n-1) dimensions separates the
sets.

Example

S1 S2 S1

S2

(a) Linearly separable patterns (b) Not Linearly separable patterns

Note : Perceptron cannot find weights for classification problems that are
not linearly separable.

83
 • XOR Problem :
Exclusive OR operation

X2

Input x1 Input x2 Output

(0, 1)

 (1, 1)

0 0 0
Even parity 
1 1 0

0 1 1

Odd parity  (0, 0)   X1

1 0 1

(0, 1)

XOR truth table

Fig. Output of XOR

in X1 , x2 plane

Even parity is, even number of 1 bits in the input

Odd parity is, odd number of 1 bits in the input

■ There is no way to draw a single straight line so that the circles are on
one side of the line and the dots on the other side.

■ Perceptron is unable to find a line separating even parity input patterns

from odd parity input patterns.

35

84
 SC - Neural Network –Single Layer learning

• Perceptron Learning
Algorithm

The algorithm is illustrated step-by-step.

■ Step 1 :

Create
a peceptron with (n+1) input neurons x0 , x1 ,.........., . xn ,

where x0 = 1 is the bias input.

xi wi
Let O be the output neuron.

■ Step 2 :

Initialize weight W = (w0 , w1 , . . . . . , . wn ) to random weights.

• Step 3 :
for
Iterate through the input patterns Xj of the training set using the weight set; ie compute the weighted sum of inputs net j =
Σn

i=1

for each input pattern j .

• Step 4 :

Compute the output y j using the step function

y j = f (net j) = 1 if net j ≥ 0 where net j = Σn x i wij

i=1
0 if net j  0

• Step 5 :

85
 Compare the computed output yj with the target output yj each
input pattern j .

If all the input patterns have been classified correctly, then output
(read) the weights and exit.

= Step 6 :

Otherwise, update the weights as given below :

If the computed outputs yj is 1 but should have been 0,

Then wi = wi - α xi , i= 0, 1, 2,........., n

If the computed outputs yj is 0 but should have been 1,

Then wi = wi + α xi , i= 0, 1, 2,........, n

where α is the learning parameter and is constant.

z Step 7 :

goto step 3

• END
36

86
 SC - Neural Network –ADALINE

6.2 ADAptive LINear Element (ADALINE)

An ADALINE consists of a single neuron of the McCulloch-Pitts type, where

its weights are determined by the normalized least mean square (LMS)
training law. The LMS learning rule is also referred to as delta rule. It is a
well-established supervised training method that has been used over a
wide range of diverse applications.

• Architecture of a simple ADALINE

87
- ADALINE Training Mechanism

(Ref. Fig. in the previous slide - Architecture of a simple ADALINE)

-
The basic structure of an ADALINE is similar to a linear neuron with an
extra feedback loop.

■ During the training phase of ADALINE, the input vector

X = [x1 , x2 , . . . , xn]T as well as desired output are presented

to the network.

■ The weights are adaptively adjusted based on delta rule.

•
After the ADALINE is trained, an input vector presented to the

network with fixed weights will result in a scalar output.

•
Thus, the network performs an n dimensional mapping to a scalar
value.

•
The activation function is not used during the training phase. Once the
weights are properly adjusted, the response of the trained unit can be
tested by applying various inputs, which are

not in the training [Link] the network produces consistent

responses to a high degree with the test inputs, it is said that the
network could generalize. The process of training and generalization
are two important attributes of this network.

Usage of ADLINE

88
 In practice, an ADALINE is used to Make binary decisions; the output is sent
through a binary threshold.

■ Realizations of logic gates such as AND, NOT and OR .

■ Realize only those logic functions that are linearly separable.

■ Applications of Neural Network

Neural Network Applications can be grouped in following categories:

Clustering:

A clustering algorithm explores the similarity between patterns and places

similar patterns in a cluster. Best known applications include data
compression and data mining.

Classification/Pattern recognition:

The task of pattern recognition is to assign an input pattern (like

handwritten symbol) to one of many classes. This category includes
algorithmic implementations such as associative memory.

•
Function approximation :

The tasks of function approximation is to find an estimate of the unknown

function subject to noise. Various engineering and scientific disciplines
require function approximation.

•
Prediction Systems:

The task is to forecast some future values of a time-sequenced data.

Prediction has a significant impact on decision support systems. Prediction
differs from function approximation by considering time factor. System
may be dynamic and may produce different results for the same input
data based on system state (time).

89
Back-Propagation Network
What is BPN ?

• A single-layer neural network has many restrictions. This network can

accomplish very limited classes of tasks.

Minsky and Papert (1969) showed that a two layer feed-forward

network can overcome many restrictions, but they did not present

a solution to the problem as "how to adjust the weights from input

to hidden layer" ?

•
An answer to this question was presented by Rumelhart, Hinton and

Williams in 1986. The central idea behind this solution is that the

errors for the units of the hidden layer are determined by back-

propagating the errors of the units of the output layer.

This method is often called the Back-propagation learning rule.

Back-propagation can also be considered as a generalization of the delta

rule for non-linear activation functions and multi-layer networks.

• Back-propagation is a systematic method of training multi-layer artificial

neural networks.

90
1. Back-Propagation Network – Background

Real world is faced with a situations where data is incomplete or noisy. To

make reasonable predictions about what is missing from the information

available is a difficult task when there is no a good theory available that may to
help reconstruct the missing data. It is in such situations the Back-propagation
(Back-Prop) networks may provide some answers.

• A BackProp network consists of at least three layers of units :

- an input layer,

- at least one intermediate hidden layer, and

- an output layer.

• Typically, units are connected in a feed-forward fashion with input units

fully connected to units in the hidden layer and hidden units fully
connected to units in the output layer.

• When a BackProp network is cycled, an input pattern is propagated

forward to the output units through the intervening input-to-hidden and
hidden-to-output weights.

• The output of a BackProp network is interpreted as a classification

decision.

91
• With BackProp networks, learning occurs during a training phase.
The steps followed during learning are :

– each input pattern in a training set is applied to the input units and then
propagated forward.

– the pattern of activation arriving at the output layer is compared

with the correct (associated) output pattern to calculate an error signal.

– the error signal for each such target output pattern is then back-
propagated from the outputs to the inputs in order to

appropriately adjust the weights in each layer of the network.

– after a BackProp network has learned the correct classification for a set
of inputs, it can be tested on a second set of inputs to see how well it
classifies untrained patterns.

•
An important consideration
in applying BackProp learning is how

well the network generalizes.

92
1.1 Learning :

AND function

Implementation of AND function in the neural network.

– there are 4 inequalities in the AND function and they must be

satisfied.

w10 + w2 0 < θ , w1 0 + w2 1 < θ ,

w11 + w2 0 < θ , w1 1 + w2 1 > θ

– one possible solution :

if both weights are set to 1 and the threshold is set to 1.5, then

(1)(0) + (1)(0) < 1.5 assign 0 , (1)(0) + (1)(1) < 1.5 assign 0

(1)(1) + (1)(0) < 1.5 assign 0 , (1)(1) + (1)(1) > 1.5 assign 1

Although it is straightforward to explicitly calculate a solution to the AND

function problem, but the question is "how the network can learn such a
solution". That is, given random values for the weights can we define an
incremental procedure which will cover a set of weights which implements
AND function.

93
1.2 Simple Learning Machines

Rosenblatt (late 1950's) proposed learning networks called Perceptron.

The task was to discover a set of connection weights which correctly
classified a set of binary input vectors. The basic architecture of the
perceptron is similar to the simple AND network in the previous example.

A perceptron consists of a set of input units and a single output unit.

in the AND network, the output of the perceptron is calculated n

i=1

If the net input is greater than the threshold θ , then the output unit is

turned on , otherwise it is turned off.

To address the learning question, Rosenblatt solved two problems.

– first, defined a cost function which measured error.

– second, defined a procedure or a rule which reduced that error by

appropriately adjusting each of the weights in the network.

However, the procedure (or learning rule) required to assesses the

relative contribution of each weight to the total error.

The learning rule that Roseblatt developed, is based on determining

the difference between the actual output of the network with the

target output (0 or 1), called "error measure"

94
 • Error Measure ( learning rule )

Mentioned in the previous slide, the error measure is the difference

between actual output of the network with the target output (0 or 1).

—
If the input vector is correctly classified (i.e., zero
error), then the weights are left unchanged, and

the next input vector is presented.

—
If the input vector is incorrectly classified (i.e., not zero
error), then there are two cases to consider :

Case 1 : If the output unit is 1 but need to be 0 then

◊ the threshold is incremented by 1 (to make it less likely that the

output unit would be turned on if the same input vector was
presented again).

◊ If the input Ii is 0, then the corresponding weight Wi is left

unchanged.

◊ If the input Ii is 1, then the corresponding weight Wi is

decreased by 1.

Case 2 : If output unit is 0 but need to be 1 then the opposite changes

are made.

09

95
 SC - NN – BPN – Background

• Perceptron Learning Rule :

Equations

The perceptron learning rules are govern by two equations, −

one that defines the change in the threshold and

– the other that defines change in the weights,

The change in the threshold is given by

∆ θ = - (tp - op) = - dp

where p specifies the presented input pattern,

op actual output of the input pattern Ipi

tp specifies the correct classification of the input pattern ie target,

dp is the difference between the target and actual outputs.

The change in the weights are given by

∆ wi = (tp - op) Ipi = - dp Ipi

10

96
 SC - NN - BPN – Background

1.3 Hidden Layer

Back-propagation is simply a way to determine the error values in hidden

layers. This needs be done in order to update the weights.

The best example to explain where back-propagation can be used is the

XOR problem.

Consider a simple graph shown below.

– all points on the right side of the line are +ve, therefore the output of
the neuron should be +ve.

– all points on the left side of the line are –ve, therefore the output of the
neuron should be –ve.

97
98
• Back Propagation Network

Learning By Example

Consider the Multi-layer feed-forward back-propagation network below.

The subscripts I, H, O denotes input, hidden and output neurons.

th th
The weight of the arc between i input neuron to j hidden layer is Vij .

The weight of the arc between i th hidden neuron to j th out layer is Wij

99
2.1 Computation of Input, Hidden and Output Layers

- Input Layer Computation

Consider linear activation function.

If the output of the input layer is the input of the input layer and the
transfer function is 1, then

{ O }I
= { I }I

ℓx1
ℓx1 (denotes matrix row, column size)

The hidden neurons are connected by synapses to the input neurons.

th th
- Let Vij be the weight of the arc between i input neuron to j hidden

layer.

■ The input to the hidden neuron is the weighted sum of the outputs of
the input neurons. Thus the equation

IHp = V1p OI1 + V2p OI2 +.........+ V1p OIℓ where (p =1, 2, 3 . . , m)
denotes weight matrix or connectivity matrix between input neurons

and a hidden neurons as [ V ].

we can get an input to the hidden neuron as ℓ x m

T
{ I }H = [ V ] { O }I

mx1 mxℓ ℓ x 1 (denotes matrix row, column size)

10
0
 SC - NN – Back Propagation Network

• Hidden Layer Computation

th
Shown below the p neuron of the hidden layer. It has input from the
output of the input neurons layers. If we consider transfer function as
th
sigmoidal function then the output of the p hidden neuron is given by

OHp =
(1+e - λ (IHP – θHP )

where
th
OHp is the output of the p hidden neuron,

th
IHp is the input of the p hidden neuron, and is
th
the threshold of the p neuron;
θHP

10
1
 Treating each component of the input of the hidden neuron separately, we
get the outputs of the hidden neuron as given by above equation .

The input to the output neuron is the weighted sum of the outputs of the
th
hidden neurons. Accordingly, Ioq the input to the q output neuron is
given by the equation

Ioq = W1q OH1 + W2q OH2 + . . . . + Wmq OHm , where (q =1, 2, 3 . . , n)

It denotes weight matrix or connectivity matrix between hidden neurons

and output neurons as [ W ], we can get input to output neuron as

T
{ I }O = [ W] { O }H

nx1 nxm mx1 (denotes matrix row, column size)

■ Output Layer Computation

th
Shown below the q neuron of the output layer. It has input from the
output of the hidden neurons layers.

If we consider transfer function as sigmoidal function then the output of

th
the q output neuron is given by

10
2
 1

OOq =
(1+e - λ (IOq – θOq )

where

OOq th
is the output of the q output neuron,

th
is the input to the q output neuron, and
IOq

th
is the threshold of the q neuron;
θOq

10
3
2.2 Calculation of Error

(refer the earlier slides - Fig. "Multi-layer feed-forward back-propagation

network" and a table indicating an 'nset' of input and out put data for the purpose
of training)

th
Consider any r output neuron. For the target out value T, mentioned in
the table- 'nset' of input and output data" for the purpose of training,
calculate output O .

th
The error norm in output for the r output neuron is

r = (1/2) e r2= (1/2) (T –O)

E1 2

1 th
where E r is 1/2 of the second norm of the error er in the
for the given training pattern.

2
e r is the square of the error, considered to make it independent

of sign +ve or –ve , ie consider only the absolute value.

1
The Euclidean norm of error E for the first training pattern is given by

1
E = (1/2)
2
Σn (Tor - Oor )

r=1

This error function is for one training pattern. If we use the same
technique for all the training pattern, we get

nset
E (V, W) =
E j (V, W, I)
Σ
r=1

where E is error function depends on

m ( 1 + n)

10
4
 weights of [W] and [V].

All that is stated is an optimization problem solving, where the

objective or cost function is usually defined to be maximized or

minimized with respect to a set of parameters. In this case, the

network parameters that optimize the error function E over the 'nset'

of pattern sets [I nset , t nset ] are synaptic weight values [ V ] and

[ W ] whose sizes are

[V] and [W]

ℓxm mxn

16

10
5
 SC - NN - BPN – Algorithm

• Back-Propagation Algorithm

The benefits of hidden layer neurons have been explained. The hidden layer
allows ANN to develop its own internal representation of input-output mapping.
The complex internal representation capability allows the hierarchical network
to learn any mapping and not just the linearly separable ones.

The step-by-step algorithm for the training of Back-propagation network is

presented in next few slides. The network is the same , illustrated before,

has a three layer. The input layer is with ℓ nodes, the hidden layer with m
nodes and the output layer with n nodes. An example for training a BPN with
five training set have been shown for better understanding.

17

10
6
 SC - NN - BPN – Algorithm

3.1 Algorithm for Training Network

The basic algorithm loop structure, and the step by step procedure of
Back- propagation algorithm are illustrated in next few slides.

■ Basic algorithm loop structure

Initialize the weights

Repeat

For each training pattern

"Train on that pattern"

End

Until the error is acceptably low.

18

10
7
 SC - NN - BPN – Algorithm

- Back-Propagation Algorithm - Step-by-step procedure

Step 1 :

Normalize the I/P and O/P with respect to their maximum values. For
each training pair, assume that in normalized form there are

ℓ inputs given by { I }I and

•x 1

outputs given by { O}O

nx1

• Step 2 :

Assume that the number of neurons in the hidden layers lie

between 1 < m < 21

19

10
8
 SC - NN - BPN – Algorithm

2. Step 3 :

Let [ V ] represents the weights of synapses connecting input neuron

and hidden neuron

Let [ W ] represents the weights of synapses connecting hidden neuron

and output neuron

Initialize the weights to small random values usually from -1 to +1;

[V] 0

= [ random weights ]

[W] 0

= [ random weights ]

[∆ V] 0 =[∆ W] 0 =[0]

λ
For
general problems can be assumed as 1 and threshold

value as
0.

20

10
9
 SC - NN - BPN – Algorithm

• Step 4 :

For training data, we need to present one set of inputs and

outputs. Present the pattern as inputs to the input layer { I }I .

then by using linear activation function, the output of the input layer
may be evaluated as

{ O }I = { I }I

ℓx1 ℓx1

2. Step 5 :

Compute the inputs to the hidden layers by multiplying corresponding

weights of synapses as

T
{ I }H = [ V] { O }I

mx1 mxℓ ℓx1

• Step 6 :

Let the hidden layer units, evaluate the output using the sigmoidal
function as

–
1

{ O }H =
(1+e - (IHi) )

11
0
 –

mx1

21

11
1
 SC - NN - BPN – Algorithm

- Step 7 :

Compute the inputs to the output layers by multiplying corresponding

weights of synapses as

T
{ I }O = [ W] { O }H

nx1 nxm mx1

■ Step 8 :

Let the output layer units, evaluate the output using sigmoidal function
as

–
1

{ O }O = (1+e - (IOj) )
–

Note : This output is the network output

22

11
2
 SC - NN - BPN – Algorithm

■ Step 9 :

Calculate the error using the difference between the network output
th
and the desired output as for the j training set as

(Tj - Ooj )2
EP = √ ∑ n

Α Step 10 :

Find a term { d } as

–
–

{d}= (Tk – OOk) OOk (1 – OOk )

–
nx1

23

11
3
 SC - NN - BPN – Algorithm

■ Step 11 :

Find [ Y ] matrix as

[ Y ] = { O }H 〈d〈

mxn mx1 1xn

■ Step 12
:
t +1
[∆ W] =
t
α [∆ W] + η[Y]
Find
mxn mxn mxn

Step 13 :

Find
{e}=[W]{d}

mx1 mxn nx1

–
(OHi) (1 – OHi )

{ d* } = e i

mx1mx1

11
4
 Find [ X ] matrix as

= { I }I 〈 d*
[ X ] = { O }I 〈 d* 〈 〈

1xm ℓ x 11 x m ℓ x 11 x m

24

11
5
 SC - NN - BPN – Algorithm

• Step 14 :

Find
[ ∆ V ] t +1 =
α
[ ∆V ] t + η[ X
1xm 1xm ]

1xm

1 Step 15 :

t +1 t t +1
Find [V] = [V ] +[∆ V]

t +1 t t +1
[W] = [W ] +[∆ W]

ƒ Step 16 :

Find error rate as

∑ Ep
error rate =

nset

• Step 17 :

Repeat steps 4 to 16 until the convergence in the error rate is less

than the tolerance value

• End of Algorithm

Note : The implementation of this algorithm, step-by-step 1 to 17,

assuming one example for training BackProp Network is illustrated in
the next section.

11
6
25

11
7
 SC - NN - BPN – Algorithm

3.2 Example : Training Back-Prop Network

• Problem :

Consider a typical problem where there are 5 training sets.

Table : Training sets

S. No. Input Output

I1 I2 O
1 0.4 -0.7 0.1

2 0.3 -0.5 0.05

3 0.6 0.1 0.3

4 0.2 0.4 0.25

5 0.1 -0.2 0.12

In this problem,

- there are two inputs and one output.

■ the values lie between -1 and +1 i.e., no need to normalize the values.

- assume two neurons in the hidden layers.

• the NN architecture is shown in the Fig. below.

0.4 0.1

0.2
0.4 -0.2 TO = 0.1

11
8
 -0.7 -0.5

0.2

Input Hidden Output

layer layer layer

Fig. Multi layer feed forward neural network (MFNN) architecture

with data of the first training set

The solution to problem are stated step-by-step in the subsequent slides.

26

11
9
 SC - NN - BPN – Algorithm

■ Step 1 : Input the first training set data (ref eq. of step 1)

0.4

{ O }I = { I }I =
-0.7
ℓx1 ℓx1
2x1

from training set [Link] 1

■ Step 2 : Initialize the weights as (ref eq. of step 3 & Fig)

0.2
0.1 0.4
0
0 [W]
=
;
[V] = -0.5
-0.2 0.2
2 x1
2x2

from fig initialization from fig initialization

T
■ Step 3 : Find { I }H = [ V] { O }I as (ref eq. of step 5)

0.1 -0.2 0.4 0.18

{ I }H = =
-0.40.2 -0.7 0.02

Values from step 1 & 2

27

12
0
 SC - NN - BPN – Algorithm

■ Step 4 (ref eq. of step 6)

:
1

(1+e - (0.18) )
0.5448

{ O }H = =

1 0.505

(1+e - (0.02))

Values from step 3 values

28

12
1
 SC - NN - BPN – Algorithm

■ Step 5 : (ref eq. of step 7)

0.5448

{ I }O = [ W] T { O }H = ( 0.2 - 0.5 ) =- 0.14354

0.505

Values from step 2 , from step 4

■ Step 6 : (ref eq. of step 8)

{ O }O =
- (0.14354) = 0.4642
(1+e )

Values from step 5

■ Step 7 : (ref eq. of step 9)

Error = (TO – OO1 )2 = (0.1 – 0.4642)2 = 0.13264

table first training set o/p from step 6

29

12
2
 SC - NN - BPN – Algorithm

■ Step 8 : (ref eq. of step 10)

■ = (TO – OO1 ) ( OO1 ) (1 – OO1 )

= (0.1 – 0.4642) (0.4642) ( 0.5358) = – 0.09058

Training o/p all from step 6

(ref eq. of step 11)

0.5448 –0.0493

[ Y ] = { O }H (d ) = (– 0.09058) =
0.505 –0.0457

from values at step 4 from values at step 8 above

■ Step 9 : (ref eq. of step 12)

1
[∆ W] = η[Y
0
α [∆ W] + ] assume η =0.6

–0.02958

=
–0.02742

from values at step 2 & step 8 above

■ Step 10 : (ref eq. of step 13)

0.2 –0.018116

{e}=[W]{d}= (– 0.09058) =

-0.5 –0.04529

from values at step 8 above

from values at step 2

12
3
30

12
4
 SC - NN - BPN – Algorithm

■ Step 11 : (ref eq. of step 13)

(–0.018116) (0.5448) (1- 0.5448) –0.00449

{ d* } = =

(0.04529) (0.505) ( 1 – 0.505) –0.01132

from values at step 10 at step 4 at step 8

■ Step 12 : (ref eq. of step 13)

0.4

[ X ] = { O }I ( d* ) = -0.7 ( – 0.00449 0.01132)

from values at step 1 from values at step 11 above

– 0.001796 0.004528

■ 0.003143 –0.007924

■ Step 13 : (ref eq. of step 14)

– 0.001077 0.002716
1 0
[ ∆V ] = α [ ∆V ] + η[X]= 0.001885–0.004754

from values at step 2 & step 8 above

31

12
5
 SC - NN - BPN – Algorithm

■ Step 14 :
(ref eq. of step 15)

0.1 0.4 – 0.001077 0.002716

1
[V] = +
-0.2 0.2 0.001885 –0.004754

from values at step 2 from values at step 13

– 0.0989 0.04027

=
0.1981 –0.19524

1 –0.02958
[W] 0.2 0.17042
=
+ =

-0.5
–0.02742 –0.52742

from values at step 2, from values at step 9

• Step 15 :

With the updated weights [ V ] and [ W ] , error is calculated again and

next training set is taken and the error will then get adjusted.

• Step 16 :

Iterations are carried out till we get the error less than the tolerance.

• Step 17 :

Once the weights are adjusted the network is ready for inferencing
new objects .

12
6
 Associative Memory

What is Associative Memory ?

• An associative memory is a content-addressable structure that maps a

set of input patterns to a set of output patterns.

— A content-addressable structure is a type of memory that allows the recall

of data based on the degree of similarity between the input pattern and
the patterns stored in memory.

— There are two types of associative memory : auto-associative and hetero-

associative.

•
An
auto-associative memory retrieves stored pattern
a
previously

that most closely resembles the current pattern.

• In a hetero-associative memory, the retrieved pattern is in general,

different from the input pattern not only in content but possibly also in
type and format.

• Neural networks are used to implement these associative memory models

called NAM (Neural associative memory).

03

12
7
 SC - AM description

1. Associative Memory

An associative memory is a content-addressable structure that maps a set of

input patterns to a set of output patterns. A content-addressable

structure refers to a memory organization where the memory is accessed by its

content as opposed to an explicit address in the traditional computer

memory system. The associative memory are of two types : auto-associative

and hetero-associative.

ƒ An auto-associative memory retrieves a previously stored pattern that most

closely resembles the current pattern.
ƒ In hetero-associative memory, the retrieved pattern is in general different

from the input pattern not only in content but possibly also in type and
format.

12
8
 SC - AM description

1.1 Description of Associative

Memory

An associative memory is a content-addressable structure that allows, the

recall of data, based on the degree of similarity between the input pattern
and the patterns stored in memory.

• Example : Associative Memory

The figure below shows a memory containing names of several people.

If the given memory is content-addressable,

Then using the erroneous string "Crhistpher Columbos" as key is

sufficient to retrieve the correct name "Christopher Colombus."

In this sense, this type of memory is robust and fault-tolerant, because

this type of memory exhibits some form of error-correction capability.

• Associative memory is a system that associates two patterns (X, Y)

such that when one is encountered, the other can be recalled. The
associative memory are of two types : auto-associative memory and
hetero-associative memory.

Auto-associative memory

Consider, y[1], y[2], y[3], . . . . . y[M], be the number of stored pattern

vectors and let y(m) be the components of these vectors, representing
features extracted from the patterns. The auto- associative memory
will output a pattern vector y(m) when inputting a noisy or incomplete
version of y(m).

12
9
 Hetero-associative memory

Here the memory function is more general. Consider, we have a

number of key-response pairs {c(1), y(1)}, {c(2), y(2)},
..............................................................................................
,
{c(M), y(M)}. The hetero-associative memory will output a pattern
vector y(m) if a noisy or incomplete verson of the c(m) is given.

• Neural networks are used to implement associative memory models.

- Linear associater is the simplest artificial neural associative memory.

- Hopfield model and Bidirectional Associative Memory (BAM)

are the other popular ANN models used as associative memories.

These models follow different neural network architectures to

memorize information.

13
0
1.2 Working of Associative Memory

■ Example

An associative memory is a storehouse of associated patterns which are

encoded in some form.

– When the storehouse is triggered or excited with a pattern, then

the associated pattern pair is recalled or appears at the output.

– The input could be an exact or distorted or partial representation of

a stored pattern.

Fig below illustrates the working of an associated memory.

The associated pattern pairs

∆ Γ (∆ , Γ ), ( , +), (7 , 4).

Input Recalled

Pattern The association is represented

Pattern
by the symbol


∆ 7 Γ

The associated pattern pairs

+ 4 are stored the memory.

.
Fig. Working of an associated memory

W e
h n

13
1
 the memory is triggered with an input pattern say

∆
then
the associated pattern Γ is retrieved automatically.

13
2
 1.3 Associative Memory - Classes

As stated before, there are two classes of associative memory:

• auto-associative and

• hetero-associative memory.

An auto-associative memory, also known as auto-associative correlator, is

used to retrieve a previously stored pattern that most closely resembles
the current pattern;

A hetero-associative memory, also known as hetero-associative correlator,

is used to retrieve pattern in general, different from the input pattern not
only in content but possibly also different in type and format.

Examples

Input Recall of Presented Recall of

pattern associated distorted perfect

presented pattern pattern pattern

Hetero-associative memory Auto-associative memory

Fig. Hetero and Auto Associative memory Correlators

08

13
3
1.4 Related Terms

Here explained : Encoding or memorization, Retrieval or recollection,

Errors and Noise, Memory capacity and Content-addressability.

■ Encoding or memorization

Building an associative memory means, constructing a connection weight

matrix W such that when an input pattern is presented, and the stored
pattern associated with the input pattern is retrieved.

This process of constructing the connection weight matrix is called

encoding. During encoding, for an associated pattern pair (Xk, Yk) , the
weight values of the correlation matrix Wk are computed as

(wij)k = (xi)k (yj)k , where

th
(xi)k represents the i component of pattern Xk , and

th
(yj)k represents the j component of pattern Yk

for i = 1, 2, . . . , m and j = 1, 2, . . . , n.

Constructing of the connection weight matrix W is accomplished

by summing up the individual correlation matrices Wk , i.e.,

Σ
W= α p Wk where

k=1

α is the proportionality or normalizing constant.

13
4
 SC - AM description

• Retrieval or recollection

After memorization, the process of retrieving a stored pattern, given an

input pattern, is called decoding.

Given an input pattern X, the decoding or recollection is accomplished by:

first compute the net input to the output units using

input j
= Σ
xi w ij
m

j=1

where
input j is weighted sum of the input or activation value of

node j , for j = 1, 2, ..., n.

then determine the units output using a bipolar output function:

+1 if input j ≥ θ j

Y j =

- 1 other wise

where θ j is the threshold value of output neuron j .

13
5
■ Errors and noise

The input pattern may contain errors and noise, or may be an incomplete
version of some previously encoded pattern.

When a corrupted input pattern is presented, the network will retrieve the
stored pattern that is closest to actual input pattern.

The presence of noise or errors results only in a mere decrease rather

than total degradation in the performance of the network.

Thus, associative memories are robust and fault tolerant because of many
processing elements performing highly parallel and distributed
computations.

13
6
 SC - AM description

- Performance Measures

The memory capacity and content-addressability are the measures of

associative memory performance for correct retrieval. These two
performance measures are related to each other.

Memory capacity refers to the maximum number of associated pattern pairs

that can be stored and correctly retrieved.

Content-addressability is the ability of the network to retrieve the correct

stored pattern.

If input patterns are mutually orthogonal - perfect retrieval is possible.

If the stored input patterns are not mutually orthogonal - non-perfect

retrieval can happen due to crosstalk among the patterns.

13
7
2. Associative Memory Models

An associative memory is a system which stores mappings of specific input

representations to specific output representations.

– An associative memory "associates" two patterns such that when one is

encountered, the other can be reliably recalled.

– Most associative memory implementations are realized as

connectionist networks.

The simplest associative memory model is Linear associator, which is a feed-

forward type of network. It has very low memory capacity and therefore not
much used.

The popular models are Hopfield Model and Bi-directional Associative Memory
(BAM) model.

The Network Architecture of these models are presented in this section.

13

13
8
 SC - AM models

2.1 Associative Memory Models

The simplest and among the first studied associative memory models

is Linear associator. It is a feed-forward type of network where the

output is produced in a single feed-forward computation. It can be

used as an auto-associator as well as a hetero-associator, but it

possesses a very low memory capacity and therefore not much used.

The popular associative memory models are Hopfield Model and

Bi-directional Associative Memory (BAM) model.

– The Hopfield model is an auto-associative memory, proposed by

John Hopfield in 1982. It is an ensemble of simple processing units that
have a fairly complex collective computational abilities and
behavior. The Hopfield model computes its output recursively in
time until the system becomes stable. Hopfield networks are

designed using bipolar units and a learning procedure.

– The Bi-directional associative memory (BAM) model is similar to linear

associator, but the connections are bi-directional and therefore allows
forward and backward flow of information between the layers. The BAM
model can perform both auto-associative and hetero-associative recall
of stored information.

The network architecture of these three models are described in the next
few slides.

14

13
9
 SC - AM models

2.2 Network Architectures of AM

Models

The neural associative memory models follow different neural network

architectures to memorize information. The network architectures

are either single layer or two layers .

• The Linear associator model, is a feed forward type network, consists,

two layers of processing units, one serving as the input layer while the
other as the output layer.

• The Hopfield model, is a single layer of processing elements where

each unit is connected to every other unit in the network other than
itself.

• The Bi-directional associative memory (BAM) model is similar to

that of linear associator but the connections are bidirectional.

In this section, the neural network architectures of these models and the
construction of the corresponding connection weight matrix W of the
associative memory are illustrated.

15

14
0
 SC - AM models

3. Linear Associator Model (two layers)

It is a feed-forward type network where the output is produced in a

single feed-forward computation. The model consists of two layers

of processing units, one serving as the input layer while the other as

the output layer. The inputs are directly connected to the outputs,

via a series of weights. The links carrying weights connect every input to
every output. The sum of the products of the weights and the

inputs is calculated in each neuron node. The network architecture

of the linear associator is as shown below.

neurons
weights wij

w11
x1 y1
w21

w12

w22
x2 y2

w2m
inputs outputs

w1m
wn1

wn2

Xn wnm Ym

Fig. Linear associator model

14
1
– all n input units are connected to all m output units via connection

weight matrix W = [wij]n x m where wij denotes the strength of the

unidirectional connection from the i th input unit to the j th output unit.

– the connection weight matrix stores the p different associated

pattern pairs {(Xk, Yk) | k = 1, 2, ..., p} .

– building
an associative memory is constructing the connection
weight matrix W such that when an input pattern is presented,

then the stored pattern associated with the input pattern is retrieved.

14
2
– Encoding : The process of constructing the connection weight matrix is
called encoding. During encoding the weight values of correlation
matrix Wk for an associated pattern pair (Xk, Yk) are computed as:

(wij)k = (xi)k (yj)k where

(xi)k is the i th component of pattern Xk for i = 1, 2, ..., m, and

th

(yj)k is the j component of pattern Yk for j = 1, 2, ..., n.

– Weight matrix : Construction of weight matrix W is

accomplished
Σ
p Wk
by summing those individual correlation matrices Wk, ie, W = α
k=1

where usually
α is the constant of proportionality, for normalizing,
set to 1/p to store p different associated pattern pairs.

– Decoding : After memorization, the network can be used for retrieval;

the process of retrieving a stored pattern, is called decoding; given an

input pattern X, the decoding or retrieving is accomplished by
computing, first the net Input as input j = Σm xi w i j where
j=1

input j stands for the weighted sum of the input or activation value of
th
node j , for j = 1, 2, . . , n. and xi is the i component of pattern Xk , and
then determine the units Output using a bipolar output function:

+1 if input j ≥ θ j

Y j =

- 1 other wise

14
3
 where θ j is the threshold value of output neuron j .

Note: The output units behave like linear threshold units; that compute
a weighted sum of the input and produces a -1 or +1 depending
whether the weighted sum is below or above a certain threshold value.

– Performance : The input pattern may contain errors and noise, or an

incomplete version of some previously encoded pattern. When such
corrupt input pattern is presented, the network will retrieve the stored
pattern that is closest to actual input pattern. Therefore, the linear
associator is robust and fault tolerant. The presence of noise or error
results in a mere decrease rather than total degradation in the
performance of the network.

14
4
 SC - AM models

2 Auto-associative Memory Model - Hopfield model (single layer)

Auto-associative memory means patterns rather than associated pattern

pairs, are stored in memory. Hopfield model is one-layer

unidirectional auto-associative memory.

– the model consists, a single layer of processing elements where

each unit is connected to every other unit in the network but not to

itself.

– connection weight between or from neuron j to i is given by a

number wij. The collection of all such numbers are represented

by the weight matrix W which is square and symmetric, ie, w i j = w j i

for i, j = 1, 2,.........., m.

– each unit has an external input I which leads to a modification

in the computation of the net input to the units as

input j = Σ m xi w i
j + I j for j = 1, 2,......, m.

i=1

and x is the i th component of pattern X

i k

– each unit acts as both input and output unit. Like linear associator,

a single associated pattern pair is stored by computing the weight

T
matrix as Wk = X k Yk where X K= Y K

– Weight Matrix : Construction of weight matrix W is accomplished by

summing those individual correlation matrices, ie, W = Σ

α p Wk where

14
5
 k=1

α is the constant of proportionality, for normalizing, usually set to 1/p to

store p different associated pattern pairs. Since the Hopfield model

is an auto-associative memory model, it is the patterns

rather than associated pattern pairs, are stored in memory.

– Decoding : After memorization, the network can be used for retrieval; the
process of retrieving a stored pattern, is called decoding; given an

input pattern X, the decoding or retrieving mis accomplished by

computing, first the net Input as input j = Σ xi w i j where input j
j=1

stands for the weighted sum of the input or activation value of node j ,
th
for j = 1, 2, ..., n. and xi is the i component of pattern Xk , and then
determine the units Output using a bipolar output function:

+1 if input j ≥ θ j

Y j =

- 1 other wise

where θ j is the threshold value of output neuron j .

Note: The output units behave like linear threshold units; that compute a
weighted sum of the input and produces a -1 or +1 depending whether the
weighted sum is below or above a certain threshold value.

Decoding in the Hopfield model is achieved by a collective and recursive

relaxation search for a stored pattern given an initial stimulus pattern.
Given an input pattern X, decoding is accomplished by computing the net
input to the units and determining the output of those units using the
output function to produce the pattern X'. The pattern X' is then fed back
to the units as an input pattern to produce the pattern X''. The pattern X''
is again fed back to the units to produce the pattern X'''. The process is

14
6
 repeated until the network stabilizes on a stored pattern where further
computations do not change the output of the units.

In the next section, the working of an auto-correlator : how to store

patterns, recall a pattern from the stored patterns and how to recognize a
noisy pattern are explained.

14
7
■ Bidirectional Associative Memory (two-layer)

Kosko (1988) extended the Hopfield model, which is single layer, by

incorporating an additional layer to perform recurrent auto-associations

as well as hetero-associations on the stored memories. The network

structure of the bidirectional associative

memory model is similar to that of the linear associator but the

connections are bidirectional; i.e.,

wij = wji , for i = 1, 2, . . . , n and j = 1, 2, . . . , m.

neurons neurons
weights wij

w11
x1 y1
w21

w12

w22
x2 y2

w2m
inputs outputs
w1m
wn1

wn2

Xn wnm Ym

Fig. Bidirectional Associative Memory model

14
8
– In the bidirectional associative memory, a single associated pattern

T
pair is stored by computing the weight matrix as Wk = X k Yk .

– the construction of the connection weight matrix W, to store p

different associated pattern pairs simultaneously, is accomplished

by summing up the individual correlation matrices Wk ,

Σ
p W
i.e., W = α k

k=1

where α is the proportionality or normalizing constant.

14
9
 SC - AM – auto correlator

• Auto-associative Memory (auto-correlators)

In the previous section, the structure of the Hopfield model has been
explained. It is an auto-associative memory model which means patterns, rather
than associated pattern pairs, are stored in memory. In this section, the working
of an auto-associative memory (auto-correlator) is illustrated using some
examples.

Working of an auto-correlator :

– how to store the patterns,

–how to retrieve / recall a pattern from the stored patterns, −

how to recognize a noisy pattern and

21

15
0
 SC - AM – auto correlator

• How to Store Patterns : Example

Consider the three bipolar patterns A1 , A2, A3 to be stored as an auto-

correlator.

A1 = (-1, 1 , -1 , 1 )

A2 = ( 1, 1 , 1 , -1 )

A3 = (-1, -1 , -1 , 1 )

Note that the outer product of two vectors U and V is

U1 U1V1 U1V2 U1V3

T
U V = U V = U2 = U2V1 U2V2 U2V3
V1 V2 V3

U3 U3V1U3V2U3V3

U4 U4V1 U4V2 U4V3

Thus, the outer products of each of these three A1 , A2, A3 bipolar patterns
are

1 -1 1 -1
T
-1 1 -1 1
1 -1 1 -1
[A1 ] 4x1 [A1 ] 1x4 =

• -1 1 -1 1

15
1
 1 1 1 -1
T
1 1 1 -1
[A2 ] 4x1 [A2 ] 1x4 = 1 1 1 -1

e -1 -1 -1 1

1 1 1 -1
T
11 1 -1
[A3 ] 4x1 [A3 ] 1x4 =
11 1 -1

• -1 -1 -1 1

Therefore the connection matrix is j

3 1 3 -3
3 T
1 3 1 -1
T = [t ij ]= Σ [Ai ] 4x1 [Ai ] 1x4 3 1 3 -3

i=1 =
i -3 -1 -33

This is how the patterns are stored .

22

15
2
 SC - AM – auto correlator

• Retrieve a Pattern from the Stored Patterns (ref. previous slide)

The previous slide shows the connection matrix T of the three

bipolar patterns A1 , A2, A3 stored as

i

3 1 3 -3
3 T

1 3 1 -1
Σ [Ai ]
T = [t ij ]= 4x1 [Ai ] 1x4 = 3 1 3 -3

i=1
j -3 -1 -3 3

and one of the three stored pattern is A2 = ( 1,1 , 1 , -1 )

ai

– Retrieve or recall of this pattern A2 from the three stored patterns.

– The recall equation is

old
anewj = ƒ (ai t ij , aj ) for ∀ j=1 , 2 , . . . , p

A2 yields α = ∑ ai t ij
Computation for the recall and

equation then find β

i=

1 2 3 4 α β
α = ∑ ai t i , j=1
α = ∑ ai t i , j=2
α = ∑ ai t i , j=3
α = ∑ ai t i , j=4
1x3 + 1x1 + 1x3 + -1x-3 = 10 1

15
3
1x1 + 1x3 + 1x1 + -1x-1 = 6 1

15
4
1x3 + 1x1 + 1x3 + -1x-3 = 10 1

1x-3 + 1x-1 + 1x-3 + -1x3 = -1 -1

a new
Therefore = ƒ (a i t
old
) for ∀ j = 1 ,
j ij , aj

2,...,p is ƒ (α , β )

anew1
=ƒ (10 , 1)

anew2 =ƒ (6 , 1)

anew3 =ƒ (10 , 1)

new
a 4=ƒ (-1 , -1)

The values of β is the vector pattern ( 1, 1 , 1 , -1 ) which is A2 .

This is how to retrieve or recall a pattern from the stored patterns.

Similarly, retrieval of vector pattern A3 as

new new new new

(a1 , a2 ,a3 , a4 ,)
= ( -1, -1 , -1 , 1 ) = A3

23

15
5
 SC - AM – auto correlator

• Recognition of Noisy Patterns (ref. previous slide)

Consider a vector A' = ( 1, 1, 1,1) which is a noisy presentation

of one among the stored patterns.

– find the proximity of the noisy vector to the stored patterns

using Hamming distance measure.

– note that the Hamming distance (HD) of a vector X from Y, where

X = (x1 , x2 , . . . , xn) and Y = ( y1, y2 , . . . , yn) is given by HD

(x , y) = Σm | (xi - yi ) |
i=1

The HDs of A' from each of the stored patterns A1 , A2, A3 are

HD (A' , A1) = ∑ |(x1 - y1 )|, |(x2 - y2)|, |(x3 - y3 )|, |(x4 - y4 )|

= ∑ |(1 - (-1))|, |(1 - 1)|, |(1 - (-1) )|, |(1 - 1)|

=4

HD (A' , A2) =2

HD (A' , A3) =6

Therefore the vector A' is closest to A2 and so resembles it.

In other words the vector A' is a noisy version of vector A2.

Computation of recall equation using vector A' yields :

15
6
 i=

1 2 3 4 α β
α = ∑ ai t i , j=1
α = ∑ ai t i , j=2 1x3 + 1x1 + 1x3 + 1x-3 =4 1
α = ∑ ai t i , j=3
1x1 + 1x3 + 1x1 + 1x-1 =4 1
α = ∑ ai t i , j=4
1x3 + 1x1 + 1x3 + 1x-3 =4 1

1x-3 + 1x-1 + 1x-3 + 1x3 = -4 -1

new old
Therefore a j = ƒ (a i t ij ,aj ) for ∀ j = 1 , 2 , . . . , p is ƒ (α , β )

anew1
=ƒ (4 , 1)

anew2 =ƒ (4 , 1)

anew3 =ƒ (4 , 1)

new
a 4=ƒ (-4 , -1)

The values of β is the vector pattern ( 1, 1 , 1 , -1 ) which is A2 .

Note : In presence of noise or in case of partial representation of vectors,

an autocorrelator results in the refinement of the pattern or removal of

noise to retrieve the closest matching stored pattern.

24

15
7
 SC - Bidirectional hetero AM

4. Bidirectional Hetero-associative Memory

The Hopfield one-layer unidirectional auto-associators have been discussed

in previous section. Kosko (1987) extended this network to two-layer

bidirectional structure called Bidirectional Associative Memory (BAM) which

can achieve hetero-association. The important performance attributes of the

BAM is its ability to recall stored pairs particularly in the presence of noise.

Definition : If the associated pattern pairs (X, Y) are different and if the

model recalls a pattern Y given a pattern X or vice-versa, then it is

termed as hetero-associative memory.

This section illustrates the bidirectional associative memory :

• Operations (retrieval, addition and deletion) ,

• Energy Function (Kosko's correlation matrix, incorrect recall of pattern),

• Multiple training encoding strategy (Wang's generalized correlation matrix).

25

15
8
 SC - Bidirectional hetero AM

4.1 Bidirectional Associative Memory (BAM) Operations

BAM is a two-layer nonlinear neural network.

Denote one layer as field A with elements Ai and the other layer as field B
with elements Bi.

The basic coding procedure of the discrete BAM is as follows.

Consider N training pairs { (A1 , B1) , (A2 , B2), . . , (Ai , Bi), . . (AN , BN) }

where Ai = (ai1 , ai2 ,..., ain) and Bi = (bi1 , bi2 ,..., bip) and

aij , bij are either in ON or OFF state.

– in binary mode , ON = 1 and OFF = 0 and

in bipolar mode, ON = 1 and OFF = -1

M0 =
N T
– the original correlation matrix of the BAM Σ [ Xi ] [ Yi ]
is
i=1

where Xi = (xi1 , xi2 ,..., xin) and Yi = (yi1 , yi2 ,..., yip)

and xij(yij) is the bipolar form of aij(bij)

for the pair

The energy function (α , β ) and correlation matrix M is
E

T
E=-αMβ

With this background, the decoding processes, means the operations

to retrieve nearest pattern pairs, and the addition and deletion of

the pattern pairs are illustrated in the next few slides.

26

15
9
 SC - Bidirectional hetero AM

• Retrieve the Nearest of a Pattern Pair, given any pair

(ref : previous slide)

Example

Retrieve the nearest of (Ai , B i) pattern pair, given any pair (α , β ) .

The methods and the equations for retrieve are :

– start with an initial condition which is any given pattern pair (α , β ),

– determine a finite sequence of pattern pairs (α ' , β ' ) , (α " , β " ) .

...

until an equilibrium point (α f , β f ) is reached, where

β ' = Φ (α M ) and
M )
α'= Φ ( β

' M T

β " = Φ (α ' M ) and β )

''
α " =Φ (

Φ (F) = G = g1 , g2 , . . . . , gr ,

F = ( f1 , f2 ,........, fr )

Mis correlation matrix

1 if fi>0

0 (binary)

16
0
 gi = , f i <0

-1 (bipolar)

previous gi, f i =0

Kosko has proved that this process will converge for any

correlation matrix M.

27

16
1
 SC - Bidirectional hetero AM

■ Addition and Deletion of Pattern

Pairs

Given a set of pattern pairs (Xi , Yi) , for i = 1 , 2, . . . , n and a set

of correlation matrix M :

– a new pair (X' , Y') can be added or

– an existing pair (Xj , Yj) can be deleted from the memory model.

Addition : add a new pair (X' , Y') , to existing correlation matrix M ,

them the new correlation matrix Mnew is given by

T T T T

Mnew = X1 Y1 + X1 Y1 + . . . . + Xn Yn + X' Y'

Deletion : subtract the matrix corresponding to an existing pair (Xj , Yj) from

the correlation matrix M , them the new correlation matrix Mnew is given by

Xj Yj
Mnew = M -( )

Note : The addition and deletion of information is similar to the functioning

of the system as a human memory exhibiting learning and forgetfulness.

28

16
2
 SC - Bidirectional hetero AM

4.2 Energy Function for BAM

Note : A system that changes with time is a dynamic system. There are two types
of dynamics in a neural network. During training phase it iteratively update
weights and during production phase it asymptotically converges to the solution
patterns. State is a collection of qualitative and qualitative items that characterize
the system e.g., weights, data flows. The Energy function (or Lyapunov function)
is a bounded function of the system state that decreases with time and the
system solution is the minimum energy.

Let a pair (A , B) defines the state of a BAM.

– to store a pattern, the value of the energy function for that pattern

has to occupy a minimum point in the energy landscape.

– also adding a new patterns must not destroy the previously

stored patterns.

The stability of a BAM can be proved by identifying the energy function E

with each state (A , B) .

– For auto-associative memory : the energy function is

T
E(A) = - AM A

– For bidirecional hetero associative memory : the energy function is

E(A, B) = - AM BT ; for a particular case A = B , it corresponds to

Hopfield auto-associative function.

16
3
 We wish to retrieve the nearest of (Ai , Bi) pair, when any (α , β ) pair is
presented as initial condition to BAM. The neurons change

their states until a bidirectional stable state (Af , Bf) is reached. Kosko has
shown that such stable state is reached for any matrix M when it
corresponds to local minimum of the energy function. Each cycle of

decoding lowers the energy E if the energy function for any point

(
T
α , β ) is given by E=α Mβ
T

If the energy E = Ai M Bi evaluated using coordinates of the pair

(Ai , Bi) does not constitute a local minimum, then the point cannot

though one starts with

be recalled, α = Ai. Thus Kosko's encoding
even

method does not ensure that the stored pairs are at a local minimum.

29

16
4
 SC - Bidirectional hetero AM

■ Example : Kosko's BAM for Retrieval of Associated

Pair The working of Kosko's BAM for retrieval of associated
pair. Start with X3, and hope to retrieve the associated pair
Y3 .

Consider N = 3 pattern pairs (A1 , B1 ) , (A2 , B2 ) , (A3 , B3 ) given by

A1 = ( 1 0 0 0 0 1 ) B1 =( 1 1 0 0 0 )

A2 = ( 0 1 1 0 0 0 ) B2 =( 1 0 1 0 0 )

A3 = ( 0 0 1 0 1 1 ) B3 =( 0 1 1 1 0 )

Convert these three binary pattern to bipolar form replacing 0s by -1s.

X1 = ( 1 -1 -1 -1 -1 1 ) Y1 =( 1 1 -1 -1 -1 )

X2 = ( -1 1 1 -1 -1 -1 ) Y2 =( 1 -1 1 -1 -1 )

X3 = ( -1 -1 1 -1 1 1 ) Y3 =( -1 1 1 1 -1 )

The correlation matrix M is calculated as 6x5 matrix

1 1 -3 -1 1

1 -3 1 -1 1

T T T
-1 -1 3 1 -1
M = X1 Y1 + X2 Y2 + X3 Y3 =

-1 -1 -1 1 3

-3 1 1 3 1

-1 3 -1 1 -1
Suppose we start with
α = X3, and we hope to retrieve the associated pair

Y3 . The calculations for the retrieval of Y3 yield :

α M = ( -1 -1 1 -1 1 1 ) ( M ) = ( -6 6 6 6 -6 )

Φ (α M) = β ' = ( -1 1 1 1 -1 )

16
5
 T
β' M = ( -5 -5 5 -3 7 5 )

T
Φ (β ' M ) = ( -1 -1 1 -1 1 1 ) =α'

α'M = ( -1 -1 1 -1 1 1 ) M = ( -6 6 6 6 -6 )

(α ' M) = β " = ( -1 1 1 1 1 -1 )
Φ

=β'

This retrieved patern β

' is same as Y3 .

(α f , β f) =
Hence, (X3 , Y3 ) is correctly recalled, a desired result .

30

16
6
 SC - Bidirectional hetero AM

• Example : Incorrect Recall by Kosko's BAM

The Working of incorrect recall by Kosko's BAM.

Start with X2, and hope to retrieve the associated pair Y2 . Consider N
= 3 pattern pairs (A1 , B1 ) , (A2 , B2 ) , (A3 , B3 ) given by

A1 = ( 1 0 0 1 1 1 0 0 0 ) B1 = ( 1 1 1 0 0 0 0 1 0 )

A2 = ( 0 1 1 1 0 0 1 1 1 ) B2 = ( 1 0 0 0 0 0 0 0 1 )

A3 = ( 1 0 1 0 1 1 0 1 1 ) B3 = ( 0 1 0 1 0 0 1 0 1 )

Convert these three binary pattern to bipolar form replacing 0s by -1s.

X1 = ( 1 -1 -1 1 1 1 -1 -1 -1 ) Y1 = ( 1 1 1 -1 -1 -1 -1 1 -1 )

X2 = ( -1 1 1 1 -1 -1 1 1 1 ) Y2 = ( 1 -1 -1 -1 -1 -1 -1 -1 1 )

X3 = ( 1 -1 1 -1 1 1 -1 1 1 ) Y3 = ( -1 1 -1 1 -1 -1 1 0 1 )

The correlation matrix M is calculated as 9 x 9 matrix

T
T T

M = X1 Y1 + X2 Y2 + X3 Y3

-1 3 1 1 -1 -1 1 1 -1

1 -3 -1 -1 1 1 -1 -1 1

-1 -1 -3 1 -1 -1 1 -3 3

3 -1 1 -3 -1 -1 -3 1 -1

= -1 3 1 1 -1 -1 1 1 -1

-1 3 1 1 -1 -1 1 1 -1

1 -3 -1 -1 1 1 -1 -1 1

-1 -1 -3 1 -1 -1 1 -3 3

16
7
 -1-1 -31-1-11-33

(Continued in next slide)

31

16
8
 SC - Bidirectional hetero AM

[Continued from previous slide]

Let the pair (X2 , Y2 ) be recalled.

X2 = ( -1 1 1 1 -1 -1 1 1 1 ) Y2 = ( 1 -1 -1 -1 -1 -1 -1 -1 1 )

Start with α = X2, and hope to retrieve the associated pair Y2 .

The calculations for the retrieval of Y2 yield :

α M = ( 5 -19 -13 -5 1 1 -5 -13 13 )

Φ (α M) = ( 1 -1 -1 -1 1 1 -1 -1 1 ) = β'

T
β' M = ( -11 11 5 5 -11 -11 11 5 5 )

T
Φ (β ' M ) = ( -1 1 1 1 -1 -1 1 1 1 ) = α'

α'M = ( 5 -19 -13 -5 1 1 -5 -13 13 )

Φ (α ' M) = ( 1 -1 -1 -1 1 1 -1 -1 1 )= β"

=β'

Here β " = β ' . Hence the cycle terminates with

α F = α' = ( -1 1 1 1 -1 -1 1 1 1 ) = X2

β F = β' = ( 1 -1 -1 -1 1 1 -1 -1 1 ) ≠ Y2

But β ' is not Y2 . Thus the vector pair (X2 , Y2) is not recalled correctly
by Kosko's decoding process.

( Continued in next slide )

32

16
9
 SC - Bidirectional hetero AM

[Continued from previous slide]

Check with Energy Function : Compute the energy functions

for the coordinates of pair (X2 , Y2) , the energy E = - X2 M Y2 = -71

T
the coordinates of pair (α F , β F) , the energy EF = - α F M β F =
for -75

However, the coordinates of pair (X2 , Y2) is not at its local

minimum can be shown by evaluating the energy E at a point which

is "one Hamming distance" way from Y2 . To do this consider a point

'

Y2 = ( 1 -1 -1 -1 1 -1 -1 -1 1 )

where the fifth component -1 of Y2 has been changed to 1. Now

′ T

E = - X2 M Y2 = - 73

which is lower than E2 confirming the hypothesis that (X2 , Y2) is not

at its local minimum of E.

Note : The correlation matrix M used by Kosko does not guarantee

that the energy of a training pair is at its local minimum. Therefore , a

pair Pi can be recalled if and only if this pair is at a local minimum of

the energy surface.

33

17
0
 SC - Bidirectional hetero AM

4.3 Multiple Training Encoding

Strategy

Note : (Ref. example in previous section). Kosko extended the unidirectional

auto-associative to bidirectional associative processes, using correlation matrix

T
M=
Σ Xi
Yi computed from the pattern pairs. The system proceeds to

retrieve the nearest pair given any pair (α , β ), with the help of recall equations.
However, Kosko's encoding method does not ensure that the stored pairs are at
local minimum and hence, results in incorrect recall.

Wang and other's, introduced multiple training encoding strategy which

ensures the correct recall of pattern pairs. This encoding strategy is an

enhancement / generalization of Kosko's encoding strategy. The Wang's

generalized correlation matrix is M = Σ qi T

Xi Yi where qi is viewed

as pair weight for

T

Xi Y i as positive real numbers. It denotes the

minimum number of times for using a pattern pair (Xi , Yi) for training to

guarantee recall of that pair.

To recover a pair (Ai , Bi) using multiple training of order q, let us

augment or supplement matrix M with a matrix P defined as

P = (q – 1) Xi Yi where (Xi , Yi) are the bipolar form of (Ai , Bi).

The augmentation implies adding (q - 1) more pairs located at (Ai , Bi) to

the existing correlation matrix. As a result the energy E' can reduced to an
arbitrarily low value by a suitable choice of q. This also ensures that the

17
1
 energy at (Ai , Bi) does not exceed at points which are one Hamming
distance away from this location.

The new value of the energy function E evaluated at (Ai , Bi) then becomes

T T T

E' (Ai , Bi) = – Ai M Bi – (q – 1) Ai Xi Yi Bi

The next few slides explains the step-by-step implementation of

Multiple training encoding strategy for the recall of three pattern pairs

(X1 , Y1 ) , (X1 , Y1 ) , (X1 , Y1 ) using one and same augmentation matrix

M . Also an algorithm to summarize the complete process of multiple

training encoding is given.

34

17
2
 SC - Bidirectional hetero AM

■ Example : Multiple Training Encoding Strategy

The working of multiple training encoding strategy which ensures the

correct recall of pattern pairs.

Consider N = 3 pattern pairs (A1 , B1 ) , (A2 , B2 ) , (A3 , B3 ) given by

A1 = ( 1 0 0 1 1 1 0 0 0 ) B1 = ( 1 1 1 0 0 0 0 1 0 )

A2 = ( 0 1 1 1 0 0 1 1 1 ) B2 = ( 1 0 0 0 0 0 0 0 1 )

A3 = ( 1 0 1 0 1 1 0 1 1 ) B3 = ( 0 1 0 1 0 0 1 0 1 )

Convert these three binary pattern to bipolar form replacing

X1 = ( 1 -1 -1 1 1 1 -1 -1 -1 ) Y1 = ( 1 1 1 -1 -1

X2 = ( -1 1 1 1 -1 -1 1 1 1 ) Y2 = ( 1 -1 -1 -1 -1

X3 = ( 1 -1 1 -1 1 1 -1 1 1 ) Y3 = ( -1 1 -1 1 -1

Let the pair (X2 , Y2) be recalled.

X2 = ( -1 1 1 1 -1 -1 1 1 1 ) Y2 = ( 1 -1 -1 -1 -1

Choose q=2, so that P = X2

Y2 , the augmented correlation m
T T

become M = X1 Y1 +2X2 Y2 + X3 Y3
s

4 2 2 0 0 2 2 -2

2 -4 -2 -2 0 0 -2 -2 2

0 -2 -4 0 -2 -2 0 -4 4
=
4 -2 0 -4 -2 -2 -4 0 0

-2 4 2 2 0 0 2 2 -2

-2 4 2 2 0 0 2 2 -2

17
3
 2 -4 -2 -2 0 0 -2 -2 2

0 -2 -4 0 -2 -2 0 -4 4

0 -2 -4 0 -2 -2 0 -4 4

( Continued in next slide )

35

17
4
 SC - Bidirectional hetero AM

[Continued from previous slide]

give
Now α = X2, and see that the corresponding pattern pair β = Y2

is correctly recalled as shown below.

αM= ( 14 -28 -22 -14 -8 -8 -14 -22 22 )

Φ (α M) = ( 1 -1 -1 -1 -1 -1 -1 -1 1 ) = β'

β'M
T
= ( -16 16 18 18 -16 -16 16 18 18 )

T
Φ (β ' M ) = ( -1 1 1 1 -1 -1 1 1 1 ) = α'

α'M = ( 14 -28 -22 -14 -8 -8 -14 -22 23 )

β
(α ' M) = ( 1 -1 -1 -1 1 1 -1 -1 1 ) = "
Φ

β"
Here = β ' . Hence the cycle terminates with

α F = α' = ( -1 1 1 1 -1 -1 1 1 1 ) = X2

β F = β' = ( 1 -1 -1 -1 1 1 -1 -1 1 ) = Y2

Thus, (X2 , Y2 ) is correctly recalled, using augmented correlation

matrix M . But, it is not possible to recall (X1 , Y1) using the same

matrix M as shown in the next slide.

( Continued in next slide )

36

17
5
 SC - Bidirectional hetero AM

[Continued from previous slide]

Note : The previous slide showed that the pattern pair (X2 , Y2 ) is
correctly recalled, using augmented correlation matrix

T T T

M = X1 Y1 + 2 X2 Y2 + X3 Y3

but then the same matrix M can not recall correctly the other

pattern pair (X1 , Y1 ) as shown below.

X1 = ( 1 -1 -1 1 1 1 -1 -1 -1 ) Y1 = ( 1 1 1 -1 -1 -1 -1 1 -1 )

Let α = X1 and to retrieve the associated pair Y1 the calculation shows

α M = ( -6 24 22 6 4 4 6 22 -22 )

Φ (α M) = ( -1 1 1 1 1 1 1 1 -1 ) = β'

β'M
T
= ( 16 -16 -18 -18 16 16 -16 -18 -18 )

T
Φ (β ' M ) = ( 1 -1 -1 -1 1 1 -1 -1 -1 ) = α'

α'M
= ( -14 28 22 14 8 8 14 22 -22 )

β
Φ (α ' M) = ( -1 1 1 1 1 1 1 1 -1 ) = "

β"=
Here β ' . Hence the cycle terminates with

α F = α' = ( 1 -1 -1 -1 1 1 -1 -1 -1 ) = X1

β F = β' = ( -1 1 1 1 1 1 1 1 -1 ) ≠ Y1

Thus, the pattern pair (X1 , Y1 ) is not correctly recalled, using augmented
correlation matrix M.

17
6
 To tackle this problem, the correlation matrix M needs to be further
augmented by multiple training of (X1 , Y1 ) as shown in the next slide.

( Continued in next slide )

37

17
7
 SC - Bidirectional hetero AM

[Continued from previous slide]

The previous slide shows that pattern pair (X1 , Y1) cannot be recalled
under the same augmentation matrix M that is able to recall (X2 , Y2).

However, this problem can be solved by multiple training of (X1 , Y1)

which yields a further change in M to values by defining

T T T

M = 2 X1 Y1 + 2 X2 Y2 + X3 Y3

-1 5 3 1 -1 -1 1 3 -3

1 -5 -3 -1 1 1 -1 -3 3

-1 -3 -5 1 -1 -1 1 -5 5

5 -1 1 -5 -3 -3 -5 1 -1

= -1 5 3 1 -1 -1 1 3 -3

-1 5 3 1 -1 -1 1 3 -3

1 -5 -3 -1 1 1 -1 -3 3

-1 -3 -5 1 -1 -1 1 -5 5

-1 -3 -5 1 -1 -1 1 -5 5

Now observe in the next slide that all three pairs can be correctly recalled.

( Continued in next slide )

38

17
8
 SC - Bidirectional hetero AM

[ Continued from previous slide ]

Recall of pattern pair(X1 ,Y1 )

X1 = (1 -1 -1 1 1
1 -1 -1 -1 ) Y1 =( 1 1 1 -1 -1 -1 -1 1 -1 )

Let α = X1 and to retrieve the associated pair Y1 the calculation shows

α M = ( 3 33 31-3-5-5-3 31 -31)
(α
M) =
Φ ( 1 1 1 -1 -1 -1-1 1-1 ) =β '

( T
β ' M ) =( 13 -13 -19 23 13 13 -13 -19 -19)
(β
T
Φ 'M) =(1-1-1111-1-1-1)=α '

'M = ( 3 33 31-3-5-5-3 31 -31)

α
(α '
(111-1-1-1-11-1)=β
Φ M) = "

Here β " = β ' . Hence the cycle terminates with

α
F = α' =(1-1-1111-1-1-1 )= X1
β
F =β ' =(111-1-1-1-11-1 )= Y1

Thus, the pattern pair (X1 ,Y1 ) is correctly recalled

Recall of pattern pair (X2 , Y2 )

X2 = ( -1 1 1 1 -1 -1 1 1 1 ) Y2 = (1 -1 -1 -1 -1 -1 -1 -1 1 )

Let α = X2 and to retrieve the associated pair Y2 the calculation shows

α M = (7 -35 -29-7-1-1-7 -29 29)

(α
M) = -1 -1 -1 -1 -1 -1 -1 1 ) = β'
Φ ( 1

(T 15 17 19 -15 -15 15 17 17 )
β ' M) =( -15

(βT
Φ' M ) =(-1111-1-1111)=α '

α' M =(7 -35 -29-7-1-1-7 -29 29)

17
9
 (α '
ΦM) =(1-1-1-1-1-1-1-11)=β "

Here β " = β ' . Hence the cycle terminates with

α
F =α ' == X2
( -1 1 1 1 -1 -1 1 1 1 )

β
( 1 -1 -1 -1 -1 -1 -1 -1 1 ) = Y2
F =β ' =

Thus, the pattern pair (X2 ,Y2 ) is correctly recalled

Recall of pattern pair (X3 , Y3 )

X3 = (1 -1 1 -1 1 1 -1 1 1 ) Y3 = ( -1 1 -1 1 -1 -1 1 0 1 )

Let α = X3 and to retrieve the associated pair Y3 the calculation shows

α M = ( -13 17-1 13-5-5 13-11)

(α
M) =
Φ ( -1 1 -1 1 -1 -1 1 -1 1 ) =β '

( T
β ' M ) =( 11 -11 27 -63 11 11 -11 27 27)
(β
T
Φ 'M) =(1-11-111-111)=α '

'M = ( -13 17-1 13-5-5 13-11)

α
(α '
(-11-11-1-11-11)=β
Φ M) = "

Here β " = β ' . Hence the cycle terminates with

α
F= α '=( 1-11-111-111 )= X3

β
F = β '=(-11-11-1-1101 )= Y3

Thus, the pattern pair (X3 ,Y3 ) is correctly recalled

( Continued in next slide )

39

18
0
 SC - Bidirectional hetero AM

[Continued from previous slide]

Thus, the multiple training encoding strategy ensures the correct of a

pair for a suitable augmentation of M . The generalization of correlation recall
matrix, for the correct recall of all training pairs, is written
the

as

M= Σ T
qi Yi where qi 's are +ve real numbers.
Xi

i=1

This modified correlation matrix is called generalized correlation matrix.

Using one and same augmentation matrix M, it is possible to recall all the
training pattern pairs .

40

18
1
 SC - Bidirectional hetero AM

• Algorithm (for the Multiple training encoding strategy)

To summarize the complete process of multiple training encoding an

algorithm is given below.

Algorithm Mul_Tr_Encode ( N , Xi , Yi , qi ) where

■ : Number of stored patterns set

X, i

Yi
X=
: the bipolar pattern pairs

xi , x
Y=
( X , X ,...., X ) where Xi = ( xi )
1 2 N 1 2 ,... i n
■
:
( Y1 , Y2, . . . . , YN) where Yj = ( x j 1 , x j 2 , . . .x j n )

is the weight vector (q1 , q2 ,........., qN )

18
2
 Step 1 Initialize correlation matrix M to null matrix M ← [0]

Step 2 Compute the correlation matrix M as

i
For ← 1 to N

M
← M⊕ [ qi ∗ Transpose ( Xi ) ⊗ ( Xi ) end

(symbols ⊕ matrix addition, ⊗ matrix multiplication, and

∗ scalar multiplication)

Step 3 Read input bipolar pattern A

Step 4 Compute A_M where A_M ← A ⊗ M

Step 5 Apply threshold function to A_M to get B'

Φ

Φ ( A_M
ie B' ← )

where Φ is defined as Φ (F) = G = g1 , g2,........, gn

Step 6 Output B' is the associated pattern pair

end

Adaptive Resonance Theory (ART)

What is ART ?

f ART stands for "Adaptive Resonance Theory", invented by Stephen

Grossberg in 1976. ART represents a family of neural networks.

g ART encompasses a wide variety of neural networks.

The basic ART System is an unsupervised learning model.

18
3
 h The term "resonance" refers to resonant state of a neural network in

which a category prototype vector matches close enough to the current

input vector. ART matching leads to this resonant state, which permits
learning. The network learns only in its resonant state.

• ART neural networks are capable of developing stable clusters of arbitrary

sequences of input patterns by self-organizing.

ART-1 can cluster binary input vectors. ART-2

can cluster real-valued input vectors.

• ART systems are well suited to problems that require online learning of
large and evolving databases.

1. Adaptive Resonance Theory (ART)

Real world is faced with a situations where data is continuously changing.

In such situation, every learning system faces plasticity-stability dilemma.

This dilemma is about :

"A system that must be able to learn to adapt to a changing environment

(ie it must be plastic) but the constant change can make the system
unstable, because the system may learn new information only by
forgetting every thing it has so far learned."

This phenomenon, a contradiction between plasticity and stability, is called

plasticity - stability dilemma.

18
4
The back-propagation algorithm suffer from such stability problem.

• Adaptive Resonance Theory (ART) is a new type of neural network,

designed by Grossberg in 1976 to solve plasticity-stability dilemma.

• ART has a self regulating control structure that allows autonomous

recognition and learning.

• ART requires no supervisory control or algorithmic implementation.

• Recap ( Supervised , Unsupervised and BackProp Algorithms )

Neural networks learn through supervised and unsupervised means.

The hybrid approaches are becoming increasingly common as well.

• In supervised learning, the input and the expected output of the system
are provided, and the ANN is used to model the relationship between the
two. Given an input set x, and a corresponding output

set y, an optimal rule is determined such that: y = f(x) + e where, e is an

approximation error that needs to be minimized. Supervised learning is
useful when we want the network to reproduce the characteristics of a
certain relationship.

• In unsupervised learning, the data and a cost function are provided. The
ANN is trained to minimize the cost function by finding a suitable

input-output relationship. Given an input set x, and a cost function g(x, y)

of the input and output sets, the goal is to minimize the cost function
through a proper selection of f (the relationship between x, and y). At each

18
5
 training iteration, the trainer provides the input to the network, and the
network produces a result. This result is put into the cost function, and the
total cost is used to update the weights. Weights are continually updated
until the system output produces a minimal cost. Unsupervised learning is
useful in situations where a cost function is known, but a data set is not
know that minimizes that cost function over a particular input space.

• In backprop network learning, a set of input-output pairs are given and

the network is able to learn an appropriate mapping. Among the

supervised learning, BPN is most used and well known for its ability to
attack problems which we can not solve explicitly. However there are
several technical problems with back-propagation type algorithms.

They are not well suited for tasks where input space changes and are

often slow to learn, particularly with many hidden units. Also the

semantics of the algorithm are poorly understood and not biologically

plausible, restricting its usefulness as a model of neural learning.

Most learning in brains is completely unsupervised.

05

18
6
 SC – ART-Competitive learning

■ Competitive Learning Neural

Networks

While no information is available about desired outputs the network

updated weights only on the basis of input patterns. The Competitive
Learning network is unsupervised learning for categorizing inputs. The
neurons (or units) compete to fire for particular inputs and then learn to
respond better for the inputs that activated them by adjusting their
weights.

T
– For an output unit j , the input vector X = [x1 , x2 , x3 ] and the

T
weight vector Wj = [w1j , w1j , w1j ] are normalized to unit length.

– The activation value aj of the output unit j is calculated by the inner

product of the weight vectors

=
a j
Σ3 xi wij = XT Wj XT
= Wj

i=1

and then the output unit with the highest activation is selected for
further processing; this implied competitive.

– Assuming that output unit k has the maximal activation, the weights
leading to this unit are updated according to the competitive, called

winner-take-all (WTA) learning rule

wk (t) + η {x (t) + wk (t)}

wk (t + 1) =
||wk (t) + η {x (t) + wk (t)}||

18
7
 which is normalized to ensure that wk (t + 1) is always of unit length;
only the weights at the winner output unit k are updated and all other
weights remain unchanged.

– Alternately, Euclidean distance as a dissimilarity measure is a more

general scheme of competitive learning, in which the activation of
output unit j is as

3 2 1/2
aj = { Σ (xi - wij ) } = || xi - wij ||

i=1

the weights of the output units with the smallest activation are
updated according to

wk (t + 1) = wk (t) + η {x (t) + wk (t)}

A competitive network, on the input patterns, performs an on-line

clustering process and when complete the input data are divided into
disjoint clusters such that similarity between individuals in the same
cluster are larger than those in different clusters. Stated above, two
metrics of similarity measures: one is Inner product and the other
Euclidean distance. Other metrics of similarity measures can be used. The
selection of different metrics lead to different clustering.

18
8
 Limitations of Competitive Learning :

– Competitive learning lacks the capability to add new clusters when

deemed necessary.

– Competitive learning does not guarantee stability in forming clusters.

If the learning rate η is constant, then the winning unit that

responds to a pattern may continue changing during training.

■ If the learning rate η is decreasing with time, it may become too

small to update cluster centers when new data of different
probability are presented.

Carpenter and Grossberg (1998) referred such occurrence as the stability-

plasticity dilemma which is common in designing intelligent learning
systems. In general, a learning system should be plastic, or adaptive in
reacting to changing environments, and should be stable to preserve
knowledge acquired previously.

• Stability-Plasticity Dilemma (SPD)

Every learning system faces the plasticity-stability dilemma.

The plasticity-stability dilemma poses few questions :

– How can we continue to quickly learn new things about the

environment and yet not forgetting what we have already learned?

– How can a learning system remain plastic (adaptive) in response to

18
9
 significant input yet stable in response to irrelevant input?

– How can a neural network can remain plastic enough to learn new
patterns and yet be able to maintain the stability of the already learned
patterns?

– How does the system know to switch between its plastic and stable
modes.

– What is the method by which the system can retain previously learned
information while learning new things.

Answer to these questions, about plasticity-stability dilemma in learning

systems is the Grossberg’s Adaptive Resonance Theory (ART).

– ART has been developed to avoid stability-plasticity dilemma in

competitive networks learning.

– The stability-plasticity dilemma addresses how a learning system can

preserve its previously learned knowledge while keeping its ability to
learn new patterns.

– ART is a family of different neural architectures. ART architecture can

self-organize in real time producing stable recognition while getting
input patterns beyond those originally stored.

19
0
 SC - ART networks

2. Adaptive Resonance Theory (ART) Networks

An adaptive clustering technique was developed by Carpenter and Grossberg in

1987 and is called the Adaptive Resonance Theory (ART) .

The Adaptive Resonance Theory (ART) networks are self-organizing competitive

neural network. ART includes a wide variety of neural networks. ART networks
follow both supervised and unsupervised algorithms.

– The unsupervised ARTs named as ART1, ART2 , ART3, . . and are similar

to many iterative clustering algorithms where the terms "nearest" and

"closer" are modified by introducing the concept of "resonance".

Resonance is just a matter of being within a certain threshold of a second

similarity measure.

The supervised ART algorithms that are named with the suffix "MAP", as

ARTMAP. Here the algorithms cluster both the inputs and targets and

associate two sets of clusters.

19
1
The basic ART system is unsupervised learning model. It typically consists of
– a comparison field and a recognition field composed of neurons, −
a vigilance parameter, and

– a reset module

Each of these are explained in the next slide.

Reset F2
Recognition Field

F2 layer

• •• •

Z W

Reset Module
Vigilance

Comparison •• • Parameter
Field •

F1 layer ρ

Normalized Input

Fig Basic ART Structure

19
2
10

19
3
 SC - ART networks

• Comparison field

The comparison field takes an input vector (a one-dimensional array of

values) and transfers it to its best match in the recognition field. Its best
match is the single neuron whose set of weights

(weight vector) most closely matches the input vector.

= Recognition field

Each recognition field neuron, outputs a negative signal proportional to

that neuron's quality of match to the input vector to each of the other
recognition field neurons and inhibits their output accordingly. In this way
the recognition field exhibits lateral inhibition, allowing each neuron in it to
represent a category to which input vectors are classified.

= Vigilance parameter

After the input vector is classified, a reset module compares the

strength of the recognition match to a vigilance parameter. The vigilance

parameter has considerable influence on the system:

– Higher vigilance produces highly detailed memories (many, fine-grained

categories), and

– Lower vigilance results in more general memories (fewer, more-general

categories).

11

19
4
 SC - ART networks

• Reset Module

The reset module compares the strength of the recognition match to the
vigilance parameter.

– If the vigilance threshold is met, then training commences.

– Otherwise, if the match level does not meet the vigilance

parameter, then the firing recognition neuron is inhibited until a new

input vector is applied;

Training commences only upon completion of a search procedure.

In the search procedure, the recognition neurons are disabled one by one
by the reset function until the vigilance parameter is satisfied by a
recognition match.

If no committed recognition neuron's match meets the vigilance threshold,

then an uncommitted neuron is committed and adjusted towards matching
the input vector.

12

19
5
 SC - ART networks

2.1 Simple ART Network

ART includes a wide variety of neural networks. ART networks follow both
supervised and unsupervised algorithms. The unsupervised ARTs as ART1,
ART2, ART3, . . . . are similar to many iterative clustering algorithms.

The simplest ART network is a vector classifier. It accepts as input a

vector and classifies it into a category depending on the stored pattern it
most closely resembles. Once a pattern is found, it is modified (trained) to
resemble the input vector. If the input vector does not match any stored
pattern within a certain tolerance, then a new category is created by
storing a new pattern similar to the input vector. Consequently, no stored
pattern is ever modified unless it matches the input vector within a certain
tolerance.

This means that an ART network has

– both plasticity and stability;

– new categories can be formed when the environment does not

match any of the stored patterns, and

– the environment cannot change stored patterns unless they

are sufficiently similar.

13

19
6
 SC - ART networks

2.2 General ART Architecture

The general structure, of an ART network is shown below.

Reset F2
Recognition Field

F2 layer, STM

• •• •

New

cluster LTM

Adaptive Filter

path
Rese
t

Expectation
Modu

Vigilance

Comparison •• • Parameter
Field •

– Bottom-up weights

F1 layer, STM ρ

Normalized Input

19
7
 Fig Simplified ART Architecture

There are two layers of neurons and a reset mechanism.

– F1 layer : an input processing field; also called comparison layer.

– F2 layer : the cluster units ; also called competitive layer.

– Reset mechanism : to control the degree of similarity of patterns placed

on the same cluster; takes decision whether or not to allow cluster unit to
learn.

There are two sets of connections, each with their own weights, called :

from each unit of layer F1 to all units of layer F2 .

– Top-down weights from each unit of layer F2 to all units of layer F1 .

14

19
8
 SC - ART networks

2.3 Important ART Networks

The ART comes in several varieties. They belong to both unsupervised

and supervised form of learning.

Unsupervised ARTs are named as ART1, ART2 , ART3, . . and are

similar to many iterative clustering algorithms.

• ART1 model (1987) designed to cluster binary input patterns.

• ART2 model (1987) developed to cluster continuous input patterns.

• ART3 model (1990) is the refinement of these two models.

Supervised ARTs are named with the suffix "MAP", as ARTMAP, that
combines two slightly modified ART-1 or ART-2 units to form a supervised
learning model where the first unit takes the input data and the second
unit takes the correct output data. The algorithms cluster both the inputs
and targets, and associate the two sets of clusters. Fuzzy ART and Fuzzy
ARTMAP are generalization using fuzzy logic.

A taxonomy of important ART networks are shown below.

ART Networks
Unsupervised ART Learning Supervised ART
Grossberg, 1976
Learning

19
9
 ART1 , ART2 Fuzzy ART ARTMAP FuzzyGaussian

ARTMAP ARTMAP

Carpenter & Carpenter & Carpenter & Carpenter & Williamson,

Grossberg,
Grossberg, Grossberg, Grossberg, 1992

etal 1987
Simplified ARTMAP
1987 etal 1987 etal 1987
Baraldi & Alpaydin, 1998

Simplified ART
Baraldi &
Alpaydin, 1998

Fig. Important ART Networks

Note : Only the unsupervised ARTs are presented in what follows in

the remaining slides.

15

20
0
 SC - ART networks

2.4 Unsupervised ARTs – Discovering Cluster Structure

Human has ability to learn through classification. Human learn new

concepts by relating them to existing knowledge and if unable to relate to
something already known, then creates a new structure. The unsupervised
ARTs named as ART1, ART2 , ART3, . . represent such human like learning
ability.

ART is similar to many iterative clustering algorithms where each pattern

is processed by

■ Finding the "nearest cluster" seed/prototype/template to that pattern

and then updating that cluster to be "closer" to the pattern;

■ Here the measures "nearest" and "closer" can be defined in different

ways in n-dimensional Euclidean space or an n-space.

How ART is different from most other clustering algorithms is that it is

capable of determining number of clusters through adaptation.

■ ART allows a training example to modify an existing cluster only if the

cluster is sufficiently close to the example (the cluster is said to
"resonate" with the example); otherwise a new cluster is formed to
handle the example

■ To determine when a new cluster should be formed, ART uses a vigilance

parameter as a threshold of similarity between patterns and clusters.

ART networks can "discover" structure in the data by finding how the data
is clustered. The ART networks are capable of developing stable clusters of
arbitrary sequences of input patterns by self-organization.

20
1
 Note : For better understanding, in the subsequent sections, first the
iterative clustering algorithm (a non-neural approach) is presented then
the ART1 and ART2 neural networks are presented.

16

20
2
 SC - Iterative clustering

• Iterative Clustering - Non Neural Approach

Organizing data into sensible groupings is one of the most fundamental mode
of understanding and learning.

Clustering is a way to form 'natural groupings' or clusters of patterns.

Clustering is often called an unsupervised learning.

— cluster analysis is the study of algorithms and methods for grouping, or

clustering, objects according to measured or perceived intrinsic characteristics
or similarity.

- Cluster analysis does not use category labels that tag objects with prior
identifiers, i.e., class labels.

- The absence of category information, distinguishes the clustering

(unsupervised learning) from the classification or discriminant analysis
(supervised learning).

- The aim of clustering is exploratory in nature to find structure in data.

17

20
3
 SC - Iterative clustering

- Example :

Three natural groups of data points, that is three natural clusters.

•••

••• •••

••• •••

••

■■

■■■■

■■■

In clustering, the task is to learn a classification from the data represented

in an n-dimensional Euclidean space or an n-space.

• the data set is explored to find some intrinsic structures in them;

• no predefined classification of patterns are required;

The K-mean, ISODATA and Vector Quantization techniques are some of the
decision theoretic approaches for cluster formation among unsupervised
learning algorithms.

20
4
 (Note : a recap of distance function in n-space is first mentioned and then vector
quantization clustering is illustrated.)

18

20
5
 SC - Recap distance functions

■ Recap : Distance Functions

Vector Space Operations

Let R denote the field of real numbers.

For any non-negative integer n, the space of all n-tuples of real

numbers forms an n-dimensional vector space over R, denotes Rn.

An element of Rn is written as X = (x1, x2, …xi…., xn), where xi is

a real number. Similarly the other element Y = (y1, y2, …yi…., yn)

n
The vector space operations on R are defined by

X + Y = (x1 + y1, X2 + y2, . . , xn + yn) and

aX = (ax1, ax2, . . , axn)

The standard inner product, called dot product, on Rn, is given by

n
X•Y= ∑ i=1 (x1 y1 + x2 y2 + . . . . + xn yn) is a real number.

n
The dot product defines a distance function (or metric) on R by

n 2
d(X , Y) = ||X – Y|| = ∑ (xi – yi)

i=1

The (interior) angle θ between x and y is then given by

X•Y

-1
θ = cos ( )
||X|| ||Y||

The dot product of X with itself is always non negative, is given by

20
6
 n 2
||X || = ∑ i (xi - xi)

[Continued in next slide]

19

20
7
 SC – Recap distance functions

■ Euclidean Distance

It is also known as Euclidean metric, is the "ordinary" distance

between two points that one would measure with a ruler.

The Euclidean distance between two points

P = (p1 , p2 , . . pi . . , xn) and

Q = (q1 , q2 , . . qi . . , qn)

in Euclidean n-space , is defined as :

(p1 – q1)2 + (p2 – q2)2 + . . + (pn – qn)2

n
■ ∑ i=1 (pi - qi)2

Example : Three-dimensional distance

For two 3D points,

P = (px, py, . . pz) and

Q = (qx, qy, . . qz)

The Euclidean 3-space , is computed as :

20
8
 2 2 2
(px – qx) + (py – qy) + (pz – qz)

20

20
9
 SC - Vector quantization

3.1 Vector Quantization Clustering

The goal is to "discover" structure in the data by finding how the data is
clustered. One method for doing this is called vector quantization for
grouping feature vectors into clusters.

The Vector Quantization (VQ) is non-neural approach to dynamic

allocation of cluster centers.

VQ is a non-neural approach for grouping feature vectors into clusters.

It works by dividing a large set of points (vectors) into groups having

approximately the same number of points closest to them. Each group is
represented by its centroid point, as in k-means and some other clustering
algorithms.

• Algorithm for vector quantization

– To begin with, in VQ no cluster has been allocated; first pattern would

hold itself as the cluster center.

p
– When ever a new input vector X as pth pattern appears, the Euclidean
distance d between it and the jth cluster center C j is calculated as

1/2

p
p N
d= |X –Cj|= Σ ( X i– CJi )2

i=1

– The cluster closest to the input is determined as

21
0
 j = 1, . . , M

p p
|X –C k |<|X –C j |= minimum

j≠k

where M is the number of allotted clusters.

p
– Once the closest cluster k is determined, the distance | X –C k |
must be tested for the threshold distance ρ as

p th
◊ |X –C k |<ρ pattern assigned to k cluster

p
2. |X –C k |>ρ a new cluster is allocated to pattern p

– update that cluster centre where the current pattern is assigned

C x = (1/Nx ) Σ X

x ∈ Sn

where set X represents all patterns coordinates (x , y) allocated to

that cluster (ie Sn) and N is number of such patterns.

21

21
1
 SC - Vector quantization

◊ Example 1 : ( Ref previous slide)

Consider 12 numbers of pattern points in Euclidean space.

Their coordinates (X , Y) are indicated in the table below.

Table Input pattern - coordinates of 12 points

Points X Y Points X Y
1 2 3 7 6 4

2 3 3 8 7 4

3 2 6 9 2 4

4 3 6 10 3 4

5 6 3 11 2 7

6 7 3 12 3 7

Determine clusters using VQ, assuming the threshold distance = 2.0.

– Take a new pattern, find its distances from all the clusters identified,

– Compare distances w.r.t the threshold distance and accordingly

decide cluster allocation to this pattern,

– Update the cluster center to which this new pattern is allocated,

– Repeat for next pattern.

Computations to form clusters

Determining cluster closest to input pattern Cluster no

Input Cluster 1 Cluster 2 Cluster 3 assigned to

Pattern Distance center Distance center Distance center i/p pattern

1, (2,3) 0 (2 , 3) 1

2, (3,3) 1 (2.5 , 3) 1

21
2
 3, (2,6) 3.041381 0 (2 , 6) 2

4, (3,6) 3.041381 1 (2.5 , 6) 2

5, (6,3) 4.5 5.408326 0 (6 , 3) 3

6, (7,3) 5.5 6.264982 1 (6.5 , 3) 3

7, (6,4) 3.640054 4.031128 1.118033 (6.333333, 3

3.3333)

8, (7,4) 4.609772 4.924428 0.942809 (6.5, 3.5) 3

9, (2,4) 1.11803 (2.33333, 2.06155 4.527692 1

3.33333)

10, (3,4) 0.942808 (2.5, 3.5) 2.0615528 3.5355339 1

11, (2,7) 3.5355339 1.1180339 (2.333333, 7.2629195 2

6.333333)

12, (3,7) 3.5707142 0.9428089 (2.5, 6.5) 4.9497474 2

The computations illustrated in the above table indicates :

– No of clusters 3

– Cluster centers C1 = (2.5, 3.5) ; C2 = (2.5, 6.5); C3 = ( 6.5, 3.5).

– Clusters Membership S(1) = {P1, P2, P9, P10}; S(2) = {P3, P4, P11, P12};

S(3) = {P5, P6, P7, P8};

These results are graphically represented in the next slide

22

21
3
 SC - Vector quantization

[Continued from previous slide]

Graphical Representation of Clustering

(Ref - Example -1 in previous slide )

Y Results of vector quantization :

Clusters formed

7 ■ ■ C2

– Number of input patterns : 12

6 ■ ■ C3

– Threshold distance assumed : 2.0

5

– No of clusters : 3
4
■ ■ ■ ■

3 – Cluster centers :
■ ■ ■ ■

2 C1 = (2.5, 3.5) ;
C1

1 X C2 = (2.5, 6.5);

0 C3 = ( 6.5, 3.5).

0123456 78 – Clusters Membership :

Fig (a) Input pattern for VQ ,

S(1)= {P1, P2, P9, P10};
Threshold distance =2.0

S(2) = {P3, P4, P11, P12};

Fig Clusters formed

S(1) = {P5, P6, P7, P8};

21
4
 Note : About threshold distance

– large threshold distance may obscure meaningful categories.

– low threshold distance may increase more meaningful categories.

– See next slide, clusters for threshold distances as 3.5 and 4.5 .

23

21
5
 SC - Vector quantization

- Example 2

The input patterns are same as of Example 1.

Determine the clusters, assuming the threshold distance = 3.5 and

4.5. − follow the same procedure as of Example 1 ;

– do computations to form clusters, assuming

the threshold distances as 3.5 and 4.5.

– The results are shown below.

Y
Y
8
8
7 C1
■ ■ ■ ■
7
6 C1
■ ■ C2

5 6 ■ ■

4 5

■ ■ ■ ■ 4

3 ■ ■ ■■
■ ■ ■ ■
3 ■ ■ ■■
2
2
1
X 1 X
0

0
0 1 2 3 4 5 6 7 8
012 345678

Fig (b) Input pattern for VQ ,

Fig (c) Input pattern for VQ ,
Threshold distance = 3.5
Threshold distance = 4.5

21
6
 Fig Clusters formed

– Fig (b) for the threshold distance = 3.5 , two clusters formed.

– Fig (c) for the threshold distance = 4.5 , one cluster formed.

24

21
7
 SC - ART Clustering

□ Unsupervised ART Clustering

The taxonomy of important ART networks, the basic ART structure, and the
general ART architecture have been explained in the previous slides. Here only
Unsupervised ART (ART1 and ART2) Clustering are presented.

ART1 is a clustering algorithm can learn and recognize binary patterns. Here

– similar data are grouped into cluster

– reorganizes clusters based upon the changes

– creates new cluster when different data is encountered

ART2 is similar
to can learn and recognize arbitrary sequences
ART1,

of analog input
patterns.

The ART1 architecture, the model description, the pattern matching cycle, and
the algorithm - clustering procedure, and a numerical example is presented in
this section.

25

21
8
 SC - ART1 architecture

4.1 ART1 Architecture

The Architecture of ART1 neural network consist of two layers of neurons.

Attentional sub system Orienting sub

system

Recognition layer F2 - STM

+
Gain +
+ 2 m - Neuron
G2

Top-DnR

wij

weights +

LTM •+

C Bottom-up

vji weights

— ρ
Gain + Comparison layer F1 - STM Reset

+ 1 G1 n - Neuron

Vigilance

parameter
+

Pattern Vector (Pi = 0 or 1)

IH=1 [1 1 0 0 Pi 0]

------------------

IH=h [1 0 0 1 Pi 0]

Fig. ART1 Network architecture

21
9
 ATR1 model consists an "Attentional" and an "Orienting" subsystem.

The Attentional sub-system consists of :

– two competitive networks, as Comparison layer F1 and Recognition

layer F2, fully connected with top-down and bottom-up weights;

– two control gains, as Gain1 and Gain2.

– Reset layer for controlling the attentional sub-system overall dynamics

based on vigilance parameter.

– Vigilance parameter ρ determines the degree of mismatch to be

tolerated between the input pattern vectors and the weights

connecting F1 and F2.

The nodes at F2 represent the clusters formed. Once the network

stabilizes, the top-down weights corresponding to each node in F2
represent the prototype vector for that node.

26

22
0
 SC - ART1 Model description

4.2 ART1 Model Description

The ART1 system consists of two major subsystem, an attentional

subsystem and an orienting subsystem, described below. The system does
pattern matching operation during which the network structure tries to
determine whether the input pattern is among the patterns previously
stored in the network or not.

• Attentional Subsystem

(a) F1 layer of neurons/nodes called or input layer or comparison layer;

short term memory (STM).

(b) F2 layer of neurons/nodes called or output layer or recognition layer;

short term memory (STM).

(c) Gain control unit , Gain1 and Gain2, one for each layer.

(d) Bottom-up connections from F1 to F2 layer ;

traces of long term memory (LTM).

(e) Top-down connections from F2 to F1 layer;

traces of long term memory (LTM).

(f) Interconnections among the nodes in each layer are not shown.

(g) Inhibitory connection (-ve weights) from F2 layer to gain control.

(h) Excitatory connection (+ve weights) from gain control to F1 and F2.

• Orienting Subsystem

22
1
 (h) Reset layer for controlling the attentional subsystem overall dynamics.

(i) Inhibitory connection (-ve weights) from F1 layer to Reset node.

(j) Excitatory connection (+ve weights) from Reset node to F2 layer

27

22
2
 SC - ART1 Model description

• Comparison F1 and Recognition F2

layers

The comparison layer F1 receives the binary external input, then F1 passes
the external input to recognition layer F2 for matching it to a

classification category. The result is passed back to F1 to find :

If the category matches to that of input, then

— If Yes (match) then a new input vector is read and the cycle starts
again

— If No (mismatch) then the orienting system inhibits the previous

category to get a new category match in F2 layer.

The two gains, control the activities of F1 and F2 layer, respectively.

Processing element x1i in layer F1 Processing element x2i in layer F2

To other nodes

To F2 in F2 (WTA)

From F2

To orient

vji
From From

orient
Gain2 Unit
x2j

G1 Unit x1i G2 in F2

22
3
 in F1
From

Gain1

wij

From F1
To all F1
Ii
and G1

1. A processing element X1i in F1

receives input from three sources: 1. A processing element X2j in F2
receives input from three sources:
(a) External input vector Ii,

(a) Orienting sub-system,

(b) Gain control signal G1
(b) Gain control signal G2
(c) Internal network input vji made
of the output from F2 multiplied (c) Internal network input wij made
appropriate connections weights. of the output from F1
multiplied appropriate
2. There is no inhibitory input to connections weights.
the neuron
2. There is no inhibitory input to
3. The output of the neuron is fed
the neuron.
to the F2 layer as well as the
orienting sub-system. 3. The output of the neuron is fed to

the F1 layer as well as G1 control.

28

'

22
4
 SC - ART1 Pattern matching

4.3 ART1 Pattern Matching

Cycle

The ART network structure does pattern matching and tries to determine
whether an input pattern is among the patterns previously stored in the
network or not.

Pattern matching consists of : Input pattern presentation, Pattern

matching attempts, Reset operations, and the Final recognition. The step-
by-step pattern matching operations are described below.

• Fig (a) show the input pattern

presentation. The sequence of effects are
:

► Input pattern I presented to

F2 =Y

the units in F1 layer. A pattern

of activation X is produced

G1
across F1.

► Same input pattern I also

―

1 0 1 0=S
excites the orientation sub-

=X
F1 system A and gain control G1.

+
► Output pattern S (which is

22
5
 inhibitory signal) is sent to A. It

1 0 1 0 =I cancels the excitatory effect of

Fig (a) Input Pattern

signal I so that A remains

inactive.

► Gain control G1 sends an excitatory signal to F1. The same signal is

applied to each node in F1 layer. It is known as nonspecific signal.

► Appearance of X on F1 results an output pattern S. It is sent through

connections to F2 which receives entire output vector S.

► Net values calculated in the F2 units, as the sum the product of the
input values and the connection weights.

► Thus, in response to inputs from F1, a pattern of activity Y develops

across the nodes of F2 which is a competitive layer that performs a
contrast enhancement on the input signal.

29

22
6
 SC - ART1 Pattern matching

• Fig (b) show the Pattern Matching Attempts.

The sequence of operations are :

► Pattern of activity Y results an

F2 =Y

output U from F2 which is an

0 0 1 0=U
inhibitory signal sent to G1. If it

receives any inhibitory signal

G1

from F2, it ceases activity.

―

► Output U becomes second

1 0 0 0 = S*

V= 0 0 0 A
1 = X* input pattern for F1 units. Output
F1

+ U is transformed to pattern V, by

LTM traces on the top-down

connections from F2 to F1.

1 0 1 0 =I

Fig (b) Pattern matching

► Activities that develop over the nodes in F1 or F2 layers are the STM
traces not shown in the fig.

22
7
 • The 2/3 Rule

► Among the three possible sources of input to F1 or F2, only two are
used at a time. The units on F1 and F2 can become active only if two out of
the possible three sources of input are active. This feature is called the 2/3
rule.

► Due to the 2/3 rule, only those F1 nodes receiving signals from both

I and V will remain active. So the pattern that remains on F1 is I ∩ V.

► The Fig shows patterns mismatch and a new activity pattern X*

develops on F1. As the new output pattern S* is different from the original S
, the inhibitory signal to A no longer cancels the excitation coming from
input pattern I.

30

22
8
 SC - ART1 Pattern matching

• Fig (c) show the Reset Operations.

The sequence of effects are :

► Orientation sub-system A
F2 =Y

becomes active due to

mismatch of patterns on F1.

► Sub-system A sends a non-

G1

specific reset signal to all nodes

―

on F2.

F1
►
Nodes on F2 responds

+
according to their present

state. If nodes are inactive,

nodes do
1 0 1 0 =I

not respond; If nodes are

Fig (c) Reset active, nodes become inactive

and remain for an extended period of time. This sustained inhibition

prevents the same node winning the competition during the next cycle.

► Since output Y no longer appears, the top-down output and the

inhibitory signal to G1 also disappears.

31

22
9
 SC - ART1 Pattern matching

• Fig (d) show the final Recognition.

The sequence of operations are :

►
The original pattern X is

F2 = Y*

reinstated on F1 and a new cycle

of pattern matching begins.

Thus

a new pattern Y* appears on F2.

G1

— ► The nodes participating in the

1 0 1 0 =S
original pattern Y remains

F1 =X inactive due to long term effects

+
of the reset signal from A.

► This cycle of pattern matching

will continue until a match is

1 0 1 0 =I

Fig (d) Final

found, or until F2 runs out of

previously stored patterns. If no

match is found, the network will

assign some uncommitted node or nodes on F2 and will begin to learn the
new pattern. Learning takes through LTM traces (modification of weights).
This learning process does not start or stop but continue while the pattern

23
0
 matching process takes place. When ever signals are sent over
connections, the weights associated with those connections are subject to
modification.

► The mismatches do not result in loss of knowledge or learning of

incorrect association because the time required for significant changes in

weights is very large compared to the time required for a complete
matching cycle. The connection participating in mismatch are not
active long enough to effect the associated weights seriously.

► When a match occurs, there is no reset signal and the network settles
down into a resonate state. During this stable state, connections remain
active for sufficiently long time so that the weights are strengthened. This
resonant state can arise only when a pattern match occurs or during
enlisting of new units on F2 to store an unknown pattern.

32

23
1
 SC - ART1 Algorithm

4.4 ART1 Algorithm - Clustering Procedure

(Ref: Fig ART1 Architecture, Model and Pattern matching explained before)

T
Example t = 4 , x(4) = {1 0 0} • Notations

■ I(X) is input data set ( ) of the form I(X) = { x(1), x(2), . . , x(t) } where
t represents time or number of vectors. Each x(t) has n elements;

th
is the 4 vector that has 3 elements .

■ W(t) = (wij (t)) is the Bottom-up weight matrix of type n x m where

i = 1, n ; j = 1, m ; and its each column is a column vector of the form

T
wj (t) = [(w1j (t) . . . . wij (t) . . . wnj (t)] , T is transpose; Example :

Each column is a column vectors of the form

Wj=1 Wj=2

W11 W12

W(t) = (wij (t)) = W21 W22 W31 W32

V(t) = (vji (t)) is the Top-down weight matrix of type m x n where j

= 1, m; i = 1, n ; and its each line is a column vector of the form

T
vj (t) = [(vj1 (t) . . . . vji (t) . . . vjn (t)] , T is transpose; Example :

Each line is a column vector of the form

vj=1 vj=2

23
2
 v11 v12 v13

v(t) = (vji (t)) =

v21 v22 v23

■ For any two vectors u and v belong to the same vector space R, say

T
u, v ∈ R the notation < u , v > = u · v = u · v is scalar product; and

T
u X v = (u1 v1 , . . . ui vi . . un vn ) ∈ R , is piecewise product , that is

component by component.

n
■ The u ∧ v ∈ R means component wise minimum, that is the minimum on
each pair of components min { ui ; vi } , i = 1, n ;

■ The 1-norm of vector u is ||u||1 = ||u|| = Σn | ui |

i=1

■ The vigilance parameter is real value ρ ∈ (0 , 1),

The learning rate is real value α ∈ (0 , 1),

33

23
3
 SC - ART1 Algorithm

• Step-by-Step Clustering Procedure

Input: Feature vectors

ƒ Feature vectors IH=1 to h , each representing input pattern to layer F1 .

ƒ Vigilance parameter ρ ; select value between 0.3 and 0.5.

Assign values to control gains G1 and G2

1 if input IH ≠ 0 and output from F2 layer = 0

G1 =

0 otherwise

1 if input IH ≠ 0

G2 =

0 otherwise

Output: Clusters grouped according to the similarity is determined

ρ
by . Each neuron at the output layer represents a cluster, and the

top-down (or backward) weights represents temp plates or prototype

of the cluster.

34

23
4
 SC - ART1 Algorithm

Step - 1 (Initialization)

■ Initially, no input vector I is applied, making the control gains,

G1 = 0, G2 = 0. Set nodes in F1 layer and F2 layer to zero.

■ Initialize bottom-up wij (t) and top-down vji (t) weights for time t.
Weight wij is from neuron i in F1 layer to neuron j in F2 layer; where i
= 1, n ; j = 1, m ; and weight matrix W(t) = (wij (t)) is of

type n x m.

Each column in W(t) is a column vector wj (t), j = 1, m ;

T
wj (t) = [(w1j (t) . . . . wij (t) . . . wnj (t)] , T is transpose and

wij = 1/(n+1) where n is the size of input vector;

Example : If n = 3; then wij = 1/4

column vectors Wj=1 Wj=2

W11 W12

W(t) = (wij (t)) = W21 W22

W31 W32

The vji is weight from neuron j in F2 layer to neuron i in F1 layer;

where j = 1, m; i = 1, n ; Weight matrix V(t) = (vji (t)) is of type

m x n.

Each line in V(t) is a column vector vj (t), j = 1, m ;

T
vj (t) = [(vj1 (t) . . . . vji (t) . . . vjn (t)] , T is transpose and vji = 1 .

Each line is a column vector vj=1 vj=2

23
5
 v11 v12 v13

v(t) = (vji (t)) =

v21 v22 v23

■ Initialize the vigilance parameter, usually

0.3 ≤ ρ ≤ 0.5

■ Learning rate α = 0, 9

■
Special Rule : Example

"While indecision, then the winner is second between equal".

35

23
6
 SC - ART1 Algorithm

Step - 2 (Loop back from step 8)

Repeat steps 3 to 10 for all input vectors IH presented to the F1

layer; that is I(X) = { x(1), x(2), x(3), . . . , x(t), }

Step – 3 (Choose input pattern vector)

Present a randomly chosen input data pattern, in a format as input vector.

Time t =1,

The First the binary input pattern say { 0 0 1 } is presented to the

network. Then

–
As input I ≠ 0 , therefore node G1 = 1 and thus activates

all nodes in F1.

–
Again, as input I ≠ 0 and from F2 the output X2 = 0 means not

producing any output, therefore node G2 = 1 and thus activates

all nodes in F2, means recognition in F2 is allowed.

36

23
7
 SC - ART1 Algorithm

Step - 4 (Compute input for each node in F2)

Compute input y j for each node in F2 layer using :

Σ
y j = n Ii x wij , If j = 1 , 2 then y j=1 and y j=1 are

i=1

W11 W12

y j=1 =
I1 I2 I3 W21 y j=2 = I1 I2 I3 W22

W31 W32

Step – 5 (Select Winning neuron)

Find k , the node in F2, that has the largest y k calculated in step 4.

no of nodes

in F

y k = Σ 2 max (y j )

j=1

If an Indecision tie is noticed, then follow note stated below.

Else go to step 6.

Note :

Calculated in step 4, y j=1 = 1/4 and y j=2 = 1/4, are equal, means an

indecision tie then go by some defined special rule.

23
8
 Let us say the winner is the second node between the equals, i.e., k = 2.

Perform vigilance test, for the F2k output neuron , as below:

T
< Vk , X(t) > Vk · X(t)

r= =
||X(t)|| ||X(t)||

If r > ρ = 0.3 , means resonance exists and learning starts as :

The input vector x(t) is accepted by F2k=2 .

Go to step 6.

37

23
9
 SC - ART1 Algorithm

Step – 6 (Compute activation in F1)

For the winning node K in F2 in step 5, compute activation in F1 as

* * * * *
X k =(x 1 , x 2 , · · · , x i=n ) where x i = vki x Ii is the

piecewise product component by component and i = 1, 2, . . . , n. ; i.e.,

*K = (vk1 I1 , . . , vki Ii . ., vkn In) T

X

Step – 7 (Similarity between activation in F1 and input)

*
Calculate the similarity between X k and input IH using :

n
*

ΣXi
X*k
i=1
=

n
IH

Σ Ii

i=1

*
Example : If X K=2 = {0 0 , IH=1 = {0 0 1}

*
then similarity between X k and IH is

Σ
* *
X K=2 n X i
i=1
= = 1

24
0
 n
IH=1
Σ Ii

i=1

38

24
1
 SC - ART1 Algorithm

Step – 8 (Test similarity with vigilance parameter )

Test the similarity calculated in Step 7 with the vigilance parameter:

X K=2

The similarity ρ
= 1 is >

IH=1

*
It means the similarity between X K=2 , I is true. Therefore,
H

Associate Input IH=1 with F2 layer node m = k

(a) Temporarily disable node k by setting its activation to 0

(b) Update top-down weights , vj (t) of node j = k = 2 , from F2 to F1

vk i (new) = vk i (t) x Ii where i = 1, 2, . . . , n ,

(c) Update bottom-up weights , wj (t) of node j = k , from F2 to F1

vk i (new)

wk i (new) = where i = 1, 2, . . , n

0.5 + || vk i (new) ||

(d) Update weight matrix W(t) and V(t) for next input vector, time t =2

vj=1 vj=2

24
2
 v11 v12 v13

v(t) = v(2) = (vji (2)) =

v21 v22 v23

Wj=1 Wj=2

W11 W12

W(t) = W(2) = (wij (t)) = W21 W22

W31 W32

If done with all input pattern vectors t (1, n) then STOP.

else Repeat step 3 to 8 for next Input pattern

39

24
3
 SC - ART1 Numerical example

4.5 ART1 Numerical Example

• Example : Classify in even or odd the

numbers 1, 2, 3, 4, 5, 6, 7

Input:

The decimal numbers 1, 2, 3, 4, 5, 6, 7 given in the BCD format.

This input data is represented by the set() of the form

I(X) = { x(1), x(2), x(3), x(4), x(5), x(6), x(7) } where

Decimal nos BCD format Input vectors x(t)

1 0 0 1 x(1) = { 0 0 1}T

2 0 1 0 x(2) = { 0 1 0}T

3 0 1 1 x(3) = { 0 1 1}T

T
4 1 0 0 x(4) = { 1 0 0}

T
5 1 0 1 x(5) = { 1 0 1}

T
6 1 1 0 x(6) = { 1 1 0}

T
7 1 1 1 x(7) = { 1 1 1}

– The variable t is time, here the natural numbers which vary from

1 to 7, is expressed as t = 1 , 7 .

– The x(t) is input vector; t = 1, 7 represents 7 vectors.

24
4
 – Each x(t) has 3 elements, hence input layer F1 contains n= 3 neurons;

– let class A1 contains even numbers and A2 contains odd numbers,

this means , two clusters, therefore output layer F2 contains m = 2

neurons.

40

24
5
 SC - ART1 Numerical example

Step - 1
(Initialization)

■ Initially, no input vector I

is applied, making the control gains,
layer and F2 layer to zero. F1
G1 = 0, G2 = 0. Set nodes in

24
6
■ Initialize bottom-up wij (t) and top-down vji (t) weights for time t.
Weight wij is from neuron i in F1 layer to neuron j in F2 layer;

where i = 1, n ; j = 1, m ; and

weight matrix W(t) = (wij (t)) is of type n x m.

Each column in W(t) is a column vector wj (t), j = 1, m ;

T
wj (t) = [(w1j (t) . . . . wij (t) . . . wnj (t)] , T is transpose

and wij = 1/(n+1) where n is the

size of input vector;

here n = 3; so wij = 1/4

column vectors Wj=1 Wj=2

W11 W12 1/4 1/4

W(t) = (wij (t)) = W21 W22 = 1/4 1/4 where t=1

W31 W32 1/4 1/4

The vji is weight from neuron j in F2 layer to neuron i in F1 layer;

where j = 1, m; i = 1, n ;

weight matrix V(t) = (vji (t)) is of type m x n.

Each line in V(t) is a column vector vj (t), j = 1, m ;

T
vj (t) = [(vj1 (t) . . . . vji (t) . . . vjn (t)] , T is transpose and vji = 1 .

Each line is a column vector vj=1 vj=2

v11 v12 v13 1 1 1

v(t) = (vji (t)) = = where t=1

1 1 1
v21 v22 v23

■
Initialize

24
7
 the vigilance parameter ρ = 0.3, usually 0.3 ≤ ρ ≤ 0.5

■
Learning rate α = 0, 9

■
Special Rule : While indecision , then the winner is second between
equal.

SC - ART1 Numerical example

Step - 2
(Loop back from step 8)

Repeat
steps 3 to 10 for all input vectors IH = 1 to h=7 presented to

the
F1 layer; that is I(X) = { x(1), x(2), x(3), x(4), x(5), x(6), x(7) }

Step – 3
(Choose input pattern vector)

Present a randomly chosen input data in B C D format as input vector.

Let us choose the data in natural order, say x(t) = x(1) = { 0 0 1 }T

Time t =1, the binary input pattern { 0 0 1 }

is presented to network.

As
– input I ≠ 0 , therefore node G1 = 1 and thus activates

24
8
 all nodes in
F1.

– Again, as input I ≠ 0 and from F2 the output X2 = 0 means not

producing any output, therefore node G2 = 1 and thus activates

all nodes in F2, means recognition in F2 is allowed.

42

24
9
 SC - ART1 Numerical example

Step - 4 (Compute input for each node in F2)

Compute input y j for each node in F2 layer using :

Σ
y j = n Ii x wij

i=1

1/4
W11

1/4
y j=1 = I1 I2 I3 W21 = 0 0 1 = 1/4

1/4

W31

W12 1/4

1/4
y j=2 = I1 I2 I3 W22 = 0 0 1 = 1/4

1/4
W32

Step – 5 (Select winning neuron)

Find k , the node in F2, that has the largest y k calculated in step 4.

no of nodes

in F

y k = Σ 2 max (y j )
j=1

If an indecision tie is noticed, then follow note stated

below. Else go to step 6.

Note :

Calculated in step 4, y j=1 = 1/4 and y j=2 = 1/4, are equal, means an

25
0
 indecision tie. [Go by Remarks mentioned before, how to deal with the tie].

Let us say the winner is the second node between the equals, i.e., k = 2.

Perform vigilance test, for the F2k output neuron , as below:

T
< Vk , X(t) > Vk · X(t) 111 0 1

1
r= = = = =1
n
||X(t)|| ||X(t)|| 1
Σ |X(t)|
i=1

r>
Thus ρ = 0.3 , means resonance exists and learning starts as :

The input vector x(t=1) is accepted by F2k=2 , ie x(1) ∈ A2 cluster.

Go to Step 6.

43

25
1
 SC - ART1 Numerical example

Step – 6 (Compute activation in F1)

For the winning node K in F2 in step 5, compute activation in F1 as

* = *
x* * ·· ·, * where xi ki is the
(
i

Xk
1 ,x 2 , x i=n ) =v xI

piecewise product component by component and i = 1, 2, . . . , n. ; i.e.,

*
X
K = (vk1 I1 , . . , vki Ii . ., vkn In) T

*
Accordingly X K=2 = {1 1 1} x {0 0 1} = {0 0 1}

Step – 7 (Similarity between activation in F1 and input)

*
Calculate the similarity between X k and input IH using :

n
*

ΣX i

X*k
i=1

=
here n = 3

n
IH
Σ Ii

i=1

*
Accordingly , while X K=2 = {0 0 1} , I = {0 0 1}
H

*
Similarity between X k and IH is

* *
X K=2 n X
Σ

i=1
= = 1

25
2
 n
IH=1
Σ Ii

i=1

44

25
3
 SC - ART1 Numerical example

Step – 8 (Test similarity with vigilance parameter )

Test the similarity calculated in Step 7 with the vigilance parameter:

X K=2

>
The similarity 1 is ρ
=

IH=1

*
It means the similarity X K=2 , IH=1 is true. Therefore,

between

Associate Input IH=1 with F2 layer node m = k = 2 , i.e., Cluster 2

(a) Temporarily disable node k by setting its activation to 0

(b) Update top-down weights , vj (t) of node j = k = 2 , from F2 to F1

vk i (new) = vk i (t=1) x Ii where i = 1, 2, . . . , n = 3 ,

vk=2 (t=2) = vk=2, i (t=1) x Ii = 111 0

001 T
=

(c) Update bottom-up weights , wj (t) of node j = k = 2 , from F2 to F1

vk i (new)

wk i (new) = where i = 1, 2, . . , n = 3 ,

0.5
+ || vk i (new) ||

wk=2 (t=2) vk=2, i (t=2) = 001

25
4
 =
0.5 + ||vk=2, i (t=2)|| 0.5 +0 0 1

T
= 0 0 2/3

(d) Update weight matrix W(t) and V(t) for next input vector, time t =2

vj=1 vj=2

v11 v12 v13

1 1 1

v(t) = v(2) = (vji (2)) = =

0 0 1
v21 v22 v23

Wj=1 Wj=2

W11 W12 1/4 0

W(t) = W(2) = (wij (t)) = W21 W22 = 1/4 0

W31W32 1/42/3

If done with all input pattern vectors t (1, 7) then stop.

else Repeat step 3 to 8 for next input pattern

45

25
5
 SC - ART1 Numerical example

• Present Next Input Vector and Do Next Iteration (step 3 to 8)

Time t =2; IH = I2 = { 0 1 0 } ;

Wj=1 Wj=2 vj=1 vj=2

1/4 0
1 1 1

W(t=2) = 1/4 0 v(t=2) =

0 0 1

1/4 2/3

1/4

1/4
y j=1 = 0 1 0 = 1/4 = 0.25

1/4

0
y j=2 = 0 1 0 = 0 = 0

2/3

Find winning neuron, in node in F2 that has max(y j =1 , m) ;

Assign k = j ; i.e., y k = y j = max(1/4, 0) ;

Decision y j=1 is maximum, so K = 1

Do vigilance test , for output neuron F2k=1,

111 1
T
V k=1 · X(t=2) 0 1

r= = = = 1
n
||X(t=2)|| 1
Σ |X(t=2)|

25
6
 i=1

r>
Resonance , since ρ = 0.3 , resonance exists ; So Start learning;

Input vector x(t=2) is accepted by F2k=1 , means x(2) ∈ A1 Cluster.

Compute activation in F1, for winning node k = 1, piecewise product

component by component

* T
X K=1 = (vk1 IH1 , . . vki IHi . , vkn IHn)
Vk=1, i x I
= {1 1 1} x {0 1 0} = { 010 }

Find similarity between K=1 = {0 1 0} and IH=2 = {0 1 0} as

X

* Σ
X K=1
n *
X i

i=1
= = 1

n
IH=2
Σ IH=2, i

i=1

[Continued in next slide]

46

25
7
 SC - ART1 Numerical example

[Continued from previous slide : Time t =2]

Test the similarity calculated with the vigilance parameter:

X K=1

Similarity = 1 is > ρ
IH=2

*
It means the similarity X K=1 , IH=2 is true.

between

So Associate input IH=2 with F2 layer node m = k = 1, i.e., Cluster 1

(a) Temporarily disable node k = 1 by setting its activation to 0

(b) Update top-down weights, vj (t=2) of node j = k = 1, from F2 to F1

vk=1, i (new) = vk =1, i (t=2) x IH=2, i where i = 1, 2, . . . , n = 3 ,

vk=1, (t=3) = vk=1, i (t=2) x IH=2, i = 111 x 1

010 T
=

(c) Update bottom-up weights, wj (t=2) of node j = k = 1, from F1 to F2

vk =1, i (new)

wk=1, i (new) = where i = 1, 2, . . , n = 3 ,

0.5 + || vk=1, i (new) ||

vk=1, i (t=3) 010

wk=1, (t=3) = =

25
8
 0.5
+ ||vk=1, i (t=3)|| 0.5
+ 010

T
= 0 2/3 0

(d) Update weight matrix W(t) and V(t) for next input vector, time t =3

vj=1 vj=2

v11 v12 v13 0 1 0

V(t) = v(3) = (vji (3)) = =

0 0 1
v21 v22 v23

Wj=1 Wj=2

W11 W12 0 0

W(t) = W(3) = (wij (3)) = W21 W22 = 2/3 0

W31 W32 0 2/3

If done with all input pattern vectors t (1, 7) then stop.

else Repeat step 3 to 8 for next input pattern

47

25
9
 SC - ART1 Numerical example

• Present Next Input Vector and Do Next Iteration (step 3 to 8)

Time t = 3; IH = I3 = { 0 1 1 } ;

Wj=1 Wj=2 vj=1vj=2

0 0
0 1 0

W(t=3) = 2/3 0 v(t=3) =

001

0 2/3

2/3
y j=1 = 011 = 2/3 = 0.666 0

y j=2 = 0110=
2/3 = 0.666 2/3

Find winning neuron, in node in F2 that has max(y j =1 , m) ;

Assign k = j ; i.e., y k = y j = max(2/3, 2/3) ; indecision tie; take

winner as second; j = K = 2

Decision K = 2

Do vigilance test , for output neuron F2k=2,

001 0

26
0
 1
T
V k=2 · X(t=3) 1 1

r= = =
n
= 0.5
||X(t=3)||

2
Σ |X(t=3)|

i=1

r>
Resonance , since ρ = 0.3 , resonance exists ; So Start learning;

Input vector x(t=3) is accepted by F2k=2 , means x(3) ∈ A2 Cluster.

Compute activation in F1, for winning node k = 2, piecewise product

component by component

* T
X K=2 = Vk=2, i x IH=3, i = (vk1 IH1 , . . vki IHi . , vkn IHn)

= {0 0 1} x {0 1 1} = { 001 }

*
Find similarity between X K=2 = {0 1 0} and IH=3 = {0 1 1} as

* *
X K=2 X
Σn

i=1

= = 1/2 = 0.5

n
IH=3
Σ IH=3, i

i=1

[Continued in next slide]

48

26
1
 SC - ART1 Numerical example

[Continued from previous slide : Time t =3]

Test the similarity calculated with the vigilance parameter:

X K=1

Similarity
= 0.5 is > ρ
IH=2

*
It means the similarity between X K=2 , IH=3 is true.

So Associate input IH=3 with F2 layer node m = k = 2, i.e., Cluster 2

(a) Temporarily disable node k = 2 by setting its activation to 0

(b) Update top-down weights, vj (t=3) of node j = k = 2, from F2 to F1

vk=2, i (new) = vk =2, i (t=3) x IH=3, i where i = 1, 2, . . . , n = 3 ,

vk=2, (t=4) = vk=2, i (t=3) x IH=3, i = 001 x 1

001 T
=

(c) Update bottom-up weights, wj (t=3) of node j = k = 2, from F1 to F2

vk =2, i (new)

wk=2, i (new) = where i = 1, 2, . . , n = 3 ,

0.5 + || vk=2, i (new) ||

vk=2, i (t=4) 001

wk=2, (t=4) = =

26
2
 0.5
+ ||vk=2, i (t=4)|| 0.5
+ 001

T
= 0 0 2/3

(d) Update weight matrix W(t) and V(t) for next input vector, time t =4

vj=1 vj=2

v11 v12 v13 0 1 0

V(t) = v(3) = (vji (3)) = =

0 0 1
v21 v22 v23

Wj=1 Wj=2

W11 W12 0 0

W(t) = W(3) = (wij (3)) = W21 W22 = 2/3 0

W31 W32 0 2/3

If done with all input pattern vectors t (1, 7) then stop.

else Repeat step 3 to 8 for next input pattern

49

26
3
 SC - ART1 Numerical example

• Present Next Input Vector and Do Next Iteration (step 3 to 8)

Time t = 4; IH = I4 = { 1 0 0 } ;

Wj=1 Wj=2 vj=1 vj=2

0 0
0 1 0

W(t=3) = 2/3 0 v(t=3) =

0 0 1

0 2/3

0 0

y j=1 =
1 0 0 2/3 = 0 y j=2 = 1 0 0 0 = 0

0 2/3

Find winning neuron, in node in F2 that has max(y j =1 , m) ;

Assign k = j for y j = max(0, 0) ; Indecision tie; Analyze both cases

Case 1 : Take winner as first; j = K = 1; Decision K=

1 Do vigilance test , for output neuron

F2k=1,

010 0
T
V k=1 · X(t=4) 0 0

r= = = = 0
n

||X(t=4)|| 1
Σ |X(t=4)|

r<ρ i=1

2
 Resonance , since = 0.3 , no resonance exists ;

Input vector x(t=4) is not accepted by F2k=1, means x(4) ∉ A1 Cluster.

Put Output O1(t = 4) = 0.

Case 2 : Take winner as second ; j = K = 2; Decision K=2

Do vigilance test , for output neuron F2k=2,

001 0
T
V k=2 · X(t=4) 0 0

r= = = = 0
n

||X(t=4)|| 1
Σ |X(t=4)|
i=1

Resonance ,
since

r < ρ = 0.3 , no resonance exists ;

Input vector x(t=4) is not accepted by F2k=2, means x(4) ∉ A2 Cluster.

Put Output O2(t = 4) = 0.

Thus Input vector x(t=4) is Rejected by F2 layer.

[Continued in next slide]

50

SC - ART1 Numerical example

[Continued from previous slide : Time t =4]

Update weight matrix W(t) and V(t) for next input vector, time t =5

T
W(4) = W(3) ; V(4) = V(3) ; O(t = 4) = { 1 1}

vj=1 vj=2

2
 v11 v12 v13 0 1 0

V(t) = v(4) = (vji (4)) = =

0 0 1
v21 v22 v23

Wj=1 Wj=2

W11W12

0 0

W(t) = W(4) = (wij (4)) = W21W22 = 2/3 0

W31W32
02/3

If done with all input pattern vectors t (1, 7) then stop.

else Repeat step 3 to 8 for next input pattern

51

2
 SC - ART1 Numerical example

• Present Next Input Vector and Do Next Iteration (step 3 to 8)

Time t =5; IH = I5 = { 1 0 1 } ;

Wj=1 Wj=2 vj=1 vj=2

0 0
0 1 0

W(t=5) = 2/3 0 v(t=5) =

0 0 1

0 2/3

0 0

y j=1 = y =
1 0 1 2/3 = 0 j=2 1 0 1 0 = 2/3

0 2/3

Find winning neuron, in node in F2 that has max(y j =1 , m) ;

Assign k = j ; i.e., y k = y j = max(0, 2/3)

Decision y j=2 is maximum, so K = 2

Do vigilance test , for output neuron F2k=2,

001 0
T
V k=2 · X(t=5) 1 1

r= = = = 0.5
n
||X(t=5)|| 2
Σ |X(t=5)|

i=1

Resonance , since r > ρ= 0.3 , resonance exists ; So Start learning;

2
 Input vector x(t=5) is accepted by F2k=2 , means x(5) ∈ A2 Cluster.

Compute activation in F1, for winning node k = 2, piecewise product

component by component

* T
X K=2 = (vk1 IH1 , . . vki IHi . , vkn IHn)
Vk=2, i x I

= {0 0 1} x {1 0 1} = { 001 }

Find similarity between K=2 = {0 0 1} and IH=5 = {1 0 1} as

X

* Σ
X K=2
n *
X i

i=1

= = 1/2 = 0.5

n
IH=5
Σ IH=5, i

i=1

[Continued in next slide]

52

2
 SC - ART1 Numerical example

[Continued from previous slide : Time t =5]

Test the similarity calculated with the vigilance parameter:

X K=1

Similarity
= 0.5 is > ρ
IH=2

*
It means the similarity between X K=2 , IH=5 is true.

So Associate input IH=5 with F2 layer node m = k = 2, i.e., Cluster 2

(a) Temporarily disable node k = 2 by setting its activation to 0

(b) Update top-down weights, vj (t=5) of node j = k = 2, from F2 to F1

vk=2, i (new) = vk =2, i (t=5) x IH=5, i where i = 1, 2, . . . , n = 3 ,

vk=2, (t=6) = vk=2, i (t=5) x IH=5, i = 001 x 0

001 T
=

(c) Update bottom-up weights, wj (t=5) of node j = k = 2, from F1 to F2

vk =2, i (new)

wk=2, i (new) = where i = 1, 2, . . , n = 3 ,

0.5 + || vk=2, i (new) ||

vk=2, i (t=6) 001

wk=2, (t=6) = =

2
 0.5
+ ||vk=2, i (t=5)|| 0.5
+ 001

T
= 0 0 2/3

(d) Update weight matrix W(t) and V(t) for next input vector, time t =6

vj=1 vj=2

v11 v12 v13 0 1 0

V(t) = v(6) = (vji (6)) = =

0 0 1
v21 v22 v23

Wj=1 Wj=2

W11 W12 0 0

W(t) = W(6) = (wij (6)) = W21 W22 = 2/3 0

W31 W32 0 2/3

If done with all input pattern vectors t (1, 7) then stop.

else Repeat step 3 to 8 for next input pattern

53

2
 SC - ART1 Numerical example

• Present Next Input Vector and Do Next Iteration (step 3 to 8)

Time t =6; IH = I6 = { 1 1 0 } ;

Wj=1 Wj=2 vj=1 vj=2

0 0
0 1 0

W(t=6) = 2/3 0 v(t=6) =

0 0 1

0 2/3

0 0

y j=1 = y =
1 1 0 2/3 = 2/3 j=2 1 1 0 0 = 0

0 2/3

Find winning neuron, in node in F2 that has max(y j =1 , m) ;

Assign k = j ; i.e., y k = y j = max(2/3 , 0)

Decision y j=1 is maximum, so K = 1

Do vigilance test , for output neuron F2k=1,

010 1
T
V k=1 · X(t=6) 0 1

r= = = = 0.5
n
||X(t=6)|| 2
Σ |X(t=6)|

i=1

Resonance , since r > ρ= 0.3 , resonance exists ; So Start learning;

2
 Input vector x(t=6) is accepted by F2k=1 , means x(6) ∈ A1 Cluster.

Compute activation in F1, for winning node k = 1, piecewise product

component by component

* T
X K=1 = (vk1 IH1 , . . vki IHi . , vkn IHn)
Vk=1, i x I

= {0 1 0} x {1 1 0} = { 010 }

*
Find similarity between X K=1 = {0 1 0} and IH=6 = {1 1 0} as

* Σ
X K=1
n *
X i

i=1

= = 1/2 = 0.5

n
IH=6
Σ IH=5, i

i=1

[Continued in next slide]

54

2
 SC - ART1 Numerical example

[Continued from previous slide : Time t =6]

Test the similarity calculated with the vigilance parameter:

X K=1

Similarity
= 0.5 is > ρ

IH=6

*
It means the similarity between X K=1 , IH=6 is true.

So Associate input IH=6 with F2 layer node m = k = 1, i.e., Cluster 1

(a) Temporarily disable node k = 1 by setting its activation to 0

(b) Update top-down weights, vj (t=6) of node j = k = 2, from F2 to F1

vk=1, i (new) = vk =1, i (t=6) x IH=6, i where i = 1, 2, . . . , n = 3 ,

vk=1, (t=7) = vk=1, i (t=6) x IH=6, i = 010 x 1

010 T
=

(c) Update bottom-up weights, wj (t=2) of node j = k = 1, from F1 to F2

vk =1, i (new)

wk=1, i (new) = where i = 1, 2, . . , n = 3 ,

0.5 + || vk=1, i (new) ||

vk=1, i (t=7) 010

wk=1, (t=7) = =

2
 0.5
+ ||vk=1, i (t=7)|| 0.5
+ 010

T
= 0 2/3 0

• Update weight matrix W(t) and V(t) for next input vector, time t =7

vj=1 vj=2

v11 v12 v13 0 1 0

V(t) = v(7) = (vji (7)) = =

0 0 1
v21 v22 v23

Wj=1 Wj=2

W11 W12 0 0

W(t) = W(7) = (wij (7)) = W21 W22 = 2/3 0

W31 W32 0 2/3

If done with all input pattern vectors t (1, 7) then stop.

else Repeat step 3 to 8 for next input pattern

55

2
 SC - ART1 Numerical example

• Present Next Input Vector and Do Next Iteration (step 3 to 8)

Time t =7; IH = I7 = { 1 1 1 } ;

Wj=1 Wj=2 vj=1 vj=2

0 0
0 1 0

W(t=7) = 2/3 0 v(t=7) =

0 0 1

0 2/3

0 0

y j=1 = y =
1 1 1 2/3 = 2/3 j=2 1 1 1 0 = 2/3

0 2/3

Find winning neuron, in node in F2 that has max(y j =1 , m) ;

Assign k = j ; i.e., y k = y j = max(2/3 , 2/3) ; indecision tie;

take winner as second; j = K = 2

Decision K = 2

Do vigilance test , for output neuron F2k=1,

001 1
T
V k=2 · X(t=7) 1 1

r= = = = 0.333
||X(t=7)|| n 3

2
 Σ |X(t=7)|

i=1

r>
Resonance , since ρ = 0.3 , resonance exists ; So Start learning;

Input vector x(t=7) is accepted by F2k=2 , means x(7) ∈ A2 Cluster.

Compute activation in F1, for winning node k = 2, piecewise product

component by component

* T
X K=2 = (vk1 IH1 , . . vki IHi . , vkn IHn)
Vk=2, i x I

= {0 0 1} x {1 1 1} = { 001 }

*
Find similarity between X K=2 = {0 0 1} and IH=7 = {1 1 1} as

* *
X K=2 Σn X i

i=1

= = 1/3 = 0.333

n
IH=7
Σ IH=7, i

i=1

[Continued in next slide]

56

2
 SC - ART1 Numerical example

[Continued from previous slide : Time t =7]

Test the similarity calculated with the vigilance parameter:

X K=2

Similarity
= 0.333 is > ρ
IH=7

*
It means the similarity X K=2 , IH=7 is true.

between

So Associate input IH=7 with F2 layer node m = k = 2, i.e., Cluster 2

■ Temporarily disable node k = 2 by setting its activation to 0

■ Update top-down weights, vj (t=7) of node j = k = 2, from F2 to F1

vk=2, i (new) = vk =2, i (t=7) x IH=7, i where i = 1, 2, . . . , n = 3 ,

vk=2, (t=8) = vk=2, i (t=7) x IH=7, i = 001 x 1

001 T

■ Update bottom-up weights, wj (t=7) of node j = k = 1, from F1 to F2

vk =2, i (new)

wk=2, i (new) = where i = 1, 2, . . , n = 3 ,

0.5 + || vk=2, i (new) ||

vk=2, i (t=8) 001

wk=2, (t=8) = =

2
 0.5
+ ||vk=2, i (t=8)|| 0.5
+ 001

T
= 0 0 2/3

- Update weight matrix W(t) and V(t) for next input vector, time t =8

vj=1 vj=2

v11 v12 v13 0 1 0

V(t) = v(8) = (vji (8)) = =

0 0 1
v21 v22 v23

Wj=1 Wj=2

W11 W12 0 0

W(t) = W(8) = (wij (8)) = W21 W22 = 2/3 0

W31 W32 0 2/3

If done with all input pattern vectors t (1, 7) then STOP.

else Repeat step 3 to 8 for next input pattern

57

2
 SC - ART1 Numerical example

[Continued from previous slide]

■ Remarks

The decimal numbers

format (patterns) have
1, 2, 3, 4, 5, 6, 7 given in the BCD
as even or odd.

been classified into two clusters (classes)

Cluster Class A1 = { X(t=2), X(t=2) }

Cluster Class A2 = { X(t=1), X(t=3) , X(t=3) , X(t=3) }

The network failed to classify X(t=4) and rejected it.

The network has learned by the :

– Top down weight matrix V(t) and

– Bottom up weight matrix W(t)

These two weight matrices, given below, were arrived after all, 1 to 7,
patterns were one-by-one input to network that adjusted the weights
following the algorithm presented.

vj=1 vj=2

V(t) = v(8) = (vji (8)) = v11 v12 v13 = 0 1 0

2
 0 0 1
W11 W12 0 0
v21 v22 v23

W(t) = W(8) = (wij (8)) = W21 W22 = 2/3 0

Wj=1 Wj=2
W31 W32 0 2/3

58

2
 SC - ART2

4.3 ART2

The Adaptive Resonance Theory (ART) developed by Carpenter and

Grossberg designed for clustering binary vectors, called ART1 have been
illustrated in the previous section.

They later developed ART2 for clustering continuous or real valued vectors.
The capability of recognizing analog patterns is significant enhancement
to the system. The differences between ART2 and ART1 are :

■ The modifications needed to accommodate patterns with

continuous- valued components.

■ The F1 field of ART2 is more complex because continuous-valued input

vectors may be arbitrarily close together. The F1 layer is split into
several sunlayers.

■ The F1 field in ART2 includes a combination of normalization and noise

suppression, in addition to the comparison of the bottom-up and top-
down signals needed for the reset mechanism.

■ The orienting subsystem also to accommodate real-valued data.

The learning laws of ART2 are simple though the network is complicated.

59

2
 Fuzzy Set Theory

What is Fuzzy Set ?

■ The word "fuzzy" means "vagueness". Fuzziness occurs when the boundary
of a piece of information is not clear-cut.

■ Fuzzy sets have been introduced by Lotfi A. Zadeh (1965) as an extension

of the classical notion of set.

•
Classical set theory allows the membership of the elements in the set

in binary terms, a bivalent condition - an element either belongs or

does not belong to the set.

Fuzzy set theory permits the gradual assessment of the membership

of elements in a set, described with the aid of a membership function

valued in the real unit interval [0, 1].

- Example:

Words like young, tall, good, or high are fuzzy.

– There is no single quantitative value which defines the term young.

– For some people, age 25 is young, and for others, age 35 is young.

– The concept young has no clean boundary.

– Age 1 is definitely young and age 100 is definitely not young;

– Age 35 has some possibility of being young and usually depends on the
context in which it is being considered.

2
3. Introduction

In real world, there exists much fuzzy knowledge;

Knowledge that is vague, imprecise, uncertain, ambiguous, inexact, or

probabilistic in nature.

Human thinking and reasoning frequently involve fuzzy information, originating

from inherently inexact human concepts. Humans, can give satisfactory
answers, which are probably true.

However, our systems are unable to answer many questions. The reason is,
most systems are designed based upon classical set theory and two-valued
logic which is unable to cope with unreliable and incomplete information and
give expert opinions.

We want, our systems should also be able to cope with unreliable and
incomplete information and give expert opinions. Fuzzy sets have been able
provide solutions to many real world problems.

Fuzzy Set theory is an extension of classical set theory where elements have
degrees of membership.

2
• Classical Set Theory

A Set is any well defined collection of objects. An object in a set is called

an element or member of that set.

– Sets are defined by a simple statement describing whether a particular

element having a certain property belongs to that particular set.

– Classical set theory enumerates all its elements using

A = { a1 , a2 , a3 , a4 ,........an }

If the elements
ai (i = 1, 2, 3, . . . n) of a set A are subset of
then
universal set X,
set A can be represented for all elements

x
∈
X by its characteristic function

1 if x∈X

A (x) =

otherwise

– A set A is well described by a function called characteristic function.

This function, defined on the universal space X, assumes :

a value of 1 for those elements x that belong to set A, and

2
 - value of 0 for those elements x that do not belong to set A.
The notations used to express these mathematically are

: Χ → [0, 1]

A(x) = 1 , x is a member of A Eq.(1)

A(x) = 0 , x is not a member of A

Alternatively, the set A can be represented for all elements

x X
∈

by its characteristic function A (x) defined as

1 if x
∈
X

A (x) =
Eq.(2)

otherwise

– Thus in classical set theory A (x) has only the values 0 ('false') and 1

('true''). Such sets are called crisp sets.

05

2
 SC - Fuzzy set theory - Introduction

■ Fuzzy Set Theory

Fuzzy set theory is an extension of classical set theory where elements

have varying degrees of membership. A logic based on the two truth
values, True and False, is sometimes inadequate when describing human
reasoning. Fuzzy logic uses the whole interval between 0 (false) and 1
(true) to describe human reasoning.

– A Fuzzy Set is any set that allows its members to have different
degree of membership, called membership function, in the interval

[0 , 1].

– The degree of membership or truth is not same as probability;

0 fuzzy truth is not likelihood of some event or condition.

1 fuzzy truth represents membership in vaguely defined sets;

– Fuzzy logic is derived from fuzzy set theory dealing with reasoning that is

approximate rather than precisely deduced from classical predicate

logic.

– Fuzzy logic is capable of handling inherently imprecise concepts.

– Fuzzy logic allows in linguistic form the set membership values to

imprecise concepts like "slightly", "quite" and "very".

– Fuzzy set theory defines Fuzzy Operators on Fuzzy Sets.

06

2
 SC - Fuzzy set theory - Introduction

• Crisp and Non-Crisp Set

– As said before, in classical set theory, the characteristic function A(x)

of Eq.(2) has only values 0 ('false') and 1 ('true'').

Such sets are crisp sets.

– For Non-crisp sets the characteristic function A(x) can be defined.

ƒ The characteristic function A(x) of Eq. (2) for the crisp set is
generalized for the Non-crisp sets.

ƒ This generalized characteristic function A(x) of Eq.(2) is called

membership function.

Such Non-crisp sets are called Fuzzy Sets.

– Crisp set theory is not capable of representing descriptions and

classifications in many cases; In fact, Crisp set does not provide
adequate representation for most cases.

– The proposition of Fuzzy Sets are motivated by the need to capture and
represent real world data with uncertainty due to imprecise
measurement.

– The uncertainties are also caused by vagueness in the language.

07

2
• Representation of Crisp and Non-Crisp Set

Example : Classify students for a basketball team

This example explains the grade of truth value.

- tall students qualify and not tall students do not qualify

- if students 1.8 m tall are to be qualified, then

1
should we exclude a student who is /10" less? or
should we exclude a student who is 1" shorter?
•
Non-Crisp Representation to represent the notion of a tall person.

Degree or grade of truth Degree or grade of truth

Not Tall Tall Not Tall Tall

1 1

0 0

1.8 m Height x
1.8 m Height x

Crisp logic Non-crisp logic

Fig. 1 Set Representation – Degree or grade of truth

A student of height 1.79m would belong to both tall and not tall sets
with a particular degree of membership.

As the height increases the membership grade within the tall set would
increase whilst the membership grade within the not-tall set would
decrease.

2
2
 SC - Fuzzy set theory - Introduction

■ Capturing Uncertainty

Instead of avoiding or ignoring uncertainty, Lotfi Zadeh introduced Fuzzy

Set theory that captures uncertainty.

■ A fuzzy set is described by a membership function A (x) of A.

This membership function associates to each element xσ ∈

a
X

number as A (xσ ) in the closed unit interval [0, 1].

The number A (xσ ) represents the degree of membership of xσ

in A.

• The notation used for membership function A (x) of a fuzzy set A is

: Χ → [0, 1]

•
Each membership function maps elements of a given universal
base set X , which is itself a crisp set, into real numbers in [0, 1] .

■ Example

c (x)
F (x)

C
F

0.5

2
 0
x

Fig. 2 Membership function of a Crisp set C and Fuzzy set F

■
In the case of Crisp Sets the members of a set are :

either out of the set, with membership of degree " 0

", or in the set, with membership of degree " 1 ",

Therefore, Crisp Sets ⊆ Fuzzy Sets

In other words, Crisp Sets are Special cases of Fuzzy Sets.

[Continued in next slide]

09

2
 SC - Fuzzy set theory - Introduction

■ Examples of Crisp and Non-Crisp

Set

Example 1: Set of prime numbers ( a crisp set)

If we consider space X consisting of natural numbers ≤ 12

ie X = {1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12}

Then, the set of prime numbers could be described as follows.

PRIME = {x contained in X | x is a prime number} = {2, 3, 5, 6, 7, 11}

Example 2: Set of SMALL ( as non-crisp set)

A Set X that consists of SMALL cannot be described;

for example 1 is a member of SMALL and 12 is not a member of SMALL.

Set A, as SMALL, has un-sharp boundaries, can be characterized by a

function that assigns a real number from the closed interval from 0 to 1 to
each element x in the set X.

10

2
 SC - Fuzzy set theory – Fuzzy Set

2. Fuzzy Set

A Fuzzy Set is any set that allows its members to have of

membership, called membership function, in the interval different degree
[0 , 1].

• Definition of Fuzzy set

A
fuzzy set A, defined in the universal space X, is a function defined

in
X which assumes values in the range [0, 1].

A fuzzy set A is written as a set of pairs

{x, A(x)} as

A = {{x , A(x)}} , x in the set X

where x is an element of the universal space X, and

A(x) is the value of the function A for this element.

The value
A(x) is the membership of the element x in a
grade

fuzzy set
A.

Example : ≤
Set SMALL in set X consisting of natural numbers to 12.

Assume:
SMALL(1) = 1, SMALL(2) = 1, SMALL(3) = 0.9, SMALL(4) = 0.6,

2
SMALL(5) = 0.4, SMALL(u) = 0 for u >= 9.
SMALL(6) = 0.3,
SMALL(7) = 0.2,
SMALL(8) = 0.1,

Then, following the notations described in the definition above :

Set SMALL = {{1, 1 }, {2, 1 }, {3, 0.9}, {4, 0.6}, {5, 0.4}, {6, 0.3}, {7, 0.2},

{8, 0.1}, {9, 0 }, {10, 0 }, {11, 0}, {12, 0}}

Note that a fuzzy set can be defined precisely by associating with each x ,
its grade of membership in SMALL.

11

2
 SC - Fuzzy set theory – Fuzzy Set

• Definition of Universal Space

Originally the universal space for fuzzy sets in fuzzy logic was defined only
on the integers. Now, the universal space for fuzzy sets and fuzzy
relations is defined with three numbers.

The first two numbers specify the start and end of the universal space,
and the third argument specifies the increment between elements. This
gives the user more flexibility in choosing the universal space.

Example : The fuzzy set of numbers, defined in the universal space

X = { xi } = {1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12} is presented as

SetOption [FuzzySet, UniversalSpace → {1, 12, 1}]

12

2
 SC - Fuzzy set theory – Fuzzy Membership

2.1 Fuzzy Membership

A fuzzy set A defined in the universal space X is a function

defined in X which assumes values in the range [0, 1].

A fuzzy set A is written as a set of pairs {x, A(x)}.

A = {{x , A(x)}} , x in the set X

where x is an element of the universal space X, and

A(x) is the value of the function A for this element.

The value A(x) is the degree of membership of the element x

in a fuzzy set A.

The Graphic Interpretation of fuzzy membership for the fuzzy sets : Small,
Prime Numbers, Universal-space, Finite and Infinite UniversalSpace, and
Empty are illustrated in the next few slides.

13

2
 SC - Fuzzy set theory – Fuzzy Membership

• Graphic Interpretation of Fuzzy Sets SMALL

The fuzzy set SMALL of small numbers, defined in the universal space

X = { xi } = {1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12} is presented as

SetOption [FuzzySet, UniversalSpace → {1, 12, 1}]

The Set SMALL in set X is :

SMALL = FuzzySet {{1, 1 }, {2, 1 }, {3, 0.9}, {4, 0.6}, {5, 0.4}, {6, 0.3},

{7, 0.2}, {8, 0.1}, {9, 0 }, {10, 0 }, {11, 0}, {12, 0}}

Therefore SetSmall is represented as

SetSmall = FuzzySet [{{1,1},{2,1}, {3,0.9}, {4,0.6}, {5,0.4},{6,0.3}, {7,0.2},

{8, 0.1}, {9, 0}, {10, 0}, {11, 0}, {12, 0}} , UniversalSpace → {1, 12, 1}]

FuzzyPlot [ SMALL, AxesLable

→ {"X", "SMALL"}]

SMALL

.8

.6

.4

.2

0 0
1 23 4 5 6 7 8 9 101112 X

Fig Graphic Interpretation of Fuzzy Sets SMALL

2
14

2
 SC - Fuzzy set theory – Fuzzy Membership

• Graphic Interpretation of Fuzzy Sets PRIME Numbers

The fuzzy set PRIME numbers, defined in the universal space

X = { xi } = {1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12} is presented as

SetOption [FuzzySet, UniversalSpace → {1, 12, 1}]

The Set PRIME in set X is :

PRIME = FuzzySet {{1, 0}, {2, 1}, {3, 1}, {4, 0}, {5, 1}, {6, 0}, {7, 1}, {8, 0}, {9, 0}, {10, 0}, {11, 1},

{12, 0}}

Therefore SetPrime is represented as

SetPrime = FuzzySet [{{1,0},{2,1}, {3,1}, {4,0}, {5,1},{6,0}, {7,1},

{8, 0}, {9, 0}, {10, 0}, {11, 1}, {12, 0}} , UniversalSpace → {1, 12, 1}]

FuzzyPlot [ PRIME, AxesLable

→ {"X", "PRIME"}]

PRIME

.8

.6

.4

.2

0 0 1 23 4 5 6 7 8 9 101112 X

Fig Graphic Interpretation of Fuzzy Sets PRIME

2
15

3
 SC - Fuzzy set theory – Fuzzy Membership

• Graphic Interpretation of Fuzzy Sets UNIVERSALSPACE

In any application of sets or fuzzy sets theory, all sets are subsets of

i fixed set called universal space or universe of discourse denoted by X.

Universal space X as a fuzzy set is a function equal to 1 for all elements.

The fuzzy set UNIVERSALSPACE numbers, defined in the universal

space X = { xi } = {1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12} is presented as

SetOption [FuzzySet, UniversalSpace

→ {1, 12, 1}]

The Set UNIVERSALSPACE in set X is :

UNIVERSALSPACE = FuzzySet {{1, 1}, {2, 1}, {3, 1}, {4, 1}, {5, 1}, {6, 1},

{7, 1}, {8, 1}, {9, 1}, {10, 1}, {11, 1}, {12, 1}}

Therefore SetUniversal is represented as

SetUniversal = FuzzySet [{{1,1},{2,1}, {3,1}, {4,1}, {5,1},{6,1}, {7,1},

{8, 1}, {9, 1}, {10, 1}, {11, 1}, {12, 1}} , UniversalSpace → {1, 12, 1}]

FuzzyPlot [ UNIVERSALSPACE, AxesLable → {"X", " UNIVERSAL SPACE "}]

UNIVERSAL SPACE

.8

.6

.4

.2

3
 0 1 2 3 4 5 6 7 8 9 10 11 12 X

Fig Graphic Interpretation of Fuzzy Set UNIVERSALSPACE

16

3
 SC - Fuzzy set theory – Fuzzy Membership

• Finite and Infinite Universal Space

Universal sets can be finite or infinite.

Any universal set is finite if it consists of a specific number of different

elements, that is, if in counting the different elements of the set, the
counting can come to an end, else the set is infinite.

Examples:

1. Let N be the universal space of the days of the week.

N = {Mo, Tu, We, Th, Fr, Sa, Su}. N is finite.

2. Let M = {1, 3, 5, 7, 9, ...}. M is infinite.

3. Let L = {u | u is a lake in a city }. L is finite.

(Although it may be difficult to count the number of lakes in a

city, but L is still a finite universal set.)

17

3
 SC - Fuzzy set theory – Fuzzy Membership

• Graphic Interpretation of Fuzzy Sets EMPTY

An empty set is a set that contains only elements with a grade of

membership equal to 0.

Example: Let EMPTY be a set of people, in Minnesota, older than

120. The Empty set is also called the Null set.

The fuzzy set EMPTY , defined in the universal space

X = { xi } = {1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12} is presented as

SetOption [FuzzySet, UniversalSpace → {1, 12, 1}]

The Set EMPTY in set X is :

EMPTY = FuzzySet {{1, 0}, {2, 0}, {3, 0}, {4, 0}, {5, 0}, {6, 0}, {7, 0}, {8,

0}, {9, 0}, {10, 0}, {11, 0}, {12, 0}}

Therefore SetEmpty is represented as

SetEmpty = FuzzySet [{{1,0},{2,0}, {3,0}, {4,0}, {5,0},{6,0}, {7,0},

{10, 0}, {11, 0}, {12, 0}} , UniversalSpace

{8, 0}, {9, 0}, → {1, 12, 1}]

FuzzyPlot [ EMPTY, AxesLable

→ {"X", " UNIVERSAL SPACE "}]

EMPTY

.8

.6

3
 .4

.2

0 0 101112 X
1 23 4 5 6 7 8 9

Fig Graphic Interpretation of Fuzzy Set EMPTY

18

3
 SC - Fuzzy set theory – Fuzzy Operation

2.2 Fuzzy Operations

A fuzzy set operations are the operations on fuzzy sets. The fuzzy set
operations are generalization of crisp set operations. Zadeh [1965]
formulated the fuzzy set theory in the terms of standard operations:
Complement, Union, Intersection, and Difference.

In this section, the graphical interpretation of the following standard fuzzy

set terms and the Fuzzy Logic operations are illustrated:

Inclusion :
FuzzyInclude [VERYSMALL, SMALL]

Equality
: FuzzyEQUALITY [SMALL, STILLSMALL]

Complement
: FuzzyNOTSMALL = FuzzyCompliment [Small]

Union :
∪
FuzzyUNION = [SMALL MEDIUM]

Intersection :
∩
FUZZYINTERSECTON = [SMALL MEDIUM]

19

3
 SC - Fuzzy set theory – Fuzzy Operation

• Inclusion

Let A and B be fuzzy sets defined in the same universal space X.

The fuzzy set A is included in the fuzzy set B if and only if for every x in

the set X we have A(x) ≤ B(x)

Example :

The fuzzy set UNIVERSALSPACE numbers, defined in the universal

space X = { xi } = {1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12} is presented as

SetOption [FuzzySet, UniversalSpace → {1, 12, 1}]

The fuzzy set B SMALL

The Set SMALL in set X is :

SMALL = FuzzySet {{1, 1 }, {2, 1 }, {3, 0.9}, {4, 0.6}, {5, 0.4}, {6, 0.3},

{7, 0.2}, {8, 0.1}, {9, 0 }, {10, 0 }, {11, 0}, {12, 0}}

Therefore SetSmall is represented as

SetSmall = FuzzySet [{{1,1},{2,1}, {3,0.9}, {4,0.6}, {5,0.4},{6,0.3}, {7,0.2},

{8, 0.1}, {9, 0}, {10, 0}, {11, 0}, {12, 0}} , UniversalSpace → {1, 12, 1}]

The fuzzy set A VERYSMALL

The Set VERYSMALL in set X is :

VERYSMALL = FuzzySet {{1, 1 }, {2, 0.8 }, {3, 0.7}, {4, 0.4}, {5, 0.2},

{6, 0.1}, {7, 0 }, {8, 0 }, {9, 0 }, {10, 0 }, {11, 0}, {12, 0}}

Therefore SetVerySmall is represented as

SetVerySmall = FuzzySet [{{1,1},{2,0.8}, {3,0.7}, {4,0.4}, {5,0.2},{6,0.1},

{7,0}, {8, 0}, {9, 0}, {10, 0}, {11, 0}, {12, 0}} , UniversalSpace → {1, 12, 1}]

3
 The Fuzzy Operation : Inclusion

Include [VERYSMALL, SMALL]

Membership Grade B A

.8

.6

.4

.2

0 1 2 3 4 5 6 7 8 9 101112 X

Fig Graphic Interpretation of Fuzzy Inclusion

FuzzyPlot [SMALL, VERYSMALL]

20

3
 SC - Fuzzy set theory – Fuzzy Operation

• Comparability

Two fuzzy sets A and B are comparable

if the condition A ⊂ B or B ⊂ A holds, ie,

if one of the fuzzy sets is a subset of the other set, they are comparable.

Two fuzzy sets A and B are incomparable

If the condition A ⊄ B or B ⊄ A holds.

Example 1:

Let A = {{a, 1}, {b, 1}, {c, 0}}

and

B = {{a, 1}, {b, 1}, {c, 1}}.

A

Then A is comparable to B, since

is a subset of B.

Example 2 :

Let C = {{a, 1}, {b, 1}, {c, 0.5}} and

D = {{a, 1}, {b, 0.9}, {c, 0.6}}.

Then C and D are not comparable since

C is not a subset of D and

D is not a subset of C.

3
 Property Related to Inclusion :

for all x in the set X, if A(x) ⊂ B(x) ⊂ C(x), then accordingly A ⊂ C.

21

3
 SC - Fuzzy set theory – Fuzzy Operation

• Equality

Let A and B
be fuzzy sets defined in the same space X.
Then A and B if
and only if
are equal, which is denoted X = Y

for all x in the set X, A(x) = B(x).

Example.

The fuzzy set B SMALL

SMALL = FuzzySet {{1, 1 }, {2, 1 }, {3, 0.9}, {4, 0.6}, {5, 0.4}, {6, 0.3},

{7, 0.2}, {8, 0.1}, {9, 0 }, {10, 0 }, {11, 0}, {12, 0}}

The fuzzy set A STILLSMALL

STILLSMALL = FuzzySet {{1, 1 }, {2, 1 }, {3, 0.9}, {4, 0.6}, {5, 0.4},

{6, 0.3}, {7, 0.2}, {8, 0.1}, {9, 0 }, {10, 0 }, {11, 0}, {12, 0}}

The Fuzzy Operation : Equality

Equality [SMALL, STILLSMALL]

Membership Grade BA

.8

.6

3
 .4

.2

0 0 1 2 3 4 5 6 7 8 9 101112 X

Fig Graphic Interpretation of Fuzzy

Equality FuzzyPlot [SMALL,

STILLSMALL]

Note : If equality A(x) = B(x) is not satisfied even for one element x in

the set X, then we say that A is not equal to B.

22

3
 SC - Fuzzy set theory – Fuzzy Operation

• Complement

Let A be a fuzzy set defined in the space X.

Then the fuzzy set B is a complement of the fuzzy set A, if and only if,

for all x in the set X, B(x) = 1 - A(x).

The complement of the fuzzy set A is often denoted by A' or Ac or A

Fuzzy Complement : Ac(x) = 1 – A(x)

Example 1.

The fuzzy set A SMALL

SMALL = FuzzySet {{1, 1 }, {2, 1 }, {3, 0.9}, {4, 0.6}, {5, 0.4}, {6, 0.3},

{7, 0.2}, {8, 0.1}, {9, 0 }, {10, 0 }, {11, 0}, {12, 0}}

The fuzzy set Ac NOTSMALL

NOTSMALL = FuzzySet {{1, 0 }, {2, 0 }, {3, 0.1}, {4, 0.4}, {5, 0.6}, {6, 0.7},

{7, 0.8}, {8, 0.9}, {9, 1 }, {10, 1 }, {11, 1}, {12, 1}}

The Fuzzy Operation : Compliment

NOTSMALL = Compliment [SMALL]

Membership Grade A Ac

.8

.6

.4

.2

3
 0

0 1 2 3 4 5 6 7 8 9 101112 X

Fig Graphic Interpretation of Fuzzy Compliment

FuzzyPlot [SMALL, NOTSMALL]

23

3
 SC - Fuzzy set theory – Fuzzy Operation

Example 2.

The empty set Φ and the universal set X, as fuzzy sets, are

complements of one another.

Φ'= X' =
X Φ
,

The fuzzy set B EMPTY

Empty = FuzzySet {{1, 0 }, {2, 0 }, {3, 0}, {4, 0}, {5, 0}, {6, 0},

{7, 0}, {8, 0}, {9, 0 }, {10, 0 }, {11, 0}, {12, 0}}

The fuzzy set A UNIVERSAL

Universal = FuzzySet {{1, 1 }, {2, 1 }, {3, 1}, {4, 1}, {5, 1}, {6, 1},

{7, 1}, {8, 1}, {9, 1 }, {10, 1 }, {11, 1}, {12, 1}}

The fuzzy operation : Compliment

EMPTY = Compliment [UNIVERSALSPACE]

Membership Grade B A

.8

.6

.4

.2

0 0 1 2 3 4 5 6 7 8 9 10 11 12 X

Fig Graphic Interpretation of Fuzzy Compliment

FuzzyPlot [EMPTY, UNIVERSALSPACE]

3
24

3
 SC - Fuzzy set theory – Fuzzy Operation

• Union

Let A and B be fuzzy sets defined in the space X.

The union is defined as the smallest fuzzy set that contains both A and
B. The union of A and B is denoted by A ∪ B.

The following relation must be satisfied for the union operation

: for all x in the set X, (A ∪ B)(x) = Max (A(x), B(x)).

Fuzzy Union : (A ∪ B)(x) = max [A(x), B(x)] for all x ∈ X

Example 1 : Union of Fuzzy A and B

A(x) = 0.6 and B(x) = 0.4 ∴ (A ∪ B)(x) = max [0.6, 0.4] = 0.6

Example 2 : Union of SMALL and MEDIUM

The fuzzy set A SMALL

SMALL = FuzzySet {{1, 1 }, {2, 1 }, {3, 0.9}, {4, 0.6}, {5, 0.4}, {6, 0.3},

{7, 0.2}, {8, 0.1}, {9, 0 }, {10, 0 }, {11, 0}, {12, 0}}

The fuzzy set B MEDIUM

MEDIUM = FuzzySet {{1, 0 }, {2, 0 }, {3, 0}, {4, 0.2}, {5, 0.5}, {6, 0.8},

{7, 1}, {8, 1}, {9, 0.7 }, {10, 0.4 }, {11, 0.1}, {12, 0}}

The fuzzy operation : Union

FUZZYUNION = [SMALL
∪ MEDIUM]

SetSmallUNIONMedium = FuzzySet [{{1,1},{2,1}, {3,0.9}, {4,0.6}, {5,0.5},

{6,0.8}, {7,1}, {8, 1}, {9, 0.7}, {10, 0.4}, {11, 0.1}, {12, 0}} ,

UniversalSpace → {1, 12, 1}]

Membership Grade FUZZYUNION = [SMALL ∪ MEDIUM]

3
 .8

.6

.4

.2

0 1 2 3 4 5 67 8 9 101112 X

Fig Graphic Interpretation of Fuzzy Union

FuzzyPlot [UNION]

The notion of the union is closely related to that of the connective "or".

Let A is a class of "Young" men, B is a class of "Bald" men.

If "David is Young" or "David is Bald," then David is associated with the

union of A and B. Implies David is a member of A ∪ B.

25

3
 SC - Fuzzy set theory – Fuzzy Operation

• Intersection

Let A and B be fuzzy sets defined in the space X. Intersection is defined

as the greatest fuzzy set that include both A and B. Intersection of A and
B is denoted by A ∩ B. The following relation must be satisfied for the

intersection operation :

for all x in the set X, (A ∩ B)(x) = Min (A(x), B(x)).

Fuzzy Intersection : (A
∩ B)(x) = min [A(x), B(x)] for all x ∈ X

Example 1 : Intersection of Fuzzy A and B

A(x) = 0.6 and B(x) = 0.4 ∴ (A ∩ B)(x) = min [0.6, 0.4] = 0.4

Example 2 : Union of SMALL and MEDIUM

The fuzzy set A SMALL

SMALL = FuzzySet {{1, 1 }, {2, 1 }, {3, 0.9}, {4, 0.6}, {5, 0.4}, {6, 0.3},

{7, 0.2}, {8, 0.1}, {9, 0 }, {10, 0 }, {11, 0}, {12, 0}}

The fuzzy set B MEDIUM

MEDIUM = FuzzySet {{1, 0 }, {2, 0 }, {3, 0}, {4, 0.2}, {5, 0.5}, {6, 0.8},

{7, 1}, {8, 1}, {9, 0.7 }, {10, 0.4 }, {11, 0.1}, {12, 0}}

The fuzzy operation : Intersection

FUZZYINTERSECTION = min [SMALL

∩ MEDIUM]

SetSmallINTERSECTIONMedium = FuzzySet [{{1,0},{2,0}, {3,0}, {4,0.2},

{5,0.4}, {6,0.3}, {7,0.2}, {8, 0.1}, {9, 0},

{10, 0}, {11, 0}, {12, 0}} , UniversalSpace → {1, 12, 1}]

Membership Grade FUZZYINTERSECTON = [SMALL ∩ MEDIUM]

3
 1

.8

.6

.4

.2

0 0
1 2 3 4 5 6 7 8 9 101112 X

Fig Graphic Interpretation of Fuzzy Union

FuzzyPlot [INTERSECTION]

26

3
 SC - Fuzzy set theory – Fuzzy Operation

■ Difference

Let A and B be fuzzy sets defined in the space X.

The difference of A and B is denoted by A ∩ B'.

Fuzzy Difference : (A - B)(x) = min [A(x), 1- B(x)] for all x ∈ X

Example : Difference of MEDIUM and SMALL

The fuzzy set A MEDIUM

MEDIUM = FuzzySet {{1, 0 }, {2, 0 }, {3, 0}, {4, 0.2}, {5, 0.5}, {6, 0.8},

{7, 1}, {8, 1}, {9, 0.7 }, {10, 0.4 }, {11, 0.1}, {12, 0}}

The fuzzy set B SMALL

MEDIUM = FuzzySet {{1, 1 }, {2, 1 }, {3, 0.9}, {4, 0.6}, {5, 0.4}, {6, 0.3},

{7, 0.2}, {8, 0.1}, {9, 0.7 }, {10, 0.4 }, {11, 0}, {12, 0}}

Fuzzy Complement : Bc(x) = 1 – B(x)

The fuzzy set Bc NOTSMALL

NOTSMALL = FuzzySet {{1, 0 }, {2, 0 }, {3, 0.1}, {4, 0.4}, {5, 0.6}, {6, 0.7}, {7, 0.8},

{8, 0.9}, {9, 1 }, {10, 1 }, {11, 1}, {12, 1}}

The fuzzy operation : Difference by the definition of Difference

FUZZYDIFFERENCE = [MEDIUM
∩ SMALL']

SetMediumDIFFERECESmall = FuzzySet [{{1,0},{2,0}, {3,0}, {4,0.2},

{5,0.5}, {6,0.7}, {7,0.8}, {8, 0.9}, {9, 0.7},

UniversalSpace
{10, 0.4}, {11, 0.1}, {12, 0}} , → {1, 12, 1}]

3
 FUZZYDIFFERENCE = [MEDIUM
Membership ∪ SMALL' ]
Grade

.8

.6

.4

.2

0 1 2 3 4 5 6 7 8 9 101112X

Fig Graphic Interpretation of Fuzzy Union

FuzzyPlot [UNION]

27

3
 SC - Fuzzy set theory – Fuzzy Properties

2.3 Fuzzy Properties

Properties related to Union, Intersection, Differences are illustrated below.

■ Properties Related to Union

The properties related to union are :

Identity, Idempotence, Commutativity and Associativity.

■ Identity:

A∪
Φ =A

input = Equality [SMALL ∪

output = True EMPTY , SMALL]

A
∪ X=X

input
= Equality [SMALL UnivrsalSpace , UnivrsalSpace]
∪

output = True

■ Idempotence :
A∪A=A

input = Equality [SMALL ∪ SMALL , SMALL]

output = True

3
 •
Commutativity :

A ∪ B =B ∪ A

input = Equality [SMALL ∪ MEDIUM, MEDIUM ∪ SMALL]

output = True

28

3
 SC - Fuzzy set theory – Fuzzy Properties

[Continued from previous slide]

-
Associativity:

A ∪ (B∪ C) = (A∪ B) ∪ C

input = Equality [Small ∪

(Medium Big) , (Small
∪
Medium) Big]
∪ ∪

output = True

Fuzzy Set Small , Medium , Big

Small = FuzzySet {{1, 1 }, {2, 1 }, {3, 0.9}, {4, 0.6}, {5, 0.4}, {6, 0.3},

{7, 0.2}, {8, 0.1}, {9, 0.7 }, {10, 0.4 }, {11, 0}, {12, 0}}

Medium = FuzzySet {{1, 0 }, {2, 0 }, {3, 0}, {4, 0.2}, {5, 0.5}, {6, 0.8},

{7, 1}, {8, 1}, {9, 0 }, {10, 0 }, {11, 0.1}, {12, 0}}

Big
= FuzzySet [{{1,0}, {2,0}, {3,0}, {4,0}, {5,0}, {6,0.1}, {7,0.2},
{8,0.4}, {9,0.6}, {10,0.8}, {11,1}, {12,1}}]

Calculate Fuzzy relations :

• Medium ∪ Big = FuzzySet [{1,0},{2,0}, {3,0}, {4,0.2}, {5,0.5},

{6,0.8},{7,1}, {8, 1}, {9, 0.6}, {10, 0.8}, {11, 1}, {12, 1}]

• Small ∪ Medium = FuzzySet [{1,1},{2,1}, {3,0.9}, {4,0.6}, {5,0.5},

{6,0.8}, {7,1}, {8, 1}, {9, 0.7}, {10, 0.4}, {11, 0.1}, {12, 0}]

3
• Small ∪ (Medium ∪ Big) = FuzzySet [{1,1},{2,1}, {3,0.9}, {4,0.6},
{5,0.5}, {6,0.8}, {7,1}, {8, 1}, {9, 0.7}, {10, 0.8}, {11, 1}, {12, 1}]

• (Small ∪ Medium) ∪ Big] = FuzzySet [{1,1},{2,1}, {3,0.9}, {4,0.6},

{5,0.5}, {6,0.8}, {7,1}, {8, 1}, {9, 0.7},{10, 0.8}, {11, 1},{12, 1}]

Fuzzy set (3) and (4) proves Associativity relation

29

3
 SC - Fuzzy set theory – Fuzzy Properties

■ Properties Related to
Intersection

Absorption, Identity, Idempotence, Commutativity, Associativity.

•
Absorption by Empty Set :

A∩
Φ =Φ

input = Equality [Small

Empty , Empty]
∩

output = True

■ Identity :

A∩X=A

input = Equality [Small ∩ UnivrsalSpace , Small]

output = True
•
Idempotence :

A∩ A=A

input = Equality [Small ∩ Small , Small]

output = True
■ Commutativity :

A
∩ B=B ∩ A

input = Equality [Small ∩ Big , Big

∩ Small]

output = True

3
 ■ Associativity :

A
∩ (B
C) = (A B) C
∩ ∩
∩

input = Equality [Small

∩ (Medium Big), (Small Medium) Big]
∩ ∩ ∩

output = True

30

3
 SC - Fuzzy set theory – Fuzzy Properties

■ Additional Properties

Related to Intersection and Union

•
Distributivity:

A ∩ (B ∪ C) = (A ∩ B) ∪ (A ∩ C)

input = Equality [Small ∩ (Medium ∪ Big) ,

(Small ∩ Medium) ∪ (Small ∩ Big)]

output = True

•
Distributivity:

A ∪ (B ∩ C) = (A ∪ B) ∩ (A ∪ C)

input = Equality [Small ∪ (Medium ∩ Big) ,

(Small ∪ Medium) ∩ (Small ∪ Big)]

output = True

•
Law of excluded middle :

A ∪ A' = X

input = Equality [Small ∪ NotSmall , UnivrsalSpace ]

output = True

=
Law of contradiction

3
 A ∩ A' = Φ

input = Equality [Small ∩ NotSmall , EmptySpace ]

output = True

31

3
 SC - Fuzzy set theory – Fuzzy Properties

aa Cartesian Product Of Two Fuzzy Sets

Cartesian Product of two Crisp

Sets

Let A and B be two crisp sets in the universe of discourse X and Y..
The Cartesian product of A and B is denoted by A x B

Defined as A x B = { (a , b) │ a ∈ A , b ∈ B }

Note : Generally A x B ≠ B x A

Example : Graphic representation of A x B

B
Let A = {a, b, c} and B = {1, 2}

2
then A x B = { (a , 1) , (a , 2) ,

(b , 1) , (b , 2) , 1

(c , 1) , (c , 2) } A

ab c

•
Cartesian product of two Fuzzy Sets

Let A and B be two fuzzy sets in the universe of discourse X and Y.

The Cartesian product of A and B is denoted by A x B

Defined by their membership function  A (x) and  B (y) as

 A x B (x , y) = min [  A (x) ,  B (y) ] =  A (x) ∧  B (y)

or  A x B (x , y) =  A (x)  B (y)

for all x∈ X and y ∈ Y

3
 Thus the Cartesian product A x B is a fuzzy set of ordered pair

and y ∈ Y,
(x , y) for all x ∈ X with grade membership of (x , y) in

• x Y given by the above equations .

In a sense Cartesian product of two Fuzzy sets is a Fuzzy Relation.

32

3
 SC - Fuzzy set theory – Fuzzy Relations

- Fuzzy Relations

Fuzzy Relations describe the degree of association of the elements;

Example : “x is approximately equal to y”.

– Fuzzy relations offer the capability to capture the uncertainty and

vagueness in relations between sets and elements of a set.

– Fuzzy Relations make the description of a concept possible.

Fuzzy Relations were introduced to supersede classical crisp relations; It

describes the total presence or absence of association of elements.

In this section, first the fuzzy relation is defined and then expressing fuzzy
relations in terms of matrices and graphical visualizations. Later the properties
of fuzzy relations and operations that can be performed with fuzzy relations are
illustrated.

33

3
 SC - Fuzzy set theory – Fuzzy Relations

3.1 Definition of Fuzzy Relation

Fuzzy relation is a generalization

of the definition of fuzzy set

from 2-D space to 3-D space.

• Fuzzy relation definition

Consider a Cartesian product

AxB
= { (x , y) | x ∈ A, y ∈ B}

where A
and B are subsets of universal sets U1 and U2.

Fuzzy relation
on A x B is denoted by R or R(x , y) is defined as the set

R = { ((x , y) , R (x , y)) | (x , y) ∈ ∈
A x B , R (x , y) [0,1] }

where R (x , y) is a function in two variables called membership function.

– It gives the degree of membership of the ordered pair (x , y) in R

associating with each pair (x , y) in A x B a real number in the interval
[0 , 1].

– The degree of membership indicates the degree to which x is in relation

to y.

3
 Note :

– Definition of fuzzy relation is a generalization of the definition of fuzzy

set from the 2-D space (x , , R (x)) to 3-D space ((x , y) , R (x , y)).

– Cartesian product A x B is a relation by itself between x and y .

– A fuzzy relation R is a sub set of R3 namely

{ ((x , y) , R (x , y)) | ∈ A x B x [0,1] ∈ U1 x U2 x [0,1] }

34

3
 SC - Fuzzy set theory – Fuzzy Relations

• Example of Fuzzy
Relation

- = { ((x1 , y1) , 0)) , ((x1 , y2) , 0.1)) , ((x1 , y3) , 0.2)) ,

((x2 , y1) , 0.7)) , ((x2 , y2) , 0.2)) , ((x2 , y3) , 0.3)) ,

((x3 , y1) , 1)) , ((x3 , y2) , 0.6)) , ((x3 , y3) , 0.2)) ,

The relation can be written in matrix form as

yy1Y2Y3

x100.1 0.2
R

X20.7 0.2 0.3

X310.6 0.2

where symbol means ' is defined as' and

the values in the matrix are the values of membership function:

R (x1 , y1) = 0 R (x1 , y2) = 0.1 R (x1 , y3) = 0.2

R (x2 , y1) = 0.7 R (x2, y2) = 0.2 R (x2 , y3) = 0.3

R (x3 , y1) = 1 R (x3 , y2) = 0.6 R (x3 , y3) = 0.2

(X, Y, ) as :
Assuming x1 = 1 , x2 = 2 , x3 = 3
the relation can be graphically

3
and y1 = 1 , y2= 2 , y3= 3 , represented
by points in 3-D space

direction of y, we therefore say that

the relation R describes x is grater

Note : Since the values of the membership than y.

function 0.7, 1, 0.6 are in the direction of x
below the major diagonal (0, 0.2, 0.2) in the
matrix are grater than those 0.1, 0.2, 0.3 in the

35

3
 SC - Fuzzy set theory – Fuzzy Relations

3.2 Forming Fuzzy Relations

Assume that V and W are two collections of objects.

A fuzzy relation is characterized in the same way as it is in a fuzzy set.

– The first item is a list containing element and membership grade pairs,

{{v1, w1}, R11}, {{ v1, w2}, R12}, ... , {{ vn, wm}, Rnm}}.

where { v1, w1}, { v1, w2}, ... , { vn, wm} are the elements of the relation are

defined as ordered pairs, and { R11 , R12 , ... , Rnm} are the membership grades
of the elements of the relation that range from 0 to 1, inclusive.

– Thesecond item is the universal space; for relations, the universal

space consists of a pair of ordered pairs,

{{ Vmin, Vmax, C1}, { Wmin, Wmax, C2}}.

where the first pair defines the universal space for the first set and the second
pair defines the universal space for the second set.

Example showing how fuzzy relations are represented

Let V = {1, 2, 3} and W = {1, 2, 3, 4}.

A fuzzy relation R is, a function defined in the space V x W, which takes

values from the interval [0, 1] , expressed as R : V x W → [0, 1]

■ = FuzzyRelation [{{{1, 1}, 1}, {{1, 2}, 0.2}, {{1, 3}, 0.7}, {{1, 4}, 0}, {{2,
1}, 0.7}, {{2, 2}, 1}, {{2, 3}, 0.4}, {{2, 4}, 0.8}, {{3, 1},
0}, {{3, 2}, 0.6}, {{3, 3}, 0.3}, {{3, 4}, 0.5},
UniversalSpace → {{1, 3, 1}, {1, 4, 1}}]

This relation can be represented in the following two forms shown below

3
 Elements of fuzzy relation are ordered pairs {vi , wj}, where vi is first and
wj is second element. The membership grades of the elements are
represented by the heights of the vertical lines.

36

3
 SC - Fuzzy set theory – Fuzzy Relations

3.3 Projections of Fuzzy

Relations

Definition : A fuzzy relation on A x B is denoted by R or R(x , y) is

defined as the set

R = { ((x , y) , R (x , y)) | (x , y) ∈ A x B , R (x , y) ∈ [0,1] }

where R (x , y) is a function in two variables called membership

function. The first, the second and the total projections of fuzzy

relations are stated below.

■ First
Projection max of R : defined as
X

(1) (1)
R = {(x) ,  R (x , y))}

= {(x) , max  R (x , y)) | (x , y) ∈ A x B }

Y

max
Y

max ■ Second Projection of R : defined as

X

(2) (2)
R = {(y) ,  (x , y))}
R

= {(y) , R (x , y)) | (x , y) ∈ AxB}

• Total Projection of R : defined as

R(T) =

3
 max max ∈
{ R (x , y) | (x , y) AxB}
X Y

Note : In all these three expression

means max with respect to y while x is considered fixed

means max with respect to x while y is considered fixed

The Total Projection is also known as Global projection

37

3
 SC - Fuzzy set theory – Fuzzy Relations

■ Example : Fuzzy Projections

The Fuzzy Relation R together with First, Second and Total Projection

of R are shown below.

yy1 y2 y3y4 Y5 R(1)

x

x1
0.1 0.3 1 0.5 0.3 1

x2 0.2 0.5 0.7 0.9 0.6 0.9

x3
0.3 0.6 1 0.8 0.2 1

R(2)
0.3 0.6 1 0.9 0.6 1 = R(T)

Note :

(1)
For R select max means max with respect to y while x is considered fixed

For R(2) select max means max with respect to x while y is considered fixed

For R(T) select max with respect to R(1) and R(2)

The Fuzzy plot of these projections are shown below.

R(1) R(2)
1 1

.8
.8

.6 .6

3
 .4
.4

.2
.2

0 x 0 y

1 2345

12 34 5

(1) (2)
Fig Fuzzy plot of 1st projection R Fig Fuzzy plot of 2nd projection R

38

3
 SC - Fuzzy set theory – Fuzzy Relations

3.4 Max-Min and Min-Max

Composition

The operation composition combines the fuzzy relations in different

x A, y z
variables, say (x , y) and (y , z) ; ∈
∈ B ∈ C.
Consider the relations : ,

R1(x , y) = { ((x , y) , R1 (x , y)) | (x , y) ∈ AxB}

R2(y , z) = { ((y , y) , R1 (y , z)) | (y , z) ∈ B x C }

The domain of R1 is A x B and the domain of R2 is B x C

◊ Max-Min Composition

Definition : The Max-Min composition denoted by R1 ο R2 with

membership function  R1 ο R2 defined as

R1
ο R2 = { ((x , z) , max(min (R1 (x , y) , R2 (y , z))))} ,

(x , z)
AxC,y ∈B
∈

Thus
R1
R2 is relation in the domain AxC
ο

An example of the composition is shown in the next slide.

39

3
 SC - Fuzzy set theory – Fuzzy Relations

◊ Example : Max-Min
Composition

Consider the relations R1(x , y) and R2(y , z) as given below.

y y1 y2 y3 z z1 z2 z3

x y

0.1 0.3 0 0.8 0.2 0

x1 y1
R1 R2

x2 0.8 1 0.3 y2 0.2 1 0.6

y3 0.5 0 0.4

Note : Number of columns in the first table and second table are equal.

Compute max-min composition denoted by R1 ο R2 :

Step -1 Compute min operation (definition in previous slide).

Consider row x1 and column z1 , means the pair (x1 , z1) for all yj ,

j = 1, 2, 3, and perform min operation

min (R1 (x1 , y1) , R2 (y1 , z1)) = min (0.1, 0.8) = 0.1,

min (R1 (x1 , y2) , R2 (y2 , z1)) = min (0.3, 0.2) = 0.2,

min (R1 (x1 , y3) , R2 (y3 , z1)) = min ( 0, 0.5) = 0,

Step -2 Compute max operation (definition in previous slide).

For x = x1 , z = z1 , y = yj , j = 1, 2, 3,

Calculate the grade membership of the pair (x1 , z1) as

{ (x1 , z1) , max ( (min (0.1, 0.8), min (0.3, 0.2), min (0, 0.5) )
i.e. { (x1 , z1) , max(0.1, 0.2, 0) }

3
 i.e. { (x1 , z1) , 0.2 }

Hence the grade membership of the pair (x1 , z1) is 0.2 .

Similarly, find all the grade membership of the pairs

(x1 , z2) , (x1 , z3) , (x2 , z1) , (x2 , z2) , (x2 , z3)

The final result is

zz1 z2 z3

R1 ο R2 = x1 0.1 0.3 0

x2 0.8 1 0.3

Note : If tables R1 and R2 are considered as matrices, the operation

composition resembles the operation multiplication in matrix calculus
linking row by columns. After each cell is occupied max-min value (the
product is replaced by min, the sum is replaced by max).

40

3
 SC - Fuzzy set theory – Fuzzy Relations

■ Example : Min-Max
Composition

The min-max composition is similar to max-min composition with the

difference that the roll of max and min are interchanged.

Definition : The max-min composition denoted by R1 R2 with membership

function  R1 R2 is defined by

min
R1 R2 = { ((x , z) , y (max (R1 (x , y) , R2 (y , z))))} ,

(x , z) ∈AxC,y∈B

Thus R1 R2 is relation in the domain A x C

Consider the relations R1(x , y) and R2(y , z) relation

as given by the same
of previous example of max-min composition,
that is

yy1 y2 y3 z z1 z2 z3

x y

R1 x1 0.1 0.3 0 y1 0.8 0.2 0

R2

x2 0.8 1 0.3 y2 0.2 1 0.6

y3 0.5 0 0.4

After computation in similar way as done in the case of max-min

3
 composition, the final result is

z z1 z2 z3

R1 R2 = x1 0.3 0 0.1

x2 0.5 0.4 0.4

◊ Relation between Max-Min and Min-Max Compositions

The Max-Min and Min-Max Compositions are related by the formula

R1 ο R2 = R1 R2

41

3
 Fuzzy Systems

What are Fuzzy Systems ?

• Fuzzy Systems include Fuzzy Logic and Fuzzy Set Theory.

• Knowledge exists in two distinct forms :

– the Objective knowledge that exists in mathematical form is used in

engineering problems; and

– the Subjective knowledge that exists in linguistic form, usually

impossible to quantify.

Fuzzy Logic can coordinate these two forms of knowledge in a logical way.

• Fuzzy Systems can handle simultaneously the numerical data and

linguistic knowledge.

• Fuzzy Systems provide opportunities for modeling of conditions which are

inherently imprecisely defined.

• Many real world problems have been modeled, simulated, and

replicated with the help of fuzzy systems.

• The applications of Fuzzy Systems are many like : Information retrieval

systems, Navigation system, and Robot vision.

• Expert Systems design have become easy because their domains

are inherently fuzzy and can now be handled better;

3
 examples : Decision-support systems, Financial planners, Diagnostic
system, and Meteorological system.

03

3
 Sc – Fuzzy System Introduction

• Introduction

Any system that uses Fuzzy mathematics may be viewed as Fuzzy system.

The Fuzzy Set Theory - membership function, operations, properties and the
relations have been described in previous lectures. These are the prerequisites
for understanding Fuzzy Systems. The applications of Fuzzy set theory is Fuzzy
logic which is covered in this section.

Here the emphasis is on the design of fuzzy system and fuzzy controller in a

closed–loop. The specific topics of interest are :

– Fuzzification of input information,

– Fuzzy Inferencing using Fuzzy sets ,

– De-Fuzzification of results from the Reasoning process, and

– Fuzzy controller in a closed–loop.

Fuzzy Inferencing, is the core constituent of a fuzzy system. A block schematic

of Fuzzy System is shown in the next slide. Fuzzy Inferencing combines the
facts obtained from the Fuzzification with the fuzzy rule base and conducts the
Fuzzy Reasoning Process.

04

3
 Sc – Fuzzy System Introduction

• Fuzzy System

A block schematic of Fuzzy System is shown below.

Fuzzy Rule Base

Input output

variables variables

X1
Fuzzy Inferencing Y1

X2 Y2
Fuzzification Defuzzification

Xn Ym

Membeship Function

Fig. Elements of Fuzzy System

Fuzzy System elements

– Input Vector : X = [x1 , x2, . . . xn ] Tare crisp values, which

are transformed into fuzzy sets in the fuzzification block.

T
– Output Vector : Y = [y1 , y2, . . . ym ] comes out from the

3
 defuzzification block, which transforms an output fuzzy set back to

a crisp value.

– Fuzzification : a process of transforming crisp values into grades of

membership for linguistic terms, "far", "near", "small" of fuzzy sets.

– Fuzzy Rule base : a collection of propositions containing linguistic

variables; the rules are expressed in the form:

If (x is A ) AND (y is B )...............THEN (z is C)

where x, y and z represent variables (e.g. distance, size) and

A, B and Z are linguistic variables (e.g. `far', `near', `small').

– Membership function : provides a measure of the degree of similarity

of elements in the universe of discourse U to fuzzy set.

– Fuzzy Inferencing : combines the facts obtained from the Fuzzification

with the rule base and conducts the Fuzzy reasoning process.

– Defuzzyfication: Translate results back to the real world values.

05

3
 Sc – Fuzzy System – Fuzzy logic

1. Fuzzy Logic

A simple form of logic, called a two-valued logic is the study of "truth tables"
and logic circuits. Here the possible values are true as 1, and false as 0.

This simple two-valued logic is generalized and called fuzzy logic which treats
"truth" as a continuous quantity ranging from 0 to 1.

Definition : Fuzzy logic (FL) is derived from fuzzy set theory dealing with
reasoning that is approximate rather than precisely deduced from classical two-
valued logic.

– FL is the application of Fuzzy set theory.

– FL allows set membership values to range (inclusively) between 0 and 1.

– FL is capable of handling inherently imprecise concepts.

– FL allows in linguistic form, the set membership values to imprecise concepts

like "slightly", "quite" and "very".

06

3
 Sc – Fuzzy System – Fuzzy logic

2.1 Classical Logic

Logic is used to represent simple facts. Logic defines the ways of putting
symbols together to form sentences that represent facts. Sentences are
either true or false but not both are called propositions.

Examples :

Sentence Truth value Is it a Proposition ?

"Grass is green" "true" Yes

"2 + 5 = 5" "false" Yes

"Close the door" - No

"Is it hot out side ?" - No

"x > 2" - No (since x is not defined)

"x = x" - No

(don't know what is "x" and "="

mean; "3 = 3" or say "air is equal

to air" or "Water is equal to water"

has no meaning)

• Propositional Logic (PL)

A proposition is a statement - which in English is a declarative sentence

and Logic defines the ways of putting symbols together to form sentences
that represent facts. Every proposition is either true or false. Propositional
logic is also called boolean algebra.

Examples: (a) The sky is blue., (b) Snow is cold. , (c) 12 * 12=144

3
 Propositional logic : It is fundamental to all logic.

• Propositions are “Sentences”; either true or false but not both.

• A sentence is smallest unit in propositional logic

• If proposition is true, then truth value is "true"; else “false”

‡ Example ;
Sentence "Grass is green";

Truth value “ true”;

Proposition “yes”

07

3
 Sc – Fuzzy System – Fuzzy logic

ƒ Statement, Variables and

Symbols

Statement : A simple statement is one that does not contain any other
statement as a part. A compound statement is one that has two or more
simple statements as parts called components.

Operator or connective : Joins simple statements into compounds, and

joins compounds into larger compounds.

Symbols for connectives

assertion P "p is true"

nagation ¬p ~ ! NOT "p is false"

p
conjunction ∧ q · && & AND "both p and q are true"

disjunction P v q || ‫׀‬ OR "either p is true,

or q is true,

or both "

implication p→q ⊃ ⇒ if . . then "if p is true, then q is

true" " p implies q "

equivalence ↔ ≡ ⇔ if and only if "p and q are either both true

or both false"

08

3
 Sc – Fuzzy System – Fuzzy logic

• Truth Value

The truth value of a statement is its truth or falsity ,

• is either true or false,

~pis either true or false,

p v q is either true or false, and so on.

"T" or "1" means "true". and

"F" or "0" means "false"

Truth table is a convenient way of showing relationship between several

propositions. The truth table for negation, conjunction, disjunction,

implication and equivalence are shown below.

p q ¬p ¬q p ∧ q p v q p→ q p↔ q q→ p

T T F F T T T T T

T F F T F T F F T

F T T F F T T F F

F F T T F F T T T

09

3
 Sc – Fuzzy System – Fuzzy logic

■ Tautology

A Tautology is proposition formed by combining other propositions

(p, q, r, . . .) which is true regardless of truth or falsehood of p, q,

r, . . . .

The important tautologies are :

¬ [p (¬p)
(p→ q) ↔ ∧ (¬q)] and (p→ q) ↔ q
∨

A proof of these tautologies, using the truth tables are given below.

¬ [p (¬p)
Tautologies (p→ q) ↔ ∧ (¬q)] and (p→ q) ↔ ∨ q

Table 1: Proof of Tautologies

p→ ¬ [p ∧ (¬p)
p q q ¬q p ∧ (¬q) (¬q)] ¬p ∨ q

T T T F F T F T

T F F T T F F F

F T T F F T T T

F
F F T T T T T

Note :

- The entries of two columns p→ q and  [p ∧ (q)] are identical,

proves the tautology. Similarly, the entries of two columns p→ q and
(p) ∨ q are identical, proves the other tautology.

3
 - The importance of these tautologies is that they express the
membership function for p→ q in terms of membership functions of
either propositions p and q or p and q.

10

3
 Sc – Fuzzy System – Fuzzy logic

■ Equivalences

Between Logic , Set theory and Boolean algebra.

Some mathematical equivalence between Logic and Set theory and the
correspondence between Logic and Boolean algebra (0, 1) are given
below.

Logic Boolean Algebra (0, 1) Set theory

T 1

F 0

∧ x ,∩
∩
∨ + , U
∪ ―
¬ ′ ie complement ( )

↔ =

p, q, r a, b, c

11

3
 Sc – Fuzzy System – Fuzzy logic

• Membership Functions obtain from facts

Consider the facts (the two tautologies)

(p→ q) ↔  [p ∧ (q)] and (p→ q) ↔ (p) ∨ q

Using these facts and the equivalence between logic and set theory,
we can obtain membership functions for p→ q (x , y) .

From 1st fact : p→ q (x , y) = 1- p ∩q

(x , y)

= 1 – min [ p(x) , 1 -  q (y)] Eq (1)

From 2nd fact : p→ q (x , y) =1- U q (x , y)

p

= max [ 1 -  p (x) ,  q (y)] Eq (2)

Boolean truth table below shows the validation membership functions

Table-2 : Validation of Eq (1) and Eq (2)

 p(x)  q(y) 1 -  p (x) 1 -  q (y) max [ 1 -  p (x) , 1 – min [ p(x) ,

 q (y)] 1 -  q (y)]

1 1 0 0 1 1

1 0
0 1 0 0

0 1 1 0 1 1

0 0 1 1 1 1

3
 Note :

■ Entries in last two columns of this table-2 agrees with the entries in
table-1 for p→ q , the proof of tautologies, read T as 1 and F as 0.

■ The implication membership functions of Eq.1 and Eq.2 are not the
only ones that give agreement with p→ q. The others are :

p
→
q (x , y) = 1 -  p (x) (1 -  q (y)) Eq (3)

p
→
q (x , y) = min [ 1, 1 -  p (x) +  q (y)] Eq (4)

12

3
 Sc – Fuzzy System – Fuzzy logic

• Modus Ponens and Modus

Tollens

In traditional propositional logic there are two important inference

rules, Modus Ponens and Modus Tollens.

Modus Ponens

Premise 1 : " x is A "

Premise 2 : " if x is A then y is B " ; Consequence : " y is B "

Modus Ponens is associated with the implication " A implies B " [A→ B]
In terms of propositions p and q, the Modus Ponens is expressed as

(p ∧ (p → q)) →
q Modus Tollens

Premise 1 : " y is not B "

Premise 2 : " if x is A then y is B " ; Consequence : " x is not A " In

terms of propositions p and q, the Modus Tollens is expressed as

(¬ q
(p
¬p
q))
→

∧ →

13

3
 Sc – Fuzzy System – Fuzzy logic

2.2 Fuzzy Logic

Like the extension of crisp set theory to fuzzy set theory, the extension of
crisp logic is made by replacing the bivalent membership functions of the
crisp logic with the fuzzy membership functions.

In crisp logic, the truth value acquired by the proposition are 2-valued,
namely true as 1 and false as 0.

In fuzzy logic, the truth values are multi-valued, as absolute true, partially
true, absolute false etc represented numerically as real value between

■ to 1.

Note : The fuzzy variables in fuzzy sets, fuzzy propositions, fuzzy relations
~
etc are represented usually using symbol ~ as P but for the purpose of
easy to write it is always represented as P .

14

3
 Sc – Fuzzy System – Fuzzy logic

- Recaps

01 Membership function  A (x) describes the membership of the elements x of the

base set X in the fuzzy set A .

• Fuzzy Intersection operator ∩ ( AND connective ) applied to two fuzzy sets A

and B with the membership functions  A (x) and  B (x) based on min/max
operations is  A ∩ B = min [  A (x) ,  B (x) ] , x ∈ X (Eq. 01)

• Fuzzy Intersection operator ∩ ( AND connective ) applied to two fuzzy sets A

and B with the membership functions  A (x) and  B (x) based on algebraic

product is  A ∩ B =  A (x)  B (x) , x ∈ X (Eq. 02)

4. Fuzzy Union operator U ( OR connective ) applied to two fuzzy sets A and B with
the membership functions  A (x) and  B (x) based on min/max
operations is  A U B = max [  A (x) ,  B (x) ] , x∈ X (Eq. 03)

• Fuzzy Union operator U ( OR connective ) applied to two fuzzy sets A and B

with the membership functions  A (x) and  B (x) based on algebraic sum is

x
 A U B =  A (x) +  B (x) -  A (x)  B (x) , ∈ X (Eq. 04)

06 Fuzzy Compliment operator ( ― ) ( NOT operation ) applied to fuzzy set A

1 -  A (x) , x
with the membership function  A (x) is  = ∈ X (Eq. 05)

4. Fuzzy relations combining two fuzzy sets by connective "min operation" is an

operation by cartesian product R : X x Y → [0 , 1].

 R(x,y) = min[ A (x),  B (y)] (Eq. 06) or

YVh-mm

 R(x,y) =  A (x)  B (y) (Eq. 07)

G 10.50.0

3
 R

Example : Relation R between fruit colour x

Y 0.3 1 0.4

R 0 0.2 1

and maturity grade y characterized by base set

linguistic colorset X = {green, yellow, red}

maturity grade as Y = {verdant, half-mature, mature}

• Max-Min Composition - combines the fuzzy relations

variables, say (x , y) and (y , z) ; x ∈ A , y ∈ B , z ∈ C .
consider the relations :

R1(x , y) = { ((x , y) , R1 (x , y)) | (x , y) ∈ A x B }

R2(y , z) = { ((y , y) , R1 (y , z)) | (y , z) ∈ B x C }

The domain of R1 is A x B and the domain of R2 is B x C

max-min composition denoted by R1

ο R2 with membership function  R1 ο R2

R1 R2 = { ((x , z) , max (min (R1 (x , y) , R2 (y , z))))} ,

ο y
(x , z) ∈ A x C , y ∈ B (Eq. 08)

Thus
R1 R2 is relation in the domain AxC
ο

15

3
 Sc – Fuzzy System – Fuzzy logic

- Fuzzy Propositional

A fuzzy proposition is a statement P which acquires a fuzzy truth value

T(P) .

Example :

P: Ram is honest

T(P) = 0.8 , means P is partially true. T(P) =

1 , means P is absolutely true.

16

3
 Sc – Fuzzy System – Fuzzy logic

■ Fuzzy Connectives

The fuzzy logic is similar to crisp logic supported by connectives.

Table below illustrates the definitions of fuzzy connectives.

Table : Fuzzy Connectves

Connective Symbols Usage Definition

Nagation ¬ ¬P 1 – T(P)

P
Disjuction ∨ ∨ Q Max[T(P) , T(Q)]

P
Conjuction ∧ ∧ Q min[T(P) , T(Q)]

Implication ⇒ P⇒Q ¬P ∨ Q = max (1-T(P), T(Q)]

Here P , Q are fuzzy proposition and T(P) , T(Q) are their truth values.

– the P and Q are related by the ⇒ operator are known as antecedents

and consequent respectively.

– as crisp logic, here in fuzzy logic also the operator ⇒ represents

IF-THEN statement like,

IF x is A THEN y is B, is equivalent to

R = (A x B) U (¬ A x Y)

the membership function of R is given by

R (x , y) = max [min (A (x) , B (y)) , 1 − A (x)]

3
– For the compound implication statement like

IF x is A THEN y is B, ELSE y is C is equivalent to

R = (A x B) U (¬ A x C)

the membership function of R is given by

R (x , y) = max [min (A (x) , B (y)) , min (1 − A (x), C (y))]

17

3
 Sc – Fuzzy System – Fuzzy logic

Example 1 : (Ref : Previous slide)

P : Mary is efficient , T(P) = 0.8 ,

Q : Ram is efficient , T(Q) = 0.65 ,

¬ P : Mary is efficient , T(¬ P) = 1 − T(P) = 1− 0.8 = 0.2

P
∧ Q : Mary is efficient and so is Ram, i.e.

T(P
∧ Q) = min (T(P), T(Q)) = min (0.8, 0.65)) = 0.65

P
∨ Q : Either Mary or Ram is efficient i.e.

T(P
∨ Q) = max (T(P), T(Q)) = max (0.8, 0.65)) = 0.8

P ⇒ Q : If Mary is efficient then so is Ram, i.e.

T(P ⇒ Q) = max (1− T(P), T(Q)) = max (0.2, 0.65)) = 0.65

18

3
 Sc – Fuzzy System – Fuzzy logic

Example 2 : (Ref : Previous slide on fuzzy connective)

Let X = {a, b, c, d} ,

A = {(a, 0) (b, 0.8) (c, 0.6) (d, 1)}

B = {(1, 0.2) (2, 1) (3, 0.8) (4, 0)}

C = {(1, 0) (2, 0.4) (3, 1) (4, 0.8)}

= { 1, 2, 3, 4} the universe of discourse could be viewed as

{ (1, 1) (2, 1) (3, 1) (4, 1) }

i.e., a fuzzy set all of whose elements x have (x) = 1

Determine the implication relations

• If x is A THEN y is B
for all x in the set
X

• If x is A THEN y is B Else y is C

Solution

To determine implication relations (i) compute :

The operator ⇒ represents IF-THEN statement like,

IF x is A THEN y is B,
is equivalent to R = (A x B) U (¬ A x Y) and

the membership function R is given by

R (x , y) = max [min (A (x) , B (y)) , 1 − A (x)]

Fuzzy Intersection A x B is defined as : Fuzzy Intersection ¬A x Y is defined as :

3
 for all x in the set X,

(A ∩ B)(x) = min [A(x),

(¬A ∩ Y)(x) = min [A(x),
B(x)],
Y(x)],

B 1 2 3 4
y 1 2 3 4

A
A

a 0 0 0 0
a 1 1 1 1

AxB= b 0.2 0.8 0.8 0 ¬A x Y = b 0.2 0.2 0.2 0.2

c 0.2 0.6 0.6 0

c 0.4 0.4 0.4 0.4

d 0.2 1 0.8 0
d 0 0 0 0

Fuzzy Union is defined as (A ∪ B)(x) = max [A(x), B(x)] for all x ∈X

Therefore R = (A x B) U (¬ A x Y) gives

y 1 2 3 4

a 1 1 1 1

b 0.2 0.8 0.8 0

R=

c 0.4 0.6 0.6 0.4

d 0.2 1 0.8 0

This represents If x is A THEN y is B ie T(A ➺ B) = max (1- T(A), T(B))

19

3
 Sc – Fuzzy System – Fuzzy logic

To determine implication relations (ii) compute : (Ref : Previous slide)

Given X = {a, b, c, d} ,

A = {(a, 0) (b, 0.8) (c, 0.6) (d, 1)}

B = {(1, 0.2) (2, 1) (3, 0.8) (4, 0)}

C = {(1, 0) (2, 0.4) (3, 1) (4, 0.8)}

Here, the operator ➺ represents IF-THEN-ELSE statement like,

IF x is A THEN y is B Else y is C, is equivalent to

R = (A x B) U (¬ A x C) and

the membership function of R is given by

R (x , y) = max [min (A (x) , B (y)) , min(1 − A (x), C (y)]

Fuzzy Intersection A x B is defined as : Fuzzy Intersection ¬A x Y is defined as :

for all x in the set X, for all x in the set X

(A ∩ B)(x) = min [A(x),

(¬A ∩ C)(x) = min [A(x),
B(x)],
C(x)],

B 1 2 3 4 y 1 2 3 4

A A

a 0 0 0 0 a 0 0.4 1 0.8

AxB= b 0.2 0.8 0.8 0 ¬A x C = b 0.2 0.2 0.2 0.2

c 0.2 0.6 0.6 0 c 0.4 0.4 0.4 0.4

d 0.2 1 0.8 0 d 0 0 0 0

Fuzzy Union is defined as (A ∪ B)(x) = max [A(x), B(x)] for all x ∈X

3
 Therefore R = (A x B) U (¬ A x C) gives

y 1 2 3 4

a 1 1 1 1

b 0.2 0.8 0.8 0

R=

c 0.4 0.6 0.6 0.4

d 0.2 1 0.8 0

This represents If x is A THEN y is B Else y is C

20

3
 Sc – Fuzzy System – Fuzzy logic

3 Fuzzy Quantifiers

In crisp logic, the predicates are quantified by quantifiers. Similarly, in

fuzzy logic the propositions are quantified by quantifiers. There are
two classes of fuzzy quantifiers :

– Absolutequantifiers
and − Relative quantifiers

Examples :

Absolute quantifiers Relative quantifiers

round about 250 almost

much greater than 6 about

some where around 20 most

21

3
 Sc – Fuzzy System – Fuzzification

□ Fuzzification

The fuzzification is a process of transforming crisp values into grades of

membership for linguistic terms of fuzzy sets.

The purpose is to allow a fuzzy condition in a rule to be interpreted.

Fuzzification of the car speed

medium speed fuzzy set

Example 1 : Speed X0 = 70km/h

Example 2 : Speed X0 =
Fig below shows the fuzzification of 40km/h
low and a medium speed fuzzy set.

Low Medium

1  Medium

A B
 V Low Low High V High

.8
1

.8
.6

.6
.4

.4
.2

.2
0

20 40 60 80 100 120 140

0
Speed X0 = 70km/h
10 20 30 40 50 60 70 80 90 00

Speed X0 =
40km/h
Characterizing two grades, low
and

3
the car speed to characterize a

Given car speed value X0=40km/h :

grade A(x0) = 0.6 belongs to fuzzy

Given car speed value X0=70km/h : grade
low, and grade B(x0) = 0.4 belongs
A(x0) = 0.75 belongs to fuzzy low, and grade
to fuzzy medium.
B(x0) = 0.25 belongs to fuzzy medium

Characterizing five grades, Very

low, low, medium, high and very

high speed fuzzy set

22

3
 Sc – Fuzzy System – Fuzzy Inference

Fuzzy Inference

Fuzzy Inferencing is the core element of a fuzzy system.

Fuzzy Inferencing combines - the facts obtained from the fuzzification with the
rule base, and then conducts the fuzzy reasoning process.

Fuzzy Inference is also known as approximate reasoning.

Fuzzy Inference is computational procedures used for evaluating linguistic

descriptions. Two important inferring procedures are

– Generalized Modus Ponens (GMP)

– Generalized Modus Tollens (GMT)

23

3
 Sc – Fuzzy System – Fuzzy Inference

• Generalized Modus Ponens (GMP)

This is formally stated as

If x is A THEN y is B

- is ¬A

- is ¬B

where A , B , ¬A , ¬B are fuzzy terms.

Note : Every fuzzy linguistic statements above the line is analytically

known and what is below the line is analytically unknown.

To compute the membership function ¬B , the max-min composition of

fuzzy set ¬A with R(x , y) which is the known implication relation ( IF-
THEN) is used. i.e. ¬B = ¬A ο R(x, y) In terms of membership function

 ¬B (y) = max (min (  ¬A (x) ,

R (x , y))) where

□ ¬A (x)is the membership function of ¬A ,

R (x , y) is the membership function of the implication relation and 

¬B (y) is the membership function of ¬B

24

3
 Sc – Fuzzy System – Fuzzy Inference

■ Generalized Modus Tollens (GMT)

This is formally stated as

If x is A THEN y is B

• is ¬B
x is ¬A

where A , B , ¬A , ¬B are fuzzy terms.

Note : Every fuzzy linguistic statements above the line is analytically

known and what is below the line is analytically unknown.

To compute the membership function ¬A , the max-min composition

of fuzzy set ¬B with R(x , y) which is the known implication relation

(IF-THEN) is used. i.e. ¬A = ¬B ο R(x, y)

In terms of membership function

 ¬A (y) = max (min (  ¬B (x) ,

R (x , y))) where

□ ¬B (x)is the membership function of ¬B ,

R (x , y) is the membership function of the implication relation and 

¬A (y) is the membership function of ¬A

3
 Sc – Fuzzy System – Fuzzy Inference

Example :

Apply the fuzzy Modus Ponens rules to deduce Rotation is quite slow?

Given :

■ If the temperature is high then then the rotation is slow.

■ The temperature is very high.

Let H (High) , VH (Very High) , S (Slow) and QS (Quite Slow) indicate the
associated fuzzy sets.

Let the set for temperatures be X = {30, 40, 50, 60, 70, 80, 90, 100} , and
Let the set of rotations per minute be Y = {10, 20, 30, 40, 50, 60} and

H = {(70, 1) (80, 1) (90, 0.3)}

VH = {(90, 0.9) (100, 1)}

QS = {10, 1) (20, 08) }

S = {(30, 0.8) (40, 1) (50, 0.6)

To derive R(x, y) representing the implication relation (i) above, compute

R (x, y) = max (H x S , ¬ H x Y)

10 20 30 40 50 60 10 20 30 40 50 60

30 0 0 0 0 0 0 30 1 1 1 1 1 1

40 0 0 0 0 0 0 40 1 1 1 1 1 1

50 0 0 0 0 0 0 50 1 1 1 1 1 1

3
 60 0 0 0 0 0 0 60 1 1 1 1 1 1

HxY=
HxS= 70 0 0 0.8 1 0.6 0 70 0 0 0 0 0 0

80 0 0 0.8 1 0.6 0 80 0 0 0 0 0 0

90 0 0 0.3 0.3 0.3 0 90 0.7 0.7 0.7 0.7 0.7 0.7

100 0 0 0 0 0 0 100 1 1 1 1 1 1

10 20 30 40 50 60

30 1 1 1 1 1 1

40 1 1 1 1 1 1

50 1 1 1 1 1 1

60 1 1 1 1 1 1

R(x,Y) =
70 0 0 0.8 1 0.6 0

80 0 0 0.8 1 0.6 0

90 0.7 0.7 0.7 0.7 0.7 0.7

100 1 1 1 1 1 1

To deduce Rotation is quite slow, we make use of the composition rule

QS = VH ο R (x, y)

10 20 30 40 50 60

30 1 1 1 1 1 1

40 1 1 1 1 1 1

3
 50 1 1 1 1 1 1

60 1 1 1 1 1 1
= [0 0 0 0 0 0 0.9 1] x

70 0 0 0 0 0 0

80 0 0 0 0 0 0

90 0.7 0.7 0.7 0.7 0.7 0.7

100 1 1 1 1 1 1

= [1 1 1 1 1 1 ]

• Fuzzy Rule Based System

The fuzzy linguistic descriptions are formal representation of systems made

through fuzzy IF-THEN rule. They encode knowledge about a system in
statements of the form :

IF (a set of conditions) are satisfied THEN (a set of consequents) can be inferred.

IF (x1 is A1, x2 is A2, xn is An ) THEN (y1 is B1, y2 is B2, yn is Bn)

where linguistic variables xi, yj take the values of fuzzy sets Ai and Bj
respectively.

Example :

IF
there is "heavy" rain and "strong" winds

THEN there must "severe" flood warnings.

3
 Here, heavy , strong , and severe are fuzzy sets qualifying the variables
rain,
wind, and flood warnings respectively.

A collection of rules referring to a particular system is known as a fuzzy

rule base. If the conclusion C to be drawn from a rule base R is the

conjunction of all the individual consequents C i of each rule , then

C = C1 ∩ C 2 ∩ . . . ∩ Cn where

∀
c (y ) = min ( c1(y ), c2(y ) , cn(y )) , y Y
∈

where Y is universe of discourse.

On the other hand, if the conclusion C to be drawn from a rule base R is the
disjunction of the individual consequents of each rule, then

C = C1 U C2 U . . . U Cn where

c (y ) = max (
∀
c1 (y ), c2(y ) , cn (y )) , y Y where
∈

• is universe of discourse.

28

3
• Defuzzification

In many situations, for a system whose output is fuzzy, it is easier to take a

crisp decision if the output is represented as a single quantity. This conversion
of a single crisp value is called Defuzzification.

Defuzzification is the reverse process of fuzzification.

The typical Defuzzification

methods are − Centroid method,

– Center of sums, −
Mean of maxima.

Centroid method

It is also known as the "center of gravity" of area method. It

obtains the centre of area (x*) occupied by the fuzzy set .
For discrete membership function, it is given by

Σn xi  (xi)
i=1

x* = where
n

Σ  (xi)

• represents the number elements in the sample,

and xi are the elements, and

 (xi) is the membership function.

3
 Genetic Algorithms & Modeling

What are GAs ?

• Genetic Algorithms (GAs) are adaptive heuristic search algorithm based on

the evolutionary ideas of natural selection and genetics.

• Genetic algorithms (GAs) are a part of Evolutionary computing, a rapidly

growing area of artificial intelligence. GAs are inspired by Darwin's theory
about evolution - "survival of the fittest".

• GAs represent an intelligent exploitation of a random search used to solve

optimization problems.

• GAs, although randomized, exploit historical information to direct the

search into the region of better performance within the search space.

• In nature, competition among individuals for scanty resources results in

the fittest individuals dominating over the weaker ones.

03

3
 SC – GA - Introduction

• Introduction

Solving problems mean looking for solutions, which is best among others.

Finding the solution to a problem is often thought :

– In computer science and AI, as a process of search through the space of

possible solutions. The set of possible solutions defines the search space
(also called state space) for a given problem. Solutions or partial solutions
are viewed as points in the search space.

– In engineering and mathematics, as a process of optimization. The problems

are first formulated as mathematical models expressed in terms of functions
and then to find a solution, discover the parameters that optimize the model
or the function components that provide optimal system performance.

• Why Genetic Algorithms ?

It is better than conventional AI ; It is more robust.

j unlike older AI systems, the GA's do not break easily even if the

inputs changed slightly, or in the presence of reasonable noise.

• while performing search in large state-space, multi-modal state-space,

or n-dimensional surface, a genetic algorithms offer significant benefits
over many other typical search optimization techniques like - linear
programming, heuristic, depth-first, breath-first.

"Genetic Algorithms are good at taking large, potentially huge search

spaces and navigating them, looking for optimal combinations of things,
the solutions one might not otherwise find in a lifetime.” Salvatore
Mangano Computer Design, May 1995.

3
1.1 Optimization

Optimization is a process that finds a best, or optimal, solution for a

problem. The Optimization problems are centered around three factors :

• An objective function : which is to be minimized or maximized;

Examples:

In manufacturing, we want to maximize the profit or minimize the

cost .

In designing an automobile panel, we want to maximize the

strength.

• A set of unknowns or variables : that affect the objective

function, Examples:

In manufacturing, the variables are amount of resources used or

the time spent.

In panel design problem, the variables are shape and dimensions

of the panel.

• A set of constraints : that allow the unknowns to take on certain

values but exclude others;
Examples:

1. In manufacturing, one constrain is, that all "time" variables to be

non-negative.

• In the panel design, we want to limit the weight and put constrain
on its shape.

An optimization problem is defined as : Finding values of the variables

that minimize or maximize the objective function while satisfying the
constraints.

3
• Optimization Methods

Many optimization methods exist and categorized as shown below. The

suitability of a method depends on one or more problem characteristics to
be optimized to meet one or more objectives like :

– low cost,

– high performance,
– low loss

These characteristics are not necessarily obtainable, and requires

knowledge about the problem.

Optimization

Methods

Linear Non-Linear

Programming Programming

Classical Enumerative Stochastic

Methods Methods Methods

Fig. Optimization Methods

Each of these methods are briefly discussed indicating the nature of the
problem they are more applicable.

3
■ Linear Programming

Intends to obtain the optimal solution to problems that are

perfectly represented by a set of linear equations; thus require

a priori knowledge of the problem. Here the

– the functions to be minimized or maximized, is called objective

functions,

– the set of linear equations are called restrictions.

– the optimal solution, is the one that minimizes (or maximizes) the
objective function.

Example : “Traveling salesman”, seeking a minimal traveling distance.

■ Non- Linear Programming

Intended for problems described by non-linear equations.

The methods are divided in three large groups:

Classical, Enumerative and Stochastic.

Classical search uses deterministic approach to find best solution. These

methods requires knowledge of gradients or higher order derivatives. In
many practical problems, some desired information

are not available, means deterministic algorithms are inappropriate.

The techniques are subdivide into:

– Direct methods, e.g. Newton or Fibonacci −

Indirect methods.

3
 Enumerative search goes through every point (one point at a time )
related to the function's domain space. At each point, all possible
solutions are generated and tested to find optimum solution. It is easy to
implement but usually require significant computation. In the field of
artificial intelligence, the enumerative methods are subdivided into two
categories:

– Uninformed methods, e.g. Mini-Max algorithm

– Informed methods, e.g. Alpha-Beta and A* ,

Stochastic search deliberately introduces randomness into the search

process. The injected randomness may provide the necessary impetus to
move away from a local solution when searching for a global optimum.
e.g., a gradient vector criterion for “smoothing” problems. Stochastic
methods offer robustness quality to optimization process. Among the
stochastic techniques, the most widely used are :

– Evolutionary Strategies (ES),

– Genetic Algorithms (GA), and
– Simulated Annealing (SA).

The ES and GA emulate nature’s evolutionary behavior, while SA is

based on the physical process of annealing a material.

09

3
 SC – GA - Introduction

1.2 Search Optimization

Among the three Non-Linear search methodologies, just mentioned

in the previous slide, our immediate concern is Stochastic search

which means

– Evolutionary Strategies (ES), −

Genetic Algorithms (GA), and −
Simulated Annealing (SA).

The two other search methodologies, shown below, the Classical and the
Enumerative methods, are first briefly explained. Later the Stochastic
methods are discussed in detail. All these methods belong to Non-Linear
search.

Search Optimization

Classical Stochastic Search Enumerativ

(Guided Random Search)
Search e Search

3
 Evolutionary Genetic Algorithms (GA)Simulated Annealing
Strategies (ES) (ES)

Fig Non- Linear search methods

10

3
 SC – GA - Introduction

- Classical or Calculus based search

Uses deterministic approach to find best solutions of an optimization

problem.

– the solutions satisfy a set of necessary and sufficient conditions of the

optimization problem.

– the techniques are subdivide into direct and indirect methods.

Direct or Numerical methods :

– example : Newton or Fibonacci,

– tries to find extremes by "hopping" around the search space

and assessing the gradient of the new point, which guides the
search.

– applies the concept of "hill climbing", and finds the best

local point by climbing the steepest permissible gradient.

– used only on a restricted set of "well behaved" functions.

• Indirect methods :

– does search for local extremes by solving usually non-linear

set of equations resulting from setting the gradient of the
objective function to zero.

– doessearch for possible solutions (function peaks), starts by

restricting itself to points with zero slope in all directions.

3
11

3
 SC – GA - Introduction

■ Enumerative Search

Here the search goes through every point (one point at a time) related to
the function's domain space.

– At each point, all possible solutions are generated and tested to find
optimum solution.

– It is easy to implement but usually require significant computation.

Thus these techniques are not suitable for applications with large
domain spaces.

In the field of artificial intelligence, the enumerative methods are

subdivided into two categories : Uninformed and Informed methods.

• Uninformed or blind methods :

– example: Mini-Max algorithm,

– search all points in the space in a predefined order,

– used in game playing.

• Informed methods :

– example: Alpha-Beta and A* ,

– does more sophisticated search

– uses domain specific knowledge in the form of a cost

function or heuristic to reduce cost for search.

3
 Next slide shows, the taxonomy of enumerative search in AI domain.

12

3
 SC – GA - Introduction

[Ref : previous slide Enumerative search]

The Enumerative search techniques follows, the traditional search and

control strategies, in the domain of Artificial Intelligence.

– the search methods explore the search space "intelligently"; means

evaluating possibilities without investigating every single possibility.

– there are many control structures for search; the depth-first search

and breadth-first search are two common search strategies.

– the taxonomy of search algorithms in AI domain is given below.

Enumerative Search

G (State, Operator, Cost)

No h(n) present User heuristics h(n)

LIFO Stack FIFO Priority

Uninformed Search Informed Search
Queue: g(n)

Breadth-First Cost-First Generate

Depth-First Hill

Search Search -and-test Climb

Search in g

3
 Impose fixed

Priority
depth limit

Queue: h(n)

Depth

Problem Constraint Mean-end-

Best
Limited
satisfactio n
sear Reduction analysis

Search

Priority

Queue: Gradually increase

f(n)=h(n)+g(n fixed depth limit

A* Search

AO* Search
Iterative Deepening DFS

Fig. Enumerative Search Algorithms in AI Domain

13

3
 SC – GA - Introduction

• Stochastic Search

Here the search methods, include heuristics and an element of

randomness (non-determinism) in traversing the search space. Unlike

the previous two search methodologies

– the stochastic search algorithm moves from one point to another in

the search space in a non-deterministic manner, guided by heuristics.

– the stochastic search techniques are usually called Guided

random search techniques.

The stochastic search techniques are grouped into two major subclasses :

– Simulated annealing and

– Evolutionary algorithms.

Both these classes follow the principles of evolutionary processes.

• Simulated annealing (SAs)

– uses a thermodynamic evolution process to search

minimum energy states.

• Evolutionary algorithms (EAs)

– use natural selection principles.

4
 – the search evolves throughout generations, improving the
features of potential solutions by means of biological
inspired operations.

– Genetic Algorithms (GAs) are a good example of this

technique.

The next slide shows, the taxonomy of evolutionary search algorithms. It

includes the other two search, the Enumerative search and Calculus based
techniques, for better understanding of Non-Linear search methodologies
in its entirety.

14

4
 SC – GA - Introduction

• Taxonomy of Search Optimization

Fig. below shows different types of Search Optimization algorithms.

Search Optimization

Calculus Enumerative Techniques

Guided Random Search techniques

Based Techniques

Indirect
Direct
Uninformed Informed
method
method
Search Search

Newton Finonac
ci

Tabu Hill Simulated Evolutionary

Climbin Annealing Algorithms

Searc
h g

Genetic Genetic
Programming Algorithms

4
 Fig. Taxonomy of Search Optimization techniques

We are interested in Evolutionary search algorithms.

Our main concern is to understand the evolutionary algorithms :

- how to describe the process of search,

- how to implement and carry out search,

- what are the elements required to carry out search, and

- the different search strategies

The Evolutionary Algorithms include :

- Genetic Algorithms and

- Genetic Programming

15

4
 SC – GA - Introduction

1.3 Evolutionary Algorithm (EAs)

Evolutionary Algorithm (EA) is a subset of Evolutionary Computation (EC)

which is a subfield of Artificial Intelligence (AI).

Evolutionary Computation (EC) is a general term for several computational

techniques. Evolutionary Computation represents powerful search and
optimization paradigm influenced by biological mechanisms of evolution : that
of natural selection and genetic.

Evolutionary Algorithms (EAs) refers to Evolutionary Computational

models using randomness and genetic inspired operations. EAs involve

selection, recombination, random variation and competition of the
individuals in a population of adequately represented potential solutions.
The candidate solutions are referred as chromosomes or individuals.

Genetic Algorithms (GAs) represent the main paradigm of Evolutionary

Computation.

• GAs simulate natural evolution, mimicking processes the nature uses :

Selection, Crosses over, Mutation and Accepting.

• GAs simulate the survival of the fittest among individuals over

consecutive generation for solving a problem.

Development History

EC = GP + ES + EP + GA

4
 Evolutionary Genetic Evolution Evolutionary Genetic

Computing Programming Strategie Programming Algorithms

Rechenberg Koza s Fogel Holland

Rechenberg
1960 1992 1965 1962 1970

16

4
 SC – GA - Introduction

1.4 Genetic Algorithms (GAs) - Basic Concepts

Genetic algorithms (GAs) are the main paradigm of evolutionary

computing. GAs are inspired by Darwin's theory about evolution – the
"survival of the fittest". In nature, competition among individuals for
scanty resources results in the fittest individuals dominating over the
weaker ones.

– GAs are the ways of solving problems by mimicking processes nature

uses; ie., Selection, Crosses over, Mutation and Accepting, to evolve a
solution to a problem.

– GAs are adaptive heuristic search based on the evolutionary ideas of

natural selection and genetics.

– GAs are intelligent exploitation of random search used in optimization

problems.

– GAs, although randomized, exploit historical information to direct the

search into the region of better performance within the search space.

The biological background (basic genetics), the scheme of evolutionary

processes, the working principles and the steps involved in GAs are
illustrated in next few slides.

17

4
 SC – GA - Introduction

■ Biological Background – Basic Genetics

Every organism has a set of rules, describing how that organism is

built. All living organisms consist of cells.

In each cell there is same set of chromosomes. Chromosomes are

strings of DNA and serve as a model for the whole organism.

A chromosome consists of genes, blocks of DNA.

Each gene encodes a particular protein that represents a trait

(feature), e.g., color of eyes.

Possible settings for a trait (e.g. blue, brown) are called alleles.

Each gene has its own position in the chromosome called its locus.

Complete set of genetic material (all chromosomes) is called a

genome.

Particular set of genes in a genome is called genotype.

The physical expression of the genotype (the organism itself after

birth) is called the phenotype, its physical and mental characteristics,
such as eye color, intelligence etc.

When two organisms mate they share their genes; the resultant
offspring may end up having half the genes from one parent and half
from the other. This process is called recombination (cross over) .

The new created offspring can then be mutated. Mutation means, that
the elements of DNA are a bit changed. This changes are mainly
caused by errors in copying genes from parents.

4
 The fitness of an organism is measured by success of the organism in
its life (survival).

18

4
 SC – GA - Introduction

[ continued from previous slide - Biological background ]

Below shown, the general scheme of evolutionary process in genetic along

with pseudo-code.

Parents

Parents
Initialization

Recombination

Population

Mutation

Termination

Offspring

Survivor

Fig. General Scheme of Evolutionary process

Pseudo-Code

BEGIN

INITIALISE population with random candidate solution.

EVALUATE each candidate;

4
 REPEAT UNTIL (termination condition ) is satisfied DO

■ SELECT parents;

■ RECOMBINE pairs of parents;

■ MUTATE the resulting offspring;

■ SELECT individuals or the next generation;

END.

19

4
 SC – GA - Introduction

• Search Space

In solving problems, some solution will be the best among others. The
space of all feasible solutions (among which the desired solution

resides) is called search space (also called state space).

– Each point in the search space represents one possible solution.

– Each possible solution can be "marked" by its value (or fitness) for the
problem.

– The GA looks for the best solution among a number of possible

solutions represented by one point in the search space.

– Looking for a solution is then equal to looking for some extreme value
(minimum or maximum) in the search space.

– At times the search space may be well defined, but usually only a few
points in the search space are known.

In using GA, the process of finding solutions generates other points

(possible solutions) as evolution proceeds.

20

4
 SC – GA - Introduction

- Working Principles

Before getting into GAs, it is necessary to explain few terms.

– Chromosome : a set of genes; a chromosome contains the solution in

form of genes.

– Gene : a part of chromosome; a gene contains a part of solution. It

determines the solution. e.g. 16743 is a chromosome and 1, 6, 7, 4
and 3 are its genes.

– Individual : same as chromosome.

– Population: number of individuals present with same length of

chromosome.

– Fitness : the value assigned to an individual based on how far or close a

individual is from the solution; greater the fitness value better the
solution it contains.

– Fitness function : a function that assigns fitness value to the individual.

It is problem specific.

– Breeding : taking two fit individuals and then intermingling there

chromosome to create new two individuals.

– Mutation : changing a random gene in an individual.

– Selection : selecting individuals for creating the next generation.

Working principles :

4
 Genetic algorithm begins with a set of solutions (represented by
chromosomes) called the population.

– Solutions from one population are taken and used to form a new
population. This is motivated by the possibility that the new population
will be better than the old one.

– Solutions are selected according to their fitness to form new solutions

(offspring); more suitable they are, more chances they have to
reproduce.

– This is repeated until some condition (e.g. number of populations or

improvement of the best solution) is satisfied.

21

4
 SC – GA - Introduction

■ Outline of the Basic Genetic

Algorithm

[Start] Generate random population of n chromosomes (i.e. suitable

solutions for the problem).

[Fitness] Evaluate the fitness f(x) of each chromosome x in the

population.

[New population] Create a new population by repeating following steps

until the new population is complete.

[Selection] Select two parent chromosomes from a population

according to their fitness (better the fitness, bigger the chance to
be selected)

[Crossover] With a crossover probability, cross over the parents to

form new offspring (children). If no crossover was performed,
offspring is the exact copy of parents.

[Mutation] With a mutation probability, mutate new offspring at

each locus (position in chromosome).

(d) [Accepting] Place new offspring in the new population

• [Replace] Use new generated population for a further run of the

algorithm

• [Test] If the end condition is satisfied, stop, and return the best
solution in current population

• [Loop] Go to step 2

Note : The genetic algorithm's performance is largely influenced by two

operators called crossover and mutation. These two operators are the
most important parts of GA.

4
22

4
 SC – GA - Introduction

■ Flow chart for Genetic Programming

Start
Seed Population

Generate N individuals

Genesis

Select two individuals

(Parent 1 Parent 2)

No
Natura

Selectio

Reproduction
Use crossover
operator
Recombination Crossover
to produce off-

Scoring : assign fitness

Crossov
to off- springs

er
Yes
Survival of Fittest

Apply replacement Yes No Natural Select one off-spring

operator to incorporate Selection

4
new
Apply Mutation
Mutation
operator to produce
No
individual into Mutated
offspring
Terminate?

Mutation Scoring : assign

Finished?
fitness to off- spring
Yes

Finish

Fig. Genetic Algorithm – program flow chart

23

4
 SC – GA - Encoding

◊ Encoding

Before a genetic algorithm can be put to work on any problem, a method is

needed to encode potential solutions to that problem in a form so that a
computer can process.

– One common approach is to encode solutions as binary strings: sequences of

1's and 0's, where the digit at each position represents the value of some
aspect of the solution.

Example :

A Gene represents some data (eye color, hair color, sight, etc.).

a Gene looks like : (11100010)

a Chromosome looks like: Gene1 Gene2 Gene3 Gene4

(11000010, 00001110, 001111010, 10100011)

A chromosome should in some way contain information about solution

which it represents; it thus requires encoding. The most popular way of
encoding is a binary string like :

Chromosome 1 : 1101100100110110

Chromosome 2 : 1101111000011110

Each bit in the string represent some characteristics of the solution.

– There are many other ways of encoding, e.g., encoding values as integer or
real numbers or some permutations and so on.

4
 – The virtue of these encoding method depends on the problem to work on .

24

4
 SC – GA - Encoding

■ Binary Encoding

Binary encoding is the most common to represent information contained.

In genetic algorithms, it was first used because of its relative simplicity.

– In binary encoding, every chromosome is a string of bits : 0 or 1, like

Chromosome 1: 101100101100101011100101

Chromosome 2: 111111100000110000011111

– Binary encoding gives many possible chromosomes even with a

small number of alleles ie possible settings for a trait (features).

– This encoding is often not natural for many problems and sometimes
corrections must be made after crossover and/or mutation.

Example 1:

One variable function, say 0 to 15 numbers, numeric values,

represented by 4 bit binary string.

Numeric 4–bit Numeric 4–bit Numeric 4–bit

value string value string value string

0 0000 6 0110 12 1100

1 0001 7 0111 13 1101

4
 2 0010 8 1000 14 1110

3 0011 9 1001 15 1111

4 0100 10 1010

5 0101 11 1011

25

4
 SC – GA - Encoding

[ continued binary encoding ]

Example 2 :

Two variable function represented by 4 bit string for each variable.

Let two variables X1 , X2 as (1011 0110) .

L U
Every variable will have both upper and lower limits as X i ≤ Xi ≤ X i
Because 4-bit string can represent integers from 0 to 15,

so (0000 0000) and (1111 1111) represent the points for X1 , X2 as

L L U U
( X1 , X2 ) and (X 1 ,X 2 ) respectively.

Thus, an n-bit string can represent integers from

n n
0 to 2 -1, i.e. 2 integers.

Binary Coding Equivalent

integer Decoded binary substring

210 Remainder
Let Xi is coded as a substring
1 0 1 0 Si of length ni. Then decoded
binary substring Si is as

25 0 0x2 0 =0
K=ni - 1

k
1
Σ 2 S k
22 1 1x2 =2
k=0

2 where Si can be 0 or 1 and the

1 0 0x2 =0
string S is represented as

3 Sn-1.........S3 S2 S1 S0
1x2 =8
10

Example : Decoding value

4
 Consider a 4-bit string (0111),

– the decoded value is equal to

23 x 0 + 22 x 1 + 21 x 1 + 20 x 1 = 7

L and X U corresponding to (0000) and (1111) ,

– Knowing X
i i

the equivalent value for any 4-bit string can be obtained as

U L
(Xi − Xi )
L
Xi =X i + --------------- x (decoded value of string)

ni
(2 −1)

L U
– For e.g. a variable Xi ; let X and X = 17, find what value the 4-
i= 2 ,
bit string Xi = (1010) would represent. First get decoded value for

Si = 1010 = 23 x 1 + 22 x 0 + 21 x 1 + 20 x 0 = 10 then

(17 -2)

Xi = 2 +----------------10 = 12

(24 - 1)

The accuracy obtained with a 4-bit code is 1/16 of search space.

By increasing the string length by 1-bit , accuracy increases to 1/32.

26

4
 SC – GA - Encoding

- Value Encoding

The Value encoding can be used in problems where values such as real
numbers are used. Use of binary encoding for this type of problems would
be difficult.

In value encoding, every chromosome is a sequence of some values.

The Values can be anything connected to the problem, such as

: real numbers, characters or objects.

Examples :

Chromosome A 1.2324 5.3243 0.4556 2.3293 2.4545

Chromosome B ABDJEIFJDHDIERJFDLDFLFEGT

Chromosome C (back), (back), (right), (forward), (left)

- Value encoding is often necessary to develop some new types

of crossovers and mutations specific for the problem.

27

4
 SC – GA - Encoding

- Permutation Encoding

Permutation encoding can be used in ordering problems, such as traveling

salesman problem or task ordering problem.

In permutation encoding, every chromosome is a string of numbers

that represent a position in a sequence.

Chromosome A 153264798

Chromosome B 856723149

□ Permutation encoding is useful for ordering problems. For some

problems, crossover and mutation corrections must be made to leave
the chromosome consistent.

Examples :

1. The Traveling Salesman problem:

There are cities and given distances between them. Traveling

salesman has to visit all of them, but he does not want to travel more
than necessary. Find a sequence of cities with a minimal traveled
distance. Here, encoded chromosomes describe the order of cities the
salesman visits.

2. The Eight Queens problem :

There are eight queens. Find a way to place them on a chess board so
that no two queens attack each other. Here, encoding describes the
position of a queen on each row.

4
28

4
 SC – GA - Encoding

• Tree Encoding

Tree encoding is used mainly for evolving programs or

expressions. For genetic programming :

– In tree encoding, every chromosome is a tree of some objects, such as

functions or commands in programming language.

– Tree encoding is useful for evolving programs or any other structures

that can be encoded in trees.

– The crossover and mutation can be done relatively easy way .

Example :

Chromosome A Chromosome B

do untill

x /

step wall

5 y

(+x(/5y)) ( do until step wall )

Fig. Example of Chromosomes with tree encoding

4
 Note : Tree encoding is good for evolving programs. The programming
language LISP is often used. Programs in LISP can be easily parsed as a
tree, so the crossover and mutation is relatively easy.

29

4
 SC – GA - Operators

3. Operators of Genetic
Algorithm

Genetic operators used in genetic algorithms maintain genetic diversity.

Genetic diversity or variation is a necessity for the process of evolution.

Genetic operators are analogous to those which occur in the natural world:

– Reproduction (or Selection) ;

– Crossover (or Recombination); and

– Mutation.

In addition to these operators, there are some parameters of GA.

One important parameter is Population size.

– Population size says how many chromosomes are in population (in one
generation).

– If there are only few chromosomes, then GA would have a few possibilities
to perform crossover and only a small part of search space is explored.

– If there are many chromosomes, then GA slows down.

– Research shows that after some limit, it is not useful to increase population
size, because it does not help in solving the problem faster. The population
size depends on the type of encoding and the problem.

4
30

4
 SC – GA - Operators

3.1 Reproduction, or Selection

Reproduction is usually the first operator applied on population. From the

population, the chromosomes are selected to be parents to crossover and
produce offspring.

The problem is how to select these chromosomes ?

According to Darwin's evolution theory "survival of the fittest" – the best

ones should survive and create new offspring.

– The Reproduction operators are also called Selection operators.

– Selection means extract a subset of genes from an existing population,

according to any definition of quality. Every gene has a meaning, so
one can derive from the gene a kind of quality measurement called
fitness function. Following this quality (fitness value), selection can be
performed.

– Fitness function quantifies the optimality of a solution (chromosome) so

that a particular solution may be ranked against all the other solutions.
The function depicts the closeness of a given ‘solution’ to the desired
result.

Many reproduction operators exists and they all essentially do same thing.
They pick from current population the strings of above average and insert
their multiple copies in the mating pool in a probabilistic manner.

The most commonly used methods of selecting chromosomes for parents

to crossover are :

4
 – Roulette wheel
selection, – Rank selection
– Boltzmann selection, −
Tournament selection, – Steady state selection.

The Roulette wheel and Boltzmann selections methods are illustrated next.

31

4
 SC – GA - Operators

• Example of Selection

Evolutionary Algorithms is
to maximize the function f(x) = x2 with x in
i.e., x = 0, 1, . . . 30, 31.
the integer interval [0 , 31],

• The first step is encoding of chromosomes; use binary representation

for integers; 5-bits are used to represent integers up to 31.

• Assume that the population size is 4.

• Generate initial population at random. They are chromosomes or

genotypes; e.g., 01101, 11000, 01000, 10011.

• Calculate fitness value for each individual.

• Decode the individual into an integer (called phenotypes),

01101 → 13; 11000 → 24; 01000 → 8; 10011 → 19;

— Evaluate the fitness according to f(x) = x2 ,

13 → 169; 24 → 576; 8 → 64; 19 → 361.

4. Select parents (two individuals) for crossover based on their fitness in

pi. Out of many methods for selecting the best chromosomes, if
th
roulette-wheel selection is used, then the probability of the i string
n

in the population is pi = F i / (Σ Fj), where

j=1

F i is fitness for the string i in the population, expressed as f(x)

pi is probability of the string i being selected,

4
 4. is no of individuals in the population, is population size, n=4
n * pi is expected count

String No Initial X value Fitness Fi p i Expected count

Population f(x) = x2 N * Prob i

1 01101 13 169 0.14 0.58

2 11000 24 576 0.49 1.97

3 01000 8 64 0.06 0.22

4 10011 19 361 0.31 1.23

Sum 1170 1.00 4.00

Average 293 0.25 1.00

Max 576 0.49 1.97

The string no 2 has maximum chance of selection.

32

4
 SC – GA - Operators

• Roulette wheel selection (Fitness-Proportionate Selection)

Roulette-wheel selection, also known as Fitness Proportionate Selection, is

a genetic operator, used for selecting potentially useful solutions for
recombination.

In fitness-proportionate selection :

– the chance of an individual's being selected is proportional to its

fitness, greater or less than its competitors' fitness.

– conceptually, this can be thought as a game of Roulette.

Fig. Roulette-wheel
Shows 8

individual with fitness

1

5%
8 2

20% 9%

13%
7

8%

8%

17%

20%

4
 – the fitness of the individuals is

The Roulette-wheel simulates 8 individuals

with fitness values Fi, marked at its calculated as the wheel is spun

circumference; e.g., n = 8 times, each time selecting

an instance, of the string, chosen

th
– the 5 individual has a higher
by the wheel pointer.
fitness than others, so the wheel

th
would choose the 5 individual more
than other individuals .

th n
Probability of i string is pi = F i / (Σ F j ) , where

j=1

th
• = no of individuals, called population size; pi = probability of i
th
string being selected; Fi = fitness for i string in the population.
Because the circumference of the wheel is marked according

to a string's fitness,

the Roulette-wheel mechanism is expected

to

F
th
make copies of the i string.

Average fitness = F Fj/n ; Expected count = (n =8 ) x pi

N=5

Cumulative Probability5 = Σ pi

i=1

33

4
 SC – GA - Operators

• Boltzmann Selection

Simulated annealing is a method used to minimize or maximize a function.

– This method simulates the process of slow cooling of molten metal to

achieve the minimum function value in a minimization problem.

– The cooling phenomena is simulated by controlling a temperature

like parameter introduced with the concept of Boltzmann probability
distribution.

– The system in thermal equilibrium at a temperature T has its energy

distribution based on the probability defined by

P(E) = exp ( - E / kT ) were k is Boltzmann constant.

– This expression suggests that a system at a higher temperature has

almost uniform probability at any energy state, but at lower

temperature it has a small probability of being at a higher energy state.

– Thus, by controlling the temperature T and assuming that the search

process follows Boltzmann probability distribution, the convergence
of the algorithm is controlled.

34

4
 SC – GA - Operators

3.2 Crossover

Crossover is a genetic operator that combines (mates) two chromosomes

(parents) to produce a new chromosome (offspring). The idea behind
crossover is that the new chromosome may be better than both of the
parents if it takes the best characteristics from each of the parents.
Crossover occurs during evolution according to a user-definable crossover
probability. Crossover selects genes from parent chromosomes and
creates a new offspring.

The Crossover operators are of many types.

– one simple way is, One-Point crossover.

– the others are Two Point, Uniform, Arithmetic, and Heuristic crossovers.

The operators are selected based on the way chromosomes are encoded.

35

4
 SC – GA - Operators

• One-Point Crossover

One-Point crossover operator randomly selects one crossover point and

then copy everything before this point from the first parent and then
everything after the crossover point copy from the second parent. The
Crossover would then look as shown below.

Consider the two parents selected for crossover.

Parent 1 11011|00100110110

Parent 2 11011|11000011110

Interchanging the parents chromosomes after the crossover points -

The Offspring produced are :

Offspring 1 1 1 0 1 1 | 1 1 0 0 0 0 1 1 1 1 0

Offspring 2 1 1 0 1 1 | 0 0 1 0 0 1 1 0 1 1 0

Note : The symbol, a vertical line, | is the chosen crossover point.

36

4
 SC – GA - Operators

► Two-Point Crossover

Two-Point crossover operator randomly selects two crossover points within

a chromosome then interchanges the two parent chromosomes between
these points to produce two new offspring.

Consider the two parents selected for crossover :

Parent 1 11011|0010011|0110

Parent 2 11011|1100001|1110

Interchanging the parents chromosomes between the crossover points -

The Offspring produced are :

Offspring 1 1 1 0 1 1 | 0 0 1 0 0 1 1 | 0 1 1 0

Offspring 2 1 1 0 1 1 | 0 0 1 0 0 1 1 | 0 1 1 0

37

4
 SC – GA - Operators

► Uniform Crossover

Uniform crossover operator decides (with some probability – know as the

mixing ratio) which parent will contribute how the gene values in the
offspring chromosomes. The crossover operator allows the parent
chromosomes to be mixed at the gene level rather than the segment level
(as with one and two point crossover).

Consider the two parents selected for crossover.

Parent 1 1 1 0 1 1 0 0 1 0 0 1 1 0 1 1 0

Parent 2 1 1 0 1 1 1 1 0 0 0 0 1 1 1 1 0

If the mixing ratio is 0.5 approximately, then half of the genes in the
offspring will come from parent 1 and other half will come from parent 2.
The possible set of offspring after uniform crossover would be:

Offspring 1 11 12 02 11 11 12 12 02 01 01 02 11 12 11 11 02

Offspring 2 12 11 01 12 12 01 01 11 02 02 11 12 01 12 12 01

Note: The subscripts indicate which parent the gene came from.

38

4
 SC – GA - Operators

• Arithmetic

Arithmetic crossover operator linearly combines two parent chromosome

vectors to produce two new offspring according to the equations:

Offspring1 = a * Parent1 + (1- a) * Parent2

Offspring2 = (1 – a) * Parent1 + a * Parent2

where a is a random weighting factor chosen before each crossover

operation.

Consider two parents (each of 4 float genes) selected for crossover:

Parent 1 (0.3) (1.4) (0.2) (7.4)

Parent 2 (0.5) (4.5) (0.1) (5.6)

Applying the above two equations and assuming the weighting factor a =
0.7, applying above equations, we get two resulting offspring. The possible
set of offspring after arithmetic crossover would be:

Offspring 1 (0.36) (2.33) (0.17) (6.87)

Offspring 2 (0.402) (2.981) (0.149) (5.842)

39

4
 SC – GA - Operators

ii Heuristic

Heuristic crossover operator uses the fitness values of the two parent
chromosomes to determine the direction of the search.

The offspring are created according to the equations:

Offspring1 = BestParent + r * (BestParent − WorstParent)

Offspring2 = BestParent

where r is a random number between 0 and 1.

It is possible that offspring1 will not be feasible. It can happen if r is

chosen such that one or more of its genes fall outside of the allowable
upper or lower bounds. For this reason, heuristic crossover has a user
defined parameter n for the number of times to try and find an r that
results in a feasible chromosome. If a feasible chromosome is not
produced after n tries, the worst parent is returned as offspring1.

40

4
 SC – GA - Operators

3.3 Mutation

After a crossover is performed, mutation takes place.

Mutation is a genetic operator used to maintain genetic diversity from one

generation of a population of chromosomes to the next.

Mutation occurs during evolution according to a user-definable mutation

probability, usually set to fairly low value, say 0.01 a good first choice.

Mutation alters one or more gene values in a chromosome from its initial
state. This can result in entirely new gene values being added to the gene
pool. With the new gene values, the genetic algorithm may be able to
arrive at better solution than was previously possible.

Mutation is an important part of the genetic search, helps to prevent the

population from stagnating at any local optima. Mutation is intended to
prevent the search falling into a local optimum of the state space.

The Mutation operators are of many type.

– one simple way is, Flip Bit.

– the others are Boundary, Non-Uniform, Uniform, and Gaussian.

The operators are selected based on the way chromosomes are encoded .

4
41

4
 SC – GA - Operators

■ Flip Bit

The mutation operator simply inverts the value of the chosen gene. i.e. 0
goes to 1 and 1 goes to 0.

This mutation operator can only be used for binary genes.

Consider the two original off-springs selected for mutation.

Original offspring 1 1 1 0 1 1 1 10 0 0 0 1 1 1 1 0

Original offspring 2
1 1 0 1 1 0 01 0 0 1 1 0 1 1 0

Invert the value of the chosen gene as 0 to 1 and 1 to 0

The Mutated Off-spring produced are :

Mutated offspring 1
1 1 0 0 1 1 10 0 0 0 1 1 1 1 0

Mutated offspring 2
1 1 0 1 1 0 11 0 0 1 1 0 1 0 0

42

4
 SC – GA - Operators

• Boundary

The mutation operator replaces the value of the chosen gene with either
the upper or lower bound for that gene (chosen randomly).

This mutation operator can only be used for integer and float genes.

• Non-Uniform

The mutation operator increases the probability such that the amount of
the mutation will be close to 0 as the generation number increases. This
mutation operator prevents the population from stagnating in the early
stages of the evolution then allows the genetic algorithm to fine tune the
solution in the later stages of evolution.

This mutation operator can only be used for integer and float genes.

• Uniform

The mutation operator replaces the value of the chosen gene with a
uniform random value selected between the user-specified upper and
lower bounds for that gene.

This mutation operator can only be used for integer and float genes.

• Gaussian

The mutation operator adds a unit Gaussian distributed random value to

the chosen gene. The new gene value is clipped if it falls outside of the
user-specified lower or upper bounds for that gene.

This mutation operator can only be used for integer and float genes.

■ Basic Genetic Algorithm :

4
Examples to demonstrate and explain : Random population, Fitness, Selection,
Crossover, Mutation, and Accepting.

Example 1 :

Maximize the function f(x) = x2 over the range of integers from 0 . . . 31.

Note : This function could be solved by a variety of traditional methods

such as a hill-climbing algorithm which uses the derivative. One way is to
:

– Start from any integer x in the domain of f

– Evaluate at this point x the derivative f’

– Observing that the derivative is +ve, pick a new x which is at a small

distance in the +ve direction from current x

– Repeat until x = 31

See, how a genetic algorithm would approach this problem ?

Genetic Algorithm approach to problem - Maximize the function f(x) = x2

1. Devise a means to represent a solution to the problem :

Assume, we represent x with five-digit unsigned binary integers.

2. Devise a heuristic for evaluating the fitness of any particular solution :

The function f(x) is simple, so it is easy to use the f(x) value itself to rate
the fitness of a solution; else we might have considered a more simpler
heuristic that would more or less serve the same purpose.

3. Coding - Binary and the String length :

4
 GAs often process binary representations of solutions. This works well,
because crossover and mutation can be clearly defined for binary solutions. A
Binary string of length 5 can represents 32 numbers (0 to 31).

4. Randomly generate a set of solutions :

Here, considered a population of four solutions. However, larger populations

are used in real applications to explore a larger part of the search. Assume,
four randomly generated solutions as : 01101, 11000, 01000, 10011. These are
chromosomes or genotypes.

5. Evaluate the fitness of each member of the population :

The calculated fitness values for each individual are -

(a) Decode the individual into an integer (called phenotypes),

11000
01101 → 13; → 24; 01000 → 8; 10011 → 19;

(b) Evaluate the fitness according to f(x) = x 2 ,

13
24
→ 169;
→ 576; 8 → 64; 19 → 361.

(c) Expected count = N * Prob i , where N is the number of

individuals in the population called population size, here N = 4.

Thus the evaluation of the initial population summarized in table below .

String No Initial X value Fitness Prob i Expected

i Population (Pheno (fraction count N *

f(x) = x2
(chromosome) types) of total) Prob i

1 01101 13 169 0.14 0.58

2 11000 24 576 0.49 1.97

3 01000 8 64 0.06 0.22

4 10011 19 361 0.31 1.23

Total (sum) 1170 1.00 4.00

Average 293 0.25 1.00

4
 Max 576 0.49 1.97

Thus, the string no 2 has maximum chance of selection.

SC – GA - Examples

6. Produce a new generation of solutions by picking from the existing pool

of solutions with a preference for solutions which are better suited than
others:

We divide the range into four bins, sized according to the relative fitness of
the solutions which they represent.

Strings Prob i Associated Bin

01101 0.14 0.0 . . . 0.14

11000 0.49 0.14 . . . 0.63

01000 0.06 0.63 . . . 0.69

10011 0.31 0.69 . . . 1.00

By generating 4 uniform (0, 1) random values and seeing which bin they fall
into we pick the four strings that will form the basis for the next generation.

Random No Falls into bin Chosen string

0.08 0.0 . . . 0.14 01101

0.24 0.14 . . . 0.63 11000

0.52 0.14 . . . 0.63 11000

0.87 0.69 . . . 1.00 10011

7. Randomly pair the members of the new generation

Random number generator decides for us to mate the first two strings
together and the second two strings together.

8. Within each pair swap parts of the members solutions to create

offspring which are a mixture of the parents :

4
 For the first pair of strings: 01101 , 11000

– We randomly select the crossover point to be after the fourth digit.

Crossing these two strings at that point yields:

01101 ➺ 0 1 1 0 |1 ➺ 01100

11000 ➺ 1 1 0 0 |0 ➺ 11001

For the second pair of strings: 11000 , 10011

– We randomly select the crossover point to be after the second digit.

Crossing these two strings at that point yields:

11000 ➺ 1 1 |0 0 0 ➺ 11011

10011 ➺ 1 0 |0 1 1 ➺ 10000
9. Randomly mutate a very small fraction of genes in the population :

With a typical mutation probability of per bit it happens that none of the bits
in our population are mutated.

10. Go back and re-evaluate fitness of the population (new generation) :

This would be the first step in generating a new generation of solutions.

However it is also useful in showing the way that a single iteration of the
genetic algorithm has improved this sample.

String No Initial X value Fitness Prob i Expected count

Population (Pheno f(x) = x2 (fraction

(chromosome) types) of total)

1 01100 12 144 0.082 0.328

2 11001 25 625 0.356 1.424

3 11011 27 729 0.415 1.660

4 10000 16 256 0.145 0.580

4
 Total (sum) 1754 1.000 4.000

Average 439 0.250 1.000

Max 729 0.415 1.660

Observe that :

1. Initial populations : At start step 5 were

01101,11000, 01000,10011

After one cycle, new populations, at step 10 to act as initial population

01100,11001, 1 1 0 11 , 10000

■ The total fitness has gone from 1170 to 1754 in a single generation.

■ The algorithm has already come up with the string 11011 (i.e x = 27) as
a possible solution.

■ Example 2 : Two bar pendulum

Two uniform bars are connected by pins at A and B and supported

at A. Let a
horizontal force P acts at C.

A
Given : Force P = 2, Length of bars ℓ1 = 2 ,

y ℓ2 = 2, Bar weights W1= 2, W2 = 2 . angles = Xi

ℓ1
θ 1
Find : Equilibrium configuration of the system if

B fiction at all joints are neglected ?

W1
ℓ2 C
θ
2 P Solution : Since there are two unknowns θ 1 and

θ 2, we use 4 – bit binary for each unknown.

U
X - XL 90 - 0

4
 W2 Accuracy = ----------- =------------= 60

24 - 1 15
Fig. Two bar pendulum

Hence, the binary coding and the corresponding angles Xi are given as

U L
Xi - Xi

L
Xi = Xi +--------------Si th
where Si is decoded Value of the i chromosome.
24 - 1

e.g. the 6th chromosome binary code (0 1 0 1) would have the corresponding

angle given by Si = 0 1 0 1 = 23 x 0 + 22 x 1 + 21 x 0 + 20 x 1 = 5

90 - 0

Xi = 0 + ----------- x 5 = 30

15

The binary coding and the angles are given in the table below.

S. No. Binary code Angle S. No. Binary code Angle

Si Xi Si Xi

1 0000 0 9 1000 48

2 0001 6 10 1001 54

3 0010 12 11 1010 60

4 0011 18 12 1011 66

5 0100 24 13 1100 72

6 0101 30 14 1101 78

7 0110 36 15 1110 84

8 0111 42 16 1111 90

Note : The total potential for two bar pendulum is written as

(c) = - P[(ℓ1 sinθ 1 + ℓ2 sinθ 2 )] - (W1 ℓ1 /2)cosθ 1 - W2 [(ℓ2 /2) cosθ 2 + ℓ1 cosθ 1]
(Eq.1)

Substituting the values for P, W1 , W2 , ℓ1 , ℓ2 all as 2 , we get ,

∏ (θ 1 , θ 2 = - 4 sinθ 1 - 6 cosθ - 4 sinθ - 2 cosθ

) 1 2 2 = function f (Eq. 2)

4
 θ1,θ2
lies between 0 and 90 both inclusive ie 0 ≤ θ 1 , θ 2 ≤ 90 (Eq. 3)

Equilibrium configuration is the one which makes ∏ a minimum .

Since the objective function is –ve , instead of minimizing the function f let us
maximize -f = f ’ . The maximum value of f ’ = 8 when θ 1 and θ 2 are zero.

Hence the fitness function F is given by F=–f–7=f’–7 (Eq. 4)

48

4
First randomly generate 8 population with 8 bit strings as shown in table below.

Population Population of 8 bit Corresponding Angles F=–f–7

No. strings (Randomly (from table above)

generated) θ1 , θ2

1 0000 0000 0 0 1

2 0010 0000 12 6 2.1

3 0001 0000 6 30 3.11

4 0010 1000 12 48 4.01

5 0110 1010 36 60 4.66

6 1110 1000 84 48 1.91

7 1110 1101 84 78 1.93

8 0111 1100 42 72 4.55

These angles and the corresponding to fitness function are shown below.

F=1 F=2.1 F=3.11 F=3.11

θ 1=0 θ 1=12 θ 1=6 θ 1=12

θ 2=0 θ 2=6 θ 2=30 θ 2=48

F=4.6 F=1.91 F=1.93 F=4.55

θ 1=36 θ 1=84 θ 1=84 θ 1=42

θ 2=60 θ 2=48 θ 2=78 θ 2=72

Fig. Fitness function F for various population

The above Table and the Fig. illustrates that :

4
 – GA begins with a population of random strings.

– Then, each string is evaluated to find the fitness value.

– The population is then operated by three operators –

– The new population is further evaluated tested for termination.

– If the termination criteria are not met, the population is iteratively operated by
the three operators and evaluated until the termination criteria are met.

– One cycle of these operation and the subsequent evaluation procedure

is known as a Generation in GA terminology.

49

4
 Hybrid Systems

Integration of NN FL GA

What is Hybridization ?

• Hybrid systems employ more than one technology to solve a problem.

• Hybridization of technologies can have pitfalls and therefore need to

be done with care.

— If one technology can solve a problem then a hybrid technology

ought to be used only if its application results in a better solution.

• Hybrid systems have been classified as :

– Sequential hybrid system: the technologies are used in pipelining

fashion;

– Auxiliary hybrid system: the one technology calls the other technology
as subroutine;

– Embedded hybrid system : the technologies participating appear to be

fused totally.

• Hybridization of fuzzy logic, neural networks, genetic algorithms has led

to creation of a perspective scientific trend known as soft computing.

– Neural networks mimic our ability to adapt to circumstances and learn

from past experience,

4
 – Fuzzy logic addresses the imprecision or vagueness in input and output,
– Genetic algorithms are inspired by biological evolution, can

systemize random search and reach to optimum characteristics.

• Each of these technologies have provided efficient solution to wide range

of problems belonging to different domains. However, each of these
technologies has advantages and disadvantages. It is therefore
appropriate that Hybridization of these three technologies are done so as
to over come the weakness of one with the strength of other.

□ Introduction :

Hybridization - Integration of NN , FL , and GA

Fuzzy logic, Neural networks and Genetic algorithms are soft computing
methods which are inspired by biological computational processes and nature's
problem solving strategies.

Neural Networks (NNs) are highly simplified model of human nervous system
which mimic our ability to adapt to circumstances and learn from past experience.
Neural Networks systems are represented by different architectures like single and
multilayer feed forward network. The networks offers back proposition
generalization, associative memory and adaptive resonance theory.

Fuzzy logic addresses the imprecision or vagueness in input and output

description of the system. The sets have no crisp boundaries and provide a
gradual transition among the members and non-members of the set elements.

Genetic algorithms are inspired by biological evolution, can systemize random

search and reach to optimum characteristics.

4
Each of these technologies have provided efficient solution to wide range of
problems belonging to different domains. However, each of these technologies
suffer from advantages and disadvantages.

It is therefore appropriate that Hybridization of these three technologies are

done so as to over come the weakness of one with the strength of other.

1.1 Hybrid Systems

Hybrid systems employ more than one technology to solve a problem.

Hybridization of technologies can have pitfalls and therefore need

to be done with care. If one technology can solve a problem then

a hybrid technology ought to be used only if its application results

in a better solution. Hybrid systems have been classified as

Sequential , Auxiliary and Embedded.

In Sequential hybrid system, the technologies are used in

pipelining fashion.

In Auxiliary hybrid system, one technology calls the other technology as

subroutine.

In Embedded hybrid system, the technologies participating appear to be

fused totally.

• Sequential Hybrid System

In Sequential hybrid system, the technologies are used in pipelining

4
 fashion. Thus, one technology's output becomes another technology's
input and it goes on. However, this is one of the weakest form of
hybridization since an integrated combination of technologies is not
present.

Example: A Genetic algorithm preprocessor obtains the optimal

parameters for different instances of a problem and hands over the
preprocessed data to a neural network for further processing.

SC – Hybrid Systems - Introduction

• Auxiliary Hybrid System

In Auxiliary hybrid system, one technology calls the other technology as

subroutine to process or manipulate information needed. The second
technology processes the information provided by the first and hands it
over for further use. This type of hybridization is better than the
sequential hybrids.

Example : A neuron-genetic system in which a neural network employs a

genetic algorithm to optimize its structural parameters that defines its
architecture.

- Embedded Hybrid System

In Embedded hybrid system, the technologies participating are integrated

in such a manner that they appear intertwined. The fusion is so complete
that it would appear that no technology can be used without the others for
solving the problem.

Example : A NN-FL hybrid system may have an NN which receives fuzzy

inputs, processes it and extracts fuzzy outputs as well.

4
1.2 Neural Networks, Fuzzy Logic, and Genetic Algorithms Hybrids

Neural Networks, Fuzzy Logic, and Genetic Algorithms are three

distinct technologies.

Each of these technologies has advantages and disadvantages. It is

therefore appropriate that hybridization of these three technologies are

done so as to over come the weakness of one with the strength of

other.

■ Neuro-Fuzzy Hybrid

Neural Networks and Fuzzy logic represents two distinct methodologies to

deal with uncertainty. Each of these has its own merits and demerits.

Neural Networks :

– Merits : Neural Networks, can model complex nonlinear relationships

and are appropriately suited for classification phenomenon into
predetermined classes.

– Demerits : Neural Network's output, precision is often limited to least

squares errors; the training time required is quite large; the training
data has to be chosen over entire range where the variables are
expected to change.

Fuzzy logic :

4
 – Merits : Fuzzy logic system, addresses the imprecision of inputs and
outputs defined by fuzzy sets and allow greater flexibility in formulating
detail system description.

Integration of NN and FL, called Neuro-Fuzzy systems, have the potential to

extend the capabilities of the systems beyond either of these two
technologies applied individually. The integrated systems have turned out to
be useful in :

– accomplishing mathematical relationships among many variables in

a complex dynamic process,

– performing mapping with some degree of imprecision, and

– controlling nonlinear systems to an extent not possible with

conventional linear control systems.

There are two ways to do hybridization :

– One, is to provide NNs with fuzzy capabilities, there by increasing the

network's expressiveness and flexibility to adapt to uncertain
environments.

– Second, is to apply neuronal learning capabilities to fuzzy systems so

that the fuzzy systems become more adaptive to changing
environments. This method is called NN driven fuzzy reasoning.

- Neuro-Genetic Hybrids

The Neural Networks and Genetic Algorithms represents two distinct

methodologies.

4
Neural Networks : can learn various tasks from examples, classify
phenomena and model nonlinear relationships.

Genetic Algorithms : have offered themselves as potential candidates for the

optimization of parameters of NN.

Integration of GAs and NNs has turned out to be useful.

– Genetically evolved nets have reported comparable results against

their conventional counterparts.

– The gradient descent learning algorithms have reported difficulties

in leaning the topology of the networks whose weights they optimize.

– GA based algorithms have provided encouraging results especially

with regard to face recognition, animal control, and others.

– Genetic algorithms encode the parameters of NNs as a string of

properties of the network, i.e. chromosomes. A large population of
chromosomes representing many possible parameters sets, for the
given NN, is generated.

– GA-NN is also known as GANN have the ability to locate the

neighborhood of the optimal solution quicker than other conventional
search strategies.

– The drawbacks of GANN algorithms are : large amount of memory

required to handle and manipulate chromosomes for a given network;
the question is whether this problem scales as the size of the networks
become large.

4
 Genetic Hybrids
Fuzzy
-

Fuzzy systems have been integrated with GAs.

The fuzzy systems like NNs (feed forward) are universal approximator in
the sense that they exhibit the capability to approximate general nonlinear
functions to any desired degree of accuracy.

4
 The adjustments of system parameters called for in the process, so that
the system output matches the training data, have been tackled using
GAs. Several parameters which a fuzzy system is involved with like
input/output variables and the membership function that define the fuzzy
systems, have been optimized using GAs.

1.3 Typical Hybrid Systems

The Systems considered are listed below.

1. Genetic algorithm based back

propagation network (Neuro Genetic
Hybrid)
2. Fuzzy back propagation network
(Neuro – Fuzzy Hybrid with Multilayer Feed forward Network as the
host architecture)

3. Simplified Fuzzy ARTMAP

(Neuro – Fuzzy Hybrid with Recurrent Network as the host architecture)

4. Fuzzy Associative Memory

( Neuro – Fuzzy Hybrid with single layer Feed forward architecture)

5. Fuzzy logic controlled Genetic

algorithm (Fuzzy – Genetic Hybrid)

• Genetic Algorithm (GA) based Back Propagation Network (BPN)

Neural networks (NNs) are the adaptive system that changes its structure based
on external or internal information that flows through the network. Neural network
solve problems by self-learning and self-organizing.

Back Propagation Network (BPN) is a method of training multi-layer neural

networks. Here learning occurs during this training phase.

4
The steps involved are:

– The pattern of activation arriving at the output layer is compared with the

correct output pattern to calculate an error signal.

– The error signal is then back-propagated from output to input for

adjusting the weights in each layer of the BPN.

– The Back-Propagation searches on the error surface using gradient descent

method to minimize error E = 1/2 Σ ( T j – O j )2 where T j is target output and

O j is the calculated output by the network.

Limitations of BPN :

– BPN can recognize patterns similar to those they have learnt, but do not
have the ability to recognize new patterns.

– BPN must be sufficiently trained to extract enough general features

applicable to both seen and unseen; over training to network may have
undesired effects.

Genetic Algorithms (GAs) are adaptive search and optimization algorithms, mimic
the principles of nature.

– GAs are different form traditional search and

– Optimization exhibit simplicity, ease of operation, minimal requirements, and

global perspective.

Hybridization of BPN and GAs

– The BPN determines its weight based on gradient search technique and

therefore it may encounter a local minima problem.

4
 – GAs do not guarantee to find global optimum solution, but are good in
finding quickly good acceptable solution.

– Therefore, hybridization of BPN and GAs are expected to provide many

advantages compare to what they alone can.

The GA based techniques for determining weights in a BPN are explained next.

15

4
2.1 GA based techniques for determining weights in a BPN

Genetic algorithms work with population of individual strings.

The steps involved in GAs are:

– each individual string represent a possible solution of the problem

considered,

– each individual string is assigned a fitness value,

– high fit individuals participate in reproduction, yields new strings as

offspring and they share some features with each parents,

– low fit individuals are kept out from reproduction and so die,

– a whole new population of possible solutions to the problem is

generated by selecting high fit individuals from current generation,

– this new generation contains characteristics which are better than

their ancestors,

– processing this way after many generation, the entire population

inherits the best and fit solution.

However, before a GA is executed :

– a suitable coding for the problem is devised,

4
 – a fitness function is formulated,

– parents have to be selected for reproduction and crossover to generate

offspring.

All these aspects of GAs for determining weights of BPN are illustrated in
next few slides.

16

4
 SC – Hybrid Systems – GA based BPN

• Coding

Assume a BPN configuration ℓ - m – n where

– ℓ is input , m is hidden and n is output neurons.

– number of weights to be determined are (ℓ + n) m.

– each weight (gene) is a real number.

– assume number of digits (gene length) in weight are d .

– a string S represents weight matrices of input-hidden and the hidden-

output layers in a linear form arranged as row-major or column-major
selected.

– population size is the randomly generated initial population of p

chromosomes.

Example :

Consider a BPN configuration ℓ - m – n where ℓ = 2 is input , m = 2 is

hidden and n = 2 is output neuron.

Input neuron Hidden neurons output neurons − number of weights is (ℓ + n) m

W11 V11 = ( 2 + 2) . 2 = 8

1 1 1
– each weight is real number and
W12 V12

W21 V21 assume number of digits in

Inputs Output
s

W22 V22 weight are d=5

4
 2 – string S representing
2 2

Input layer Hidden layer output layer chromosome of weights is 8 x 5

= 40 in length
Fig. BPN with 2 – 2 - 2

– Choose a population size p = 40

ie choose 40 chromosomes

Gene Gene Gene Gene Gene Gene Gene Gene

← k=0 k=1 → k=3 → k=4 → k=5

→ ← ← k=2 → ←← ← → ← k=6 → ← k=7 →

84321 46234 78901 32104 42689 63421 46421 87640

Chromosome

32478 76510 02461 84753 64321 14261 87654 12367

Chromosome

Fig. Some randomly generated chromosome made of 8 genes

representing 8 weights for BPN

17

4
 SC – Hybrid Systems – GA based BPN

• Weight Extraction

Extract weights from each chromosomes, later to determine the fitness

values.

Let x1 , x2 , . . . . x d , .... x L represent a chromosome and

th
Let xkd+1 , xkd+2 , . . x(k + 1)d represent k gene (k ≥ 0) in the chromosomes.

The actual weight wk is given by

d-2 d-3
xkd+2 10 + xkd +3 10 + . . . + x(k + 1)d , if 5 ≤ xkd +1 ≤ 9

+ 10d-2

wk =
d-2 d-3
xkd +2 10 + xkd +3 10 + . . . + x(k + 1)d , if 0 ≤ xkd +1 < 5

10d-2

Example : [Ref Fig. BPN previous slide]

The Chromosomes are stated in the Fig. The weights extracted from all

the eight genes are :

• Gene 0 : 84321 ,

Here we have, k = 0 , d = 5 , and xkd +1 is x1 such that

5 ≤ x1 = 8 ≤ 9. Hence, the weight extracted is

4 x 103 + 3 x 102 + 2 x 10 + 1
W0 = + 3 = +4.321
10

■ Gene 1 : 46234 ,

Here we have, k = 1 , d = 5 , and xkd +1 is x6 such that

4
 0 ≤ x6 = 4 ≤ 5. Hence, the weight extracted is

6 x 103 + 2 x 102 + 3 x 10 + 4

W1 = −
103 = − 6.234

• Similarly for the remaining genes

Gene 2 : 78901 yields W2 = + 8.901

Gene 3 : 32104 yields W3 = − 2.104

Gene 4 : 42689 yields W4 = − 2.689

Gene 5 : 63421 yields W5 = + 3.421

Gene 6 : 46421 yields W6 = − 6.421

Gene 7 : 87640 yields W7 = + 7.640

18

4
 SC – Hybrid Systems – GA based BPN

■ Fitness Function :

A fitness is devised for each problem.

Example :

The matrix on the right, represents a set of input I (I11 , I21) (T11 , T21)

(I12 , I22) (T12 , T22)

and output T for problem P to be solved.

(I13 , I23) (T13 , T23)

Generate initial population P0 of size p =
40.

Let C0 1, C0 1, . . . , C0 represent the 40 chromosomes.

Let 0 0 0
w 1, w 2 ,....w 40 be the weight sets extracted, using the Eq.
0
in the previous slides, from each of the chromosome C i , i = 1, 2, . . . , 40 .

Let
o 01 ,
o
0
2 , o 03 be the calculated outputs of BPN.

Compute root mean square error :

E 1 = (T11 – O11)2 + (T21 – O21)2 ,

E 2 = (T12 – O12)2 + (T22 – O22)2

E3 = (T13 – O13)2 + (T23 – O23)2

The root mean square of error is

E = [(E1 + E2 + E3) / 3 ] 1/2

Compute Fitness F1 :

4
The fitness for the chromosome
F1
C01 is given by

F1 = 1 / E .

Similarly, find the fitness F2 for the chromosome C02

and

so on the fitness Fn for the chromosome C0n

4
Algorithm

Let ( Ii , Ti ) , i = 1 , 2 , . . . , N represents the input-output pairs of the

problem to be solved by BPN with configuration ℓ - m – n ; where

I = (I1i , I2i ,, ..., I ℓi ) and

T i = (T1i , T2i ,,..., Tn i )

For each chromosome C i , i = 1,2,. . . , p belonging to current the

population P i whose size is p

Extract weights w i form C i using Eq. 2.1 in previous slide;

Keeping w i as a fixed weight, train the BPN for the N input instances;

Calculate error E i for each of the input instances using the formula below

2
E i =Σ ( T j i – O ji ) where O
i is the output vector calculated by BPN;
j

Find the root mean square E of the errors E i ,i=1,2,...,N

1/2
i.e. E =(( Σ Ei )/N)

Calculate the Fitness value F i for each of the individual string of the

population as F i = 1 / E

Output F i for each C i , i = 1 , 2 , . . . , p ;

4
}

Thus
the Fitness values Fi for all chromosomes in the initial

population
are computed. The population so
is p = 40, Fi,i=1
size

■ 2 , . . , 40 are computed.

A schematic for the computation of fitness values is illustrated below.

Initial

Population of Extracted

Chromosomes weight

sets

C01 w 01
Compute
Training BPN
C02
Fitness
w 02

Extract Input Output

C040 F i =1/E
---

weights weights Error E

---

W 040
Fitness

Values

4
 Fig. Computation of Fitness values for the population

21

4
 SC – Hybrid Systems – GA based BPN

• Reproduction of Offspring

Before the parent chromosomes reproduce offspring :

First, form a mating pool by excluding that chromosome C ℓ with least

fitness F min
and then replacing it with a duplicate copy of C k with

highest fitness F max ;

i.e., the best fit individuals have multiple copies while worst fit

individuals die off.

Having formed the mating pool, select parent pair at random. Chromosomes
of respective pairs are combined using crossover operator. Fig. below shows
:

– two parent chromosomes Pa and Pb,

– the two point crossover,

– exchange of gene segments by the parent pairs, and

– the offspring Oa and Ob are produced.

Pa Pb

Parent

Chromosomes A B

Crossover Crossover Crossover Crossover

Point 1 Point 1 Point 1 Point 1

4
 Offspring B A

Oa Ob

Fig. Two – point crossover operator

22

4
 Example :

– Consider the initial population of chromosomes P0 generated, with

their fitness value F i , where i = 1 , 2 , . . , 40 ,

– Let F max = Fk be maximum and F min =F ℓ be minimum fitness value

for 1 ≤ ℓ , k ≤ 40 where ℓ≠k

min
– Replace all chromosomes having fitness value F with copies of

max
chromosomes having fitness value F

Fig. below illustrates the Initial population of chromosomes and the

formation of the mating pool.

Initial population P0 Mating pool

C01 F1 C01 F1
C02
F2 C02 F2
Max Fitness

value Fmax

Chromosomes C0k Fk C0k Fk

C0ℓ C0ℓ
C0 1 to C0 F ℓ F max

Min Fitness

---- value Fmin ----

C040 F40 C040 F40

Fig. Formation of Mating pool

max
F min is replaced by F

4
k Selection of Parent Chromosomes

The previous slide illustrated Reproduction of the Offspring.

Here, sample "Selection Of Parents" for the "Two Points Crossover" operator

to produce Offspring Chromosomes are illustrated.

Chromosomes - Mating Pool

C1 1 C1 2 C1 k C1 ℓ C140

Selected Parent Pairs

Fig. Random Selection of Parent Chromosomes

The Crossover Points of the Chromosomes are randomly chosen for each
parent pairs as shown in the Fig. below.

Chromosomes -Mating Pool

C1 1 C1 2 C1 k C1 ℓ C140

Crossover

points

4
 Selected Parent Pairs

Fig. Randomly chosen Crossover points of Parent Chromosomes

The Genes are exchanged for Mutation as shown in the Fig. below.

Chromosomes -Mating Pool

C1 1 C1 2 C1k C1 ℓ C140

C1 1 C1 2 C1 k C1ℓ C140

New Population P1

Fig. New population P1 after application of two point Crossover operator

Thus new population P1 is created comprising 40 Chromosomes which are

the Offspring of the earlier population generation P0 .

24

4
 SC – Hybrid Systems – GA based BPN

• Convergence

For any problem, if GA is correctly implemented, the population evolves over

successive generations with fitness value increasing towards the global
optimum.

Convergence is the progression towards increasing uniformity.

A population is said to have converged when 95% of the individuals

constituting the population share the same fitness value.

Example :

Let a population P1 undergoes the process of selection, reproduction,

and crossover.

– the fitness values for the chromosomes in P1 are computed.

– the best individuals replicated and the reproduction carried out using two-
point crossover operators form the next generation P2 of the
chromosomes.

– the process of generation proceeds until at one stage 95% of the

chromosomes in the population Pi converge to the same fitness value.

– at that stage, the weights extracted from the population Pi are the final
weights to be used by BPN.

4
• Fuzzy Back Propagation Network

Neural Networks and Fuzzy logic (NN-FL) represents two distinct methodologies
and the integration of NN and FL is called Neuro-Fuzzy systems.

Back Propagation Network (BPN) is a method of training multi-layer neural

networks where learning occurs during this training phase.

Fuzzy Back Propagation Network (Fuzzy-BPN) is a hybrid architecture. It is,

Hybridization of BPN by incorporating fuzzy logic.

Fuzzy-BPN architecture, maps fuzzy inputs to crisp outputs. Here, the Neurons
uses LR-type fuzzy numbers.

The Fuzzy-Neuron structure, the architecture of fuzzy BP, its learning

mechanism and algorithms are illustrated in next few slides.

3.1 LR-type Fuzzy Numbers

The LR-type fuzzy number are special type of representation of fuzzy

numbers. They introduce functions called L and R.

• Definition
~

A fuzzy member M is of L-R type if and only if

m–x
L
for x≤m, α 0
µ ~ (x) = α
m–x
M
R for x ≤ m , β 0

4
 β

where L is a left reference

R is a right reference, ~

m, is called mean of M is a real number,

α, β are left and right spreads respectively.

~
µ~ is the membership function of fuzzy member M

The functions L and R are defined as follows:

m–x
m–x
L = max ( 0,1 - )
α α
m–x
m–x
R = max ( 0 , 1 - )

α α

~
can be represented as (m, α, β) LR shown below.
LR-type fuzzy number

Member

ship deg µ ~

(x)

00 α m, β x

Fig. A triangular fuzzy number (m, α, β).

4
Note : If α and β are both zero, then L-R type function indicates a crisp
value. The choice of L and R functions is specific to problem.

4
• Operations on LR-type Fuzzy Numbers
~ ~ = (n, γ , δ)
Let M = (m, α , β) LR and N LR be two L R-type fuzzy

numbers. The basic operations are

• Addition

(m, α , β) LR (n, γ , δ) LR = (m + n, α + γ , β + δ ) LR

• Substraction

(m, α , β) LR (n, γ , δ) LR = (m - n, α + δ , β + γ ) LR

■ Multiplicaion

(m , α , β ) LR (n , γ , δ) LR = (mn , mγ + nα , mδ + nβ) LR for m≥ 0 , n≥ 0

(m,
α
, β) LR ,
, δ) LR = (mn , mα - mδ , nβ - mγ ) RL for m<0 , n≥ 0
(n,

β) LR γ

, δ) LR = (mn , - nβ - mδ , -nα - mγ ) LR for m<0 ,

(m, (n, γ

n<0

■ Scalar Multiplicaion

λ*(m, α , β) LR = (λm,
∀
λ
≥

λα , λβ) LR , λ 0 , R
∈

λ*(m, ∀
λ
α , β) LR = (λm, - , -λβ) RL , λ<0 , R
∈
λα

4
4
 SC – Hybrid Systems – Fuzzy BPN

■ Fuzzy Neuron

The fuzzy neuron is the basic element of Fuzzy BP network. Fig. below
shows the architecture of the fuzzy neuron.

The fuzzy neuron computes the crisp output given by

n ~ ~
~
O = f (NET) = f )) where I0 = (1, 0, 0) is the bias.
( CE ( Σ Wi . Ii

i=1

Here, the fuzzy weighted summation is given by

~
n ~ ~
net ▪
Σ Wi Ii is first computed and
=

i=0
~

NET
= CE ( net ) is computed next

The function CE is the centroid of triangular fuzzy number, that has

m as mean and α , β as left and right spreads explained before, can

be treated as defuzzification operation, which maps fuzzy weighted

summation to crisp value.

4
 ~ ~ ~ ~

If net = ( netm , netα , netβ ) is the fuzzy weighted summation

Then function CE is given by

~ ~ ~ ~ ~ ~ ~

, net ne ) = netm + 1/3 (

CE ( net ) = CE ( netm
α t netβ – netα ) = NET
, β

The function f is a sigmoidal function that performs nonlinear mapping

between the input and output. The function f is obtained as :

f (NET) = 1 / ( 1 + exp ( - NET ) ) = O is final crisp output value.

■ Architecture of Fuzzy BP

Fuzzy Back Propagation Network (BP) is a 3-layered feed forward

architecture. The 3 layers are: input layer, hidden layer and output layer.
Considering a configuration of ℓ-input neurons, m-hidden neurons and n-
output neurons, the architecture of Fuzzy BP is shown below.

4
4
 SC – Hybrid Systems – Fuzzy AM

• Fuzzy Associative Memory

A fuzzy logic system contains the sets used to categorize input data (i.e.,
fuzzification), the decision rules that are applied to each set, and then a way
of generating an output from the rule results (i.e., defuzzification).

In the fuzzification stage, a data point is assigned a degree of

membership (DOM) determined by a membership function. The member-

ship function is often a triangular function centered at a given point.

The Defuzzification is the name for a procedure to produce a real (non-

fuzzy) output .

Associative Memory is a type of memory with a generalized addressing

method. The address is not the same as the data location, as in

traditional memory. An associative memory system stores mappings

of specific input representations to specific output representations.

Associative memory allows a fuzzy rule base to be stored. The inputs are

the degrees of membership, and the outputs are the fuzzy system’s output.

Fuzzy Associative Memory (FAM) consists of a single-layer feed-forward

fuzzy neural network that stores fuzzy rules "If x is Xk then y is Yk" by means
of a fuzzy associative matrix.

FAM has many applications; one such application is modeling the

operations of washing machine.

33

4
 The problem indicates, that there are two inputs and one-output
variables. The inference engineer is constructed based on fuzzy rule :

“ If < input variable > AND < input variable >

THEN < output variable >”

According to the above fuzzy rule, the Fuzzy Associative Memory

(FSM) of X, Y, and T variables are listed in the Table below.

Weight (X)
Washing time (T)

S M L

S M L L

Stream (Y) M S M L

L S S L

Table 1. Fuzzy associative memory (FSM) of Washing Machine

■ Operations : To wash the clothes −

Turn on the power,

The machine automatically detects the weight of the clothes as

(X) = 3.2 K.g. ,

– The machine adjusts the water stream (Y) to 32 liter/min.,

35

4
• Fuzzy Representation :

The fuzzy sets representation, while X = 3.2 Kg and Y = 32 liter/min.,

according to the membership functions, are as follows:

The fuzzy set of X3.2 Kg = { 0.8/S, 0.2/M, 0/L }

The fuzzy set of Y32 liters/min. = { 0.4/S, 0.8/M, 0/L }

■ Defuzzification

The real washing time is defuzzied by the Center of gravity (COG)

defuzzification formula. The washing time is calculated as :

Z COG = Σn µc (Z j ) Z j / Σn µc (Z j ) where

j=1 j=1

j = 1, . . . , n , is the number of quantization levels of the output,

Z j
is the control output at the quantization level j,

µc (Z j )
represents its membership value in the output fuzzy set.

Referring to Fig in the previous slide and the formula for COG, we
get the fuzzy set of the washing time as w = { 0.8/20, 0.4/35, 0.2/60
} The calculated washing time using COG formula T = 41.025 min.

4
■ Simplified Fuzzy ARTMAP

ART is a neural network topology whose dynamics are based on

Adaptive Resonance Theory (ART). ART networks follow both
supervised and unsupervised algorithms.

– The Unsupervised ARTs are similar to many iterative clustering

algorithms where "nearest" and "closer" are modified slightly by
introducing the concept of "resonance". Resonance is just a matter
of being within a certain threshold of a second similarity measure.

– The Supervised ART algorithms that are named with the suffix
"MAP", as ARTMAP. Here the algorithms cluster both the inputs and
ART1:
targets and associate two sets of clusters.
ART2 :

The basic ART system is an unsupervised learning model.

The ART systems have many variations : ART1, ART2, Fuzzy ART,
ARTMAP.

The simplest variety of ART networks, accepting only binary

inputs.

It extends network capabilities to support continuous inputs.

ARTMAP : Also known as Predictive ART. It combines two slightly modified

ART-1 or ART-2 units into a supervised learning structure. Here, the first
unit takes the input data and the second unit takes the correct output
data, then used to make the minimum possible adjustment of the
vigilance parameter in the first unit in order to make the correct
classification.

4
 The Fuzzy ARTMAP model is fuzzy logic based computations incorporated
in the ARTMAP model.

Fuzzy ARTMAP is neural network architecture for conducting supervised

learning in a multidimensional setting. When Fuzzy ARTMAP is used on a
learning problem, it is trained till it correctly classifies all training data.
This feature causes Fuzzy ARTMAP to ‘over-fit’ some data sets, especially
those in which the underlying pattern has to overlap. To avoid the
problem of ‘over-fitting’ we must allow for error in the training process.

• Supervised ARTMAP System

ARTMAP is also known as predictive ART. The Fig. below shows a

supervised ARTMAP system. Here, two ART modules are linked by an

inter-ART module called the Map Field. The Map Field forms predictive
associations between categories of the ART modules and realizes a
match tracking rule. If ARTa and ARTb are disconnected then each
module would be of self-organize category, groupings their respective
input sets.

[Link] ARTMAP system

4
 In supervised mode, the mappings are learned between input vectors
a and b. A familiar example of supervised neural networks are feed-
forward networks with back-propagation of errors.

• Comparing ARTMAP with Back-Propagation Networks

ARTMAP networks are self-stabilizing, while in BP networks the new

information gradually washes away old information. A consequence of
this is that a BP network has separate training and performance
phases while ARTMAP systems perform and learn at the same time

– ARTMAP networks are designed to work in real-time, while BP

networks are typically designed to work off-line, at least during
their training phase.

4
– ARTMAP systems can learn both in a fast as well as in a slow match
configuration, while, the BP networks can only learn in slow
mismatch configuration. This means that an ARTMAP system learns,
or adapts its weights, only when the input matches an established
category, while BP networks learn when the input does not match
an established category.

– In BP networks there is always a danger of the system getting

trapped in a local minimum while this is impossible for ART
systems. However, the systems based on ART modules learning
may depend upon the ordering of the input patterns.

4
Scoring : assign fitness to each individual