S8-SOFT COMPUTING ​ ​ ​ ​ ​ ​ ​ 2019 SCHEME

SEMESTER VIII
CST 444

ST
SOFT COMPUTING
SI
(2019 SCHEME)
AD
LE

Prepared by,
NIKHILA A
ASSISTANT PROFESSOR
DEPARTMENT OF COMPUTER SCIENCE AND ENGINEERING
SARABHAI INSTITUTE OF SCIENCE AND TECHNOLOGY
Module – 1 (Introduction to Soft Computing & Artificial Neural Network)
Syllabus:Introduction to Soft Computing. Difference between Hard Computing &
Soft Computing. Applications of Soft Computing. Artificial Neurons Vs Biological
Neurons. Basic models of artificial neural networks – Connections, Learning,
Activation Functions. McCulloch and Pitts Neuron. Hebb network.

Module – 1
1.1 Introduction to Soft Computing

Two major problem solving techniques are:

ST
●​ Hard computing

It deals with precise model where accurate solutions are achieved.

SI
AD

Figure 1.1: Hard Computing

●​ Soft computing

It deals with an approximate model to give solutions for complex problems.

LE

The term “soft computing" was introduced by Professor Lorfi Zadeh with the
objective of exploiting the tolerance for imprecision, uncertainty and partial
truth to achieve tractability, robustness, low solution cost and better rapport
with reality. The ultimate goal is to be able to emulate the human mind as
closely as possible.

Soft computing has three main branches:

►fuzzy Systems,

►artificial neural Networks

CSE DEPT,
SARABHAI INSTITUTE OF SCIENCE AND TECHNOLOGY
1
 ►Genetic Algorithms

ST
Figure 1.2: Soft Computing

The difference between hard computing and soft computing are as follows

Hard Computing Soft Computing

The analytical model required by hard It is based on uncertainty partial truth

Computation time is more

SI
computing must be precisely represented tolerant of imprecision and approximation.

Computation time is less

It depends on binary logic, numerical Based on approximation and dispositional.

systems, crisp software.
AD

Sequential computation Parallel computation

Gives exact output Gives appropriate output

Examples: Traditional methods of Example: Neural networks like Adaline,

computing using our personal computer. Madaline, ART networks, etc.

Applications of Soft Computing

LE

►Handwriting Recognition

►Image Processing and Data Compression

►Automotive Systems and Manufacturing

►Soft Computing to Architecture

►Decision-support Systems

►Soft Computing to Power Systems

►Neuro Fuzzy systems

CSE DEPT,
SARABHAI INSTITUTE OF SCIENCE AND TECHNOLOGY
2
►Fuzzy Logic Control

►Machine Learning Applications

►Speech and Vision Recognition Systems

1.1 Biological Neurons

ST
Figure 1.3: Schematic diagram of a biological neuron
SI
A neuron is the basic processing unit in a neural network sitting on our brain.

The biological neuron consists of :

●​ Soma or cell body-where cell nucleus is located

●​ Dendrites –Tree like network made of nerve fibre connected to cell
AD

body which receives signals

●​ Axon-Single long connection extending from the cell body which carries
impulses of the neuron and transmit signals
●​ Synapse –The end of the axon splits into fine strands, each strand
terminates into a small bulb-like organ called Synapse.
LE

It is through synapse that the neuron introduces its signals to other nearby
neurons. The receiving ends of these synapses on the nearby neurons can be
found both on the dendrites and on the cell body. There are approximately

4
10 synapses per neuron in the human body. Electric impulse is passed

between synapse and dendrites. It is a chemical process which results in

increase/decrease in the electric potential inside the body of the receiving cell.
If the electric potential reaches a thresh hold value, receiving cell fires & pulse
/ action potential of fixed strength and duration is send through the axon to

CSE DEPT,
SARABHAI INSTITUTE OF SCIENCE AND TECHNOLOGY
3
 synaptic junction of the cell. After that, cell has to wait for a period called

refractory period.

ST
Figure 1.4: Mathematical model of artificial neuron

Biological neuron SI Artificial neuron

Cell Neuron

Dendrites Weights or interconnections

Soma Net input

Axon Output
AD

Table 1.1: Terminology relationships between biological and artificial neurons

In this model net input is calculated as

LE

th
Where, i represents i processing element. The activation function applied

over it to calculate the output. The weight represents the strength of synapses
connecting the input and output.

1.1 Artificial neural networks

CSE DEPT,
SARABHAI INSTITUTE OF SCIENCE AND TECHNOLOGY
4
An artificial neural network (ANN) is an efficient information processing system
which resembles the characteristics of biological neural network. ANNs contain
large number of highly interconnected processing elements called nodes or
neurons or units. Each neuron is connected with other by connection link and each
connection link is associated with weights which contain information about the
input signal. This information is used by neuron net to solve a particular problem.
ANNs have ability to learn, recall and generalize training pattern or data similar to
that of human brain. The ANN processing elements called neurons or artificial
neurons.

ST
SI
Figure 1.5: Architecture of a simple artificial neuron net

Each neuron has an internal state of its own, called activation or activity level
AD

of neuron which is the function of the inputs the neuron receives. The
activation signal of a neuron is transmitted to other neurons. A neuron can
send only one signal at a time which can be transmitted to several neurons.
LE

Consider the figure 1.5, here X 1 and X2 are input neurons, Y is the output

neuron W 1 and W2 are the weights net input is calculated as

𝑦𝑖𝑛 = 𝑥1𝑤1 + 𝑥2𝑤2

where x 1 and x2 are the activation of the input neurons X1 and X2, i.e., is the
output of the input signals. The output y of the output neuron Y can be obtained by
applying activations over the net input.
CSE DEPT,
SARABHAI INSTITUTE OF SCIENCE AND TECHNOLOGY
5
 𝑦 = 𝑓(𝑦𝑖𝑛)

Output = Function (net input calculated)

The function to be applied over the net input is called activation function. The
net input calculation is similar to the calculation of output of a pure linear
straight line equation y=mx

ST
Figure 1.6: Neural net of pure linear equation

SI
AD

Figure 1.7: Graph for y = mx

The weight involve in the ANN is equivalent to the slope of the straight line.

1.4 Comparison between Biological neuron and Artificial neuron

LE

Term Brain Computer

Speed Execution time is few milliseconds Execution time is few nano seconds

Perform massive parallel Perform several parallel operations

simultaneously. It is faster the
Processing operations simultaneously
biological neuron

CSE DEPT,
SARABHAI INSTITUTE OF SCIENCE AND TECHNOLOGY
6
 11
Size and Number of Neuron is 10 and It depends on the chosen application
complexity and network designer.
number of interconnections is

15
10 . So complexity of brain is

higher than computer

· Information is stored in
· Stored in continuous memory

ST
interconnections or in synapse location.
strength.
· Overloading may destroy older
· New information is stored
locations.
Storage without destroying old one.
· Ca be easily retrieved
· Sometimes fails to recollect
capacity
information
SI
· No fault tolerance
· Fault tolerant
· Information corrupted if the
Tolerance · Store and retrieve
AD

network connections
information even
disconnected.
interconnections fails
· No redundancies
· Accept redundancies

Control Depends on active chemicals and CPU

mechanism neuron connections are strong or
LE

weak Control mechanism is very simple

Table 1.2: Comparison between Biological neuron and Artificial neuron

Characteristics of ANN:

⚫ It is a neurally implemented mathematical model

⚫ Large number of processing elements called neurons exists here.
⚫ Interconnections with weighted linkage hold informative knowledge.
⚫ Input signals arrive at processing elements through connections and
connecting weights.
CSE DEPT,
SARABHAI INSTITUTE OF SCIENCE AND TECHNOLOGY
7
 ⚫ Processing elements can learn, recall and generalize from the given data.
⚫ Computational power is determined by the collective behavior of neurons.
o ANN is a connection models, parallel distributed processing
models, self-organizing systems, neuro-computing systems and
neuro - morphic system.

1.1 Evolution of neural networks

Year Neural network Designer Description

ST
1943 McCulloch McCulloch Arrangement of neurons is
and Pitts and Pitts combination of logic gate.
Unique feature is thresh hold
neuron SI
1949 Hebb network Hebb If two neurons are active, then
their connection strengths
should be increased.
AD

1958, Perceptron Frank Here the weights on the

Rosenblatt, connection path can be adjusted.
1959,
Block, Minsky
1962, and Papert
1988,
LE

1960 Adaline Widrow and Here the weights are adjusted to

Hoff reduce the difference between
the net input to the output unit
and the desired output.

CSE DEPT,
SARABHAI INSTITUTE OF SCIENCE AND TECHNOLOGY
8
1972 Kohonen self- Kohonen Inputs are clustered to obtain a
organizing fired output neuron.
feature map

1982, Hopfield John Hopfield Based on fixed weights.

network and Tank
1984, Can act as associative memory
nets
1985,

ST
1986,

1987

Rumelhart, · Multilayered
1986 Back
propagation Hinton
SI · Error propagated backward

from output to the hidden

network and
units
Williams
AD

1988 Counter Grossberg Similar to kohonen network.

propagati
on
network
LE

1987- Adaptive Carpenter Designed for binary and analog

resonance and inputs.
1990 Theory(ART)
Grossberg

1988 Radial basis Broomhead Resemble back propagation

function and Lowe network, but activation function
used is Gaussian function.
network

CSE DEPT,
SARABHAI INSTITUTE OF SCIENCE AND TECHNOLOGY
9
1988 Neo cognitron Fukushima For character recognition.

Table 1.3: Evolution of neural networks

1.6 Basic models of artificial neural networks

Models are based on three entities

· The model’s synaptic interconnections.

· The training or learning rules adopted for updating and adjusting the
connection weights.

ST
· Their activation functions

1.6.1 Connections

The arrangement of neurons to form layers and the connection pattern

SI
formed within and between layers is called the network architecture. There
exist five basic types of connection architecture.

They are:
AD

1. Single layer feed forward network

2. Multilayer feed-forward network
3. Single node with its own feedback
4. Single-layer recurrent network
5. Multilayer recurrent network
LE

Feed forward network: If no neuron in the output layer is an input to a

node in the same layer / proceeding layer.

Feedback network: If outputs are directed back as input to the processing

elements in the same layer/proceeding layer.

Lateral feedback: If the output is directed back to the input of the same layer.

Recurrent networks: Are networks with feedback networks with closed loop.

1. Single layer feed forward network

CSE DEPT,
SARABHAI INSTITUTE OF SCIENCE AND TECHNOLOGY
10
Layer is formed by taking processing elements and combining it with other
processing elements. Input and output are linked with each other Inputs are
connected to the processing nodes with various weights, resulting in a series of
outputs one per node.

ST
SI
Figure 1.8: Single-layer feed-forward network

When a layer of processing nodes is formed the inputs can be connected to these
nodes with various weights, resulting in a serious of outputs, one per node. This is
called single layer feedforward network.
AD

2. Multilayer feed-forward network

This network is formed by the interconnection of several layers. Input layer

receives input and buffers input signal. Output layer generated output. Layer
LE

between input and output is called hidden layer. Hidden layer is internal to the
network. There are Zero to several hidden layers in a network. More the hidden
layer more is the complexity of network, but efficient output is produced.

CSE DEPT,
SARABHAI INSTITUTE OF SCIENCE AND TECHNOLOGY
11
 Figure 1.9: Multilayer feed-forward network

ST
1. Single node with its own feedback

It is a simple recurrent neural network having a single neuron with

feedback to itself.
SI
AD

Figure 1.10: Single node with own feedback

3. Single layer recurrent network

LE

A single layer network with feedback from output can be directed to processing
element itself or to other processing element/both.

CSE DEPT,
SARABHAI INSTITUTE OF SCIENCE AND TECHNOLOGY
12
 ST
SI
Figure 1.11: Single-layer recurrent network

5. Multilayer recurrent network

AD

Processing element output can be directed back to the nodes in the preceding layer,
forming a multilayer recurrent network.
LE

Figure 1.12: Multilayer recurrent network

⚫ Maxnet –competitive interconnections having fixed weights.

CSE DEPT,
SARABHAI INSTITUTE OF SCIENCE AND TECHNOLOGY
13
 Figure 1.13: Competitive nets

⚫ On-center-off-surround/lateral inhibition structure – each

processing neuron receives two different classes of inputs- “excitatory”

ST
input from nearby processing elements & “inhibitory” elements from
more distantly located processing elements. This type of
interconnection is shown below
SI
AD

Figure 1.14: Lateral inhibition structure

1.6.2 Learning

Learning or Training is the process by which a neural network adapts itself to a

stimulus by making proper parameter adjustments, resulting in the production of
desired response.
LE

Two broad kinds of learning in ANNs is:

i) Parameter learning – updates connecting weights in a neural net.

ii) Structure learning – focus on change in the network.

Apart from these, learning in ANN is classified into three categories as

i) Supervised learning
ii) Unsupervised learning
iii) Reinforcement learning
CSE DEPT,
SARABHAI INSTITUTE OF SCIENCE AND TECHNOLOGY
14
i) Supervised learning

The Learning here is performed with the help of a teacher. Example: Consider the
learning process of a small child. Child doesn’t know how to read/write. Their each
and every action is supervised by a teacher. Actually a child works on the basis of
the output that he/she has to produce. In ANN, each input vector requires a
corresponding target vector, which represents the desired output. The input vector
along with target vector is called training pair. Input vector results in output vector.
The actual output vector is compared with desired output vector. If there is a

ST
difference means an error signal is generated by the network. It is used for
adjustment of weights until actual output matches desired output.

SI
AD

Figure 1.15: Supervised learning

iI) Unsupervised learning

Learning is performed without the help of a teacher. Example: tadpole – learn to

LE

swim by itself. In ANN, during training process, network receives input patterns and
organize it to form clusters.

Figure 1.16: Unsupervised learning

From the above Fig.1.16 it is observed that no feedback is applied from

environment to inform what output should be or whether they are correct. The
network itself discover patterns, regularities, features/ categories from the input

CSE DEPT,
SARABHAI INSTITUTE OF SCIENCE AND TECHNOLOGY
15
data and relations for the input data over the output. Exact clusters are formed by
discovering similarities & dissimilarities so called as self – organizing.

iii) Reinforcement learning

It is similar to supervised learning. Learning based on critic information is called

reinforcement learning & the feedback sent is called reinforcement signal. The
network receives some feedback from the environment. Feedback is only evaluative.

ST
SI
Figure 1.17: Reinforcement learning

The external reinforcement signals are processed in the critic signal generator,
and the obtained critic signals are sent to the ANN for adjustment of weights
AD

properly to get critic feedback in future.

1.6.3 Activation Functions

To make work more efficient and for exact output, some force or activation is given.
LE

Like that, activation function is applied over the net input to calculate the output of an
ANN. Information processing of processing element has two major parts: input and
output. An integration function (f) is associated with input of processing element.

Several activation functions are there.

1. Identity function: It is a linear function which is defined as

𝑓(𝑥) = 𝑥 𝑓𝑜𝑟 𝑎𝑙𝑙 𝑥

The output is same as the input.

CSE DEPT,
SARABHAI INSTITUTE OF SCIENCE AND TECHNOLOGY
16
2. Binary step function: This function can be defined as

Where, θ represents thresh hold value. It is used in single layer nets to convert the
net input to an output that is binary (0 or 1).

3. Bipolar step function: This function can be defined as

ST
Where, θ represents threshold value. It is used in single layer nets to convert the net
input to an output that is bipolar (+1 or -1).

4. Sigmoid function: It is used in Back propagation nets.

Two types:
SI
a) Binary sigmoid function: It is also termed as logistic sigmoid function or
unipolar sigmoid function. It is defined as
AD

where, λ represents steepness parameter. The derivative of this function is

LE

The range of sigmoid function is 0 to 1.

b) Bipolar sigmoid function: This function is defined as

Where λ represents steepness parameter and the sigmoid range is between -1 and
+1. The derivative of this function can be

CSE DEPT,
SARABHAI INSTITUTE OF SCIENCE AND TECHNOLOGY
17
 It is closely related to hyperbolic tangent function, which is written as

ST
The derivative of the hyperbolic tangent function is

SI
5. Ramp function: The ramp function is defined as
AD

The graphical representation of all these function is given in the upcoming figure
1.18
LE

CSE DEPT,
SARABHAI INSTITUTE OF SCIENCE AND TECHNOLOGY
18
 ST
SI
AD

Figure 1.18: Depiction of activation functions: (A) identity function; (B) binary step function;(C)
LE

bipolar step function; (D) binary sigmoidal function; (E) bipolar sigmoidal function; (F)ramp
function.

1.7 McCulloch and Pitts Neuron

It is discovered in 1943 and usually called as M-P neuron. M-P neurons are
connected by directed weighted paths. Activation of M-P neurons is binary
(i.e) at any time step the neuron may fire or may not fire. Weights associated
with communication links may be excitatory (wgts are positive)/inhibitory (wgts
are negative). Threshold plays major role here. There is a fixed threshold for
each neuron and if the net input to the neuron is greater than the threshold
then the neuron fires. They are widely used in logic functions. A simple M-P
CSE DEPT,
SARABHAI INSTITUTE OF SCIENCE AND TECHNOLOGY
19
 neuron is shown in the figure. It is excitatory with weight w (w>0) / inhibitory

with weight –p (p<0). In the Fig.,inputs from x 1 to xn possess excitatory

weighted connection and X n+1 to x n+m has inhibitory weighted

interconnections.

ST
SI
AD
LE

Figure 1.19: McCulloch-Pins neuron model

Since the firing of neuron is based on threshold, activation function is defined as

For inhibition to be absolute, the threshold with the activation function should
satisfy the following condition:

θ > nw –p

CSE DEPT,
SARABHAI INSTITUTE OF SCIENCE AND TECHNOLOGY
20
 Output will fire if it receives “k” or more excitatory inputs but no inhibitory inputs
where

kw ≥ θ>(k-1) w

The M-P neuron has no particular training algorithm. An analysis is performed

to determine the weights and the threshold. It is used as a building block
where any function or phenomenon is modeled based on a logic function.

1.8 Hebb network

Donald Hebb stated in 1949 that “In brain, the learning is performed by the change

ST
in the synaptic gap”. When an axon of cell A is near enough to excite cell B, and
repeatedly or permanently takes place in firing it, some growth process or
metabolic change takes place in one or both the cells such that A’s efficiency, as
one of the cells firing B, is increased. According to Hebb rule, the weight vector is
found to increase proportionately to the product of the input and the learning
SI
signal. In Hebb learning, two interconnected neurons are ‘on’ simultaneously. The
weight update in Hebb rule is given by
AD

Hebbs network is suited more for bipolar data. If binary data is used, the weight
updation formula cannot distinguish two conditions namely:

1. A training pair in which an input unit is “on” and the target value is “off”.
2. A training pair in which both the input unit and the target value is “off”.
LE

Training algorithm

The training algorithm is used for the calculation and adjustment of weights. The
flowchart for the training algorithm of Hebb network is given below

Step 0: First initialize the weights. Basically in this network they may be set to zero,
i.e., w; =0, for i= 1 to n where "n" may be the total number of input neurons.

Step 1: Steps 2-4 have to be performed for each input training vector and
target output pair, s: t.

CSE DEPT,
SARABHAI INSTITUTE OF SCIENCE AND TECHNOLOGY
21
Step 2: Input units activations are set. Generally, the activation function of input

i i
layer is identity function: x = s for i=1 to n

Step 3: Output units activations are set: y = t.

Step 4: Weight adjustments and bias adjustments are performed:

wi(new) =wi(old)+xiy

b(new)=b(old)+y

ST
In step 4, the weight updation formula can be written in vector form as

w(new) = w(old)+y

Hence, Change in weight is expressed as

SI Δw = xy

As a result,

w(new)=w(old)+Δw
AD

Hebb rule is used for pattern association, pattern categorization, pattern

classification and over a range of other areas.

Flowchart of Training algorithm

LE

CSE DEPT,
SARABHAI INSTITUTE OF SCIENCE AND TECHNOLOGY
22
 ST
SI
AD

Figure 1.20: Flowchart of Hebb training algorithm

LE

CSE DEPT,
SARABHAI INSTITUTE OF SCIENCE AND TECHNOLOGY
23