SOFT COMPUTING

RECOMMENDED BOOKS

1. Fuzzy Logic with Engineering Applications- By [Link], 3nd

edition, John Wiley Sons, 2010.
2. Principles of Soft Computing- [Link] and
[Link], Wiley India edition, 2008.
3. Neural Networks, Fuzzy Logic and Genetic Algorithms:
Synthesis and Applications- By [Link] and G.A.V.
Pai, PHI 2007.
4. Genetic Algorithms: Search, Optimization and Machine
Learning- Davis E. Goldberg, Pearson Education, Fourth
Impression, 2009.
5. Fuzzy Sets and Fuzzy Logic: Theory and Applications-
[Link] and [Link], PHI 1997
 SOFT COMPUTING

• Soft Computing: Coined by [Link] in 1994

• 1. [Link]: Fuzzy logic, Neural Networks and Soft

Computing, communications of the ACM, March 1994, pp.77
-84.

• 2. [Link]: Soft Computing and Fuzzy Logic, IEEE Software,

vol.11, no.6, 1994, pp.48-56.
 SOFT COMPUTING

• It is a collection of methodologies that aim

to exploit the tolerance for imprecision and uncertainty
to achieve tractability, robustness and low cost solution.

• Its principal constituents (In the beginning) were

• Fuzzy logic
• Neuro-computing and
• Probabilistic reasoning

• The role model for soft computing is the human mind

 ELABORATION

• Tractability: Being handled easily or controlled

• Robustness: Ability to resist disorder

• Low cost solution:

• We live in a world that is pervasively imprecise, uncertain and
hard to be categorical
• Precision and certainty carry a cost
• For high precision and low uncertainty we have to pay some
price
 AN EXAMPLE

• Let us consider the parking of a car.

• We find it easy to park a car because the final position of the

car is not specified precisely.

• If the final position is specified then the difficult of parking

would increase geometrically with the increase in precision.

• Eventually the parking may become impossible

 THE GUIDING PRINCIPLE

• Exploit the tolerance for imprecision, uncertainty and partial

truth to achieve tractability, robustness and low solution cost.

• HARD COMPUTING:
• The traditional computing is called hard computing.
• In this imprecision of the real world is not accommodated.

• AIM OF SOFT COMPUTING: To emulate human mind as

closely as possible
 EVERYDAY USE OF APPROXIMATION

We find that because of the exploitation of the tolerance for

imprecision and uncertainty human have
• The remarkable ability to understand distorted speech
• Decipher sloppy handwriting
• Comprehend nuances of natural language
• Summarize text
• Recognize and classify images
• Drive a vehicle in dense traffic
• More generally: Make rational decisions in an environment
of uncertainty and imprecision
 HARD COMPUTING SOFT COMPUTING

Conventional computing techniques Non conventional approaches those

those are deterministic and have are stochastic and have vague
sharp boundary boundary

Precise, certain and has two valued Imprecise, uncertain and has multi
(Boolean) logic valued logic

Needs exact input Can handle ambiguous and noisy data

Not tractable Tractable solution

High computational cost Low computational cost

low Intelligence Quotient (MIQ) High Machine Intelligence Quotient (MIQ)

Precise Approximate reasoning

 SOME APPLICATION AREAS OF SOFT
COMPUTING
• Data clustering
• Database anonymisation
• Rule generation
• Image processing
• Medical diagnosis
• Pattern recognition
• Social networks
• Distributed computing
• Parallel processing
• Machine learning and
• Granular computing
 NEURAL NETWORKS

• A neural network is a processing device

• It is either an algorithm or an actual hardware
• Its design is inspired by the design and functioning of animal
brains and components thereof
• It has the ability to learn by example
• It has made them very flexible and powerful
• The networks are also well suited for real-time systems
• They have fast response and computational times
• They have a parallel architecture
 HUMAN BRAIN

• It is an amazing processor
• Its exact working is still a mystery
• The basic elements of the human brain are specific type of
cells, known as neurons
• Neurons do not regenerate
• Neurons provide us with abilities to remember, think and
apply previous experiences to our every action
• Human brain comprises of about 100 billion neurons (10^11)
• Each neuron is connected to 200,000 other neurons
• THE POWER OF HUMAN MIND COMES FROM THE SHEER
NUMBER OF NEURONS AND THEIR MULTIPLE
INTERCONNECTIONS
 ARTIFICIAL NEURAL NETWORKS (ANN)

• Artificial Neural Networks are information processing

systems
• Constructed and implemented to model the human brain

• The main objective of the neural network research is

• To develop a computational device
• For modeling the brain
• To perform various computational tasks
• At a faster rate than the traditional systems
 ARCHITECTURE OF A SIMPLE ANN

• Fig. yin  x1w1  x2 w2 y  f ( yin ) f : Activation function

X 1 , X 2 : Input neurons Y : Output neuron
X 1 , X 2 transmit signals , Y receives signal
x1 X1
w1

Y y

x2 w2
X2

x1 , x2 : activations of the input neurons : output of input signals

w1 , w2 : associated weights, which contain information about theinput signals
 TASKS PERFORMED BY ANN

• Pattern-matching
• Classification
• Optimization of functions
• Approximation
• Vector quantization
• Data Clustering
 ANN CONTD…

• ANN is defined as an information-processing model

• Inspired by the way biological nervous systems such as the
brain processes information
• It tries to replicate the most basic functions of the brain
• It comprises of a large number of highly interconnected
processing elements(neurons) working in unison to solve
specific problem
• An ANN is configured for a specific application, such as
pattern recognition or data classification through a learning
process
 COMPONENTS OF BIOLOGICAL NEURAL
NETWORKS
• SOMA or CELL BODY:
• Inside the cell body we have the cell nucleus
• DENDRITES:
• Where the nerve is connected to the cell body
• Are tree like networks made of nerve fiber connected to the
cell body
• AXON:
• Which carries the impulses of the neuron
• A single long connection extending from the cell body and
carrying signals from the neuron
 COMPONENTS OF BIOLOGICAL NN CONTD…
• SYNAPSE:
• A small bulb-like organ into which each strand terminates
• The axon dendrite contact organ is called a synapse.
• It is where neuron introduces its signal for neighbouring
neurons,which is an electrical impulse caused by means of a
chemical process.
• The neuron aggregates the total of its inputs within a short
period of time called period of latent summation.
• THROUGH SYNAPSES THE NEURON INTRODUCES ITS
SIGNALS TO OTHER NEARBY NEURONS

• THE RECEIVING ENDS OF THESE SYNAPSES ON THE NEARBY

NEURONS CAN BE FOUND BOTH ON THE DENDRITES AND
ON THE CELL BODY
The neuron fires when the aggregate of thee incoming
signals exceeds threshold value of the neuron(i.e.
40mV).

Incoming signals are excitatory if they cause the firing

or inhibitory if they hinder the firing of the neuron.

The characteristic feature of biological neuron is that

the information is transmitted between nerve cells by
means of binary signals,
SCHEMATIC DIAGRAM OF A BIOLOGICAL
NEURON

Synapse : bulb like organ at the end of strands

Strands : The splits at the end of axons
Synapse
Nucleus Strands

Axon
.

Cell body (Soma)

Dendrites

Dendrites : Wherethe nerve is connected to the cell body

Axon : Which carries the impulses of the neuron
X1
X1
w1

Y y
w2
x2
X2
Neuron and a sample of pulse train
 BIOLOGICAL NN CONTD…

• In the human brain there are approximately 10000 synapses

per neuron

• Mathematical representation of the above process in ANN is:

• Suppose there are n inputs from n neurons with
activations respectively X 1 , X 2 ,... X n
x1 , xof
• Let the weights 2 ,...
thexninterconnections between
and the connecting neuron Y be X 1 , X 2 ,... X n
respectively
w1 , w2 ,...wn
 ARTIFICIAL NEURAL NETWORKS CONTD…

• The net input to the neuron Y is given by the formula:

yin  x1w1  x2 w2  ...xn wn
• The activation function is applied to yin to compute the
output
• The weight represents the strength of synapse connecting
the input and the output neurons
• The weights may be positive or negative
• +ve weight means the synapse is excitatory
• -ve weight means the synapse is inhibitory
TERMINOLOGICAL RELATIONSHIP BETWEEN
BIOLOGICAL NN AND ANN
Biological Neuron Artificial Neuron
Cell Neuron

Dendrites Weights or Interconnections

Soma Net input

Axon Output
COMPARISON BETWEEN BIOLOGICAL NEURON
AND ARTIFICIAL NEURON
• SPEED:
• The cycle time of execution in the AN is of few nanoseconds
• In biological neurons the cycle time is of a few milliseconds

• PROCESSING:
• Biological neuron can perform massive parallel operations
simultaneously
• AN can also perform several parallel operations
simultaneously.
• PROCESSING IN ANN IS FASTER THAN BIOLOGICAL NN
COMPARISON BETWEEN BIOLOGICAL NEURON
AND ARTIFICIAL NEURON
• SIZE AND COMPLEXITY:
• IN BRAIN:
• The total no. of neurons is 10
11

• The total no. of interconnections is 1015

• The complexity of a biological neuron is high
• The size of a biological neuron is high
• The computational work is carried not only inside the cell
body but also in axon, synapse etc.
• IN ANN:
• The size and complexity depends upon the chosen application
and the network designer
COMPARISON BETWEEN BIOLOGICAL NEURON
AND ARTIFICIAL NEURON
• STORAGE CAPACITY (MEMORY)
• The biological neuron stores the information in its
interconnections or in synapse strength
• AN stores in its contiguous memory locations
• In AN the loading may sometimes overload the memory
• Some of the information in the contiguous memory may be
lost
• In biological neuron the new information can be added in the
interconnections by adjusting the strength without the
information being lost
• In Biological N, the recalling may fail, but in AN it never fails
COMPARISON BETWEEN BIOLOGICAL NEURON
AND ARTIFICIAL NEURON
• TOLERENCE
• The biological N has fault tolerance capability where as the
AN has no fault tolerance
• In biological N the storage and retrieval continues even if the
interconnections get disconnected
• In AN the information gets corrupted if the interconnections
are disconnected
• Biological N accepts redundancies
• This is not possible for AN
COMPARISON BETWEEN BIOLOGICAL NEURON
AND ARTIFICIAL NEURON
• CONTROL MECHANISM
• In AN there is a control unit in CPU, which can transfer and
control precise scalar values from unit to unit
• There is no such control unit in the brain
• The strength of the N in the brain depends on the active
chemicals present and whether neuron connections are
strong or weak
• So in turn on the structure of the layer rather than the
individual synapses
• ANs possess simpler interconnections and is free from
chemical actions similar to those taking place in brain
• Control mechanism in AN is simpler than that in brain
 BASIC MODELS OF ARTIFICIAL NEURAL
NETWORK
• The models are specified by the three basic entities

 The model’s synaptic (through synapses) interconnections

 The training or learning rules adopted for updating and
adjusting the connection weights
 The activation functions
 CONNECTIONS

• The neurons can be visualised for their arrangements in layers

• An ANN consists of a set of highly interconnected processing
elements called neurons
• Output of each processing element is found to be connected
through weights to the other processing elements or itself
• Arrangement of these processing elements and the geometry
of their interconnections are essential for an ANN
• The point where the connection originates and terminates
should be noted
• The function of each processing element in an ANN should be
specified
 BASIC NEURON CONNECTION ARCHITECTURES

• There are five types of basic connections:

1. SINGLE-LAYER FEED FORWARD NETWORK

[Link]-LAYER FEED FORWARD NETWORK

[Link] NODE WITH ITS OWN FEEDBACK

[Link]-LAYER RECURRENT NETWORK

[Link]-LAYER RECURRENT NETWORK

 SINGLE LAYER FEED FORWARD NETWORK

• Architecture
X1 w11
Y1
w21
w12
X2 w22
Y2
Input wn1 Output
neurons w2m wn2 neurons

w1m

Xn Ym
wnm

Input Layer Output Layer

 DETAILED DESCRIPTIONS

• Neural nets are classified into single-layer or multi-layer

neural nets
• A layer is formed by taking a processing element and
combining it with other processing elements
• Practically a layer implies a stage, going stage by stage
• This means the input stage and output stage are linked with
each other
• These linked interconnections lead to various network
architectures
 SINGLE LAYER FEED FORWARD NETWORK

• From each node in the input layer we have connections to

each of the output nodes with various weights
• This architecture is called the single-layer feed forward
network
• The input nodes are say X i , i  1, 2,...n
• The output nodes be say Y j , j  1, 2...m
• The connections from ‘n’ input nodes to the ‘m’ output nodes
may be assigned with weights be given by
wij , i  1, 2...n ; j  1, 2...m
 MULTI-LAYER FEED FORWARD NETWORK
• It is formed by the interconnection of several layers
• Input layer is the one which receives the input
• It simply buffers the input signal
• The output layer generates the output of the network
• Any layer that is formed between the input and the output
layer are called the hidden layers
• The hidden layers are internal to the network and do not have
any connection with the external environment
• More the number of hidden layers more is the complexity of
the network
• This may provide an efficient output response
• Every output from one layer is connected to each and every
node in the next layer
 MULTI-LAYER FEED FORWARD NETWORK

• Architecture
weights weights weights

Input
neurons Output
neurons

Input Output
Layer Hidden Layers Layer
 FEEDBACK NETWORKS

• Networks are said to be feed forward if no neuron in the

output layer sends an input to a node in the same layer or in
the preceding layer
• When the outputs can be directed back as inputs to same or
preceding layer nodes then it results in the formation of
feedback networks
• When the feedback is directed as input to the nodes in the
same layer it is called lateral feedback
• When each node gets two types of inputs; excitatory (from
nearby processing elements) and inhibitory (from most
distinctly located processing elements) it is called lateral
inhibition structure network
SINGLE NODE WITH ITS OWN FEEDBACK

Output
Input

Feedback
 SINGLE LAYER RECURRENT NETWORK

• slr

X1 w11
Y1

X2 w22
YY2

wnm
Xn Ym
 LEARNING

• Main property of ANN is learning

• There are two kinds of learning in ANNs
• Parameter learning
It updates the connecting weights in a neural network
• Structure Learning
It focuses on the change in structure in the network
Structure:
 Number of processing elements
 The types of connection between nodes
 LEARNING CATEGORIES

• Supervised learning:
Learning through a teacher/ supervisor

• Unsupervised learning:
Learning without a teacher /supervisor

• Reinforcement learning:
Learning basing upon a critic information
It is similar to supervised learning
 SUPERVISED LEARNING

• Example: A child learning with the help of a teacher or parents

• Here each and every action of the child is supervised by a
teacher
• The child works on the basis of the output that he/she has to
produce
• In ANN context, each input vector requires a corresponding
target vector, which represents the output vector
• Training pair:
• A pair of input vector and the output vector
 SUPERVISED LEARNING CONTD…

• During training an input vector generates a actual output

vector
• This actual output vector is compared with the desired output
vector
Input Neural
Network W Y
Actual Output

Error
D-Y signals Error Signal D
generator Desired
Output
 SUPERVISED LEARNING CONTD…

• If there is a difference between the two vectors, then an error

signal is generated by the network
• This error signal is used for adjustment of weights until the
output matched the desired output

• NOTE: It is assumed that correct target output values are

known for each input pattern
 UNSUPERVISED LEARNING

• Example: A fish learns swimming by itself, not taught by its

mother
• In the context of ANN
• The input vectors of similar type are grouped without the use
of training data to specify how a member of each group looks
or to which group a member belongs
• In the training process the network receives the input patterns
and organises these patterns into clusters
• When a new pattern arrives the ANN gives an output
response indicating the class to which the pattern belongs
 UNSUPERVISED LEARNING CONTD…

• If for a pattern no class could be found then a new class is

formulated

Artificial Neural Network W Y

X (Input) Actual
Output
 REINFORCEMENT LEARNING

• Similar to supervised learning

• In some cases instead of the exact output vector less
information may be available
• This information is called “critic information”
• The learning based upon critic information is called
reinforcement learning and the feedback sent is called
“reinforcement signal”
• The external reinforcement signals are processed in the critic
signal generator and the obtained critic signals are sent to the
ANN for adjustment of weights properly
 REINFORCEMENT LEARNING CONTD…

• Adjustment of weights are done by the ANN so as to get

better critic feedback in future

X Neural Network W
Y
Input
Actual Output

Error
Signals Error signal R
generator Reinforcement
signal
 ACTIVATION FUNCTIONS

• It helps in achieving the exact output

• The activation function is applied over the net input to
calculate the output of an ANN
• A processing element in an ANN comprises of two major parts:
the input and the output
• Mostly non-linear functions are used to achieve the advantages
of a multilayer network over a single layer network
• If a linear activation function is used in a multilayer ANN then
the output remains same as that obtained from a single layer
network
 TYPES OF ACTIVATION FUNCTIONS

• Identity function: It is a linear function f(x) = x for all x

• : The output remains the same as the input
• Binary step function: This function is mostly used in single
layer ANN to convert the net input into a binary output
• 1, if x   ;
f ( x)  
0, if x   .
• Here  represents the threshold value
• Bipolar step function: This function is also used in single layer
ANN to convert the net input to an output that is bipolar (+1
or -1)
1, if x   ;
f ( x)  
1, if x   .
 TYPES OF ACTIVATION FUNCTIONS CONTD…

• SIGMOIDAL FUNCTIONS: These are used in back propagation

ANN because of the relationship between the value of the
functions at a point and the value of the derivative at that
point which reduces the computational burden during training
• Binary sigmoid function: (Also called as logistic sigmoid
function or unipolar sigmoid function)
1
f ( x)   x
, 
• 1 e where is the steepness parameter
f '( x)   . f ( x)[1  f ( x)]
• Derivative is
• The range of this function is 0 to 1.
Derivative of Sigmoidal function
 BIPOLAR SIGMOID FUNCTION

• This function is defined as

1  e  x
• f ( x)   x
, where  is the steepness parameter
1 e
• The value of this function lies between -1 and +1

• f ' ( x)  ( / 2)[1  f ( x)][1  f ( x)]

 HYPERBOLIC TANGENT FUNCTION

• It is closely related to the bipolar sigmoid function

e x  e x
• h( x )  x
e  e x
1  e 2 x
h( x ) 
1  e 2 x
• The derivative of this function is given by
h '( x)  [1  h( x)][1  h( x)]
 RAMP FUNCTION

• The ramp function is given by

1, if x  1;

f ( x)   x, if 0  x  1;
0, if x  0.

 IMPORTANT TERMINOLOGIES

• Weights:
• In ANN architecture each neuron is connected to other
neurons by means of directed communication links
• Each communication link is associated with a weight
• The weights contain information about input signal
• This information is used by the ANN to solve a problem
• The weight can be represented by a matrix
• This matrix is called the connection matrix
 THE CONNECTION MATRIX

• Suppose there are ‘n’ processing elements in an ANN and

each processing element has exactly ‘m’ adaptive weights
• The weight matrix W is defined by

 w1T   w11 w12 . . w1m 

 T 
 w2   w21 w22 . . w2 m 
W   .  . . . . . 
  
 .  . . . . . 
 wT   wn1 wn 2 . . wnm 
 n
 WEIGHTS CONTD…

• The vector wi  [ wi1 , wi 2 ,...wim ]T , i  1, 2,...n is the weight vector

of the processing element
• wij is the weight from the processing element ‘i’ to the
processing element ‘j’

• The set of all weight matrices will determine the set of all
possible information processing configurations for the ANN

• The ANN can be realised by finding an appropriate matrix W

 BIAS
• The bias included in the ANN has its impact in calculating the net input
• The bias is included by adding a component x0 =1 to the input vector X
• The vector becomes X = (1, x1 , x2 ,...xn )
• The bias is considered like another weight
• Thus n n
( yin ) j   xi wij  b j   xi wij w0 j  b j
i 0 i 1

• On the other hand Bias is like the intercept added in a linear equation. It is
an additional parameter in the Neural Network which is used to adjust the
output along with the weighted sum of the inputs to the neuron.
• Therefore Bias is a constant which helps the model in a way that it can fit
best for the given data.
• bias helps in controlling the value at which activation function will trigger.
 THRESHOLD

• Threshold is a pre-defined value based upon which the final

output of the ANN is to be computed
• It is used in the activation function
• A comparison is made between the net input and the
threshold to obtain the net output
• For each and every application there is a threshold
• The activation functions are defined in terms of the threshold
values and the outputs are computed
• Ex.
1, if input   ;
• f(input)  
1, if input   .
 LEARNING RATE

• The learning rate is denoted by ' '

• It is used to control the amount of weight adjustment at each
step of training
• It ranges from 0 to 1
• Determines the rate of learning at every step
 MOMENTUM FACTOR

• The convergence is made faster if a momentum factor is

added to the weight updation process
• This is generally done in the back propagation ANN
• For momentum to be used, the weights from one or more of
previous training patterns must be saved
• Momentum helps the net in reasonably large weight
adjustments until the corrections are in the same general
direction for several patterns
 McCulloch-Pitts Neuron
• It is the earliest neuron discovered in 1943
• Popularly called M-P neuron
• Here, the neurons are connected by directed weighted paths
• Activation function for M-P neuron is binary
• This means at every time step the neuron may or may not fire
• The weights attached with the communication links may be
excitatory (+ve) or inhibitory (-ve)
• All the excitatory connection weights are same in value
• The threshold plays a major role here
• If the net input to the neuron is greater than the threshold
then the neuron fires
• Any non-zero inhibitory input would prevent the neuron from
firing
 THE ARCHITECTURE

• The M-P neuron model architecture

x1
X1

x2 w
X2
w

xn w Y y
Xn
-p

xn+1 -p
Xn+1

xn+m Xn+m
 M-P NEURON CONTD…

• It has excitatory weight ‘w’

• It has inhibitory weight ‘-p’
• The activation function is defined by
1, if yin   ;
f ( yin )  
0, if yin   .
• The output will fire if it receives ‘k’ or more excitatory inputs
but no inhibitory inputs, where
k .w    (k  1).w
• It is used as building blocks on which we can model any
function or phenomenon, which can be represented as a logic
function
McCulloch-Pitts Neuron Model
 LINEAR SEPARABILITY

• An ANN does not provide exact solution for a nonlinear

problems
• It provides possible approximation solutions to nonlinear
problems
• Linear separability is the concept wherein the separation of
the input space into regions is based on whether the network
response is positive or negative
• A decision line is drawn to separate positive and negative
responses
• Decision line = decision-making line= decision-support line =
linear-separable line
 LINEAR SEPARABILITY CONTD…

• We have the formula for the net input given by

n
yin  b   xi wi
i 1

• It is clear that there exists a boundary between the regions

where yin  0 and yin  0
• The region may be called as decision boundary and can be
determined by the relation n
b   xi wi  0
i 1
• Basing upon the number of input units in the ANN the above
equation may represent a line or plane or a hyperplane
 LINEAR SEPARABILITY CONTD…

• The linear separability of the network is based on the decision

boundary line
• If there exist weights for which the training input vectors
having positive response, +1, lie on one side of the decision
boundary and all the other vectors having negative response ,
-1, on the other side of the decision boundary then we can
conclude that the problem is “linearly separable”
• Let us consider the ANN in the next slide
 EXAMPLE OF LINEAR SEPARABILITY

• Consider the single layer neural network

x0
1
b

x1 w1
X1 Y y
w2 Output

x2
X2

• The net input is yin  b  x1.w1  x2 .w2

• The separating line is b  x .w  x .w  0
1 1 2 2
 EXAMPLE OF LINEAR SEPARABILITY

• The requirements for positive response of the ANN is

b  x1.w1  x2 .w2  0
• During the training process, the values of w1 , w2 , b are
determined so that the ANN will produce a positive response
for the training data
•
 EXAMPLE FOR M-P ANN

• Implement AND function using M-P Neuron

• We have the truth table for the AND function given by

x1 x2 y
1 1 1
1 0 0
0 1 0
0 0 0
 EXAMPLE CONTD…

• Let us assume that the two weights be w1=1, w2=1

• Here
yin (1,1)  x1 .w1  x2 .w2  2
yin (1, 0)  x1 .w1  x2 .w2  1
yin (0,1)  x1 .w1  x2 .w2  1
yin (0, 0)  x1 .w1  x2 .w2  0

• For an AND input the output is high if both the inputs are high
• We set the threshold basing upon the net input
• So,   2 . Thus, the output of neuron Y can be written as
1 if yin  2;
y  f ( yin )  
0 if yin  2.
 EXAMPLE CONTD…

• The computation of the value of  can be seen as follows:

• If there are ‘n’ excitatory weights of value ‘w’ and ‘m’
inhibitory weights of value ‘p’ then we should have
  n.w  m. p
• Here, n = 2, w = 1 and m = 0 . So,   2 1  0  p or   2
• So, we take the threshold to be 2
 HEBB NETWORK

• The Hebb learning rule was put forth by Donald Hebb in 1949
• Statement:
• When an axon of cell A is near enough to excite cell B and
repeatedly or permanently takes place in firing it, then some
growth process or metabolic change takes place in one or
both the cells such that A’s efficiency, as one of the cells
firing B is increased
• The weight update in Hebb rule is given by
wi (new)  wi (old )  xi . y
• The Hebb rule is more suited for bipolar data than binary data
 HEBB NETWORK

• If binary data is used then the two following cases cannot be

distinguished
• Case-1: A training pair in which an input unit is ‘on’ and the
target value is ‘off’
• Case-2: A training pair in which both the input and the target
value are ‘off’
• The Training Algorithm for Hebb ANN:
• STEP-1: first initialize the weights. In this network they may be
set to 0. ( wi  0 , i = 1,2,…n)
• STEP-2: Steps 2 to 4 have to be performed for each input
training vector and target output pair (s:t)
 TRAINING ALGORITHM CONTD…

• STEP-2: Input units activations are set. Generally the

activation function of input layer is identity function
• xi  si , for i  1, 2,...n
• STEP-3: Output activations are set: y=t

• STEP-4: Weight adjustments and bias adjustments are

performed
wi (new)  wi (old )  xi . y
• B(new)=b(old) + y
 TRAINING ALGORITHM CONTD…

• Weight formula can be given in the vector form as

• w(new)= w(old)+ x.y
• Change in weight can be represented as w  x. y
• So, the vector form of the weight formula can be written as
• W(new) = w(old)+ w

• The flow chart is

Start

Initialize weights
 FLOW CHART CONTD…

• Contd.
For
No
Each
s:t

Yes

Activate input units xi = si

Activate output units y = t

 FLOW CHART CONTD…

• Contd..
Weight update
wi(new) = wi(old)+ xi.y

bias update
b(new) = b(old)+y

STOP
 EXAMPLE FOR HEBB NETWORK

• Design a Hebb net to implement logical AND function by using

bipolar inputs and targets
• The training data for the logical AND function is

Inputs------ -> -> Target

x1 x2 b y
1 1 1 1
1 -1 1 -1
-1 1 1 -1
-1 -1 1 -1
 EXAMPLE CONTD…

• Using the Hebb Network training algorithm:

• Initially the weights and bias are zeros. So, we have
w1  w2  b  0
• First Input: [ x1 x2 b]  [111] Target: 1 (i.e. y =1)
• Using Hebb Rule: wi (new)  wi (old )  xi . y
• We have w (new)  w (old )  x . y  0  11  1
1 1 1

w2 (new)  w2 (old )  x2 . y  0  11  1

b(new)  b(old )  y  0  1  1
• These inputs are used in the next iteration
• Here,
w1  x1. y  11  1, w2  x2 . y  1 1  1, b  y  1
 EXAMPLE CONTD…

• Second Input: [ x1 x2 b]  [1  11] target: y = -1

• The weight changes are:
w1  x1. y  1 1  1, w2  x2 . y  1 1  1, b  y  1
• The new weights here are:
w1 (new)  w1 (old )  w1  1  1  0
w2 (new)  w2 (old )  w2  1  1  2
b(new)  b(old )  b  1  1  0
• Third Input: [ x1 x2 b]  [111] target: y = -1
• The weight changes are:
w1  x1. y  1 1  1, w2  x2 . y  1 1  1, b  y  1
 EXAMPLE CONTD…

• The new weights are:

w1 (new)  w1 (old )  w1  0  1  1
w2 (new)  w2 (old )  w2  2  1  1
b(new)  b(old )  b  0  1  1
• Fourth input: [ x1 x2 b]  [1  11] target: y =-1
• The weight changes are:
w1  x1. y  1 1  1, w2  x2 . y  1 1  1, b  y  1
• The new weights are: w (new)  w (old )  w  1  1  2
1 1 1

w2 (new)  w2 (old )  w2  1  1  2

b(new)  b(old )  b  1  1  2