Module 2
Supervised learning network
Perceptron network
Perceptron networks come under single-layer feed-forward networks and are also
called simple perceptrons.
Properties of perceptron network:
1. The perceptron network consists of three units, namely, sensory unit (input
unit), associator unit (hidden unit), response unit (output unit).
2. The sensory units are connected to associator unit with fixed weights
having values 1, 0 or -l, which are assigned at random.
3. The binary activation function is used in sensory unit and associator unit.
4. The response unit has an'activation of l, 0 or -1. The binary step with fixed threshold
Ø is used as activation for associator. The output signals that are sent from the
associator unit to the response unit are only binary.
Perceptron Learning Rule
Flowchart for Training Process
Perceptron Training Algorithm for Single Output Classes
The perceptron algorithm can be used for either binary or bipolar input vectors,
having bipolar targets,threshold being fixed and variable bias.
Perceptron Training Algorithm for Multiple Output Classes
Perceptron Network Testing Algorithm
Adaptive Linear Neuron (Adaline)
Delta Rule for Single Output Unit
● The Widrow-Hoff rule is very similar to perceptron learning rule. However,
their origins are different. The perceptron learning rule originates from the
Hebbian assumption while the delta rule is derived from the
gradient~descent method.
● Also, the perceptron learning rule stops after a finite number of learning
steps, but the gradient~descent approach continues forever, converging
only asymptotically to the solution.
The delta rule updates the weights between the connections so as to minimize the
difference between the net input to the output unit and the target value. The
major aim is to minimize the error over all training patterns. This is done by reducing
the error for each pattern, one at a time.
Testing Algorithm
It is essential to perform the testing of a network that has been trained. When
training is completed, the Adaline can be used to classify input patterns. A step
function is used to test the performance of the network.
Multiple Adaptive Linear Neurons
The multiple adaptive linear neurons (Madaline) model consists of many
Adalinel with a single output unit whose value is based on certain
selection rules. 'It may use majority vote rule. On using this rule, the
output would have as answer either true or false.
Backpropagation network
Backpropagation is a supervised learning technique which is applied to multi
layer feedforward networks consisting of processing elements with continuous
differentiable activation functions. The networks associated with back propagation
training are called BPN or back propagation networks.
A backpropagation network is a supervised learning neural network in which the error
between the actual output and desired output is propagated backwards from
the output layer to the hidden layers to update the weights.
When the hidden layers are increased the network training becomes more
complex. To update weights, the error must be calculated. The error, Which
is the difference between the actual (calculated) and the desired (target)
output, is easily measured at the output layer. It should be noted that at the
hidden layers, there is no direct information of the error. Therefore, other
techniques should be used to calculate an error at the hidden layer, which
will cause minimization of the output error, and this is the ultimate goal.
The training of the BPN is done in three stages -
1. Feed-forward of the input training pattern
2. The calculation and back-propagation of the error
3. updation of weights.
The testing of the BPN involves the computation of feed-forward phase
only.,There can be more than one hidden layer (more beneficial) bur one
hidden layer is sufficient. Even though the training is very slow, once the
network is trained it can produce its outputs very rapidly.
The commonly used activation functions are binary sigmoidal and bipolar
sigmoidal activation functions.
These functions are used in the BPN because of the following
characteristics:
(i) continuity; (ii) differentiability (iii) nondecreasing monotony (its
output never decreases when its input increases.)
The range of binary sigmoid is from 0 to 1, and for bipolar sigmoid it is from
-1 to+ 1.
update the change in weights and bias:
Learning Factors of Back Propagation Network
The training of a BPN is based on the choice of various parameters.
The convergence of the BPN is based on some important learning factors such as
the initial weights, the learning rate, the updation rule, the size and nature of
the training set, and the architecture (number of layers and number of
neurons per layer).
Initial Weights:
The ultimate solution may be affected by the initial weights of a multilayer feed-forward
network. They are initialized at small random values. The choice of weights
determines how fast the network converges.
The initial weights cannot be very high because of the sigmoid activation function here
may get saturated from the beginning itself and the system may be stuck at a local
minima or at a very flat plateau at the starting point itself.
Learning Rate:
The learning rate (a) affects the convergence of the BPN. A larger value of a may
speed up the convergencebut might result in overshooting, while a smaller value of a
has vice-versa effecr. The range of a from 10- 3to 10 has been used successfulfy for
several back-propagation algorithmic experiments. Thus, a large learning rate leads to
rapid learning bm there is oscillation of wei_g!lts, while the lower learning rare leads to
slower learning.