UNIT-2: Learning Algorithms

2.1 Learning and Memory

• One of the primary functions of the brain is associative memory. Learning can be considered
as a process of forming associations between related patterns. The associative memory is
composed of a cluster of units which represent a simple model of a real biological neuron.

• An associative memory, also known as Content-Addressable Memory (CAM) can be

searched for a value in a single memory cycle rather than using a software loop.

• Associative memories can be implemented using networks with or without feedback. Such
associative neural networks are used to associate one set of vectors with another set of vectors,
say input and output patterns.

• The aim of an associative memory is to produce the associated output pattern whenever one
of the input patterns is applied to the neural network. The input pattern may be applied to the
network either as input or as initial state and the output pattern is observed at the outputs of
some neurons constituting the network.

• Associative memories belong to class of neural network that learn according to a certain
recording algorithm. They require information a priori and their connectivity matrices most
often need to be formed in advance. Writing into memory produces changes in the neural
interconnections. Reading of the stored info from memory named recall, is a transformation of
input signals by the network.

• All memory information is spatially distributed throughout the network. Associative memory
enables a parallel search within a stored data. The purpose of search is to output one or all
stored items that matches the search argument and retrieve it entirely or partially.

• The Fig. 2.1.1 shows a block diagram of an associative memory.

Fig. 2.1.1 Block diagram of an associative memory

• In the initialization phase of the associative memory no information is stored because the
information is represented in the w weights they are all set to zero.

• The advantage of neural associative memories over other pattern storage algorithms like
lookup tables or hash codes is that the memory access can be fault tolerant with respect to
variation of the input pattern.
• In associative memories many associations can be stored at the same time. There are different
schemes of superposition of the memory traces formed by the different associations. The
superposition can be simple linear addition of the synaptic changes required for each
association (as in the Hopfield model) or nonlinear.

• The performance of neural associative memories is usually measured by a quantity called

information capacity, that is, the maximum content that can be learned and retrieved, divided
by the number of synapses required.

• An associative memory is a content-addressable structure that maps specific input

representations to specific output representations. It is a system that associates two patterns (X,
Y) such that when one is encountered, the other can be recalled.

• Associative network memory can be static or dynamic.

○ Static: Networks recall an output response after input has been applied in one feed-forward
pass and theoretically without delay. They were termed instantaneous.

○ Dynamic: Memory networks produce recall as a result of output/input feedback interactions,

which requires time.

• There are two classes of associative memory: Auto-associative and hetero-associative.

• Whether auto- or hetero-associative, the net can associate not only the exact pattern pairs used
in training, but is also able to obtain associations if the input is similar to one on which it has
been trained.

2.1.1 Auto-associative Memory

• Auto-associative networks are a special subset of the hetero-associative networks, in which

each vector is associated with itself, i.e. yᵢ = xᵢ for i = 1, …, m. The function of such networks
is to correct noisy input vectors.

• Fig. 2.1.2 shows auto-associative memory.

• Auto-associative memories are content based memories which can recall a stored sequence
when presented with a fragment or a noisy version of it. They are very effective in de-noising
the input or removing interference from the input which makes them a promising first step in
solving the cocktail party problem.

• The simplest version of auto-associative memory is linear associator which is a two-layer

feed-forward fully connected neural network where the output is constructed in a single feed-
forward computation.

Fig. 2.1.2 Auto-associative memory

• Artificial neural networks can be used as associative memories. One of the simplest artificial
neural associative memory is the linear associator. (The Hopfield model and Bidirectional
Associative Memory (BAM) models are some of the other popular artificial neural network
models used as associative memories.)

2.1.2 Hetero-associative Memory Network

• Hetero-associative networks map “m” input vectors X¹, X², …, Xᵐ in n-dimensional space to
m output vectors y¹, y², …, yᵐ in k-dimensional space, such that Xⁱ → yⁱ.

• If ‖Xⁱ − Xʲ‖ < ε then Xⁱ → yʲ. This should be achieved by the learning algorithm, but becomes
very hard when the number of vectors to be learned is too high.

• Fig. 2.1.3 shows block diagram of hetero-associative network.

Fig. 2.1.3 Auto-associative memory

• Fig. 2.1.4 shows the structure of a hetero-associative network without feedback.

Fig. 2.1.4 Hetero-associative network without feedback

• Fig. 2.1.5 shows hetero-associative network with feedback.

Fig. 2.1.5 Hetero-associative network with feedback

2.1.3 The Hopfield Network

• The Hopfield model is a single-layered recurrent network. Like the associative memory, it is
usually initialized with appropriate weights instead of being trained.

• Hopfield Neural Network (HNN) is a model of auto-associative memory. It is a single layer

neural network with feedbacks. Fig. 2.1.6 shows Hopfield network of three units. The Hopfield
network is created by supplying input data vectors, or pattern vectors, corresponding to the
different classes. These patterns are called class patterns.

Fig. 2.1.6 Hopfield network of three units

• Hopfield model consists of a single layer of processing elements where each unit is connected
to every other unit in the network other than itself.

• The output of each neuron is binary number in {−1, 1}. The output vector is the state vector.
Starting from an initial state (given as input vector), the state of the network changes from one
state to another like an automaton. If the state converges, the point to which it converges is
called the attractor.

• In its simplest form, the output function is the sign function, which yields 1 for arguments ≥
0 and −1 otherwise.

• The connection weight matrix W of this type of network is square and symmetric.

• The units in the Hopfield model act as both input and output units.

• A Hopfield network consists of “n” totally coupled units. Each unit is connected to all other
units except itself. The network is symmetric because the weight wᵢⱼ for the connection between
unit i and unit j is equal to the weight wⱼᵢ of the connection from unit j to unit i. The absence of
a connection from each unit to itself avoids a permanent feedback of its own state value.

• Hopfield networks are typically used for classification problems with binary pattern vectors.

• Hopfield model is classified into two categories:

1. Discrete Hopfield Model

2. Continuous Hopfield Model
• In both discrete and continuous Hopfield network weights trained in a one-shot fashion and
not trained incrementally as was done in case of Perceptron and MLP.

• In the discrete Hopfield model, the units use a slightly modified bipolar output function where
the states of the units, i.e., the output of the units remain the same if the current state is equal
to some threshold value.

• The continuous Hopfield model is just a generalization of the discrete case. Here the units use
a continuous output function such as the sigmoid or hyperbolic tangent function. In the
continuous Hopfield model, each unit has an associated capacitor Cᵢ and resistance rᵢ that model
the capacitance and resistance of real neuron’s cell membrane, respectively.

2.1.4 Bidirectional Associative Memory (BAM)

• BAM consists of two layers, x and y. Signals are sent back and forth between both layers
until an equilibrium is reached. Equilibrium is reached if the x and y vectors no longer change.
Given an x vector the BAM is able to produce the y vector and vice versa.

• BAM consists of bi-directional edges so that information can flow in either direction. Since
the BAM network has bidirectional edges, propagation moves in both directions, first from one
layer to another and then back to the first layer. Propagation continues until the nodes are no
longer changing values.

• Fig. 2.1.7 shows BAM network.

Fig. 2.1.7 BAM network

• Since the BAM also uses the traditional Hebb’s learning rule to build the connection weight
matrix to store the associated pattern pairs, it too has a severely low memory capacity.

• BAM can be classified into two categories :

1. Discrete BAM : The network propagates an input pattern X to the Y layer where the
units in the Y layer will compute their net input.
2. Continuous BAM : The units use the sigmoid or hyperbolic tangent output function.
The units in the X layer have an extra external input Iₓ, while the units in the Y layer
have an extra external input Iᵧ for i = 1, 2, …, m and j = 1, 2, …, n.

• These extra external inputs lead to a modification in the computation of the net input to the
units.
2.1.5 Difference between Auto-associative Memory and
Hetero-Associative Memory
Auto-associative memory | Hetero-associative memory
Auto-associative memory Hetero-associative memory

The inputs and output vectors s and t are the

The inputs and output vectors s and t are different.
same.

Recalls a memory of the same modality as the Recalls a memory that is different in character from
one that evoked it. the input.

A picture of a favorite object might evoke a A particular smell or sound, for example, might
mental image of that object in vivid detail. evoke a visual memory of some past event.

An auto-associative memory retrieves the same Hetero-associative memory retrieves the stored
pattern. pattern.

Example : 1. Space transforms : Fourier, 2.

Example : color correction, color constancy.
Dimensionality reduction : PCA

2.2 Hebbian Learning

• In 1949, Donald Hebb proposed one of the key ideas in biological learning, commonly known
as Hebb’s Law. Hebb’s Law states that if neuron i is near enough to excite neuron j and
repeatedly participates in its activation, the synaptic connection between these two neurons is
strengthened and neuron j becomes more sensitive to stimuli from neuron i.

• Hebb’s Law can be represented in the form of two rules :

1. If two neurons on either side of a connection are activated synchronously, then the
weight of that connection is increased.
2. If two neurons on either side of a connection are activated asynchronously, then the
weight of that connection is decreased.

• Hebb’s Law provides the basis for learning without a teacher. Learning here is a local
phenomenon occurring without feedback from the environment.

• Using Hebb’s Law we can express the adjustment applied to the weight wᵢⱼ at iteration p in
the following form :

Δwij(p)=F[yi(p),xj(p)]

• As a special case, we can represent Hebb’s Law as follows :

Δwij(p)=α yi(p) xj(p)

where α is the learning rate parameter. This equation is referred to as the activity product rule.
• Hebbian learning implies that weights can only increase. To resolve this problem, we might
impose a limit on the growth of synaptic weights. It can be done by introducing a non-linear
forgetting factor into Hebb’s Law :

Δwij(p)=α yi(p) xj(p)−ϕ yi(p) wij(p)

where φ is the forgetting factor.

• Forgetting factor usually falls in the interval between 0 and 1, typically between 0.01 and 0.1,
to allow only a little “forgetting” while limiting the weight growth.

Hebbian learning algorithm

Step 1 : Initialisation

Set initial synaptic weights and thresholds to small random values, say in an interval [0, 1].

Step 2 : Activation

Compute the neuron output at iteration p

where n is the number of neuron inputs, and θⱼ is the threshold value of neuron j.

Step 3 : Learning

Update the weights in the network :

wij(p+1)=wij(p)+Δwij(p)

where wᵢⱼ(p) is the weight correction at iteration p. The weight correction is determined by the
generalised activity product rule :

Δwij(p)=ϕ yi(p) [λ xi(p)−wij(p)]

Step 4 : Iteration

Increase iteration p by one, go back to Step 2.

Hebbian learning example

To illustrate Hebbian learning, consider a fully connected feed forward network with a single
layer of five computation neurons. Each neuron is represented by a McCulloch and Pitts model
with the sign activation function. The network is trained on the following set of input vectors:
Initial and final weight matrices
Initial Weight Matrix (W₀)

Final Weight Matrix (W)

• A test input vector, or probe, is defined as:

• When this probe is presented to the network, we obtain:

2.3 Competitive Learning

• Fig. 2.3.1 shows basic architecture of a competitive learning system.

• It consists of a set of hierarchically layered units in which each layer connects, via excitatory
connections, with the layer immediately above it, and has inhibitory connections to units in its
own layer.

• In the most general case each unit in a layer receives an input from each unit in the layer
immediately below it and projects to each unit in the layer immediately above it.

• Moreover, within a layer, the units are broken into a set of inhibitory clusters in which all
elements within a cluster inhibit all other elements in the cluster.

• Thus the elements in a cluster at one level compete with one another to respond to the pattern
appearing on the layer below. The more strongly any particular unit responds to an incoming
stimulus, the more it shuts down the other members of its cluster.

• Units are represented in the diagram as dots. Units may be active or inactive. Active units are
represented by filled dots, inactive ones by open dots.

• Properties :

1. The units in a given layer are broken into several sets of non-overlapping clusters. Each
unit within a cluster inhibits every other unit within a cluster. Within each cluster, the
unit receiving the largest input achieves its maximum value while all other units in the
cluster are pushed to their minimum value. We have arbitrarily set the maximum value
to 1 and the minimum value to 0.

2. Every unit in every cluster receives inputs from all members of the same set of input
units.
3. A unit learns if and only if it wins the competition with other units in its cluster.
 4. A stimulus pattern Sj consists of a binary pattern in which each element of the pattern
is either active or inactive. An active element is assigned the value 1 and an inactive
element assigned the value 0.
5. Each unit has a fixed amount of weight (all weights are positive) that is distributed
among its input lines.

• There are several characteristics of a competitive learning mechanism :

1. Each cluster classifies the stimulus set into M groups, one for each unit in the cluster.
Each of the units captures roughly an equal number of stimulus patterns.
2. If there is structure in the stimulus patterns, the units will break up the patterns along
structurally relevant lines.
3. If the stimuli are highly structured, the classifications are highly stable. If the stimuli
are less well structured, the classifications are more variable, and a given stimulus
pattern will be responded to first by one and then by another member of the cluster.
4. The particular grouping done by a particular cluster depends on the starting value of the
weights and the sequence of stimulus patterns actually presented.
5. To a first approximation, the system develops clusters that minimize within-cluster
distance, maximize between-cluster distance, and balance the number of patterns
captured by each cluster.

2.4 Error Correction

• Learning is a process where unknown ANN parameters are adapted through continuous
process of stimulation from the environment. Learning is determined by the way how change
of parameters takes place. A set of rules that are solution to the learning problem is called a
learning algorithm.

• Error correction belongs to the supervised learning paradigm. Fig. 2.4.1 shows error
correction learning.

• Neuron (k) is driven by a signal vector x(n) produced by one or more layers of hidden neurons,
which are themselves driven by an input vector applied to the source nodes of the neural
network.

• Let dₖ(n) be desired output of neuron k at moment n

• Let yₖ(n) be obtained output of neuron k at moment n

• Output yₖ(n) is obtained using input vector x(n).

• Input vector x(n) and desired output dₖ(n) represent an example that is presented to ANN at
moment n.

• Error is the difference between desired and obtained output of neuron k at moment n :

ek(n)=dk(n)−yk(n)

• Error signal eₖ(n) actuates a control mechanism.

• The goal of error-correction learning is to minimize an error function derived from errors
eₖ(n) so that the obtained output of all neurons approximates the desired output in some
statistical sense. A frequently used error function is mean square error :

where E[.] is the statistical expectation operator and summation is for all neurons in the output
layer.

2.5 Gradient Decent Rules

• Much of machine learning can be written as an optimization problem.

• Example loss functions : Logistic regression, linear regression, principle component analysis,
neural network loss.

• A very efficient way to train logistic models is with Stochastic Gradient Descent (SGD).

• One challenge with training on power law data (i.e. most data) is that the terms in the gradient
can have very different strengths.

• The idea behind stochastic gradient descent is iterating a weight update based on the gradient
of loss function :

w(k+1) = w(k) − γ∇L(w)

• Logistic regression is designed as a binary classifier (output say {0, 1}) but actually outputs
the probability that the input instance is in the "1" class.

• A logistic classifier has the form :

where
X=(X1,....,Xn) is a vector of features.

• Stochastic gradient has some serious limitations however, especially if the gradients vary
widely in magnitude. Some coefficients change very fast, others very slowly.

• This happens for text, user activity and social media data (and other power-law data), because
gradient magnitudes scale with feature frequencies, i.e. over several orders of magnitude.

• It is not possible to set a single learning rate that trains the frequent and infrequent features at
the same time.

• An example of stochastic gradient descent with perceptron loss is shown as follows :

from sklearn.linear_model import SGDClassifier

2.5.1 Finding the Optimal Hyper-Parameters through Grid Search

• In statistics, hyperparameter is a parameter from a prior distribution; it captures the prior

belief before data is observed.

• In any machine learning algorithm, these parameters need to be initialized before training a
model.

• Model hyperparameters are the properties that govern the entire training process.

• Hyperparameters are important because they directly control the behaviour of the training
algorithm and have a significant impact on the performance of the model is being trained.

• Choosing appropriate hyperparameters plays a crucial role in the success of our neural
network architecture. Since it makes a huge impact on the learned model.

• For example, if the learning rate is too low, the model will miss the important patterns in the
data. If it is high, it may have collisions.

• Choosing good hyperparameters gives two benefits

1. Efficiently search the space of possible hyperparameters.

2. Easy to manage a large set of experiments for hyperparameter tuning.

• The process of finding most optimal hyperparameters in machine learning is called

hyperparameter optimisation.

• Grid search is a very traditional technique for implementing hyperparameters. It brute force
all combinations. Grid search requires to create two set of hyperparameters.

1. Learning rate
2. Number of layers
• Grid search trains the algorithm for all combinations by using the two set of hyperparameters
and measures the performance using "Cross Validation" technique.

• This validation technique gives assurance that our trained model got most of the patterns from
the dataset.

• One of the best methods to do validation by using "K-Fold Cross Validation" which helps to
provide ample data for training the model and ample data for validations.

• With this technique, we simply build a model for each possible combination of all of the
hyperparameter values provided, evaluating each model and selecting the architecture which
produces the best results.

• For example, say you have two continuous parameters α and β, where manually selected
values for the parameters are the following :

α ∈ {0, 1, 2}
β ∈ {25, 50, 75}

• Then the pairing of the selected hyperparametric values, H, can take on any of the following:

H ∈ {(0, 25), (0, 50), (0, 75), (1, 25), (1, 50), (1, 75), (2, 25), (2, 50), (2, 75)}

• Grid search will examine each pairing of and to determine the best performing combination.
The resulting pairs, H, are simply each output that results from taking the Cartesian product of
α and β.

• While straightforward, this "brute force" approach for hyperparameter optimization has some
drawbacks. Higher-dimensional hyperparameter spaces are far more time consuming to test
than the simple two-dimensional problem presented here.

• Also, because there will always be a fixed number of training samples for any given model,
the model's predictive power will decrease as the number of dimensions increases. This is
known as a Hughes phenomenon.

2.5.2 Vanishing Gradient Problem

• When back-propagation is used, the earlier layers will receive very small updates compared
to the later layers. This problem is referred to as the vanishing gradient problem.

• The vanishing gradient problem is essentially a situation in which a deep multilayer feed-
forward network or a Recurrent Neural Network (RNN) does not have the ability to propagate
useful gradient information from the output of the model back to the layers near the input end
of the model.

• Weight initialization is one technique that can be used to solve the vanishing gradient
problem. It involves artificially creating an initial value for weights in a neural network to
prevent the backpropagation algorithm from assigning weights that are unrealistically small.
• The most important solution to the vanishing gradient problem is a specific type of neural
network called Long Short-Term Memory Networks (LSTMs).

• Indication of vanishing gradient problem :

a) The parameters of the higher layers change to a great extent, while the parameters of lower
layers barely change.
b) The model weights could become 0 during training.
c) The model learns at a particularly slow pace and the training could stagnate at a very early
phase after only a few iterations.

• Some methods that are proposed to overcome the vanishing gradient problem :
a) Residual neural networks (ResNets)
b) Multi-level hierarchy
c) Long short term memory (LSTM)
d) Faster hardware
e) ReLU
f) Batch normalization

2.6 Supervised Learning

• Supervised learning is the machine learning task of inferring a function from supervised
training data. The training data consist of a set of training examples. The task of the supervised
learner is to predict the output behavior of a system for any set of input values, after an initial
training phase.

• Supervised learning in which the network is trained by providing it with input and matching
output patterns. These input-output pairs are usually provided by an external teacher.

• Human learning is based on the past experiences. A computer does not have experiences.

• A computer system learns from data, which represent some "past experiences" of an
application domain.

• To learn a target function that can be used to predict the values of a discrete class attribute,
e.g., approve or not-approved and high-risk or low risk. The task is commonly called :
Supervised learning, Classification or inductive learning.

• Training data includes both the input and the desired results. For some examples the correct
results (targets) are known and are given in input to the model during the learning process. The
construction of a proper training, validation and test set is crucial. These methods are usually
fast and accurate.

• Have to be able to generalize : give the correct results when new data are given in input
without knowing a priori the target.

• Supervised learning is the machine learning task of inferring a function from supervised
training data. The training data consist of a set of training examples. In supervised learning,
each example is a pair consisting of an input object and a desired output value.
• A supervised learning algorithm analyzes the training data and produces an inferred function,
which is called a classifier or a regression function. Fig. 2.6.1 shows supervised learning
process.

• The learned model helps the system to perform task better as compared to no learning.

• Each input vector requires a corresponding target vector.

Training Pair = (Input Vector, Target Vector)

• Fig. 2.6.2 shows input vector.

• Supervised learning denotes a method in which some input vectors are collected and presented
to the network. The output computed by the network is observed and the deviation from the
expected answer is measured. The weights are corrected according to the magnitude of the
error in the way defined by the learning algorithm.

• Supervised learning is further divided into methods which use reinforcement or error
correction. The perceptron learning algorithm is an example of supervised learning with
reinforcement.

Data formats in supervised learning :

• Supervised learning uses a dataset to define finite set of real vectors with m features each :

• Considering that user approach is always probabilistic, we need to consider each X as drawn
from a statistical multivariate distribution D. It is also useful to add an important condition
upon the whole dataset X. Here we consider that all samples to be independent and identically
distributed. This means all variables belong to the same distribution D and considering an
arbitrary subset of m values, it happens that :

• The corresponding output values can be both numerical - continuous and categorical. In the
first case, the process is called regression, while in the second, called classification.

• Example : Dataset contains city populations by year for the past 100 years and we want to
know what the population of a specific city will be four years from now. The outcome uses
labels that already exist in the data set : population, city and year.

• In order to solve a given problem of supervised learning, following steps are performed :

1. Find out the type of training examples.

2. Collect a training set.
3. Determine the input feature representation of the learned function.
4. Determine the structure of the learned function and corresponding learning algorithm.
5. Complete the design and then run the learning algorithm on the collected training set.
6. Evaluate the accuracy of the learned function. After parameter adjustment and learning,
the performance of the resulting function should be measured on a test set that is
separate from the training set.

• Supervised learning is divided into two types : Classification and Regression.

1. Classification :

• Classification predicts categorical labels (classes), prediction models continuous - valued

functions. Classification is considered to be supervised learning.

• Classifies data based on the training set and the values in a classifying attribute and uses it in
classifying new data. Prediction means models continuous-valued functions, i.e., predicts
unknown or missing values.

• Preprocessing of the data in preparation for classification and prediction can involve data
cleaning to reduce noise or handle missing values, relevance analysis to remove irrelevant or
redundant attributes and data transformation, such as generalizing the data to higher level
concepts or normalizing data.

• Numeric prediction is the task of predicting continuous values for given input. For example,
we may wish to predict the salary of college employee with 15 years of work experience or the
potential sales of a new product given its attributes.

• Some of the classification methods like back - propagation, support vector machines and k -
nearest - neighbor classifiers can be used for prediction.

Regression :
• For an input x, if the output is continuous, this is called a regression problem. For example,
based on historical information of demand for tooth paste in supermarket, user are asked to
predict the demand for the next month.

• Regression is concerned with the prediction of continuous quantities. Linear regression is the
oldest and most widely used predictive model in the field of machine learning. The goal is to
minimise the sum of the squared errors to fit a straight line to a set of data points.
• Regression algorithms used in supervised learning is linear regression, Bayesian linear
regression, polynomial regression, regression tree etc.

2.6.1 Advantages and Disadvantages of Supervised Learning

1. Advantages of supervised learning

• It performs classification and regression tasks.
• It allows estimating or mapping the result to a new sample.
• We have complete control over choosing the number of classes we want in training
data.
2. Disadvantages of supervised learning
• Supervised learning cannot handle all complex tasks in Machine Learning.
• Computation time is vast for supervised learning.
• It requires a labelled data set.
• It requires a training process.

2.6.2 Difference between Supervised and Unsupervised Learning

S.
Supervised learning Unsupervised learning
No.
1. Desired output is given. Desired output is not given.
It is not possible to learn larger and It is possible to learn larger and more
2. more complex models than with complex models than with unsupervised
supervised learning. learning.
3. Use training data to infer model. No training data is used.
Every input pattern that is used to train
The target output is not presented to the
4. the network is associated with an output
network.
pattern.
Trying to predict a function from labelled
5. data.
Try to detect interesting relations in data.

Supervised learning requires that the target For unsupervised learning typically either the
6. variable is well defined and that a sufficient target variable is unknown or has only been
number of its values are given. recorded for too small a number of cases.
7. Example: Optical character recognition. Example: Find a face in an image.
8. We can test our model. We cannot test our model.
Supervised learning is also called
9. Unsupervised learning is also called clustering.
classification.

2.7 Backpropagation – backward propagation

• Backpropagation is a training method used for a multi-layer neural network also called the
generalized delta rule. It is a gradient descent method which minimizes the total squared error
of the output computed by the net.

• The backpropagation algorithm looks for the minimum value of the error function in weight
space using a technique called the delta rule or gradient descent. The weights that minimize the
error function is then considered to be a solution to the learning problem.

• Backpropagation is a systematic method for training multiple layer ANN. It is a generalization

of Widrow-Hoff error correction rule. 80% of ANN applications use backpropagation.

• Consider a simple neuron :

a. Neuron has a summing junction and activation function.
b. Any non linear function which differentiable everywhere and increases everywhere with
sum can be used as activation function.
c. Examples : Logistic function, Arc tangent function, Hyperbolic activation function.

• These activation function makes the multilayer network to have greater representational
power than single layer network only when non-linearity is introduced.

Need of hidden layers :

1. A network with only two layers (input and output) can only represent the input with
whatever representation already exists in the input data.
2. If the data is discontinuous or non-linearly separable, the innate representation is
inconsistent, and the mapping cannot be learned using two layers (Input and Output).
3. Therefore, hidden layer(s) are used between input and output layers.

• Weights connects unit (neuron) in one layer only to those in the next higher layer.
• The output of the unit is scaled by the value of the connecting weight and it is fed forward to
provide a portion of the activation for the units in the next higher layer.
• Backpropagation can be applied to an artificial neural network with any number of hidden
layers. The training objective is to adjust the weights so that the application of a set of inputs
produces the desired outputs.
Training procedure : The network is usually trained with a large number of input-output pairs.

1. Generate weights randomly to small random values (both positive and negative) to
ensure that the network is not saturated by large values of weights.
2. Choose a training pair from the training set.
3. Apply the input vector to network input.
4. Calculate the network output.
5. Calculate the error, the difference between the network output and the desired output.
6. Adjust the weights of the network in a way that minimizes this error.
7. Repeat steps 2 – 6 for each input-output in the training set until the error in the system
is acceptably low.

Forward pass and backward pass :

• Backpropagation neural network training involves two passes.

1. In the forward pass, the input signals moves forward from the network input to the
output.
2. In the backward pass, the calculated error signals propagate backward through the
network, where they are used to adjust the weights.
3. In the forward pass, the calculation of the output is carried out, layer by layer, in the
forward direction. The output of one layer is the input to the next layer.

• In the reverse pass,

a. The weights of the output neuron layer are adjusted first since the target value of each output
neuron is available to guide the adjustment of the associated weights, using the delta rule.
b. Next, we adjust the weights of the middle layers. As the middle layer neurons have no target
values, it makes the problem complex.

• Selection of number of hidden units : The number of hidden units depends on the number of
input units.

1. Never choose h to be more than twice the number of input units.

2. You can load p patterns of l elements into log₂ p hidden units.
3. Ensure that we must have at least 1/6 times as many training examples.
4. Feature extraction requires fewer hidden units than inputs.
5. Learning many examples of disjointed inputs requires more hidden units than inputs.
6. The number of hidden units required for a classification task increases with the number
of classes in the task. Large networks require longer training times.

Factors influencing backpropagation training :

• The training time can be reduced by using :

1. Bias : Networks with biases can represent relationships between inputs and outputs
more easily than networks without biases. Adding a bias to each neuron is usually
desirable to offset the origin of the activation function. The weight of the bias is
trainable similar to weight except that the input is always +1.
2. Momentum : The use of momentum enhances the stability of the training process.
Momentum is used to keep the training process going in the same general direction
analogous to the way that momentum of a moving object behaves. In backpropagation
 with momentum, the weight change is a combination of the current gradient and the
previous gradient.

2.7.1 Advantages and Disadvantages

Advantages of backpropagation :

1. It is simple, fast and easy to program.

2. Only numbers of the input are tuned and not any other parameter.
3. No need to have prior knowledge about the network.
4. It is flexible.
5. A standard approach and works efficiently.
6. It does not require the user to learn special functions.

Disadvantages of backpropagation :

1. Backpropagation possibly be sensitive to noisy data and irregularity.

2. The performance of this is highly reliant on the input data.
3. Needs excessive time for training.
4. The need for a matrix-based method for backpropagation instead of mini-batch.

2.8 Feed Forward Neural Networks

• Feed Forward Neural Network is an artificial neural network in which the connections
between nodes does not form a cycle. The feed forward model is the simplest form of neural
network as information is only processed in one direction.
While the data may pass through multiple hidden nodes, it always moves in one direction and
never backwards.

• They are called feed forward because information only travels forward in the network (no
loops), first through the input nodes, then through the hidden nodes (if present) and finally
through the output nodes.

• Feed-forward networks tends to be simple networks that associates inputs with outputs. It can
be used in pattern recognition. This type of organization is represented as bottom-up or top-
down.

• Fig. 2.8.1 shows basic structure of a Feed Forward (FF) Neural Network.

• Input layer contains one or more input nodes. For example, suppose we want to predict
whether it will rain tomorrow and base our decision on two variables, humidity and wind speed.
In that case, our first input would be the value for humidity and the second input would be the
value for wind speed.
• Hidden layer : This layer contains an activation function.
• Output layer contains one or more output nodes.

• Feed forward neural networks are primarily used for supervised learning in cases where the
data to be learned is neither sequential nor time-dependent.

• Feed-forward networks have the following characteristics :

1. Perceptron’s are arranged in layers, with the first layer taking in inputs and the last layer
producing outputs. The middle layers have no connection with the external world and
hence are called hidden layers.
2. Each perceptron in one layer is connected to every perceptron on the next layer. Hence
information is constantly ‘fed’ forward from one layer to the next and this explains why
these networks are called feed-forward networks.
3. There is no connection among perceptron’s in the same layer.

2.8.1 Feedback Neural Networks

• Feedback networks also known as recurrent neural network or interactive neural network are
the deep learning models in which information flows in backward direction. It allows feedback
loops in the network. Feedback networks are dynamic in nature, powerful and can get much
complicated at some stage of execution.

• Fig. 2.8.2 shows feedback neural network.

• Signals can travel in both the directions in feedback neural networks. Feedback neural
networks are very powerful and can get very complicated. Feedback neural networks are
dynamic.
• The ‘state’ in such network keep changing until they reach an equilibrium point. They remain
at the equilibrium point until the input changes and a new equilibrium needs to be found.

• Feedback neural network architecture is also referred to as interactive or recurrent, although

the latter term is often used to denote feedback connections in single-layer organizations.
Feedback loops are allowed in such networks. They are used in content addressable memories.

2.8.3 Example and Applications of Feedforward NN

• Physiological feedforward system : during this, the feedforward management is epitomized

by the conventional preventive regulation of heartbeat prior to work out by the central
involuntary.

• Gene regulation and feedforward : during this, a motif preponderantly seems altogether the
illustrious networks and this motif has been shown to be a feedforward system for the detection
of the non-temporary modification of atmosphere.

• Automation and machine management : Feedforward control may be discipline among the
sphere of automation controls.

• Parallel feedforward compensation with derivative : This a rather new technique that changes
the part of AN open-loop transfer operates of a non-minimum part system into the minimum
part.

2.8.4 Difference between RNNs and feed-forward NN

• In a feed-forward neural network, the information only moves in one direction from the input
layer, through the hidden layers, to the output layer. The information moves straight through
the network.

• Feed-forward neural networks have no memory of the input they receive and are bad at
predicting what’s coming next. Because a feedforward network only considers the current
input, it has no notion of order in time. It simply can’t remember anything about what happened
in the past except its training.

• In a RNN the information cycles through a loop. When it makes a decision, it considers the
current input and also what it has learned from the inputs it received previously.