Module3

Artificial Neural Networks

 Introduction
• Artificial Neural Networks (ANN)
1. Information processing paradigm inspired by biological
nervous systems
2. ANN is composed of a system of neurons connected by
synapses
3. ANN learn by example
• Adjust synaptic connections between neurons
• Computational models inspired by the human brain:
1. Massively parallel, distributed system, made up of simple
processing units (neurons)
2. Synaptic connection strengths among neurons are used to
store the acquired knowledge.
3. Knowledge is acquired by the network from its environment
through a learning process
 Properties of ANNs
1. Learning from examples – labeled or unlabeled
2. Adaptivity – changing the connection strengths to learn
things
3. Non-linearity – the non-linear activation functions are
essential
4. Fault tolerance – if one of the neurons or connections is
damaged, the whole network still works quite well
• Thus, they might be better alternatives than classical
solutions for problems characterized by:
high dimensionality, noisy, imprecise or imperfect data
a lack of a clearly stated mathematical solution or
algorithm
• A neural network is a massively parallel, distributed processor
made up of simple processing units (artificial neurons).
• It resembles the brain in two respects:
– Knowledge is acquired by the network from its environment
through a learning process
– Synaptic connection strengths among neurons are used to
store the acquired knowledge.
Artificial Neural Networks – Basic
Structure
•x
•X
1 •w
1
1
•Y •y
•x •w
•X
2
2
2
 Comparison between Human brain
verses computer (ANN)
Human brain ANN
Speed Few ms. Few nano sec. massive
parallel processing
Size and complexity 1011 neurons & 1015 Depends on designer
interconnections
Storage capacity Stores information in its Contiguous memory
interconnection or in locations
synapse. loss of memory may
No Loss of memory happen sometimes.
Tolerance Has fault tolerance No fault tolerance Inf
gets disrupted when
interconnections are
disconnected
Control mechanism Complicated involves Simpler in ANN
chemicals in biological
neuron
 Activation Function
• In artificial neural networks, the activation function of a node
defines the output of that node given an input or set of inputs. A
standard integrated circuit can be seen as a digital network of
activation functions that can be "ON" (1) or "OFF" (0), depending
on input. This is similar to the linear perceptron in neural
networks. However, only nonlinear activation functions allow such
networks to compute nontrivial problems using only a small
number of nodes, and such activation functions are called
nonlinearities.
 • Types of Activation Functions : The activation function can be
broadly classified into 2 categories.
1. Binary Step Function
2. Linear Activation Function
• Binary Step Function : A binary step function is generally used in
the Perceptron linear classifier. It thresholds the input values to 1
and 0, if they are greater or less than zero, respectively.
 • Linear Activation Function :
• The equation for Linear activation function is:
f(x) = a.x
When a = 1 then f(x) = x and this is a special case known as
identity.
• Properties:
1. Range is -infinity to +infinity
2. Provides a convex error surface so optimization can be
achieved faster
3. df(x)/dx = a which is constant. So cannot be optimized with
gradient descent
• Limitations:
1. Since the derivative is constant, the gradient has no relation
with input
2. Back propagation is constant as the change is delta x
 Non-Linear Activation Functions
• Modern neural network models use non-linear activation
functions. They allow the model to create complex mappings
between the network’s inputs and outputs, such as images,
video, audio, and data sets that are non-linear or have high
dimensionality.
• Majorly there are 3 types of Non-Linear Activation functions.
1. Sigmoid Activation Functions
2. Rectified Linear Units or ReLU
3. Complex Nonlinear Activation Functions
• Sigmoid Activation Functions :
• Sigmoid(or Logistic) functions are bounded, differentiable, real
functions that are defined for all real input values, and have a
non-negative derivative at each point.
• The sigmoid function is a logistic function and the output is ranging
between 0 and 1.
• The output of the activation function is always going to be in range
(0,1) compared to (-inf, inf) of linear function. It is non-linear,
continuously differentiable, monotonic, and has a fixed output
range. But it is not zero centred.
• Hyperbolic Tangent : The function produces outputs in scale
of [-1, 1] and it is a continuous function. In other words,
function produces output for every x value.
Y = tanh(x)
tanh(x) = (ex – e-x) / (ex + e-x)
• Inverse Hyperbolic Tangent (arctanh) :
It is similar to sigmoid and tanh but the output ranges from
[-pi/2,pi/2].
 Optimization algorithm- Gradient
decent
• Machine learning in its most reduced form is sometimes
referred to as glorified curve fitting. In a way, it is true.
Machine learning models are typically founded on the
principles of convergence; fitting data to the model. Whether
this approach will lead to AGI is still a debatable subject.
However, for now, deep neural networks are the best
possible solution, and they use optimization methods to
arrive at the target.
• Fundamental optimisation methods are typically categorised
into first-order, high-order and derivative-free optimisation
methods. One usually comes across methods that fall into the
category of the first-order optimisation such as the gradient
descent and its variants.
 Gradient Descent
• The gradient descent method is the most popular
optimization method. The idea of this method is to update
the variables iteratively in the (opposite) direction of the
gradients of the objective function. With every update, this
method guides the model to find the target and gradually
converge to the optimal value of the objective function.
 Stochastic Gradient Descent
• Stochastic gradient descent (SGD) was proposed to address the
computational complexity involved in each iteration for large scale data.
The equation is given as:

Taking the values and adjusting them iteratively based on different

parameters in order to reduce the loss function is called back-propagation.
In this method, one sample randomly used to update the gradient(theta) per
iteration instead of directly calculating the exact value of the gradient. The
stochastic gradient is an unbiased estimate of the real gradient.
This optimisation method reduces the update time for dealing with large
numbers of samples and removes a certain amount of computational
redundancy.
 Adaptive Learning Rate Method
• Learning rate is one of the key hyper-parameters that
undergo optimization. Learning rate decides whether the
model will skip certain portions of the data. If the learning
rate is high, then the model might miss on subtler aspects of
the data. If it is low, then it is desirable for real-world
applications. Learning rate has a great influence on SGD.
Setting the right value of the learning rate can be challenging.
Adaptive methods were proposed to this tuning
automatically.
• The adaptive variants of SGD have been widely used in DNNs.
Methods like AdaDelta, RMSProp, Adam use the exponential
averaging to provide effective updates and simplify the
calculation.
1. Adagrad: weights with a high gradient will have low learning
rate and vice versa
2. RMSprop: adjusts the Adagrad method such that it reduces
its monotonically decreasing learning rate.
3. Adam is almost similar to RMSProp but with momentum
4. Alternating Direction Method of Multipliers (ADMM) is
another alternative to Stochastic Gradient Descent (SGD) .
• The difference between gradient descent and AdaGrad
methods is that the learning rate is no longer fixed. It is
computed using all the historical gradients accumulated up
to the latest iteration.
 Conjugate Gradient Method
• The conjugate gradient (CG) approach is used for solving large
scale linear systems of equations and nonlinear optimization
problems. The first-order methods have a slow convergence
speed. Whereas, the second-order methods are
resource-heavy. Conjugate gradient optimization is an
intermediate algorithm, which combines the advantages of
first-order information while ensuring the convergence
speeds of high-order methods.
 Network Perceptron
• A perceptron is a simple model of a biological neuron in
an artificial neural network. Perceptron is also the name of an
early algorithm for supervised learning of binary classifiers.
• The perceptron algorithm was designed to classify visual
inputs, categorizing subjects into one of two types and
separating groups with a line. Classification is an important
part of machine learning and image processing. Machine
learning algorithms find and classify patterns by many
different means. The perceptron algorithm classifies patterns
and groups by finding the linear separation between different
objects and patterns that are received through numeric or
visual input.
 • The perceptron consists of 4 parts.
1. Input values or One input layer
2. Weights and Bias
3. Net sum
4. Activation Function
• The Neural Networks work the same way as the perceptron. So, if you
want to know how neural network works, learn how perceptron works.
• But how does it work?
• The perceptron works on these simple steps
1 . All the inputs x are multiplied with their weights w. Let’s call
it k.

Fig: Multiplying inputs with weights for 5 inputs

2. Add all the multiplied values and call them Weighted Sum.

3. Apply that weighted sum to the correct Activation

Function.
For Example: Unit Step Activation Function.
 • Weights shows the strength of the particular node.
• A bias value allows you to shift the activation function curve
up or down.

• The activation functions are used to map the input

between the required values like (0, 1) or (-1, 1).
 Adaptive Linear Neuron (Adaline)
• Adaline which stands for Adaptive Linear Neuron, is a network
having a single linear unit. It was developed by Widrow and
Hoff in 1960. Some important points about Adaline are as
follows −
1. It uses bipolar activation function.
2. It uses delta rule for training to minimize the Mean-Squared
Error (MSE) between the actual output and the
desired/target output.
3. The weights and the bias are adjustable.
• Architecture
• The basic structure of Adaline is similar to perceptron having
an extra feedback loop with the help of which the actual
output is compared with the desired/target output. After
comparison on the basis of training algorithm, the weights
and bias will be updated.
 • Training Algorithm :
Step 1 − Initialize the following to start the training −
1. Weights
2. Bias
3. Learning rate α
• For easy calculation and simplicity, weights and bias must be set
equal to 0 and the learning rate must be set equal to 1.
Step 2 − Continue step 3-8 when the stopping condition is not true.
Step 3 − Continue step 4-6 for every bipolar training pair s:t.
Step 4 − Activate each input unit as follows −
xi = si (i=1 to n)
Step 5 − Obtain the net input with the following relation −

yin = b + ∑xiwi
Here ‘b’ is bias and ‘n’ is the total number of input neurons.
Step 6 − Apply the following activation function to obtain the final
output −
f(yin)= 1 if yin⩾0
−1 if yin<0
Step 7 − Adjust the weight and bias as follows −
Case 1 − if y ≠ t then,
wi(new)=wi(old)+α(t−yin)xi
b(new)=b(old)+α(t−yin)
Case 2 − if y = t then,
wi(new)=wi(old)
b(new)=b(old)
✔ Here ‘y’ is the actual output and ‘t’ is the desired/target output.
✔ (t−yin)(t−yin) is the computed error.
Step 8 − Test for the stopping condition, which will happen when
there is no change in weight or the highest weight change occurred
during training is smaller than the specified tolerance.
 Multi layer perceptron
• Multi layer perceptron (MLP) is a supplement of feed forward
neural network. It consists of three types of layers—the input
layer, output layer and hidden layer. The input layer receives
the input signal to be processed. The required task such as
prediction and classification is performed by the output layer.
An arbitrary number of hidden layers that are placed in
between the input and output layer are the true
computational engine of the MLP. Similar to a feed forward
network in a MLP the data flows in the forward direction from
input to output layer. The neurons in the MLP are trained
with the back propagation learning algorithm. MLPs are
designed to approximate any continuous function and can
solve problems which are not linearly separable. The major
use cases of MLP are pattern classification, recognition,
prediction and approximation.
The computations taking place at every neuron in the output
and hidden layer are as follows,
• with bias vectors b(1), b(2); weight matrices W(1), W(2) and
activation functions G and s. The set of parameters to learn is
the set θ = {W(1), b(1), W(2), b(2)}. Typical choices for s
include tanh function with tanh(a) = (ea − e− a)/(ea + e− a) or the
logistic sigmoid function, with sigmoid(a) = 1/(1 + e− a).
 Backpropogation Algorithms
• Backpropogation is the essence of neural network training. It
is the method of fine-tuning the weights of a neural network
based on the error rate obtained in the previous epoch (i.e.,
iteration). Proper tuning of the weights allows you to reduce
error rates and make the model reliable by increasing its
generalization.
• Backpropogation in neural network is a short form for
"backward propagation of errors." It is a standard method of
training artificial neural networks. This method helps calculate
the gradient of a loss function with respect to all the weights
in the network.
• How Backpropogation Algorithm Works :
• The Back propagation algorithm in neural network computes the
gradient of the loss function for a single weight by the chain rule. It
efficiently computes one layer at a time, unlike a native direct
computation. It computes the gradient, but it does not define how
the gradient is used. It generalizes the computation in the delta
rule.
• Consider the following Back propagation neural network example
diagram to understand:
1. Inputs X, arrive through the preconnected path
2. Input is modeled using real weights W. The weights are
usually randomly selected.
3. Calculate the output for every neuron from the input layer,
to the hidden layers, to the output layer.
4. Calculate the error in the outputs : ErrorB= Actual Output –
Desired Output
5. Travel back from the output layer to the hidden layer to
adjust the weights such that the error is decreased.
• Keep repeating the process until the desired output is
achieved
 • Why We Need Backpropogation?
Most prominent advantages of Backpropagation are:
1. Backpropagation is fast, simple and easy to program
2. It has no parameters to tune apart from the numbers of
input
3. It is a flexible method as it does not require prior knowledge
about the network
4. It is a standard method that generally works well
5. It does not need any special mention of the features of the
function to be learned.
 Tuning the Network Size
• Hyper parameters ( Tuning ) related to Network structure :
1. Number of Hidden Layers and units :
• Hidden layers are the layers between input layer and output layer.
• “Very simple. Just keep adding layers until the test error does not
improve anymore.”
• Many hidden units within a layer with regularization techniques
can increase accuracy. Smaller number of units may cause under
fitting.
2. Dropout :
• Dropout is regularization technique to avoid overfitting (increase
the validation accuracy) thus increasing the generalizing power.
• Generally, use a small dropout value of 20%-50% of neurons with
20% providing a good starting point. A probability too low has
minimal effect and a value too high results in under-learning by the
network.
• Use a larger network. You are likely to get better performance
when dropout is used on a larger network, giving the model
more of an opportunity to learn independent representations.
3. Network Weight Initialization :
• Ideally, it may be better to use different weight initialization
schemes according to the activation function used on each
layer.
• Mostly uniform distribution is used.
4. Activation function :
• Activation functions are used to introduce nonlinearity to
models, which allows deep learning models to learn nonlinear
prediction boundaries.
• Generally, the rectifier activation function is the most
popular.
• Sigmoid is used in the output layer while making binary
predictions. Softmax is used in the output layer while
making multi-class predictions.
5. Learning Rate :
• The learning rate defines how quickly a network updates its
parameters.
• Low learning rate slows down the learning process but converges
smoothly. Larger learning rate speeds up the learning but may not
converge.
• Usually a decaying Learning rate is preferred.
6. Momentum :
Momentum helps to know the direction of the next step with the
knowledge of the previous steps. It helps to prevent oscillations. A
typical choice of momentum is between 0.5 to 0.9.
7. Number of epochs :
• Number of epochs is the number of times the whole training data
is shown to the network while training.
• Increase the number of epochs until the validation accuracy starts
decreasing even when training accuracy is increasing(overfitting).
8. Batch size :
• Mini batch size is the number of sub samples given to the
network after which parameter update happens.
• A good default for batch size might be 32. Also try 32, 64,
128, 256, and so on.