A Brief Overview of Neural
Networks
By
Rohit Dua, Samuel A. Mulder, Steve E.
Watkins, and Donald C. Wunsch
Overview
• Relation to Biological Brain: Biological Neural Network
• The Artificial Neuron
• Types of Networks and Learning Techniques
• Supervised Learning & Backpropagation Training
Algorithm
• Learning by Example
• Applications
• Questions
Biological Neuron
Artificial Neuron
W
Neuron
I W
N
P Σ f(n) Outputs
U
T
W Activation
S
W Function
W=Weight
Transfer Functions
1
Output
SIGMOID : f (n) = 1
−n
1 +e
0
Input
LINEAR : f (n) = n
Types of networks
Multiple Inputs and Multiple Inputs
Single Layer and layers
Types of Networks – Contd.
Feedback
Recurrent Networks
Learning Techniques
• Supervised Learning:
Expected
Inputs from the Output
environment Actual System
Actual +
Output
Neural Network Σ
-
Training
Error
Multilayer Perceptron
Output
Inputs First Hidden Second Layer
layer Hidden Layer
Signal Flow
Backpropagation of Errors
Function Signals
Error Signals
Learning by Example
• Hidden layer transfer function: Sigmoid function
= F(n)= 1/(1+exp(-n)), where n is the net input to
the neuron.
Derivative= F’(n) = (output of the neuron)(1-
output of the neuron) : Slope of the transfer
function.
• Output layer transfer function: Linear function=
F(n)=n; Output=Input to the neuron
Derivative= F’(n)= 1
Learning by Example
• Training Algorithm: backpropagation of
errors using gradient descent training.
• Colors:
– Red: Current weights
– Orange: Updated weights
– Black boxes: Inputs and outputs to a neuron
– Blue: Sensitivities at each layer
First Pass
G1= (0.6225)(1- G2= (0.6508)(1-
0.6225)(0.0397)(0.5)(2)=0.0093 0.6508)(0.3492)(0.5)=0.0397
0.6225 0.6225 0.6508
0.5 0.5
0.5 0.6508
0.5
1 0.5 0.5
0.5 0.5 0.6508
0.5 0.6225 0.5 0.6225 0.6508
G3=(1)(0.3492)=0.3492
Gradient of the neuron= G Gradient of the output
=slope of the transfer neuron = slope of the
function×[Σ{(weight of the transfer function × error
neuron to the next neuron) ×
(output of the neuron)}] Error=1-0.6508=0.3492
Weight Update 1
New Weight=Old Weight + {(learning rate)(gradient)(prior output)}
0.5+(0.5)(0.0397)(0.6225) 0.5+(0.5)(0.3492)(0.6508)
0.5+(0.5)(0.0093)(1)
0.5124
0.5047 0.6136
0.5124 0.5124
0.5047 0.6136
0.5124
Second Pass
G1= (0.6236)(1- G2= (0.6545)(1-
0.6236)(0.5124)(0.0273)(2)=0.0066 0.6545)(0.1967)(0.6136)=0.0273
0.6236 0.6391
0.6545
0.5047 0.5124
0.6136 0.8033
0.5047
1 0.5124 0.5124
0.5047 0.6136 0.8033
0.5047 0.5124
0.6236 0.6391 0.6545
G3=(1)(0.1967)=0.1967
Error=1-0.8033=0.1967
Weight Update 2
New Weight=Old Weight + {(learning rate)(gradient)(prior output)}
0.5124+(0.5)(0.0273)(0.6236) 0.6136+(0.5)(0.1967)(0.6545)
0.5047+(0.5)(0.0066)(1)
0.5209
0.508 0.6779
0.5209 0.5209
0.508 0.6779
0.5209
Third Pass
0.6243 0.6504
0.508 0.6571
0.5209
0.6779 0.8909
0.508
1 0.5209 0.5209
0.508 0.6779
0.508 0.5209 0.6571 0.8909
0.6243 0.6504
Weight Update Summary
Weights Output Expected Output
Error
w1 w2 w3
Initial conditions 0.5 0.5 0.5 0.6508 1 0.3492
Pass 1 Update 0.5047 0.5124 0.6136 0.8033 1 0.1967
Pass 2 Update 0.508 0.5209 0.6779 0.8909 1 0.1091
W1: Weights from the input to the input layer
W2: Weights from the input layer to the hidden layer
W3: Weights from the hidden layer to the output layer
Training Algorithm
• The process of feedforward and
backpropagation continues until the
required mean squared error has been
reached.
• Typical mse: 1e-5
• Other complicated backpropagation
training algorithms also available.
Why Gradient?
O1 Output 1
W1
O3 = 1/[1+exp(-N)]
N= Error =
W2
O2 (O1×W1) Actual
+(O2×W Output – O3
O = Output of the neuron 0
2) Input
W = Weight
N = Net input to the neuron
• To reduce error: Change in weights:
o Learning rate
o Rate of change of error w.r.t rate of change of weight
▪ Gradient: rate of change of error w.r.t rate of change of ‘N’
▪ Prior output (O1 and O2)
Gradient in Detail
• Gradient : Rate of change of error w.r.t rate of change in net input to
neuron
o For output neurons
▪ Slope of the transfer function × error
o For hidden neurons : A bit complicated ! : error fed back in terms of
gradient of successive neurons
▪ Slope of the transfer function × [Σ (gradient of next neuron ×
weight connecting the neuron to the next neuron)]
▪ Why summation? Share the responsibility!!
o Therefore: Credit Assignment Problem
An Example
G1=0.66×(1-0.66)×(0.34)= 0.0763
1 0.731 0.6645 1
0.5 0.66
Error = 1-0.66 = 0.34
0.5 0.5
0.4
0.5 0.6645 0.66 0
0.598
Error = 0-0.66 = -0.66
Reduce more G1=0.66×(1-0.66)×(-0.66)= -0.148
Increase less
Improving performance
• Changing the number of layers and
number of neurons in each layer.
• Variation in Transfer functions.
• Changing the learning rate.
• Training for longer times.
• Type of pre-processing and post-
processing.
Applications
• Used in complex function approximations,
feature extraction & classification, and
optimization & control problems
• Applicability in all areas of science and
technology.