Module 3

Neural Networks

Compiled By:
Dr. Radhika Kotecha
Professor and Head,
Department of Information Technology,
K J Somaiya Institute of Technology
Neural Networks
What is a Neural Network?
•Biologically motivated approach to
machine learning / data mining
Similarity with biological network
Fundamental processing elements of a neural
network is a neuron
[Link] inputs from other source
[Link] them in someway
[Link] a generally nonlinear operation on the
result [Link] the final result

Similarity with Biological Network

• Fundamental processing element of a neural network is a neuron • A human
brain has 100 billion neurons
• An ant brain has 250,000 neurons
Neural Network

● Neural Network is a set of connected

INPUT/OUTPUT UNITS, where each
 connection has a WEIGHT associated with it.

● NeuralNetwork learning is also called

CONNECTIONIST learning due to the connections
between units.

● Itis by default a case of SUPERVISED,

INDUCTIVE or CLASSIFICATION
learning.
Sample
Data
Input to
Neural
Network
Neural
Network Learning
● NeuralNetwork learns by adjusting the
weights so as to be able to correctly classify
 the training data and hence, after testing
phase, to classify unknown data.

● Neural Network needs long time for training.

● Neural Network has a high tolerance to noisy

and incomplete data
Neural Network Topology Designing
● Input: Classification data :: It contains classification attribute ● Data
is divided, as in any classification problem :: [Training data and Testing
data]
● All data must be normalized.
(i.e. all values of attributes in the database are changed to contain values in
the internal [0,1] or[-1,1])

● There are no clear rules as to “best” number of hidden layer units. ●

Network design is a trial-and-error process and may affect the accuracy of the
resulting trained network.
● The initial values of the weights may also affect the resulting accuracy. ●
Once a network has been trained and its accuracy is not considered
acceptable, it is common to repeat the training process with a different
network topology or a different set of initial weights.
Perceptron -
One Neuron as a
Network
● Here x1 and x2 are normalized attribute value of data.

●y is the output of the neuron , i.e the class label.

● x1 and x2 values multiplied by weight values w1 and w2 are input to the neuron x.

● Value of x1 is multiplied by a weight w1 and values of x2 is multiplied by a weight w2.

● Given that

• w1 = 0.5 and w2 = 0.5

 • Say value of x1 is 0.3 and value of x2 is 0.8,
• So, weighted sum is :
• sum= w1 x x1 + w2 x x2 = 0.5 x 0.3 + 0.5 x 0.8 = 0.55
One Neuron as a Network

• The neuron receives the weighted sum as input and

calculates the output as a function of input as follows :

• y = f(x) , where f(x) is defined as

• f(x) = 0 { when x< 0.5 }
• f(x) = 1 { when x >= 0.5 }
• For our example, x ( weighted sum ) is 0.55, so y = 1 ,
• That means corresponding input attribute values are
classified in class 1.
 • If for another input values , x = 0.45 , then f(x) =
0, • So we could conclude that input values are
classified to class 0.

Bias as extra input

x2 function v 
Input
w0 W1
∑ ϕ(−)
Attribute values Summing
x0 = +1 x1 w2 function
Activation Output class y

∑
weights
xm v wjx
wm m
=
 j =
0 j

A Multilayer Feed-Forward
Neural Network

A Multilayer Feed-Forward
Neural Network: Example
A Multilayer Feed-Forward
Neural Network: Example
 Diabetes Prediction: Create your own Neural Network

A Multilayer Feed-Forward
Neural Network: Example
Neural Network Learning

● The inputs are fed simultaneously into the

 input layer.

● The weighted outputs of these units are fed

into hidden layer.

● The weighted outputs of the last hidden layer

are inputs to units making up the output
layer.
A Multilayered Feed – Forward Network
● INPUT:records without class attribute with
normalized attributes values.

● INPUTVECTOR: X = { x1, x2, …. xn}

where n is the number of (non class) attributes.
 ● INPUT LAYER – there are as many nodes as non
class attributes

● HIDDEN LAYER – the number of nodes in the

hidden layer and the number of hidden layers
depends on implementation.
A Multilayered Feed–Forward
Network

● OUTPUT LAYER – corresponds to the

class attribute.
● Generally, there are as many nodes as
classes (values of the class attribute).
 Okk= 1, 2,.. #classes
• Network is fully connected, i.e. each unit provides
input to each unit in the next forward layer.
Classification by Back propagation

● BackPropagation learns by iteratively

processing a set of training data (samples).

● Foreach sample, weights are modified

to minimize the error between network’s
 classification and actual classification.
Steps in Back propagation
Algorithm
● STEP ONE: initialize the weights and biases.

● Theweights in the network are initialized to

random numbers from the interval [-1,1].

● Each unit has a BIAS associated with it

● Thebiases are similarly initialized to random

numbers from the interval [-1,1].

● STEP TWO: feed the training sample.

Steps in Back propagation Algorithm ( cont..)

● STEP THREE: Propagate the inputs forward;

we compute the net input and output of each
unit in the hidden and output layers.

● STEP FOUR: back propagate the error.

● STEP FIVE: update weights and biases to

reflect the propagated errors.

● STEP SIX: verify terminating conditions.

A Multilayer Feed-Forward
Neural Network
Propagate the inputs forward
● For each input unit j in the input layer,
its output is equal to its input, that is,
O=I
jj

The net input to each unit in the hidden and output layers is
computed as follows:
•Given each unit j in a hidden or output layer, the net input
is = ∑ +
Iθ
j wijOi j
i

where wij is the weight of the connection from unit i in the previous layer
to unit j; Oiis the output of unit i from the previous layer; and; θis the bias
 of the unit
j

Propagate the inputs forward

● Each unit in the hidden and output layers takes its

net input and then applies an activation function.
The function symbolizes the activation of the
neuron represented by the unit. It is also called a
logistic, sigmoid, or squashing function.
● Given a net input Ijto unit j, then
Oj = f(Ij),
=
11
the output of unit j, is computed as j

O−
jI
 + e
Back propagate the error
● When reaching the Output layer, the error is
computed and propagated backwards. ● For a
unit k in the output layer the error is computed
by a formula:
•
(1 )( ) Errk= Ok−OkTk−Ok
where Ok – actual output of unit k ( computed by activation function.
=
11
O−
kI
+ e k

Tk – True output based of known class label;

 Ok(1-Ok) – is a Derivative ( rate of change ) of activation function.
Back propagate the error

● The error is propagated backwards by updating

weights and biases to reflect the error of the
network classification .
● For a unit j in the hidden layer the error is
computed by a formula:
Errj= rrkw k
)
Oj(1−Oj ∑E jk

where wjk is the weight of the connection from unit j to unit k

in the next higher layer, and Errkis the error of unit k.
 Update weights and biases

● Weights are updated by the following equations,

where l is a constant between 0.0 and 1.0
reflecting the learning rate:
Δw = (l)
ij ErrjOi

wij= wij +Δwij

• Biases are updated by the following equations

Δθ = (l)
jErrj

θ j=θ j+ Δθ j
Update weights and biases
● We are updating weights and biases after the
presentation of each sample.
● This is called case updating.

● Epoch --- One iteration through the training set is called an epoch.

● Epoch updating: Alternatively, the weight and bias

increments could be accumulated in variables and the
weights and biases updated after all of the samples of
the training set have been presented.

● Case updating is more accurate

Terminating Conditions ●
 Training stops
• All in the previous epoch are below some
Δwij d, or
threshol
•The percentage of samples misclassified in the previous
epoch is below some threshold, or

• a pre specified number of epochs has expired.

• In practice, several hundreds of thousands of epochs may

be required before the weights will converge.
Summary: Backpropagation FormulasOutput

vector
Output nodes
(1 )( k k ) Errk= Ok−OkT −O
Errj = jk
= )
11 Oj(1−Oj ∑Errkw k
O −jI
j
+ e
= j wijOi j i j j Errj θ =θ + (l)
Hidden nodes ∑+

Iθ wij
ij ij ErrjOi w = w + (l)
Input nodes

Input vector: xi
Example of Back propagation Input =
3, Hidden
 Neuron = 2 Output
=1

Initialize weights
:

Random Numbers
from -1.0 to 1.0

Initial Input and

weight

x1 x2 x3 w14 w15 w24 w25 w34 w35 w46 w56 1 0 1 0.2 -0.3 0.4 0.1 -0.5

0.2 -0.3 -0.2

Example ( cont.. )
● Bias added to Hidden
● + Output nodes

● Initialize Bias

● Random Values from

● -1.0 to 1.0

● Bias ( Random )

θ4 θ5 θ6

-0.4 0.2 0.1

x1 x2 x3 w14 w15 w24 w25 w34 w35 w46 w56 θ4 θ5 θ6 1 0 1 0.2 -0.3 0.4 0.1 -0.5 0.2 -0.3 -0.2 -0.4 0.2

0.1
 Unitj Net Input Ij Output Oj = ∑ +
Iθ
O −jI
j wijOi j
=
11
j
i + e
0 - j
4 O+
1

= 0.332 = 0.525
1 1
0.7
5 Oj e
=
−
=
e 0.1

1
 +

6 (-0.3)0.332-
1
(0.2)(0.525)+0.1= j
-0.105 O+
= e =
0.105
1 0.475

Calculation of Error at Each Node Unit j

Error j

6
(1 )( k k ) Errk= Ok−OkT −O
0.475(1-0.475)(1-0.475) =0.1311
(GIVEN) T 6 = 1
5 Errj = )
Oj(1−Oj ∑E
 rrkw k jk

0.525 x (1- 0.525)x 0.1311x (-0.2) = -0.0065

) rrkwk
4 Oj(1−Oj ∑E
Errj = jk

0.332 x (1-0.332) x 0.1311 x (-0.3) = -0.0087

Calculation of Weights and Bias Updating ij ij ErrjOi w
= w + (l)j j Errj θ =θ + (l)
Learning Rate l =0.9
w14 w15 w24 w25 w34 w35 w46 w56 θ4 θ5 θ6 0.2 -0.3 0.4 0.1 -0.5 0.2 -0.3 -0.2 -0.4

0.2 0.1

Weight New Values

w46 -0.3 + 0.9(0.1311)(0.332) = -0.261

w56 -0.2 + (0.9)(0.1311)(0.525) = -0.138
 w14 0.2 + 0.9(-0.0087)(1) = 0.192

w15 -0.3 + (0.9)(-0.0065)(1) = -0.306

……..similarly ……… similarly
θ6 0.1 +(0.9)(0.1311)=0.218
……..similarly ……… similarly

Applications

● HandwrittenDigit Recognition
● Face recognition

● Time series prediction

● Process identification

● Process control
● Optical character recognition
 References
1. S. Rajasekaran and G. A. Vijayalakshmi Pai, Neural
Networks, Fuzzy Logic and Genetic Algorithm:
Synthesis and Applications, PHI.
2. S. N. Sivanandam and S. N. Deepa, Principles of
Soft Computing, 2nd ed., Wiley India.
3. J. Zurada, Introduction to Artificial Neural Systems,
Jaico Publishing House.
Example Questions
● Explain the design of Neural Network topology.
● State the role of Bias and Learning rate in Neural Networks.

● Design a Neural Network for given application (E.g. Design a Neural Network for
diabetes prediction).
● Consider a Multilayer Feed-forward Neural Network with architecture 3 – 2 – 1
(no. of nodes in input layer, hidden layer, output layer). Let the learning rate be
0.9. The initial weight and bias values of the network can be assumed randomly.
 Consider the first training tuple, X = (1, 0, 1), whose class label is 1. Calculate
the net input, output and error of each unit in hidden and output layer once the
tuple is fed into the network. Also show updated values of weights and bias after
first iteration calculating the error.
● Differentiate Case Updating and Epoch Learning.

● Explain Backpropagation algorithm in detail. Argue why weights are modified in

Neural Networks.
● Explain terminating conditions for Neural Network.