Math Behind Artificial Neural

Networks

1
History of the Artificial Neural Networks
 history of the ANNs stems from the 1940s- the decade of the first electronic
computer.
 The first important step took place in 1957 when Rosenblatt introduced the
first concrete neural model, the perceptron. Rosenblatt also took part in
constructing the first successful neurocomputer, the Mark I Perceptron.

2
 Computers vs. Neural Networks
“Standard” Computers Neural Networks

 one CPU highly parallel processing

fast processing units slow processing units

reliable units unreliable units

static infrastructure dynamic infrastructure

3
 Computers vs. Neural Networks
“Standard” Computers Neural Networks

 one CPU highly parallel processing

fast processing units slow processing units

reliable units unreliable units

static infrastructure dynamic infrastructure

4
 Artificial Neural Networks

• The “building blocks” of neural networks are the

neurons.
• In technical systems, we also refer to them as units or
nodes.
• Basically, each neuron
 receives input from many other neurons.
 changes its internal state (activation) based on the
current input.
 sends one output signal to other neurons

5
 How do ANNs work?
 Either a hardware implementation or a computer program
 Simulates the information processing capabilities of its biological
exemplar
 Composed of a great number of interconnected artificial neurons
 Technique for solving problems by constructing software that
works like our brains.

6
 How do our brains work?
 The Brain is a massively parallel information processing
system
 Our brains are a huge network of processing elements
 A typical brain contains a network of 10 billion neurons.

7
How do our brains work?
A processing element

Dendrites: Input
Cell body: Processor
Synaptic: Link
Axon: Output
8
 How do our brains work?
A processing element

A neuron is connected to other neurons through about 10,000

synapses

9
Analogy between Artificial NN and
Biological NN
• A neuron receives input
from other neurons
• Once input exceeds a
critical level the neuron
discharges a spike
• Spike is an electrical
pulse that travels from
the body, down the
axon, to the next
neuron(s)
• The axon endings almost
touch the dendrites or
cell body of the next
neuron 10
Analogy between Artificial NN and
Biological NN
• Transmission of an electrical
signal from one neuron to the
next is effected by
neurotransmitters
• Neurotransmitters are chemicals
which are released from the first
neuron and which bind to the
Second
• This link is called a synapse
• The strength of the signal that
reaches the next neuron
depends on factors such as the
amount of neurotransmitter
available

11
Analogy between Artificial NN and
Biological NN

• Accept input
Dendrites  Input
• Process input
Soma  Node
• Electrochemical contact between
neurons
Synapse  Weight
• Turns processed input to output
Axon  output 12
 Artificial NN
• Attributes of neuron
– m binary inputs and one output (0 or 1)
– Synaptic weights wij
– Threshold I
• Output is ‘1’ if and only if weighted sum of inputs is greater
than threshold

13
 What is a Perceptron?
• Is a neural network without any hidden layer
• Has an input layer and an output layer only

14
 Where can perceptrons be used?
• Mostly used for simple decision making
• Give access if a person is employee and
deny access if a person is an outsider
• Provide entry for humans only
• Implementation of logic gates

15
Perceptron

16
Perceptron

17
Perceptron

18
Perceptron

19
Perceptron

20
Perceptron

21
Perceptron

22
Perceptron

23
Perceptron

24
 Perceptron

• Learning refers to the method of modifying weights of

connections
• Learning ability of a neural network is determined by its
25
architecture and by the algorithm chosen for training
Perceptron Bias, b

26
 Perceptron Bias, b

Output is dependent on Step Function i.e. it is either 0 or 1

depending on threshold

27
 Perceptron

Sigmoid function

28
 Perceptron

Sigmoid function

29
 Perceptron

30
 Perceptron

31
 Perceptron

32
 Perceptron

33
 Perceptron

34
 Perceptron

35
Example: AND function

X1 1
AND
Y
X1 X2 Y
1 1 1
X2 1
1 0 0
AND Function
0 1 0
0 0 0

Threshold(Y) = 2

36
Example: AND function
X1 1
AND
Y
X1 X2 Y
1 1 1
X2 1
1 0 0
AND Function
0 1 0
0 0 0

Threshold(Y) = 2

37
 Example: OR function
OR
X1 2
X1 X2 Y
Y
1 1 1
X2 2 1 0 1
AND Function
OR Function
0 1 1
0 0 0

Threshold(Y) = 2

38
 Example: OR function
OR
X1 2
X1 X2 Y
Y
1 1 1
X2 2
1 0 1
0 1 1
AND Function
OR Function
0 0 0

Threshold(Y) = 2

39
 Example: AND-NOT function
AND
X1 2 NOT
Y X1 X2 Y
X2
1 1 0
-1
1 0 1
AND NOT Function
0 1 0
0 0 0
Threshold(Y) = 2

40
 Example: AND-NOT function
AND
X1 2 NOT
Y X1 X2 Y
X2
1 1 0
-1
1 0 1
AND NOT Function
0 1 0
0 0 0
Threshold(Y) = 2

41
Limitations of Single-Layer Perceptron

• There are two major problems:

– Single-Layer Percpetrons cannot classify non-
linearly separable data points.
– Complex problems, that involve a lot of
parameters cannot be solved by Single-Layer
Perceptrons.

42
 Limitations of Single-Layer Perceptron

• Single-Layer Percpetrons cannot classify non-linearly separable

data points
Y=0
Y=1

X2

X1

43
 Limitations of Single-Layer Perceptron

• Single-Layer Percpetrons cannot classify non-linearly separable

data points
Y=0
Y=1

X2

X1

Cannot separate the high and low points with a single

straight line 44
 Limitations of Single-Layer Perceptron

• Complex problems, that involve a lot of parameters cannot be

solved by Single-Layer Perceptrons
• Ex: for E-commerce firm, you have noticed a decline in sales
• Form a marketing team who would market the products for
increasing the sales.
• The marketing team can market your product through various
ways, such as:
– Google Ads
– Personal emails
– Sale advertisement on relevant sites
– Reference program
– Blogs and so on . . .
45
 Limitations of Single-Layer Perceptron

• marketing team has to decide a strategy to do optimal and efficient

marketing
• but this task is too complex for a human to analyse, because number of
parameters is quite high.
• This problem will have to be solved using Deep Learning

46
Marketing strategy using Deep Learning

47
Multilayer Perceptron

48
Multilayer Perceptron

XOR
X1 X2 Y
1 1 0
1 0 1
0 1 1
0 0 0
σ(x) = 0, if x<0
σ(x) = 1 if x>0

49
Multilayer Perceptron

XOR
X1 X2 Y
1 1 0
1 0 1
0 1 1
0 0 0
σ(x) = 0, if x<0
σ(x) = 1 if x>0

50
Multilayer Perceptron

XOR
X1 X2 Y
1 1 0
1 0 1
0 1 1
0 0 0
σ(x) = 0, if x<0
σ(x) = 1 if x>0

51
What is Multi-Layer Perceptron?

52
 General Structure of (ANN)
• Three layers

inputs

53
 ANN Example
• Two inputs, two hidden neurons, two output
neurons
• Hidden and output neurons include bias

54
 ANN Example
• Given:
• Initial weights, the biases, inputs 0.05 and 0.10
• Activation function is sigmoid
• Actual output are 0.01 and 0.99.

55
 ANN Example
• z21 is first neuron of layer 2 (hidden layer)
• z21 = i1× w1 + i2× w2 + b1
= 0.05 × 0.15 + 0.10 × 0.20 + 0.35
= 0.3775
• h1 = 1/(1+e-z21) = 0.593269992

56
 ANN Example
• z22 = i1× w3 + i2× w4 + b1
= 0.05 × 0.25 + 0.10 × 0.30 + 0.35
=?
• h2 = 1/(1+e-z22) = 0.596884378

57
 ANN Example
• z31 = h1× w5 + h2× w6 + b2
= 0.593269992 × 0.40 + 0.596884378 × 0.45 + 0.60
= 1.105905967
• o1 = 1/(1+e-z31) = 0.75136507

58
 ANN Example
• z32 = h1× w7 + h2× w8 + b2
• o2 = 1/(1+e-z32) = 0.772928465

59
 ANN Example
• o1 = 0.75136507 and o2 = 0.772928465
• Actual outputs for the given inputs are 0.01 and 0.99
• Total error = (0.01- 0.75136507)2 + (0.99- 0.772928465)2
= 0.298371109
• Weights and biases should be changed to reduce error
• This is called training the network

60
 Artificial Neural Network
• Has at least three layers
• Has at least one hidden layer

61
Artificial Neural Network

62
 Real Life example
• Data set of animals with Fur Color and Weight
• Classify animals into two categories

63
 Real Life example
• Initialize network with weights

64
 Real Life example
• Assign inputs for known outputs

65
Real Life example

66
 Real Life example
• Output of each node is given to activation function

67
 Real Life example
• For node 1 at output layer , predicted value is 0.3512 and actual is 1
Error = 1- 0.3512
• For node 2 at output layer, predicted value is 0.7819 and actual is 0
• Error = 0- 0.7819

68
 Real Life example
• After a few iterations error at the output will reduce
• Apply next sample data to ANN and adjust weights and biases
• Calculate output values

69
 Activation Functions
• Binary Step Function
• threshold based classifier
• whether or not the neuron should be activated based on
the value from the linear transformation
• if the input to the activation function is greater than a
threshold, then the neuron is activated
• else it is deactivated, i.e. its output is not considered for the
next hidden layer
f(x) =0, x<o
= 1, x>1

70
 Activation Functions
x is input to the node

Sigmoid Tanh
A = 1/(1 + e-x)
Output is always positive. A= {2/(1 + e-x)} -1
Therefore input to next layer is always positive
71
 Activation Functions
• x is input to the node

Sigmoid Tanh
A = 1/(1 + e-x) A = {2/(1 + e-x)} -1
Output is positive/negative

72
 Activation Functions
• For the negative input values, the result is zero, that means the
neuron does not get activated
• Since only a certain number of neurons are activated, the ReLU
function is far more computationally efficient when compared to
the sigmoid and tanh function

Rectified Linear Unit(ReLU)

stands for A = max(0, x)
73
 Activation Functions
• Leaky ReLU
• Improved version of the ReLU function
• For ReLU, gradient is 0 for x<0, which would deactivate the
neurons in that region
• Leaky ReLU is defined to address this problem
• For negative values of x, output is an extremely small linear
component of x

74
 Real Life example
• Learning rate
• The amount of change in weights and biases
• Momentum
• The amount of effect of past weights and
biases on the current weights and biases

75
 Deep Neural Network

It is an ANN with more than one hidden layer

76
 Deep Neural Network
• Takes the data
• Train network to identify patterns
• Predict the output

77
The training phase of a neural network

78
 Steps involved in the implementation
of a neural network
A neural network executes in 2 steps:
1. Feedforward:
• Initially, specific weight required by inputs are not
known
• Have a set of input features and some random weights
• Weight decides how vital is that feature for prediction
• The higher the weight, the greater the importance
2. Backpropagation:
• Calculate the error between predicted output and target
output
• Use an algorithm (gradient descent) to update the
weight values

79
 Example: Deep Learning
• Data set of circle, square and a triangle is given

80
 Example: Deep Learning
• Predict shape of the image, square, circle or triangle?

81
 Example: Deep Learning
• Activation function decides a particular neuron will get activated or
not

82
 Example: Deep Learning
• Neuron with highest value (probability) determines the output
• This is called forward propagation

83
 Example: Deep Learning
• Network is yet to be trained
• Arrows show predicted value should be higher or lower than the
actual value

84
 Example: Deep Learning
• Error is transferred back to the network, called backpropagation
• Based on this information weights are adjusted
• After first iteration, error for circle has reduced from 0.6 to 0.4
• Similarly error for other outputs has reduced

85
 Example: Deep Learning
• Cycle of forward propagation and back propagation is iteratively
performed
• Till error is minimum

86
Backpropagation using Gradient Descent
• Handwritten alphabets are present as images of
28x28 pixels

87
Backpropagation using Gradient Descent

88
Backpropagation using Gradient Descent
• Predicted probability is compared with actual
probability and error is calculated

89
Backpropagation using Gradient Descent
• Magnitude indicated amount of change
• Sign indicates increase or decrease in the weights
• This information is transmitted back to the network
• This is called backpropagation

90
Backpropagation using Gradient Descent
• Weights in the network are adjusted in order
to reduce the loss in prediction

91
Backpropagation using Gradient Descent
• Keep training the network until it is able to
predict with high accuracy

92
Backpropagation using Gradient Descent
• We have different ‘a’
• Repeat training for ‘b’ and ‘c’

93
 Example: Training Neural Network
• After 2nd iteration, loss for
‘a’ = 0.16
‘b’ = 0.04
‘c’ = 0.01

• After 3rd iteration, loss for

‘a’ = 0.04
‘b’ = 0.01
‘c’ = 0.01
• Keep iterating for better accuracy
• Can use hundreds of letter, ‘a’
94
Example: Training Neural Network

95
 Example: Training Neural Network
• With gradient descent and backpropagation network is
completely trained
• Training process takes longer than the process required
for prediction

96
 Example: Training
• Single neuron with weight is ‘w’

desired output

97
 Example: Training
• Output = input × weight
• Network starts training itself by choosing random value of ‘w’

98
 Example: Training
• Loss = (actual value – predicted value)2
• Loss function is measurement of error in
predicted outputs

99
 Example: Training
• For practical neural networks, we process massive amount
of data
• May not get slope which is zero
• Therefore network is trained to get minimum slope instead
of zero
• Slope and magnitude of error is fed back to the network

100
 Training ANN with one weight
• Training algorithm increases or decreases weight to get
minimum error
• Choose weight which results in minimum error
• Adjustment of weight is back propagation

101
 Gradient Descent
• Cost = J(w)
= addition of errors for all samples at input

102
 Training ANN
• In real-life data, the situation can be a bit
more complex
• In real-life data, weight values are often
decimal (non-integer)
• Use a gradient descent algorithm with a low
learning rate
so that we can try different weight values
and obtain the best predictions from model
103
 What is Gradient Descent?
• Algorithm that operates iteratively to find the
optimal values for weights
• Requires user-defined learning rate, and initial
weight values
• Steps: (Iterative)
1. Start with initial values of weights
2. Calculate cost
3. Update values using learning rate and update
function if cost is not acceptable
4. Stop if cost is acceptable

104
 Gradient descent
• Common cost functions:
• Mean squared error
• Cross-entropy loss (log loss)
• Cost, J(θ) is dependent on weight, θ
• To determine minimum point determine derivative of J(θ)
with respect to θ

105
 Gradient descent
• Process of gradient descent is

where

• ∆θi is the change in weight of θi

• Set a learning rate, α, to control the size of
the change (step)
106
Learning Rate

107
Learning Rate

108
Learning Rate

109
 Gradient descent

• For one weight, determine derivative (gradient) for

only one weight
• For multiple weights, determine partial derivative
(gradient) with respect to each weight
• As you reach a local optima, the slope will approach
zero
• Once the slope of current parameter reaches zero,
parameter value stops updating
• It results in convergence and signifies that we can stop
the iterative process
110
 Gradient descent
• It is computationally expensive for a model
with large number of parameters (weights)
• Ex: if number of weights is 100 it may take
hours to determine optimum value of weight

111
Gradient Descent for ANN with one node
• Start with simplest neural network
• one neuron in input, hidden and output layers
• First use forward propagation
• Then Backward propagation

112
Gradient Descent for ANN with one node
• forward propagation

113
Gradient Descent for ANN with one node
• Forward propagation step is a series of functions
• Output of one layer/neuron is fed as input to the next layer

• Cost function is the error in prediction

114
Gradient Descent for ANN with one node
• Cost function, J(θ) is squared error

• Minimize the difference between the target

and actual output (output of neural network)

115
 Relating the weights to the cost
function
• In order to minimize the difference between
neural network's output and the target output,
• need to know how the cost function (error)
changes with respect to weights
• In other words, determine partial derivative of
cost function with respect to each weight

• Update these weights in an iterative process to

minimize the error
116
Gradient Descent for ANN with one node
• Output at NN is a(3) at third layer
• and target is y for the given input
• Cost function is J(θ)

117
Gradient Descent for ANN with one node
• Start from the output layer
• Apply the chain rule to determine partial
derivative for dependence of cost function on θ2

118
Gradient Descent for ANN with one node

• apply the chain rule to determine partial

derivative for dependence of cost function on θ1

119
Gradient Descent for ANN with two nodes
• Neural network with two neurons in input
layer, one hidden layer, output layer
• Disregard the bias for hidden layer

120
Gradient Descent for ANN with two nodes
• First, compute the output of a neural network
via forward propagation

121
Gradient Descent for NN with two nodes

First two layers

122
Gradient Descent for NN with two nodes
Last two layers

123
Gradient Descent for NN with two nodes

124
Gradient Descent for NN with two nodes

• Next, compute cost function, J(θ), for the last

layer in the network
• Cost function consist of all of the partial
derivatives that will be used in calculating new
weights

125
Gradient Descent for NN with two nodes
(3)

• ‘m’ is the number of examples in training

dataset
• Add cost function for all ‘m’ number of
samples
• Normalize by 1/m

126
Gradient Descent for NN with two nodes
• Layer 2 weights

127
Gradient Descent for NN with two nodes
• Layer 2 Parameters

128
Gradient Descent for NN with two nodes
• Layer 2 Parameters

129
Gradient Descent for NN with two nodes
• Layer 2 Parameters

130
Gradient Descent for NN with two nodes
• Layer 2 Parameters

131
Gradient Descent for NN with two nodes
• Layer 2 Parameters

132
Gradient Descent for NN with two nodes
• Layer 1 weights
The derivative chain for the blue dash path is:

133
Gradient Descent for NN with two nodes
• Layer 1 weights
The derivative chain for the red dash path is

134
Gradient Descent for NN with two nodes
Layer 1 weights
• Complete derivative chain (blue and red dash) is:

135
Gradient Descent for NN with two nodes
Layer 1 weights

• Partial derivatives describe how changing each

weight affects the cost function
• Use this knowledge to change all of the
weights parameter values
• that continues to decrease the cost function
till it converges to minimum value

136
Gradient Descent for NN with two nodes
• Layer 1 weights (all)

137
Gradient Descent for NN with two nodes

• Neural network with 2 neurons requires

computation of 8 partial derivatives
• Neural network with 8 neurons requires 52
partial derivative terms
• Generally neural networks that are used in
practice use many weights
• Mathematical complexity increases with the
number of weights
138
 Putting it all together
• Process of gradient descent for computation of
new weight is

where

• ∆θi is incremental change in weight

• Learning rate, α, controls the size of step
(change)
139
 Training ANN using gradient Descent

• Use gradient descent to update each of the

weights
• Recompute cost function with the new weights
• Repeat cost function converges to minimum
value
• During each iteration we perform forward
propagation to compute the outputs and
backward propagation to compute the errors
• One complete iteration is known as an epoch
• Keep checking cost function after each epoch to
watch the amount of error as network is trained 140
 Example: Backpropagation
Target/actual values
𝑥1 = 0.05
𝑡1 = 0.01
𝑥2 = 0.10
𝑏1

𝑡2 = 0.99

𝑏2

(2) (1) (1)

𝑧2 = 𝑥1 𝜃21 + 𝑥2 𝜃22
• Sigmoid is used as an activation function at each node of hidden
and output layer
(2) 1
=
1 𝑎2 = (2)
−𝑧2
1 + 𝑒− 1+𝑒
141
 Example: Backpropagation
𝑥1 = 0.05
𝑥2 = 0.10
𝑦1
Randomly initialize
weights and biases
(2) 𝑏1
𝑏1 = 0.35
(2) 𝑦2
𝑏2 = 0.6
(1) (2)
𝜃11 = 0.15 𝜃11 = 0.40 𝑏2
Forward Propagation
(1) (2)
𝜃12 = 0.20 𝜃12 = 0.45 (2) (1) (1)
𝑧1 = 𝑥1 𝜃11 + 𝑥2 𝜃12 + 𝑏12
(1) (2)
𝜃21 = 0.25 𝜃21 = 0.50 (2)
𝑧1 = (0.05 × 0.15) + (0.10 × 0.20) + 0.35 = 0.3775
(1) (2)
𝜃22 = 0.55 (2) 1 (2) 1
𝜃22 = 0.30 𝑎1 = 𝑎1 =
(2)
−𝑧1 1 + 𝑒 −0.3775 = 0.5933
1+𝑒
142
 Example: Backpropagation
𝑥1 = 0.05
𝑥2 = 0.10
𝑦1
Randomly initialize
weights and biases
(2) 𝑏1
𝑏1 = 0.35
(2) 𝑦2
𝑏2 = 0.6
(1) (2)
𝜃11 = 0.15 𝜃11 = 0.40 𝑏2
Forward Propagation
(1) (2)
𝜃12 = 0.20 𝜃12 = 0.45
(2) (1) (1)
(1) (2)
𝑧2 = 𝑥1 𝜃21 + 𝑥2 𝜃22 + 𝑏2
𝜃21 = 0.25 𝜃21 = 0.50
(1) (2) (2) 1 (2)
𝜃22 = 0.30 𝜃22 = 0.55 𝑎2 = (2) 𝑎2 = 0.5969
−𝑧2
1+𝑒
143
 Example: Backpropagation
𝑥1 = 0.05
𝑥2 = 0.10

Randomly initialize (3)

𝑦1 = 𝑎1
weights and biases
(2) 𝑏1
𝑏1 = 0.35
(2)
𝑏2 = 0.6
(3)
(1)
𝜃11 = 0.15
(2)
𝜃11 = 0.40 𝑏2 𝑦2 = 𝑎2
Forward Propagation
(1) (2)
𝜃12 = 0.20 𝜃12 = 0.45 (3) (2) (2) (2) (2)
(1) (2)
𝑧1 = 𝑎1 𝜃11 + 𝑎2 𝜃12 + 𝑏2
𝜃21 = 0.25 𝜃21 = 0.50 = (0.5933 × 0.40) + (0.5969 × 0.45) + 0.6
(1)
𝜃22 = 0.30
(2)
𝜃22 = 0.55 = 1.105
(3) 1 1
(3) = −1.105 = 0.7514
𝑎1 =
1+𝑒 −𝑧1 1+𝑒

144
 Example: Backpropagation
𝑥1 = 0.05
𝑥2 = 0.10

Randomly initialize (3)

weights and biases 𝑦1 = 𝑎1
(2) 𝑏1
𝑏1 = 0.35
(2)
𝑏2 = 0.6
(3)
(1) (2) 𝑦2 = 𝑎2
𝜃11 = 0.15 𝜃11 = 0.40 𝑏2
Forward Propagation
(1) (2)
𝜃12 = 0.20 𝜃12 = 0.45
(3) (2) (2) (2) (2)
(1)
𝜃21 = 0.25 (2)
𝜃21 = 0.50 𝑧2 = 𝑎1 𝜃21 + 𝑎2 𝜃22 + 𝑏2
(1)
𝜃22 = 0.30
(2)
𝜃22 = 0.55 (3) 1
𝑎2 = (3) = 0.7729
−𝑧2
1+𝑒
145
 Example: Backpropagation

1
𝑡𝑜𝑡𝑎𝑙 𝑒𝑟𝑟𝑜𝑟 = 𝐽(𝜃) = ෍ 𝑡𝑎𝑟𝑔𝑒𝑡 − 𝑜𝑢𝑡𝑝𝑢𝑡 2
2
1 1
= (𝑡1 − 𝑦1 )2 + (𝑡2 − 𝑦2 )2
2 2
1 1
= (0.01 − 0.75)2 + (0.99 − 0.77)2
2 2
=0.2748 + 0.0235 = 0.2983

146
 Example: Backpropagation
Recalculate weights using backpropagation
(2) 𝜕𝐽(𝜃)
𝑒𝑟𝑟𝑜𝑟 𝑑𝑢𝑒 𝑡𝑜 𝜃11 = (2)
𝜕𝜃11
𝑦1
1 1
𝐽(𝜃) = (𝑡1 − 𝑦1 )2 + (𝑡2 − 𝑦2 )2
2 2 𝑏1
(3)
𝑦1 = 𝑎1
𝑦2
• 𝐽(𝜃) does not have 𝑏2
weight term
• Use chain rule for
differentiation
(3)
𝜕𝐽(𝜃) 𝜕𝐽(𝜃) 𝜕𝑦1 𝜕𝑧1
(2)
=
𝜕𝜃11 𝜕𝑦1 𝜕𝑧 (3) 𝜕𝜃 (2)
1 11
147
 Example: Backpropagation
(3)
𝜕𝐽(𝜃) 𝜕𝐽(𝜃) 𝜕𝑦1 𝜕𝑧1 (3) (3) (3)
= 𝑦1 = 𝑎1 𝑎1 = sigmoid(𝑧1 )
(2) 𝜕𝑦1 𝜕𝑧 (3) 𝜕𝜃 (2)
𝜕𝜃11 1 11
𝜕𝑦1
• determine 𝜕𝐽(𝜃)
• determine 𝜕𝑧1
(3)
𝜕𝑦1 (3)
𝜕𝑦1 𝜕𝑎1
1 1 (3) = (3)
𝐽(𝜃) = (𝑡1 − 𝑦1 ) + (𝑡2 − 𝑦2 )2
2 𝜕𝑧1 𝜕𝑧1
2 2 1
(3)
𝑎1 = (3) 𝑖𝑠 𝑎 𝑠𝑖𝑔𝑚𝑜𝑖𝑑 𝑓𝑢𝑛𝑐𝑡𝑖𝑜𝑛
𝜕𝐽(𝜃) 1 −𝑧1
= 2 × 𝑡1 − 𝑦1 −1 + 0 1+𝑒
𝜕𝑦1 2
𝜕𝑠𝑖𝑔(𝑥)
given = 𝑠𝑖𝑔(𝑥) {1 - 𝑠𝑖𝑔(𝑥)}
𝜕𝑥
= − 𝑡1 − 𝑦1 (3)
𝜕𝑎1 (3) (3)
= − 0.01 − 0.75 = 0.7413 (3) = 𝑎1 (1 - 𝑎1 )
𝜕𝑧1

= 0.75(1- 0.75)
= 0.1868 148
 Example: Backpropagation
(3)
𝜕𝐽(𝜃) 𝜕𝐽(𝜃) 𝜕𝑦1 𝜕𝑧1 𝜕𝐽(𝜃) 𝜕𝑦1
(2)
= = 0.7413 = 0.1868
𝜕𝜃11 𝜕𝑦1 𝜕𝑧 (3) 𝜕𝜃 (2) 𝜕𝑦1 (3)
𝜕𝑧1
1 11

(3) (2) (2) (2) (2)

𝑧1 = 𝑎1 𝜃11 + 𝑎2 𝜃12 + 𝑏2
(3)
𝜕𝑧1 (2)
(2)
= 𝑎1
𝜕𝜃11
= 0.5933

(3)
𝜕𝐽(𝜃) 𝜕𝐽(𝜃) 𝜕𝑦1 𝜕𝑧1
(2)
=
𝜕𝜃11 𝜕𝑦1 𝜕𝑧 (3) 𝜕𝜃 (2)
1 11

= 0.7413 × 0.1868 ×0.5933

= 0.0821 149
 Example: Backpropagation
𝜕𝐽(𝜃)
(2)
= 0.0821
𝜕𝜃11

(𝟐)
update 𝜽𝟏𝟏
𝑦1

(2) (2) 𝜕𝐽(𝜃) 𝑏1

𝜃11 = 𝜃11 −𝛼 (2)
𝜕𝜃11
𝑦2
• α is learning rate for weight
• Range is 0 to 1 𝑏2
• Assume it to be 0.5 In the same way other
weights are calculated
(2)
𝜃11 = 0.4 – 0.5(0.0821) (2)
𝜃12 = 0.4086
= 0.3589 (2)
𝜃21 = 0.5113
(2)
𝜃22 = 0.0613
150
 Example: Backpropagation
Recalculate weights for input to hidden layer
(1) 𝜕𝐽(𝜃)
𝑒𝑟𝑟𝑜𝑟 𝑑𝑢𝑒 𝑡𝑜 𝜃11 = (1)
𝜕𝜃11
𝑦1
1 1
𝐽(𝜃) = (𝑡1 − 𝑦1 )2 + (𝑡2 − 𝑦2 )2
2 2 𝑏1

(2) (2) 𝑦2
𝜕𝐽(𝜃) 𝜕𝐽(𝜃) 𝜕𝑎1 𝜕𝑧1
(1)
= (2) (2) (1)
𝜕𝜃11 𝜕𝑎1 𝜕𝑧1 𝜕𝜃11
𝑏2
𝜕𝐽(𝜃) 𝜕𝐽1 (𝜃) 𝜕𝐽2 (𝜃) (3) (2) (2) (2) (2) (3)
(2)
= (2)
+ (2)
𝑠𝑖𝑛𝑐𝑒 𝑧1 = 𝜃11 𝑎1 +𝜃12 𝑎2 𝜕𝐽1 (𝜃) 𝜕𝐽1 (𝜃) 𝜕𝑎1
𝜕𝑎1 𝜕𝑎1 𝜕𝑎2 =
(3) (3) (3) (3)
(3) 𝜕𝑧1 (2) 𝜕𝑧1 𝜕𝑎1 𝜕𝑧1
𝜕𝐽1 (𝜃) 𝜕𝐽1 (𝜃) 𝜕𝑧1 (2)
= 𝜃11
= 𝜕𝑎1
(2)
𝜕𝑎1 𝜕𝑧1
(3)
𝜕𝑎1
(2) = 0.7413 ×0.1868
= 0.4 = 0.1384
151
 Example: Backpropagation
Recalculate weights of hidden layer
(1) 𝜕𝐽(𝜃)
𝑒𝑟𝑟𝑜𝑟 𝑑𝑢𝑒 𝑡𝑜 𝜃11 = (1)
𝜕𝜃11
𝑦1
1 1
𝐽(𝜃) = (𝑡1 − 𝑦1 )2 + (𝑡2 − 𝑦2 )2
2 2 𝑏1

(2) (2) 𝑦2
𝜕𝐽(𝜃) 𝜕𝐽(𝜃) 𝜕𝑎1 𝜕𝑧1
(1)
= (2) (2) (1)
𝜕𝜃11 𝜕𝑎1 𝜕𝑧1 𝜕𝜃11
𝑏2
𝜕𝐽1 (𝜃)
𝜕𝐽(𝜃) 𝜕𝐽1 (𝜃) 𝜕𝐽2 (𝜃) (3)
= 0.1384
(2)
= (2)
+ (2) 𝜕𝑧1
𝜕𝑎1 𝜕𝑎1 𝜕𝑎1
(3) 𝜕𝐽1 (𝜃) 𝜕𝐽1 (𝜃) (2)
𝜕𝐽1 (𝜃) 𝜕𝐽1 (𝜃) 𝜕𝑧1 (2)
= (3)
𝜃11
(2)
= (3) (2) 𝜕𝑎1 𝜕𝑧1
𝜕𝑎1 𝜕𝑧1 𝜕𝑎1
= 0.1384 × 0.4
= 0.055 152
 Example: Backpropagation

𝜕𝐽1 (𝜃) 𝜕𝐽2 (𝜃) (2) (2)

= - 0.0190 𝜕𝐽(𝜃) 𝜕𝐽(𝜃) 𝜕𝑎1 𝜕𝑧1
(2)
= 0.055 (2) =
𝜕𝑎2 (1) (2) (2) (1)
𝜕𝑎1 𝜕𝜃11 𝜕𝑎1 𝜕𝑧1 𝜕𝜃11

𝜕𝐽(𝜃) 𝜕𝐽1 (𝜃) 𝜕𝐽2 (𝜃) (2)

𝜕𝑎1 (2) (2)
(2)
= (2)
+ (2) (2) = 𝑎1 (1 - 𝑎1 )
𝜕𝑎1 𝜕𝑎1 𝜕𝑎2 𝜕𝑧1

= 0.5932(1-0.5932)
𝜕𝐽(𝜃) = 0.2413
(2)
= 0.055 − 0.0190
𝜕𝑎1
(2) (1) (1)
𝑧1 = 𝑥1 𝜃11 +𝑥2 𝜃12 +𝑏1

(2)
𝜕𝑧1
(1) = 𝑥1 = 0.05
𝜕𝜃11
153
 Example: Backpropagation
(2) (2) In the same way, updated weights are
𝜕𝐽(𝜃) 𝜕𝐽(𝜃) 𝜕𝑎1 𝜕𝑧1
(1)
= (2) (2) (1)
𝜕𝜃11 𝜕𝑎1 𝜕𝑧1 𝜕𝜃11 (1)
𝜃12 = 0.199566
(1)
= 0.03635 × 0.2413×0.05 𝜃21 = 0.2497
(1)
= 0.0004 𝜃22 = 0.2995

(1) (1) 𝜕𝐽(𝜃)

𝜃11 = 𝜃11 − 𝛼 (1)
𝜕𝜃11
= 0.15 – 0.5 × 0.0004
= 0.1497

154
 Example: Backpropagation
• Using updated weights, use forward
propagation and calculate 𝐽(𝜃).
• Go backwards and repeat the process till
output is equal to target output
• Or error is minimum

155