ARTIFICIAL

NEURAL
NETWORKS
ARTIFICIAL NEURAL
NETWORKS

Topics :
 Neural Network
Representation
 Differences between
Biological and Neural
Network 2
 NETWORK
 An Artificial Neural Network(ANN) may be
defined as an information – processing model
that is inspired by the Biological Nervous
System , such as Brain.
 The model tries to replicate only the most basic
functions
of the Brain.
 An ANN is composed of a large no. of
highly inter connected processing units
(Neurons) working to solve the specific
3

Problem.
 D..
 Ie, Just as we useour brains to identify
patterns and
classify different types of information
 By using the Neural Networks, it allowing
computers to observe, learn, and react to
complex situations faster than humans.
 An ANN is configured as different
types of Applications, such as Face
Recognition, Pattern Recognition etc.
through a learning process. 4
 WORKS?
Working procedure of Human Brain and ANN:
 The term "Artificial Neural Network" is
derived from Biological neural networks that
develop the structure of a human brain.
 Similar to the human brain that has
neurons interconnected to one another.
A neuron is the fundamental unit which is
used for to is build the brain
 A typical brain contains something like 100
billion microscopic cells called neurons. Each
5

neuron is made up of a cell body (the central

 D..
 There are over 100 billion neurons inside a
human brain. Neurons are responsible for
everything that we do. Our conscious &
subconscious is controlled by these small
power stations.
 The Structure of Biological Neural Network
is shown Below:

6
 CO
NT
D..

Figure: Biological Neural

Network
 D..
 Soma: This is also called the cell body. It is
where the cell nucleus is located.
 Dendrites: These are tree-like networks
that are connected to the cell body. ie, it
receives the signal from other Neurons.
(Input)
 Axon: Axon carries the signal from the cell
body. It splits into strands (Chain or a single
unit of a molecule such as DNA) and each
strand ends in a bulb-like structure called
8
synapse. (output)
D..

9
 D..

Table: Terminology b/w Human Brain

and ANN

10
 OF ANN
 Artificial Neural Networks are processing elements
either in
the form of algorithms or hardware devices
modeled.
 Artificial Neural Network primarily consists of Three
Layers:
[Link] Layer
[Link] Layer
[Link] Layer
 Each Layer consists of more Neurons
 The nodes of the Input Layer can take input data
and perform operations on it and send the
11

results of these operations to other neurons of the

CO
NT
D..

12
WHAT HAPPENS
INSIDE THE NEURON

13
WHAT HAPPENS
INSIDE THE NEURON

14
 BIAS IN NEURAL
 Bias is a systematic error that occurs due to wrong
NETWORK
assumptions in the machine learning process.
 Bias is simply defined as the inability of the model because of
that there is some difference or error occurring between the
model’s predicted value and the actual value.
 These differences between actual or expected values and the
predicted values are known as error or bias error or error
due to bias.
 Biases provide a critical additional layer of flexibility to
neural networks. Biases are essentially constants associated
15

with each neuron. Unlike weights, biases are not connected to

 CO
NT
 Biases provide a critical additional layer of flexibility to
neural networks. D..
 Biases are essentially constants associated with each neuron.
Unlike weights, biases are not connected to specific inputs
but are added to the neuron’s output.
 It is used to adjust the output along the weighted sum of the
inputs of the neuron.
 This Bias Value allows you to shift the Activation Function to
either Left or Right.
16
 CO
NT
D..

17

outpu = su (weight * inputs + bia

 D..
 wecan represent the sigmoid activation function with
the
mathematical expression as

18
 D..
1. Let us vary different values of w and fix the b value to
0

when the value of b =0 and

w = 0.3 — the blue line in the plot
w= 0.5 — the red line in the plot
w = 0.7 — the green line in the
plot

19
 D..
2. Let us vary different values of b and fix w value to 0.5
b = -1 — the red line in the plot
b= -5 — the green line in the plot
b = 1 — the blue line in the plot
b = 5 — the yellow line in the
plot

20
EXAMPLE

21
NETWORK
Three basic components are used in ANN are
[Link] 2. Learning 3. Activation Function

22

EXPLANATION OF LAYERS IN
The input layer has two input neurons, while
ANN
the output
layer consists of three neurons.

2
3
 ..
 Each connection between two neurons is
represented by a
different weight w. Each of these weight w has
indices.
 The firstvalue of the indices stands
for the numberof neurons in the layer from
which the connections originate.
 The second value for the number of the neurons in
the layer to which the connection leads.
 All weights between two neural network
layers can be represented by a matrix
2
called the weight matrix. 4
 ..

In this particular example, the number of

rows of the weight matrix corresponds to the
size of the input layer which is two and the
number of columns to the size of the output
layer which is three.
2
5
 NN
 For a given inputfeature vector x,
the neural network

calculates a prediction vector, which we call

here as h.

2
Figure: Forward Propagation 6
 ..
 This step is also referred to as the forward
propagation. With the input vector x
and the weight matrix W connecting the
two neuron layers, we compute the dot
product between the vector x and the matrix
W. The result of this dot product is again a vector,
which we call it as Z

2
7
 CONTD
..
 The final prediction vector h is obtained by
applying a so- called activation function to the
vector z. In this case, the activation function is
represented by the letter Sigma.
 Here neuron is simply a representation of a
numeric value.

2
8
 ..
 ACTIVATION FUNCTION: It is also known as
Transfer Function. It can also be attached with 2
neural
is also anetworks.
non linearItFunction.
 Activation Functions are Functions
Mathematical that

 determine the output of a Neural

The Activation Function is attached to each
Network
neuron in a neural network and determines
whether neuron should be activated or not.

2
9
 ..
 The main purpose is to convert a input signal
of a node in an ANN to an Output Signal.
 It translates the input signals to output signals. It
maps the output values on a range like 0 to 1 or -1
to 1.

3
0
 ..
 Some of Different Activation Functions used
in Machine Learning are:
..
..

3
3
 ..
 we can extend our knowledge to a deeper
architecture that
consists of 5 layers.
 h1 is now considered as the input for the
upcoming third layer. The whole procedure from
before is repeated until we obtain the final output
y:

3
4
 COMPARISON
 1. Speed of Operation
 2. Way of Processing Data
 3. Size and Complexity of the Neural Network
 4. Storage Capacity or Memory
 5. Fault Tolerance
 6. Control System

35
 D..
 1. Speed of Operation: The Execution time of
ANN is of the order of Nano Seconds.
 But in case of Biological Neural Network
(BNN) or Human Brain, execution time is
of the order of MilliSeconds.
 So ANN is faster than Brain.
 2. Data Processing: In both Brain and ANN
processing of
data takes place parallelly.
 i.e, Multiple Instructions are executed at the
36
same time , Simultaneously.
 But ANN is faster than Brain in data
 D..
 3. Size and Complexity of the Network:
Number of Neurons in the Brain about 1011 and
there are about 1015 inter connecting links.
 But in ANN it can 10-1000 neurons are
used and the
complexity is reduced when compared to
Humans
 So, Brain is More Complex as compared to
ANN

37
 D..
 4. Storage Capacity and Memory: In Human
Brain, data is stored in its synaptic
interconnections.
 But in ANN, data is stored in
specifically allotted memory locations
 Storage Capacity of ANN is limited, But that
of brain is Unlimited.
 Data once stored in ANN can be retrieved
easily at any
time. But in case of Brain , Data 38

loss canoccur due to memory Failure.

 D..
 5. Fault Tolerance: In ANN, if one of the
neurons or connecting links get damaged or
disconnected. i.e, In the event of the system
failure, Corrupted Data cannot be
recovered.
 But in Human Brain, if some of the Neuron
Cells get damaged or Die, it doesn’t cause
much damage to the nervous system.

39
 D..
 6. Control System : In ANN are
modeled using computers and the main
controlling unit is the Central Processing
Unit(CPU)
 But in Human Brain, control lies in the active
chemicals
present and the resultant chemical reactions
taking place.

40
 APPLICATIONS OF
ARTIFICIAL NEURAL NETWORK

41
THANK
YOU

42
 (APPROPRIATE
PROBLEMS IN
ANN)

1
 TOPIC
S

Topics :
 Appropriate Problems for
NN Learning

2
 NEURAL NETWORK
LEARNING
 ANN learning is well-suited to problems in
which the training data corresponds to noisy,
complex sensor data, such as inputs from
cameras and microphones.
 Artificial neural networks (ANNs) provide a
general, practical method for learning real-
valued, discrete-valued, and vector-valued
functions from examples.
 Algorithms such as BACKPROPAGATION3
gradient descent to tune network parameters
 D..
 ANN learning is robust to errors in the
training data and has been successfully
applied to problems such as interpreting
visual scenes, speech recognition, and
 learning
One ofrobot controlApplication
the strategies. in ANN
is
Best Land
in a
ALVINN(Autonomo Vehicle
Neural
 It
usisNetwork)
Designed by Pomerleau in the year 1993,
is a Neural Network that has performed well in a
Domain .

4
CO
NT
D..

5
CO
NT
D..

6
CO
NT
D..

7
 D..
 Neural network learning to steer an
autonomous vehicle.
 The ALVINN system uses Back propagation to
learn to steer an autonomous vehicle (photo
at top right) driving at speed up to 70 miles per
hour.
 The diagram on the left shows how the
image of a forward forward‐mounted
mounted camera is mapped to 960 neural
network inputs, which are fed forward to 4
8
hidden units, connected to 30 output units.
 Network output encoded the commanded
 D..
 The 30 x 32 weights into the hidden unit are
displayed in the large matrix with white
blocks indicating positive and black
indicating negative weights.
 The weights from this hidden unit to the 30
output units are depicted by the smaller
rectangular block directly above the large
block.
 The most appropriate for problems with the
following characteristics: 9
 D..
 1. Instances are represented by many
attribute-value pairs.
 2. The target function output may be
discrete-valued, real-valued, or a
vector of several real- or discrete-
valued attributes.
 3. The training examples may contain
errors.
 4. Long training times are acceptable
 5. Fast evaluation of the learned target
10
function may be required.
 6. The ability of humans to understand
 D..
 1. Instances are represented by many
attribute-value pairs:
 If the particular attribute has many attribute
value pairs or the Continuous Values for the
particular attribute, in such case the
Algorithms i.e, Decision Tree, Candidate
Elimination Algorithms.
 In that case we can use the Neural Networks.
 In ALVINN Example, input attributes may
be highly correlated or independent of one
11

another.
 D..
 Here the main use of Correlation is
“Prediction”.
 If there is a relationship between two
variables, we can
make predictions about one from another.
 Some times input values can be real values.

12
 D..
[Link] target function output may be
discrete-valued, real- valued, or a vector of
several real- or discrete-valued attributes.
 Suppose the Target Function is the Discrete
Values i.e, it means, each and every value is
independent each other.
 Eg: Number of traffic accidents that
occur in a specific city on a given
day.
 Suppose the Target Function is the
13

Continous Values i.e, it means, each and

 D..
 In the ALVINN System the output is a
vector of 30 attributes , each
corresponding to a recommendation
regarding the Steering Direction.
 The value of each output is some real number
between 0 and 1, which in this case corresponds
to the confidence in predicting the
corresponding steering direction.
 We can also train a single network to output
of both the steering command and suggested
14

acceleration, simply by concatenating the

 D..
 3. The training examples may contain errors.
 If the Training Examples like
 Missing Values
 Outliers, In such cases the ANN is the
Best one to solve these types of Errors.

15
 D..
 4. Long training times are acceptable :
 Network training algorithms typically
require longer training times than, say,
decision tree learning algorithms.
 Whenever we use Artificial Neural Networks ,
we expect the data set is very large. i.e, the
attribute may contain large value pairs.
Then we have to process all the
attributes it takes some time to process all
these values
16
 In such situations , The training time is also
high. So here
 D..
 Training times can range from a
few seconds to many hours,
depending on factors such as the
 Number of weights in the network,
 The number of training examples
considered, and
 The settings of various learning
algorithm parameters

17
 D..
 5. Fast evaluation of the learned target
function may be required.
 In ANN, Training it will take more time. But
once learn,
it evaluated is very fast.
 For eg: Suppose we learn a subject , it takes a
time . But when we are revising it for second
time , i.e, before the exam, it will not take
so much time when compared to Learning
Time.
18
 Although ANN learning times are
relatively long, evaluating the learned
 D..
 For example, ALVINN applies its neural
network several times per second to
continually update its steering command
as the vehicle drives forward.

19
 D..
 6. The ability of humans to understand the
learned
Target Function is not important.
 What ever the Target Function that
the Machine is
Learning, is very complicated
 The weights learned by neural networks are
often
difficult for humans to interpret.
 Learned neural networks are less easily
communicated to humans than learned rules.

20
 Mr. Frank ARCHITECTURE OF THE
Rosenblatt invented the perceptron
model as aPERCEPTRON
binary classifier.

5
 The idea CON
was to use different weights to
represent the importance of each input,TD..
and that
the sum of the values should be greater than
a threshold value before making a decision like
true or false (0 or 1).

6
7
 More precisely, the given inputs x1 through xn,CON
the output o1(x1,-------xn) TD..
computed by the
perceptron is

 The Perceptron receives multiple input signals, and

if the sum of the input signals exceeds a
certain threshold, it either outputs a signal or
does not return an output. 8

 In the context of supervised learning and

 PERCEPTRON
Frank Rosenblatt suggested this
LEARNING ALGORITHM
algorithm:
 Step-1: Set a threshold value
 Step-2: Multiply all inputs with its
weights
 Step-3: Sum all the results
 Step-4: Activate the output

9
 Here, First, multiply all input values with
corresponding WORKING PRINCIPLE
weightvaluesand then add them to determine
the
weighted sum. Mathematically,
we can calculate the
weighted sum as follows:
∑wi*xi = x1*w1 + x2*w2 +…wn*xn
 and that the sum of the values should be
greater than a threshold value before
making a decision like true or false (0 or
1).
 Add a special term called bias 'b' to this
10

weighted sum to
CON
TD..

11
 E
 Imagine a perceptron (in your brain).
 The perceptron tries to decide if
you should go to a concert.
 Is the artist good?
 Is the weather good?
 What weights should these facts have?

12
CO
NT
D..

13
 D..
 Perceptron Terminology:
 1. Inputs : Here Inputs called Nodes
In the above example, the node values are: 1, 0,
1, 0, 1 The binary input values (0 or 1) can be
interpreted as (no or yes) or (false or true).
 2. Node Weights: Weights shows the
strength of each node.
In the above example, the Weights are 0.7, 0.6,
0.5, 0.3, 0.4

14
 D..
 3. Activation Function: The activation
functions maps the result (the weighted sum)
into a required value like 0 or 1.
 In the above Example,the activation
function is simple: (sum > 1.5)
 The binary output (1 or 0) can be interpreted
as (yes or no)
or (true or false).
 Note: It is obvious that a decision is NOT
made by one neuron alone. 15

 Threshold Value: We have to define a reliable

 D..
 Bias: Model bias refers to the presence of
systematic errors in a model that can cause it
to consistently make incorrect predictions.
 These errors can arise from many sources,
including the selection of the training data, the
choice of features used to build the model,
or the algorithm used to train the model.

16
 (MULTI LAYER
NETWORKS AND
THE BACK
1 PROPAGATION
ALGORITHM)
 TOPIC
S

Topics :
 Multi Layer Neural
Networks
 Back Propagation
Algorithm
 Example Problem

2
 NETWORKS
 A fully connected Multi-Layer Neural Network
is called a Multilayer Perceptron (MLP).
 The Multi-Layer-Perceptron was first
introduced by
M. Minsky and S. Papert in 1969. It is
an extended Perceptron and has one ore
more hidden neuron layers between its input
and output layers
 Due to its extended structure, a Multi-Layer-
Perceptron is able to solve every logical
operation, including the XOR problem.
3
 D..
 Multilayer perceptron's (MLPs) are feed forward
neural networks trained with the standard
back propagation algorithm.
 By using the Back propagation, is a
procedure to repeatedly adjust the weights
so as to minimize the difference between
actual output and desired output

4
 CO
NT
 In the multi-layer perceptron diagram above, we
D..
can see that there are three input nodes and the
hidden layer has three nodes. The output layer
gives two outputs, therefore there are two
output nodes.
 The nodes in the input layer take input and
forward it for further process, in the diagram
above the nodes in the input layer forwards
their output to each of the three nodes
 In the hidden layer, and in the same way, the
5

hidden layer processes the information and

 CO
NT
 Hence, a multi-layered perceptron model has
considered as artificial neural D..
networks
having various layers in which activation
function does not remain linear, similar to a
single layer perceptron model.
 In this Networks, we used the Non Linear
Activation Functions [Link], Sigmoid, Relu,
Tanh etc. Since the probability of any event lies
between 0 and 1.
 In Multi Layer Networks, We can follow the
Supervised Learning. 6
DIFFERENT NON
LINEAR ACTIVATION
FUNCTIONS

7
 BACK PROPAGATION
ALGORITHM
 In an Artificial Neural Network, the values
of weights and biases are randomly initialized.
 Due to random initialization, the neural
network probably has errors for the given
inputs.
 So, We need to reduce error values as much as
possible.
 So, for reducing these error values, we need a
mechanism that can compare the desired
8
output of the neural network with the
Target network’s output
 CONTD..
 Suppose, that consists of errors and adjusts
its weights and biases such that it gets
closer to the desired output after each
iteration.
 For this, we train the network such that it back
propagates and updates the weights and
biases. This is the concept of the back
propagation algorithm.
 Below are the steps that an artificial neural
9

network follows to gain maximum accuracy

 CONTD..
 Steps:
 1. Parameter
Initialization
 2. Feed Forward
Propagation
 3. Back Propagation

10
 CONTD..
One way to train our model is called as Back propagation.
Consider the diagram below:

11
 CONTD..
Let me summarize the steps for you:
Calculate the error – How far is your model output
from the actual output.
Minimum Error – Check whether the error is
minimized or
not.
Update the parameters – If the error is huge
then, update the parameters (weights and
biases). After that again check the error. Repeat the
process until the error becomes minimum.
12

Model is ready to make a prediction –

 BACK PROPAGATION
ALGORITHM
 The below Figure show you the
Network: i.e, Feed Forward Network.

13
 CO
NT
 In the above Figure,
 Input layer with two inputs neurons
D..
 I1, I2 are the Input Layers
 One hidden layer with two neurons
 H1, H2 are the Hidden Layers
 Output layer with two neurons
 O1,O2 are the Output Layers
 Target O1, Target O2 are the Desired
Output (or)the Expected Output
14
 CO
NT
 Here I1, I2 are the Input Layers and W1,
D..
W2, W3, W4 are the corresponding Weights.
 From the above Network, First
we canCalculate the Net H1 and Net
H2.

15
 CO
NT
 After Calculating net H1 and net H2.
compute Out H1 and Out H2 as follows:
D..

16
 CO
NT
 Here we can use the Activation Function is the
D..
sigmoid function ,the values for which it is used
in the range, 0 to 1.
 It is used for models where we have to
predict the probability. Since the probability
of any event lies between 0 and 1, the
sigmoid function is the right choice.

17
 CO
NT
 Similarly, we canCalculating net O1
and net O2. D..

18
 CO
NT
 From the above Equations, we can calculate
Out O1 and Out O2 as follows:
D..

19
 CO
NT
 Now , Set some Target Values for both
D..
Outputs and Calculate Error Outputs based
on Target and Calculated Outputs.

Here, Eo1 is the Error at Output O1

Here, Eo2 is the Error at Output O2

 Next, we can Calculating the
Total Error
20
 CO
NT
 Our main goal of the training is to reduce the
D..
error or the difference between prediction and
actual output.
 Suppose thereis an Error occurred ,
we will use “ Back Propagation”

21
 BACK
 Back propagation, short for “backward
propagation
PROPAGATION
of errors”, is a
mechanism used to update the weights
using gradient descent.
 Gradient descent is an iterative optimization
algorithm
for finding the minimum of a function.
 In our case we want to minimize the error
function.
 It calculates the gradient of the error function
22
with respect to the neural network’s weights.
 CO
NT
 The Formula for the Gradient
Descent is: D..

23
 CO
NT
 2. Back ward Pass:
D..
 When we got the Error. i.e, Target Output O1
and Actual Output O1.
 Similarly, Target Output O2 and Actual Output
O2.
 So that we have to move Backward and we
have to adjust the weights of W5, W6, W7, W8.
 Now we have to Calculate the Output Values.
 If we get the Expected Output and
Actual Outputs are Same, We will stop.24
 CO
NT
 Otherwise again we will back and Update the
weights of D..
W1, W2, W3, W4 also.
 This process continues unless until the Actual
Output and Target Output are the Same.
 Update the Weights: Here we can use
the Gradient Descent.
 i.e, for suppose we have to update the
weights : W5
 1. For W5:
25
 CO
NT
 Here, W5 has to be updated [Link] the Total
Error. D..
 So here first calculate the

 This can be
written as

 So, here will calculate each and every term 26

separately.
 CO
NT
 Here we can apply the Chain
Rule. D..

 First
term,

27
 CO
NT
 we will written
as, D..

 After apply the partial Derivation, we

will get :

28
 CO
NT
 After
Simplifying, D..

29
 CO
NT
 Second
term, D..

30
 CO
NT
 Apply the partial Derivation, we
will get : D..

31
 CO
NT
 After apply the partial Derivation, we
will get : D..

32
 CO
NT
 Third
term, D..

 By combining, all these

terms

33
 CO
NT
 After Calculating
the D..

 Update the Weight ,

W5 as

 Here,
 W5 is the Old Value
 a – is the Learning Rate or 34

Step Rate.
CO
NT
D..

35
EXAMPLE
PROBLEM

36
 CO
NT
The above network contains the following:
1. two inputs : X1 and X2
D..
2. two hidden neurons : h1 and h2
3. two output neurons : o1 and o2
4. two biases : b1 and b2
Below are the steps involved in Back propagation:
Step – 1: Forward Propagation
Step – 2: Backward Propagation
Step – 3: Putting all the values together and
37
calculating the updated weight value
CO
NT
D..

38
 CO
NT
Step – 1: Forward
Propagation : D..
We will start by propagating
forward.

39
CO
NT
D..

40
 D..
We will repeat this process for the output layer neurons,
using the output from the hidden layer neurons as inputs.

41
CO
NT
D..

42
 CO
NT
Calculating the Total error:
D..

43
 CO
NT
Step – 2: Backward Propagation
D..
Now, we will propagate backwards. This way we will try to
reduce the error by changing the values of weights and
biases.
Consider W5, we will calculate the rate of change of error w.r.t
change in weight W5.

44
 CO
NT
Calculate Each and Every Term:
First Term:
D..

45
 CO
NT
Calculate Second Term:
D..

Calculate Third Term:

46
 D..
Step – 3: Putting all the values together and
calculating the updated weight value

47
 CO
NT
Let’s calculate the updated value of W5:
D..

48
 CO
NT
 Similarly, we can calculate the other weight values as well.
D..
 After that we will again propagate forward and calculate the
output. Again, we will calculate the error.
 If the error is minimum we will stop right there, else we will
again propagate backwards and update the weight values.
 This process will keep on repeating until error
becomes
minimum.

49
 EXAMPLE
PROBLEM - 2

The correct output from output node o1 and o2 be y1 and y2 respectively.

Let's assume the value of y1 = 0.05 and the value of y2 = 0.95 which are
50
the correct outputs labeled for the given inputs.