Chapter 2: Artificial Neural Network (ANN)

Biological Neural Network: In living organisms, the brain is the control unit of the neural
network, and it has different subunits that take care of vision, senses, movement, and hearing. The
brain is connected with a dense network of nerves to the rest of the body’s sensors and actors.
There are approximately 10¹¹ neurons in the brain, and these are the building blocks of the
complete central nervous system of the living body.

The neuron is the fundamental building block of neural networks. In the biological systems, a
neuron is a cell just like any other cell of the body, which has a DNA code and is generated in the
same way as the other cells. Though it might have different DNA, the function is similar in all the
organisms. A neuron comprises three major parts: the cell body (also called Soma), the dendrites,
and the axon. The dendrites are like fibers branched in different directions and are connected to
many cells in that cluster.

Dendrites receive the signals from surrounding neurons, and the axon transmits the signal to the
other neurons. At the ending terminal of the axon, the contact with the dendrite is made through a
synapse. Axon is a long fiber that transports the output signal as electric impulses along its length.
Each neuron has one axon. Axons pass impulses from one neuron to another like a domino effect.

Figure 2.1: Biological Neuron

At age 20, the average weight of the male brain is approximately 1400 g, and by the age of 65
brain weight is approximately 1300 g. Brain weight for females follows a similar trend, although
the total weight is 100–150 g less than that of males.
Our body is composed of millions to billions of nerve cells. Some nerve cells can be comparatively
smaller by 0.1 millimeters or can be longer by 1 meter.

Artificial Neural Network (ANN): The term "Artificial Neural Network" is derived from
Biological neural networks that develop the structure of a human brain. Similar to the human brain
that has neurons interconnected to one another, artificial neural networks also have neurons that
are interconnected to one another in various layers of the networks. These neurons are known as
nodes.
 Figure 2.2: Representation of Biological vs. Artificial Neurons
An Artificial Neural Network (ANN) is a computational model inspired by the structure and
functioning of biological neural networks, such as the human brain. ANNs are used in machine
learning and artificial intelligence to solve a wide range of tasks, including pattern recognition,
classification, regression, and more. Here are some key features and components of artificial neural
networks:
Neurons (Nodes): Neurons in an ANN are the fundamental units of computation. They receive
input signals, perform mathematical operations on these inputs, and produce an output signal.
Neurons are also referred to as nodes or units.
Layers: ANNs are organized into layers. Typically, there are three main types of layers:

Input Layer: This layer receives the initial data or features.

Hidden Layers: These intermediate layers process the data through a series of weighted
connections and apply activation functions to produce complex representations of the
input.

Output Layer: This layer produces the final result or prediction.

Figure 2.3: Artificial Neural Networks (ANN)

Weights and Connections: The connections between neurons in different layers have associated
weights. These weights determine the strength of the connections. During the training process,
these weights are adjusted to make the network produce accurate output.

Activation Functions: Each neuron in a neural network typically has an activation function, which
determines its output based on the weighted sum of its inputs. Common activation functions
include Sigmoid, ReLU (Rectified Linear Unit), and Tanh.
Feedforward and Backpropagation: In the feedforward process, data is passed through the
network from the input layer to the output layer. During training, backpropagation is used to update
the weights based on the error between the predicted output and the actual target values.
Training: Training an ANN involves using a dataset with known inputs and desired outputs to
adjust the network's weights and biases. Optimization techniques like gradient descent and
algorithms like backpropagation are commonly used to update these parameters and minimize the
prediction error.

Artificial Neuron: An artificial neuron is a mathematical function inspired by biological neurons

in the human brain. In artificial intelligence (AI), particularly in the context of neural networks,
neurons are the basic unit of computation. Every artificial neuron is composed of at least one input,
a weight, a bias, and an activation function. Most neurons have more than one input.

Each input comes with its own weight, which is adjusted during training to reflect the input’s
importance. In addition to weighted inputs, each neuron has a bias. This is an extra input that has
a set value that doesn’t change. The bias allows the neuron to produce outputs other than zero,
even if all the inputs are zero. Here is an example of how an artificial neuron works:
o The neuron receives multiple input values.
o Each of these inputs is multiplied by its corresponding weight.
o All the weighted inputs are added up, and the bias is added to this sum.
o The result is passed through the activation function to produce the neuron’s output. The
activation function introduces non-linearity to the model, enabling it to learn from errors
and make adjustments.

Figure 2.4: Artificial Neuron (AN)

Perceptron Algorithm: Perceptron is Machine Learning algorithm for supervised learning of

various binary classification tasks. Further, Perceptron is also understood as an Artificial Neuron
or neural network unit that helps to detect certain input data computations in business intelligence.
Perceptron model is also treated as one of the best and simplest types of Artificial Neural networks.
However, it is a supervised learning algorithm of binary classifiers. Hence, we can consider it as a
single-layer neural network with four main parameters, i.e., input values, weights and Bias, net
sum, and an activation function.

Mr. Frank Rosenblatt invented the perceptron model as a binary classifier which contains three
main components. These are as follows:
Input Nodes or Input Layer: This is the primary component of Perceptron which accepts
the initial data into the system for further processing. Each input node contains a real
numerical value.
Wight and Bias: Weight parameter represents the strength of the connection between units.
This is another most important parameter of Perceptron components. Weight is directly
proportional to the strength of the associated input neuron in deciding the output. Further,
Bias can be considered as the line of intercept in a linear equation.

Activation Function: These are the final and important components that help to determine
whether the neuron will fire or not. Activation Function can be considered primarily as a
step function.

Types of Activation functions:

o Sign function
o Step function, and
o Sigmoid function

Figure 2.5: Activation Function

1 𝑖𝑓 𝑆𝑢𝑚 > 0
Step Function: 𝒀 = { }
0 𝑖𝑓 𝑆𝑢𝑚 ≤ 0
+1 𝑖𝑓 𝑆𝑢𝑚 > 0
Sign Function: 𝒀 = { }
−1 𝑖𝑓 𝑆𝑢𝑚 ≤ 0
 𝟏
Sigmoid Function: 𝒀 = 𝟏+𝒆−𝑺𝒖𝒎

Types of Perceptron Models: Based on the layers, Perceptron models are divided into two types.
These are as follows:

o Single-layer Perceptron Model

o Multi-layer Perceptron model

Single-Layer Perceptron (SLP): A single layer perceptron (SLP) is a feed-forward network based
on a threshold transfer function. SLP is the simplest type of artificial neural networks and can only
classify linearly separable cases with a binary target (1, 0).

Figure 2.6: Single-Layer Perceptron (SLP)

Input x1, x2 and the step function is

Input value or One input layer: The input layer of the perceptron is made of artificial input
neurons and takes the initial data into the system for further processing.

Weights and Bias:

Weight: It represents the dimension or strength of the connection between units. If the
weight to node 1 to node 2 has a higher quantity, then neuron 1 has a more considerable
influence on the neuron.
Bias: It is the same as the intercept added in a linear equation. It is an additional parameter
which task is to modify the output along with the weighted sum of the input to the other
neuron.

Net sum: It calculates the total sum.

Activation Function: A neuron can be activated or not, is determined by an activation function.
The activation function calculates a weighted sum and further adding bias with it to give the result.

Algorithm
1. Create a perceptron with (n + 1) input neurons x0, x1, . . . xn, where x0 = 1 is the bias input.
Let O be the output neuron.
2. Initialize w = (w0, w1, . . ., wn) to random weights.
3. Iterate through the input patterns xj of the training set using the weight set, i, e. Compute
the weighted sum of inputs netj = ∑𝑛𝑖=0(𝑥𝑖 w𝑖 ) for each input pattern j.
4. Compute the output yj using the step function –

1 𝑖𝑓 𝑛𝑒𝑡𝑖 > 0
𝒀={ }
0 𝑖𝑓 𝑛𝑒𝑡𝑖 ≤ 0

5. Compare the computed output yj with the target output yj for each input pattern j. If all the
input patterns have been classified correctly, output the weights and exit.
6. Otherwise, update the weighs as given below: If the computed output yj is 1 but should
have been 0, wi = wi + αxi, i = 0, 1, 2, . . ., n. If the computed output yj is 1 but should have
been 1, wi = wi + αxi, i = 0, 1, 2, . . ., n. Here α is the learning parameter and is a constant.
7. Go to step 3.

Perceptron Weight Adjustment

wnew = wold + α (yt-yj) xi
Here α is Learning rate
yj is Computed output
yt is Target Output
x is Input data
SLP sums all the weighted inputs and if the sum is above the threshold (some predetermined value),
SLP is said to be activated (output=1).

The input values are presented to the perceptron, and if the predicted output is the same as the
desired output, then the performance is considered satisfactory and no changes to the weights are
made. However, if the output does not match the desired output, then the weights need to be
changed to reduce the error.
Because SLP is a linear classifier and if the cases are not linearly separable the learning process
will never reach a point where all the cases are classified properly. The most famous example of
the inability of perceptron to solve problems with linearly non-separable cases is the XOR
problem.

Error Calculation: e = (yt - yc)

Problem-1: Solve the following problem by using Single Lyer Perceptron Algorithm (SLP) for
the following logical XOR operation. Where learning rate α = .2.

Input Output
X1 X2 Yt
0 0 0
0 1 1
1 0 1
1 1 0
Solution:
Iteration-1:

Input: x1 = 0 and x2 = 0
Random Weight: w1 = 0.3 w2 = 0.3

Net= ∑ wixi
= w1x1 + w2x2

= 0.3*0 + 0.3*0
=0
Apply Step Activation Function to calculate the computed output yc. The step activation function
is:
1 𝑖𝑓 𝑛𝑒𝑡𝑖 > 0
𝒀𝒄 = { }
0 𝑖𝑓 𝑛𝑒𝑡𝑖 ≤ 0
Since the Net= 0 so that yc=0. Here yt = 0 and yc= 0 Here, Yt is target Output, and yc= Computed
Output. Since both values are same so neurons can successfully computed the output. So that this
iteration has finished.
Iteration-2:
Input: x1 = 0 and x2 = 1

Random Weight: w1 = 0.3 w2 = 0.3

 Net= ∑ wixi
= w1x1 + w2x2

= .03*0 + 0.3*1
= 0.3

Apply Step Activation Function to calculate the computed output yc. The step activation function
is:
1 𝑖𝑓 𝑛𝑒𝑡𝑖 > 0
𝒀𝒄 = { }
0 𝑖𝑓 𝑛𝑒𝑡𝑖 ≤ 0

Since the Net= .3 so that yc=1. Here yt = 1 and yc= 1 Here, Yt is target Output, and yc= Computed
Output. Since both values are same so neurons can successfully compute the output. So that this
iteration has finished.
Iteration-3:

Input: x1 = 1 and x2 = 0
Random Weight: w1 = 0.3 w2 = 0.3

Net= ∑ wixi
= w1x1 + w2x2

= .03*1 + 0.3*0
= 0.3

Apply Step Activation Function to calculate the computed output yc. The step activation function
is:
1 𝑖𝑓 𝑛𝑒𝑡𝑖 > 0
𝒀𝒄 = { }
0 𝑖𝑓 𝑛𝑒𝑡𝑖 ≤ 0
Since the Net= .3 so that yc=1. Here yt = 1 and yc= 1 Here, Yt is target Output, and yc= Computed
Output. Since both values are same so neurons can successfully compute the output. So that this
iteration has finished.
Iteration-4:

Input: x1 = 1 and x2 = 1
Random Weight: w1 = 0.3 w2 = 0.3

Net= ∑ wixi
= w1x1 + w2x2
 = .03*1 + 0.3*1
= 0.6

Apply Step Activation Function to calculate the computed output yc. The step activation function
is:
1 𝑖𝑓 𝑛𝑒𝑡𝑖 > 0
𝒀𝒄 = { }
0 𝑖𝑓 𝑛𝑒𝑡𝑖 ≤ 0

Since the Net= .6 so that yc=1. Here yt = 0 and yc= 1 Here, Yt is target Output, and yc= Computed
Output. Since both values are not same so that need to update weight and Calculate this iteration
again.

Now Update the weight –

Winew= wiold + α(yt- yc)xi [Here α is Learning rate, and α = .2]

W1new = w1old + α(yt- yc)x1

= .3 + .2(0 - 1) 1 [Here α is Learning rate, and α = .2]
= .3 - .2 = 0.1
W2 = w2old – α(yt- yc)x2
new

= .3 + .2(0 - 1) 1 [Here α is Learning rate, and α = .2]

= .3 - .2 = 0.1
Iteration-4.1:
Input: x1 = 1 and x2 = 1
Updated Weight: w1 = 0.1 w2 = 0.1

Net= ∑ wixi
= w1x1 + w2x2

= .01*1 + 0.1*1
= 0.2

Apply Step Activation Function to calculate the computed output yc. The step activation function
is:
1 𝑖𝑓 𝑛𝑒𝑡𝑖 > 0
𝒀𝒄 = { }
0 𝑖𝑓 𝑛𝑒𝑡𝑖 ≤ 0

Since the Net= .1 so that yc=1. Here yt = 0 and yc= 1 Here, Yt is target Output, and yc= Computed
Output. Since both values are not same so that need to update weight and calculate this iteration
again.
Now Update the weight –
Winew= wiold+α(yt- yc)xi [Here α is Learning rate, and α = .2]

W1new = w1old + α(yt- yc)x1

= .2 + .2(0 - 1) 1 [Here α is Learning rate, and α = .2]

=0
W2new = w2old – α(yt- yc)x2

= .2 - .2(0 - 1) 1 [Here α is Learning rate, and α = .2]

=0

Iteration-4.2:
Input: x1 = 1 and x2 = 1

Random Weight: w1 = 0 w2 = 0
Net= ∑ wixi
= w1x1 + w2x2

= 0*1 + 0*1
=0

Apply Step Activation Function to calculate the computed output yc. The step activation function
is:
1 𝑖𝑓 𝑛𝑒𝑡𝑖 > 0
𝒀𝒄 = { }
0 𝑖𝑓 𝑛𝑒𝑡𝑖 ≤ 0
Since the Net= 0 so that yc=0. Here yt = 0 and yc= 0 Here, Yt is target Output, and yc= Computed
Output. Since both values are same so neurons can successfully computed the output. So that this
iteration has finished.