IV [Link] VII Sem.

Deep Learning Notes Unit-II 1

1. Discuss about Biological Neuron?

In the context of a neural network, a neuron is the most fundamental unit of processing. It’s also called a
perceptron. A neural network is based on the way a human brain works. So, we can say that it simulates the
way the biological neurons signal to one another.
Biological Neuron: We can define neurons as the information carriers that use electrical impulses and
chemical signals to transmit information. The neurons transmit the information in the following two areas:
1. different parts of the brain
2. the brain and the nervous system
Thus, whatever we think, feel, and later do is all due to the working of the neurons.
The following figure shows a typical biological neuron:

A neuron has the following three basic parts:

1. Cell body
2. Cell extension Axon
3. Cell extension Dendrite
The following figure shows the architecture of a biological neuron:

The nucleus in the cell body controls the cell’s functioning. The axon extension (having a long tail)
transmits messages from the cell. Dendrites extension (like a tree branch) receive messages for the cell.
So, in a nutshell, we can summarize that the biological neurons communicate with each other by sending
chemicals, called neurotransmitters, across a tiny space, called a synapse, between the axons and dendrites
of adjacent neurons.

Biological Neural Network: The biological neural network is also composed of several processing pieces
known as neurons that are linked together via synapses. These neurons accept either external input or the
results of other neurons. The generated output from the individual neurons propagates its effect on the entire
network to the last layer, where the results can be displayed to the outside world.
IV [Link] VII Sem. Deep Learning Notes Unit-II 2

Every synapse has a processing value and weight recognized during network training. The performance and
potency of the network fully depend on the neuron numbers in the network, how they are connected to each
other (i.e., topology), and the weights assigned to every synapse.

Advantages and Disadvantages of Biological Neural Network: There are various advantages and
disadvantages of the biological neural network. Some advantages and disadvantages of the biological neural
network are as follows:
Advantages:
1. It can handle extremely complex parallel inputs.
2. The input processing element is the synapses.
Disadvantages:
1. As it is complex, the processing speed is slow.
2. There is no controlling mechanism in this network.

2. Illustrate Idea of computational units?

Deep Learning Activation Functions: Activation functions are a core concept to understand in deep
learning.
They are what allow neurons in a neural network to communicate with each other through their synapses.
In this section, you will learn to understand the importance and functionality of activation functions in deep
learning.
Activation Functions in Deep Learning: A weighted sum of these signals is fed into the neuron's
activation function, and then the activation function's output is passed onto the next layer of the network.
There are four main types of activation functions:
1. Threshold functions
2. Sigmoid functions
3. Rectifier functions, or ReLUs
4. Hyperbolic Tangent functions
Threshold Functions: Threshold functions compute a different output signal depending on whether or not
its input lies above or below a certain threshold. Remember, the input value to an activation function is the
weighted sum of the input values from the preceding layer in the neural network.
Mathematically speaking, here is the formal definition of a deep learning threshold function:

As the image above suggests, the threshold function is sometimes also called a unit step function.
Threshold functions are similar to Boolean variables in computer programming. Their computed value is
either 1 (similar to True) or 0 (equivalent to False).
The Sigmoid Function: The sigmoid function is well-known among the data science community because of
its use in logistic regression, one of the core deep learning techniques used to solve classification problems.
The sigmoid function can accept any value, but always computes a value between 0 and 1.
IV [Link] VII Sem. Deep Learning Notes Unit-II 3

Here is the mathematical definition of the sigmoid function:

One benefit of the sigmoid function over the threshold function is that its curve is smooth. This means it is
possible to calculate derivatives at any point along the curve.
The Rectifier Function: The rectifier function does not have the same smoothness property as the sigmoid
function from the last section. However, it is still very popular in the field of deep learning.
The rectifier function is defined as follows:
 If the input value is less than 0, then the function outputs 0
 If not, the function outputs its input value
Here is this concept explained mathematically:

Rectifier functions are often called Rectified Linear Unit activation functions, or ReLUs for short.
The Hyperbolic Tangent Function: The hyperbolic tangent function is the only activation function, that is
based on a trigonometric identity.
It’s mathematical definition is below:
IV [Link] VII Sem. Deep Learning Notes Unit-II 4

The hyperbolic tangent function is similar in appearance to the sigmoid function, but its output values are all
shifted downwards.

3. Explain about liner Perceptron?

Perceptron: Perceptron is Deep Learning algorithm for supervised learning of various binary classification
tasks. Further, Perceptron is also understood as an Artificial Neuron or neural network unit that helps to
detect certain input data computations in business intelligence.
Perceptron model is also treated as one of the best and simplest types of Artificial Neural networks.
However, it is a supervised learning algorithm of binary classifiers. Hence, we can consider it as a single-
layer neural network with four main parameters, i.e., input values, weights and Bias, net sum, and an
activation function.
Binary classifier in Deep Learning?
In Deep Learning, binary classifiers are defined as the function that helps in deciding whether input data can
be represented as vectors of numbers and belongs to some specific class.
Binary classifiers can be considered as linear classifiers. In simple words, we can understand it as
a classification algorithm that can predict linear predictor function in terms of weight and feature vectors.
Basic Components of Perceptron
Mr. Frank Rosenblatt invented the perceptron model as a binary classifier which contains three main
components. These are as follows:

Input Nodes or Input Layer: This is the primary component of Perceptron which accepts the initial data
into the system for further processing. Each input node contains a real numerical value.
IV [Link] VII Sem. Deep Learning Notes Unit-II 5

Weight and Bias: Weight parameter represents the strength of the connection between units. This is another
most important parameter of Perceptron components. Weight is directly proportional to the strength of the
associated input neuron in deciding the output. Further, Bias can be considered as the line of intercept in a
linear equation.
Activation Function: These are the final and important components that help to determine whether the
neuron will fire or not. Activation Function can be considered primarily as a step function.
Types of Activation functions:
o Sign function
o Step function, and
o Sigmoid function

The data scientist uses the activation function to take a subjective decision based on various problem
statements and forms the desired outputs. Activation function may differ (e.g., Sign, Step, and Sigmoid) in
perceptron models by checking whether the learning process is slow or has vanishing or exploding gradients.

4. Explain about Perceptron learning algorithm?

Perceptron is a linear supervised machine learning algorithm. It is used for binary classification. The
perceptrons, which forms the basis for the most popular machine learning models nowadays – the n
eural networks.
Perceptron Learning Algorithm is also understood as an Artificial Neuron or neural network unit that
helps to detect certain input data computations in business intelligence. The perceptron learning algorithm is
treated as the most straightforward Artificial Neural network. It is a supervised learning algorithm of binary
classifiers. Hence, it is a single-layer neural network with four main parameters, i.e., input values, weights
and Bias, net sum, and an activation function.
There are four significant steps in a perceptron learning algorithm:
1. First, multiply all input values with corresponding weight values and then add them to determine the
weighted sum. Mathematically, we can calculate the weighted sum as follows:
wi∗xi=x1∗w1+x2∗w2+…+wn∗xn.
Add another essential term called bias 'b' to the weighted sum to improve the model performance.
wi∗xi+b.
2. Next, an activation function is applied to this weighed sum, producing a binary or a continuous-
valued output. Y=f(∑wi∗xi+b)

3. Next, the difference between this output and the actual target value is computed to get the error
term, E, generally in terms of mean squared error. The steps up to this form the forward propagation
part of the algorithm.
E=(Y−Yactual)2
IV [Link] VII Sem. Deep Learning Notes Unit-II 6

4. We optimize this error (loss function) using an optimization algorithm. Generally, some form of
gradient descent algorithm is used to find the optimal values of the hyperparameters like learning
rate, weight, Bias, etc. This step forms the backward propagation part of the algorithm.
An overview of this algorithm is illustrated in the following Figure:

In a more standardized notation, the perceptron learning algorithm is as follows:

P <-- inputs with label 1
N <-- inputs with label 0
Initialise w randomly;
while !converge do:
$\hspace{2em}$ Pick random x $\in P \cup N;$
$\hspace{2em}$ if x $\in$ P and w.x $<$ 0 then
$\hspace{3em}$ w = w+x
$\hspace{2em}$ end
$\hspace{3em}$ if x $\in$ N and w.x $\ge$ 0 then
$\hspace{3em}$ w = w-x
$\hspace{2em}$ end
end

Basic Components of Perceptron:

1. Input Nodes or Input Layer: Primary component of Perceptron learning algorithm, which accepts
the initial input data into the model. Each input node contains an actual value.
2. Weight and Bias: The weight parameter represents the strength of the connection between units.
Bias can be considered as the line of intercept in a linear equation.
3. Activation Function: Final and essential components help determine whether the neuron will fire.
The activation function can be primarily considered a step function. There are various types of
activation functions used in a perceptron learning algorithm. Some of them are the sign function, step
function, sigmoid function, etc.

[Link] Convergence Theorem for perceptron learning?

Perceptron Convergence Theorem:
In the classification of linearly separable patterns belonging to two classes only, the training task for the
classifier was to find the weight w such that.
(w^tx>0\hspace{0.4cm} for\hspace{0.2cm}each \hspace{0.2cm}x\in X_1\ w^tx<0\hspace{0.4cm}
for\hspace{0.2cm}each \hspace{0.2cm}x\in X_2\)
Completion of training with the fixed correction training rule for any initial weight vector and any correction
increment constant leads to the following weights:
w∗=wk0=wk0+1=wk0+2.....
with w∗ as the solution vector for equation.
IV [Link] VII Sem. Deep Learning Notes Unit-II 7

Integer k0 is the training step number starting at which no more misclassification occurs, and thus no right
adjustments take place for (k_0>=0)
This theorem is called as the "Perceptron Convergence Theorem".
Perceptron Convergence theorem states that a classifier for two linearly separable classes of patterns is
always trainable in a finite number of training steps.
In summary, the training of a single discrete perceptron two class classifier requires a change of weights if
and only if a misclassification occurs.
In the reason for misclassification is (w^tx<0\) then all weights are increased in proportion wo xi.
If \(w^tx>0) then all weights are decreased in proportion to xi
Summary of the Perceptron Convergence Algorithm:
Variables and Parameters: x(n)=(m+1) by 1 input vector
=[+1,x1(n),x2(n),.....xm(n)]T
w(n)=(m+1)= by 1 weight vector
=[b(n),w1(n),w2(n),.....wm(n)]T
b(n)= bias
y(n)= actual response
d(n)= desired response
η= learning rate parameter, a +ve constant less than unity
1. Initialization: Set w(0)=0 , then perform the following computations for time step n=1,2
2. Activation: At time step n, activate the perceptron by applying input vector x(n) and desired response
d(n).
3. Computation of actual response: Compute the actual response of the perceptron:
y(n)=sgn[wT(x)x(n)]
4. Adaptation of weight vector: Update the weight vector of the perceptron:
w(n+1)=w(n)+η[d(n)−y(n)]x(n)
5. Continuation: Increment time step n by 1, go to step 1

[Link] about Linear seperability?

Linear Separability: Linear separability is an important concept in machine learning, particularly in the
field of supervised learning. It refers to the ability of a set of data points to be separated into distinct
categories using a linear decision boundary. In other words, if there exists a straight line that can cleanly
divide the data into two classes, then the data is said to be linearly separable.
Linear separability is a concept in machine learning that refers to the ability to separate data points in binary
classification problems using a linear decision boundary. If the data points can be separated using a line,
linear function, or flat hyperplane, they are considered linearly separable. Linear separability is an important
concept in neural networks, and it is introduced in the context of linear algebra and optimization theory.
In the context of machine learning, linear separability is an important property because it makes
classification problems easier to solve. If the data is linearly separable, we can use a linear classifier, such as
logistic regression or support vector machines (SVMs), to accurately classify new instances of data.
Linearly separable data points can be separated using a line, linear function, or flat hyperplane. In practice,
there are several methods to determine whether data is linearly separable. One method is linear
programming, which defines an objective function subjected to constraints that satisfy linear separability.
Another method is to train and test on the same data - if there is a line that separates the data points, then the
IV [Link] VII Sem. Deep Learning Notes Unit-II 8

accuracy or AUC should be close to 100%. If there is no such line, then training and testing on the same data
will result in at least some error. Multi-layer neural networks can learn hidden features and patterns in data
that linear classifiers cannot
To understand the concept of linear separability, it is helpful to first consider a simple two-dimensional
example. Imagine we have a set of data points in a two-dimensional space, where each point is labeled either
"red" or "blue". If these data points can be separated by a straight line, such that all the red points are on one
side of the line and all the blue points are on the other side, then the data is linearly separable.
Python provides several methods to determine whether data is linearly separable. One method is linear
programming, which defines an objective function subjected to constraints that satisfy linear separability.
Another method is clustering, where if two clusters with cluster purity of 100% can be found using some
clustering methods such as k-means, then the data is linearly separable.
However, not all data sets are linearly separable. In some cases, it may be impossible to draw a straight line
that can separate the data into distinct categories. For example, imagine a set of data points that are arranged
in a circular pattern, with red and blue points interspersed throughout. In this case, it is impossible to draw a
straight line that separates the data into two classes.
When faced with data that is not linearly separable, machine learning algorithms must use more complex
decision boundaries to accurately classify the data. For example, a decision tree or a neural network may be
able to accurately classify data that is not linearly separable.
Linear separability is not only important in the context of machine learning, but it also has applications in
other fields such as physics, biology, and economics. For example, in physics, linear separability can be
used to analyze the relationship between two physical quantities. In biology, it can be used to study the
behavior of animals or to analyze genetic data. In economics, it can be used to analyze the relationship
between two economic variables.
Example:
One way to test for linear separability is to use linear programming. Linear programming defines an
objective function subject to constraints that satisfy linear separability. The [Link]() function
in Python can be used to solve linear programming problems. Here's an example of using
[Link]() to test for linear separability:

import numpy as np
from [Link] import linprog
# Define the data points
X = [Link]([[1, 2], [2, 3], [3, 1], [4, 3]])
y = [Link]([1, 1, -1, -1])
# Define the objective function and constraints
c = [Link]([Link][1] + 1)
c[-1] = 1
A = [Link](([Link], [Link][1] + 1))
A[:, :-1] = -y[:, [Link]] * X
A[:, -1] = -y
b = -[Link]([Link])
# Solve the linear programming problem
res = linprog(c, A_ub=A, b_ub=b)
if [Link]:
print("The data is linearly separable.")
else:
IV [Link] VII Sem. Deep Learning Notes Unit-II 9

print("The data is not linearly separable.")

[Link] about Linear & Non-Linear Classification?

Introduction: Linear Classification refers to categorizing a set of data points to a discrete class based on
a linear combination of its explanatory variables. On the other hand, Non-Linear Classification refers to
separating those instances that are not linearly separable.
Linear Classification:
Linear Classification refers to categorizing a set of data points into a discrete class based on a linear
combination of its explanatory variables. Some of the classifiers that use linear functions to separate classes
are Linear Discriminate Classifier, Naive Bayes, Logistic Regression, Perceptron, SVM (linear kernel).

In the figure above, we have two classes, namely 'O' and '+.' To differentiate between the two classes, an
arbitrary line is drawn, ensuring that both the classes are on distinct sides. Since we can tell one class apart
from the other, these classes are called ‘linearly-separable.’ However, an infinite number of lines can be
drawn to distinguish the two classes. The exact location of this plane/hyperplane depends on the type of the
linear classifier.

Linear Discriminate Classifier

→ It is a dimensionality reduction technique in the domain of Supervised Machine Learning.
→ It is crucial in modeling differences between two groups, i.e., classes.
→ It helps project features in a high dimensions space in a lower-dimensional space.
→ Technique - Linear Discriminate Analysis (LDA) is used, which reduced the 2D graph into a 1D graph
by creating a new axis. This helps to maximize the distance between the two classes for differentiation.

X
In the above graph, we notice that a new axis is created, which maximizes the distance between the mean of
the two classes. As a result, variation within each class is also minimized. However, the problem with LDA
is that it would fail in case the means of both the classes are the same. This would mean that we would not
be able to generate a new axis for differentiating the two.
Non-Linear Classification:
Non-Linear Classification refers to categorizing those instances that are not linearly separable. Some of the
classifiers that use non-linear functions to separate classes are Quadratic Discriminant Classifier, Multi-
Layer Perceptron (MLP), Decision Trees, Random Forest, and K-Nearest Neighbours (KNN).
IV [Link] VII Sem. Deep Learning Notes Unit-II 10

In the figure above, we have two classes, namely 'O' and 'X.' To differentiate between the two classes, it is
impossible to draw an arbitrary straight line to ensure that both the classes are on distinct sides. We notice
that even if we draw a straight line, there would be points of the first-class present between the data points of
the second class. In such cases, piece-wise linear or non-linear classification boundaries are required to
distinguish the two classes.

7. Discuss about Perceptron and its models?

Perceptron: Perceptron is Machine Learning algorithm for supervised learning of various binary
classification tasks. Further, Perceptron is also understood as an Artificial Neuron or neural network unit
that helps to detect certain input data computations in business intelligence.
Perceptron model is also treated as one of the best and simplest types of Artificial Neural networks.
However, it is a supervised learning algorithm of binary classifiers. Hence, we can consider it as a single-
layer neural network with four main parameters, i.e., input values, weights and Bias, net sum, and an
activation function.
Types of Perceptron Models: Based on the layers, Perceptron models are divided into two types. These are
as follows:
1. Single-layer Perceptron Model
2. Multi-layer Perceptron model
Single Layer Perceptron Model:
This is one of the easiest Artificial neural networks (ANN) types. A single-layered perceptron model
consists feed-forward network and also includes a threshold transfer function inside the model. The main
objective of the single-layer perceptron model is to analyze the linearly separable objects with binary
outcomes.
In a single layer perceptron model, its algorithms do not contain recorded data, so it begins with inconstantly
allocated input for weight parameters. Further, it sums up all inputs (weight). After adding all inputs, if the
total sum of all inputs is more than a pre-determined value, the model gets activated and shows the output
value as +1.
If the outcome is same as pre-determined or threshold value, then the performance of this model is stated as
satisfied, and weight demand does not change. However, this model consists of a few discrepancies
triggered when multiple weight inputs values are fed into the model. Hence, to find desired output and
minimize errors, some changes should be necessary for the weights input.
"Single-layer perceptron can learn only linearly separable patterns."
Multi-Layered Perceptron Model:
Like a single-layer perceptron model, a multi-layer perceptron model also has the same model structure but
has a greater number of hidden layers.
The multi-layer perceptron model is also known as the Back propagation algorithm, which executes in two
stages as follows:
IV [Link] VII Sem. Deep Learning Notes Unit-II 11

o Forward Stage: Activation functions start from the input layer in the forward stage and terminate on
the output layer.
o Backward Stage: In the backward stage, weight and bias values are modified as per the model's
requirement. In this stage, the error between actual output and demanded originated backward on the
output layer and ended on the input layer.
Hence, a multi-layered perceptron model has considered as multiple artificial neural networks having
various layers in which activation function does not remain linear, similar to a single layer perceptron
model. Instead of linear, activation function can be executed as sigmoid, TanH, ReLU, etc., for deployment.
A multi-layer perceptron model has greater processing power and can process linear and non-linear patterns.
Further, it can also implement logic gates such as AND, OR, XOR, NAND, NOT, XNOR, NOR.
Multi-layer Perceptron in TensorFlow: Multi-Layer perceptron defines the most complex architecture of
artificial neural networks. It is substantially formed from multiple layers of the perceptron. TensorFlow is a
very popular deep learning framework released by, and this notebook will guide to build a neural network
with this library. If we want to understand what is a Multi-layer perceptron, we have to develop a multi-layer
perceptron from scratch using Numpy.
The pictorial representation of multi-layer perceptron learning is as shown below-

8. Explain about Back Propagation?

Backpropagation: is the essence of neural network training. It is the method of fine-tuning the weights of a
neural network based on the error rate obtained in the previous epoch (i.e., iteration). Proper tuning of the
weights allows you to reduce error rates and make the model reliable by increasing its generalization.
Backpropagation in neural network is a short form for “backward propagation of errors.” It is a standard
method of training artificial neural networks. This method helps calculate the gradient of a loss function
with respect to all the weights in the network.
How Backpropagation Algorithm Works
The Back propagation algorithm in neural network computes the gradient of the loss function for a single
weight by the chain rule. It efficiently computes one layer at a time, unlike a native direct computation. It
computes the gradient, but it does not define how the gradient is used. It generalizes the computation in the
delta rule.
Consider the following Back propagation neural network example diagram to understand:

How Backpropagation Algorithm Works

IV [Link] VII Sem. Deep Learning Notes Unit-II 12

1. Inputs X, arrive through the preconnected path

2. Input is modeled using real weights W. The weights are usually randomly selected.
3. Calculate the output for every neuron from the input layer, to the hidden layers, to the output layer.
4. Calculate the error in the outputs
ErrorB= Actual Output – Desired Output
5. Travel back from the output layer to the hidden layer to adjust the weights such that the error is
decreased.
Keep repeating the process until the desired output is achieved
Why We Need Backpropagation?
Most prominent advantages of Backpropagation are:
 Backpropagation is fast, simple and easy to program
 It has no parameters to tune apart from the numbers of input
 It is a flexible method as it does not require prior knowledge about the network
 It is a standard method that generally works well
 It does not need any special mention of the features of the function to be learned.
What is a Feed Forward Network?
A feedforward neural network is an artificial neural network where the nodes never form a cycle. This kind
of neural network has an input layer, hidden layers, and an output layer. It is the first and simplest type of
artificial neural network.
Types of Backpropagation Networks
Two Types of Backpropagation Networks are:
 Static Back-propagation
 Recurrent Backpropagation
Static back-propagation:
It is one kind of backpropagation network which produces a mapping of a static input for static output. It is
useful to solve static classification issues like optical character recognition.
Recurrent Backpropagation:
Recurrent Back propagation in data mining is fed forward until a fixed value is achieved. After that, the
error is computed and propagated backward.
The main difference between both of these methods is: that the mapping is rapid in static back-propagation
while it is nonstatic in recurrent backpropagation.