St.

Joseph’s Institute of Technology 4653 – Deep Learning

UNIT – I – DEEP LEARNING – IT4653

1
St. Joseph’s Institute of Technology 4653 – Deep Learning

UNIT I BASICS OF NEURAL NETWORKS

Basic Concept of Neurons – Perceptron Algorithm - Linear Classifiers and Gradient Descent– Feed
Forward and Backpropagation Networks.

1 Definition of Machine Learning [ML]:

Well posed learning problem: "A computer program is said to learn from experience E with
respect to some class of tasks T and performance measure P, if its performance at tasks in T, as
measured by P, improves with experience E."(Tom Michel)
Machine Learning (ML) is an algorithm that works on consequently through experience and
by the utilization of information. It is viewed as a piece of AI. ML calculations assemble a model
dependent on example information (Data), known as "training Data or information", to settle on
forecasts or choices without being unequivocally customized to do as such. AI calculations are
utilized in a wide assortment of utilizations, for example, in medication, email sifting, discourse
acknowledgment, and Computer vision, where it is troublesome or impractical to foster customary
models to play out the needed tasks.
ML includes PCs finding how they can perform tasks without being expressly modified to
do as such. It includes Systems that can perform tasks without being expressly modified to do as
such. It includes models gaining data so they can do certain specific applications. For basic
undertakings appointed to Models, it is feasible to models advising the machine how to execute all
means needed to tackle the current issue; on the systems part, no learning is required. For further
developed undertakings, it tends to be trying for a human to physically make the required
calculations. For more advanced tasks, it can be challenging for a human to manually create the
needed algorithms. In practice, it can turn out to be more effective to help the machine develop its
own algorithm, rather than having human programmers specify every needed step.
The term machine learning was coined in 1959 by Arthur Samuel, an American IBMer
and pioneer in the field of computer gaming and artificial intelligence.

a. Fundamentals of ANN
Neural computing is an information processing paradigm, inspired by biological system,
composed of a large number of highly interconnected processing elements(neurons) working in
unison to solve specific problems.
Artificial neural networks (ANNs), like people, learn by example. An ANN is configured
for a specific application, such as pattern recognition or data classification, through a learning
process. Learning in biological systems involves adjustments to the synaptic connections that exist
between the neurons. This is true of ANNs as well.
b. The Biological Neuron
The human brain consists of a large number, more than a billion of neural cells that process
information. Each cell works like a simple processor. The massive interaction between all cells and

2
St. Joseph’s Institute of Technology 4653 – Deep Learning
their parallel processing only makes the brain’s abilities possible. Figure 1 represents a human
biological nervous unit. Various parts of biological neural network(BNN) is marked in Figure 1.

Figure 1: Biological Neural Network

Dendrites are branching fibres that extend from the cell body or soma.
Soma or cell body of a neuron contains the nucleus and other structures, support chemical
processing and production of neurotransmitters.
Axon is a singular fiber carries information away from the soma to the synaptic sites of
other neurons (dendrites ans somas), muscels, or glands.
Axon hillock is the site of summation for incoming information. At any moment, the
collective influence of all neurons that conduct impulses to a given neuron will determine whether
or n ot an action potential will be initiated at the axon hillock and propagated along the axon.
Myelin sheath consists of fat-containing cells that insulate the axon from electrical activity.
This insulation acts to increase the rate of transmission of signals. A gap exists between each
myelin sheath cell along the axon. Since fat inhibits the propagation of electricity, the signals jump
from one gap to the next.
Nodes of Ranvier are the gaps (about 1 μm) between myelin sheath cells. Since fat serves as
a good insulator, the myelin sheaths speed the rate of transmission of an electrical impulse along
the axon.
Synapse is the point of connection between two neurons or a neuron and a muscle or a gland.
Electrochemical communication between neurons take place at these junctions.
Terminal buttons of a neuron are the small knobs at the end of an axon that release
chemicals called neurotransmitters.
3
St. Joseph’s Institute of Technology 4653 – Deep Learning
Information flow in a neural cell
The input/output and the propagation of information are shown below.
1.3. Artificial neuron model
An artificial neuron is a mathematical function conceived as a simple model of a real (biological)
neuron.
 The McCulloch-Pitts Neuron
This is a simplified model of real neurons, known as a Threshold Logic Unit.
 A set of input connections brings in activations from other neuron.
 A processing unit sums the inputs, and then applies a non-linear activation function (i.e.
squashing/transfer/threshold function).
 An output line transmits the result to other neurons.
1.3.1 Basic Elements of ANN:
Neuron consists of three basic components –weights, thresholds and a single activation
function. An Artificial neural network(ANN) model based on the biological neural sytems is shown
in figure 2.

Figure 2: Basic Elements of Artificial Neural Network

1.4 Different Learning Rules

A brief classification of Different Learning algorithms is depicted in figure 3.
 Training: It is the process in which the network is taught to change its weight
and bias.
 Learning: It is the internal process of training where the artificial neural
system learns to update/adapt the weights and biases.
4
St. Joseph’s Institute of Technology 4653 – Deep Learning
Different Training /Learning procedure available in ANN are
 Supervised learning
 Unsupervised learning
 Reinforced learning
 Hebbian learning
 Gradient descent learning
 Competitive learning
 Stochastic learning
1.4.1. Requirements of Learning Laws:
• Learning Law should lead to convergence of weights
• Learning or training time should be less for capturing the information from the
training pairs
• Learning should use the local information
• Learning process should able to capture the complex non linear mapping available
between the input & output pairs
• Learning should able to capture as many as patterns as possible
• Storage of pattern information's gathered at the time of learning should be high for
the given network

Figure 3: Different Training methods of Artificial Neural Network

[Link]. Supervised learning :

5
St. Joseph’s Institute of Technology 4653 – Deep Learning
Every input pattern that is used to train the network is associated with an output pattern which is
the target or the desired pattern.
A teacher is assumed to be present during the training process, when a comparison is made
between the network’s computed output and the correct expected output, to determine the [Link]
error can then be used to change network parameters, which result in an improvement in
performance.
[Link] Unsupervised learning:
In this learning method the target output is not presented to the [Link] is as if there is no
teacher to present the desired patterns and hence the system learns of its own by discovering and
adapting to structural features in the input patterns.
[Link] Reinforced learning:
In this method, a teacher though available, doesnot present the expected answer but only
indicates if the computed output correct or [Link] information provided helps the network in
the learning process.
[Link] Hebbian learning:

learning mechanism inspired by [Link] this, the input-output pattern pairs (𝑥𝑖, 𝑦𝑖) are
This rule was proposed by Hebb and is based on correlative weight [Link] is the oldest

associated by the weight matrix W, known as the correlation matrix.

It is computed as
𝑛
𝑖=
1
𝑥𝑖𝑦𝑖
∑
W= ------------eq(1)
𝑇

Here 𝑦𝑖𝑇 is the transposeof the associated output vector 𝑦𝑖.Numerous variants of the rule have
been proposed.
[Link] Gradient descent learning:
This is based on the minimization of error E defined in terms of weights and activation function
of the [Link] it is required that the activation function employed by the network is
differentiable, as the weight update is dependent on the gradient of the error E.
Thus if ∆𝑤𝑖𝑗 is the weight update of the link connecting the 𝑖𝑡ℎ and 𝑗𝑡ℎ neuron of the two
neighbouring layers, then ∆𝑤𝑖𝑗 is defined as,

∆𝑤𝑖𝑗 =
𝛛𝐸ɳ
-----------eq(2)
𝛛𝑤
𝑖𝑗

𝛛𝐸
Where, ɳ is the learning rate parameter and
𝛛𝑤𝑖𝑗
weight 𝑤𝑖𝑗.
is the error gradient with reference to the

1.5 Perceptron Model

6
St. Joseph’s Institute of Technology 4653 – Deep Learning
1.5.1 Simple Perceptron for Pattern Classification
Perceptron network is capable of performing pattern classification into two or more
categories. The perceptron is trained using the perceptron learning rule. We will first consider
classification into two categories and then the general multiclass classification later. For
classification

7
St. Joseph’s Institute of Technology 4653 – Deep Learning
into only two categories, all we need is a single output neuron. Here we will use bipolar neurons.
The simplest architecture that could do the job consists of a layer of N input neurons, an output
layer with a single output neuron, and no hidden layers. This is the same architecture as we saw
before for Hebb learning. However, we will use a different transfer function here for the output
neurons as given below in eq (7). Figure 7 represents a single layer perceptron network.

eq (7)

Figure 4: Single Layer Perceptron

Equation 7 gives the bipolar activation function which is the most common function used in
the perceptron networks. Figure 7 represents a single layer perceptron network. The inputs arising
from the problem space are collected by the sensors and they are fed to the aswociation
[Link] units are the units which are responsible to associate the inputs based on their
similarities. This unit groups the similar inputs hence the name association unit. A single input
from each group is given to the summing [Link] are randomnly fixed intially and assigned to
this inputs. The net value is calculate by using the expression
x = Σ wiai – θ eq(8)
This value is given to the activation function unit to get the final output [Link] actual
output is compared with the Target or desired .If they are same then we can stop training else the
weights haqs to be updated .It means there is error .Error is given as δ = b-s , where b is the desired

8
St. Joseph’s Institute of Technology 4653 – Deep Learning
/ Target output and S is the actual outcome of the machinehere the weights are updated based on the
perceptron Learning law as given in equation 9.
Weight change is given as Δw= η δ ai. So new weight is given as
Wi (new) = Wi (old) + Change in weight vector (Δw) eq(9)
1.5.2. Perceptron Algorithm
Step 1: Initialize weights and [Link] simplicity, set weights and bias to [Link] learning
rate in the range of zero to one.
• Step 2: While stopping condition is false do steps 2-6
• Step 3: For each training pair s:t do steps 3-5
• Step 4: Set activations of input units xi = ai
• Step 5: Calculate the summing part value Net = Σ aiwi-θ
• Step 6: Compute the response of output unit based on the activation functions
• Step 7: Update weights and bias if an error occurred for this pattern(if yis not equal to t)
Weight (new) = wi(old) + atxi , & bias (new) = b(old) + at
Else wi(new) = wi(old) & b(new) = b(old)
• Step 8: Test Stopping Condition
1.5.3. Limitations of single layer perceptrons:
• Uses only Binary Activation function
• Can be used only for Linear Networks
• Since uses Supervised Learning ,Optimal Solution is provided
• Training Time is More
• Cannot solve Linear In-separable Problem

1.5.4. Multi-Layer Perceptron Model:

Figure 8 is the general representation of Multi layer Perceptron [Link] the
input and output Layer there will be some more layers also known as Hidden layers.

9
St. Joseph’s Institute of Technology 4653 – Deep Learning

Figure 5: Multi-Layer Perceptron

1.5.5. Multi Layer Perceptron Algorithm

1. Initialize the weights (Wi) & Bias (B0) to small random values near Zero
2. Set learning rate η or α in the range of “0” to “1”
3. Check for stop condition. If stop condition is false do steps 3 to 7
4. For each Training pairs do step 4 to 7
5. Set activations of Output units: xi = si for i=1 to N
6. Calculate the output Response
yin = b0 + Σ xiwi
7. Activation function used is Bipolar sigmoidal or Bipolar Step functions
For Multi Layer networks, based on the number of layers steps 6 & 7 are repeated
8. If the Targets is (not equal to) = to the actual output (Y), then update weights and bias
based on Perceptron Learning Law
Wi (new) = Wi (old) + Change in weight vector
Change in weight vector = ηtixi
Where η = Learning Rate
ti = Target output of ith unit
xi = ith Input vector
b0(new) = b0 (old) + Change in Bias
Change in Bias = ηti
Else Wi (new) = Wi (old)
b0(new) = b0 (old)
10
St. Joseph’s Institute of Technology 4653 – Deep Learning

9. Test for Stop condition

11
St. Joseph’s Institute of Technology 4653 – Deep Learning
1.6. linearly seperable & Linear in separable tasks:

Figure 6: Representation of Linear seperable & Linear-in separable Tasks

Perceptron are successful only on problems with a linearly separable solution [Link] 9
represents both linear separable as well as linear in seperable [Link] cannot handle, in
particular, tasks which are not linearly separable.(Known as linear inseparable problem).Sets of
points in two dimensional spaces are linearly separable if the sets can be seperated by a straight
[Link], a set of points in n-dimentional space are that can be seperated by a straight
[Link] called Linear seperable as represented in figure 9.
Single layer perceptron can be used for linear [Link] AND [Link] it cant be
used for non linear ,inseparable problems.(Example XOR Gate).Consider figure 10.

Figure 7: XOR representation (Linear-in separable Task)

Here a single decision line cannot separate the Zeros and Ones [Link] least Two lines
are required to separate Zeros and Onesas shown in Figure 10. Hence single layer networks can not
be used to solve inseparable problems. To over come this problem we go for creation of convex
regions.

12
St. Joseph’s Institute of Technology 4653 – Deep Learning
Convex regions can be created by multiple decision lines arising from multi layer
[Link] layer network cannot be used to solve inseparable [Link] we go for
multilayer network there by creating convex regions which solves the inseparable problem.
1.6.1 Convex Region:
Select any Two points in a region and draw a straight line between these two points. If the
points selected and the lines joining them both lie inside the region then that region is known as
convex regions.
1.6.2. Types of convex regions
(a) Open Convex region (b) Closed Convex region

Figure 8: Open convex region

Figure 9 A: Circle - Closed convex region Figure 9 B: Triangle - Closed convex region
1.7. Logistic Regression
Logistic regression is a probabilistic model that organizes the instances in terms of
probabilities. Because the classification is probabilistic, a natural method for optimizing the
parameters is to ensure that the predicted probability of the observed class for each training
occurrence is as large as possible. This goal is achieved by using the notion of maximumlikelihood
estimation in order to learn the parameters of the model. The likelihood of the training data is
defined as the product of the probabilities of the observed labels of each training instance. Clearly,
larger values of this objective function are better. By using the negative logarithm of this value, one
obtains a loss function in minimization form. Therefore, the output node uses the negative log-
likelihood as a loss function. This loss function replaces the squared error used in the Widrow-Hoff
method. The output layer can be formulated with the sigmoid activation function, which is very
common in neural network design.
13
St. Joseph’s Institute of Technology 4653 – Deep Learning
 Logistic regression is another supervised learning algorithm which is
used to solve the classification problems. In classification problems, we
have dependent variables in a binary or discrete format such as 0 or 1.

 Logistic regression algorithm works with the categorical variable such as

0 or 1, Yes or No, True or False, Spam or not spam, etc.

 It is a predictive analysis algorithm which works on the concept of probability.

 Logistic regression is a type of regression, but it is different from the

linear regression algorithm in the term how they are used.

 Logistic regression uses sigmoid function or logistic function which is a

complex cost function. This sigmoid function is used to model the data in
logistic regression. The function can be represented as:

Where f(x)= Output between the 0 and 1 value.

x= input to the function
e= base of natural logarithm.
When we provide the input values (data) to the function, it gives the S-
curve as follows: It uses the concept of threshold levels, values above the
threshold level are rounded up to 1, and values below the threshold level are
rounded up to 0.

Figure 10: Circle – Logistic Function

14
St. Joseph’s Institute of Technology 4653 – Deep Learning
1.8. Support Vector Machines
Support Vector Machine or SVM is one of the most popular Supervised Learning
algorithms, which is used for Classification as well as Regression problems. However,
primarily, it is used for Classification problems in Machine Learning. The goal of the SVM
algorithm is to create the best line or decision boundary that can segregate n-dimensional space
into classes so that we can easily put the new data point in the correct category in the future.
This best decision boundary is called a hyperplane. SVM chooses the extreme points/vectors
that help in creating the hyperplane. These extreme cases are called as support vectors, and
hence algorithm is termed as Support Vector Machine. Consider the below diagram in which
there are two different categories that are classified using a decision boundary or hyperplane :

Figure 11: SVM – Classification

1.8.1. SVM can be of two types:

Linear SVM: Linear SVM is used for linearly separable data, which means if a dataset can be
classified into two classes by using a single straight line, then such data is termed as linearly
separable data, and classifier is used called as Linear SVM classifier.

Non-linear SVM: Non-Linear SVM is used for non-linearly separated data, which means if a
dataset cannot be classified by using a straight line, then such data is termed as non-linear data
and classifier used is called as Non-linear SVM classifier
Support Vectors:
The data points or vectors that are the closest to the hyperplane and which affect the position of
the hyperplane are termed as Support Vector. Since these vectors support the hyperplane, hence
called a Support vector.

15
St. Joseph’s Institute of Technology 4653 – Deep Learning

1.8.2. Linear SVM:

The working of the SVM algorithm can be understood by
using an example. Suppose we have a dataset that has two tags
(green and blue), and the dataset has two features x1 and x2. We
want a classifier that can classify the pair (x1, x2) of coordinates
in either green or blue. Consider the below image figure11. It is
2-d space so by just using a straight line, we can easily separate
these two classes. But there can be multiple lines that can
separate these classes. Consider the below image:

Figure 12A: SVM – Input Space

Hence, the SVM algorithm helps to find the best line or
decision boundary; this best boundary or region is called as a
hyperplane. SVM algorithm finds the closest point of the lines
from both the classes. These points are called support vectors.
The distance between the vectors and the hyperplane is called as
margin. And the goal of SVM is to maximize this margin. The
hyperplane with maximum margin is called the optimal
hyperplane.

Figure 12B: SVM – Linear Classification

16
 St. Joseph’s Institute of Technology 4653 – Deep Learning

1.9 Gradient Descent

Gradient Descent is known as one of the most commonly used optimization algorithms to train
machine learning models by means of minimizing errors between actual and expected results.
Further, gradient descent is also used to train Neural Networks.
In mathematical terminology, Optimization algorithm refers to the task of minimizing/maximizing an
objective function f(x) parameterized by x. Similarly, in machine learning, optimization is the task of
minimizing the cost function parameterized by the model's parameters. The main objective of
gradient descent is to minimize the convex function using iteration of parameter updates. Once these
machine learning models are optimized, these models can be used as powerful tools for Artificial
Intelligence and various computer science applications.
In this tutorial on Gradient Descent in Machine Learning, we will learn in detail about gradient
descent, the role of cost functions specifically as a barometer within Machine Learning, types of
gradient descents, learning rates, etc.
What is Gradient Descent or Steepest Descent?
Gradient descent was initially discovered by "Augustin-Louis Cauchy" in mid of 18th
century. Gradient Descent is defined as one of the most commonly used iterative optimization
algorithms of machine learning to train the machine learning and deep learning models. It helps
in finding the local minimum of a function.
The best way to define the local minimum or local maximum of a function using gradient descent is
as follows:
o If we move towards a negative gradient or away from the gradient of the function at the
current point, it will give the local minimum of that function.
o Whenever we move towards a positive gradient or towards the gradient of the function at the
current point, we will get the local maximum of that function.

17
 St. Joseph’s Institute of Technology 4653 – Deep Learning

This entire procedure is known as Gradient Ascent, which is also known as steepest descent. The
main objective of using a gradient descent algorithm is to minimize the cost function using
iteration. To achieve this goal, it performs two steps iteratively:
o Calculates the first-order derivative of the function to compute the gradient or slope of that
function.
o Move away from the direction of the gradient, which means slope increased from the current
point by alpha times, where Alpha is defined as Learning Rate. It is a tuning parameter in the
optimization process which helps to decide the length of the steps.
What is Cost-function?
The cost function is defined as the measurement of difference or error between actual values and
expected values at the current position and present in the form of a single real number. It helps to
increase and improve machine learning efficiency by providing feedback to this model so that it can
minimize error and find the local or global minimum. Further, it continuously iterates along the
direction of the negative gradient until the cost function approaches zero. At this steepest descent
point, the model will stop learning further. Although cost function and loss function are considered
synonymous, also there is a minor difference between them. The slight difference between the loss
function and the cost function is about the error within the training of machine learning models, as
loss function refers to the error of one training example, while a cost function calculates the average
error across an entire training set.
The cost function is calculated after making a hypothesis with initial parameters and modifying these
parameters using gradient descent algorithms over known data to reduce the cost function.
How does Gradient Descent work?
Before starting the working principle of gradient descent, we should know some basic concepts to
find out the slope of a line from linear regression. The equation for simple linear regression is given
as:
1. Y=mX+c
Where 'm' represents the slope of the line, and 'c' represents the intercepts on the y-axis.

18
 St. Joseph’s Institute of Technology 4653 – Deep Learning

The starting point(shown in above fig.) is used to evaluate the performance as it is considered just as
an arbitrary point. At this starting point, we will derive the first derivative or slope and then use a
tangent line to calculate the steepness of this slope. Further, this slope will inform the updates to the
parameters (weights and bias).
The slope becomes steeper at the starting point or arbitrary point, but whenever new parameters are
generated, then steepness gradually reduces, and at the lowest point, it approaches the lowest point,
which is called a point of convergence.
The main objective of gradient descent is to minimize the cost function or the error between expected
and actual. To minimize the cost function, two data points are required:
o Direction & Learning Rate
These two factors are used to determine the partial derivative calculation of future iteration and allow
it to the point of convergence or local minimum or global minimum. Let's discuss learning rate
factors in brief;

Learning Rate:
It is defined as the step size taken to reach the minimum or lowest point. This is typically a small
value that is evaluated and updated based on the behavior of the cost function. If the learning rate is
high, it results in larger steps but also leads to risks of overshooting the minimum. At the same time,
a low learning rate shows the small step sizes, which compromises overall efficiency but gives the
advantage of more precision.

19
 St. Joseph’s Institute of Technology 4653 – Deep Learning

Types of Gradient Descent

Based on the error in various training models, the Gradient Descent learning algorithm can be
divided into Batch gradient descent, stochastic gradient descent, and mini-batch gradient
descent. Let's understand these different types of gradient descent:
1. Batch Gradient Descent:
Batch gradient descent (BGD) is used to find the error for each point in the training set and update
the model after evaluating all training examples. This procedure is known as the training epoch. In
simple words, it is a greedy approach where we have to sum over all examples for each update.
Advantages of Batch gradient descent:
o It produces less noise in comparison to other gradient descent.
o It produces stable gradient descent convergence.
o It is Computationally efficient as all resources are used for all training samples.
2. Stochastic gradient descent
Stochastic gradient descent (SGD) is a type of gradient descent that runs one training example per
iteration. Or in other words, it processes a training epoch for each example within a dataset and
updates each training example's parameters one at a time. As it requires only one training example at
a time, hence it is easier to store in allocated memory. However, it shows some computational
efficiency losses in comparison to batch gradient systems as it shows frequent updates that require
more detail and speed. Further, due to frequent updates, it is also treated as a noisy gradient.
However, sometimes it can be helpful in finding the global minimum and also escaping the local
minimum.
Advantages of Stochastic gradient descent:
In Stochastic gradient descent (SGD), learning happens on every example, and it consists of a few
advantages over other gradient descent.
o It is easier to allocate in desired memory.
o It is relatively fast to compute than batch gradient descent.
20
 St. Joseph’s Institute of Technology 4653 – Deep Learning

o It is more efficient for large datasets.

3. MiniBatch Gradient Descent:
Mini Batch gradient descent is the combination of both batch gradient descent and stochastic
gradient descent. It divides the training datasets into small batch sizes then performs the updates on
those batches separately. Splitting training datasets into smaller batches make a balance to maintain
the computational efficiency of batch gradient descent and speed of stochastic gradient descent.
Hence, we can achieve a special type of gradient descent with higher computational efficiency and
less noisy gradient descent.
Advantages of Mini Batch gradient descent:
o It is easier to fit in allocated memory.
o It is computationally efficient.
o It produces stable gradient descent convergence.
Challenges with the Gradient Descent
Although we know Gradient Descent is one of the most popular methods for optimization problems,
it still also has some challenges. There are a few challenges as follows:
1. Local Minima and Saddle Point:
For convex problems, gradient descent can find the global minimum easily, while for non-convex
problems, it is sometimes difficult to find the global minimum, where the machine learning models
achieve the best results.

Whenever the slope of the cost function is at zero or just close to zero, this model stops learning
further. Apart from the global minimum, there occur some scenarios that can show this slop, which is
saddle point and local minimum. Local minima generate the shape similar to the global minimum,
where the slope of the cost function increases on both sides of the current points.
21
 St. Joseph’s Institute of Technology 4653 – Deep Learning

In contrast, with saddle points, the negative gradient only occurs on one side of the point, which
reaches a local maximum on one side and a local minimum on the other side. The name of a saddle
point is taken by that of a horse's saddle.
The name of local minima is because the value of the loss function is minimum at that point in a
local region. In contrast, the name of the global minima is given so because the value of the loss
function is minimum there, globally across the entire domain the loss function.
2. Vanishing and Exploding Gradient
In a deep neural network, if the model is trained with gradient descent and backpropagation, there
can occur two more issues other than local minima and saddle point.
Vanishing Gradients:
Vanishing Gradient occurs when the gradient is smaller than expected. During backpropagation, this
gradient becomes smaller that causing the decrease in the learning rate of earlier layers than the later
layer of the network. Once this happens, the weight parameters update until they become
insignificant.
Exploding Gradient:
Exploding gradient is just opposite to the vanishing gradient as it occurs when the Gradient is too
large and creates a stable model. Further, in this scenario, model weight increases, and they will be
represented as NaN. This problem can be solved using the dimensionality reduction technique, which
helps to minimize complexity within the model.

22
 St. Joseph’s Institute of Technology 4653 – Deep Learning

FEED FORWARD NETWORK

BACKPROPAGATION:

23
 St. Joseph’s Institute of Technology 4653 – Deep Learning

ARCHITECTURE:

24
 St. Joseph’s Institute of Technology 4653 – Deep Learning

TRAINING ALGORITHM:

Steps for Training algorithm of back Propagation

25
St. Joseph’s Institute of Technology 4653 – Deep Learning

26