FOUNDATIONS OF DEEP

LEARNING
AIT 401
Reference Books
1. Goodfellow, I., Bengio,Y., and Courville, A., Deep Learning, MIT Press, 2016.
2. Neural Networks and Deep Learning, Aggarwal, Charu C., c Springer
International Publishing AG, part of Springer Nature 2018
3. Deep Learning, Core Conceps, Methods and Applications- M Gopal, Pearson
Education
[Link] of Deep Learning: Designing Next-Generation Machine
Intelligence Algorithms (1st. ed.). Nikhil Buduma and Nicholas Locascio. 2017.
O'Reilly Media, Inc.
Module 1: Introduction to Neural Networks and Deep learning
Introduction, The Basic Architecture of Neural Networks - Single
Computational Layer: The Perceptron, Multilayer Neural Networks.
Activation functions – Sign, Sigmoid, Tanh, ReLU, leaky ReLU, Hard Tanh,
Softmax. Loss function. Training a Neural Network with Backpropagation.
Practical issues in neural network training. Overfitting, Underfitting, Hyper
parameters and Validation sets,Estimators -Bias and [Link] to
deep learning, Deep feed forward network.
INTRODUCTION
 A machine learning model transforms its input data into a meaningful
output, a process which is "learned" from exposure to known examples
of inputs and outputs.
 Deep learning - is a subset of machine learning, which is essentially a neural
network with three or more layers. These neural networks attempt to simulate
the behavior of the human brain, allowing it to “learn” from large amounts of
data. While a neural network with a single layer can still make approximate
predictions, additional hidden layers can help to optimize and refine for
accuracy.
Machine Learning Basics
Machine Learning Basics
• Artificial Intelligence or AI, the broadest term of the three, is used to classify machines that mimic human
intelligence and human cognitive functions like problem-solving and learning. AI uses predictions and
automation to optimize and solve complex tasks that humans have historically done, such as facial and speech
recognition, decision-making and translation.
• Machine Learning is a branch of Artificial Intelligence, which focuses on methods that learn from data and make
predictions on unseen data

Machine Learning
Labeled Data algorithm

Training
Prediction

Learned
Labeled Data Prediction
model

Picture from: Ismini Lourentzou – Introduction to Deep Learning 6

7
8
ML vs. Deep Learning
Introduction to Deep Learning

• Deep learning (DL) is a machine learning subfield that uses multiple layers for learning data representations
 DL is exceptionally effective at learning patterns

Picture from: [Link] 9

ML vs. Deep Learning
Introduction to Deep Learning

• DL applies a multi-layer process for learning rich hierarchical features (i.e., data representations)
 Input image pixels → Edges → Textures → Parts → Objects

Low- Mid- High- Trainabl

Output
Level Level Level e
Features Features Features Classifie
r

Slide credit: Param Vir Singh – Deep Learning 10

Why is DL Useful?
Introduction to Deep Learning

• DL provides a flexible, learnable framework for representing visual, text, linguistic

information
 Can learn in supervised and unsupervised manner
• DL represents an effective end-to-end learning system
• Requires large amounts of training data
• Since about 2010, DL has outperformed other ML techniques
 First in vision and speech, then NLP, and other applications

13
 Neuron

•Neurons are the fundamental unit of the

nervous system specialized to transmit
information to different parts of the body
What is a Neuron?
• Neurons are the building blocks of the nervous system.
• They receive and transmit signals to different parts of the body.
• This is carried out in both physical and electrical forms.
• There are several different types of neurons that facilitate the transmission
of information.
• The sensory neurons carry information from the sensory receptor cells
present throughout the body to the brain.
• The motor neurons transmit information from the brain to the muscles.
• The interneurons transmit information between different neurons in the
body.
 Neuron Structure
• All neurons have three different parts – dendrites, cell body and
axon.

Parts of Neuron

• Following are the different parts of a neuron:

Dendrites
• These are branch-like structures that receive messages from other neurons
and allow the transmission of messages to the cell body.
Cell Body
• Each neuron has a cell body with a nucleus, Golgi body, endoplasmic
reticulum, mitochondria and other components.
Axon
• Axon is a tube-like structure that carries electrical impulse from the cell body
to the axon terminals that pass the impulse to another neuron.
• Synapse
It is the chemical junction between the terminal of one neuron and the
dendrites of another neuron.
 Neuron Functions
The important functions of a neuron are:

• Chemical Synapse
Chemical synapses use neurotransmitters to activate postsynaptic receptors
and deliver excitatory or inhibitory messages. The receptors amplify the
message. Chemical synapses allow unidirectional transmission by using
chemicals called neurotransmitters to send signals along the neurons.

• Electrical Synapse
When two neurons are connected by a gap junction, it results in an electrical
synapse. These gaps include ion channels that help in the direct transmission
of a positive electrical signal. These are much faster than chemical
synapses. It allows for transmission in both directions
 Artificial Neural Network
• An Artificial neural network is usually a computational network
based on biological neural networks that construct the structure of
the human brain.
• Similar to a human brain, it has neurons interconnected to each
other, artificial neural networks also have neurons that are linked to
each other in various layers of the networks.
• These neurons are known as nodes.

2
1
Dendrites from Biological Neural Network represent inputs in Artificial
Neural Networks, Cell nucleus represents Nodes, Synapse represents
Weights, and Axon represents Output.
Biological Neural Network Artificial Neural Network

Dendrites Inputs

Cell nucleus Nodes

Synapse Weights

Axon Output
 Artificial Neural Network
• Artificial Neural Network primarily consists of three layers:
Input Layer:
• As the name suggests, it accepts inputs in several different formats
provided by the programmer.
• It brings the initial data into the system for further processing by
subsequent layers of artificial neurons. (It is the only layer where no
computation happens i.e. no activation function is applied.)
Hidden Layer:
• The hidden layer presents in-between input and output layers. It performs
all the calculations to find hidden features and patterns.
Output Layer:
• The input goes through a series of transformations using the hidden layer,
which finally results in output that is conveyed using this layer.
Weights and Biases:

 Weights are numerical parameters which determine how strongly each of the
neurons affects the other.
 Bias is special neuron with the value 1. It is added to each pre-output layer.

Activation function:

 An activation function is a non-linear mathematical function which converts

the input values to an output.
 Without activation functions, the working of a neural network will be like linear

models.
• The artificial neural network takes input and computes the
weighted sum of the inputs and includes a bias. This
computation is represented in the form of a transfer function.

What is Artificial Neural Network

• It determines weighted total is passed as an input to an
activation function to produce the output. Activation
functions choose whether a node should fire or not. Only
those who are fired make it to the output layer. There are
distinctive activation functions available that can be applied
upon the sort of task we are performing.
 How do artificial neural networks work?
• Artificial Neural Network can be best represented as a
weighted directed graph, where the artificial neurons form
the nodes.
• The association between the neurons outputs and neuron
inputs can be viewed as the directed edges with weights.
• The Artificial Neural Network receives the input signal from
the external source in the form of a pattern and image in the
form of a vector. These inputs are then mathematically
assigned by the notations x(n) for every n number of inputs.
• The activation function refers to the set of transfer functions
used to achieve the desired output.
• There is a different kind of the activation function, but primarily
either linear or non-linear sets of functions.
• Some of the commonly used sets of activation functions are the
Binary, linear, and Tan hyperbolic, sigmoidal activation
functions.
• Afterward, each of the input is multiplied by its corresponding
weights ( these weights are the details utilized by the artificial
neural networks to solve a specific problem ).
• In general terms, these weights normally represent the strength
of the interconnection between neurons inside the artificial
neural network.
• All the weighted inputs are summarized inside the computing
unit.
• If the weighted sum is equal to zero, then bias is added to make
the output non-zero or something else to scale up to the
system's response.
• Bias has the same input, and weight equals to 1.
• Here the total of weighted inputs can be in the range of 0 to
positive infinity.
 Types of Artificial Neural Network:
Feedforward ANN

Its flow is uni-directional, meaning that the information in the model flows in
only one direction—forward—from the input nodes, through the hidden nodes
(if any) and to the output nodes, without any cycles or loops. FNNs are
primarily used for pattern recognition, classification, and regression tasks.
Recurrent Neural Network or Feedback ANN
 In this type of ANN, the output returns into the network to accomplish the

best-evolved results internally

 The feedback networks feed information back into itself and are well suited to

solve optimization issues.

 The Internal system error corrections utilize feedback ANN.

 RNNs are widely used for natural language processing, speech

recognition, and sentiment analysis tasks.
Advantages of Artificial Neural Network (ANN)

• Parallel processing capability:

ANNs can process information parallel across multiple neurons and
layers. This similar processing capability allows for efficient
computation and scalability, making ANNs suitable for handling large
and complex datasets
• Storing data on the entire network:
• Capability to work with incomplete knowledge:
• After ANN training, the information may produce output even with
inadequate data. The loss of performance here relies upon the
significance of missing data.
• Having a memory distribution:
• For ANN is to be able to adapt, it is important to determine the
examples and to encourage the network according to the desired
output by demonstrating these examples to the network.
• Having fault tolerance:
• Extortion of one or more cells of ANN does not prohibit it from
generating output, and this feature makes the network fault-
tolerance.
Disadvantages of Artificial Neural Network:
• Assurance of proper network structure:
• There is no particular guideline for determining the structure of artificial
neural networks. The appropriate network structure is accomplished
through experience, trial, and error.
• Unrecognized behavior of the network:
• It is the most significant issue of ANN. When ANN produces a testing
solution, it does not provide insight concerning why and how. It decreases
trust in the network.
• Hardware dependence:
• Artificial neural networks need processors with parallel processing
power, as per their structure. Therefore, the realization of the equipment is
dependent.
Difficulty of showing the issue to the network:
• ANNs can work with numerical data. Problems must be
converted into numerical values before being introduced to
ANN. The presentation mechanism to be resolved here will
directly impact the performance of the network. It relies on the
user's abilities.
• It needed high processing time for big neural networks.
The Basic Architecture of Neural
Networks
Single-layer and Multi-layer neural networks
• In the single layer network, a set of inputs is directly mapped to an
output using a generalized linear function. This simple instantiation
of a neural network is also referred to as the perceptron.
• In multi-layer neural networks, the neurons are arranged in a
layered fashion, in which the input and output layers are separated
by a group of hidden layers. This layer-wise architecture of the neural
network is also referred to as a feed-forward network.
 PERCEPTRON
• It is one of the oldest and first introduced neural networks.
• It was proposed by Frank Rosenblatt in 1958.
• Perceptron is also known as an artificial neural network.
• Perceptron is mainly used to compute the logical gate like AND, OR,
and NOR which has binary input and binary output.
• Perceptron is the basic building block of an Artificial Neural Network.
• Perceptron is a linear machine learning algorithm used for
supervised learning for various binary classifiers.
The main functionality of the perceptron is:-
• Takes input from the input layer
• Weight them up and sum it up.
• Pass the sum to the function to produce the output.
Input Nodes or Input Layer:
• This is the primary component of Perceptron which accepts the
initial data into the system for further processing. Each input node
contains a real numerical value.
Weight and Bias:
• Weight parameter represents the strength of the connection
between units.
• Weight is directly proportional to the strength of the associated
input neuron in deciding the output.
• Bias can be considered as the line of intercept in a linear equation.
Activation Function:
• These are the final and important components that help to
determine whether the neuron will fire or not. Activation
Function can be considered primarily as a step function.

Types of Activation functions:

• Sign function
• Step function
• Sigmoid function
How does Perceptron work?

• In Machine Learning, Perceptron is considered as a single-layer

neural network that consists of four main parameters named input
values (Input nodes), weights and Bias, net sum, and an activation
function.
• The perceptron model begins with the multiplication of all input
values and their weights, then adds these values together to create
the weighted sum.
• Then this weighted sum is applied to the activation function 'f' to
obtain the desired output. This activation function is also known as
the step function and is represented by 'f'.
Perceptron model works in two important steps as
follows:

Step-1
• In the first step first, multiply all input values with corresponding weight values
and then add them to determine the weighted sum. Mathematically, we can
calculate the weighted sum as follows:
∑wi*xi = x1*w1 + x2*w2 +…wn*xn
• Add a special term called bias 'b' to this weighted sum to improve the model's
performance.
∑wi*xi + b
Step-2
• In the second step, an activation function is applied with the above-mentioned
weighted sum, which gives us output either in binary form or a continuous value
as follows:
Y = f(∑wi*xi + b)
 Types of Perceptron Models

• Based on the layers, Perceptron models are divided into two types.

• These are as follows:

• Single-layer Perceptron Model

• Multi-layer Perceptron model
Single Layer Perceptron Model:
• This is one of the easiest Artificial neural networks (ANN) types.
• A single-layered perceptron model consists feed-forward network and also
includes a threshold transfer function inside the model.
• The main objective of the single-layer perceptron model is to analyze the
linearly separable objects with binary outcomes.
• In a single layer perceptron model, its algorithms do not contain recorded
data, so it begins with inconstantly allocated input for weight parameters.
• Further, it sums up all inputs (weight).
• After adding all inputs, if the total sum of all inputs is more than a pre-
determined value, the model gets activated and shows the output value as
+1.
• If the outcome is same as pre-determined or threshold value, then the
performance of this model is stated as satisfied, and weight demand does not
change.
• Hence, to find desired output and minimize errors, some changes should be
necessary for the weights input.
Single Computational Layer: The Perceptron

• This neural network contains a single input layer and an output node.
• The basic architecture of the perceptron is shown in Figure 1.3(a).

• Each training instance is of the form (X,y), where

each X= [x1,...xd] contains d feature variables,
and y€{−1, +1} contains the observed value of
the binary class variable.
• By “observed value” we refer to the fact that it is
given to us as a part of the training data, and our
goal is to predict the class variable for cases in
which it is not observed.
• The input layer contains d nodes that transmit the
d features X = [x1 ...xd] with edges of weight W =
[w1 ...wd] to an output node.
• The linear function is computed
at the output node
• There is an invariant part of the prediction, which is
referred to as the bias.
• The binary class distribution is highly imbalanced
• We need to incorporate an additional bias variable ‘b‘
that captures this invariant part of the prediction:

• The bias can be incorporated as the weight of an edge

by using a bias neuron.
• This is achieved by adding a neuron that always
transmits a value of 1 to the output node.
• The weight of the edge connecting the bias neuron to
the output node provides the bias variable.
"Single-layer perceptron can learn only linearly separable
patterns."
• One of the main disadvantages of using a single-layer perceptron is
its limited expressive power and generalization ability.
• It cannot learn to classify non-linearly separable patterns, such as
XOR, circles, or spirals.
• It is also prone to overfitting and noise, as it tries to fit a straight line
to the data.
• It does not have any hidden layers that can introduce non-linearity
and flexibility to the model.
Explain the limitation of single layer perceptron.

One of the main disadvantages of using a single-layer perceptron is its limited expressive
power and generalization ability.
It cannot learn to classify non-linearly separable patterns, such as XOR, circles, or spirals.
It is also prone to overfitting and noise, as it tries to fit a straight line to the data.
It does not have any hidden layers that can introduce non-linearity and flexibility to the
model.
A "single-layer" perceptron can't implement XOR. The reason is because the classes in
XOR are not linearly separable. You cannot draw a straight line to separate the points
(0,0),(1,1) from the points (0,1),(1,0).
Truth table of XOR
• XOR is where if one is 1 and other is 0 but not both.

• Need:
• 1.w1 + 0.w2 cause a fire, i.e. >= t
• 0.w1 + 1.w2 >= t
• 0.w1 + 0.w2 doesn't fire, i.e. < t
• 1.w1 + 1.w2 also doesn't fire, < t

• w1 >= t
• w2 >= t
• 0<t
• w1+w2 < t which is a contradiction.
• A "single-layer" perceptron can't implement XOR. The reason is because the classes in XOR are
not linearly separable. You cannot draw a straight line to separate the points (0,0),(1,1) from the
points (0,1),(1,0).
MULTI LAYER NEURAL NETWORK
• Multilayer neural networks contain more than one computational
layer.
• The additional intermediate layers (between input and output) are
referred to as hidden layers because the computations performed are
not visible to the user.
• The specific architecture of multilayer neural networks is referred to
as feed-forward networks, because successive layers feed into one
another in the forward direction from input to output.
• The default architecture of feed-forward networks assumes that all
nodes in one layer are connected to those of the next layer.
• Therefore, the architecture of the neural network is almost fully
defined, once the number of layers and the number/type of nodes in
each layer have been defined.
• The only remaining detail is the loss function that is optimized in the
output layer.
• The loss function is the function that computes the distance
between the current output of the algorithm and the
expected output. It’s a method to evaluate how your algorithm
models the data.
Examples of multilayer networks with or
without the bias neurons
• The number of units in each layer is referred to as the dimensionality
of that layer.
• Deep feedforward networks, also often called feedforward neural
networks, or multilayer perceptrons (MLPs).
• To be accurate a fully connected Multi-Layered Neural Network is
known as Multi-Layer Perceptron.
• A Multi-Layered Neural Network consists of multiple layers of artificial
neurons or nodes.
• A multi-layer perceptron model has greater processing power and can
process linear and non-linear patterns. Further, it can also implement
logic gates such as AND, OR, XOR, NAND, NOT, XNOR, NOR.
Formula for Multi-Layered Neural Network

• Suppose we have xm inputs(x1, x2….xm) and a bias unit. Let the weight applied
to be w1, w2…..wm. Then find the summation and bias unit on performing dot
product among inputs and weights as:

• On feeding the r into activation function F(r) we find the output for the hidden
layers. For the first hidden layer h1, the neuron can be calculated as:

• For all the other hidden layers repeat the same procedure. Keep repeating the
process until reach the last weight set.
Advantages of Multi-Layer Perceptron:

• A multi-layered perceptron model can be used to solve complex non-

linear problems.
• It works well with both small and large input data.
• It helps us to obtain quick predictions after the training.
• It helps to obtain the same accuracy ratio with large as well as small
data.
 ACTIVATION FUNCTIONS
• It’s a function that we use to get the output of node. It is also
known as Transfer Function.
• The primary role of the Activation Function is to transform the
summed weighted input from the node into an output value
to be fed to the next hidden layer or as output.
• If activation function activates the node, the value of the
node is used to find the next layer nodes.
• If not the output is no longer used as an input for the next
layer.
 In multilayer neural networks, activation functions bring non-linearity
into hidden layers, which increases the complexity of the model.
 A neural network with any number of layers but only linear activations
can be shown to be equivalent to a single-layer network.
 The Activation Functions can be basically divided into 2 types-
1)Linear Activation Function
2)Non-linear Activation Functions
Binary Step Function

 Binary step function depends on a threshold value that decides whether a

neuron should be activated or not.

 The input fed to the activation function is compared to a certain threshold; if the
input is greater than it, then the neuron is activated, else it is deactivated,
meaning that its output is not passed on to the next hidden layer.
Activation: Linear Function
Introduction to Neural Networks

• Linear function means that the output signal is proportional to the input signal to the neuron
ℝ𝑛 → ℝ𝑛
 If the value of the constant c is 1, it is also called
identity activation function
𝑓 𝑥 = 𝑐𝑥
 The linear activation function is the
simplest activation function, defined as:
f(x) = x

 This activation type is used in regression

problems
o E.g., the last layer can have linear activation
function, in order to output a real number
(and not a class membership)
o It doesn’t help with the complexity or
various parameters of usual data that is fed
to the neural networks.

78
 Non-Linear Activation Functions
 Non-linear activation functions solve the limitations and drawbacks of
simpler activation functions, such as the vanishing gradient problem.
 Non-linear activation functions introduce a non-linear relationship
between input and output, allowing the network to learn complex
patterns.
 Examples:
 Sigmoid, ReLU (Rectified Linear Unit), Tanh, etc.
It makes it easy for the model to generalize or adapt with variety of data and
to differentiate between the output.
• The main terminologies needed to understand for nonlinear
functions are:

• Derivative or Differential: Change in y-axis w.r.t. change in x-

[Link] is also known as slope.

• Monotonic function: A function which is either always

increasing or always decreasing on its domain.
• 1. Sigmoid or Logistic Activation Function
• The Sigmoid Function curve looks like a S-shape.
• The main reason why we use sigmoid function is because it exists
between (0 to 1).
• Therefore, it is especially used for models where we have to predict
the probability as an output.
• Since probability of anything exists only between the range of 0 and
1, sigmoid is the right choice.
• The function is differentiable. That means, we can find the
slope of the sigmoid curve at any two points.
• The function is monotonic but function’s derivative is not.
• 2. Tanh or hyperbolic tangent Activation Function
• tanh is also like logistic sigmoid but better. The range of the tanh
function is from (-1 to 1). tanh is also sigmoidal (s - shaped).
• The function is differentiable.

• The function is monotonic while its derivative is not monotonic.

• The tanh function is mainly used classification between two classes.

• Both tanh and logistic sigmoid activation functions are used in feed-
forward nets.
• 3. ReLU (Rectified Linear Unit) Activation Function
• The ReLU is the most used activation function in the world right now. Since, it is
used in almost all the convolutional neural networks or deep learning.
• As you can see, the ReLU is half rectified (from bottom). f(z) is zero
when z is less than zero and f(z) is equal to z when z is above or
equal to zero.

• Range: (0 to infinity)

• The function and its derivative both are monotonic.

• But the issue is that all the negative values become zero
immediately which decreases the ability of the model to fit or train
from the data properly.

• That means any negative input given to the ReLU activation function
turns the value into zero immediately in the graph, which in turns
affects the resulting graph by not mapping the negative values
appropriately.
• 4. Leaky ReLU
• It is an attempt to solve the dying ReLU problem
• The leak helps to increase the range of the ReLU function. Usually,
the value of a is 0.01 or so.

• When a is not 0.01 then it is called Randomized ReLU.

• Therefore the range of the Leaky ReLU is (-infinity to infinity).

• Both Leaky and Randomized ReLU functions are monotonic in nature.

Also, their derivatives also monotonic in nature.
LEAKY ReLU

• F(x)=max(0.01x,x)
• 5. Softmax activation function
• The softmax activation function takes in a vector of raw
outputs of the neural network and returns a vector
of probability scores.
• z is the vector of raw outputs from the neural network
• The value of e ≈ 2.718
• Example, N = 3 as we have 3 classes: {seal, panda, duck}, and the
valid indices are 0, 1, and 2. Suppose you’re given the vector z =
[0.25, 1.23, -0.8] of raw outputs from the neural network.

• Let’s apply the softmax formula on the vector z, using the steps below:

• Calculate the exponent of each entry.

• Divide the result of step 1 by the sum of the exponents of all entries
• In the vector z of raw outputs, the maximum value is 1.23, which on
applying softmax activation maps to 0.664: the largest entry in the
softmax output vector. Likewise, 0.25 and -0.8 map to 0.249 and 0.087:
the second and the third largest entries in the softmax output
respectively. Thus, applying softmax preserves the relative ordering of
scores.

• All entries in the softmax output vector are between 0 and 1.

• In a multiclass classification problem, where the classes are mutually

exclusive, notice how the entries of the softmax output sum up to 1:
0.664 + 0.249 + 0.087 = 1.
• 6. Hardtanh Activation Function

• Hardtanh is an activation function used for neural networks:

• The hard tanh activation function is a modified version of the tanh
function that applies a threshold to the output to produce an output
between -1 and 1.
• The hard tanh function is faster to compute than the tanh function
and is commonly used in embedded systems and real-time
applications.
• The ReLU and hard tanh activation functions have largely replaced
the sigmoid and soft tanh activation functions in modern neural
networks because of the ease in training multilayered neural networks with
these activation functions
Refer the link for activation functions
[Link]
University Question
• Specify the advantages of ReLU over sigmoid activation function
Tutorial - 1
• 1. Problem
Loss Functions

• A loss function is a function that compares the target and predicted

output values; measures how well the neural network models the
training data.
• When training, we aim to minimize this loss between the predicted
and target outputs.
• The hyperparameters are adjusted to minimize the average
loss — we find the weights, wT, and biases, b, that minimize
the value of J (average loss).
 Types of Loss Functions

• In supervised learning, there are two main types of loss functions :

regression and classification loss functions
• Regression Loss Functions — used in regression neural networks;
given an input value, the model predicts a corresponding output value
(rather than pre-selected labels); Ex. Mean Squared Error, Mean
Absolute Error
• Classification Loss Functions — used in classification neural
networks; given an input, the neural network produces a vector of
probabilities of the input belonging to various pre-set categories — can
then select the category with the highest probability of belonging; Ex.
Binary Cross-Entropy, Categorical Cross-Entropy
 Mean Squared Error (MSE)

• One of the most popular loss functions, MSE finds the average of the
squared differences between the target and the predicted outputs

• The difference is squared, which means it does not matter whether the
predicted value is above or below the target value; however, values with a
large error are penalized.
• However, one disadvantage of this loss function is that it is very sensitive
to outliers; if a predicted value is significantly greater than or less than its
target value, this will significantly increase the loss.
 Mean Absolute Error (MAE)
• MAE finds the average of the absolute differences between the target
and the predicted outputs.
• A loss function called a Huber Loss was developed, which has the
advantages of both MSE and MAE.

If the absolute difference between the actual and predicted value is less than or
equal to a threshold value, 𝛿, then MSE is applied. Otherwise — if the error is
sufficiently large — MAE is applied.
Binary Cross-Entropy/Log Loss

• This is the loss function used in binary classification models — where

the model takes in an input and has to classify it into one of two pre-
set categories.

 In binary classification, there are only two possible actual values of y

— 0 or 1. Thus, to accurately determine loss between the actual and
predicted values, it needs to compare the actual value (0 or 1) with
the probability that the input aligns with that category (p(i) =
probability that the category is 1; 1 — p(i) = probability that the
category is 0)
Categorical Cross-Entropy Loss

• In cases where the number of classes is greater than two, we utilize

categorical cross-entropy — this follows a very similar process to
binary cross-entropy.
UNIVERSITY QUESTION
• Derive weight updating rule in gradient descent when error function
is
• a. Mean squared error
• b. cross entropy
 TRAINING A NEURAL NETWORK WITH
BACKPROPAGATION
• Backpropagation is the essence of neural network training.
• It is the method of fine-tuning the weights of a neural network based
on the error rate obtained in the previous epoch .
• Proper tuning of the weights allows you to reduce error rates and
make the model reliable by increasing its generalization.
• The Back propagation algorithm in neural network computes the
gradient of the loss function for a single weight by the chain rule.
• It efficiently computes one layer at a time
Backpropagation Algorithm
Backpropagation is a supervised learning algorithm for training artificial neural
networks, including feedforward and recurrent neural networks.
It is used to adjust the network's weights and biases to minimize the error
between predicted and actual output.
In the single-layer neural network, the training process is relatively
straightforward because the error (or loss function) can be computed as a direct
function of the weights,
In the case of multi-layer networks, the problem is that the loss is a complicated
composition function of the weights in earlier layers
The training of the BPN is done in three stages

 The feed-forward of the input training pattern

 The calculation and back-propogation of the error

 Updation of weights
Step 1: Forward Pass

•Start by passing an input through the network to compute the predicted

output.

•The forward pass involves two main calculations:

• Weighted sum of inputs and biases at each neuron.

• Application of an activation function to the weighted sum to produce the neuron's

output.
ADVANTAGES
• Backpropagation is fast, simple and easy to program
• It has no parameters to tune apart from the numbers of input
• It is a flexible method as it does not require prior knowledge about
the network
• It is a standard method that generally works well.
• The main features of Backpropagation are the iterative, recursive and
efficient method through which it calculates the updated weight to
improve the network
BACKPROPAGATION EXAMPLE
• Input values
• X1=0.05
• X2=0.10

• Initial weight
• W1=0.15 w5=0.40
• W2=0.20 w6=0.45
• W3=0.25 w7=0.50
• W4=0.30 w8=0.55

• Bias Values
• b1=0.35 b2=0.60

• Target Values
• T1=0.01
• T2=0.99
 Types of Neural Network Architectures
Standard neural networks
•Perceptron - A neural network that applies a mathematical operation
to an input value, providing an output variable.
•Feed-Forward Networks - A multi-layered neural network where the
information moves from left to right, or in other words, in a forward
direction. The input values pass through a series of hidden layers on
their way to the output layer.
•Residual Networks (ResNet) - A deep feed-forward network with
hundreds of layers.
Recurrent neural networks
Recurrent neural networks (RNNs) remember previously learned
predictions to help make future predictions with accuracy.
•Long short term memory network (LSTM) - LSTM adds extra
structures, or gates, to an RNN to improve memory capabilities.
Echo state network (ESN) - A type of RNN hidden layers that are
sparsely connected.
• Convolutional Neural Networks

• Convolutional neural networks (CNNs) are a type of feed-forward

network that are used for image analysis and language processing.
There are hidden convolutional layers that form ConvNets and detect
patterns. CNNs use features such as edges, shapes, and textures to
detect patterns.
• Examples of CNNs include:
•AlexNet - Contains multiple convolutional layers designed for
image recognition.
•Visual geometry group (VGG) - VGG is similar to AlexNet, but has
more layers of narrow convolutions.
•Capsule networks - Contain nested capsules (groups of neurons)
to create a more powerful CNN.
• Generative Adversarial Networks

• Generative Adversarial Networks (GAN) are a type of unsupervised learning

where data is generated from patterns that were discovered from the input
data.
• GANs have two main parts that compete against one another:
• Generator - creates synthetic data from the learning phase of the model. It
will take random datasets and generate a transformed image.
• Discriminator - decides whether or not the images produced are fake or
genuine.
• GANs are used to help predict what the next frame in a video might be,
text to image generation, or image to image translation.
• Transformer neural networks

• Unlike RNNs, transformer neural networks do not have a concept of

timestamps. This enables them to pass through multiple inputs at
once, making them a more efficient way to process data.
• Transformers don't rely on sequential processing, allowing for parallel
computation and better handling of long-range dependencies.
• This makes them highly effective for tasks such as machine
translation, text generation, and image analysis.
Basic Terms
1. Bias in Machine Learning

 It is the error due to the model’s inability to represent the true relationship
between input and output accurately.
 Bias is simply defined as the inability of the model because of that there is some
difference or error occurring between the model’s predicted value and the actual
value.
 These differences between actual or expected values and the predicted values are
known as error or bias error or error due to bias.
 Bias is a systematic error that occurs due to wrong assumptions in the machine
learning process.
 When a model has poor performance both on the training and testing data means
high bias because of the simple model, indicating underfitting.
• Low bias – low error while training with training set

• High bias – high error while training with training set.

• Models with high bias tend to underfit the training data. They are too

simplistic and cannot capture the complexity of the true patterns in the

data.

• As a result, they have poor performance not only on the training data but

also on new, unseen data.

2. Variance, on the other hand, is the error due to the model’s sensitivity to
fluctuations in the training data.
 It’s the variability of the model’s predictions for different instances of training data.
 High variance occurs when a model learns the training data’s noise and random
fluctuations rather than the underlying pattern.
 As a result, the model performs well on the training data but poorly on the testing
data, indicating overfitting.

Variance: If the machine learning model performs well with the training dataset, but does
not perform well with the test dataset, then variance occurs.
 High Variance : high variations in error while considering training and testing set.

 Low Variance : low variations in error while considering training and testing set.

 Models with high variance tend to overfit the training data. They capture not only
the underlying patterns but also the random fluctuations in the data.

 While they may perform exceptionally well on the training data, they
generalize poorly to new data because they have learned noise rather than
signal.
 Ways to Reduce the Variance in Machine Learning:

• Cross-validation: By splitting the data into training and testing sets multiple times, cross-validation can help
identify if a model is overfitting or underfitting and can be used to tune hyperparameters to reduce variance.
• Feature selection: By choosing the only relevant feature will decrease the model’s complexity. and it can
reduce the variance error.
• Regularization: We can use L1 or L2 regularization to reduce variance in machine learning models
• Ensemble methods: It will combine multiple models to improve generalization performance. Bagging,
boosting, and stacking are common ensemble methods that can help reduce variance and improve
generalization performance.
• Simplifying the model: Reducing the complexity of the model, such as decreasing the number of
parameters or layers in a neural network, can also help reduce variance and improve generalization
performance
• Early stopping: Early stopping is a technique used to prevent overfitting by stopping the training of the deep
learning model when the performance on the validation set stops improving.
How the trade-off between bias and variance affects machine learning algorithms:

•High Bias, Low Variance (Underfitting): When a model has high bias and low variance, it is too simplistic and
cannot capture the complexity of the data. It underfits the training data. The model will perform poorly both on the
training data and new data. Solutions for underfitting include using more complex models or adding relevant
features.

•Low Bias, High Variance (Overfitting): Models with low bias and high variance are overly complex and fit the
training data closely, including the noise. They overfit the training data and perform well on it but poorly on new
data. Solutions for overfitting include using simpler models, more training data, regularization techniques, or early
stopping.

•Balancing Bias and Variance: The goal is to find a balance between bias and variance that results in a model
with good generalization. This often involves tuning hyperparameters, such as model complexity, learning rate,
and regularization strength. Cross-validation and careful hyperparameter tuning help strike the right balance.

•Bias-Variance Trade-off Curve: In practice, there's often a trade-off between bias and variance. As you reduce
bias (e.g., by using a more complex model), variance tends to increase, and vice versa. Machine learning
practitioners aim to find the sweet spot on the bias-variance trade-off curve where the model achieves the best
3. Overfitting and Underfitting

 Overfitting and Underfitting are the two main problems that occur in machine
learning and degrade the performance of the machine learning models.

 The main goal of each machine learning model is to generalize well.

 Generalization defines the ability of an ML model to provide a suitable output by

adapting the given set of unknown input. It means after providing training on the
dataset, it can produce reliable and accurate output.

 Hence, the underfitting and overfitting are the two terms that need to be checked for
the performance of the model and whether the model is generalizing well or not.
 Overfitting
A statistical model is said to be overfitted when the model does not make accurate predictions on
testing data.

When a model gets trained with so much data, it starts learning from the noise and inaccurate data
entries in our data set.

When testing with test data results in High variance.

Then the model does not categorize the data correctly, because of too many details and noise.

Reasons for Overfitting:

1. High variance and low bias.
[Link] model is too complex.
[Link] size of the training data.
Underfitting
A statistical model or a machine learning algorithm is said to have underfitting when a model is too simple to
capture data complexities.

It represents the inability of the model to learn the training data effectively result in poor performance
both on the training and testing data.

In simple terms, an underfit model’s are inaccurate, especially when applied to new, unseen examples.

It mainly happens when we uses very simple model with overly simplified assumptions.

Reasons for Underfitting:

1. High bias and low variance.
[Link] model is too simple.
[Link] input features which is used to train the model is not the adequate representations of underlying factors
[Link] size of the training dataset used is not enough
[Link] are not scaled.
• Techniques to Reduce Underfitting
[Link] model complexity.
[Link] the number of features, performingfeature
engineering.
[Link] noise from the data.
[Link] the number of epochs or increase the duration of
training to get better results.
HYPERPARAMETERS AND VALIDATION SETS
• Hyperparameters in Machine learning are those parameters that are explicitly
defined by the user to control the learning process.
• These hyperparameters are used to improve the learning of the model, and
their values are set before starting the learning process of the model.
• Here the prefix "hyper" suggests that the parameters are top-level parameters
that are used in controlling the learning process.
• The value of the Hyperparameter is selected and set by the machine learning
engineer before the learning algorithm begins training the model.
• Hence, these are external to the model, and their values cannot be
changed during the training process.
 • Some examples of Hyperparameters in Machine Learning

o The k in kNN or K-Nearest Neighbour algorithm

o Learning rate for training a neural network
o Train-test split ratio
o Batch Size
o Number of Epochs
o Branches in Decision Tree
o Number of clusters in Clustering Algorithm
 Model Hyperparameters:

• Some key points for model parameters are as follows:

• These are usually defined manually by the machine learning engineer.

• One cannot know the exact best value for hyperparameters for the given
problem. The best value can be determined either by the rule of thumb
or by trial and error.

• Some examples of Hyperparameters are the learning rate for training

a neural network, K in the KNN algorithm,
Categories of Hyperparameters

• Broadly hyperparameters can be divided into two categories,

which are given below:
[Link] for Optimization
[Link] for Specific Models
 Hyperparameter for Optimization

 The process of selecting the best hyperparameters to use is known as

hyperparameter tuning, and the tuning process is also known as
hyperparameter optimization.
 Optimization parameters are used for optimizing the model.
• Learning Rate: The learning rate is the hyperparameter in optimization
algorithms that controls how much the model needs to change in
response to the estimated error for each time when the model's weights
are updated.
• It is one of the crucial parameters while building a neural network, and also
it determines the frequency of cross-checking with model parameters.
• Selecting the optimized learning rate is a challenging task because if the
learning rate is very less, then it may slow down the training process.
• On the other hand, if the learning rate is too large, then it may not optimize
the model properly.
• Note: Learning rate is a crucial hyperparameter for optimizing the model, so
if there is a requirement of tuning only a single hyperparameter, it is
suggested to tune the learning rate
• Batch Size: To enhance the speed of the learning process, the training set
is divided into different subsets, which are known as a batch.

• Number of Epochs: An epoch can be defined as the complete cycle for

training the machine learning model.
• Epoch represents an iterative learning process.
• The number of epochs varies from model to model, and various
models are created with more than one epoch.
• To determine the right number of epochs, a validation error is taken
into account. The number of epochs is increased until there is a
reduction in a validation error.
• If there is no improvement in reduction error for the consecutive
epochs, then it indicates to stop increasing the number of epochs.
 Hyperparameter for Specific Models
• Hyperparameters that are involved in the structure of the model are known as hyperparameters
for specific models. These are given below:
• Number of Hidden Units: Hidden units are part of neural networks, which refer to the
components comprising the layers of processors between input and output units in a neural
network.
• It is important to specify the number of hidden units hyperparameter for the neural network.
• It should be between the size of the input layer and the size of the output layer.
• More specifically, the number of hidden units should be 2/3 of the size of the input layer, plus the
size of the output layer.
• For complex functions, it is necessary to specify the number of hidden units, but it should not
overfit the model.
• Number of Layers: A neural network is made up of vertically arranged components, which are
called layers. There are mainly input layers, hidden layers, and output layers. A 3-layered
neural network gives a better performance than a 2-layered network. For a Convolutional Neural
network, a greater number of layers make a better model.
Validation Set
• Training set: The data you will use to train your model. This will be fed into an
algorithm that generates a model. It maps inputs to outputs.

• Validation set: This is smaller than the training set, and is used to evaluate the
performance of models with different hyperparameter values. It's also used to
detect overfitting during the training stages.

• Test set: This set is used to get an idea of the final performance of a model after
hyperparameter tuning. It's also useful to get an idea of how different models
(SVMs, Neural Networks, Random forests...) perform against each other.
• The validation and test sets are usually much smaller than the training set.
• The validation and test sets are put aside at the beginning of the project and are
not used for training.
• The validation set is used to fine-tune the hyperparameters of the model and is
considered a part of the training of the model.
• The model only sees this data for evaluation but does not learn from this data

• The size of the validation set should be large enough to provide reliable
performance estimates but not too large, as it reduces the size of the training
set.

• Common splits are 70-80% training, 10-15% validation, and 10-15% testing.
Practical Issues in Neural Network Training
I. The Problem of Overfitting
II. The Vanishing and Exploding Gradient Problems
III. Difficulties in Convergence
IV. Local and Spurious Optima
V. Computational Challenges
I. The Problem of Overfitting

• The primary objective in deep learning is to have a network that performs its best on
both training data & the test data/new data it hasn’t seen before.
• Overfitting & Underfitting is a common occurrence encountered .
• Overfitting during training can be spotted when the error on training data decreases to a
very small value but the error on the new data or test data increases to a large value
• There is always a gap between the training and test data performance, which is
particularly large when the models are complex and the data set is small.
• When the network tries to learn too much or too many details in the training data
along with the noise from the training data which results in poor performance on
unseen or test dataset.
• When this happens the network fails to generalize the features/pattern found in the
training data.
• The error vs iteration graph shows how a deep neural network
overfits on training data.
• The blue curve indicates the error on training data & the red curve
the error on test data.
• The point where the green line intersects is the instance the network
begins to overfit.
• As you can see, the error on test data increases sharply while error
on training data decreases.
Error vs iteration graph
• A new set of data points will result in the model/network performing
poorly as it is very close to all the training points which are noise &
outliers.
• The error on the training points is minimum or very small but the
error on the new data points will be high.
• One of the main reasons for the network to overfit is if the size of the
training dataset is small.
• When the network tries to learn from a small dataset it will tend to
have greater control over the dataset & will make sure to satisfy all
the datapoints exactly.
• In order to understand this point, consider a simple single-layer
neural network on a data set with five attributes, where we use the
identity activation to learn a real-valued target variable.
• Consider a situation in which the observed target value is real and is
always twice the value of the first attribute, whereas other
attributes are completely unrelated to the target. However, we have
only four training instances, which is one less than the number of
features. For example, the training instances could be as follows:
• The correct parameter vector in this case is W = [2, 0, 0, 0, 0] based on the known
relationship between the first feature and target.
• The training data also provides zero error with this solution, although the
relationship needs to be learned from the given instances
• However, the problem is that the number of training points is fewer than the
number of parameters and it is possible to find an infinite number of solutions with
zero error.
• For example, the parameter set [0, 2, 4, 6, 8] also provides zero error on the training
data.
• However, if we used this solution on unseen test data, it is likely to provide very
poor performance because the learned parameters are spuriously inferred and are
unlikely to generalize well to new points in which the target is twice the first
attribute (and other attributes are random).
• As a result, the solution does not generalize well to unseen test data
Measures to prevent overfitting
1. Decrease the network complexity
• Deep neural networks like CNN are prone to overfitting because of
the millions or billions of parameters it encloses.
• By removing certain layers or decreasing the number of neurons
(filters in CNN) the network becomes less prone to overfitting as the
neurons contributing to overfitting are removed or deactivated.
• There is no general rule as to how many layers are to be removed or
how many neurons must be in a layer before the network can
overfit.
• The popular approach for reducing the network complexity is
Grid search can be applied to find out the number of neurons
and/or layers to reduce or remove overfitting.
The overfit model can be pruned (trimmed) by removing nodes
or connections until it reaches suitable performance on test data.
2. Data Augmentation
• One of the best strategies to avoid overfitting is to increase the size
of the training dataset.
• As discussed, when the size of the training data is small the network
tends to have greater control over the training data.
• Data augmentation provides techniques to increase the size of
existing training data without any external addition.
• If our training data consists of images, image augmentation
techniques like rotation, horizontal & vertical flipping, translation,
increasing or decreasing the brightness or adding noise, cutouts
etc can be applied to the existing training images to increase the
number of instances.
• By applying the above-mentioned data augmentation strategies, the
network is trained on multiple instances of the same class of
object in different perspectives.
• An augmented result of a lion’s photograph will have an instance of
a lion being viewed in a rotated manner, a lion being viewed up-
side-down or cutting out the portion of an image which encloses
the mane of a lion.
• By applying the last augmentation (cutout) the network learns to
associate the feature that male lions have a mane with its class.
3. Weight Regularization

• Weight regularization is a technique which aims to stabilize an overfitted network

by penalizing the large value of weights in the network.
• An overfitted network usually presents with problems with a large value of
weights as a small change in the input can lead to large changes in the output.
• For instance, when the network is given new or test data, it results in incorrect
predictions.
• Weight regularization penalizes the network’s large weights & forcing the
optimization algorithm to reduce the larger weight values to smaller weights, and
this leads to stability of the network & presents good performance.
• In weight regularization, the network configuration remains unchanged only
modifying the value of weights.
• Weight Regularization reduces overfitting by penalizing or adding a
constraint to the loss function.
• In Deep Learning there are two well-known regularization techniques:
• L1 and L2 regularization.
• Both add a penalty to the cost based on the model complexity, so
instead of calculating the cost by simply using a loss function, there
will be an additional element (called “regularization term”) that
will be added in order to penalize complex models.
Regularization to prevent over-fit

Weight decay: To prevent overfitting, every time we update a

weight w with the gradient ∇J in respect to w, we also subtract from
it λ∙w. This gives the weights a tendency to decay towards zero, hence
the name.
4. Dropouts
• Dropout is a regularization strategy that prevents deep neural networks
from overfitting.
• Deactivate a certain number of neurons at a layer from firing during
training.
• At each iteration different set of neurons are deactivated & this results in
a different set of results.
• Many deep learning frameworks implement dropouts as a layer which
receives inputs from the previous layer, the dropout layer randomly
selects neurons which are not fired to the next layer.
• By deactivating certain neurons which might contribute to overfitting the
performance of the network on test data improves.
• Dropouts reduce overfitting in a variety of problems like image
classification, image segmentation, word embedding etc.
5. Early Stopping
• While training a neural network using an optimization algorithm like
Gradient Descent, the model parameters (weights) are updated to reduce
the training error.
• At the end of each forward propagation, the network parameters are
updated to reduce error in the next iteration.
• Too much training can result in network overfitting on the training data.
• Early stopping provides guidance as to how many iterations can be run
before the network begins to overfit.
• The graph indicates the point after which the network begins to
overfit.
• The network parameters at the point of early termination are the best
fit for the model.
• To decrease the test error beyond the point of early termination can
be done by
• Decreasing the learning rate. Applying a learning rate scheduler
algorithm would be recommended.
• Applying a different optimization algorithm.
• Applying regularization.
6. Neural Architecture and Parameter Sharing
• The most effective way of building a neural network is by
constructing the architecture of the neural network after giving
some thought to the underlying data domain.
• For example, the successive words in a sentence are often related to
one another, whereas the nearby pixels in an image are typically
related.
• These types of insights are used to create specialized architectures
for text and image data with fewer parameters.
• Furthermore, many of the parameters might be shared. For
example, a convolutional neural network uses the same set of
parameters to learn the characteristics of a local block of the image.
7. Trading Off Breadth for Depth
• Networks with more layers (i.e., greater depth) tend to require far
fewer units per layer because the composition functions created by
successive layers make the neural network more powerful.
• Increased depth is a form of regularization, as the features in later
layers are forced to obey a particular type of structure imposed by
the earlier layers
• The number of units in each layer can typically be reduced to such
an extent that a deep network often has far fewer parameters even
when added up over the greater number of layers.
8. Ensemble Methods
 A variety of ensemble methods like bagging are used in order to increase the
generalization power of the model.
These methods are applicable not just to neural networks but to any type of
machine learning algorithm.
 However, in recent years, a number of ensemble methods that are specifically
focused on neural networks have also been proposed.
Two such methods include Dropout and Dropconnect.
 When training with Dropout, a randomly selected subset of activations are set to
zero within each layer.

 DropConnect instead sets a randomly selected sub-set of weights within the

network to zero. Each unit thus receives input from a random subset of units in
the previous layer.
 These methods can be combined with many neural network architectures to

obtain an additional accuracy improvement of about 2% in many real settings.

 However, the precise improvement depends to the type of data and the nature
of the underlying training.
[Link] Vanishing and Exploding Gradient Problems

In neural networks, gradients are used to update the model's

weights via backpropagation.
They are calculated from the loss function with respect to the
weights.
If these gradients are too small or too large, learning becomes
inefficient or unstable.
Propagating backwards using the chain rule has its drawbacks in
networks with a large number of layers in terms of the stability of the
updates.
 In particular, the updates in earlier layers can either be negligibly
small (vanishing gradient) or they can be increasingly large (exploding
gradient) in certain types of neural network architectures.
2. VANISHING and EXPLODING GRADIENTS PROBLEM
EXPLODING GRADIENTS
III. Difficulties in Convergence
Sufficiently fast convergence of the optimization process is difficult
to achieve with very deep networks, as depth leads to increased
resistance to the training process in terms of letting the gradients
smoothly flow through the network.
This problem is somewhat related to the vanishing gradient problem,
but has its own unique characteristics.
4. IV. LOCAL OPTIMA, SPURIOUS OPTIMA, GLOBAL OPTIMA
IV. Local and Spurious Optima
When the parameter space is large, and there are many local optima, it
makes sense to spend some effort in picking good initialization points.
One such method for improving neural network initialization is referred to
as pretraining.
The basic idea is to use either supervised or unsupervised training on
shallow sub-networks of the original network in order to create the initial
weights.
This type of pretraining is done in a greedy and layerwise fashion in which
a single layer of the network is trained at one time in order to learn the
initialization points of that layer.
This type of approach provides initialization points that ignore drastically
irrelevant parts of the parameter space to begin with.
 Furthermore, unsupervised pretraining often tends to avoid problems
associated with overfitting
 V. COMPUTATIONAL CHALLENGES
Training neural networks, especially deep neural networks, can be computationally challenging due to
several factors.
Model Size and Complexity:
• Deep neural networks can have millions or even billions of parameters, making them computationally intensive to
train. The sheer size and complexity of these models require substantial computational resources.
Data Size:
• Deep learning models often benefit from large amounts of training data. Handling large datasets can be challenging in
terms of storage, memory, and processing power.
Computational Power:
• Training deep neural networks requires powerful hardware, such as GPUs (Graphics Processing Units) or TPUs (Tensor
Processing Units). These specialized hardware accelerators are necessary for the fast matrix operations required for
training.
Memory Requirements:
• Deep learning models require significant memory for storing model parameters, intermediate activations, and
gradients during training. This can be a limitation, especially when working with limited GPU memory.
Hyperparameter Tuning:
Finding the best hyperparameters for a neural network often involves experimenting with multiple
configurations. This process can be computationally expensive, as each configuration requires
training and evaluation.
Training Time:
Training deep neural networks can take a long time, ranging from hours to weeks, depending on the
model size, dataset size, and complexity. Long training times can slow down research and
development.
Overfitting Mitigation:
Techniques to combat overfitting, such as dropout and regularization, can increase training time
since they often require training for more epochs to achieve good generalization.
Energy Consumption:
•Training deep learning models consumes a significant amount of energy, which can be a concern in
terms of environmental impact and operational costs.
DEEP LEARNING VS MACHINE LEARNING
Machine Learning Deep Learning
Apply statistical algorithms to learn the hidden Uses artificial neural network architecture to
patterns and relationships in the dataset. learn the hidden patterns and relationships in
the dataset.
Can work on the smaller amount of dataset Requires the larger volume of dataset compared
to machine learning
Better for the low-label task. Better for complex task like image processing,
natural language processing, etc.
Takes less time to train the model. Takes more time to train the model.
A model is created by relevant features which Relevant features are automatically extracted
are manually extracted from images to detect from images. It is an end-to-end learning
an object in the image. process.
Less complex and easy to interpret the result. More complex, it works like the black box
interpretations of the result are not easy.
It can work on the CPU or requires less It requires a high-performance computer with
computing power as compared to deep GPU.
learning.
DEEP FEED FORWARD NETWORK
• A Feed Forward Neural Network is an artificial neural network in
which the connections between nodes does not form a cycle.
• The feed forward model is the simplest form of neural network as
information is only processed in one direction.
• While the data may pass through multiple hidden nodes, it always
moves in one direction and never backwards.
• The structure of a DFF is very similar to that of an FF.
• The major difference between them is the number of hidden layers.
Currently, people refer to a Neural Network with one hidden layer as
a “shallow” network or simply a Feed-Forward network.
• Feedforward neural networks perform well when solving basic
problems like identifying simple patterns or classifying information.
However, they will struggle with more complex tasks.
• On the other hand, deep learning algorithms can process and
analyze vast data volumes due to several hidden layers of
abstraction
IMPLEMENTATION OF SINGLE-LAYER PERCEPTRON

• Let us now implement a single-layer perceptron using the “MNIST”

dataset using the TensorFlow library.

• Step1: Import necessary libraries

• Numpy – Numpy arrays are very fast and can perform large
computations in a very short time.
• Matplotlib – This library is used to draw visualizations.
• TensorFlow – This is an open-source library that is used for Machine
Learning and Artificial intelligence and provides a range of functions to
achieve complex functionalities with single lines of code.
import numpy as np
import tensorflow as tf
from tensorflow import keras
import [Link] as plt
%matplotlib inline
• Step 2: Now load the dataset using “Keras” from the imported
version of tensor flow.

• (x_train, y_train),(x_test, y_test) = [Link].load_data()

• Step 3: Now display the shape and image of the single image in the
dataset. The image size contains a 28*28 matrix and length of the
training set is 60,000 and the testing set is 10,000.

len(x_train)
len(x_test)
x_train[0].shape
[Link](x_train[0])
• Step 4: Now normalize the dataset in order to compute the calculations in a
fast and accurate manner.

• # Normalizing the dataset

x_train = x_train/255
x_test = x_test/255
•
• # Flattening the dataset in order to compute for model building
x_train_flatten = x_train.reshape(len(x_train), 28*28)
x_test_flatten = x_test.reshape(len(x_test), 28*28)
Step 5: Building a neural network with single-layer perceptron. Here we
can observe as the model is a single-layer perceptron that only contains
one input layer and one output layer there is no presence of the hidden
layers.

model = [Link]([
[Link](10, input_shape=(784,),
activation='sigmoid')
])
[Link](
optimizer='adam',
loss='sparse_categorical_crossentropy',
metrics=['accuracy'])

[Link](x_train_flatten, y_train, epochs=5)

• Step 6: Output the accuracy of the model on the testing data.

• [Link](x_test_flatten, y_test)
•THANK YOU