Introduction to Deep Learning

(23AIE234M)

Ms. Shalini Tiwari

Department of Computer Science & Engineering,
Amrita School of Computing, Bengaluru

1
 Unit-1
ANN

2
What is Artificial Intelligence
Artificial intelligence (AI) refers to the field of computer science focused on developing
intelligent machines that can perform complex tasks, that would typically require
human level of intelligence. These tasks include analysis, pattern recognition, decision
making, prediction, etc.

Key AI techniques:
• Machine Learning
• Generative AI
• Deep Learning
• Natural Language Processing
• Reinforcement Learning
• Expert Systems
• Knowledge Representation and Reasoning

3
Machine Learning
Machine Learning is a subset of Artificial Intelligence that focuses on developing
algorithms and models enabling computers to learn from data using statistical
techniques, allowing them to make predictions or decisions without being
explicitly programmed.

4
Deep Learning
Deep learning is a subset of machine learning that uses multilayered neural networks,
called deep neural networks, to simulate the complex decision-making power of the
human brain.

• Deep Learning uses

Representation learning
(automatic feature extraction
and selection)

5
Why is Deep Learning has become famous in recent years?
• Massive Dataset Availability

• Powerful Hardware/Processors Availability

• Large Collection of Prebuilt Frameworks and

Libraries

• Availability of state-of-the-art architectures

6
Why is Deep Learning famous?
• Massive Dataset Availability
Dataset Name Description Organization
MNIST Handwritten digit images (0–9) Yann LeCun / NYU
Large-scale image dataset for object
ImageNet ImageNet / Stanford
recognition
Images with object detection,
COCO Microsoft
segmentation, captions

IMDB Reviews Sentiment analysis on movie reviews Stanford

Google Audio Set Labeled sound clips from YouTube Google

videos
SQuAD Question-answering dataset based on Stanford
Wikipedia
Open Images Annotated images for classification and Google
detection

7
Why is Deep Learning famous?
• Powerful Hardware/Processors Availability
Processor Type Types Use Best For

CPU Desktop, Server, General-purpose • Light AI tasks

(Central Processing Unit) Mobile CPUs computing • Data preprocessing
• Running small models
GPU Gaming GPUs, Parallel processing • Training deep learning models
(Graphics Processing Unit) Mobile GPUs, & deep learning • Fast parallel computation
Datacenter GPUS, • High-performance computing
TPU TPU v2/v3 (For Powerful Google's custom chip • Training massive DL models on Google
(Tensor Processing Unit) Google Cloud), for Deep Learning Cloud
TPU Edge (For low • Fast matrix operations (e.g., in
power Transformers)
AI on Edge devices)
NPU Neural Engine by Apple, AI acceleration on • Optimized for on-device AI tasks (e.g.,
(Neural Processing Unit) Kirin NPU by Huawei, mobile/embedded face unlock, object detection).
Exynos NPU (Samsung) devices • Real-time processing (voice, camera AI)

8
Why is Deep Learning famous?
Availability of Built in libraries:
The availability of various Deep Learning Frameworks gives flexibility to the user to focus on the mail
problem to be solve and reduce the burden of coding deep learning components from scratch.

Libraries/Frameworks Language Best For

TensorFlow (by Google) Python, C++ • Most popular DL framework; scalable; used in
research & production

PyTorch (by Meta/Facebook) Python • Very popular in research

Keras (integrated with Python • High-level API, beginner-friendly; great for

TensorFlow) prototyping and learning

9
Types of Learning Algorithms
Supervised Learning
Supervised learning is a machine learning technique that uses human-
labeled input and output datasets to train the machine. The trained
model learns the underlying relationships between inputs and outputs,
enabling it to predict correct outputs based on new, unlabeled real-
world input data. The major types of supervised learning algorithms
are:
• Regression
• Classification

Examples:
Using the pixels of an image to detect the presence or absence of a cat.
Using someone’s words to predict whether they’re happy or sad
Using weather sensor data to predict the probability of rain
Using news data to predict tomorrow’s stock price
Using a raw audio file to predict a transcript of the audio

10
Types of Learning Algorithms
Unsupervised Learning
In unsupervised learning the machine is given with data
that has no labels or categories. It analyzes the input data
on its own to find patterns, groups or relationships without
any labeled output. The machine learns by discovering
hidden structures within the data without being told what
the correct output should be. The major types of
unsupervised learning algorithms are:
• Clustering
• Association

Examples:
Recommendation System
Customer segmentation
Anomaly Detection

11
The inspiration from biological neuron
• The Dendrites receives input from other neurons in form
of chemical signals called the neurotransmitters.
• These inputs create electric impulses which are
processed at the SOMA(cell body), which is collected at
the Axon Hillock.
• Depending on the strength of the signals created in the
SOMA, the neuron decides to passes this further down
via axon. This electrical signal is called action potential.
• The Synapse or Axon terminals further pass the
Fig1. The Biological Neuron
neurotransmitter to the dendrites of the next connected
neurons.

12
The inspiration from biological neuron

To perform its complex cognitive functions the brain relies on the complex network of multiple neurons where,
each neuron may only fire when its intended criteria is met i.e., a neuron may perform a certain role to a certain
stimulus .

13
The inspiration from biological neuron

Fig2. The Artificial Neuron

14
 McCulloch-Pitts Neuron (MP Neuron)
• Its the first computational model of a neuron.
• Proposed by Warren McCulloch (neuroscientist) and Walter Pitts
(logician) in 1943.
• MP Neuron was designed to work with binary inputs and outputs.
• The processing at the neuron is divided into 2 main parts.
• g(x) is a simple aggregation of the inputs.
• Theta θ here is called thresholding parameter, which maps the
output to either 0 or 1 depending upon the function f(g(x))

Example: Prediction of watching a cricket match

• Inputs: isItaWorldCup, isOneDay, isIndiaPlaying, isNotAtHome,
isTomorrowExam {all can be either 0 or 1}
• Outputs: 1 is I will watch, 0 if I will not watch

15
McCulloch-Pitts Neuron (MP Neuron)
Excitatory inputs are NOT the ones that will make the neuron fire on their own, but they might fire it when
combined.
Inhibitory inputs have maximum effect on the decision making irrespective of other inputs i.e.,
if any of isNotHome, isTomorrowExam is 1 then my output will always be 0 i.e., the neuron will never fire. So,
these inputs can be inhibitory.

Boolean Functions using MP Neurons:

NOR Function
A Function With
AND Function OR Function
An
Inhibitory Input

16
McCulloch-Pitts Neuron (MP Neuron)
Geometrical Interpretations of Boolean Functions using MP Neurons:

17
McCulloch-Pitts Neuron (MP Neuron)
Implementation of 2-input AND Function using MP Neurons:
• You Need to get the theta (threshold value)
1. For x (1,1) g(x) = 1+1 = 2
2. For x (1,0) g(x) = 1+0 = 1
3. For x (0,1) g(x) = 0+1 = 1
4. For x (0,0) g(x) = 1+1 = 0

Keeping the thresholding logic in mind

Theta here should be >=2

• So, we can say

18
McCulloch-Pitts Neuron (MP Neuron)
Implementation of 2-input x1 AND !x2 Function using MP Neurons:

• You Need to get the theta (threshold

value)
1. For x (0,0) g(x) = 0+!0 = 0+1= 1
2. For x (0,1) g(x) = 0+!1 = 0+0 =0
3. For x (1,0) g(x) = 1+!0 = 1+1=2
4. For x (1,1) g(x) = 1+!1 = 1+0=1

From the truth table we can see that in

3rd case the neuron is fired and that is
because x1 = 1. Keeping the
thresholding logic in mind
Theta here should be >=1

19
McCulloch-Pitts Neuron (MP Neuron) : Limitations

• Works only for binary inputs and outputs.

• Fixed weights (usually 1) limits its applicability to represent the real–world problems
where, the inputs may have different influence on the final output.
• MP neurons do not have mechanisms for learning or adjusting their weights based on
experience or training data, making them unsuitable for tasks requiring adaptation or
optimization.
• Inability to Handle Non-Linearities due to linear threshold logic.
• MP neurons may not scale well to handle large amounts of data or complex networks, as
their binary nature and fixed weights can lead to computational inefficiencies.

20
Linear Separability
Linearly Separable Data Linearly Non-Separable Data

Data points belong to different classes can be clearly Data points belong to different classes can not be
separated by linear boundary (line in 2D, plane in 3D separated by any linear boundary (line in 2D, plane in
and hyperplane in higher dimensional data) 3D and hyperplane in higher dimensional data)

21
Perceptron
• One of the simplest artificial neural network architectures, introduced by Frank Rosenblatt in 1958 and
refined by Minsky and Papert in 1969.
• Primarily used for binary classification (prediction between two classes) in Supervised Machine Learning.
• Overcomes some of the limitations of MP Neuron.
• It takes an input, aggregates it (weighted sum) and returns 1 only if the aggregated sum is more than
some threshold else returns 0. Weights
are
important

22
Perceptron

Theta Θ is the bias (threshold here) which is negated when

moved to left side of the equation.
Weight w0 represents the bias

23
Components of a Perceptron
• Input Features: The perceptron takes multiple input features, each representing a characteristic of the input
data.
• Weights: Each input feature is assigned a weight that determines its influence on the output. These weights are
adjusted during training to find the optimal values.
• Summation Function: The perceptron calculates the weighted sum of its inputs, combining them with their
respective weights.
• Activation Function: The weighted sum is passed through the Heaviside step function, comparing it to a
threshold to produce a binary output (0 or 1).
• Output: The final output is determined by the activation function, often used for binary classification tasks.
• Bias: The bias term helps the perceptron make adjustments independent of the input, improving its flexibility in
learning.
• Learning Algorithm: The perceptron adjusts its weights and bias using a learning algorithm, such as
the Perceptron Learning Rule, to minimize prediction errors.
24
Perceptron Learning Algorithm : The basic transformation

[w0, w1, w2,……………wn]

Here w0 represents the bias as

discussed in the previous slide no
23

Visualization
[Link]

25
Perceptron Learning Algorithm w= w +x or w = w – x
Directly adding or subtracting x in w may cause
bigger transformations in the linear boundary,
which may affect the model convergence

For slower changes in weights we do:

if x ∈ P and w.x < 0 then
w_new = w_old + lr * x
End
if x ∈ N and w.x ≥ then
w_new= w_old – lr *x
end
In one line these update statements can be re
written as:
𝑊𝑖_𝑛𝑒𝑤 = 𝑊𝑖_𝑜𝑙𝑑 +∝ 𝑦𝑡𝑎𝑟𝑔𝑒𝑡 − 𝑦𝑝𝑟𝑒𝑑 𝑥𝑖

Where, i is the index of input feature α is learning rate,

𝑦𝑡𝑎𝑟𝑔𝑒𝑡 is actual output and 𝑦𝑃𝑟𝑒𝑑 is the predicted output

26
Updated Perceptron Learning Algorithm
Algorithm: Perceptron Learning
Input: Input feature matrix X(m,n) where, m is the number of inputs and n is the number of features per
input.
Initialize weights from w1 to wn and bias b with small random values
Initialize learning rate α
Repeat till convergence
For ep in range (epochs)
For i in range([Link][0])
current input x = x[i]
Compute the weighted sum wᵀx
Apply the step function on the weighted sum to get predicted value ŷ
Update weight by 𝑊𝑖_𝑛𝑒𝑤 = 𝑊𝑖_𝑜𝑙𝑑 +∝ 𝑦𝑡𝑎𝑟𝑔𝑒𝑡 − 𝑦𝑝𝑟𝑒𝑑 𝑥𝑖 and bias by 𝑏𝑛𝑒𝑤 = 𝑏𝑜𝑙𝑑 +∝ 𝑦𝑡𝑎𝑟𝑔𝑒𝑡 − 𝑦𝑝𝑟𝑒𝑑

Return the final weights and bias

27
Limitations of Perceptron (Single layer)
• Limited to Linearly Separable Data : it can only learn to classify data that can be separated by a line/plane/hyperplane
• Cannot learn complex features in the data
• No Probabilistic Interpretation : As perceptron uses step activation function which is not smooth, not continuous and not
differentiable. Perceptron only give a hard decision (0 or 1) and don’t provide a confidence level or probability.
• Can not learn the non – linearity in data like XOR Function

The 4th condition contradicts 2nd and 3rd

Thus, this inequality has no solution

28
MultiLayer Perceptron
• A multi-layer perceptron (MLP) is a neural network that has at least three layers: an input layer, a
hidden layer and an output layer. Each layer operates on the outputs of its preceding layer.
• The intermediate layers between the input and the output are called hidden layers since they are
not visible outside of the network.
• The input layer is typically not counted in the number of layers in the network. For example, a 3-
layer network has one input layer, two hidden layers, and an output layer.
• It handles the limitations of Rosenblatt’s single perceptron by creating a network of multiple
perceptrons to better learn multiple complex patterns hidden in the input data.
• The number of neurons in the final output layer depends on the type of problem.
Single Perceptron for binary class classification or prediction problems
Multiple perceptrons for multiclass classification problem
29
MultiLayer Perceptron: The simple Feed-Forward Neural
Network

A two layer MLP with one hidden layer A three-layer MLP with two hidden layers
and single output neuron and 3 output neurons

30
 MultiLayer Perceptron
• MLP works on two major operations:
1. Forward Propagation (aka Forward pass) 2. Backward Propagation (aka Backpropagation)
Forward Propagation is a process where the input data is fed through the network in a forward direction, layer-
by-layer, until it generates the output. At each Layer the data and corresponding weights are accepted from the
previous layer and the following major operations are performed at each perceptron (neuron) on this layer.
Summation z = w.x +b
a = f(z)
Where, a is the output produced by the neuron and f is the activation function applied to map the linear
summation output to a non-linear output. This output a in then forwarded to the neurons in the next layer.

Backpropagation is the process in which the error in the final output is propagated back through the network to
adjust the weights and biases. The gradient of the error function gets calculated and propagated back using
chain rule to each layer to learn the optimal weights and biases.

Due to the above operations a MLP is also called a Feed-Forward Neural Network with Backpropagation
31
Universal Approximation Theorem
• The universal approximation theorem states that an MLP with one hidden layer (with a
sufficient number of neurons) can approximate any continuous function of the inputs
arbitrarily well. With two hidden layers, it can even approximate discontinuous functions.
This means that even very simple network architectures can be extremely powerful.

32
Activation Functions: What and Why?
• The activation function is a mathematical function that is used within neural networks to add non-
linearity.
• To decide whether a neuron is activated or not, the activation function processes the weighted sum of
the neuron’s inputs and calculates a new value to determine how strongly the signal is passed on to the
next layer in the network.
• Activation functions are necessary for any neural network to generate high-dimensional nonlinear
patterns.
• Activation functions are also known as Transfer functions.
• A neural network without the activation function behaves like a linear regression model, because every
neuron will only be performing a linear transformation on the inputs using the weighted summation.
Thus, it doesn’t matter how many hidden layers we attach in the neural network; all layers will behave in
the same way because the composition of two or more linear functions is a linear function itself.
33
Activation Functions: What and Why?
• While choosing the activation functions keep the following in the mind:
• Should be non-linear
• Should be differentiable
• Should be computationally inexpensive
• Should be zero centered
• Should be non-saturating (saturating (within a range) causes vanishing gradient problem)
However, these requirements are not mandatory but recommended.

34
Activation Functions: What and Why?
Activation functions in Deep Learning are mainly of three types:
• Binary Step Activation Function
• Linear Activation Function
• Non-Linear Activation Function

Binary Step Activation Function: The Binary Step is a very

basic activation function that decides whether a neuron
should activate or not based on a threshold. If the input is
above the threshold, it activates(sends a signal); if not, it
does not fire.

Limitations:
• It cannot provide multi-value outputs—thus not suitable for
multi-class classification problems.
• The gradient of the step function is zero, which causes a
hindrance in the backpropagation process.

35
Activation Functions: Types
Bipolar Step Activation Function: In Bipolar Step function a neuron should activate if the input is
above the threshold else it does not fire.

OR

Limitations:
• It cannot provide multi-value outputs—thus not suitable for multi-class classification problems.
• The gradient of the step function is zero, which causes a hindrance in the backpropagation process.

36
Activation Functions: Types
Linear or Identity Activation Function: The linear activation function, also known as "no activation,"
or "identity function" , is where the activation is proportional to the input.
Its Range is –infinity to +infinity.
This activation function is better suitable for linear regression
problem.

Limitations:
• Can not help the network to learn complex non- linear patterns in the data, whereas most of the
real-world data are non-linear in nature
• It’s not possible to use backpropagation as the derivative of the function is a constant and has no
relation to the input x.

37
Non-linear activation functions and their types

Why are Non-Linear Activation Functions Needed?

• The non-linear activation functions are differentiable, and their derivative function is
related to the input thus, allowing backpropagation to learn the optimal set of
weights.

• They allow the stacking of multiple layers of neurons to make the final output a non-
linear combination of input passed through multiple layers. Using this any output
can be represented as a functional computation in a neural network.

38
Non-linear activation functions and their types
Sigmoid / Logistic Activation Function
• This function ensures that the input value is mapped to a range between 0 and 1.

• The larger the input (more positive), the closer the output value will be to 1, whereas the smaller
the input (more negative), the closer the output will be to 0.

• This range of values makes the sigmoid function particularly suitable for binary class
classification/ prediction.

• This function provide smoother and differentiable output, allowing for more effective training.

Derivative : f′(x)= f(x) (1 − f(x) )

39
Non-linear activation functions and their types
Sigmoid / Logistic Activation Function : Limitations

• The function is monotonic (either always increasing or always

decreasing) but function’s derivative is not. That means

• Gradient values will be significant only for small range (around -3

to 3). After that, graph is getting saturated to a constant value.
Therefore, its gradient will approach towards 0 which will lead to
vanishing gradient problem.

• The output of the logistic function is not symmetric around zero.

So, the output of all the neurons will be of the same sign. This
makes the training of the neural network more difficult and
unstable.

40
Non-linear activation functions and their types
Tanh/ hyperbolic tangent Activation Function
• This function is a shifted version of the sigmoid, ensures that the input value is mapped to a range
between -1 and 1.
• The larger the input (more positive), the closer the output value will be to 1, whereas the smaller
the input (more negative), the closer the output will be to -1.
• The output of the tanh activation function is Zero centered; hence we can easily map the output
values as strongly negative, neutral, or strongly positive.

41
Non-linear activation functions and their types
Tanh/ hyperbolic tangent Activation Function
: Limitations

• It also faces the problem of vanishing gradients

similar to the sigmoid activation function. Even
the gradient of the tanh function is much steeper
as compared to the sigmoid function.

42
Non-linear activation functions and their types
Rectified Linear Unit (ReLU) Activation Function
• Introduced to solve the vanishing gradient problem
• It keeps positive values and sets negative input values equal to
zero.
• The most used in computer vision and CNN.

Advantages:
• Computationally efficient when compared to the sigmoid and
tanh
Functions as it does not activate all the neurons at the same time.
Neurons will be activated only if the output of the linear Derivative :
transformation
is greater than 0.
• ReLU accelerates the convergence of gradient descent towards
the global minimum of the loss function due to its linear, non
saturating property.
43
Non-linear activation functions and their types
ReLU Limitations
The Dying ReLU Problem : The negative side of the graph makes
the gradient value zero (Fig.5). Due to this reason, during the
backpropagation process, the weights and biases for some neurons
are not updated. This can create dead neurons which never get
activated.

44
Non-linear activation functions and their types
Leaky ReLU Activation Function:

An improved version of ReLU function to solve the Dying ReLU

problem as it has a small positive slope in negative area.

The gradient of the left side of the graph comes out to be a

nonzero value. Therefore, we would no longer encounter dead
neurons in that region. Derivative :
it does enable backpropagation, even for negative input values.

Limitations:
The gradient for negative values is a small value that makes the
learning of model parameters time-consuming.

45
Non-linear activation functions and their types
Parametric ReLU Activation Function:

Another variant of ReLU that aims to solve the problem of

gradient’s becoming zero or very small for the left half of the axis in
original ReLU and Leaky ReLU.
This function provides the slope of the negative part of the function
as an argument a. By performing backpropagation, the most
appropriate value of a is learnt.

Used when the leaky ReLU function still fails at solving the problem
of dead neurons, and the relevant information is not successfully
passed to the next layer.

Other Variants for ReLU are:

1. Exponential Linear Units (ELU)
2. Gaussian Error Linear Unit (GELU)
3. Scaled Exponential Linear Unit (SELU)

46
Non-linear activation functions and their types
Softmax Activation Function:

• It is most commonly used as an activation function for the

last layer of the neural network in the case of multi-class
classification.

• It transforms raw output scores from a neural network

into probabilities.

• The Softmax function ensures that each class is assigned a

probability, helping to identify which class the input
belongs to.

• The number of neurons in the output layer must be equal

to the number of classes present in the dataset.

• Outputs will be non-negative and sums upto 1.

47
How to choose activation functions
The choice of activation function for each layer in a neural network depends on the type of layer, the problem
you're solving, and the output range you desire.

For Hidden Layers:

• The most used is ReLU (Rectified Linear Unit) and variants due to its faster convergence and efficient
computation.
• Tanh and sigmoid function can also be used , but they become saturated due to vanishing gradient problem.
However, Tanh is preferred over sigmoid for hidden layers.

For Output Layer:

• For Regression Problem : Identity Function (If target values are unbounded real numbers.), ReLU (If outputs
are non-negative such as house price).
• For Binary classification problem : Sigmoid is preferred
• For Multiclass classification Problem: Softmax

Web Resources For Activation Functions:

1. [Link]
2. [Link]
cb20414c04e5/
48
Loss Functions / Error Functions in Deep Learning
• Loss function also known error function is a mathematical function which is used to evaluate how well
our algorithm is modelling the dataset.

• It is used to measure the difference (variation) between the target and predicted output value.

• The lower value of loss function means the better prediction by the neural network.

• The loss function is a function of network parameters ( Weights and bias). So, whenever there is a
change in weight , bias or both the loss function will also change.

• It guides the learning algorithm in adjusting the model’s parameter i.e weights and biases during the
training time.

• The correct choice of loss function plays a crucial role in the training of any neural network.

• Sometimes Loss Functions (L) and Cost Function (J) are used interchangeably, however, loss function
is measured with respect to single training data point whereas cost function is calculated with respect
to the batch of data points. So, it is basically the average of loss functions for ‘n’ data points.
49
Types of loss functions
The choice of correct loss function depends majorly on the type of problem we are solving.
Following are some important loss functions used in different type of Neural Networks.
However, there use is not limited to the specific problem only.

50
Loss Functions in Detail: Mean Squared Error (MSE)
• MSE finds the average of the squared differences between the target and the predicted
outputs. Also known as Squared Loss or L2 loss.

• The squared term is used for 2 main reasons 1. To penalize the predicted outputs with
larger difference and 2. To avoid the –ve values.

• One of the most widely used loss functions for regression tasks. It can be used only if
output layer neuron is using identity activation function.

Advantages:
• Easy to understand and interpret .
• Due to square term, we get the derivative (differentiable) which helps in
backpropagation using Gradient Descent. [Not recommended if dataset has
outliers]

51
Loss Functions in Detail: Mean Squared Error (MSE)
outliers are data points that significantly deviate from the general pattern or
distribution of the training data. These unusual values can negatively impact model
performance if not handled properly during training.

Limitation: Sensitive to outliers in the dataset because the errors are squared which can
significantly affect the loss.

52
Loss Functions in Detail: Mean Absolute Error (MAE)
• MAE finds the average of the absolute difference between the target and the predicted outputs. It
is also known as L1 loss. Also used commonly is regression problem.

• Can be used only if output layer neuron is using identity activation function.
• Advantages:
• Easy to understand and interpret .
• Unlike MSE, MAE is not sensitive to outliers in the dataset. [Can be used if dataset has
outliers]
• Limitation: Due to absolute term, we don’t get the derivative (non differentiable) which becomes
a problem in using gradient descent while doing backpropagation. Thus, sub gradients are used
here which makes it complex.

53
Loss Functions in Detail: Huber Loss
• Huber loss combines both MSE and MAE .It behaves like MSE for small deviations from original
value and for large deviations from original value it behaves like MAE.

• Huber Loss is mainly used when there are more number of outliers present in the dataset.

• It behaves like MSE, if the absolute difference between the actual and predicted value is less than
or equal to a threshold value, 𝛿. Otherwise – if the error is sufficiently large – it behaves like MAE.

• The threshold 𝛿 is again a hyperparameter which must be

tuned to understand the outliers.

54
Loss Functions in Detail: Binary Cross Entropy
• This is the loss function used in binary classification models – where the model takes in an input
and has to classify it into one of two pre-set categories. Also known as log loss or Negative log
likelihood.

• Whenever we are using binary cross entropy loss then the output neuron must have sigmoid as
activation function.

• The log function grows slowly for large values and sharply for small values. This helps gradients
respond more effectively based on the difference between y and y hat.

55
Intuition behind the log loss
• Log-loss is indicative of how close the prediction probability is to the corresponding actual/true value (0 or 1 in case
of binary classification). The more the predicted probability diverges from the actual value, the higher is the log-loss
value.
The goal of the logistic loss function is:
Create a large loss if the target = 1 and the prediction = ~0
Create a small loss if the target = 1 and the prediction = ~1
Create a large loss if the target = 0 and the prediction = ~1
Create a small loss if the target = 0 and the prediction = ~0

56
 Intuition behind the log loss
Why Consider log loss:
• Log loss penalty: The log loss would penalize the incorrect predictions severely when the predicted
probability is high. A model overconfident in its predictions will produce high log loss score than a model that
is more uncertain about its predictions.
• Scale-invariant: Log loss was not affected by the scale of the predictions, which is a valuable property when
comparing models that might make different scale predictions.
• Following a proper scoring rule: Log loss is optimized by the actual class probabilities compared to accuracy,
which is only optimized when the predicted class is the same as the actual class.

The limitations:
• Log loss only measures the performance of a model in probability. If you need an accuracy score, then it
might not be suitable.
• Log loss is sensitive to class imbalance. Suppose there is a significant imbalance in the distribution of classes;
the performance of the majority class may dominate log loss.
• Log loss can be computationally expensive for large datasets. As log loss is calculated for each data row, it
could be a burden during calculation.

refer this link [Link]

[Link]

57
Loss Functions in Detail: Categorical Cross Entropy
• In Multiclass classification we use Categorical Cross entropy . Also known as Multi log loss.

• Number of output neurons must be equal to the number of classes. The target label is represented
in one-hot encoded form. For example, if we have 4 classes and the correct class is class 2, then:
y=[0,1,0,0]

• Output layer should use Softmax activation function.

58
Loss Functions in Detail: Sparse Categorical Cross Entropy
• Similar to Categorical Cross Entropy. It is computationally faster than Categorical Cross Entropy.

• Everything is same as Categorical Cross Entropy.

• The only difference is that the target label is represented in single integer value. For example, if we
have 4 classes and the correct class is class 2, then: y = 2

• Output layer should use Softmax activation function.

Web Resources For Loss Functions:

1. [Link]
2. [Link]

59
Optimization in Deep Learning
What is Optimization?
Optimization is the process of finding the best value of a variable that either minimizes or
maximizes a function. In general, an optimization problem is of two types:
1. Minimization problem : Objective is to find the best possible value of x where value of f(x) is
minimum.
2. Maximization problem : Objective is to find the best possible value of x where value of f(x) is
maximum.
Where does optimization fit in neural network/ deep learning?
• As, we discusses before, that the loss/cost function measures the amount of error done in the
output produced by any NN and it is desired to keep the error lowest possible to improve the
network performance.
• Thus, the main objective of any neural network is to minimize the value of the loss/cost function
through multiple iterations/epochs.
• There are many optimization algorithms available , but ML and Deep learning highly rely on Gradient
Descent Optimization algorithm to minimize the cost function.
• As, we know that cost function is a function of network parameters(weights and bias), the gradient
descent algorithm tries to find the best possible values of weight and bias to minimize the value of cost
function.
60
What is Gradient/Derivative :
The term Gradient also known as Derivative/Slope refers to the instantaneous rate of change of a
quantity with respect to the other. To define more mathematically it is the varying rate of change
of a function f(x) with respect to an independent variable x.

Some Examples: f(a) = 3a then d f(a)/ d(a) = 3

Lets say, a = 2 then f(a) = 6

If a is changed to 2.001 then f(a) = 6.003, then slope =
0.003/0.001 = 3

Another point a is 5 then f(a) = 15

If a is changed to 5.002 then f(a) = 15.006, then slope =
0.006/0.002 = 3

So, for this function the derivative/ slope is constant at any

point.

The term Gradient is used for derivative when you have more than one dependent variables.

61
What is Gradient/Derivative
𝑑 𝑓(𝑎)
𝑓(𝑎) = 𝑎2 then = 2𝑎
𝑑𝑎

Lets say, a = 2 then f(a) = 4

If a is changed to 2.001 then f(a) = 4.004001, then slope = 0.004001 /0.001 = 4.001

Another point a is 5 then f(a) = 25

If a is changed to 5.001 then f(a) = 25.010001, then slope = 0.010001/0.001 = 10.001

So, for this function the derivative/ slope changes at different points.

Thus, we can say that,

• The derivative of a function is slope of a function

• And for different functions the derivative can vary at different points.

Chain rule of differentiation:

The chain rule tells you how to take the derivative of a composite function.
𝑑𝑦 𝑑𝑦 𝑑𝑢
If 𝑦 = 𝑓 𝑢 , 𝑤ℎ𝑒𝑟𝑒 𝑢 = 𝑔 𝑥 𝑡ℎ𝑒𝑛 𝑑x = 𝑑𝑢 . 𝑑𝑥

62
Convex and Non-Convex Functions
A cost function is said to be convex if it is shaped like a bowl, with a single minimum point. On
the other hand, a non-convex cost function has multiple local minimum points, and the global
minimum may not be easily identifiable.
When selecting an optimization algorithm, it is essential to consider whether the loss
function is convex or non-convex.

Convex Plot in 3D Non-Convex Plot in 3D Convex and Non-Convex Plot in 2D

63
Gradient Descent Algorithm
What?
• Gradient descent is a method for
unconstrained mathematical optimization. It is
a first order iterative algorithm for minimizing
a differential multivariate function.

• The idea is to take repeated steps in the

opposite direction of the gradient of the
function at the current point, because this is the
direction of steepest descent.

• It is particularly useful in machine learning for

minimizing the cost or loss function.

• Originally proposed by CAUCHY in 1847

64
Gradient Descent Algorithm
Goal : Minimize a loss/cost function J(θ), where θ represents the model parameters (weights and
biases).

Step 1: Initialize model parameters θ (e.g., weights w, bias b). Commonly with small value close to
zero.
Step 2: Initialize a learning rate α>0. This controls how big the updation step is.

Repeat till convergence or max number of epochs

Step 3: For each input 𝑥𝑖 ​, compute the predicted output 𝒚_𝒉𝒂𝒕𝒊 = 𝝈 𝒛𝒊 , where, 𝒛𝒊 = 𝒘𝒊 𝒙𝒊 + 𝒃𝒊

Step 4: Compute the loss function J(θ), such as MSE, MAE, BCE, etc., to evaluate how far the
predictions are from the actual targets.

𝜕𝐽(𝜃)
Step 5: Calculate the gradient of the loss function with respect to each parameter: 𝜕𝜃
Step 6: Adjust each parameter θ (weight and bias) in the opposite direction of the gradient using:
𝜕𝐽(𝜃)
𝜃𝑛𝑒𝑤 = 𝜃𝑜𝑙𝑑 − ∝
𝜕𝜃

65
Types of Gradient Descent Algorithm
The choice of gradient descent algorithm depends on the problem at hand and the size of the dataset.
There are majorly three types of gradient descent algorithms.
1. Batch Gradient Descent
2. Stochastic Gradient Descent
3. Minibatch Gradient Descent

Batch Gradient Descent:

• Also known as vanilla gradient descent.
• Batch gradient descent calculates the error for each example in the training dataset. But the
parameters only gets updated after all training examples have been evaluated. This process is
called a training epoch.
• It basically computes the average gradient of the loss over all training examples (cost function) and
Update parameters using that full-batch gradient

66
Types of Gradient Descent Algorithm
Batch Gradient Descent (Advantage):
• It’s computationally efficient, it produces a stable error gradient and a stable convergence.

• When the data fits into memory (small dataset), batch gradient descent is efficient and stable.

• It has straight trajectory towards the minimum, and it is guaranteed to converge in theory to the global
minimum if the loss function is convex and to a local minimum if the loss function is not convex.

• We can use fixed learning rate during training without worrying about learning rate decay.

Batch Gradient Descent (Limitations):

• Slow for large datasets as it needs to compute the gradient using the entire dataset before each
update, which can be very slow.

• Since updates are made only once per full pass through the data, convergence can be slower.

• Requires loading the entire dataset into memory at once — not feasible for very large datasets.
• Sometimes, its stable error gradient can lead to an unfavorable convergence state.
67
Types of Gradient Descent Algorithm
Stochastic Gradient Descent:
• Stochastic Gradient Descent (SGD) updates the parameters for each training example one
by one within the dataset.

• Much faster than batch gradient descent.

• Frequent updates allow it to have a pretty detailed rate of improvement.

• Computationally more expensive than the batch gradient descent due to frequent updates.

• Additionally, the frequency of those updates can result in noisy gradients, which may cause
the error rate to jump around instead of slowly decreasing.

68
Types of Gradient Descent Algorithm
Limitations of SGD

High Variance in Updates

Because updates depend on individual samples (or small batches), the path to convergence is noisy and
unstable.
Sensitive to Learning Rate
Too high → diverges, too low → very slow convergence. Careful tuning is needed.
Slow Convergence to Minima
While it moves quickly at first, it can slow down near the minima due to noisy updates.
May Get Stuck in Poor Local Minima/Saddle Points
Without improvements like momentum, RMSProp, or Adam, SGD may converge to suboptimal solutions.

69
Types of Gradient Descent Algorithm
Minibatch Gradient Descent:
• Mini-batch gradient descent combines concepts from both batch gradient descent and stochastic gradient
descent.
• It splits the training dataset into small batch sizes and performs updates on each of those batches.
• This approach strikes a balance between the computational efficiency of batch gradient descent and the
speed of stochastic gradient descent.
• Note that it is the go-to algorithm when you are training a neural network and it is the most common type of
gradient descent within deep learning.
• The batch size is something we can tune. It is usually chosen as power of 2 such as 32, 64, 128, 256, 512, etc.
The reason behind it is because some hardware such as GPUs achieve better run time with common batch
sizes such as power of 2.
• Common mini-batch sizes range between 50 and 256, but like for any other machine learning techniques,
there is no clear rule, because they can vary for different applications.
• Mini-batch Gradient Descent sums up over lower number of examples based on the batch size. Therefore,
learning happens on each mini-batch of b examples:

70
Types of Gradient Descent Algorithm
Minibatch Gradient Descent:
The main advantages:
Faster than Batch version because it goes through a lot less samples than Batch (all samples).
Randomly selecting samples will help avoid redundant samples that are very similar that don’t
contribute much to the learning.
With batch size < size of training set, it adds noise to the learning process that helps improving
generalization error.
Even though with more samples the estimate would have lower standard error, the return is
less than linear compared to the computational burden we incur.

The main disadvantages:

On each iteration, the learning step may go back and forth due to the noise. Therefore, it
wanders around the minimum region but never converges.
Due to the noise, the learning steps have more oscillations and requires adding learning-decay
to decrease the learning rate as we become closer to the minimum.

71
Types of Gradient Descent Algorithm
Below is a graph that shows the gradient descent’s variants and their direction towards the
minimum.
As the figure shows, SGD direction is very noisy compared to mini-batch.
Also, the table in the right-hand side shows the comparison between all the three variants.

72
Impact of Learning Rate on Optimization
• How big the steps gradient descent takes in the direction of the local minimum is determined by the learning
rate, which figures out how fast or slow we will move towards the optimal weights.

• For the gradient descent algorithm to reach the local minimum, we must set the learning rate to an
appropriate value, which is neither too low nor too high. This is important because
• If the steps it takes are too big, it may never converge (never reach the local minimum) because it will
oscillate between the boundaries of the convex function.
• If we set the learning rate to a very small value, gradient descent will eventually reach the local minimum
but that may take a while.

73
 Impact of Learning Rate on Optimization
To make sure the gradient descent algorithm runs
properly, plot the cost function with each epoch. This
helps us visualize the value of your cost function after
each iteration of gradient descent and provides a way
to easily spot how appropriate the learning rate is.
The image on the right illustrates the difference
between good and bad learning rates.
If the gradient descent algorithm is working properly,
the cost function should decrease after every
iteration.

Resources for Gradient Descent:

• [Link]
• [Link]
2d8ef6ecad6f
• [Link]
8335f15fdee2
74
Steps involved in Neural Network Learning

Step 1: Do Forward Propagation by calculating the weighted sum then apply activation function on this sum for all the
neurons.
Step 2: At the output layer find the predicted output Y_hat.
Step 3: Calculate the error or loss by applying suitable loss function.
Step 4: Use gradient descent to calculate the gradient of this loss with respect to all the network learnable
parameters(weights and bias)
Step 5: Perform backpropagation till the input layer to update all the weights and biases.

75
 Forward Pass/ Forward Propagation
Forward propagation is the process where a neural network transforms input data into predictions or outputs.

76
Forward Pass/ Forward Propagation

77
Forward Pass/ Forward Propagation

78
Forward Pass/ Forward Propagation

Matrix dimensions in layer L:

z = (3,1) = (𝑛𝐿 , 1)
x = (2,1) = (𝑛𝐿−1 , 1)
w = (3,2) = (𝑛𝐿 , 𝑛𝐿−1 )
b = (3,1) = (𝑛𝐿 , 1)
Where, n is number of neurons

79
Forward Pass/ Forward Propagation

80
Forward Pass/ Forward Propagation

Matrix dimensions in layer L:

z = (2,1) = (𝑛𝐿 , 1)
o = (3,1) = (𝑛𝐿−1 , 1)
w = (2,3) = (𝑛𝐿 , 𝑛𝐿−1 )
b = (2,1) = (𝑛𝐿 , 1)
Where, n is number of neurons

81
Forward Pass/ Forward Propagation

82
Forward Pass/ Forward Propagation

Matrix dimensions in layer L:

z = (1,1) = (𝑛𝐿 , 1)
o = (2,1) = (𝑛𝐿−1 , 1)
w = (1,2) = (𝑛𝐿 , 𝑛𝐿−1 )
b = (1,1) = (𝑛𝐿 , 1)
Where, n is number of neurons

83
 Backpropagation
• Backpropagation algorithm is probably the most fundamental building block in a neural network. It was first
introduced in 1960s and almost 30 years later (1989) popularized by Rumelhart, Hinton and Williams in a
paper called “Learning representations by back-propagating errors”.

• Backpropagation, short for "backward propagation of errors," is an algorithm for supervised learning
of artificial neural networks using gradient descent. Given an artificial neural network and an error function,
the method calculates the gradient of the error function with respect to the neural network's weights.

• It is a generalization of the delta rule for perceptrons to multilayer feedforward neural networks.

• It is a method to calculate how changes to any of the weights or biases of a neural network will affect the
accuracy of model predictions.

• It’s essential to the use of supervised learning, semi-supervised learning or self-supervised learning to train
neural networks.

Backpropagation is the essence of neural network training. It is the practice of fine-tuning the weights of a neural
network based on the error rate (i.e. loss) obtained in the previous epoch (i.e. iteration.) Proper tuning of the
weights ensures lower error rates, making the model reliable by increasing its generalization.

84
Backpropagation: Explained using Linear Regression loss
Here we have explained the process using MSE loss for linear regression problem but can be adapted for other losses
and classification problem too. Also, for simplicity the activation function at all layers is chosen as RELU…. So its
derivative is 1.

85
Backpropagation: Explained using Linear Regression loss: Output layer

86
Backpropagation: Explained using Linear Regression loss: Output layer

87
Backpropagation: Explained using Linear Regression loss: Hidden layer2

88
Backpropagation:Explained using Linear Regression loss: Hidden layer2

Neuron 1
Similarly, it will
proceed for
other neurons of
each layer to
update
corresponding
weights and bias.

89
Backpropagation: General Formula to solve problems :

90
Backpropagation: Why?
• It’s memory-efficient in calculating the derivatives, as it uses less memory compared to other
optimization algorithms. This is a very important feature, especially with large networks.

• The backpropagation algorithm is fast, especially for small and medium-sized networks. As more
layers and neurons are added, it starts to get slower as more derivatives are calculated.

• This algorithm is generic enough to work with different network architectures, like convolutional
neural networks, generative adversarial networks, fully-connected networks, and more.

• There are no parameters to tune the backpropagation algorithm, so there’s less overhead. The only
parameters in the process are related to the gradient descent algorithm, like learning rate.

Limitations:
• Working through multiple iterations can make backpropagation time consuming.
• The success of this process requires high-quality data—"noisy" data can lead to irregularities.

91
 Optimization Algorithms
In Deep Learning
Optimizers other than Gradient Descent

92
 Some Basics for understanding
Contour Plot/Contour Map:
A contour map is a useful alternative for representing plots in 2D space. Contour map uses contours or color-coded regions
helps us to visualize 3D data in two dimensions. Contour maps are also used to visualize the error surfaces in deep
learning/machine learning optimization techniques like Gradient descent, Momentum gradient descent, Adam, etc…

Refer this articles for better understanding:

[Link]

93
Some Basics for understanding
Examples of Contour Maps:

94
Some Basics for understanding
Gradient Updates for Batch Gradient Descent, SGD and MBGD:

95
Some Basics for understanding
Gradient Updates for Batch Gradient Descent, SGD and MBGD
Comparison between BGD , Stochastic GD and Mini-Batch GD:
Batch GD Stochastic GD Mini batch GD
BGD performs model updates at the end SGD is faster than Batch Gradient Descent MBGD provides a good balance between
of each training epoch. This means that and Mini Batch GD since it uses only one the stability of Batch Gradient Descent and
the weights of the model are updated only example to update the parameters. the speed and memory efficiency of
once per epoch, after all instances of the Stochastic Gradient Descent.
dataset have been processed.
Memory intensive because it needs to
store complete datapoints at once.
computationally intensive and time memory efficient because it considers one
consuming for large datasets observation at a time from the complete
dataset.
Descent often leads to more stable and introduce high variance in parameter
reliable convergence to the minimum of updates, leading to oscillations in the
the cost function due to fewer number of convergence path leading to slower
updates convergence and the updates are noisy
(due to update on each datapoint in one
epoch).

96
Challenges in basic variants of gradient descent

97
 Challenges in basic variants of gradient descent
2. Saddle points (flat surface): The slope becomes approximately zero on these points and
does not allow vanilla gradient descent to get a proper direction. These points can
decelerate (slows down) the optimization progress as it is hard to decide the direction.

3. Some optimization landscapes could have steep and shallow regions. Navigating through
these can be tough because the optimization process can get stuck in the flat parts or steep
parts as its struggle to climb which make convergence difficult.

4. Optimization becomes trickier & challenging as we work with more parameters. This can
lead to the “curse of dimensionality” as it is like finding a path inside a complex maze.

5. Noisy Gradient : Real-world data isn’t always perfectly clean as it can have noise or
imperfections. Therefore, optimization methods should be able to handle these types of
real-world data.

6. Fixed Learning Rate across all parameters: Some parameters need big steps (they rarely
get gradients) and Others need small steps (they get frequent large gradients). Thus, Fixed
learning rate cannot adapt to this imbalance.

98
 Improved Optimizers in Deep Learning:
To deal with the limitations and challenges involved in basic gradient descent algorithms many other optimization algorithms
were proposed. The major improvements are done either by adding momentum or adaptive learning rate during the training.

1. Momentum-based → adds inertia to updates

• SGD with Momentum
• Nesterov Accelerated Gradient (NAG)

2. Adaptive learning rate-based → adjusts learning rate for each parameter

• AdaGrad
• RMSProp
• Adadelta

3. Combined (Momentum + Adaptive) → Adam family optimizers.

• Adam
• Adamax : improvement over adam
• Nadam : adds Nesterov momentum to Adam.

99
 SGD with Momentum:
• Momentum methods in the context of machine learning refer to a group of tricks and techniques designed to dampen
oscillations and speed up convergence of first order optimization methods like gradient descent (and its many variants).
• They essentially work by adding what’s called the momentum term to the update formula for gradient descent, thereby
improve its natural “zigzagging behavior,” especially in long narrow valleys of the cost function.
• The figure below shows the progress of gradient descent - with and without momentum - towards reaching the minimum
of a cost function, located at the center of the concentric elliptical contours.

100
 SGD with Momentum:
• Momentum or SGD with momentum is a method which helps accelerate gradients vectors in the right directions, thus
leading to faster converging.
• In the context of gradient optimization it refers to a method that smoothens the optimization trajectory by adding a term
that helps the optimizer remember the past gradients.
In mathematical terms the momentum-based gradient descent updates can be described as:

• Learning Rate (η): The learning rate determines the size of the step taken during each update. It plays a crucial role in
both standard gradient descent and momentum-based optimizers.

• Momentum Factor (β): This controls how much of the past gradients are remembered in the current update. A value
close to 1 means the optimizer will have more inertia while a value closer to 0 means less reliance on past gradients.

101
 SGD with Momentum:
Advantages
• Faster Convergence: It helps to accelerate the convergence by considering past gradients, which helps the model navigate
through flat regions more efficiently.
• Reduces Oscillation: Traditional gradient descent can oscillate when there are steep gradients in some directions and flat
gradients in others. Momentum reduces this oscillation by maintaining the direction of previous updates.
• Improved Generalization: By smoothing the optimization process, momentum-based methods can lead to better
generalization on unseen data, preventing overfitting.
• Helps Avoid Local Minima: The momentum term can help the optimizer escape from local minima by maintaining a strong
enough "velocity" to continue moving past these suboptimal points.

Challenges:
• Choosing Hyperparameters: Selecting the appropriate values for the learning rate and momentum factor can be
challenging. Typically, a momentum factor of 0.9 is common but it may vary based on the specific problem or dataset.
• Potential for Over-Accumulation: If the momentum term becomes too large it can lead to the optimizer overshooting the
minimum, especially in the presence of noisy gradients.

[Link]

[Link]
guide-0252ede605b4
102
 Nesterov Accelerated Gradient
Nesterov Accelerated Gradient is a refined version of momentum-based
optimization, designed to further smooth and speed up convergence. It
does so to dampen the oscillation occurs by large momentum produced
by momentum based gradient update.

NAG helps the optimizer to avoid overshooting the minimum. It does so

by looking ahead at the future gradient and adjusting accordingly, which
helps reduce overshooting and improves stability.

nstead of calculating the gradient at the current point, NAG takes a look-
ahead step using the momentum term. It then computes the gradient at
this new look-ahead position and adjusts the update based on this
gradient. This approach makes it easier to predict when the optimizer is
likely to overshoot, allowing for more controlled updates.

Example:
Imagine rolling a ball down a hill. With simple momentum, you only
know the direction you’re moving in and rely on past speed to guide
you. But with NAG, it’s like you’re peeking a few steps ahead to see if
you’re about to overshoot. This helps you adjust the speed to avoid
missing the target.
103
Nesterov Accelerated Gradient: Mathematical Formulation

104
Nesterov Accelerated Gradient: Mathematical Formulation
This adjustment helps NAG to better control the update, especially when the model is nearing an optimal point, reducing
oscillations that could lead to overshooting.

Advantages of NAG
Reduced Oscillations: By peeking ahead, NAG reduces oscillations around the minimum, which can be especially useful in
error surfaces with multiple peaks and valleys.
Faster Convergence: NAG often converges faster than simple momentum-based methods because it reduces unnecessary
steps around the optimum.

105
 AdaGrad (Adaptive Gradient Algorithm):
AdaGrad (Adaptive Gradient Algorithm): A well-known optimization technique that’s especially useful for sparse data. Sparse
features are those where most values are zero, such as a dataset with lots of missing or inactive features.
AdaGrad modifies the plain gradient descent by adjusting the learning rate dynamically for each parameter, based on its
historical gradients.
The parameter updated formula for AdaGrad is

Where, η is the initial learning rate

Gt is the accumulated squared gradients up to the time step t. and it is calculated as 𝐺𝑡 = 𝐺𝑡−1 + 𝑔𝑡 2
ϵ is a small value to avoid division by zero.
gt is the gradient at the current time step.

Intuition
It accumulates the gradient of each parameter; it divides the learning rate by the square root of the sum of past gradients
squared. This effectively reduces the learning rate for parameters with high gradients, allowing it to focus on features that
change less.
If a parameter has been updated a lot before (large 𝐺𝑡 ​), its denominator is big → smaller step.
If a parameter hasn’t been updated much (small 𝐺𝑡 ​), denominator is small → larger step.
Hence: rarely active parameters learn faster, frequently active ones slow down.

106
 AdaGrad (Adaptive Gradient Algorithm):
Advantage:
• Great for Sparse Data: AdaGrad works well for features with many zeros because it adapts the learning rate for each
feature based on its gradient history.
• No Manual Tuning of Learning Rates: The adaptive learning rate means you don’t have to adjust the learning rate
manually for each parameter.

Weaknesses:
• Converges Slowly due to diminishing Learning rate: Over time, the accumulated gradients make the learning rate so small
that the algorithm stops making meaningful updates.
• Limited Use in Neural Networks: AdaGrad is rarely used in deep neural networks because it slows down too much due to
its shrinking learning rate.
• Gives equal weightage to all the previous gradients in the formula.

To overcome this, optimizers like RMSProp and Adam were developed to address AdaGrad’s limitations.

107
 RMSProp (Root Mean Square Propagation)
• RMSProp keeps a moving average of the squared gradients to normalize the gradient updates. By doing so it prevents the
learning rate from becoming too small which was a drawback in AdaGrad and ensures that the updates are appropriately
scaled for each parameter.
• RMSProp keeps track of an exponentially decaying average of the squared gradients. Instead of accumulating all past
squared gradients as AdaGrad does, RMSProp uses a moving average. This means that RMSProp gives more weight to
recent gradients, allowing it to adapt more effectively without diminishing the learning rate too quickly.

Advantages:
• RMSProp performs well with complex, non-convex optimization problems common in deep learning, unlike AdaGrad,
which works best in convex settings.
• Stable Learning Rate: RMSProp prevents the learning rate from diminishing too quickly, allowing the optimizer to move
steadily towards the minimum.

Challenges:
While RMSProp is a highly effective optimizer, it has largely been superseded by Adam (Adaptive Moment Estimation), which
combines the benefits of both RMSProp and Momentum (another optimization technique). Adam generally outperforms
RMSProp but comes with additional computational overhead.

108
RMSProp (Root Mean Square Propagation)

109
Adam (Adaptive Moment Estimation)
• Combines the advantages of Momentum and RMSprop (Adaptive Learning Rate based) techniques to adjust learning
rates during training. It works well with large datasets and complex models because it uses memory efficiently and
adapts the learning rate for each parameter automatically.

110
Adam (Adaptive Moment Estimation)

111
Comparative Performance

112
Underfitting and Overfitting Model
Underfitting and overfitting are two common challenges faced in machine learning.
Underfitting occurs when a model is too simplistic to grasp the underlying patterns in the data. It lacks the complexity
needed to adequately represent the relationships present, resulting in poor performance on both the training and new
data.
Reason Behind Underfitting
• Model is too simple : Using linear regression for a nonlinear problem., Shallow neural network with too few neurons.
• Insufficient training time: Model hasn’t learned enough (too few epochs/early stopping).
• Too much regularization: L1/L2 penalties, dropout, or pruning may over-restrict the model.
• Important features are Missing: Features don’t capture the relationship (poor feature engineering).

Overfitting happens when a model learns not only the true pattern but also the noise in the training data. Overfitting
happens when a machine learning model becomes overly intricate, essentially memorizing the training data. While this
might lead to high accuracy on the training set, the model may struggle with new, unseen data due to its excessive focus on
specific details. An Overfit model lacks generalization.
Reason Behind Overfitting
• Model is too complex: High-degree polynomial regression (e.g., degree=15), Deep neural network without control.
• Less training data: With fewer samples, model learns specific quirks instead of general rules.
• Training too long: Too many epochs → model adapts to noise.
• No regularization: Lack of dropout, weight decay, or constraints.

113
Bias and Variance
• Bias and variance are two types of errors that can occur in machine learning
models.
• They are both important factors to consider when building a model that can
accurately predict new data.
• Bias and variance are two key sources of error in machine learning models that directly
impact their performance and generalization ability.

How to identify:
The rules

Overfitting/ High Variance: Training error is low, but testing error is significantly higher. Also,
the testing error varies a lot with small change in train set.

Underfitting/ High Bias: Errors are consistently high across training and testing data sets.
114
What is Bias
Bias is the error due to overly simplistic assumptions in the model.
Bias is the error in a model's prediction that's caused by
incorrect assumptions.

High bias typically leads to underfitting, where the model performs

poorly on both training and testing data because it fails to learn
enough from the data.

High Bias:
• Model is too simple.
• Unable to learn important patterns.
• Both training error and test error are high.

Low Bias:
• Model is flexible and can capture complex patterns.
• Training error is low (but variance may increase).

Both the models are underfitting the

data and have high Bias.

115
What is Variance
Error due to high sensitivity to small fluctuations in training data.
Variance measures how much a model’s predictions change if you train it again on a
different sample of data from the same distribution.

If the model is very sensitive to training data (small changes in input → big changes in
predictions), it has high variance.
If the model is stable across different training sets, it has low variance.
Basically, high variance is with respect to the performance on test data. If by changing
train data, there is a noticeable change in test data performance then the model is
considered to have high variance.

A high-variance model learns not only the patterns but also the noise in the training data,
which leads to poor generalization on unseen data.
High variance typically leads to overfitting, where the model performs well on training
data but poorly on testing data.

High Variance:
•Model learns noise along with patterns (overfitting).
•Training accuracy is high but test accuracy is poor.
Both the models are Overfitting the
Low Variance:
•Model generalizes better to unseen data. data and may have high Variance.
•Predictions remain stable even with new training samples.

116
Some Examples

This is a typical classification problem where the two variables (in this case, dots and cross) are two be
separated by a best line.
In case of an underfit model, the line is too straight and doesn’t account for many data points (i.e. high bias and
high variance). Where as, in consideration of Overfit line, the line is so accurate w.r.t. the training data (low
bias) that when the same model is input with a test data, the chances of the prediction of test data going wrong is
high (i.e. high variance).
But with a good fit line, the training data is also classified with quite high accuracy and the testing data is also
predicted with a fair percentage of accuracy (i.e. low bias and low variance).

117
Some Examples

118
Techniques to reduce underfitting/ reduce bias
1. Increase model complexity.
2. Increase the number of features, performing feature engineering.
3. Remove noise from the data.
4. Increase the number of epochs or increase the duration of training to get better results.

Techniques to reduce Overfitting/ reduce Variance

1. Improving the quality of training data reduces overfitting by focusing on meaningful patterns, mitigate the risk of fitting
the noise or irrelevant features.

2. Increase the training data can improve the model's ability to generalize to unseen data and reduce the likelihood of
overfitting.

3. Reduce model complexity.

4. Early stopping during the training phase (have an eye over the loss over the training period as soon as loss begins to
increase stop training).

5. Ridge Regression (L2 Regularization) and Lasso Regression (L1 Regularization).

6. Use dropout for neural networks to tackle overfitting.

119
Bias and Variance Trade-off
The bias-variance tradeoff is the delicate equilibrium between underfitting and overfitting. The goal is to find the
optimal level of complexity that allows a model to generalize effectively to unseen data.
The Tradeoff
•High Bias, Low Variance → The model is too simple (Underfitting).
•High Variance, Low Bias → The model is too complex (Overfitting).
•Goal: Low Bias, Low Variance : Find a balance where both bias and variance are minimized to achieve the lowest total
error. Techniques such as regularization, cross-validation, can help to mitigate the bias-variance tradeoff.
This tradeoff is often visualized as a curve, known as the validation error curve shown below:

120
Regularization
Regularization is an important technique in machine learning that helps to improve model accuracy by preventing
overfitting.
The main idea behind the regularization technique is to penalize complex models, i.e.- to define a penalty function to
quantify complexity of the model.
The more complex models will have a greater penalty associated with them.
Since most of the training algorithms are considered an optimization problem where the loss is minimized we add the
penalty term and minimize the whole expression together.
There are multiple types of regularization techniques, Out of which few important ones and widely used are: Lasso
Regression (L1 Regularization) , Ridge Regression (L2 Regularization) and Elastic Net.

121
Regularization
Regularization tries to shrink the effect of very large weights in the model.

122
Regularization
Example: Predicting House Prices: Why Large weights to specific features can be the problem
Suppose we have a neural network trying to predict house prices based on the following features:
•Size (sq. ft.)
•Number of bedrooms
•Neighborhood rating

123
Regularization

124
Ridge Regression/ L2 Regularization
• Add a term in cost function
• Coefficient regularization
• It adds the squared magnitude of the coefficient as a penalty term to the loss function(L) or cost function.
• It handles multicollinearity by shrinking the coefficients of correlated features instead of eliminating them.
• Also known as weight decay

125
Ridge Regression/ L2 Regularization: How it helps
•By shrinking large weights, L2 regularization prevents extreme parameter updates.
•This reduces the model’s sensitivity to noise and small fluctuations

126
Ridge Regression/ L2 Regularization
When to Use L2 (Ridge) Regularization
•Handling Multicollinearity:
L2 is effective at dealing with highly correlated features (multicollinearity) by shrinking their coefficients
more evenly. This helps to stabilize unstable coefficients.

•Model Robustness:
L2 regularization creates more robust models that are less sensitive to small changes in the data.

•All Features Contribute:

When you believe all features in your dataset are potentially relevant and should contribute to the
outcome, L2 is a good choice to keep them all in the model but with smaller weights.

•Generalization:
L2 regularization helps to prevent overfitting by shrinking large coefficients, leading to a more
generalizable and accurate model.

127
Lasso Regression/ L1 Regularization
• L1 Regularization is a technique used to prevent overfitting by adding a penalty to the absolute values of the
weights.
• This encourages sparsity, meaning some weights become exactly zero, effectively removing less important
features from the model.

128
Lasso Regression/ L1 Regularization
•Unlike L2 regularization (which shrinks weights but does not make them zero), L1 forces some weights to be exactly
zero.
•This leads to a sparse model, where only the most important features are kept.

Effect: The model automatically selects the most relevant features and removes unnecessary ones.

When to Use L1 (Lasso) Regularization

•Feature Selection:
L1 regularization can force the weights of some features to be exactly zero, effectively removing those features from the
model. This is ideal for simplifying complex models and understanding which features are most important.
•High-Dimensional Data:
When you have a large number of features, L1 helps to identify and remove irrelevant ones, creating a sparser model.
•Model Interpretability:
By zeroing out features, L1 can make the model more interpretable, as you can focus on the features with non-zero
weights.
•Applications:
It's particularly useful in domains like financial modeling, image processing, and computer vision where identifying key
features is crucial.

129
Lasso Regression/ L1 Regularization: How to use in code

130
Early Stopping
Early stopping is a regularization technique used to prevent overfitting by stopping the training process once the model’s
performance on the validation data starts to degrade. Instead of setting a fixed number of epochs, we monitor the model’s
performance on a separate validation set after each epoch.

Why Use Early Stopping?

Avoid Overfitting: Early stopping prevents the model from memorizing the training data by halting the process before overfitting occurs.
Save Resources: Stopping early reduces training time and computational costs, especially in large models.
Improves Generalization: Models trained with early stopping tend to generalize better, performing well on unseen data.

131
Dropout
Dropout is a technique that randomly disables (or “drops”) a fraction of
neurons from (input and hidden layer) during each training iteration in a
neural network (as seen in Figure). All the forward and backwards
connections with a dropped node are temporarily removed, thus creating a
new network architecture which is a subset of the parent network. The
nodes are dropped by a dropout probability of p.

This prevents the network from becoming too dependent on certain nodes
and encourages it to learn more generalized features, which helps it perform
better on new data.
How Does Dropout Work?
[Link] Training: At each training iteration, dropout randomly disables a fraction of neurons in the network.
This effectively creates a new, smaller neural network with fewer neurons for that iteration. As a result, the model
learns to work without depending on any one specific neuron, which reduces overfitting.

[Link] Testing: During testing or inference (when the model is predicting on new data), all neurons are active.
However, the weights of the neurons are scaled down by the dropout rate to account for the different training
structure. This scaling is typically by a factor of (1 — dropout rate) to ensure the same output range as during
training.

132
Dropout
Why Dropout Works
By randomly disabling neurons, dropout effectively trains multiple “subnetworks” within the main network, which makes
the overall model more adaptable and prevents it from memorizing specific data points (overfitting). This technique has
shown to improve model accuracy by up to 2%, especially on complex datasets.

Practical Tips for Using Dropout

• Choosing Dropout Rate: Common dropout rates range from 0.2 to 0.5. A higher dropout rate may be beneficial in very
complex networks with many layers, while a lower rate is better for simpler models.

• Using Dropout with Other Regularization: Dropout is often combined with other regularization methods like L2
regularization to further improve performance.

• Early Stopping: Track model performance during training to determine when it starts to overfit. Early stopping allows
you to halt training at the optimal point, preventing unnecessary overfitting.

133
Dropout: Code Snippet using keras
import tensorflow as tf
from [Link] import Sequential
from [Link] import Dense, Dropout

# Simple MLP with Dropout

model = Sequential([
Dense(128, activation='relu', input_shape=(784,)), # Input layer (e.g., flattened 28x28 image)
Dropout(0.5), # Dropout: randomly drop 50% neurons during training
Dense(64, activation='relu'),
Dropout(0.3), # Dropout: 30%
Dense(10, activation='softmax') # Output layer (10 classes)
])

[Link](optimizer='adam', loss='categorical_crossentropy', metrics=['accuracy'])

[Link]()

134
How to improve the performance of a Neural Network
1. Fine tunning of hyperparameters:
• Number of hidden layers: It is better to have more number of hidden layers with sufficient number of neurons
than keeping one hidden layer with too many neurons.
• Number of neurons per layer: Must be sufficient to capture the patterns in the training data.
• Number of Epochs
• Batch Size
• Value of Learning rate : if set high than model performance may oscillate or become poor, if set too low then
causes slower convergence. Typically a range between (0.1 to 0.001 is preferred in many cases.)
• Optimizer ( GD, MBGD, SGD, SGD with Momentum, RMSProp, AdaGrad, Adam etc.)
• Activation Function

2. Solving other problems:

• Solving Vanishing and Exploding Gradient Problems
• Solving problem of insufficient size of dataset
• Slow training
• Problem of underfitting
• Problem of overfitting

3. Weight Initialization Techniques : [Link]

techniques-in-neural-networks-582e80a1e839
[Link]
6066403140e9
135
Parameters of the Artificial Neural Network / MLP
The parameters of any neural network is divided into 2 categories:
Trainable/Learnable Parameters and Hyperparameters

1. Trainable/ Learnable Parameters:

Trainable parameters are the parameters of a neural network (such as weights and biases) that are learned from
data during training by adjusting their values through optimization algorithms (e.g., gradient descent) in order to minimize
the loss function.

In an Artificial Neural Network (ANN) or a Multi-Layer Perceptron (MLP), the parameters that are learned/updated
during training are:

•Weights (W) (connection between two neurons from one layer to the next layer). These weights control the
strength/importance of the input signal. During backpropagation, the weights are updated to minimize the loss.

•Bias (b): Each neuron (except input neurons) usually has a bias term. The bias allows shifting of the
activation function, improving flexibility of the model. Biases are also updated during training.

136
Parameters of the Artificial Neural Network / MLP
How to find total trainable/learnable
parameters in a fully connected / dense ANN.

A fully connected or dense ANN is the one

where each neuron in previous layer is
connected to each neuron in the next layer.

To calculate the total number of trainable

parameters in ANN, we need to follow the
below steps:
For each layer (from input to output):
• Weights = (number of inputs to the
layer) × (number of neurons in the
layer)
• Biases = (number of neurons in the
layer)
• Repeat the above calculation for every
layer in the network.
• Total Trainable Parameters = Sum of
all weights and biases across layers

137
Parameters of the Artificial Neural Network / MLP

138
Parameters of the Artificial Neural Network / MLP
2. Hyperparameters:
Hyperparameters are the parameters of a neural network that are set before training begins and are not updated
during training. They control how the learning process is performed rather than what the network learns.

In an Artificial Neural Network (ANN) or a Multi-Layer Perceptron (MLP), the hyperparameters are:

139