What is Deep learning

Deep Learning is a subfield of artificial intelligence

and machine learning that is inspired by the
structure of a human brain.

DL attempt to draw similar conclusion as human

would by continually analyzing data with a given
logical structure called neural network.
 What is Deep learning
➢ Layers: Deep learning models have multiple layers an input layer,
several hidden layers, and an output layer.
➢ Learning: Each layer learns to extract increasingly complex features
from the data.
➢ Data: Works best with large datasets and high computational
power.
➢ Training: Uses techniques like backpropagation and gradient
descent to minimize error.
 Deep learning

In image recognition, the

first layer may detect edges,
the next shapes, and later
layers recognize objects like
faces or cars.
 What is Deep learning

Applications
➢ Speech recognition
➢ Image classification
➢ Natural Language Processing
➢ Recommendation systems
What is Neuron
 What is Perceptron

A Perceptron is the simplest type of artificial neural network model,

it’s the building block of more complex networks. It was introduced

by Frank Rosenblatt in 1958 and is used mainly for binary

classification.
 Perceptron

A perceptron mimics a biological neuron. It takes several inputs,

multiplies each by a weight, sums them up, adds a bias, and passes

the result through an activation function to produce an output

Perceptron can use any activation function to classify data.

Perceptron & Neuron
 Limitations of perceptron

Cannot Solve Non-Linear Problems

XOR problem

Uses Step Activation Function

 What is Deep learning

x1

x1

x1

x1 3x2 2x1
4x3
 ANN

b is bias, i is layer and j is node

O is output, i is layer and j is node

W represents the weight connecting node i to node j,

where k indicates the layer in which the weight is
applied
 What is Deep learning

x1

x1

x1

x1 3x2 2x1
4x3
 Propagation

In neural networks, the term propagation refers to

the flow of information through the network from
inputs to outputs and back again during the learning
process.
Forward propagation calculation
Forward propagation calculation
Forward propagation
Label the given MLP

Calculate CGPA for student 1

 ANN LOSS FUNCTION

MSE

MAE

BINARY CROSS ENTROPY

Categorical Cross Entropy

 Gradient
The gradient tells us which direction and how strongly to
adjust the model’s parameters (weights) to reduce the error.

If your loss function (error) is

𝑳 𝒘
then the gradient is
𝒅𝑳
𝛁𝑳 𝒘 =
𝒅𝒘
It tells how the loss changes with respect to the weight 𝒘.
 Gradient
The gradient tells us which direction and how strongly to
adjust the model’s parameters (weights) to reduce the error.
 Gradient
The gradient tells us which direction and how strongly to
adjust the model’s parameters (weights) to reduce the error.
 Back propagation
Epochs=5

for i in range (epochs):

Gradient Descent
for j in range (input or X):

select 1 row (random)

predict using forward propagation

calculate loss (using loss function)

update bias and weight using GD

 Gradient Descent Types

Batch GD

Stochastic GD

Mini Batch GD
 Gradient Descent Types

Batch GD
Uses the entire training dataset to compute the

gradient and update the parameters once per epoch.

 Batch GD
Epochs=10
for i in range (10):
calculate Y_pred using dot product for all data in training
calculate loss (using loss function)
update bias and weight using GD
Repeat above step for 10 times
 Stochastic GD
Updates parameters for each training sample individually.

Epochs=5
for i in range (epochs):
for j in range (input or X):
select 1 row (random)
predict using forward propagation
calculate loss (using loss function)
update bias and weight using GD
Which one is faster Batch or Stochastic ??

Which is the faster to converge on same epochs??

 Mini Batches
Splits training data into small batches (e.g.32, 64, 128 samples) and
updates weights after each batch.
Epochs=5
for i in range (epochs):
for j in range number of batches:
select one batch
calculate dot product
calculate loss (using loss function)
update bias and weight using GD
Repeat for number of batches
 Back propagation
Backpropagation (short for backward propagation of errors) is
used to train neural networks by minimizing the error (loss) using
gradient descent.
Std Q1 A1 GPA

1 5 6 3

2 7 9 4

3 2 3 2

4 1 2 1
 Back propagation steps
1. Forward Propagation
2. Compute the Loss (Error)
3. Compute Gradients
4. Update the Weights and Biases
5. Repeat Steps 1–4 for many epochs (iterations) until:
➢ The loss is minimized.
➢ The network converges.
 Back propagation
Back propagation is an algorithm used to Train Neural Network
Std Q1 A1 GPA

1 5 6 3
 Back propagation
Back propagation is an algorithm used to Train Neural Network
Std Q1 A1 GPA

1 5 6 3

We know about W(old) and b(old), learning

rate could be 0.1. We need to find
derivative of W(old) and b old
 Back propagation
Back propagation is an algorithm used to Train Neural Network
Std Q1 A1 GPA

1 5 6 3

We will start from the output and

calculate all weights and biases
 Back propagation
Back propagation is an algorithm used to Train Neural Network
Std Q1 A1 GPA

1 5 6 3

Total Nine parameters details are given:

 Back propagation
Back propagation is an algorithm used to Train Neural Network
Std Q1 A1 GPA

1 5 6 3

Total Nine parameters details are given:

 Back propagation
Std Q1 A1 GPA

1 5 6 3
 Back propagation
Std Q1 A1 GPA

1 5 6 3
 Back propagation
Std Q1 A1 GPA

1 5 6 3
 Back propagation
Std Q1 A1 GPA

1 5 6 3
 Back propagation
Std Q1 A1 GPA

1 5 6 3

These three terms are same in all

 Back propagation
Std Q1 A1 GPA

1 5 6 3

These three terms are same in all

 Back propagation
Std Q1 A1 GPA

1 5 6 3

These three terms are same in all

 Back propagation
Std Q1 A1 GPA

1 5 6 3

These three terms are same in all

 Back propagation
Std Q1 A1 GPA

1 5 6 3

These three terms are same in all

 Back propagation
Std Q1 A1 GPA

1 5 6 3

These three terms are same in all

 Back propagation
Std Q1 A1 GPA

1 5 6 3

These three terms are same in all

 Back propagation
Std Q1 A1 GPA

1 5 6 3
 Back propagation
Std Q1 A1 GPA

1 5 6 3
 Back propagation
Std Q1 A1 GPA

1 5 6 3
 Back propagation
Std Q1 A1 GPA

1 5 6 3
 Back propagation
Std Q1 A1 GPA

1 5 6 3
 Back propagation
Std Q1 A1 GPA

1 5 6 3
 Back propagation
Std Q1 A1 GPA

1 5 6 3
 Back propagation
Std Q1 A1 GPA

1 5 6 3
Back propagation for classification

Std Q1 A1 GPA

1 5 6 Pass

2 2 3 Fail
Back propagation for classification

Std Q1 A1 GPA

1 5 6 Pass

2 2 3 Fail
Back propagation for classification

Std Q1 A1 GPA

1 5 6 Pass

2 2 3 Fail

These are same

Back propagation for classification
Back propagation for classification
Back propagation for classification
Back propagation for classification
Back propagation for classification
Back propagation for classification

Std Q1 A1 GPA

1 5 6 Pass

2 2 3 Fail
Back propagation for classification
Back propagation for classification
Back propagation for classification
Back propagation for classification

Std Q1 A1 GPA

1 5 6 Pass

2 2 3 Fail
Back propagation for classification
Back propagation for classification
Back propagation for classification
Back propagation for classification
Performance of Neural Network?
 Performance of Neural Network

1. Tunning of Hyperparameters

2. Vanishing Gradient/Exploding gradient

3. Not enough data

4. Slow Training

5. Overfitting
 Hyperparameters

Hyperparameters are external configurations that control how a neural

network learns , they are not learned during training but are set before

training.
 1. Tunning of Hyperparameters
i. Number of Hidden layers

ii. Neuron per layer

iii. Learning rate

iv. Optimizer

v. Batch Size

vi. Activation function

vii. Epochs
 1. Tunning of Hyperparameters
i. Number of Hidden layers
➢ Determines the depth of the network.

➢ Each layer extracts higher-level features from the previous one

Model Type Example Hidden Layers

Shallow Network Simple Perceptron 0–1

Deep Network CNN / RNN / LSTM 3–100+

 1. Tunning of Hyperparameters

i. Number of Hidden layers

How many hidden layers will be sufficient in neural network??

 1. Tunning of Hyperparameters

ii. Neuron per layer and

Neurons per Layer

➢ Defines the width of each layer.
➢ Each neuron learns a different aspect of the input pattern.
Rule of Thumb
➢ Too few → underfitting (can’t learn patterns).
➢ Too many → overfitting (memorizes data).
 1. Tunning of Hyperparameters

ii. Neuron per layer and & arrangement

 1. Tunning of Hyperparameters
iii. Learning rate
➢ Controls how much weights change after each step.
➢ It’s the most critical hyperparameter.
Gradient Descent Update Rule:
𝜕𝐿
𝑊𝑡+1 = 𝑊𝑡 − 𝜂
𝜕𝑊𝑡

Learning Rate Behavior

Too small Slow convergence

Too large May overshoot minima

 1. Tunning of Hyperparameters

Activation function

An activation function is a mathematical function used in neural

networks to decide whether a neuron should be activated or not

Why We Need Activation Functions?

 1. Tunning of Hyperparameters

Activation function
An activation function is a mathematical function used in neural
networks to decide whether a neuron should be activated or not

Why We Need Activation Functions?

Without activation functions, a neural network would just be a linear

model (like simple regression) no matter how many layers it has.
 Types of Activation function

Linear Activation Functions

Sigmoid
Tanh
ReLU
LeakyReLU
 Types of Activation function

Function Range Note

Sigmoid (0, 1) For probabilities

Tanh (-1, 1) Zero-centered

ReLU [0, ∞) Fast & widely used

Leaky ReLU (-∞, ∞) Avoids dead neurons

 Tanh Activation function

The tanh (hyperbolic tangent) activation

𝑒 𝑥 + 𝑒 −𝑥
function is a non-linear function used in 𝑇𝑎𝑛ℎ = 𝑥
𝑒 − 𝑒 −𝑥
neural networks. It maps input values to a
range between –1 and +1
 Tanh Activation function

Graphical Behavior
For large positive 𝒙 … … … + 𝟏
For large negative 𝒙 … … . . −𝟏
 Tanh Activation function
Advantages of Tanh

Zero-centered output helps faster convergence during training.

Stronger gradients than sigmoid for mid-range inputs.

Useful for hidden layers in deep networks.

Provides both positive and negative activations.

 Tanh Activation function

Limitations of Tanh

Vanishing Gradient Problem

For very large or small input values, gradients become nearly zero.

Computationally expensive

Involves exponentials

Not ideal for very deep networks , may cause slow learning.
 Tanh Activation function

When to Use Tanh

Hidden layers of shallow or moderate-depth networks.
Output layers when the target range is between –1 and +1.
 ReLU Activation Function

ReLU (Rectified Linear Unit)

It introduces non-linearity
Range: [0, ∞)
Non-Zero-centered
Shape: Linear for positive values, flat (zero) for negative values

f(x)=max(0,x)
 ReLU Activation Function

Graphical Behavior
For x > 0, output increases
linearly with slope
For x ≤ 0, output = 0.
 ReLU Activation Function

Advantages of ReLU
➢ Simple and fast computation, just
thresholding at zero.
➢ Sparse activation: many neurons output
zero, making the model efficient.
 ReLU Activation Function
Limitations of ReLU
➢ Dying ReLU Problem
Neurons can get stuck outputting zero (when weights make inputs
negative).
➢ Not zero-centered
Can slow down convergence for some optimizers.
➢ Sensitive to learning rate
Too high a rate may kill many neurons.
 ReLU Activation Function

Variants of ReLU
1. Leaky ReLU: allows small slope for negative inputs
2. Parametric ReLU (PReLU) :α is learned during training
3. ELU (Exponential Linear Unit):smooth curve for negatives
 ReLU Activation Function

Variants of ReLU

1. Leaky ReLU

2. Parametric ReLU (PReLU)

3. ELU (Exponential Linear Unit)

 Leaky ReLU Activation Function

Leaky ReLU is a modified version of the ReLU (Rectified

Linear Unit) activation function. It was designed to solve the

“dying ReLU” problem, where some neurons stop learning

because their output becomes permanently zero.

 Leaky ReLU Activation Function

➢ For positive inputs, it behaves just like ReLU outputs the

input directly.
➢ For negative inputs, it outputs a small negative value
instead of zero.
 Leaky ReLU Activation Function

Graph Behavior
Right side (x > 0): straight
line with slope.
Left side (x < 0): small slope
(α), not flat like in ReLU.
 Leaky ReLU Activation Function
Advantages

➢ Prevents dying neurons (unlike ReLU).

➢ Allows some learning for negative inputs.

➢ Simple to implement, only adds one parameter (α).

➢ Performs better than ReLU in many deep networks.

 Leaky ReLU Activation Function
Limitations
The slope value α must be chosen manually, not learned.

If α is too small, behaves almost like ReLU.

If α is too large, may pass too much negative signal, reducing

performance.

Still not zero-centered (outputs mostly positive).

 PReLU (Parametric Rectified Linear Unit)

PReLU (Parametric Rectified Linear Unit) is an improved version of

Leaky ReLU. Unlike Leaky ReLU, where the negative slope (α) is a
fixed constant, PReLU learns α automatically during training through
backpropagation.
 PReLU (Parametric Rectified Linear Unit)

Instead of manually choosing how much the negative side

“leaks,” the network learns the best α value from the data
itself. This makes PReLU more flexible and adaptive than ReLU
or Leaky ReLU.
PReLU (Parametric Rectified Linear Unit)
PReLU (Parametric Rectified Linear Unit)

So, what is alpha ?

 PReLU (Parametric Rectified Linear Unit)

Advantages

➢ Automatically learns the best slope α during training.

➢ Solves the dying ReLU problem.

➢ Can adapt to different layers or neurons, improving accuracy.

➢ Often leads to faster convergence and better performance.

 PReLU (Parametric Rectified Linear Unit)

Working Mechanism

➢ For positive inputs, behaves like ReLU (output = input).

➢ For negative inputs, multiplies input by a trainable slope (α).

➢ During training, α adjusts itself to minimize the overall loss.

PReLU (Parametric Rectified Linear Unit)

𝑥, 𝑥>0
𝐹 𝑥 =
𝑎 𝑥 , 𝑥≤0
 PReLU (Parametric Rectified Linear Unit)

Limitations
Introduces extra parameters (α values) slightly more computation.
Risk of overfitting if many α values are learned independently.
Still not zero-centered.
 ELU(Exponential Linear Unit)
➢ ELU is an activation function designed to fix two major issues of
ReLU
➢ Dying neurons (when ReLU outputs only zeros for negative
inputs).
➢ Non–zero-centered outputs (which can slow down learning).
➢ ELU introduces an exponential curve for negative inputs to keep
small negative outputs instead of zeros.
 ELU(Exponential Linear Unit)

𝑥, 𝑥>0
𝐹 𝑥 =
𝑎 𝑒𝑥 − 1 , 𝑥≤0
where 𝜶 is a positive constant (usually α = 1).
 ELU(Exponential Linear Unit)

ELU allows small negative outputs instead of hard zeros,

which keeps the mean activations closer to zero-centered

improving learning speed and reducing bias shift.

 ELU(Exponential Linear Unit)
Advantages

➢ Solves dying ReLU problem, neurons continue to learn for x ≤ 0.

➢ Produces zero-centered outputs, improving convergence.

➢ Smooth gradient, avoids sudden jumps.

➢ Improves training speed and accuracy.

 ELU(Exponential Linear Unit)
Limitations
➢ Slightly slower computation (because of exponential term).
➢ Needs a proper choice of α (usually α=1).
➢ If α is too large, outputs can become unstable.
➢ Can cause vanishing gradients for large negative x values (since
e^x becomes very small)