First Activation: Step Function

For 𝑥 < 0:
•the function is constant
• Step function: •derivative is 0

For 𝑥 > 0:
•the function is constant
•derivative is 0

So everywhere except the threshold, the

derivative is zero.

Problems with Step Activation

• Not continuous: The function has an abrupt jump (discontinuity) at the threshold (usually zero).

• Not differentiable: Because of this sharp jump, it does not have a defined derivative at the threshold point.

• Gradient is zero everywhere else: The function's output is constant in the regions before and after the threshold (e.g., 0 for
inputs < 0 and 1 for inputs > 0). The derivative (gradient) of a constant is zero.

• Learning via gradient descent is impossible: Since the gradient is zero everywhere, the backpropagation algorithm receives no
useful information to update the network's weights.
 First Activation: Step Function

For 𝑥 < 0:
•the function is constant
• Step function: •derivative is 0

For 𝑥 > 0:
•the function is constant
•derivative is 0
So in particular, 𝑓 0 = 1
So everywhere except the threshold, the
derivative is zero.

Substitute f 𝑥0 = 1:
𝑓 ℎ −1 Evaluate the left hand limit 𝒙 → 𝟎−
𝑓 ′ 0 = lim If 𝑥 < 0, then:
ℎ→0 ℎ
𝑓 𝑥 =0
This limit must exist and have a single finite value.
Evaluate the right hand limit 𝐱 → 𝟎+ Substitute:
To check differentiability at 𝑥 = 0, we look at slopes from both If x > 0, then x ≥ 0, so: 𝑓 𝑥 − 1 0 − 1 −1
= =
sides 𝑓 𝑥 =1 ℎ ℎ ℎ
Substitute: Now take the limit:
𝑓 𝑥 −1 1−1 0 • 𝑥 → 0− means 𝑥 is negative and approaching
= = =0 zero
ℎ ℎ ℎ
Therefore, • −1/ℎ becomes a large positive number
𝑓 𝑥 −1 So,
lim + =0 −1
ℎ→0 ℎ lim − = +∞
ℎ→0 ℎ
RHS is finite (0); LHS diverges to infinity; The two limits are not equal.
𝑓′ 𝑥 does not exist at x=0
 Sigmoid Activation Function

Sigmoid Activation Function is smooth version of step

function characterized by ‘S’ shape
1
• σ 𝑥 = 𝑠𝑖𝑔𝑚𝑜𝑖𝑑 𝑥 =
1+𝑒 −𝑥
𝑑
• σ′ 𝑥 = σ 𝑥 = σ 𝑥 (1 − σ 𝑥 )
𝑑𝑥 Sigmoid Fails in Deep Networks
Sigmoid derivative: Always < 0.25
In deep networks:
Why Exponentiation? Any other base would Gradients multiply, so Gradients shrink
Ensures smooth curve technically work. But base 𝑒 exponentially
Always positive gives:
Monotonic growth • simplest derivative
• clean relationship with
Result: Early layers stop learning
maximum likelihood
 Sigmoid Activation Function

Sigmoid Activation Function is smooth version of step

function characterized by ‘S’ shape
1
• σ 𝑥 = 𝑠𝑖𝑔𝑚𝑜𝑖𝑑 𝑥 =
1+𝑒 −𝑥 a binary event 𝑦 ∈ 0 1 and a probability 𝑝 = 𝑃 𝑦
𝑑
• σ′ 𝑥 = 𝑑𝑥
σ 𝑥 = σ 𝑥 (1 − σ 𝑥 )
Odds measure how much more likely the event is to happen than not happen:
𝑝
Why Exponentiation? Any other base would odds =
1−𝑝
Ensures smooth curve technically work. But base 𝑒
Always positive gives: Take the logarithm: Solve for 𝑝:
𝑝 𝑝
Monotonic growth • simplest derivative logit 𝑝 = log
1−𝑝 𝑥 = log
• clean relationship with 1−𝑝
The range is: − ∞ + ∞ Exponentiate:
maximum likelihood This is crucial because: 𝑝
• probabilities are bounded 𝑒𝑥 =
1−𝑝
Sigmoid Fails in Deep Networks • linear models produce unbounded Rearrange:
Sigmoid derivative: Always < 0.25 outputs 1
So we have: 𝑝=
In deep networks: 1 + 𝑒 −𝑥
𝑝 That is the sigmoid.
Gradients multiply, so 𝑥 = log
Gradients shrink exponentially 1−𝑝
Result: Early layers stop learning This says: a linear score equals log odds.
 Sigmoid Fails in Deep Networks
Sigmoid derivative: Always < 0.25
Sigmoid Activation Function In deep networks:
Gradients multiply, so
Gradients shrink exponentially
1 Result: Early layers stop learning
• σ 𝑥 = 𝑠𝑖𝑔𝑚𝑜𝑖𝑑 𝑥 =
1+𝑒 −𝑥
𝑑 In a deep network, backprop multiplies derivatives
• σ′ 𝑥 = σ 𝑥 = σ 𝑥 (1 − σ 𝑥 )
𝑑𝑥

Maximum possible gradient

Let:
𝑔 𝑥 = 𝜎 𝑥 1−𝜎 𝑥 Each sigmoid derivative satisfies: 0 < 𝜎 ′ 𝑥 ≤ 0.25
This is a concave function in 𝜎. So the product satisfies:
Maximum occurs at:
𝜎 𝑥 = 0.5
So:
max 𝜎 ′ 𝑥 = 0.25
That means: even in the best case, sigmoid shrinks “sigmoid saturates for |x| > 4” : Its not a magic constant.
gradients by at least 4× 𝜎 4 ≈ 0.982 ⇒ 𝜎 ′ 4 ≈ 0.018
𝜎 5 ≈ 0.993 ⇒ 𝜎 ′ 5 ≈ 0.0067
𝜎 6 ≈ 0.998 ⇒ 𝜎 ′ 6 ≈ 0.0025
Exponential decay in the tails By the time ∣ 𝑥 ∣≥ 4:
Consider large positive 𝑥: • output is almost 0 or 1
𝜎 𝑥 ≈ 1 ⇒ 𝜎 ′ 𝑥 ≈ 𝑒 −𝑥 • derivative is almost zero
For large negative 𝑥: • learning is effectively gone
𝜎 𝑥 ≈ 0 ⇒ 𝜎′ 𝑥 ≈ 𝑒 𝑥
So: Here “4” means: beyond this scale, gradients are numerically useless
𝜎 ′ 𝑥 → 0 exponentially fast
This is not linear decay. It is exponential collapse
 Sigmoid Activation Function

Sigmoid is still used:

•Binary classification output layer
•Probabilistic decision modeling
•Medical risk prediction
•Calibrated probability outputs

Advantages Disadvantages
• It allows neural networks to model • Prone to exponential decay problem for large or small
complex patterns that linear equations inputs. (the derivatives are less than 1 for most input
cannot. values. Multiplying many such small derivatives causes the
gradient to shrink exponentially as it moves backward
• Converts inputs into probabilities. through layers.)
• Outputs are not zero-centered. (The optimization
• The output ranges between 0 and 1, hence algorithm may take longer to converge because the
useful for binary classification. updates are not symmetrically balanced, which can lead to
oscillations or inefficiency.)
 tanh function (hyperbolic tangent function)-1990s
Need was:
It was introduced to fix sigmoid’s optimization bias. • zero-centered output
Problem: Non zero-centered activations • symmetric range
For sigmoid: So apply an affine transformation:
𝔼𝜎 𝑧 >0 𝒈 𝒙 = 𝒂 ⋅ 𝝈 𝒃𝒙 + 𝒄
That means: 𝜎(⋅) sigmoid
•Activations are always positive b Horizontal scaling : Controls steepness of the curve
•Gradients accumulate bias in one direction 𝑎 Vertical scaling: Controls output range size
•If 𝑥>0 then Δw has consistent sign bias c Vertical shift : Controls where the midpoint lands

1. Midpoint should be Choose constants so that:

2. Output range should (-1,1)
zero • output range becomes (-1, 1). 𝔼 𝜎 . ≅ 0
Original sigmoid range is (0,1)
Since 𝜎 0 = 0.5, • midpoint shifts from 0.5 to 0. Positive and
After scaling and shifting:
𝑔 0 = 𝑎 ⋅ 0.5 + 𝑐 = 0 negative gradients balance
•Minimum: 𝑎 ⋅ 0 + 𝑐 = 𝑐
So: •Maximum: 𝑎 ⋅ 1 + 𝑐 = 𝑎 + 𝑐
𝑎
𝑐=− We want:
2 𝑐 = −1
𝑎+𝑐 =1 3. Choose 𝒃 Does not affect centering or
Solving gives: range. It only controls slope near zero.
𝑎 = 2, 𝑐 = −1 Choosing 𝑏 = 2 makes the slope at the
origin
tanh function (hyperbolic tangent function)-1990s

Recognition by hyperbolic functions (not redefine).

Mathematical Definition
Substitute sigmoid definition:
1
2 −1
1 + 𝑒 −2𝑥
Put everything over a common denominator:
2 − 1 + 𝑒 −2𝑥
=
1 + 𝑒 −2𝑥
Simplify numerator:
1 − 𝑒 −2𝑥
=
1 + 𝑒 −2𝑥

Now multiply numerator and denominator by 𝑒 𝑥 :

𝑒 𝑥 − 𝑒 −𝑥
= 𝑥
𝑒 + 𝑒 −𝑥
tanh function (hyperbolic tangent function)-1990s

Recognition by hyperbolic functions (not redefine).

Mathematical Definition
Substitute sigmoid definition:
1
2 −1
1 + 𝑒 −2𝑥
Put everything over a common denominator:
2 − 1 + 𝑒 −2𝑥
=
1 + 𝑒 −2𝑥
Simplify numerator:
1 − 𝑒 −2𝑥
=
1 + 𝑒 −2𝑥

Now multiply numerator and denominator by 𝑒 𝑥 :

𝑒 𝑥 − 𝑒 −𝑥
= 𝑥
𝑒 + 𝑒 −𝑥

Why multiply by 𝑒 𝑥 ?
• Removes negative exponents
• Balances powers of 𝑒 𝑥 and 𝑒 −𝑥
• Reveals odd symmetry around zero
This is algebraic normalization
 tanh function (hyperbolic tangent function)-1990s
Maximum derivative at 𝑥 = 0: tanh’ 0 = 1
Compare with sigmoid: 𝜎 ′ 0 = 0.25
This alone made tanh much easier to train than sigmoid.

Step 1: Let

Step 2: Apply the quotient rule

𝑑 𝑢 𝑢′ 𝑣 − 𝑢𝑣 ′
=
𝑑𝑥 𝑣 𝑣2
Compute derivatives:
𝑢′ = 𝑒 𝑥 + 𝑒 −𝑥
Let 𝑎 = 𝑒 𝑥 , 𝑏 = 𝑒 −𝑥 :
𝑣 ′ = 𝑒 𝑥 − 𝑒 −𝑥
ቀ𝑒 𝑥 + 𝑒 −𝑥 )2 − ൫𝑒 𝑥 − 𝑒 −𝑥 )2 = 4
So:

Step 3: Simplify the numerator

Using identities:
Step 4: Rewrite in terms of tanh
ቀ𝑎 + 𝑏)2 − ൫𝑎 − 𝑏)2 = 4𝑎𝑏
Then:
 tanh function (hyperbolic tangent function)-1990s

What Problem tanh Actually Solved Why tanh Still Suffers from decay?
• Solved problem at derivative:
• Non zero-centered activations
• Gradient bias in updates For large |x|:
• Faster convergence than sigmoid
• Better conditioning of Hessian
In deep networks:
• Did NOT solve
• Vanishing gradient
• Saturation at large |x| Same issue as sigmoid.

Use tanh when:

Do NOT use tanh:
•Inputs are zero-centered
•In very deep feedforward networks
•Features contain signed information
•When gradients must flow across many layers
•Network is shallow or moderately deep
•When training speed matters more than smoothness
•You want symmetric activations
 ReLU function (Rectified Linear Unit)- 2010
Sigmoid and Tanh had exponential decay (vanishing ReLU definition
ReLU 𝑥 = max 0 𝑥
gradient), and no optimizer can fix it at this time.
Derivative
0 𝑥<0
So Researchers, ReLU′ 𝑥 = ቊ
1 𝑥>0
stopped focusing on: “What curve looks smooth?” This derivative was the real breakthrough.
started focusing on: “What derivative keeps gradients alive?”

Requirements ReLU was designed to satisfy ReLU Solved the Core Problem of Gradient behavior
•Positive region:
Need of an activation 𝑓 𝑥 such that:
ReLU′ 𝑥 = 1 ⇒ no gradient decay
[Link]-linear •No saturation for 𝑥 > 0
[Link] cheap •Deep networks now trainable for more layers
[Link] = constant (or large) over wide region
[Link] saturation for positive inputs Sparsity
ReLU outputs zero for half the space:
𝑃 ReLU 𝑥 = 0 ≈ 0.5
Deriving ReLU Effects:
•Sparse activations
Simplest non-saturating Suppress negative signals
•Better generalization
function
•Faster computation
𝑓 𝑥 =𝑥
Derivative:
𝑓′ 𝑥 = 1
 ReLU function (Rectified Linear Unit)

Leaky ReLU: The First Fix

• What ReLU Did NOT Solve
• Non-differentiable at zero Allow small gradient for negative inputs.
• The derivative at 𝑥 = 0 is undefined. (handled in practice as 0 or 1)

• Zero gradient for negative region

• For 𝑥 < 0: ReLU′ 𝑥 = 0. This is the real problem.
• So Output = 0, Gradient = 0, Weights do not update

• Can permanently deactivate neurons

• A neuron dies when: 𝑧 = 𝑊𝑥 + 𝑏 ≪ 0 for all inputs
• Then: ReLU(z) = 0 always ; ReLU′(z) = 0 always ;
• No gradient ever reaches 𝑊, 𝑏

• This leads to the dead neuron problem.

tanh saturation → gradients are small but nonzero

ReLU negative region → gradients are exactly zero
 ReLU function (Rectified Linear Unit)
ELU (Exponential Linear Unit): The Zero Mean Fix

Leaky ReLU: The First Fix ReLU variants still: Produce non-zero mean ELU fixes this.

Allow small gradient for negative inputs. • The output of a ReLU-activated layer will always have a positive mean.
• When you feed this +ve mean data into the next layer, the subsequent
neurons receive biased inputs (again sign bias issue but now in inputs).
• This can lead to all, or most of the weights in that layer updating in the
same direction.

Parametric ReLU (PReLU): The Learnable Fix

Why fix 𝛼 manually, let model decide

Trade-off
• More flexibility
• More parameters
 ReLU function (Rectified Linear Unit)

A(x)=max(0,x)

Range: [0,∞)

• The most commonly used activation functi on

in hidden layers.

Advantages: Disadvantages
• ReLU is very fast to compute, as it only requires a max(0, • Can result in "dead neurons" (neurons that output 0 for all inputs).
x) operation • Not Zero-Centered
• Mitigates Vanishing Gradient
• Since ReLU sets negative inputs to zero, it produces
sparse representations, which can lead to better
generalization and reduced overfitting.
 Softmax

• The softmax function, often used in the final layer of a neural

network model for classification tasks.
• It converts raw output scores into probabilities by taking the
exponential of each output and normalizing these values by
dividing by the sum of all the exponentials.
• This process ensures the output values are in the range (0,1)
and sum up to 1, making them interpretable as probabilities.

Cons of softmax
Pros of softmax 1. Softmax often outputs very high probabilities even when the model is uncertain.
[Link] acceptable probability distributions 2. Large values can cause numerical overflow without stabilization tricks like
[Link] subtracting max [softmax (x- c) where c = max (xi) ].
[Link] relative ranking of classes 3. Poor uncertainty estimation: High probability does not mean high confidence in
[Link] outputs real world sense.
4. Class competition: Increasing one class probability decreases all others, which is
bad for multi label tasks.

Sparsemax-Unlike softmax, it produces sparse outputs where irrelevant classes receive probability 0.
Entmax - Sparsemax is often too sparse. Softmax is never sparse. Entmax generalizes both and lets you choose sparsity level.
Function Output range Differentiable Zero centered Sparsity Main use Key problems
Scenario Best choice
Not Perceptron theory Step
Classical
Step {0, 1} No No High differentiable,
perceptron
no gradients Binary probability output Sigmoid
Binary Vanishing
Sigmoid (0, 1) Yes No None classificationgradients, not Zero centered hidden
output zero centered Tanh
layers (legacy)
Hidden layers Vanishing
Tanh (-1, 1) Yes Yes None
(older nets) gradients Deep CNNs and MLPs ReLU
Almost Default hidden
ReLU [0, ∞) No High Dying ReLU Avoid dead neurons Leaky ReLU
everywhere layers
Almost Negative slope
Leaky ReLU (-∞, ∞) No Low Improved ReLU Multi class classification Softmax
everywhere tuning
Multi class Overconfidence
Softmax (0, 1), sum = 1 Yes No None
output , dense

Function Gradient near 0 Gradient for large |x|

Step Undefined 0

Sigmoid Small →0

Tanh Moderate →0

ReLU 1 (x > 0) Constant

Leaky ReLU α (x < 0) Constant

[Link]
Softmax Coupled across classes Can saturate