Activation Functions in Deep
Learning
Beginner-Friendly Explanation
What is an Activation Function?
• • A mathematical function applied to a
neuron’s output
• • Decides whether the neuron should be
activated or not
• • Introduces non-linearity so the network can
learn complex patterns
Why Do We Need Activation
Functions?
• • Without them, neural networks behave like
simple linear models
• • Help model complex data like images,
speech, and text
• • Enable deep networks to work effectively
Types of Activation Functions
• 1. Linear Activation Function
• 2. Sigmoid
• 3. Tanh
• 4. ReLU (Rectified Linear Unit)
• 5. Leaky ReLU
• 6. Softmax
Sigmoid Function
• • Formula: 1 / (1 + e^-x)
• • Output range: (0,1)
• • Used for binary classification
• • Problem: Vanishing gradient in deep
networks
Tanh Function
• • Formula: (e^x - e^-x)/(e^x + e^-x)
• • Output range: (-1,1)
• • Zero-centered but still suffers from vanishing
gradients
ReLU Function
• • Formula: f(x) = max(0,x)
• • Most widely used in deep learning
• • Pros: Simple, avoids vanishing gradient for
positive values
• • Cons: Dying ReLU problem (neurons output
0 always)
Leaky ReLU
• • Formula: f(x) = x if x>0 else αx (small α)
• • Fixes dying ReLU problem by allowing small
negative values
Softmax Function
• • Converts outputs into probability
distribution
• • Used in multi-class classification problems
How to Choose Activation
Functions?
• • Hidden layers: ReLU or Leaky ReLU
commonly used
• • Binary classification: Sigmoid in the output
layer
• • Multi-class classification: Softmax in the
output layer