Different activation functions and when to choose which
Short summary (purpose)
Activation functions introduce non-linearity so a network can learn complex mappings.
Common activations, properties, and when to use them
Identity / Linear
o f(x)=x — use in output layer for regression (predicting real values).
Sigmoid ([Link])
o Range (0,1). Good for binary probabilities (but prefer BCEWithLogitsLoss with
raw logits). Problems: vanishing gradients for large magnitude inputs. Use
only on single-output probability outputs.
Tanh ([Link])
o Range (-1,1). Zero-centered (better than sigmoid). Still suffers from vanishing
gradients for deep nets.
ReLU ([Link])
o max(0,x). Fast, simple, works well in many situations. Can suffer dead
neurons (outputs become 0).
LeakyReLU / ParametricReLU ([Link])
o Small slope for negative inputs (prevents dead neurons). Use when ReLU
leads to many zero activations.
ELU / SELU
o Smooth, sometimes improves learning; SELU used with special initialization
and architecture (for self-normalizing nets).
Softmax ([Link])
o Multi-class probability distribution across classes (use in output for multi-class
classification; pair with CrossEntropyLoss expects raw logits so don't apply
softmax before CrossEntropyLoss).
GELU
o Used in Transformers; smooth version of ReLU with probabilistic intuition.
Practical tip
Hidden layers: ReLU (or LeakyReLU) as default.
Output layer: Identity for regression, Sigmoid/BCEWithLogitsLoss for binary,
CrossEntropyLoss with raw logits for multi-class (no softmax applied manually).
Exact step-by-step understanding of how [Link] is subclassed to make model
architecture (complete Python code)
Key ideas
Create a class inheriting from [Link].
Define layers in __init__.
Implement forward pass in forward(self, x) — PyTorch uses forward to compute
outputs and __call__ to handle hooks/backprop.
Minimal example: small MLP for binary classification
import torch
import [Link] as nn
import [Link] as F
class SimpleMLP([Link]):
def __init__(self, input_dim, hidden_dim, output_dim):
super(SimpleMLP, self).__init__() # init base class
# define layers
self.fc1 = [Link](input_dim, hidden_dim)
self.fc2 = [Link](hidden_dim, hidden_dim)
self.fc_out = [Link](hidden_dim, output_dim)
[Link] = [Link](0.2)
# initialize weights optionally
[Link].kaiming_uniform_([Link], nonlinearity='relu')
def forward(self, x):
# x: [batch_size, input_dim]
x = [Link](self.fc1(x)) # activation after layer
x = [Link](x)
x = [Link](self.fc2(x))
logits = self.fc_out(x) # raw outputs (logits)
return logits
# usage
model = SimpleMLP(input_dim=10, hidden_dim=64, output_dim=1)
print(model)
Why in __init__ and not forward
Layers are created in __init__ so that PyTorch can register parameters and move
them with .to(device) and include them in optimizers.
forward contains only computations (can reuse layers multiple times).
Full training loop skeleton (shows how forward integrates)
device = [Link]("cuda" if [Link].is_available() else "cpu")
model = SimpleMLP(10, 64, 1).to(device)
optimizer = [Link]([Link](), lr=1e-3)
criterion = [Link]() # for binary; expects logits
for epoch in range(epochs):
[Link]()
for X_batch, y_batch in train_loader:
X_batch, y_batch = X_batch.to(device), y_batch.to(device).float().unsqueeze(1)
optimizer.zero_grad() # clear gradients
logits = model(X_batch) # forward
loss = criterion(logits, y_batch)
[Link]() # compute gradients
[Link]() # update parameters
Optimizers, their role and effect of learning rate (lr) [hyperparameter]
Role of optimizer
The optimizer uses gradients (from [Link]()) to update model parameters.
Different optimizers use different strategies (momentum, adaptive moments, etc.).
Common optimizers & when to use
SGD ([Link])
o Simple, may require careful lr tuning. Use momentum (momentum=0.9) for
faster convergence.
SGD + Momentum / Nesterov
o Good general-purpose when you want stable convergence and control.
RMSprop ([Link])
o Adaptive per-parameter learning rates; used historically for RNNs.
Adam ([Link])
o Works well out-of-the-box for many problems. Fast convergence.
AdamW ([Link])
o Adam with decoupled weight decay — preferred for transformer-style models
and modern training.
AdaGrad, Adadelta — less common now.
Learning rate (lr) effects
Too large lr: training diverges (loss goes up, gradients explode).
Too small lr: training is slow; may get stuck in poor minima.
Rule of thumb: start with lr=1e-3 for Adam; 0.1–0.01 for SGD depending on scale.
Use learning rate schedulers: StepLR, ExponentialLR, ReduceLROnPlateau,
CosineAnnealing.
Example: optimizer + scheduler
optimizer = [Link]([Link](), lr=1e-3, weight_decay=1e-2)
scheduler = [Link].lr_scheduler.ReduceLROnPlateau(optimizer, mode='min',
patience=3, factor=0.5)
# After each validation epoch:
val_loss = ...
[Link](val_loss) # reduces lr when val_loss plateaus
Practical tip
Use learning rate finder (e.g., fastai approach) to find a good initial lr. If not available,
run small experiments and watch loss curves.
Pre-processing with scaling (how to choose, what they do, why?)
Why scale?
Many ML algorithms assume features on similar scales. Unscaled data can cause slow
or unstable training.
Neural nets: scaling improves gradient behavior and convergence.
Common scalers (scikit-learn)
StandardScaler: (x - mean) / std. Use when features are roughly Gaussian. Centers
data to zero mean.
MinMaxScaler: scales to a range [0,1] (or custom). Use when you want all features in
same bounded interval.
RobustScaler: uses median and IQR — robust to outliers.
MaxAbsScaler: scales by max absolute value, useful for sparse data.
Choosing a scaler
If outliers: RobustScaler.
If activation uses ReLU and data positive: MinMaxScaler can be fine.
If using BatchNorm or activation that benefits from zero mean: StandardScaler.
Example (sklearn)
from [Link] import StandardScaler, MinMaxScaler, RobustScaler
import numpy as np
X = [Link]([[1, 200], [2, 300], [3, 400]], dtype=float)
scaler = StandardScaler().fit(X)
X_scaled = [Link](X)
Important: fit on training data only
Fit scaler on training set, transform train/val/test using same scaler to avoid leakage.
PyTorch works on "tensors" [WHAT IS A TENSOR?]
Definition
A tensor is a generalization of scalars (0D), vectors (1D), and matrices (2D) to potentially
higher dimensions. It's the basic data structure in PyTorch, holding data and optionally
gradient information.
Dimensions explained
0D: scalar — [Link](3.0) (shape [Link]([]))
1D: vector — [Link]([1,2,3]) (shape [3])
2D: matrix — shape [batch_size, features] or [H, W] for images
3D: sequence data ([seq_len, batch, features]) or [channels, H, W] for images in CHW
4D: batch of images [batch_size, channels, H, W]
Example shapes
Tabular batch: [batch, num_features]
Image batch (PyTorch default): [batch, channels, height, width]
Reshaping — why and how?
Reshape when layers expect certain shapes (e.g., linear expects [batch, features]).
Methods: .view(), .reshape(), .permute(), .unsqueeze(), .squeeze().
o view requires contiguous memory; reshape is safer.
o permute changes order of axes (useful for NHWC <-> NCHW).
o unsqueeze(dim) adds a dimension; squeeze(dim) removes dimension of size 1
Examples
x = [Link](32, 3, 28, 28) # [batch, ch, H, W]
x_flat = [Link](32, -1) # flatten to [batch, features]
x_perm = [Link](0,2,3,1) # to [batch, H, W, ch]
x_unsq = [Link](10).unsqueeze(1) # shape [10,1]
requires_grad and zero_grad
requires_grad=True marks a tensor to track operations for gradient computation.
Model parameters have requires_grad=True by default.
optimizer.zero_grad() or model.zero_grad() clears accumulated gradients before
backprop for the current step.
If you don't zero gradients, they accumulate across .backward() calls (useful for
gradient accumulation but often unintended).
Example showing grad attr
x = [Link]([2.0], requires_grad=True)
y = x**2
[Link]() # dy/dx = 2x -> 4
print([Link]) # tensor([4.])