DEEP LEARNING
LOSS FUNCTIONS COMPLETE REFERENCE
Comprehensive Formula List with Properties & Applications
This comprehensive reference document provides all standard loss functions used in deep learning networks. Each
loss function includes mathematical formulas, properties, advantages/disadvantages, and practical applications.
CONTENTS:
Section 1: Regression Loss Functions
Section 2: Classification Loss Functions
Section 3: Specialized Loss Functions
Section 4: Custom & Weighted Loss Functions
Section 5: Comparison Table & Selection Guide
Section 6: Quick Reference Cheat Sheet
Last Updated: 2025 | For BTech & Advanced Courses
SECTION 1: REGRESSION LOSS FUNCTIONS
1.1 Mean Squared Error (MSE) / L2 Loss
Formula: L_MSE = (1/n) × Sigma(y-hat - y)^2 Alternative Forms: L_MSE = (1/n) ×
||y-hat - y||_2^2 L_MSE = E[(y-hat - y)^2]
Property Description
Range [0, infinity)
Symmetry Penalizes positive & negative errors equally
Outlier Sensitivity Very sensitive (quadratic penalty)
Differentiability Differentiable everywhere
Gradient dL/dy-hat = 2(y-hat - y)/n
Scale Squared units of target variable
Advantages:
- Mathematically convenient (smooth gradient)
- Fast computation
- Well-understood statistical properties
Disadvantages:
- Heavily penalizes outliers (not robust)
- Can be dominated by few large errors
Use Cases: House prices, Temperature, Stock prices, Linear regression
1.2 Mean Absolute Error (MAE) / L1 Loss
Formula: L_MAE = (1/n) × Sigma |y-hat - y| Alternative Forms: L_MAE = (1/n) ×
||y-hat - y||_1 L_MAE = E[|y-hat - y|]
Property Description
Range [0, infinity)
Symmetry Penalizes all errors equally
Outlier Sensitivity Robust to outliers (linear penalty)
Differentiability Not differentiable at y=y-hat
Gradient dL/dy-hat = sign(y-hat - y)/n
Scale Same units as target variable
Advantages: Robust to outliers, Interpretable, Stable training
Disadvantages: Not differentiable at zero, Slower convergence
Use Cases: Robust regression, Anomaly detection, Datasets with outliers
1.3 Huber Loss (Smooth L1)
Formula: L_Huber = { (1/2)(y-hat - y)^2 if |y-hat - y| <= delta delta|y-hat - y| -
(1/2)delta^2 if |y-hat - y| > delta } Where: delta: threshold parameter (typically
1.0) Combines MSE (small errors) + MAE (large errors)
Advantages: Combines MSE and MAE benefits, Robust to outliers, Smooth gradient
Use Cases: Datasets with some outliers, When both smoothness and robustness needed
1.4 Root Mean Squared Error (RMSE)
Formula: L_RMSE = sqrt[(1/n) × Sigma(y-hat - y)^2] Relationship: L_RMSE =
sqrt(L_MSE) Properties: Same units as target variable Always non-negative More
interpretable than MSE
Use Cases: Model evaluation, When interpretability is important
1.5 Mean Absolute Percentage Error (MAPE)
Formula: L_MAPE = (100/n) × Sigma |y - y-hat| / |y| Properties: Scale-independent
(percentage error) Only applicable when y != 0 Range: [0, infinity) in percent
Use Cases: Forecasting, Sales prediction, Demand estimation
SECTION 2: CLASSIFICATION LOSS FUNCTIONS
2.1 Binary Cross-Entropy (BCE) / Log Loss
Formula: L_BCE = -(1/n) × Sigma[y × log(y-hat) + (1-y) × log(1-y-hat)] Individual
Loss: L_BCE(i) = -[y × log(p) + (1-y) × log(1-p)] where p = sigmoid(z) is predicted
probability Activation: Sigmoid σ(z) = 1/(1+e^(-z))
Advantages: Standard for binary classification, Probabilistically motivated
Use Cases: Binary classification, Disease detection, Spam detection
2.2 Categorical Cross-Entropy (CCE)
Formula (Multi-class): L_CE = -(1/n) × Sigma Sigma y_ij × log(y-hat_ij) For Single
Sample: L_CE = -Sigma y_j × log(p_j) Properties: y_ij: one-hot encoded true label
(binary) y-hat_ij: predicted probability for class j c: number of classes Softmax
activation: p_j = e^z_j / Sigma e^z_k
Advantages: Standard for multi-class problems, Works naturally with softmax
Use Cases: Image classification, Object recognition, Multi-class problems
2.3 Sparse Categorical Cross-Entropy
Key Difference: - Labels are integer class indices (not one-hot) - Example: y = [0,
2, 1] instead of [[1,0,0], [0,0,1], [0,1,0]] - Mathematically equivalent to CE -
More memory efficient for many classes Use Cases: Large number of classes, Memory
constraints
2.4 Focal Loss (for Imbalanced Data)
Formula: L_Focal = -(1/n) × Sigma alpha_t × (1 - p_t)^gamma × log(p_t) Where: p_t:
predicted probability for true class alpha (α): weighting factor (typically 0.25)
gamma (γ): focusing parameter (typically 2.0) (1 - p_t)^γ: modulating factor
down-weights easy examples Effect: Reduces weight of easy examples Focuses on hard,
misclassified examples Better for imbalanced datasets
Advantages: Addresses class imbalance, Focuses on hard examples
Use Cases: Class imbalanced data, Object detection, Rare disease detection
2.5 Hinge Loss / SVM Loss
Formula (Binary): L_Hinge = (1/n) × Sigma max(0, 1 - y × y-hat) Multiclass Hinge:
L_Hinge = (1/n) × Sigma Sigma max(0, 1 + s_j - s_y_i) Where: y in {-1, +1}: true
class labels y-hat: raw scores (not probabilities) margin: typically 1.0
Advantages: Does not require probability outputs, Margin-based optimization
Use Cases: SVMs, Classification with raw scores, Maximum margin classifiers
SECTION 3: SPECIALIZED LOSS FUNCTIONS
3.1 Triplet Loss (Metric Learning)
Formula: L_Triplet = max(0, d(a, p) - d(a, n) + margin) Where: a: anchor sample p:
positive sample (same class) n: negative sample (different class) d: distance
metric (usually L2 Euclidean) margin: minimum separation (typically 1.0) Objective:
Minimize distance between similar samples Maximize distance between dissimilar
samples Learn metric space directly
Advantages: Learns metric space directly, Scalable to many classes
Use Cases: Face recognition, Person re-identification, Metric learning
3.2 Contrastive Loss (Siamese Networks)
Formula: L_Contrastive = (1 - y) × (1/2) × d^2 + y × (1/2) × max(0, m - d)^2 Where:
d = ||f(x) - f(x')||_2: Euclidean distance y: 0 if similar, 1 if dissimilar m:
margin for dissimilar pairs (typically 1.0) Effect: If similar (y=0): minimize
distance If dissimilar (y=1): maximize distance beyond margin
Use Cases: Siamese Networks, Signature verification, One-shot learning
3.3 Dice Loss / F1 Loss
Formula: L_Dice = 1 - (2 × |X ∩ Y| + epsilon) / (|X| + |Y| + epsilon) Pixel-wise
Version: L_Dice = 1 - (2 × Sigma(y × y-hat) + eps) / (Sigma y + Sigma y-hat + eps)
Where: epsilon: small constant (typically 1) to avoid division by zero Similar to
Sorensen-Dice coefficient
Advantages: Robust to class imbalance, Focus on intersection
Use Cases: Semantic segmentation, Medical image analysis, Imbalanced classification
3.4 Intersection over Union (IoU) / Jaccard Loss
Formula: L_IoU = 1 - |X ∩ Y| / |X ∪ Y| Expanded: L_IoU = 1 - Sigma(y × y-hat) /
(Sigma y + Sigma y-hat - Sigma(y × y-hat)) Properties: Jaccard Index = 1 - IoU Loss
Range: [0, 1] Higher IoU = Better segmentation
Use Cases: Object detection, Image segmentation, Instance segmentation
3.5 Kullback-Leibler (KL) Divergence
Formula: L_KL = (1/n) × Sigma Sigma p_j × log(p_j / q_j) Simplified for
Classification: L_KL = (1/n) × Sigma Sigma y_j × [log(y_j) - log(y-hat_j)]
Properties: p: true probability distribution q: predicted probability distribution
Measures distribution divergence Non-symmetric (D_KL(p||q) != D_KL(q||p))
Use Cases: Knowledge distillation, Distribution matching, VAE training
SECTION 4: WEIGHTED & CUSTOM LOSS FUNCTIONS
4.1 Weighted Cross-Entropy Loss
Formula: L_WCE = -(1/n) × Sigma Sigma w_j × y_ij × log(y-hat_ij) Binary Version:
L_WBCE = -(1/n) × Sigma [w_1 × y × log(y-hat) + w_0 × (1-y) × log(1-y-hat)] Weight
Calculation: w_j = n / (c × n_j) where n_j is count of class j Purpose: Handles
class imbalance Different misclassification costs per class
Use Cases: Imbalanced datasets, Medical diagnosis, Different error costs
4.2 Cross-Entropy with Label Smoothing
Formula: y-tilde_j = y_j × (1 - epsilon) + epsilon/c Then: L_LS = -(1/n) × Sigma
Sigma y-tilde_j × log(y-hat_j) Where: epsilon: smoothing factor (typically 0.1) c:
number of classes Replaces hard one-hot labels with soft targets Effect: Reduces
overfitting Prevents overconfident predictions Improves generalization
Use Cases: Large-scale classification, ImageNet, Preventing overconfidence
4.3 Multi-task Learning Loss
Example: Segmentation + Classification L_total = λ_1 × L_CE + λ_2 × L_Dice + λ_3 ×
L_Reg Example: Object Detection (YOLO-style) L_total = λ_1 × L_bbox + λ_2 × L_conf
+ λ_3 × L_class Typical Approach: λ_i ∈ [0, 1], Sigma λ_i = 1 Can be optimized or
manually weighted
Use Cases: Multi-task learning, Object detection, Joint segmentation+classification
SECTION 5: COMPARISON TABLE & SELECTION GUIDE
5.1 Loss Functions Quick Comparison
Loss Function Type Activation Best For
MSE Regression Linear Standard regression
MAE Regression Linear Robust regression
Huber Regression Linear Mixed outliers
BCE Binary Sigmoid Binary classification
CE Multiclass Softmax Multi-class problems
Sparse CE Multiclass Softmax Integer labels
Focal Imbalanced Softmax Class imbalance
Hinge Margin None SVM-style
Triplet Metric Embedding Metric learning
Dice Segmentation Sigmoid Image segmentation
IoU Segmentation Sigmoid Detection tasks
KL Divergence Distribution Softmax KL matching
5.2 Selection Guide by Problem Type
REGRESSION: Default choice: MSE With outliers: MAE or Huber Percentage errors: MAPE Interpretable scale:
RMSE BINARY CLASSIFICATION: Default: BCE + Sigmoid Imbalanced: Focal Loss + Sigmoid SVM-style: Hinge
Loss Weighted: Weighted BCE MULTI-CLASS CLASSIFICATION: Default: Categorical CE + Softmax Integer labels:
Sparse CE + Softmax Imbalanced: Focal Loss + Softmax With label smoothing: CE + LS + Softmax IMBALANCED
DATASETS: First try: Focal Loss Alternative: Weighted Cross-Entropy For segmentation: Dice Loss + CE METRIC
LEARNING: Face recognition: Triplet Loss Siamese networks: Contrastive Loss Embedding space: Triplet or
Contrastive SEGMENTATION: Balanced dataset: Cross-Entropy Imbalanced: Dice + CE Bounding boxes: IoU Loss
DETECTION: YOLO/RetinaNet: Focal Loss + IoU Standard: CE + IoU Loss SPECIAL: Knowledge distillation: KL
Divergence Distribution matching: KL Div or Wasserstein Multi-task: Custom weighted sum
SECTION 6: QUICK REFERENCE CHEAT SHEET
6.1 Activation Function Pairs
LINEAR activation + MSE/MAE For regression problems Output unbounded SIGMOID activation + BCE For binary
classification Output: [0, 1] probability SOFTMAX activation + CE For multi-class classification Output: probability
distribution TANH activation + MSE Hidden layers or bounded regression Output: [-1, 1] RELU activation + Custom
loss Hidden layers (not for output) Output unbounded RAW SCORES (no activation) + Hinge Loss SVM-style
classification Direct score comparison
6.2 Default Loss Choice by Task
House price prediction → MSE Disease probability → BCE Image classification → CE (1000
classes) Rare disease (imbalanced) → Focal Loss Face recognition → Triplet Loss Image
segmentation → Dice + CE Object detection → Focal + IoU Handwriting recognition → CTC Loss
Demand forecasting → MAE Stock price (with outliers) → Huber Loss
6.3 Common Mistakes to Avoid
✗ Using sigmoid + CE (use softmax + CE for multi-class) ✗ Using Hinge Loss with softmax (use raw scores) ✗
Forgetting log() in Cross-Entropy ✗ Using MSE with class probabilities (use CE instead) ✗ Not normalizing labels
before custom loss ✗ Ignoring class imbalance (use Focal or Weighted loss) ✗ Using BCE for multi-class problems ✗
Not scaling targets for regression ✗ Forgetting margin parameter in Triplet/Contrastive loss ✗ Using same weight for
all classes in imbalanced data