EMPIRICAL RISK MINIMIZATION
& REGULARIZATION METHODS
Learning Theory · Bias-Variance · L1/L2 · Dropout · Early Stopping · Data Augmentation
Based on Goodfellow, Bengio & Courville — Deep Learning (MIT Press, 2016)
Chapters 5 (ML Basics) · 7 (Regularization) · 8 (Optimization)
AGENDA
1–3 Learning Theory: Risk, ERM, and Generalization
4–5 Bias-Variance Tradeoff — Concepts & Numerical Example
6–7 L2 Regularization (Weight Decay) — Theory & Worked Example
8–9 L1 Regularization — Sparsity & Comparison with L2
10–11 Dropout — Mechanism, Theory & Example
12 Early Stopping & Dataset Augmentation
13–14 Applications: NLP, Vision, Tabular Data
15 Summary & Key Takeaways
LEARNING THEORY: RISK & EMPIRICAL RISK MINIMIZATION
"The goal of a machine learning algorithm is to reduce the expected generalization error known as the risk." — Goodfellow et
al., §5.2
True Risk R(f) Empirical Risk R̂(f) ERM Principle
R(f) = E(x,y)~p_data[ L(f(x), y) ] ERM: f* = argmin_f R̂(f)
R̂(f) = (1/m) Σᵢ L(f(xᵢ), yᵢ)
Expected loss over the true (unknown) Find the function that minimizes training
Average loss over the m training samples.
data distribution p_data. The gold loss. Justified by the Law of Large
Tractable proxy for R(f) since we can't
standard — minimizing R(f) gives the best Numbers: as m→∞, R̂(f)→R(f).
compute expectations over p_data.
model. Goodfellow §5.2
GENERALIZATION GAP, UNDERFITTING & OVERFITTING
Generalization Gap = R(f) − R̂(f) | True goal: minimize R(f), not R̂(f) (Goodfellow §5.2)
← Underfitting ✓ Sweet Spot → Overfitting
High Bias Low Bias, Low Variance High Variance
Model is too simple to capture the Model complexity matches data Model memorizes training data, fails to
underlying pattern. complexity. generalize.
Training error is HIGH. Training error is LOW. Training error is VERY LOW.
Test error is HIGH. Test error is LOW. Test error is HIGH.
Fix: increase capacity, train longer, add Goal: find this regime through Fix: regularization, more data, simpler
features. regularization. model.
BIAS-VARIANCE TRADEOFF — NUMERICAL EXAMPLE
Goodfellow §5.4 — MSE decomposition: MSE = Bias² + Variance + Irreducible Noise
MSE(θ̂) = E[(θ̂ − θ)²] = (E[θ̂] − θ)² + E[(θ̂ − E[θ̂])²] = Bias(θ̂)² + Var(θ̂)
Worked Example: Estimating μ from Gaussian data
True parameter: μ = 3.0
Estimator A (biased): θ̂_A = 0.8 × x̄ → E[θ̂_A] = 0.8 × 3.0 = 2.4
Bias_A = 2.4 − 3.0 = −0.6 → Bias²_A = 0.36
Var(θ̂_A) = 0.64 × σ²/n = 0.64 × 1.0/10 = 0.064
MSE_A = 0.36 + 0.064 = 0.424
Estimator B (unbiased): θ̂_B = x̄ → E[θ̂_B] = 3.0
Bias_B = 0 → Bias²_B = 0
Var(θ̂_B) = σ²/n = 1.0/10 = 0.100
MSE_B = 0 + 0.100 = 0.100
Conclusion: MSE_B (0.100) < MSE_A (0.424)
Unbiased estimator wins here, but biased can win
if variance reduction exceeds bias penalty.
L2 REGULARIZATION (WEIGHT DECAY) — GOODFELLOW §7.1
Objective: J ̃(w; X, y) = J(w; X, y) + (λ/2) ǁwǁ²₂
λ ≥ 0 controls regularization strength. λ=0 → standard ERM. Larger λ → stronger shrinkage toward 0.
Eigendecomposition View (Goodfellow
Gradient Update (Goodfellow §7.1) Bayesian Interpretation
§7.1)
Without L2: Optimal w* satisfies:
w ← w − α ∇_w J (H + λI) w* = H w_ML L2 penalty ≡ MAP estimation with a
Gaussian prior on weights:
With L2 (weight decay): where H = Hessian of J at w_ML
w ← w − α (∇_w J + λw) log p(w|X,y) ∝ log p(y|X,w) − (λ/2) ǁwǁ²
= (1 − αλ) w − α ∇_w J w*_i = (d_i / (d_i + λ)) w_ML_i
Maximizing MAP ≡ minimizing J ̃.
Effect: each step first shrinks w by (1−αλ), Eigenvalue d_i >> λ: weight kept λ encodes prior precision (inverse variance).
then takes gradient step. Shrinks weights Eigenvalue d_i << λ: weight shrunk to ~0 See Goodfellow §7.1 for full derivation.
toward zero. Regularizes directions of low curvature.
L2 REGULARIZATION — NUMERICAL WORKED EXAMPLE
Linear regression: f(x) = w₁x₁ + w₂x₂ + b. Three training points. λ = 0.5
x₁ x₂ y Step-by-step L2 solution
1 2 5
Design matrix X = [[1,2],[3,1],[2,3]] (ignoring bias for simplicity)
3 1 7 XᵀX = [[1,3,2],[2,1,3]]·[[1,2],[3,1],[2,3]] = [[14,11],[11,14]]
Xᵀy = [[1,3,2],[2,1,3]]·[5,7,8] = [42,39]
2 3 8
Without regularization (λ=0):
w_ML = (XᵀX)⁻¹ Xᵀy → Δ = 14²−11² = 75
w₁ = (14·42 − 11·39)/75 = (588−429)/75 = 159/75 ≈ 2.12
w₂ = (14·39 − 11·42)/75 = (546−462)/75 = 84/75 ≈ 1.12
With L2 regularization (λ=0.5):
(XᵀX + λI) = [[14.5, 11],[11, 14.5]], Δ = 14.5²−11² = 89.25
w₁* = (14.5·42 − 11·39)/89.25 = (609−429)/89.25 ≈ 2.02
w₂* = (14.5·39 − 11·42)/89.25 = (565.5−462)/89.25 ≈ 1.16
Regularization shrinks w₁: 2.12 → 2.02 and w₂: 1.12 → 1.16 (trade-off).
L1 REGULARIZATION (LASSO) — GOODFELLOW §7.1.2
Objective: J ̃(w; X, y) = J(w; X, y) + λ ǁwǁ₁ = J(w; X, y) + λ Σᵢ |wᵢ|
L1 Gradient & Sparse Solutions L1 vs L2 Comparison
Property L1 (Lasso) L2 (Ridge)
Gradient update: Penalty term λǁwǁ₁ λǁwǁ²₂ / 2
∇_w J ̃ = ∇_w J + λ sign(w)
Solution Sparse (zeros) Dense (small)
Key property: L1 penalty produces SPARSE solutions — many
weights become exactly zero. This is feature selection. Prior Laplace Gaussian
Why sparsity? The L1 ball (diamond shape) touches the loss Feature select. Yes (built-in) No
contours at corners on axes, where coordinates are exactly 0.
Differentiable? No (at 0) Yes
Goodfellow §7.1.2: L1 corresponds to a MAP estimate with a
Laplace prior: p(wᵢ) ∝ exp(−λ|wᵢ|) Optimization Subgradient/CD Closed-form
Use when Many irrelevant All features useful
features
L1 REGULARIZATION — SOFT-THRESHOLDING EXAMPLE
Soft-thresholding (coordinate descent solution for Lasso — Goodfellow §7.1.2):
w*ᵢ = sign(ŵᵢ) · max(|ŵᵢ| − λ, 0) where ŵᵢ = unconstrained (OLS) estimate of weight i
5 features, λ = 0.8
Feature OLS ŵ |ŵ| − λ w* (L1)
w₁ (relevant) 2.5 2.5−0.8=1.7 1.7
w₂ (relevant) 1.2 1.2−0.8=0.4 0.4
w₃ (noise) 0.6 0.6−0.8=−0.2 0 ← zeroed
w₄ (noise) 0.3 0.3−0.8=−0.5 0 ← zeroed
w₅ (relevant) L1 keeps
Interpretation: −1.9
features 1, 2, 5 and1.9−0.8=1.1 −1.1
discards noise features 3, 4 (automatic
feature selection).
DROPOUT — GOODFELLOW §7.12
"Dropout provides an inexpensive approximation to training and evaluating a bagged ensemble of exponentially many neural
networks." — Goodfellow §7.12
Mechanism Why it works
⊘ During training, each unit is randomly set to zero with ⊗ Forces redundant representations — no unit can rely on
probability p (typically p=0.5 for hidden, p=0.2 for input). A others. Equivalent to training 2ⁿ different networks and
different sub-network is sampled each mini-batch. averaging predictions (ensemble).
Weight Scaling (Inference) Inverted Dropout
✕ At test time: keep all units but multiply each weight by keep ⊕ Modern practice: divide activations by (1−p) during training
probability (1−p). Approximates averaging the ensemble so weights remain the same scale at test time. Used in
without multiple forward passes. PyTorch, TensorFlow default.
DROPOUT — NUMERICAL EXAMPLE (INVERTED DROPOUT)
2-layer network, hidden layer h = [3.0, 1.5, 4.2, 0.8], dropout keep-rate = 0.5
Training (Inverted Dropout, p=0.5) Test Time (No dropout, no scaling needed)
h = [3.0, 1.5, 4.2, 0.8] h = [3.0, 1.5, 4.2, 0.8] (all units active)
Step 1: Sample mask M ~ Bernoulli(0.5) No mask applied.
M = [1, 0, 1, 0] (units 2,4 dropped) Inverted dropout ensures expected
activation scale matches training:
Step 2: Apply mask and scale by 1/keep_rate
h_drop = h ⊙ M / 0.5 E[h_drop_i] = p × (h_i/p) = h_i
h_drop = [3.0×1, 1.5×0, 4.2×1, 0.8×0] / 0.5 E[h_test_i] = h_i (same scale)
h_drop = [6.0, 0.0, 8.4, 0.0]
Outcome: no adjustment needed at test
Step 3: Forward pass with h_drop as input time — just pass h through unchanged.
Gradient computed only through active units
Contrast with vanilla dropout:
Multiply all weights by (1−p) = 0.5
Different mask sampled each mini-batch →
at test time to match training scale.
Trains 2⁴ = 16 sub-networks implicitly
Result: exact ensemble approximation
EARLY STOPPING
EARLY STOPPING&&DATASET
DATASETAUGMENTATION
AUGMENTATION——GOODFELLOW
GOODFELLOW§7.8,
§7.4
§7.8, §7.4
Early Stopping (§7.8) Dataset Augmentation (§7.4)
Geometric:
Crop, flip, rotate, scale, shear. Core for image tasks (ImageNet
Idea: Monitor validation error during training. Stop when it starts baseline).
rising.
Color jitter:
Algorithm: Random brightness, contrast, saturation, hue. Helps with lighting
1. Keep track of best validation error so far variation.
2. Save model checkpoint at each best
Noise injection:
3. If no improvement after p steps → stop
4. Return best checkpoint Add Gaussian noise to inputs or hidden units. Equivalent to Tikhonov
regularization.
Goodfellow insight: Early stopping is equivalent to L2 regularization
Mixup (§7.4):
under some conditions. Number of training steps acts like 1/λ.
x̃ = λxᵢ + (1−λ)xⱼ, ỹ = λyᵢ + (1−λ)yⱼ. Trains on convex combinations.
Practical tips: patience p=10–20 epochs; use learning rate schedule
(cosine decay). Free regularizer — always use it. Feature dropout:
Randomly mask input features. Common for tabular and NLP tasks.
ADDITIONAL REGULARIZATION METHODS — GOODFELLOW CH. 7
Parameter Norm Penalties (§7.1) Multi-Task Learning (§7.7) Sparse Representations (§7.3)
Share representations across related tasks. Penalize activations: Ω(h) = λǁhǁ₁
Ω(θ) = λ₁ǁwǁ₁ + λ₂ǁwǁ²₂/2 (Elastic Net)
Acts as implicit regularizer — shared layers Sparse codes force distributed
Combines sparsity (L1) and stability (L2).
are constrained by all task losses representation. Used in autoencoders and
Useful when correlated features exist.
simultaneously. dictionary learning.
Max-Norm Constraints (§7.2) Batch Normalization (§8.7) Noise Robustness (§7.5)
Constrain: ǁwǁ₂ ≤ c (project weights onto L2 Normalizes layer inputs μ=0, σ=1. Has strong Inject noise into weights: w̃ = w + ε, ε~N(0,
ball if violated). More stable than penalty — regularization side effect: reduces need for η²I). Equivalent to regularizer proportional to
prevents explosion, used with dropout. dropout. Standard in deep networks. Fisher information matrix.
APPLICATIONS OF REGULARIZATION ACROSS DOMAINS
Computer Vision Natural Language Processing
L2 + Dropout + Augmentation Dropout + Early Stopping
ResNet uses L2 weight decay (λ=1e-4). Dropout in FC layers. BERT/GPT use dropout (p=0.1) on attention + FFN layers. Early
Aggressive augmentation (crop, flip, color jitter) is the strongest stopping on validation perplexity. Input dropout (word masking) is
regularizer for ImageNet. implicit augmentation.
Tabular / Finance Medical / Small Data
L1 (feature selection) + L2 Strong L2 + Transfer Learning
Credit scoring: Lasso selects relevant features from 1000s. Elastic Net With few samples, bias-variance demands high regularization.
balances sparsity and multicollinearity. L2 for stable coefficients in Fine-tuning (transfer learning) implicitly regularizes via pretrained
risk models. weights initialization.
Reinforcement Learning Graph Neural Networks
Entropy Regularization Dropout + DropEdge
Policy regularized by H(π) = −E[log π(a|s)]. Prevents premature DropEdge randomly removes edges during training, regularizing the
collapse to deterministic policy. Used in SAC, A3C. Equivalent to graph adjacency. Prevents over-smoothing in deep GNNs (each layer
KL-divergence from uniform. blends neighborhoods).
KEY TAKEAWAYS
1 ERM is the foundation —
Minimizing empirical risk is how neural networks learn. Justified by LLN as m→∞. All training algorithms optimize some form of
ERM (Goodfellow §5.2).
2 Bias-Variance Tradeoff —
MSE = Bias² + Variance. Regularization trades variance for bias. No free lunch: choose the right balance for your data regime (§5.4).
3 L2 shrinks, L1 selects —
L2 keeps all features small (Gaussian prior). L1 zeros out irrelevant features (Laplace prior). Elastic Net combines both (§7.1).
4 Dropout = cheap ensemble —
Randomly zeroing units during training approximates averaging 2ⁿ networks. Inverted dropout scales at train time for free inference
(§7.12).
5 Free lunch: early stopping & augmentation —
Early stopping ≡ L2 regularization in some settings. Augmentation is the strongest regularizer for vision/NLP. Always use both (§7.4,
§7.8).
Reference: Goodfellow, Bengio & Courville — Deep Learning, Chs. 5, 7, 8 (MIT Press, 2016)