DEEP LEARNING • CHAPTER 8

Optimization for
Training Deep Models
Ill Conditioning · Local Minima · Plateaus · Saddle Points
Cliffs · Exploding Gradients · Basic & Adaptive Algorithms

Reference: Ian Goodfellow, Yoshua Bengio and Aaron Courville — Deep Learning, MIT Press, 2016
Table of Contents

Overview of Optimization in Deep Learning Cliffs & Exploding Gradients

01 The unique challenges compared to pure mathematical optimization 05 Sudden steep regions and gradient instability

Ill Conditioning of the Hessian Basic Optimization Algorithms

02 How curvature slows gradient descent convergence 06 SGD, Momentum, Nesterov, and beyond

Local Minima & Saddle Points Adaptive Learning Rate Algorithms

03 Why local minima may not be the real problem 07 AdaGrad, RMSProp, Adam and their properties

Plateaus and Flat Regions Summary & Key Takeaways

04 Regions of near-zero gradient that stall learning 08 Putting it all together
SECTION 01

Overview of Optimization
in Deep Learning

Why training neural networks is fundamentally different from classical optimization

Goodfellow, Bengio & Courville — Deep Learning (2016) | Chapter 8

What Makes Deep Learning Optimization Unique? Goodfellow et al. (2016)

Empirical Risk Surrogate Loss Batch vs. Online

Minimization Functions Learning

We minimize a surrogate loss on Direct optimization of 0–1 loss is Mini-batch SGD is the workhorse: large
training data as a proxy for true intractable. Cross-entropy and MSE enough for stable gradient estimates,
generalization error. The gap between serve as smooth, differentiable small enough for frequent updates and
training loss and test performance is surrogates enabling gradient-based implicit regularization (§8.1.3).
the generalization gap (§8.1). methods (§8.1.2).
Core Challenges in Neural Network Optimization Goodfellow et al. (2016)

The key insight from Goodfellow et al. (§8.2): Neural network optimization faces challenges unlike those in convex optimization. These
interact in complex, poorly-understood ways.

Ill Conditioning Local Minima Plateaus & Saddle Pts

Hessian eigenvalue spread causes slow Many local optima exist; may not harm Flat regions with near-zero gradient
convergence generalization dominate

Exploding Gradients Vanishing Gradients Poor Init / LR

Cliff-like surfaces cause unstable Gradients shrink through deep layers, Bad starting points or step sizes derail
parameter updates blocking learning training
SECTION 02

Ill Conditioning
of the Hessian

How the curvature landscape makes gradient descent inefficient

Goodfellow, Bengio & Courville — Deep Learning (2016) | Chapter 8

Ill Conditioning — The Hessian Problem (§8.2.1) Goodfellow et al. (2016)

The Hessian Matrix & Curvature Condition Number

κ(H) = λ_max / λ_min

Large κ → Ill-conditioned
• The Hessian H of the loss w.r.t. parameters encodes second-order curvature Small κ → Well-conditioned
information.
• A Taylor expansion shows: f(x − εg) ≈ f(x) − ε gᵀg + ½ε² gᵀHg
• When gᵀHg is large relative to gᵀg, the quadratic term dominates and a
gradient step overshoots.
• Ill-conditioning: the ratio λ_max / λ_min (condition number) is large, meaning Practical Consequence
different directions have wildly different curvature.
• Result: gradient descent must use very small learning rates globally, even
when most directions permit larger steps. Even if SGD reaches a region near the optimum,
• The gradient norm may NOT decrease reliably — training 'stalls' not from a are computationally infeasible for large nets.
flat region but from oscillation in high-curvature directions. ill-conditioning causes oscillatory behaviour.
Second-order methods (Newton) solve this but

→ Motivates momentum and adaptive methods.

SECTION 03

Local Minima &

Saddle Points

The real obstacles in high-dimensional non-convex optimization

Goodfellow, Bengio & Courville — Deep Learning (2016) | Chapter 8

Local Minima — Are They Actually the Problem? (§8.2.2) Goodfellow et al. (2016)

Key Insight (Goodfellow et al., §8.2.2)

In high-dimensional spaces, most critical points with small gradient are saddle points, not local minima. True local minima with high cost
are exponentially rare. Local minima with low cost are essentially equivalent to global minima for practical purposes.

Model Identifiability Issue Theoretical Support Practical Concern

Neural networks are not identifiable —

Theoretical results for random Gaussian error For modern over-parameterized networks, the
permuting hidden units or negating paired
surfaces show: at any critical point, the worry is NOT getting stuck at high-cost local
weights yields equivalent models. This
fraction of negative curvature directions minima.
generates a combinatorial number of
(indicating a saddle) grows with the loss value.
equivalent local minima, all with the same loss
Low-loss critical points are overwhelmingly Instead the concern is:
value (§8.2.2).
saddle points or global minima. • Saddle points with near-zero gradient
• Flat regions that slow convergence
This means that local minima proliferation is a
Empirical experiments confirm small, final • Poor generalization (training ≠ test loss)
structural artifact, not a real obstacle to
training losses at convergence. • Cliff-like regions causing instability
finding good solutions.
Saddle Points — The Dominant Obstacle (§8.2.3) Goodfellow et al. (2016)

What is a Saddle Point? Saddle Point Geometry

∇f = 0
λ₁ > 0 (upward curvature)
λ₂ < 0 (downward curvature)
• A saddle point is a critical point (∇f = 0) where the Hessian has both
positive and negative eigenvalues.
• In n-dimensional space, a random critical point has probability 2⁻ⁿ of being
a local minimum — saddle points overwhelmingly dominate.
• First-order methods (SGD) can take an extremely long time to escape
saddle points because the gradient is near zero. Dauphin et al. (2014) Hypothesis
• Empirical observation: gradient norms often reach small values long before
a true minimum is found — indicating prolonged time near saddle points.
• Second-order information (negative curvature) can escape saddle points Critical points with high loss are almost always saddle
but requires expensive Hessian computation. points. As training loss decreases, critical points
• Stochastic gradient noise in mini-batch SGD provides a beneficial increasingly resemble local/global minima. This explains
perturbation that helps escape saddle points over time. why training eventually converges despite the presence
of saddle points.

(Referenced in Goodfellow et al. §8.2.3)

SECTION 04

Plateaus and
Flat Regions

Near-zero gradient regions that dramatically slow learning

Goodfellow, Bengio & Courville — Deep Learning (2016) | Chapter 8

Plateaus, Vanishing Gradients & Flat Regions (§8.2.5) Goodfellow et al. (2016)

A plateau is a broad, flat region of the loss landscape where gradients are very small but no critical point exists. Unlike saddle
points, plateaus do not require ∇f = 0 — they just have ∥∇f∥ ≈ 0 over an extended region.

Saturating Activations Long Plateau Traversal Effect on Training Dynamics

Sigmoid & tanh saturate in their extremes: σ(x) SGD must traverse these plateau regions taking Learning curves show extended flat sections
→0 or 1, tanh(x)→±1. In these regions, the many small steps with little improvement in loss. followed by sudden drops when the optimizer
gradient of the activation is nearly zero regardless This gives the misleading impression that training escapes the plateau. This non-monotone,
of the upstream gradient. Deep networks amplify has converged when it has not. Practitioners may step-like behavior is characteristic and expected
this across many layers, blocking weight updates prematurely terminate training or assume the in deep network training, especially in early
in early layers (§6.3.1, §8.2.5). model has reached its capacity. epochs with poor initialization.

Mitigation via Activation Choice Batch Normalization Momentum's Role

By normalizing layer inputs, batch normalization Momentum accumulates velocity across plateau
ReLU (max(0,x)) does not saturate for positive
keeps activations in the linear (non-saturating) steps, allowing the optimizer to coast through flat
inputs, largely solving the vanishing gradient
regime of activation functions. This reduces regions with diminishing per-step gradient. This is
plateau problem in forward layers. However, the
plateau effects and allows much higher learning a key practical advantage of momentum-based
'dying ReLU' (permanently zero output) creates a
rates, dramatically accelerating training through methods over vanilla gradient descent in plateau
different kind of dead neuron plateau (§6.3.1).
flat regions (§8.7). traversal.
SECTION 05

Cliffs & Exploding

Gradients

Sudden loss surface discontinuities and catastrophic gradient instability

Goodfellow, Bengio & Courville — Deep Learning (2016) | Chapter 8

Cliffs and Exploding Gradients (§8.2.4) Goodfellow et al. (2016)

⚠ Critical Issue: Cliffs create regions where the gradient is enormous. A single gradient step can move parameters far outside
any reasonable range, causing catastrophic divergence.

Solution: Gradient Clipping

What Are Loss Cliffs? Clip by value: g ← clip(g, −v, +v)

Clip by norm: g ← g · (v / ∥g∥) if ∥g∥ > v

• Cliffs occur when the loss surface has extremely steep walls adjacent to Norm clipping (Pascanu et al., 2013) is preferred: it
relatively flat regions — common in recurrent neural networks (RNNs). preserves gradient direction while bounding
• Caused by multiplying many large weight matrices during backpropagation magnitude.
through time (BPTT): gradient grows exponentially with sequence length.
• Mathematically: if the largest eigenvalue of the Jacobian |λ| > 1 repeatedly,
RNN Context (§10.7)
gradients explode. If |λ| < 1 repeatedly, they vanish.
• A cliff step causes the optimizer to 'jump over' the minimum, potentially
landing far on the other side with a large loss value. RNNs are particularly susceptible due to
• This can catastrophically undo many iterations of careful progress in a single parameter reuse across timesteps. The same
weight matrix W is applied repeatedly, so its
update step.
spectral radius determines whether gradients
explode or vanish. LSTM gating mechanisms were
specifically designed to mitigate this.
SECTION 06

Basic Optimization
Algorithms

SGD, Momentum, Nesterov Momentum — the foundational methods

Goodfellow, Bengio & Courville — Deep Learning (2016) | Chapter 8

Stochastic Gradient Descent (§8.3.1) Goodfellow et al. (2016)

SGD — The Core Algorithm Learning Rate Schedules

θ ← θ − ε · ∇_θ (1/m) Σ L(f(x⁽ⁱ⁾;θ), y⁽ⁱ⁾) • Constant ε (simplest, rarely optimal)

• Linear decay: ε_k = (1 − k/K)ε₀ + (k/K)ε_τ
• Exponential decay: ε_k = ε₀ · γᵏ
• Cosine annealing with warm restarts

• ε is the learning rate (step size); m is the mini-batch size; the sum is over
mini-batch samples.
• Mini-batch size: typically 32–256 samples. Larger batches give more accurate
gradient estimates but fewer updates per epoch. Bottleneck: Noisy Gradients
• SGD introduces gradient noise which can help escape sharp local minima and
saddle points (implicit regularization). Standard error of mini-batch gradient ∝ 1/√m.
• Convergence guarantee: for convex functions, learning rate must satisfy Σε_k Doubling batch size halves variance but halves
= ∞ and Σε_k² < ∞ (diminishing learning rate schedule). update frequency — marginal benefit requires
• For non-convex deep nets, no global convergence guarantees exist — doubling the learning rate to compensate (linear
practical convergence is empirically observed. scaling rule).
• Learning rate is the single most important hyperparameter. Decay schedules:
linear, exponential, 1/t, cosine annealing. This motivates momentum: averaging gradients over
time to reduce variance without reducing update
frequency.
Momentum & Nesterov Momentum (§8.3.2 – §8.3.3) Goodfellow et al. (2016)

Momentum Nesterov Momentum (NAG)

v ← αv − ε∇_θ L(θ) v ← αv − ε∇_θ L(θ + αv)

θ ← θ + v θ ← θ + v

• Key difference: gradient is evaluated at the 'lookahead'

• v is the velocity (exponential moving average of gradients); α position θ + αv, not at the current θ.
is the momentum coefficient (typically 0.9). • Provides corrective feedback: if the momentum step
• Velocity accumulates in directions of consistent gradient, overshoots, the gradient at the new location corrects it.
accelerating through flat regions and plateaus. • Nesterov momentum has provably faster convergence than
• In ill-conditioned directions, opposing gradients cancel in the standard momentum for convex functions: O(1/k²) vs
velocity, reducing oscillation. O(1/k).
• Terminal velocity: v_∞ = ε∥g∥ / (1−α), so momentum • Intuition: standard momentum is 'blind' — it applies
amplifies effective step size by 1/(1−α)×. momentum then corrects. NAG applies correction before
• α = 0.9 gives 10× amplification; α = 0.99 gives 100× — useful the full momentum step.
for long flat plateau traversal. • In practice, often gives slightly better convergence than
standard momentum, especially in the later stages of
training.
SECTION 07

Adaptive Learning
Rate Algorithms

AdaGrad, RMSProp, Adam — per-parameter learning rate adaptation

Goodfellow, Bengio & Courville — Deep Learning (2016) | Chapter 8

Motivation for Adaptive Learning Rates (§8.5) Goodfellow et al. (2016)

Core Problem: A single global learning rate ε is suboptimal. Some parameters (e.g., rare features) need large steps; others
(frequent, well-trained) need small steps. Adaptive methods maintain a per-parameter learning rate scaled by the history of
gradients for that parameter.

AdaGrad (§8.5.1) RMSProp (§8.5.2) Adam (§8.5.3)

m ← β₁m + (1−β₁)g
r ← r+g⊙g r ← ρr + (1−ρ)g ⊙ g
v ← β₂v + (1−β₂)g⊙g
θ ← θ − (ε/√(r+δ)) ⊙ g θ ← θ − (ε/√(r+δ)) ⊙ g
θ ← θ − ε·m̂/√(v̂)+δ

Strengths: Strengths: Strengths:

• Eliminates manual learning rate tuning • Exponentially weighted moving average • Bias-corrected momentum + RMSProp
• Great for sparse features & NLP • Fixes AdaGrad's vanishing LR problem • Robust to hyperparameter choices
• Monotonically decreasing per-parameter LR • Works well for non-stationary objectives • De facto standard for deep learning

Limitations: Limitations: Limitations:

• Accumulates all past gradients forever
• Still lacks bias correction • Can generalize worse than SGD+momentum
• LR → 0, training stalls in deep nets
• ρ (decay rate ≈ 0.9) is an extra hyperparameter • Weight decay must be decoupled (AdamW)
• Works well for convex but less for deep
• Can diverge with large gradients • More memory: stores m and v for all θ
non-convex
Adam in Depth & Comparison of Algorithms (§8.5.3) Goodfellow et al. (2016)

Adam — Full Algorithm Algorithm Comparison

Require: ε (lr), β₁ (≈0.9), β₂ (≈0.999), δ (≈10⁻⁸) Algorithm LR Adapt Momentum Memory

Initialize: m₀=0, v₀=0, t=0
1. t ← t + 1
2. g_t ← ∇_θ L(θ_{t-1}) SGD None None O(θ)
3. m_t ← β₁·m_{t-1} + (1−β₁)·g_t
4. v_t ← β₂·v_{t-1} + (1−β₂)·g_t²
5. m̂_t ← m_t / (1−β₁ᵗ) [bias correction]
6. v̂_t ← v_t / (1−β₂ᵗ) [bias correction] SGD+Moment None 1st order 2×O(θ)
7. θ_t ← θ_{t-1} − ε · m̂_t / (√v̂_t + δ) um

Nesterov None Lookahead 2×O(θ)

• Bias correction compensates for zero initialization of m and v, AdaGrad Global hist. None 2×O(θ)
especially critical in early iterations.
• β₂ = 0.999 makes v a very slow-moving average — stable
denominator. RMSProp Exp. decay Optional 2×O(θ)
• δ prevents division by zero; also acts as a minimum effective
learning rate.
• AdamW: decouples weight decay from gradient update for Adam Exp. decay 1st order 3×O(θ)
better regularization behavior.
SECTION 08

Summary &
Key Takeaways

Connecting the challenges to the algorithms

Goodfellow, Bengio & Courville — Deep Learning (2016) | Chapter 8

Connecting Challenges to Solutions Goodfellow et al. (2016)

Challenge Algorithm / Solution Core Insight

Ill Conditioning Momentum, Nesterov, Natural Gradient Momentum damps oscillations in high-curvature directions; 2nd order methods
(2nd order) normalize curvature but are too expensive.

Local Minima Mini-batch SGD noise; large Modern theory: local minima with high loss are rare; focus shifts to saddle points and
over-parameterized models generalization.

Saddle Points SGD noise; saddle-free Newton Mini-batch noise provides implicit perturbation to escape saddle points; first-order
methods methods eventually escape.

Plateaus / Flat Regions Momentum, Batch Norm, ReLU Momentum coasts through flat regions; BN keeps activations in linear regime; ReLU
activations avoids saturation.

Cliffs / Exploding Grad. Gradient clipping, LSTM gating Clip by norm preserves direction; LSTM learns when to gate gradient flow to prevent
amplification.

Global Learning Rate AdaGrad, RMSProp, Adam Per-parameter adaptive rates solve the one-size-fits-all problem; Adam combines
momentum + RMSProp + bias correction.
Practical Guidelines from Goodfellow et al. (§8.6 & §8.7) Goodfellow et al. (2016)

01 Start with Adam 02 Tune Learning Rate First 03 Use Gradient Clipping for RNNs

Adam with default hyperparameters (ε=0.001,

The learning rate is the most critical Always clip gradients by norm (threshold ≈
β₁=0.9, β₂=0.999) works well across most
hyperparameter. Search on log scale: {0.1, 1–5) when training recurrent networks.
architectures. Switch to SGD+momentum if
0.01, 0.001, 0.0001}. Use learning rate Monitor gradient norms during training to
generalization matters more than fast
warmup then decay for stability. detect explosions early.
convergence.

Batch Normalization Transforms SGD+Momentum for Best

04 05 06 Monitor, Don't Assume Convergence
Landscape Generalization

BN makes the loss surface significantly Well-tuned SGD with momentum and learning Plateaus cause misleading training curves.
smoother, eliminates most plateau and rate decay often outperforms Adam on test Track gradient norms, parameter norms, and
ill-conditioning issues, and allows larger accuracy, especially for vision tasks (ResNets, validation loss — not just training loss. Use
learning rates. Use it by default in deep VGGs). SGD finds flatter minima that learning rate range tests to diagnose
feedforward nets. generalize better. landscape issues.
References & Further Reading
Ian Goodfellow, Yoshua Bengio, Aaron Courville. Deep Learning. MIT Press, 2016. Chapter 8: Optimization for Training Deep
Primary
Models (§8.1–§8.6).

Razvan Pascanu, Tomas Mikolov, Yoshua Bengio. On the difficulty of training recurrent neural networks. ICML 2013. [Gradient
Pascanu et al.
clipping]

Yann Dauphin et al. Identifying and attacking the saddle point problem in high-dimensional non-convex optimization. NIPS 2014.
Dauphin et al.
[Saddle point analysis]

Duchi et al. John Duchi, Elad Hazan, Yoram Singer. Adaptive Subgradient Methods for Online Learning. JMLR 2011. [AdaGrad]

Tieleman & Hinton Tieleman & Hinton. Lecture 6.5 — RMSProp. COURSERA: Neural Networks for Machine Learning. 2012. [RMSProp]

Kingma & Ba Diederik P. Kingma, Jimmy Ba. Adam: A Method for Stochastic Optimization. ICLR 2015. [Adam optimizer]

Nesterov Yurii Nesterov. A method of solving a convex programming problem with convergence rate O(1/k²). 1983. [Nesterov Momentum]

All content derived from: Goodfellow, Bengio & Courville — Deep Learning, MIT Press, 2016 | Chapter 8