M.A.M.
COLLEGE OF ENGINEERING AND TECHNOLOGY
SIRUGANUR, TIRUCHIRAPPALLI - 621 105
DEPARTMENT OF ARTIFICIAL INTELLIGENCE & DATA SCIENCE
AA3501-Deep Learning
PRESENTATION TITLE : Optimization Techniques: SGD, Adam, and
Beyond
NAME OF THE
REGISTER NUMBER
STUDENT
Pavithra N
812024243029
Praveen P
812024243030
Priyadarshini P
812024243031
Under the esteemed guidance of Supervisor
Name : Mr.R. Kishore Designation : Lecturer 1
Optimizer Primer
How SGD, Adam, and newer methods shape
model training
Gradients to Updates
Optimizers map gradients g_t to parameter deltas, determining
how θ_t evolves. They differ in update rule, sensitivity to learning
rate, and implicit regularization. For example, SGD uses θ_{t+1} =
θ_t − η · g_t, while momentum adds a velocity term v_t = μ · v_{t-1} +
η · g_t to accelerate along consistent gradients. Adaptive methods
further adjust per-parameter step sizes to stabilize and speed 训练
Adam Fast Early, SGD Better Generalizes
Uses biased first/second moment estimates (m_t, v_t) Updates scale raw gradients by a global learning rate with
and per-parameter updates: eta * m_t / (sqrt(v_t) + eps). momentum, exposing training dynamics to noise that can
Often yields faster initial loss reduction and is less help generalization. With careful LR schedules and larger
sensitive to starting eta, especially with sparse or batches, SGD often achieves stronger final test accuracy
heterogeneous parameters. on vision tasks after proper decay.
Early progress speed vs. final generalization — the choice trades speed of gains for potential generalization gains.
Tuning Flow
Candidate 10% Sanity LR Sweep Decay Tune Schedule Pick Final Validate
Opts
Choose Run a 10% epoch Do a learning- Evaluate weight Choose between Run full training
SGD+momentum with default rate finder from decay candidates step, cosine, or with the chosen
and hyperparams; 1e-6 to 1 with 20– {0, 1e-4, 1e-3, 1e- warmup based config; monitor
Adam/AdamW as check for 100 mini- 2} with short on observed loss train/val loss,
initial monotonic loss, batches; pick a runs; select plateaus and gradient norms,
contenders; no divergence, stable, fast- decay that gradient and weight
balance and early stability learning starting minimizes behavior. norms to confirm
generalization vs signals. point. overfitting and stability.
speed of instability.
progress.
Decay vs L2
In adaptive optimizers, naive L2 penalty is not equivalent to true weight decay, because
regularization interacts with moment estimates and scales with parameter magnitude.
Adam mixes the lambdaw term into gradients, altering m_t and v_t, producing scale-
dependent shrinkage. AdamW decouples decay: w <- w - eta (m_t / (sqrt(v_t) + eps) +
lambda * w). Empirically, switching from Adam with L2 to V
Diagnostics 3.5×
Initial loss drop after the first 10% of epochs;
Adam typically accelerates loss reduction 2–5× vs
SGD.
8.2
Peak gradient norm (L2) during early training; if >1e3,
consider gradient clipping or smaller learning rate.
Quantitative benchmarks guide optimizer
choice and tuning with three key metrics. 2.4%
Final generalization gap (validation − training
accuracy); target <3% for well-regularized vision
models.
Optimizer Checklist
Gradient Check Divergence Fix Overfitting Handle
Do a single-step forward/backward If loss explodes, reduce LR by 10× and If validation gap grows, adjust
and assert non-zero grads: y = apply clip: lr = lr * 0.1; regularization or optimizer: try
model(x); loss = loss_fn(y, t); [Link].clip_grad_norm_(mode weight_decay in {1e-4,1e-3} or switch
[Link](); for p in [Link](), 1.0) + re-run a small to SGD with momentum 0.9 and a
cosine decay schedule.
[Link](): assert [Link] is batch.
not None and [Link]().sum() > 0.
Slow Start Prod Repro
If progress is slow, enable adaptive For production: fix seeds (torch,
optimizers with warmup: use AdamW numpy, Python), log optimizer state
with warmup_steps = int(0.01–0.05) ×
total_steps (e.g., 1%–5%). dumps, and export hyperparameters
([Link]) to ensure
identical runs.
Own the Optimizer
- Understand the math and practical behavior of SGD with
momentum and Adam to predict updates, not just rely on
curves.
- Know when to choose each optimizer and how weight decay
interacts with adaptive methods to control generalization.
- Apply a concise 4-step tuning procedure (learning rate,
Thanks!