Complete Guide — Chapter 11: Practical Methodology

Deep Learning (Goodfellow, Bengio, Courville) — study notes in paraphrased form

Note: The uploaded file contains only Chapter 11 of the book, so this guide covers that chapter. Its theme:
success in applied deep learning depends less on knowing exotic algorithms and more on disciplined methodology
— knowing what to try next instead of guessing blindly.

0. The Core Design Process (The Chapter’s Backbone)

The authors recommend a four-part loop for any applied ML project:

1. Define goals first — pick an error metric and a target value for it, driven by the real problem you’re solving.
2. Build a working end-to-end pipeline early — including the machinery to measure your chosen metrics.
3. Instrument the system — diagnose which component is underperforming and whether the cause is overfitting,
underfitting, bad data, or a software bug.
4. Iterate incrementally — add data, tune hyperparameters, or change algorithms based on evidence from your
instrumentation, not intuition.

The running example throughout the chapter is Google’s Street View address-number transcription system,
which reads house numbers from photos so addresses can be placed correctly on Google Maps.

1. Performance Metrics (§11.1)

Why metrics come first

Your metric guides every later decision. You also need a realistic target, because zero error is impossible:

Bayes error is the theoretical floor — the minimum error even with infinite data, caused by incomplete input
features or inherent randomness in the system.
Finite training data limits you further. In product settings you can usually buy more data (at a cost in time,
money, or even human suffering, e.g. invasive medical tests); in academic benchmark settings the dataset is
fixed.

Setting a target

Academic context: anchor expectations to previously published benchmark results.

Real-world context: anchor to what makes the application safe, cost-effective, or attractive to users.

Beyond plain accuracy

Accuracy/error rate is the default, but many applications need more:

Situation Better metric Why

Errors have unequal costs (e.g. spam Weighted total cost Blocking a legitimate email is far worse
filter) than letting one spam through, so the
two mistake types should not count
equally

Detecting rare events (e.g. 1-in-a- Precision & recall A classifier that always says “no
million disease) disease” scores 99.9999% accuracy
while being useless

System may abstain from answering Coverage Fraction of inputs the system is willing
to answer
Key definitions: - Precision — of the detections the model reported, what fraction were correct. - Recall — of the true
events, what fraction were detected. - Degenerate extremes: predicting “never” gives perfect precision but zero recall;
predicting “always” gives perfect recall but tiny precision. - PR curve — precision (y-axis) vs. recall (x-axis), traced by
sweeping the decision threshold on the model’s score. - F-score — collapses the trade-off into one number:

F = 2pr / (p + r)

Alternative: area under the PR curve. - Coverage — lets you trade answering fewer cases for higher accuracy on the
ones you do answer. Refusing everything gives 100% accuracy at 0% coverage. Street View’s target: human-level
(98%) accuracy at ≥95% coverage.

Other metrics exist (click-through rates, user-satisfaction surveys, domain-specific criteria). What matters is
committing to one metric in advance so you can tell whether changes are progress.

2. Default Baseline Models (§11.2)

Once metrics are chosen, get a reasonable end-to-end system running quickly. Sensible defaults:

Do you even need deep learning?

If the problem might be solvable with a few well-chosen linear weights → start with logistic regression or
similar simple statistical models.
If it’s an “AI-complete” task (object recognition, speech recognition, machine translation) → start with an
appropriate deep model.

Choosing architecture by data structure

Input structure Default model

Fixed-size vectors, supervised Feedforward net with fully connected layers

Known topology (e.g. images) Convolutional network

Sequential input or output Gated recurrent net (LSTM or GRU)

Use piecewise-linear activations (ReLU or variants: Leaky ReLU, PReLU, maxout) as the default nonlinearity.

Optimization defaults

SGD with momentum + decaying learning rate. Common decay schemes: linear decay to a floor,
exponential decay, or cutting the rate by 2–10× whenever validation error plateaus.
Adam is a reasonable alternative.
Batch normalization can dramatically help optimization (especially convnets and sigmoidal networks). Fine to
omit in the very first baseline, but add it fast if optimization struggles.

Regularization defaults

Unless you have tens of millions of examples, include mild regularization from the start.
Early stopping — use it almost universally.
Dropout — excellent, easy, widely compatible.
Batch norm’s statistical noise sometimes acts as a regularizer itself and can let you drop dropout.

Reuse what exists

If your task resembles a well-studied one, copy the best-known model — possibly the trained weights too
(e.g. ImageNet-pretrained convnet features reused for other vision tasks).
Unsupervised pretraining? Domain-dependent. NLP benefits enormously (e.g. word embeddings); computer
vision mostly doesn’t, except semi-supervised settings with very few labels. Include it in the first baseline only if
your domain is known to need it — otherwise reserve it for later, e.g. if the baseline overfits.
3. Should You Gather More Data? (§11.3)

Novices try new algorithms; practitioners often just get more data. Decision procedure:

1. Is training-set performance acceptable?

No → more data won’t help (the model isn’t even exploiting what it has). Instead:
Increase model size (layers, hidden units).
Improve optimization (e.g. tune the learning rate).
If big, well-tuned models still fail → suspect data quality (too noisy, missing informative inputs) →
collect cleaner data or richer features.
Yes → move to step 2.
2. Is test-set performance acceptable?
Yes → you’re done.
No (big train–test gap) → more data is one of the most effective fixes. Weigh:
Cost/feasibility of collecting data (cheap at large internet companies; expensive in medicine).
Cost of alternatives: shrink the model or strengthen regularization (weight decay, dropout).
If the gap stays unacceptable after tuning regularization → gather more data.
3. How much data? Plot generalization error vs. training-set size and extrapolate. Small additions rarely move the
needle — scale the dataset logarithmically (e.g. doubling between experiments).
4. If more data is simply infeasible, the only remaining lever is improving the learning algorithm itself — which is
research territory, not routine practice.

4. Selecting Hyperparameters (§11.4)

Two philosophies: manual (needs understanding, cheap compute) vs. automatic (needs less understanding,
expensive compute).

4.1 Manual tuning

Goal: minimize generalization error within a runtime/memory budget by adjusting effective capacity to match task
complexity. Effective capacity is limited by three things:

1. Representational capacity of the model (more layers/units = more expressible functions),

2. The optimizer’s ability to actually find good functions,
3. How strongly the cost function/training procedure regularizes.

The U-shaped curve: generalization error vs. a hyperparameter typically forms a U — one end underfits (high
training error), the other overfits (large train–test gap), with optimal capacity in the middle. Caveats: - Many
hyperparameters are discrete or binary, so they sample only a few points on the curve. - Some hyperparameters have
one-sided ranges — e.g. weight decay’s minimum is zero, so it can only subtract capacity. If the model underfits at
zero decay, this knob can’t help.

The learning rate is the single most important hyperparameter. If you can tune only one thing, tune it: -
Capacity is maximized when the rate is correct, not large or small. - Training error vs. learning rate is itself U-shaped:
too large and gradient steps can increase training error (in the idealized quadratic case, when the rate exceeds
roughly twice the optimum); too small and training is slow and can even get permanently stuck at high error.

Everything else: monitor train and test error to diagnose the regime, then adjust capacity: - Training error above
target → must add capacity (more layers/units) — at higher compute cost. - Test error above target → test error =
training error + gap, so trade off the two. Networks usually do best with very low training error (high capacity) plus
strong regularization (dropout, weight decay) to shrink the gap without inflating training error. - Brute-force guarantee:
keep growing both model and dataset until solved — feasible only with resources.

Capacity effects of common hyperparameters (Table 11.1, condensed): | Hyperparameter | Raises capacity
when… | Notable caveat | |—|—|—| | Hidden units | increased | Raises time & memory of nearly every operation | |
Learning rate | tuned optimally | Wrong rate (either direction) lowers effective capacity via optimization failure | | Conv
kernel width | increased | Narrows output unless zero padding compensates; more memory/runtime | | Implicit zero
padding | increased | Raises time & memory of most operations | | Weight decay coefficient | decreased | — | | Dropout
rate | decreased | Fewer dropped units let units co-adapt to fit training data |
4.2 Automatic hyperparameter optimization

Finding hyperparameters is itself an optimization problem (objective: validation error). Wrapper algorithms can hide
hyperparameters from the user — but introduce secondary hyperparameters (e.g. search ranges), which are
fortunately easier to set robustly. Neural nets sometimes need forty-plus hyperparameters tuned, so automation
matters when no good starting point exists.

4.3 Grid search

Practical with ≤3 hyperparameters. Pick a small value set per hyperparameter; train on every combination in
the Cartesian product; keep the best on validation error.
Choose ranges conservatively and space values logarithmically (e.g. learning rates {0.1, 0.01, 10⁻³, 10⁻⁴,
10⁻⁵}; hidden units {50, 100, 200, 500, 1000, 2000}).
Iterate: if the best value sits at a grid edge, shift the grid outward; if it’s interior, zoom in and refine.
Fatal flaw: cost grows O(nᵐ) for m hyperparameters × n values each — exponential, and parallelism only
partially rescues it.

4.4 Random search

As easy to program, more convenient, and converges to good values much faster (Bergstra & Bengio, 2012).
Define a marginal distribution per hyperparameter (Bernoulli/multinoulli for discrete; log-uniform for positive
reals), e.g. sample the log of the learning rate uniformly on (−1, −5), then exponentiate.
Don’t discretize the values — continuous sampling explores more of the space at no extra cost.
Why it wins: no wasted runs. When a hyperparameter barely matters, grid search re-runs effectively identical
experiments; random search gives every trial fresh values of every influential hyperparameter.
Like grid search, benefits from repeated, refined rounds.

4.5 Model-based (Bayesian) hyperparameter optimization

Gradients of validation error w.r.t. hyperparameters occasionally exist but are usually unavailable (compute cost,
or intrinsically non-differentiable/discrete hyperparameters).
Workaround: fit a model (typically Bayesian regression) predicting validation error plus uncertainty, then
choose the next trial by balancing exploration (high-uncertainty regions, big potential gains) vs. exploitation
(near known good configurations).
Tools of the era: Spearmint, TPE, SMAC.
Verdict in the book: not yet reliably recommended — sometimes matches or beats human experts, sometimes
fails catastrophically. Worth trying, not mature.
Structural weakness of most sophisticated methods: they need each run to finish before learning from it,
whereas humans spot pathological settings early. Freeze–thaw approaches (Swersky et al., 2014) pause
unpromising runs and resume them later.

5. Debugging Strategies (§11.5)

Why ML is uniquely hard to debug

1. You don’t know the intended behavior — the whole point of learning is to discover behavior you couldn’t
specify. Is 5% test error on a new task good, or is something broken?
2. Adaptive parts mask each other’s failures. Example: a botched bias update b ← b − α (no gradient at all!)
may go unnoticed because the weights adapt to compensate.

Most strategies either (a) construct cases simple enough to predict the correct behavior, or (b) test one component in
isolation.

The debugging toolkit

1. Visualize the model in action. Look at detections drawn on images; listen to generated speech. Metrics alone
can hide evaluation bugs — the most devastating kind, because they convince you a broken system works.
2. Visualize the worst mistakes. Sort errors by the model’s confidence. Confidence scores from softmax/max-
 likelihood are overestimates but still rank examples usefully. In Street View, the most confident training-set
errors exposed over-tight image cropping that chopped off digits.
3. Reason from train/test error patterns.
Low train, high test → training works; genuine overfitting — or a broken evaluation path (model save/reload
issues, test data preprocessed differently).
High train and high test → can’t yet distinguish software defect from underfitting; run the next test.
4. Fit a tiny dataset. Even tiny models should nail one example (a single-example classification task is solvable
by output biases alone). Failure to fit one example — classifier, autoencoder, or generative model — indicates a
software defect, not underfitting. Extend to a handful of examples.
5. Check gradients against finite differences. If you hand-implement gradients or a new op’s backward pass:
f′(x) ≈ [f(x + ε) − f(x)] / ε, improved by the centered difference f′(x) ≈ [f(x + ½ε) − f(x − ½ε)] / ε.
ε must be big enough to survive floating-point rounding.
For vector functions g: ℝᵐ→ℝⁿ, avoid mn finite-difference runs by testing f(x) = uᵀg(vx) with random vectors
u, v; repeat with several u, v to catch errors orthogonal to one projection.
With complex arithmetic available: use f(x + iε); the imaginary part / ε recovers f′(x) with no cancellation,
allowing absurdly tiny ε (e.g. 10⁻¹⁵⁰).
6. Monitor histograms of activations and gradients over ~an epoch:
How often are ReLUs off — any permanently dead? How saturated are tanh units (mean |preactivation|)?
Exploding/vanishing gradient magnitudes hamper optimization.
Compare update size to parameter size: aim for updates ≈1% of parameter magnitude per minibatch —
not 50%, not 0.001%. Watch for parameter groups that stall, and remember sparse data means some
parameters update rarely.
7. Test algorithmic guarantees. Many algorithms promise invariants (objective never increases per step; certain
gradients zero after a step; all gradients zero at convergence). Test them — with numerical tolerance, since
rounding breaks exact equalities.

6. Case Study: Street View Multi-Digit Recognition (§11.6)

A worked demonstration of the whole methodology:

1. Data: cars photographed buildings; humans labeled; substantial curation, including an upstream ML detector to
locate house numbers before transcription.
2. Metrics: maps need high accuracy → fixed accuracy at human-level 98%, then optimized coverage as the main
metric (goal ≥95%).
3. Baseline: convolutional net with ReLUs; output layer = n independent softmax units predicting n characters (the
simplest thing that could work — sequence-output convnets weren’t yet standard).
4. Principled refinement: the network abstained whenever p(y|x) < t, but p(y|x) was initially an ad-hoc product of
softmaxes. Replacing it with an output layer and cost computing a proper log-likelihood made the rejection
mechanism far more effective.
5. Diagnosis: coverage still <90%. Train and test error were nearly identical (thanks to tens of millions of labels) →
problem was underfitting or data, not overfitting.
6. Debugging win: visualizing the most confident training errors revealed over-tight crops (e.g. “1849” cropped to
“849”). Rather than spend weeks perfecting the detector, the team simply widened the crop region — +10
percentage points of coverage from one pragmatic change.
7. Final gains: hyperparameter adjustment, mainly growing the model under compute constraints; the persistent
train≈test equality kept confirming underfitting as the bottleneck.
8. Outcome: hundreds of millions of addresses transcribed faster and cheaper than human effort could achieve.

7. Key Takeaways (One-Page Mental Model)

1. Metric before model. Choose one metric, one target, and let it drive everything.
2. Baseline before brilliance. A correctly applied common algorithm beats a sloppily applied clever one.
3. Diagnose before acting. Train vs. test error tells you whether to grow capacity, regularize, fix data, or hunt
bugs.
4. Data beats algorithms — when the train–test gap is the problem and collection is affordable. Scale it
logarithmically.
5. Learning rate first. It’s the highest-leverage hyperparameter; its effect on capacity is U-shaped via
optimization quality.
6. Random > grid for hyperparameter search; Bayesian methods are promising but unreliable (as of the book’s
writing).
7. ML bugs hide. Adaptive components compensate for each other — so visualize behavior, fit tiny datasets, verify
gradients numerically, and monitor activation/gradient statistics.
8. Be pragmatic. The widened-crop fix in Street View exemplifies choosing the cheap systematic fix over the
elegant expensive one.