Selecting the Best Regression Equation
Chapter 15 Notes — Draper & Smith | Applied Regression Analysis
What is 'Best' Regression?
When we have k predictor variables X■, X■, …, Xk (and derived forms like squares, cross-products, logs),
we need to decide which subset to include in the final model. Two competing goals create the tension:
• Goal 1 (Bias control): Include enough variables to keep prediction bias small and fitted values reliable.
• Goal 2 (Variance + Cost): Keep the model lean — average variance of ■■ is pσ²/n, so more parameters
→ larger variance; also, measuring & monitoring many variables is expensive.
The compromise is 'selecting the best regression equation.' No single unique procedure exists; personal
judgment is always needed.
Three Model-Fit Criteria Used Across All Methods
Criterion Symbol What It Measures Goal
R-squared R² Proportion of variation explained by the model Maximise
Residual Mean Square s² Estimate of σ² — average unexplained variance Minimise / stabilise
Mallows Cp C■ Total mean-square error (variance + bias²) estimate Find C■ ≈ p
■ All three criteria are mathematically related. The choice of which to emphasise is a matter of context and personal
preference.
1. All Possible Regressions
Definition
Every possible regression equation involving any subset of the r candidate predictor variables Z■, Z■, …, Zr
is fitted and evaluated. Since each Z■ can be either in or out, there are 2■ total equations (including the
intercept-only model). With r = 10 predictors that is 1,024 equations; with r = 20 it exceeds a million.
Procedure — Step by Step
Step 1 — Partition
Divide all equations into sets: Set A (intercept only), Set B (1-variable models), Set C (2-variable models), … Set
E (all r variables).
Step 2 — Rank within sets
Within each set, order models by R², s², or C■ value. The best model in each set is the 'leader.'
Step 3 — Look for a pattern
Examine the leaders across sets. Identify the point at which adding another variable brings only marginal
improvement.
Step 4 — Decide
Choose the equation that best balances fit and parsimony, using domain knowledge alongside statistical criteria.
Using the R² Statistic
After forming the sets and ranking, scan across sets. If R² increases substantially from Set B → C → D, more
variables are needed. Once R² barely changes (e.g., 0.979 with 2 vars vs 0.982 with 3), additional variables are
not worth the cost.
• Correlated predictors (|r■■| close to 1) explain the same variance — once one enters, the other adds little.
• Look for which individual predictor is the leader in Set B to guide later interpretation.
Using the Residual Mean Square (s²)
Plot average s²(p) against p (number of parameters including β■). The plot typically drops sharply, then levels
off and approaches σ² asymptotically as all important variables are included. Overfitting occurs when extra,
unneeded predictors are added — s² stabilises rather than continues to decrease.
• The minimum individual s² in any set can be lower than the set average — always look at single-equation
values, not just averages.
• Useful rule: choose the fewest variables whose s² is close to the asymptotic (stabilised) level.
Using the Mallows C■ Statistic
C■ was proposed by C. L. Mallows as an estimate of total discrepancy (variance error + bias error) between the
fitted model and the true model.
C■ = RSS■ / s² − (n − 2p)
where RSS■ = residual sum of squares for a p-parameter model, s² = residual mean square from the full model
(all r variables), n = number of observations.
Model type Expected C■ Interpretation
Adequate (unbiased) ≈p Points fall near or on the C■ = p line
Biased (important vars missing) >> p Points fall far above the C■ = p line
Full model (all vars) =r+1 Always perfect by definition — ignore this point
Plot C■ vs p. Models near the 45° line (C■ ≈ p) are unbiased. The 'best' model is the one with the smallest C■
that is also close to p. When choice is not clear-cut, you must decide between a smaller biased model and a
larger unbiased but higher-variance model — judgment call.
'Best Subset' Regression — a Practical Variant
Instead of examining all 2■ equations, a computer program identifies the top K equations for each model size
(1-variable, 2-variable, etc.). 'Best' can be defined by maximum R², maximum adjusted R², or minimum C■. This
saves computation while still surfacing near-optimal models.
• Drawback 1: Tends to suggest too many predictors.
• Drawback 2: If K is too small, the truly best model may not appear.
• Drawback 3: No information on how each subset was found.
Properties of All Possible Regressions
• Completely exhaustive — guaranteed to find the globally optimal subset.
• Computationally expensive for large r (2■ models); impractical for r > ~20.
• Gives a full picture of trade-offs between model size and fit.
• Does NOT automatically select one model — still requires human judgment.
• Highly correlated predictors produce many near-equivalent models, making selection ambiguous — other
knowledge must break the tie.
■ Opinion: While exhaustive, all-possible-regressions is generally unwarranted for large problems. The sheer
volume of output is difficult to review meaningfully. Best-subset programs offer a more manageable alternative for
quick screening.
2. Stepwise Regression
Definition
Stepwise regression is a sequential variable selection procedure that builds up a model by adding one
variable at a time, but after each addition it checks whether any variable already in the model has become
redundant and should be removed. It combines the ideas of Forward Selection and Backward Elimination.
Core Mechanism
Variable entry and removal are both governed by partial F-values. At each step, the partial F-statistic for variable
Z■, given all other variables already in the model, measures Z■'s marginal contribution.
Partial F = [Change in RSS from adding Z■] / s²
• F-to-enter (F■■■): threshold that a variable's partial F must exceed to be admitted.
• F-to-remove (F■■■): threshold below which a variable's partial F triggers removal.
■ Recommended practice: set F■■■ ≈ F■■■, e.g., both at α = 0.05 or F = 4. Setting F■■■ < F■■■ can cause
cycling (variable added then immediately removed).
Step-by-Step Algorithm
Step Action Decision rule
1 Find Z most correlated with Y — compute its partial F (= If F > F■■■: enter it. Else: stop, use ■ only.
simple regression F).
2 With Z■ in model, compute partial F for every remaining Enter the variable with the highest partial F if it
variable. exceeds F■■■.
3 Check ALL variables currently in the model for removal. Remove the variable with the smallest partial F if it
is < F■■■. Remove only one at a time, then
recheck.
4 Repeat Steps 2–3 until no variable can enter or leave. Stop. Report the final equation.
Key Properties
Advantages Disadvantages / Cautions
• Economical — examines far fewer models than • Does NOT guarantee the globally optimal subset.
all-possible-regressions. • Final model depends heavily on the chosen α (or
• Avoids working with more variables than F) levels.
necessary at any stage. • Large α (small F) → more variables enter; small α
• Improves the equation at every step. → fewer.
• Output is easy to read and trace step by step. • At α ≈ 0.15, columns of pure random numbers are
• Can handle moderately large predictor sets (r up to sometimes selected!
~30+). • Partial F-tests at each step do not follow a
standard F-distribution (the correct null
distribution is very hard to obtain).
• Easy to abuse — blind trust in automatic output is
dangerous.
Difference from Forward Selection
Forward Selection is simpler: it only adds variables and never removes any once admitted. Stepwise regression
re-checks all variables in the model after each addition — this is its key advantage over pure forward selection,
because a variable useful when it entered may become redundant after a later, correlated variable is added.
Significance Level Guidance
• Use α = 0.05 for both entry and removal as a starting point.
• Using a smaller F (larger α) admits more variables and gives a bigger printout to inspect — useful for
exploratory work.
• The 'actual' α corresponding to a given F-to-enter is substantially larger than the nominal α because the
selection inflates Type I error.
• MINITAB defaults: Fenter = 4, Fremove = 4 (equivalent to α ≈ 0.05–0.10).
■ Opinion: Stepwise regression is one of the best practical procedures. It is economical, improves the model at
every step, and produces a clear audit trail. However, sensible judgment is still required for the initial variable pool
and for residual diagnostics. Never rely blindly on the computer output.
3. Backward Elimination
Definition
Backward Elimination starts with the full regression model containing all r candidate predictor variables and
then eliminates them one by one, removing the least useful variable at each step, until no further elimination
is warranted.
Step-by-Step Algorithm
Step Action Decision
1 Fit the full model with all r predictors. —
2 Compute partial F for every predictor, treating each as if it Find the variable with the smallest partial F, call
were the last variable to enter. it F■.
3a If F■ < F■ (pre-set threshold, e.g. F(1,n-p,0.95)) Remove that variable; refit model without it;
return to Step 2.
3b If F■ ≥ F■ Stop. The current model is the final selected
equation.
Key Properties
Advantages Disadvantages / Cautions
• Every variable gets a chance to be in the model — • Once a variable is eliminated it is GONE
nothing is missed at entry. FOREVER — no re-entry is allowed.
• Useful when the analyst wants to 'see' every • All alternative models using the eliminated variable
predictor at least once. are never examined.
• Computationally economical compared to • If X'X is nearly singular (ill-conditioned), the full
all-possible-regressions. model may be meaningless due to rounding
• Good estimate of σ² from the full model guides all errors.
subsequent partial F-tests. • Like stepwise, the partial F-tests are not strictly
• Satisfying for statisticians who like to start with valid statistically because of the sequential
everything and then prune. selection.
• Generally not as flexible as stepwise regression.
Comparison: Backward vs Forward vs Stepwise
Feature Forward Selection Backward Elimination Stepwise
Starting model Empty (intercept only) Full (all predictors) Empty (intercept only)
Direction Add only Remove only Add AND remove
Can revisit? No — once in, always in No — once out, always out Yes — variables can re-enter
Needs full model No Yes — must fit full model No
first?
Recommended? Not generally (variables not Useful when r is moderate Yes — preferred by Draper &
re-tested) Smith
■ Opinion: Backward elimination is satisfactory, especially for statisticians who like to see all variables in the
equation at least once. However, once a variable is removed it cannot re-enter — a serious limitation compared to
stepwise regression.
4. Forward Selection (Simplified Stepwise)
Definition
Forward Selection is the simplest sequential method. It starts with no predictors and adds them one at a time
based on the highest partial F-value, but it never removes a variable once it has been admitted. It was the
first edition of what became stepwise regression.
Algorithm
• Step 1: Start with the intercept-only model.
• Step 2: Compute the partial F for every candidate predictor not yet in the model.
• Step 3: Select the predictor with the highest partial F. If it exceeds F■■■, add it.
• Step 4: Repeat until no remaining variable has a partial F exceeding F■■■.
• Step 5: Stop. The current model is final — no removals are ever performed.
Properties
Advantages Disadvantages
• Simple to implement and explain. • Cannot detect that a previously entered variable
• Fast — no removal testing needed. has become redundant after a later correlated
variable enters.
• Useful when it is specifically desired that variables
once retained are never removed. • Often retains too many variables.
• Generally NOT recommended — stepwise
regression strictly dominates it.
■ In the Hald data example: Forward Selection at Step 6 would test only X■ and just reject it (at α = 0.05). It would
NOT discover that X■ had become the weakest variable. At α = 0.10, X■ would have been retained, giving a
3-variable model — different from stepwise. This illustrates how the two procedures can diverge.
5. Cautionary Remarks & Overall Summary
Dangers of Unplanned (Observational) Data
All selection procedures work best on designed experiments. With observational/unplanned data, several
dangers exist:
• Latent variables: The error term may be partly explained by unmeasured confounders. Their omission can
make a visible variable appear significant when it is not.
• Narrow operating ranges: Key predictors may be kept in a small range to maintain process stability. This
artificially suppresses their regression coefficient, making them appear 'non-significant' even when
practitioners know they matter.
• Induced correlations: Operating policies (e.g., 'if X■ rises, reduce X■') create large inter-predictor
correlations, making it impossible to separate the effects of X■ and X■.
General Warnings for ALL Selection Methods
Warning Detail
No unique 'best' model Different methods often select different equations. The same method with different α
levels selects different equations.
p-values are not trustworthy After selection, the partial F-tests no longer follow standard F-distributions. True α is
much larger than nominal α because of multiple implicit comparisons.
Statistical ≠ Scientific A selected equation may not reflect the true causal structure. Domain knowledge must
guide interpretation.
High α risk with random At α = 0.15, columns of pure random numbers are often selected. Keep α ≤ 0.05 for
predictors serious work.
Collinearity issues Highly correlated predictors leave many near-equivalent models. Check Variance
Inflation Factors (VIFs) and X'X determinant.
Residual analysis is mandatory No selection method replaces thorough residual diagnostics. Patterns in residuals
may suggest transformations or missing terms.
Recommended Order of Preference (Draper & Smith)
# Method When to use
1 Stepwise Regression Default choice for most practical problems. Economical and improves at every
step.
2 Best Subsets (with C■) Use after stepwise to explore 'nearby' models and confirm. Manageable for r ≤
~30.
3 Backward Elimination When you want all variables examined at least once; moderate r and
well-conditioned X'X.
4 All Possible Regressions Only when r is very small (≤ 10). Complete picture but impractical for large r.
— Forward Selection Generally not recommended; use stepwise instead.
Quick-Reference Summary Card
Method Start Direction Criterion Complexity Optimal?
All Possible — All subsets R², s², C■ O(2■) Yes (global)
Best Subsets — All subsets (top K) R², adj-R², C■ O(2■) approx Near-optimal
Forward Sel. Empty Add only Partial F O(r²) Not guaranteed
Backward Elim. Full Remove only Partial F O(r²) Not guaranteed
Stepwise Empty Add + Remove Partial F (both) O(r²) Not guaranteed
The Golden Rule of Variable Selection
All selection procedures are tools for the orderly display and review of data. Applied with common sense,
they produce useful results. Applied mechanically and blindly, they may be useless or even misleading.
Domain knowledge, basic understanding of the data-generating process, and careful residual analysis can
never be set aside — no algorithm replaces these.
Reference: Draper, N.R. & Smith, H. (1998). Applied Regression Analysis, 3rd Edition. Wiley. Chapter 15.