Variable Screening Methods
Chapter 6
Chapter 06 1
Contents
Introduction: Why Use a Variable Screening
Method?
Step Regression
All-Possible-Regressions Selection
Procedure
Caveats
Chapter 06 2
1
6.1 Introduction: Why Use a Variable
Screening Method?
The model with numerous multivariable
interactions and squared terms is too
complex to be of use in practice.
Variable screening procedures
Stepwise regression
All-possible-regressions-selection
Chapter 06 3
6.2 Stepwise Regression
To run a stepwise regression, the user first
identifies the dependent variable (response) y,
and the set of potentially important
independent variables, x1 , x2 , , xk , where k is
generally large.
Chapter 06 4
2
6.2 Stepwise Regression
Step 1.
The software program fits all possible one-
variable models to the data,
E y 0 1 xi
where xi is the ith independent variable,
i 1, 2, , k.
Chapter 06 5
6.2 Stepwise Regression
For each model, test
H 0 : 1 0
H a : 1 0
Using the t-test (or the equivalent F-test) for a
single parameter.
The independent variable that produces the
largest (absolute) t-value is declared the best one-
*
variable predictor of y . Call this independent
variable x1 .
Chapter 06 6
3
6.2 Stepwise Regression
Step 2.
The stepwise program now begins to search
through the remaining (k-1) independent variables
for the best two-variable model of the form
E y 0 1 x1 2 xi
The t-values for the test H 0 : 2 0 are computed
for each of the (k-1) models, and the variable
having the largest t is retained. Call this
variable x2 .
Chapter 06 7
6.2 Stepwise Regression
Before proceeding to Step 3, the stepwise routine
will go back and check the t-value of ̂1 after ˆ2 x2
has been added to the model.
If the t-value has become insignificant at some
specified level (say 0.05 ), the variable x1 is
removed and a search is made for the
independent variable with a parameter that will
yield the most significant t-value in the presence
of ˆ2 x2 .
Chapter 06 8
4
6.2 Stepwise Regression
Step 3.
The stepwise regression procedure now checks
for a third independent variable to include in the
model with x1 and x2 .
E y 0 1 x1 2 x2 3 xi
The criterion is again to include the independent
variable with the largest t-value. Call this best third
variable x3 .
Chapter 06 9
6.2 Stepwise Regression
The programs now recheck the t-values
corresponding to the x1 and x2 coefficients,
replacing the variables that yield nonsignificant t-
values.
This procedure is continued until no further
independent variables can be found that yield
significant t-values (at the specified level) in the
presence of the variables already in the model.
Chapter 06 10
5
Example 6.1
Chapter 06 11
Figure 6.1 StatTools stepwise regression
results for executive salaries
Stepwise Regression for Y Multiple Adjusted StErr of
R-Square
Summary R R-Square Estimate
0.9595 0.9206 0.9164 0.075117921
Degrees of Sum of Mean of
F-Ratio p-Value
ANOVA Table Freedom Squares Squares
Explained 5 6.1523 1.2305 218.0606 < 0.0001
Unexplained 94 0.5304 0.0056
Standard Confidence Interval 95%
Coefficient t-Value p-Value
Regression Table Error Lower Upper
Constant 9.9619 0.1011 98.5777 < 0.0001 9.7613 10.1626
X1 0.0273 0.0010 26.5005 < 0.0001 0.0252 0.0293
X3 0.2247 0.0164 13.7424 < 0.0001 0.1922 0.2572
X4 0.0005 0.0000 11.0643 < 0.0001 0.0004 0.0006
X2 0.0291 0.0033 8.7188 < 0.0001 0.0225 0.0357
X5 0.0020 0.0005 3.9469 0.0002 0.0010 0.0029
Chapter 06 12
6
6.2 Stepwise Regression
Forward selection
Nearly identical to the stepwise procedure.
The only difference is that the forward selection
technique provides no option for rechecking the t-
values corresponding to the x’s that have entered
the model in an earlier step.
Chapter 06 13
6.2 Stepwise Regression
Backward elimination
Initially fits a model containing terms for all
potential independent variables.
The variable with the smallest t (or F) statistics is
identified and dropped from the model if the t-
value is less than some specified critical value.
This process is repeated until no further
nonsignificant independent variables can be found.
Chapter 06 14
7
Figure 6.2 StatTools backward stepwise
regression for executive salaries
Backward Stepwise Regression for Y Multiple Adjusted StErr of
R-Square
Summary R R-Square Estimate
0.9595 0.9206 0.9164 0.075117921
Degrees of Sum of Mean of
F-Ratio p-Value
ANOVA Table Freedom Squares Squares
Explained 5 6.1523 1.2305 218.0606 < 0.0001
Unexplained 94 0.5304 0.0056
Standard Confidence Interval 95%
Coefficient t-Value p-Value
Regression Table Error Lower Upper
Constant 9.9619 0.1011 98.5777 < 0.0001 9.7613 10.1626
X1 0.0273 0.0010 26.5005 < 0.0001 0.0252 0.0293
X2 0.0291 0.0033 8.7188 < 0.0001 0.0225 0.0357
X3 0.2247 0.0164 13.7424 < 0.0001 0.1922 0.2572
X4 0.0005 0.0000 11.0643 < 0.0001 0.0004 0.0006
X5 0.0020 0.0005 3.9469 0.0002 0.0010 0.0029
Chapter 06 15
6.3 All-Possible-Regressions Selection
Procedure
R 2 Criterion
SSE
R2 1
SS Total
The objective of the R 2 criterion is to find a subset
model (i.e., a model containing a subset of the k
independent variables) so that adding more
variables to the model will yield only small
increases in R 2 .
Chapter 06 16
8
6.3 All-Possible-Regressions Selection
Procedure
2
Adjusted R or MSE Criterion
MSE
Ra2 1 n 1
SS Total
Ra2 increases only if MSE decreases.
An equivalent procedure is to search for the
model with the minimum, or near minimum, MSE.
Chapter 06 17
6.3 All-Possible-Regressions Selection
Procedure
Cp Criterion
Total Mean Square Error (TMSE)
n 2
n n
TMSE E yˆi E yi E yˆi E yi Var yˆ
2
i
i 1 i 1 i 1
The objective is to compare the TMSE for the
subset regression model with , the variance of
2
the random error for the true model, using the
ratio
TMSE
2
Chapter 06 18
9
6.3 All-Possible-Regressions Selection
Procedure
A good estimator of the ratio is given by
SSE p
Cp 2 p 1 n
MSEk
The C p criterion selects as the best model the
subset model with
A small value of C p (i.e., a small total mean square error)
A value of C p near p+1, a property that indicates that
slight or no bias exists in the subset regression model.
Chapter 06 19
6.3 All-Possible-Regressions Selection
Procedure
PRESS Criterion
n
2
PRESS yi yˆ
i
i 1
Since small differences yi yˆ i indicates that the
model is predicting well, we desire a model with a
small PRESS.
Chapter 06 20
10
Figure 6.3 MINITAB all-possible-regressions
selection results for executive salaries
Chapter 06 21
Example 6.2
Chapter 06 22
11
Figure 6.4 MINITAB plots of all-possible-
regressions selection criteria for Example 6.2
Chapter 06 23
6.4 Caveats
Be wary of using the stepwise (or best subset)
model as the final model for predicting y for
several reasons.
First, the probability of making at least one Type I
error or at least one Type II error is often quite
high.
Second, analysts typically do not include higher-
order terms or interactions in the list of potential
predictors for stepwise regression.
Chapter 06 24
12
6.4 Caveats
Third, even if the analyst includes some higher-
order terms and interactions as potential
predictors, the stepwise and best subsets
procedures will more than likely select a
nonsensical model.
Finally, make sure you do not abandon your
common sense or intuition when applying
stepwise regression.
Chapter 06 25
13