Best Subsets Regression, Adjusted R-Sq, Mallows Cp

In this section, we learn about the best subsets

regression procedure (or the all possible subsets
regression procedure).

Whenever possible, researchers would prefer to allow theory to

drive their decisions about what predictors constitute a "best"
regression model. Sometimes, however, lack of strong theory
requires researchers to allow the data to speak.

In such cases in multiple linear regression, a researcher might use

statistical methods to help determine which predictors are most
useful and most worthy of continued future empirical consideration.
A number of methods exist for this process in regression, including
selection methods (e.g., forward, backward, stepwise), and best-
subset regression techniques.

In most cases, some criterion is required to help determine which

predictors and/or which models are best among the options
provided by the data. Most commonly, fit criteria of some kind are
used in multiple linear regression. Such criteria include methods to
determine the maximum strength of relationship fit and methods
based on minimizing error variation, most of which include
penalties for models that include more predictors of little value
(and sometimes combined with penalties for smaller samples).
The primary purpose of this section is to investigate how well
various subset selection criteria identify the correct regression
model under varying conditions.

Theoretical Perspectives

This section applies to standard multiple linear regression analysis,

where all possible subsets of predictors are entered in order to
create all possible models. In all subsets regression, researchers
attempt to identify, based on some criterion, the "best" subset
model of predictors. That is, researchers attempt to identify a
subset model that works well, perhaps even as well or better than
the full model of all available predictors. Some statistical programs
have procedures available for such an analysis.

One approach to best-subset regression is to compare all possible

regression models on the chosen criterion. There are 2 k-1 possible
models with predictors that can be created from a set of k
predictors. Another approach is to compare the best subset of
predictors of each possible model size.

For k predictors, there are k possible "best" models, one model of

each size from 1 to k. Researchers choose a criterion statistic to use
for the model comparisons.

For example, by default SPSS provides several statistics in its

"Model Summary" table that might be used for such a purpose: R2,
adjusted R2, standard error of the estimate (SSE), and perhaps even
R2 change and F change statistics. With the "selection" syntax
option included, SPSS also will provide the Akaike Information
Criterion (AIC), the Schwarz Bayesian Criterion (BIC), and Mallows'
Prediction Criterion (CP).
Variable Selection Methods

Variable selection generally refers to systematic techniques for

selecting a subset of predictor variables, from among a specified
set of predictors, that adequately explains or predicts the given
criterion variable. Generally, the most important tool in selecting a
subset of variables for a multiple linear regression model is careful
logical analysis based on the analyst's knowledge of theory and
research in the area of study (Gordon, 1968).

Not all researchers see the same relationships, same order of

variable importance, and same models – perhaps based on
different levels of theoretical expertise, experience, and creativity.

Variable selection constitutes strategies by which a subset of

“better” variables is chosen from among a “larger constellation of
predictors” (Thompson, 1995, p. 525). Breiman (1995) suggested
that these subsets are useful for two primary reasons: variance
reduction and simplicity (i.e., parsimony). More regression
coefficients increase the overall variance and the prediction errors.
Weisberg (1985) also indicated that deletion of predictors
from a prediction model can improve it and reduce
apparent multicollinearity.

In stepwise regression procedures, linear models are developed in a

sequence of steps by adding and/or deleting predictor variables.
That is, a path through possible models is chosen based on an
appropriate statistical criterion, identifying first a subset of one size
and then adding/deleting predictors until a final model is reached
based on some stopping criterion. Stepwise regression techniques
differ from all-subsets regression techniques because only a limited
number of models of each size are examined. All-subsets
regression, on the other hand, provides analysis of certain criterion
statistics computed for every possible model of every size. Best-
subsets regression, which we focus on in this study, then identifies
the best model at each size, from among all possible models of that
size, based on that criterion.

Best-Subsets Regression

Whereas in stepwise methods successive models are limited by

variables already in the model from previous steps of the analysis,
all-subsets regression provides analysis of certain statistical criteria
(e.g., AIC, adjusted R2, Mallow’s CP) computed for every possible
model of every size.

Then, in the best-subsets approach, for each model of a given size

the best-subset model of predictors is chosen based on the chosen
statistical criterion.

The number of total models of all sizes is 2 k-1 (where k is the

number of predictors), while the number of best-subset models
is equal to the number of predictors.

For example, for five predictors there will be 31 possible regression

models for all subsets, but only five best-subset models based on
some criterion such as the highest adjusted R2 or lowest AIC:

one best one-predictor model,

one best two-predictor model,

one best three-predictor model,

one best four-predictor model,

and the one full, five-predictor model.

Because best-subsets approaches are computer intensive, because
all possible regressions must be created, it is not always feasible to
use the method, especially as k increases.

The general idea behind best subsets regression is that we select

the subset of predictors that do the best at meeting some well-
defined objective criterion, such as having the largest adjusted
R2 value or the smallest MSE.

Again, our hope is that we end up with a reasonable and

useful regression model. There is one sure way of ending up
with a model that is certain to be underspecified—and that's if the
set of candidate predictor variables doesn't include all the variables
that actually predict the response.

Therefore, just as is the case for the selection methods in

regression, a fundamental rule of the best subsets regression
procedure is that the list of candidate predictor variables must
include all the variables that predict the response. Otherwise, we
are sure to end up with a regression model that is underspecified
and therefore misleading.

The procedure of using subset model is as follows:

A regression analysis utilizing the best subsets regression

procedure involves the following steps:

Step #1. First, identify all of the possible regression models

derived from all of the possible combinations of the candidate
predictors.

Unfortunately, this can be a huge number of possible models.

For the sake of example, suppose we have k=3 candidate
predictors x1, x2, and x3 for our final regression model. Then, there
are 23=8 possible regression models we can consider:

 the one (1) model with no predictors

 the three (3) models with only one predictor each

o the model with x1 alone;

o the model with x2 alone;
o and the model with x3 alone

 the three (3) models with two predictors each

o the model with x1 and x2;

o the model with x1 and x3;
o and the model with x2 and x3

 and the one (1) model with all three predictors

o that is, the model with x1, x2 and x3

That's 1 + 3 + 3 + 1 = 8 possible models to consider.

It can be shown that when there are four candidate predictors—

x1, x2, x3 and x4—there are 16 possible regression models to
consider.

In general, if there are k possible candidate predictors, then there

are 2k possible regression models containing the predictors.

The good news is that statistical software can be used to

find all possible models.

Step #2. From the possible models identified in the first step:
 o determine the one-predictor models that do the "best" at
meeting some well-defined criteria,
o the two-predictor models that do the "best," the three-
predictor models that do the "best," and so on.

For example, suppose we have three candidate predictors—x1, x2,

and x3—for our final regression model.

Of the three possible models with one predictor, identify the one or
two that does "best." Of the three possible two-predictor models,
identify the one or two that does "best."

By doing this, it cuts down considerably the number of possible

regression models to consider!

But, have you noticed that we have not yet even defined
what we mean by "best"?

What do you think "best" means?

"best" means, you might have thought of any of the following:

 the model with the largest R2

 the model with the largest adjusted R2

 the model with the smallest MSE (or S = square root of MSE)

There are other criteria you probably didn't think of, but we could
consider, too, for example, Mallows' Cp-statistic,
the PRESS statistic, AIC,BICand Predicted R2 (which is calculated
from the PRESS statistic).

To make matters even worse—the different criteria quantify

different aspects of the regression model, and therefore often yield
different choices for the best set of predictors. We should use
best subsets regression as a screening tool that is, as a way
to reduce the large number of possible regression models
to just a handful of models that we can evaluate further
before arriving at one final model.

Step #3. Further evaluate and refine the handful of models

identified in the last step. This might entail performing residual
analyses, transforming the predictors and/or response, adding
interaction terms, and so on. Do this until you are satisfied that you
have found a model that meets the model conditions, does a good
job of summarizing the trend in the data, and most importantly
allows you to answer your research question.

USING THE CP CRITERION TO IDENTIFY "BEST"

MODELS
Recalling that k denotes the number of predictor or independent
terms in the model:

 Subset models with small Cp values have a small estimated

total (standardized) variation in predicted responses.

 When the Cp value is ...

o ... near k+1=p, the bias is small (next to none)

o ... much greater than k+1, the bias is substantial/large.

o ... below k+1, it is due to sampling error; interpret as no

bias

 For the largest model containing all of the candidate

predictors, Cp = k+1 (always).
  Therefore, you shouldn't use Cp to evaluate the full model (the
model containing all of the candiate predictors).

That all said, here's a reasonable strategy for using Cp to identify

"best" models:

 Identify subsets of predictors for which the Cp value

is near k+1 (if possible).

 The full model always yields Cp = k+1, so don't select the full
model based on Cp.

 If all models, except the full model, yield a

large Cp not near k+1, it suggests some
important predictor(s) are missing from the
analysis. In this case, we are well-advised to identify the
predictors that are missing!

 If a number of models have Cp near k+1, choose the model

with the smallest Cp value, thereby insuring that the
combination of the bias and the variance is at a minimum.

 When more than one model has a small value of Cp value near
k+1, in general, choose the simpler model or the model that
meets your research needs.

 The Mallows Cp formula

Adjusted R2 statistic

Recall from Chapter 3 that we can measure the prediction quality in linear regression
via the R2 statistic.

This is the proportional reduction in the residual sum of squares – an estimate of the
proportional reduction of error that the predictor allows us to achieve. The R2 statistic is
a standardised measure of the quality of the prediction.

There are two reasons, however, why it is not advisable to use R2 for model
comparison:

1. If we compare two models that are nested, then the larger model will
never have a smaller R2 than the smaller model, for a given data set. In
other words, adding a predictor to a model can never make our
prediction of the outcome in the same data set worse. So if we used
R2 for model comparison, we would always decide in favour of
the larger model.
2. R2 measures only one aspect of model quality, namely, the fit of the
model. It fails to take account of parsimony.

A measure of model quality that avoids these two disadvantages of R2 is the

adjusted R2 .

The adjusted R2 statistic may be used to compare nested as well as non-nested

models.

NB: Adjusted R2 is the better model when you compare models that have
a different amount of variables. The logic behind it is, that R 2 always
increases when the number of variables increases. Meaning that
even if you add a useless variable to your model, your R 2 will still increase.
 2
adjusted
( 1−r 2 ) ×df total
r =1−
df error
Adjusted 𝑅-squared is the value of 𝑅-squared which has been penalized
for thenumber of variables added to the model

Why do we have to Adjust 𝑅2 ?

For multiple linear regression there are 2 problems:

Problem 1: Every time you add a predictor to a model, the R-squared

increases, even if due to chance alone. It never decreases. Consequently, a
model with more terms may appear to have a better fit simply because it has
more terms.

Problem 2: If a model has too many predictors and higher order polynomials,
it begins to model the random noise in the data. This condition is known as
over-fitting the model and it produces misleadingly high R-squared values and
a lessened ability to make predictions.

Therefore for Multiple Linear Regression you need to report the Adjust 𝑅2 which
accounts for the number of predictors you had to added herefore Adjusted 𝑅-
squared is always smaller than 𝑅2
 Adjusted R2 does not always increase as more variables are added. In fact, if
unnecessary terms are added, Adjusted R2 may decrease.

Akaike’s Information Criterion (AIC)

The model fit (AIC value) is measured ask likelihood of the parameters being correct for
the population based on the observed sample.
The number of parameters is derived from the degrees of freedom that are left
AIC value roughly equals the number of parameters minus the likelihood of the overall
model – Therefore the smaller the AIC value the better the model
Allows us to balance over- and under-fitting in our modelled relationships – We want a
model that is as simple as possible, but no simpler
A reasonable amount of explanatory power is traded off against model size – AIC
measures the balance of this for us.

AIC=nln( SSE )+2 p−nln(n)

F-TESTS AND NESTED MODELS:

A core concept in statistics is comparing nested models. Consider the model

The following reduced models are special cases (nested within) the full or complete
model (1):

By setting certain coefficients in (1) to zero, we get the various reduced models.

Assessing Predictors and Testing Coefficients via Nested Models: By comparing a

given reduced model with the complete model we can assess the usefulness of one
or more predictors and formally test whether or not the corresponding coefficients
are nonzero. There are various methods and statistics for comparing nested
statistical models. The most popular method for comparing nested regression
 models uses the Sum of Squares Error.

For example, suppose we want to compare the following two models:

The bottom, reduced model (3) is nested in the top, complete model (2), that is, (3)
is the special case of (2) when β2 = 0. Let SSE(reduced) denote the SSE for model
(3) and SSE(full) denote the SSE for model (2).

THE GENERAL F TEST FOR COMPARING NESTED MODELS

 Again, as usual, we are going to follow six steps to perform this test.

However, it is important to, first of all, write down the two models that we compare in
order to be able to perform the six steps of the hypothesis test.

We must also remember that both models will be fitted to the same data and
therefore there will be two printouts (one for each model) that we will use to
perform the test.

EXAMPLE

We want to know whether the additional terms in the

complete model (i.e. those terms that does not appear
in the reduced model), is important to include or not.

STEPS

E(y)=β0+β1x1 (reduced model)

E(y)=β0+β1x1+β2x2+β3x3 (complete

model) Then, the hypotheses are:

H0:β2=β3=0

H1: At least one of the betas differ from zero

CRITICAL VALUE

F(df(redued)-df(full);df(full))

DECISION:

Reject/ fail to reject the null hypothesis

CONCLUSION

We want to know whether the additional terms in the complete model (i.e.
those terms that does not appear in the reduced model), is important to
include or not.
Number C(p) R- SSE Variables in Model
in Square
Model
2 0.7580 3.49804 Age Mileage
1 0.6707 4.76024 Age
1 0.0788 13.31748 Mileage
 Number of Observations Read 15
Number of Observations Used 15

Analysis of Variance
Source D Sum of Mean F Value Pr > F
F Squares Square
Model 2 10.95929 5.4796 18.80 0.0002
4
Error 12 3.49804 0.2915
0
Corrected Total 14 14.45733

Root MSE 0.53991 R-Square 0.7580

 Dependent 1.68667 Adj R-Sq 0.7177
Mean
Coeff Var 32.01052

Parameter Estimates
Variable DF Parameter Standard t Val Pr > |t| 95% Confidence
Estimate Error ue Limits
Intercept 1 -0.84490 0.49645 -1.70 0.1145 -1.92656 0.23677
Age 1 0.52528 0.09050 5.80 <.0001 0.32809 0.722
Mileage 1 0.03994 0.01920 2.08 0.0595 -0.00188 0.08177