Chapter 9: Selection of variables

Table of Contents
9.1. . Problem formulation ........................................................................................................................ 1
9.2. All possible regressions ..................................................................................................................... 2
[Link] ........................................................................................................................................ 2
9.2.2. Example ...................................................................................................................................... 3
9.2.3. Mallow’s C(P) ............................................................................................................................ 5
9.3. Model selection criteria ................................................................................................................... 10
9.3.1. Partial F test .............................................................................................................................. 10
9.3.2. Akaike information criterion .................................................................................................... 12
9.4. Selection-methods ........................................................................................................................... 13
9.4.1. Introduction .............................................................................................................................. 13
9.4.2. Example .................................................................................................................................... 15
9.5. Exercise .......................................................................................................................................... 18
9.6. Solution ........................................................................................................................................... 20

Chapter 9 : Regression: selection of variables 9-0

9.1. . Problem formulation

In practice: there are sometimes a lot of possible explanatory variables to

explain the response variable.

 How to reduce this number of explanatory variables?

 Examine whether all of the potential explanatory variables are needed
or whether a subset of them is adequate.
 Search the optimal subset of explanatory variables.

m candidate explanatory variables X1, X2,……Xm

response variable Y

look for the optimal subset of p X-variables ?

criterion ?

Chapter 9 : Regression: selection of variables 9-1

9.2. All possible regressions
[Link]

The all-possible-regressions procedure consider all possible subsets of

the pool of potential X variables .
Then identify a small group of regression models which are “good”
according to a specified criterion.
A detailed examination of these models can lead to the selection of the
final model.

All possible subsets?

If there are m candidate explanatory variables:
 2m regressions for all possible subsets

(e.g. if m=10, then there are 1024 possible regression models)

 A lot of computation

 Only possible for models with not too many candidates (m < 25)

Possible criteria?
 The smallest Error SS or highest R²
(Remember: R²=1-(SSE/SST))

 The highest adjusted R²

Chapter 9 : Regression: selection of variables 9-2

9.2.2. Example

Insurance

Loss Amount of damage, caused by the driver

(1000 $)
Age Age of the driver
Exper Number of years the driver has a driver license
Towork Distance from home to work (expressed in
miles)
Miles Total number of miles in one year (1000 miles)

We want to explain the LOSS of the driver based on following possible

variables: AGE, EXPER, TOWORK, MILES (m=4)

Part of the data:

> insur[1:10,c(2,4,5,8,9)]
LOSS AGE EXPER TOWORK MILES
1 0.432 59 10 14 1
2 0.488 70 42 10 6
3 0.491 54 26 0 6
4 1.030 35 17 16 4
5 0.853 33 12 6 5
6 0.128 72 30 8 7
7 0.297 63 33 7 3
8 0.542 60 35 8 4
9 0.874 42 8 14 2
10 0.558 51 17 5 20

Regression with only intercept: 1 regression

All regressions with 1 variable: 4 regressions

All regressions with 2 variables: 6 regressions

All regressions with 3 variables: 4 regressions

All regressions with 4 variables: 1 regression

In total: 16 regressions

Chapter 9 : Regression: selection of variables 9-3

loss <- insur$LOSS
age <- insur$AGE
exper <- insur$EXPER
miles <- insur$MILES
towork <- insur$TOWORK

#all possible subsets

#package leaps is used here
library(leaps)
leap <- leaps(x=cbind(age,exper, miles, towork), y=loss, method=c("r2"))

combine <- cbind(leap$which,leap$size, leap$r2)

dimnames(combine) <-
list(1:15,c("age","exper","miles","towork","size","r2"))
round(combine, digits=3)

Interpretation
(i)size = number of parameters in the model. This is number of
explanatory variables + 1
(ii) The best model with 2 parameters is the first one: with AGE (R² =
0.87)
(iii) The best model with 3 parameters in the models is the one with AGE
and TOWORK (R² = 0.90)
(iv) The best model with 4 parameters in the models is the one with AGE,
TOWORK and MILES (R² = 0.90)

(v) There is only one model with 5 parameters.

Chapter 9 : Regression: selection of variables 9-4

9.2.3. Mallow’s C(P)
This criterion is concerned with the total mean squared error of the fitted
values for each subset regression model.
This statistic is invented by Mallows in 1973.

This can be used to assure

- that all important explanatory variables are in the model
- that there aren’t too many or too less variables in the model;

C(p) = ErrorSS (RM) / MSE (FM) - (n - 2(p+1) )

p= number of variables in the reduced model.

where

 FM Full Model
regression model with all m candidate variables
 MSE is the Mean Square Error (is thus an estimate of the variance ²
of Y) for the model with m variables
or MSE(FM) = ErrorSS(FM) / n-m-1

 RM Reduced Model
regression model with p regressors
Error SS is the Error Sum of Squares

 n is the total number of observations

Chapter 9 : Regression: selection of variables 9-5

Theory of Mallow’s:

When there is no bias in the regression model with p X variables, the

expected value of Cp is approximately p+1 (number of parameters).

Hence we make a plot of Cp values against p+1 and detect:

- those models with little bias will tend to fall near the line
Cp = p+1.
- Models with substantial bias will tend to fall considerably above
this line.
- Models with Cp values below this line are interpreted as showing
no bias, being below this line due to sampling error.

In using the Cp criterion, we seek to identify subsets of X variables for

which
(1) the Cp value is small
(2) the Cp value is near p+1

 If Cp > p+1 , the model is under specified. The error is because there are
too less variables in the model.

 If Cp < p + 1 , the model is over specified. There are too many variables
in the model.

there where Cp crosses the other line (p+1), gives you an indication
about the number of variables in the model.

Chapter 9 : Regression: selection of variables 9-6

Insurance

[Link] <- leaps(x=cbind(age,exper, miles, towork), y=loss,

method="Cp")
[Link] <- cbind([Link]$which,[Link]$size, [Link]$Cp)
dimnames([Link]) <-
list(1:15,c("age","exper","miles","towork","size","cp"))
round([Link], digits=3)

age exper miles towork size cp

1 1 0 0 0 2 1944.741
2 0 1 0 0 2 15623.407
3 0 0 0 1 2 47658.806
4 0 0 1 0 2 49639.573
5 1 0 0 1 3 2.821
6 1 1 0 0 3 1945.972
7 1 0 1 0 3 1946.434
8 0 1 0 1 3 13698.865
9 0 1 1 0 3 15620.675
10 0 0 1 1 3 47653.857
11 1 0 1 1 4 3.552
12 1 1 0 1 4 4.288
13 1 1 1 0 4 1947.654
14 0 1 1 1 4 13693.331
15 1 1 1 1 5 5.000

Interpretation:
(i) How to obtain the optimal number of variables in the model?
We make a plot of cp versus size (=p+1)

plot([Link]$size, [Link]$Cp, ylim=c(1,5))

abline(a=0, b=1)

Chapter 9 : Regression: selection of variables 9-7

A model with 3 parameters (and hence 2 variables) is the best model.
This is the model with variables AGE and TOWORK.

(ii) How to obtain the value cp=1944.741 for the regression with
age in the model.

> reg.lm1 <- lm(loss~age)

> reg.lm2 <- lm(loss~age+exper+miles+towork)
>
> anova(reg.lm1)
Analysis of Variance Table

Response: loss
Df Sum Sq Mean Sq F value Pr(>F)
age 1 802.29 802.29 34348 < 2.2e-16 ***
Residuals 4998 116.74 0.02
---
Signif. codes: 0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

> anova(reg.lm2)

Chapter 9 : Regression: selection of variables 9-8

Analysis of Variance Table

Response: loss
Df Sum Sq Mean Sq F value Pr(>F)
age 1 802.29 802.29 47699.0673 <2e-16 ***
exper 1 0.01 0.01 0.7687 0.3807
miles 1 0.01 0.01 0.3182 0.5727
towork 1 32.71 32.71 1944.6540 <2e-16 ***
Residuals 4995 84.01 0.02

C(p) = ErrorSS (RM) / MSE (FM) - (n - 2(p+1) )

= [SSE(AGE)/MSE(age, exper, miles, towork)] – (5000 – 2 (2))

= [[116.74/(84.01/4995)] –(5000-2*2)

= 1901

(the difference with 1944 is possible due to rounding errors)

Chapter 9 : Regression: selection of variables 9-9

9.3. Model selection criteria
When the set of candidate variables is too large, we have to make use of
automatic stepwise selection techniques (see 9.4). What is the criterion
used to determine whether a certain model is better or worse than another
model? The partial F test or the Akaike Information criterion can be
used.

9.3.1. Partial F test

Consider two nested models for Y with p > k:

Y= β0 + β1 X1 + β2 X2 +… +βk Xk (with k+1 parameters)

and
Y= β0 + β1 X1 + β2 X2 +… +βk Xk + βk+1 Xk+1 + …+βp Xp (with p+1
parameters)

We want to test
H0 : βk+1 = … = βp = 0 versus H1: not all βi = 0, i= k+1, …p

Construction of a test statistic

Since p > k  ESS(p) < ESS(k)

Extra sum of squares= ESS(k)-ESS(p) (> 0)

If the new parameters are not really important (if H0 is true), then there
should be little difference between the sums of squares when computed
with or without the new parameters.
If they are important (if H0 is not true), then there should be a big
difference.
To measure big or small, we divide the extra sum of squares by the
residual sum of square for the largest model.

This ratio:
[ESS(k)-ESS(p)]/ESS(p) should measure the influence of the extra
parameters. If we divide the extra sum of squares by their respective
degrees of freedom, we obtain the partial F test statistic:

𝑒𝑥𝑡𝑟𝑎 𝑠𝑢𝑚 𝑜𝑓 𝑠𝑞𝑢𝑎𝑟𝑒𝑠

𝑝−𝑘
F= 𝐸𝑆𝑆(𝑝)
𝑛−𝑝−1

which has an F distribution with (p-k) and (n-p-1) degrees of freedom.

Chapter 9 : Regression: selection of variables 9-10

This partial F can be used to test hypothesis (for nested models only!)
H0 : βk+1 = … = βp = 0 versus H1: not all βi = 0, i= k+1, …p

Based on the corresponding p-value, we come to conclusions.

In R

 Consider the model with 4 explanatory variables:

AGE, EXPER, MILES, TOWORK

 Consider the model with 2 explanatory variables:

AGE, TOWORK

How to compare these models (these are nested here):

#comparison of nested models
reg.lm1 <- lm(loss ~age+towork)
reg.lm2 <- lm(loss ~age+towork +miles +exper)
anova(reg.lm1, reg.lm2)

Analysis of Variance Table

Model 1: loss ~ age + towork

Model 2: loss ~ age + towork + miles + exper

[Link] RSS Df Sum of Sq F Pr(>F)

1 4997 84.045
2 4995 84.015 2 0.031 0.9104 0.4024

Interpretation:
Model 1: Loss= β0 + β1 Age + β2 Towork
Model 2: Loss= β0 + β1 Age + β2 Towork+ β3 Miles + β4 Exper
H0 : β3 = β4 = 0 versus H1: not both β3 and β4 are 0

(i) The p-value (= 0.402) is large (> 0.05) , hence the H0 is not
rejected.
We can use model1 with only 2 explanatory variables Age and Towork.

(ii) How to obtain the value of the F test statistic (F = 0.9104)

ESS= Extra sum of squares= 84.045-84.015
F = [ESS/(4-2)]/[84.015/4995]= 0.89
Small differences are due to rounding errors.

Chapter 9 : Regression: selection of variables 9-11

9.3.2. Akaike information criterion
We have to realize that there are no ‘true’ models. The proposed models
only approximate reality!

The more parameters that there are in the model, the better the fit. You
can obtain a perfect fit when you have a separate parameter for every data
point, but this model will not have any explanatory power.

There is always a trade-off between the goodness of fit of the model and
the number of parameters required.
AIC (Akaike’s Information Criterion, 1973) is useful because it explicitly
penalizes any superfluous parameters in the model.
In a regression context, AIC (k) for a model with (k+1) parameters
(hence k explanatory variables ) and sample size n can be written as:

AIC(k) = n * ln(ESS(k)) – n*ln(n) + 2*(k+1)

Remark
1. When k increases, then ESS(k) decreases
2. n *ln(n) is independent of the choice of k
3. 2*(k+1) increases when k increases.

When comparing two models, the smaller the AIC, the better the fit.
An advantage of the AIC is that it can be used to compare models that are
not nested. This was the restriction of the partial F test.

In R
The function AIC() will compute the AIC value for a given model.

reg.lm1 <- lm(loss ~age+exper+miles+towork)

reg.lm2 <- lm(loss ~age+towork)

#compare the AIC value for 2 models

AIC(reg.lm1)
AIC(reg.lm2)
> AIC(reg.lm1)
[1] -6229.622
> AIC(reg.lm2)
[1] -6231.8
Interpretation:
The AIC value for the model with AGE and TOWORK is smaller. This
model is to be preferred.

Chapter 9 : Regression: selection of variables 9-12

9.4. Selection-methods
9.4.1. Introduction

In those cases when the pool of potential X variables contains 40 to 60 (or

even more) variables, an automatic procedure that develops the “best”
subset of X variables sequentially may be helpful.
Therefore we have the selection methods.

These methods develop a sequence of regression models. At each step,

they add or delete an X variable. The criterion used in R for comparing
the sequential models is the AIC criterion.

Weakness of the procedure:

The search methods end with a ‘single’ regression model, while the all-
possible-regression procedure can identify several regression models as
good for final consideration.

There are 3 selection methods:

 FORWARD selection
Start with a model with no variables.
Step1: try to add a variable in terms of a certain criterion.
We then obtain a regression model with one variable
Step2: try to add another variable to the given model in terms of a
certain criterion, given the fact that the first variable is already in the
model. We then obtain a model with two variables.
 Step 3 …..
Variables are added one by one to the model

Chapter 9 : Regression: selection of variables 9-13

 BACKWARD selection:
Start with a model with all possible variables in it.
Try to delete one variable (step by step) in terms of a certain criterion.

 STEPWISE selection:
Try to add variables one by one (as in the Forward procedure).
Once there are variables in the model, try to delete variables (as in the
Backward procedure). Continue by adding variables, deleting, …

The criterion which is used in R is the AIC criterion.

Chapter 9 : Regression: selection of variables 9-14

9.4.2. Example

#stepwise selection methods

#forward
[Link] <- step(lm(loss ~1, data=insur),
scope=~age+exper+miles+towork, direction="forward")

> [Link] <- step(lm(loss ~1, data=insur),

scope=~age+exper+miles+towork, direction="forward")
Start: AIC=-8467.38
loss ~ 1

Df Sum of Sq RSS AIC

+ age 1 802.3 116.7 -18782.2
+ exper 1 572.2 346.8 -13338.0
+ towork 1 33.4 885.6 -8650.4
<none> 919.0 -8467.4
+ miles 1 0.1 919.0 -8465.8

Step: AIC=-18782.15
loss ~ age

Df Sum of Sq RSS AIC

+ towork 1 32.7 84.0 -20423.2
<none> 116.7 -18782.2
+ exper 1 0.012929 116.7 -18780.7
+ miles 1 0.005154 116.7 - 18780.4

Step: AIC=-20423.19
loss ~ age + towork

Df Sum of Sq RSS AIC

<none> 84 -20423
+ miles 1 0.021336 84 -20423
+ exper 1 0.008956 84 -20422

Chapter 9 : Regression: selection of variables 9-15

#backward
reg.lm1 <- lm(loss ~age+exper+miles+towork)
[Link] <- step(reg.lm1, direction="backward")

> [Link] <- step(reg.lm1, direction="backward")

Start: AIC=-20421.01
loss ~ age + exper + miles + towork

Df Sum of Sq RSS AIC

- exper 1 0.00929 84.0 -20422.5
- miles 1 0.02167 84.0 -20421.7
<none> 84.0 -20421.0
- towork 1 32.7 116.7 -18778.9
- age 1 230.3 314.3 -13826.5

Step: AIC=-20422.45
loss ~ age + miles + towork

Df Sum of Sq RSS AIC

- miles 1 0.02134 84.0 -20423.2
<none> 84.0 -20422.5
- towork 1 32.7 116.7 -18780.4
- age 1 801.5 885.5 -8649.1

Step: AIC=-20423.19
loss ~ age + towork

Df Sum of Sq RSS AIC

<none> 84.0 -20423.2
- towork 1 32.7 116.7 -18782.2
- age 1 801.6 885.6 -8650.4
>

Chapter 9 : Regression: selection of variables 9-16

#stepwise
reg.lm1 <- lm(loss ~age+exper+miles+towork)
[Link] <- step(reg.lm1,direction="both")

> [Link] <- step(reg.lm1,direction="both")

Start: AIC=-20421.01
loss ~ age + exper + miles + towork

Df Sum of Sq RSS AIC

- exper 1 0.00929 84.0 -20422.5
- miles 1 0.02167 84.0 -20421.7
<none> 84.0 -20421.0
- towork 1 32.7 116.7 -18778.9
- age 1 230.3 314.3 -13826.5

Step: AIC=-20422.45
loss ~ age + miles + towork

Df Sum of Sq RSS AIC

- miles 1 0.02134 84.0 -20423.2
<none> 84.0 -20422.5
+ exper 1 0.00929 84.0 -20421.0
- towork 1 32.7 116.7 -18780.4
- age 1 801.5 885.5 -8649.1

Step: AIC=-20423.19
loss ~ age + towork

Df Sum of Sq RSS AIC

<none> 84.0 -20423.2
+ miles 1 0.021336 84.0 -20422.5
+ exper 1 0.008956 84.0 -20421.7
- towork 1 32.7 116.7 -18782.2
- age 1 801.6 885.6 -8650.4
>

Chapter 9 : Regression: selection of variables 9-17

9.5. Exercise
We use the cars database: [Link]

93 cars are investigated in order to compare their prices. There is also

information on the specifications of the car. In this way, a customer can
decide which car to take.

Bron: [Link]/publication/jse.

Variable Description
Manufacturer Name of manufacturer
Model Model
Type Small, sporty, compact, midsize, large
Min price Price basic version (in 1000 $)
Midrange price Price in between min. en max. price
Max price Max price (in 1000 $)
City MPG ‘Miles per Gallon’ in city
Highway MPG ‘Miles per Gallon’ on highway
Air bags standard 0 : none
1 : driver only
2: driver & passenger
Drive train type 0 : rear wheel drive
1 : front wheel drive
2 : all wheel drive
Number of cylinders Number of cylinders
Engine size Engine size (in liters)
Horsepower Horsepower (max)
RPM Revs. Per minute at maximum
horsepower

Chapter 9 : Regression: selection of variables 9-18

 Engine revolutions per Engine revolutions per mile
mile
Manual transmission 0 : no
available 1 : yes
Fuel tank capacity (in gallons)
Passenger capacity Number of passengers
Length Length, in inches
wheelbase In inches
width Width, in inches
U-turn space Expressed in feet
Rear seat room Expressed in inches
Luggage capacity Luggage capacity
Weights Weights (in pond)
Domestic 0 : non-US
1 : US

Formulate a good regression model for the MidrangePrice. You may use
following variables: Horsepower+ Length+ Luggage+Uturn +
Wheelbase+Width.

a. Use all possible subsets selection

b. Use one of the automatic selection techniques.

Chapter 9 : Regression: selection of variables 9-19

9.6. Solution
 Use of all possible subsets

#import dataset [Link]

cars <- [Link](file=[Link](), header=TRUE)
names(cars)

#regression
attach(cars)
leap <- leaps(x=cbind(Horsepower, Length, Luggage,Uturn,
Wheelbase,Width),
y=MidrangePrice, method=c("r2"), nbest=3)
combine <- cbind(leap$which,leap$size, leap$r2)
n <- length(leap$size)
dimnames(combine) <-
list(1:n,c("horsep","length","Lug","Uturn","WB","Width","size","r2"))
round(combine, digits=3)

> round(combine, digits=3)

horsep length Lug Uturn WB Width size r2
1 1 0 0 0 0 0 2 0.620
2 0 0 0 0 1 0 2 0.395
3 0 1 0 0 0 0 2 0.305
4 1 0 0 0 1 0 3 0.640
5 1 0 1 0 0 0 3 0.628
6 1 0 0 0 0 1 3 0.624
7 1 0 0 0 1 1 4 0.699
8 1 0 0 1 1 0 4 0.670
9 1 1 0 0 0 1 4 0.658
10 1 0 0 1 1 1 5 0.703
11 1 0 1 0 1 1 5 0.701
12 1 1 0 0 1 1 5 0.699
13 1 0 1 1 1 1 6 0.705
14 1 1 0 1 1 1 6 0.703
15 1 1 1 0 1 1 6 0.701
16 1 1 1 1 1 1 7 0.705

Chapter 9 : Regression: selection of variables 9-20

[Link] <- leaps(x=cbind(Horsepower, Length, Luggage,Uturn,
Wheelbase,Width),
y=MidrangePrice, nbest=3)
[Link] <- cbind([Link]$which,[Link]$size, [Link]$Cp)
dimnames([Link]) <-
list(1:n,c("horsep","length","Lug","Uturn","WB","Width","size","cp"))
round([Link], digits=3)
plot([Link]$size, [Link]$Cp, ylim=c(1,7))
abline(a=0, b=1)

horsep length Lug Uturn WB Width size cp

1 1 0 0 0 0 0 2 18.747
2 0 0 0 0 1 0 2 75.975
3 0 1 0 0 0 0 2 98.886
4 1 0 0 0 1 0 3 15.605
5 1 0 1 0 0 0 3 18.696
6 1 0 0 0 0 1 3 19.677
7 1 0 0 0 1 1 4 2.503
8 1 0 0 1 1 0 4 10.032
9 1 1 0 0 0 1 4 13.106
10 1 0 0 1 1 1 5 3.646
11 1 0 1 0 1 1 5 4.028
12 1 1 0 0 1 1 5 4.483
13 1 0 1 1 1 1 6 5.133
14 1 1 0 1 1 1 6 5.462
15 1 1 1 0 1 1 6 6.022
16 1 1 1 1 1 1 7 7.000

Chapter 9 : Regression: selection of variables 9-21

Chapter 9 : Regression: selection of variables 9-22
  Use of stepwise techniques

#stepwise selection methods

#forward
[Link] <- step(lm(MidrangePrice ~1, data=cars),
scope=~Horsepower+ Length+ Luggage+Uturn + Wheelbase+Width,
direction="forward")

#backward
reg.lm1 <- lm(MidrangePrice ~Horsepower+ Length+ Luggage+Uturn +
Wheelbase+ Width)
[Link] <- step(reg.lm1, direction="backward")

#stepwise
reg.lm1 <- lm(MidrangePrice ~Horsepower+ Length+ Luggage+Uturn +
Wheelbase+Width)
[Link] <- step(reg.lm1,direction="both")

Chapter 9 : Regression: selection of variables 9-23

> #forward
> [Link] <- step(lm(MidrangePrice ~1, data=cars),
scope=~Horsepower+ Length+ Luggage+Uturn + Wheelbase+Width,
direction="forward")
Start: AIC=377.95
MidrangePrice ~ 1

Df Sum of Sq RSS AIC

+ Horsepower 1 4979.3 3054.9 300.7
+ Wheelbase 1 3172.3 4862.0 338.8
+ Length 1 2448.8 5585.4 350.1
+ Width 1 1969.2 6065.0 356.9
+ Uturn 1 1450.2 6584.0 363.6
+ Luggage 1 1079.6 6954.7 368.1
<none> 8034.2 377.9

Step: AIC=300.66

MidrangePrice ~ Horsepower

Df Sum of Sq RSS AIC

+ Wheelbase 1 162.36 2892.54 298.18
<none> 3054.90 300.66
+ Luggage 1 64.74 2990.16 300.90
+ Width 1 33.76 3021.14 301.75
+ Length 1 28.64 3026.26 301.89
+ Uturn 1 26.58 3028.32 301.94

Step: AIC=298.18
MidrangePrice ~ Horsepower + Wheelbase

Df Sum of Sq RSS AIC

+ Width 1 476.86 2415.68 285.41
+ Uturn 1 239.12 2653.42 293.10
+ Length 1 103.05 2789.49 297.21
<none> 2892.54 298.18
+ Luggage 1 2.01 2890.53 300.12

Step: AIC=285.41

MidrangePrice ~ Horsepower + Wheelbase + Width

Chapter 9 : Regression: selection of variables 9-24

 Df Sum of Sq RSS AIC
<none> 2415.68 285.41
+ Uturn 1 27.07 2388.61 286.48
+ Luggage 1 15.01 2400.67 286.90
+ Length 1 0.62 2415.06 287.39
>

Chapter 9 : Regression: selection of variables 9-25

> #backward
> reg.lm1 <- lm(MidrangePrice ~Horsepower+ Length+ Luggage+Uturn
+ Wheelbase+ Width)
> [Link] <- step(reg.lm1, direction="backward")
Start: AIC=289.78
MidrangePrice ~ Horsepower + Length + Luggage + Uturn + Wheelbase
+
Horsepower + Width

Df Sum of Sq RSS AIC

- Length 1 4.2 2372.4 287.9
- Luggage 1 14.6 2382.8 288.3
- Uturn 1 32.3 2400.5 288.9
<none> 2368.2 289.8
- Wheelbase 1 242.0 2610.2 295.8
- Width 1 272.6 2640.8 296.7
- Horsepower 1 2305.1 4673.4 343.5

Step: AIC=287.93
MidrangePrice ~ Horsepower + Luggage + Uturn + Wheelbase + Width

Df Sum of Sq RSS AIC

- Luggage 1 16.2 2388.6 286.5
- Uturn 1 28.2 2400.7 286.9
<none> 2372.4 287.9
- Width 1 280.4 2652.8 295.1
- Wheelbase 1 413.1 2785.5 299.1
- Horsepower 1 2301.5 4673.9 341.5

Step: AIC=286.48
MidrangePrice ~ Horsepower + Uturn + Wheelbase + Width

Df Sum of Sq RSS AIC

- Uturn 1 27.1 2415.7 285.4
<none> 2388.6 286.5
- Width 1 264.8 2653.4 293.1
- Wheelbase 1 630.3 3018.9 303.7
- Horsepower 1 2421.5 4810.1 341.9

Step: AIC=285.41
MidrangePrice ~ Horsepower + Wheelbase + Width

Chapter 9 : Regression: selection of variables 9-26

 Df Sum of Sq RSS AIC
<none> 2415.7 285.4
- Width 1 476.9 2892.5 298.2
- Wheelbase 1 605.5 3021.1 301.7
- Horsepower 1 2402.2 4817.9 340.0
>

Chapter 9 : Regression: selection of variables 9-27

> #stepwise
> reg.lm1 <- lm(MidrangePrice ~Horsepower+ Length+ Luggage+Uturn
+ Wheelbase+Width)
> [Link] <- step(reg.lm1,direction="both")
Start: AIC=289.78
MidrangePrice ~ Horsepower + Length + Luggage + Uturn + Wheelbase
+
Horsepower + Width

Df Sum of Sq RSS AIC

- Length 1 4.2 2372.4 287.9
- Luggage 1 14.6 2382.8 288.3
- Uturn 1 32.3 2400.5 288.9
<none> 2368.2 289.8
- Wheelbase 1 242.0 2610.2 295.8
- Width 1 272.6 2640.8 296.7
- Horsepower 1 2305.1 4673.4 343.5

Step: AIC=287.93
MidrangePrice ~ Horsepower + Luggage + Uturn + Wheelbase + Width

Df Sum of Sq RSS AIC

- Luggage 1 16.2 2388.6 286.5
- Uturn 1 28.2 2400.7 286.9
<none> 2372.4 287.9
+ Length 1 4.2 2368.2 289.8
- Width 1 280.4 2652.8 295.1
- Wheelbase 1 413.1 2785.5 299.1
- Horsepower 1 2301.5 4673.9 341.5

Step: AIC=286.48
MidrangePrice ~ Horsepower + Uturn + Wheelbase + Width

Df Sum of Sq RSS AIC

- Uturn 1 27.1 2415.7 285.4
<none> 2388.6 286.5
+ Luggage 1 16.2 2372.4 287.9
+ Length 1 5.8 2382.8 288.3
- Width 1 264.8 2653.4 293.1
- Wheelbase 1 630.3 3018.9 303.7
- Horsepower 1 2421.5 4810.1 341.9

Chapter 9 : Regression: selection of variables 9-28

Step: AIC=285.41
MidrangePrice ~ Horsepower + Wheelbase + Width

Df Sum of Sq RSS AIC

<none> 2415.7 285.4
+ Uturn 1 27.1 2388.6 286.5
+ Luggage 1 15.0 2400.7 286.9
+ Length 1 0.6 2415.1 287.4
- Width 1 476.9 2892.5 298.2
- Wheelbase 1 605.5 3021.1 301.7
- Horsepower 1 2402.2 4817.9 340.0

Chapter 9 : Regression: selection of variables 9-29