Chapter 8

Multiple Linear Regression

INDE3151: Forecasting Methods ISIK University Dr. Nuri Volkan Kayaçetin

❑ Multiple Regression Basics

❑ Model Assumptions

❑ Coefficient Estimates

❑ Multicollinearity

❑ Coefficient of Determination

❑ Significance Tests

❑ Additional Aspects

❑ Case Study

Roadmap 2

INDE3151: Forecasting Methods ISIK University Dr. Nuri Volkan Kayaçetin

❑ The use of a multiple regression model for forecasting purposes
assumes there exists a well-defined statistical model that can be used
to predict the variable of interest using empirical data.

❑ The theoretical form of the multiple linear regression model is:

𝐾

𝑌𝑖 = 𝛼 + ෍ 𝛽𝑘 𝑋𝑘,𝑖 + 𝜀𝑖
𝑘=1

o 𝑋𝑘,𝑖 𝑖th observation for explanatory variable 𝑘

o 𝛼: fixed population intercept parameter
o 𝛽𝑘 : fixed population slope parameter for variable 𝑘
o 𝜀𝑖 : random forecast error for observation 𝑖.

Multiple Linear Regression 3

INDE3151: Forecasting Methods ISIK University Dr. Nuri Volkan Kayaçetin

❑ The linear regression model represents a line in the case of a single
explanatory variable and a plane in case of two explanatory variables.

❑ In case of more than two explanatory variables, the dependent

variable is mapped into a higher dimensional surface, a hyperplane,
which is a conceptual geometrical form that is impossible to illustrate
in our three-dimensional world and difficult to interpret visually.

Geometry of Linear Regression 4

INDE3151: Forecasting Methods ISIK University Dr. Nuri Volkan Kayaçetin

❑ The population parameters will be estimated from a small sample.
The estimated model will be:
𝐾

𝑌𝑖 = 𝑎 + ෍ 𝑏𝑘 𝑋𝑘,𝑖 + 𝑒𝑖
𝑘=1

where 𝑌𝑖 , 𝑋1,𝑖 , …, 𝑋𝑘,𝑖 represent the 𝑖𝑡ℎ observation of each variable,

𝑎 𝑎𝑛𝑑 𝑏0 , 𝑏1 , … , 𝑏𝑘 are the sample estimates of unknown population
intercept and slope parameters, and 𝑒𝑖 is a normally distributed
regression residual with mean zero and variance 𝑆𝑒2 .

❑ The form of the regression model is linear in the coefficients, which

means that coefficient estimates can be calculated using the ordinary
least squares method. This computation, however, is more complex.

Multiple Regression: Basics 5

INDE3151: Forecasting Methods ISIK University Dr. Nuri Volkan Kayaçetin

❑ The explanatory variables in the model are assumed to be either fixed
numbers measured without noise or random numbers measured with
noise that is uncorrelated with the dependent variable.

❑ The explanatory variables are assumed to be linearly related to the

dependent variable.

❑ The regression residuals are assumed to be serially uncorrelated.

❑ The regression residuals should not be correlated with the dependent

variable or any of the independent variables.

❑ The error terms should be approximately normally distributed with a

mean of zero and constant variance.

Multiple Linear Regression Assumptions 6

INDE3151: Forecasting Methods ISIK University Dr. Nuri Volkan Kayaçetin

❑ If the assumed form is incorrect, then the forecasts may be inaccurate
and the F-test, t-tests, and confidence intervals are not strictly valid.
The model needs to be re-estimated in a more appropriate form.

❑ If the regression residuals are not independent, the use of the F and t-
tests and confidence intervals is not strictly valid, and the coefficient
estimates may be unstable.

❑ If the regression residuals are not homoscedastic, the use of the F and
t-tests and confidence intervals is not valid. The heteroscedasticity
problem can often be corrected using a mathematical transformation.

❑ The violation of the normality makes no difference to the estimates of

the coefficients but affects the F- and t-tests and confidence intervals.

Violation of Model Assumptions 7

INDE3151: Forecasting Methods ISIK University Dr. Nuri Volkan Kayaçetin

❑ A computer program is normally used to solve for the coefficients in
a multiple regression model. It is, however, important to get a good
understanding of what is behind the method.

❑ The method of least squares is used to find the minimum sum of

squares of the error terms, that is, find 𝑏0 , 𝑏1 , … , 𝑏𝑘 to minimize
𝑛 𝑛

𝑆𝑆𝐸 = ෍(𝑌𝑖 − 𝑌෠𝑖 )2 = ෍(𝑌𝑖 − 𝑏0 −𝑏1 𝑋1,𝑖 −𝑏2 𝑋2,𝑖 − ⋯ − −𝑏𝑘 𝑋𝑘,𝑖 )2
𝑖=1 𝑖=1

❑ The problem is solved by taking partial derivative of 𝑆𝑆𝐸 with respect

to each of the unknown coefficients 𝑏0 , 𝑏1 , … , 𝑏𝑘 , setting these partial
derivatives equal to zero, and solving a set of 𝑘 + 1 equations in 𝑘 + 1
unknowns to get the estimated values for 𝑏0 , 𝑏1 , … , 𝑏𝑘 .

Coefficient Estimates 8

INDE3151: Forecasting Methods ISIK University Dr. Nuri Volkan Kayaçetin

❑ Before getting down to quantitative analysis of multiple regression ,
it is advisable to observe two sets of correlations:
• simple correlations between 𝑌 and each of 𝑋 variables
• simple correlations between all pairs of 𝑋 variables

❑ The first set of correlations indicates how each 𝑋 variable, on its

own, relates to 𝑌, while the second set is helpful in selecting the
appropriate 𝑋 variables for a regression model and, at a deeper level
of analysis, is critical for examining multicollinearity.

❑ Multicollinearity occurs when an 𝑋 variable or a linear combination

of a subset of 𝑋 variables is highly correlated with another 𝑋 variable
or a linear combination of another subset of 𝑋 variables.

Correlations between Variables 9

INDE3151: Forecasting Methods ISIK University Dr. Nuri Volkan Kayaçetin

❑ The reason for concern about multicollinearity is, first and foremost,
a computational one. If perfect multicollinearity exists in a regression
problem, it is not possible to carry out the least squares procedure.

❑ The first concern regarding multicollinearity is that, if it exists in near

perfect form, the least squares solutions would be affected by round-
off error problems in some calculators and computer packages.

❑ The major concern however is about the stability of the coefficients.

❑ As multicollinearity becomes closer to perfect, regression coefficients

computed by standard regression programs will become (a) unstable
as measured by the coefficient standard errors, and (b) unreliable, in
that different computer programs are likely to give different estimates.

Problem of Multicollinearity 10

INDE3151: Forecasting Methods ISIK University Dr. Nuri Volkan Kayaçetin

❑ To understand the problem of multicollinearity, it is useful to look at
the simplest possible setting in which it can occur, i.e., a regression
with only two explanatory variables. In this special case, the standard
error of the coefficient estimates can be written as:
𝜐𝑒∗
𝑠𝑒𝑏 =
1 − 𝑟𝑋21,𝑋2

❑ Clearly, as the correlation between the two 𝑋 variables approaches +1

or –1, the denominator in the equation above approaches 0.

❑ Dividing by a number that approaches 0 means exploding the standard

error; if the standard error of a coefficient estimate is very large, the
analyst cannot put much faith in the value of that coefficient.

Problem of Multicollinearity 11

INDE3151: Forecasting Methods ISIK University Dr. Nuri Volkan Kayaçetin

❑ Multicollinearity does not affect the forecasting ability of a model.

❑ It is only problematic if the individual regression coefficients are of

interest or if the contribution of one explanatory variable needs to be
determined without the influence of other explanatory variables.

❑ A common false belief is that an examination of the correlations

between all pairs of explanatory variables in a multiple regression
can decisively reveal the presence or absence of multicollinearity.

❑ A close to perfect negative/positive correlation between any pair of

explanatory variables suggests the presence of multicollinearity; but
it cannot decisively be inferred that multicollinearity does not exist
based on absence of such correlations between explanatory variables.

Implications of Multicollinearity 12

INDE3151: Forecasting Methods ISIK University Dr. Nuri Volkan Kayaçetin

❑ The correlation between 𝑌 and 𝑌෠ designated 𝑅, is referred to as the
multiple correlation coefficient, while the square of this correlation,
or 𝑅2 , is called the coefficient of determination.

❑ Both multiple correlation coefficient and coefficient of determination

are bounded between 0 and 1, where a value of zero (one) indicates
that none (all) of the variation in Y is explained by the X variables.

❑ As in the case of simple regression, the coefficient of determination

measures the proportion of total variability in the forecast variable
that can be accounted for by the explanatory variables in the model.
2
2
σ 𝑌෠𝑖 − 𝑌ത 𝐸𝑥𝑝𝑙𝑎𝑖𝑛𝑒𝑑 𝑆𝑢𝑚 𝑜𝑓 𝑆𝑞𝑢𝑎𝑟𝑒𝑠 𝑆𝑆𝑅
𝑅 = = =
σ 𝑌𝑖 − 𝑌ത 2 𝑇𝑜𝑡𝑎𝑙 𝑆𝑢𝑚 𝑜𝑓 𝑆𝑞𝑢𝑎𝑟𝑒𝑠 𝑆𝑆𝑇

Coefficient of Determination 13

INDE3151: Forecasting Methods ISIK University Dr. Nuri Volkan Kayaçetin

❑ The coefficient of determination of a model can be mechanically
increased through including additional explanatory variables in the
regression even when the added variables are not important. For this
reason, analysts tend prefer the adjusted coefficient of determination.
𝑛−1
𝑅ത 2 = 1 − (1 − 𝑅2 )
𝑛−𝑘−1

❑ The adjusted and unadjusted coefficients of determination will only

be equal in a model with an intercept and no explanatory variables.
For any other model, we will have 0 ≤ 𝑅ത 2 ≤ 𝑅2 .

❑ The greater the number of predictors in the model, the lower the
adjusted R2 will be compared to the unadjusted R2.

Adjusted Coefficient of Determination 14

INDE3151: Forecasting Methods ISIK University Dr. Nuri Volkan Kayaçetin

❑ After estimating the coefficients of a regression model to determine
the 𝑌෠ values, there will be a set of errors of fit, 𝑒𝑖 = 𝑌𝑖 − 𝑌෠𝑖 , for the ith
observation. One way to discuss the goodness of fit is to partition the
deviation of actuals from the mean value into two parts:
𝑌𝑖 − 𝑌ത𝑖 = (𝑌𝑖 −𝑌෠𝑖 ) + (𝑌෠𝑖 − 𝑌ത𝑖 )

❑ The degrees of freedom for this partition satisfy the following relation:
𝑇𝑜𝑡𝑎𝑙 𝐷𝑜𝐹 = 𝑈𝑛𝑒𝑥𝑝𝑙𝑎𝑖𝑛𝑒𝑑 𝐷𝑜𝐹 + 𝐸𝑥𝑝𝑙𝑎𝑖𝑛𝑒𝑑 𝐷𝑜𝐹

❑ The total degrees of freedom in a multiple regression equals 𝑛 − 1,

i.e., the number of observations minus one. The explained degrees of
freedom equals the number of estimated slope coefficients, 𝑘. Finally,
the unexplained degrees of freedom is 𝑛 − 𝑘 − 1.

F-Test for Overall Significance 15

INDE3151: Forecasting Methods ISIK University Dr. Nuri Volkan Kayaçetin

❑ It is now possible to construct an overall F-test to check on the
statistical significance of the regression model.

❑ Since the F statistic of the regression model is defined as the ratio of

two variances (or “mean squares” as referred to by statisticians), we
have to convert “sums of squares” to “mean squares” as follows:

σ(𝑌෠ 𝑖 −𝑌ത 𝑖 )2 /𝑘 𝑅2 /𝑘
𝐹= σ(𝑌𝑖 −𝑌෠ 𝑖 )2 /(𝑛−𝑘−1)
=
(1−𝑅2 )/(𝑛−𝑘−1)

❑ The F-test is sensitive to the relative strengths of the numerator and

denominator. If the regression model is not doing well, unexplained
variance of error terms will be large, and F-statistic becomes smaller.
The vice versa occurs if explained variance of error terms is large.

F-Test for Overall Significance 16

INDE3151: Forecasting Methods ISIK University Dr. Nuri Volkan Kayaçetin

F-Test for Overall Significance 17

INDE3151: Forecasting Methods ISIK University Dr. Nuri Volkan Kayaçetin

❑ If the p-value in the 𝐴𝑁𝑂𝑉𝐴 table is small (less than adopted α), this
suggests that, with confidence (1–α), that the explanatory variables
account for a significant part of the variation in the forecast variable.

❑ The 𝐴𝑁𝑂𝑉𝐴 table does not show whether all explanatory variables
are important, or which one is more important; but it does show that
there is a significant relation between 𝑌 and explanatory variables.

❑ If the p-value is not given, it would be looked up from the F table.

F-Test for Overall Significance 18

INDE3151: Forecasting Methods ISIK University Dr. Nuri Volkan Kayaçetin

❑ After examining the overall significance of the regression model, it is
sometimes useful to study the significance of individual regression
coefficients. There is one very important point to bear in mind:

❑ A t-test on an individual coefficient is a test of its significance in the

presence of all other explanatory variables.

❑ Since multiple regression makes use of the interdependence of the

explanatory variables to model 𝑌, it is improper to treat individual
coefficients as if they could stand alone (except in the very special
case where all explanatory variables are uncorrelated with each other.

❑ Hence, the confidence interval for each coefficient is computed in the

presence of the other explanatory variables in the model.

Significance of Individual Coefficients 19

INDE3151: Forecasting Methods ISIK University Dr. Nuri Volkan Kayaçetin

❑ For each regression coefficient 𝑏𝑗 we can determine a standard error
(a measure of the stability of the coefficient), calculate its confidence
interval, and assess its significance in the regression equation.

❑ If a coefficient has no real effect in the regression model, then (given

the normality and constancy assumptions), it is known that 𝑡, defined
below, has a t-distribution with 𝑛 − 𝑘 − 1 degrees of freedom.

❑ The t-test is thus a test of whether the explanatory variable in question

is helping significantly in the prediction of 𝑌 in the presence of the
other explanatory variables.

Significance of Individual Coefficients 20

INDE3151: Forecasting Methods ISIK University Dr. Nuri Volkan Kayaçetin

❑ To calculate confidence intervals for the individual coefficients in the
regression, the standard errors from the regression analysis are used.
The following equation is used to construct the confidence intervals:

❑ The multiplying factor, 𝑡 ∗ , can be obtained from the student’s t table

and depends on the number of observations used (𝑛), the number of
explanatory variables (𝑘), and the level of confidence required (𝛼).
The degrees of freedom is 𝑛 − 𝑘 − 1, the same as that for the t-test.

❑ In a multiple regression setting, the standard errors of the coefficient

estimates are computed using a statistical package and not by using
mathematical expressions as we did in a simple regression setting.

Confidence Intervals of Coefficient Estimates 21

INDE3151: Forecasting Methods ISIK University Dr. Nuri Volkan Kayaçetin

❑ There are two additional aspects to consider in dealing with tests of
individual coefficients. First, the stability of the regression coefficients
depends upon the intercorrelation among the explanatory variables.

❑ Given two explanatory variables, the higher the correlation between

them the more unstable will be their coefficient estimates. In the case
of more than two 𝑋 variables the situation is similar but more subtle:
it is possible to have unstable coefficients without large correlations.

❑ The second aspect to consider is the estimated correlations among the

regression coefficients themselves. In the practical regression model,
coefficients 𝑏0 through 𝑏𝑘 are all random variables: they fluctuate
from sample to sample and have a joint probability distribution.

Multiple Regression: Additional Aspects 22

INDE3151: Forecasting Methods ISIK University Dr. Nuri Volkan Kayaçetin

❑ It is technically possible to determine the correlations among the
estimated regression coefficients. Many statistical packages do not
readily provide this information, but it can be very useful.

❑ For instance, the slope and the intercept estimates in a simple linear
regression are always going to be correlated because a least squares
regression line goes through the means of both 𝑌 and 𝑋.

❑ In multiple regression, the situation is more complicated.

❑ If two coefficients are found to be significantly correlated (positive or

negative), then the investigator should be warned that individual t-
tests on these coefficients should not be considered in isolation since
the two coefficients are dependent on each other.

Multiple Regression: Additional Aspects 23

INDE3151: Forecasting Methods ISIK University Dr. Nuri Volkan Kayaçetin

❑ In 1993, there was considerable concern about mutual savings banks:
deposits were getting smaller, and withdrawals were getting bigger.

❑ Thus, it was of interest to develop a short-term forecasting model to

forecast changes in end-of-month balance over the next few months.

❑ To illustrate the multiple regression model in forecasting, data from a

mutual savings bank in a large metropolitan area is examined.

❑ Of interest to the bank was the change in the end-of-month balances.

It was hypothesized that the yields in corporate and government bond
markets had influence on the end-of-the-month balance.

❑ The end-of-the-month balance is measured monthly over a five-year

sample period and is hence comprised of sixty monthly observations.

Example: Mutual Savings Bank Deposits 24

INDE3151: Forecasting Methods ISIK University Dr. Nuri Volkan Kayaçetin

❑ The end-of-the-month balances are plotted in the top right figure and
the AAA bond rates and the rates on 3 to 4-year U.S. bonds below.

Example: Mutual Savings Bank Deposits 25

INDE3151: Forecasting Methods ISIK University Dr. Nuri Volkan Kayaçetin

❑ The bank needs to forecast the direction and the magnitude of the
changes in its end-of-month balances. Thus, it is more appropriate to
first difference the balances and focus on monthly changes.

❑ From the figure above, it’s clear that the end-of-month balances have
been volatile in the last two years. The challenge to the forecaster is
to predict at least part of these changes to help the bank plan better.

Example: Mutual Savings Bank Deposits 26

INDE3151: Forecasting Methods ISIK University Dr. Nuri Volkan Kayaçetin

❑ In preparation for the regression analyses ahead, D(EOM) designates
the monthly first difference of end-of-month balances. The model
will use three explanatory variables: X1, X2, and X3.

❑ X1 and X2 are the yields on AAA rated corporate and 3 to 4-year

government bonds at the beginning of the month. X3 is the change in
the government bond yield within the month of observation.

Example: Mutual Savings Bank Deposits 27

INDE3151: Forecasting Methods ISIK University Dr. Nuri Volkan Kayaçetin

Example: Mutual Savings Bank Deposits 28

INDE3151: Forecasting Methods ISIK University Dr. Nuri Volkan Kayaçetin

❑ The bank could forecast 𝐷𝐸𝑂𝑀 with X1 alone, or using a combination
of X1, X2, and X3. The most straightforward way these variables can
be linked to the forecast variable is through a linear model of form:

𝐷𝐸𝑂𝑀 = 𝑏0 + 𝑏1 𝐴𝐴𝐴 + 𝑏2 𝐺𝑂𝑉 + 𝑏3 𝐷𝐺𝑂𝑉 + 𝑒𝑖

❑ Such a model would express the expected change in end-of-month

balances as a linear function of the lagged corporate and government
bond yields and the same month change in government bond yield.

❑ Plotting different combinations of variables against one another, with

each panel containing a scatterplot of one variable against the other
three, is a great way to visualize the bivariate associations between
the variables included in the forecasting procedure.

Example: Mutual Savings Bank Deposits 29

INDE3151: Forecasting Methods ISIK University Dr. Nuri Volkan Kayaçetin

Example: Mutual Savings Bank Deposits 30

INDE3151: Forecasting Methods ISIK University Dr. Nuri Volkan Kayaçetin

❑ For the bank data, we will use the first 53 rows for model estimation
and reserve the last seven rows for testing its out-of-sample accuracy.

❑ If the model has valuable predictive content, it should perform well

in forecasting the end-of-month balances in the last seven months.

❑ Using the data at hand, the model can be solved using least squares
method to yield the following coefficients:

෡𝐸𝑂𝑀 = −4.34 + 3.37 𝐴𝐴𝐴 − 2.83 𝐺𝑂𝑉 − 1.96𝐷𝐺𝑂𝑉

𝐷

❑ ෡𝐸𝑂𝑀 indicates that this is a forecast and not the actual observation.
𝐷
The difference between the observed and the estimated values, called
the regression error, tells us something about the “fit” of the model.

Example: Mutual Savings Bank Deposits 31

INDE3151: Forecasting Methods ISIK University Dr. Nuri Volkan Kayaçetin

❑ The correlation between 𝑌 and 𝑌෠ can be computed as
෠
𝑐𝑜𝑣 𝑌,𝑌
𝑅 = 𝑟𝑌𝑌෠ = ෠
= 0.749
𝑣𝑎𝑟(𝑌)

❑ The coefficient of determination, or the regression R2 then equals

𝑅2 = 0.7492 = 0.561

❑ Once could also divide the sum of squared deviations of the 𝑌෠ values
from the actual observations by the sum of squared deviations of the
actual observations from the mean value of 𝑌.

Example: Mutual Savings Bank Deposits 32

INDE3151: Forecasting Methods ISIK University Dr. Nuri Volkan Kayaçetin

❑ For the bank problem, the ANOVA table would be as follows:

❑ The F-statistic could also have been computed as:

𝑅2 /𝑘 0.561/3
𝐹= = = 20.85
(1 − 𝑅2 )/(𝑛 − 𝑘 − 1) (1 − 0.561)/(53 − 3 − 1)

❑ Since the p-value that corresponds to the computed F-statistic is very

small (less than 0.0005), the three explanatory variables are jointly
highly significant in predicting changes in end-of-month balances.

Example: Mutual Savings Bank Deposits 33

INDE3151: Forecasting Methods ISIK University Dr. Nuri Volkan Kayaçetin

❑ For the bank data and the linear regression of 𝐷𝐸𝑂𝑀 on 𝐴𝐴𝐴, 𝐺𝑂𝑉
and 𝐷𝐺𝑂𝑉 , the full output from a regression program includes:

❑ Note that for each estimate, there is a standard error, a t statistic, and
a p-value that can be used to assess the estimate’s significance.

❑ The p-value for each of 𝐴𝐴𝐴, 𝐺𝑂𝑉 and 𝐷𝐺𝑂𝑉 is below conventional
𝛼 levels of 0.01 or 0.05, indicating that each explanatory variable is
highly significant in presence of the other two explanatory variables.

Example: Mutual Savings Bank Deposits 34

INDE3151: Forecasting Methods ISIK University Dr. Nuri Volkan Kayaçetin

❑ We can construct confidence intervals using the data obtained:

❑ The residual degrees of freedom, 53 –3 –1 = 49, is the relevant row in

the student’s t table and the relevant column is selected based on the
desired confidence. A 95% confidence interval requires tcrit = 2.01.

Example: Mutual Savings Bank Deposits 35

INDE3151: Forecasting Methods ISIK University Dr. Nuri Volkan Kayaçetin

 None of the explanatory
variables has particularly
high correlation with ∆EOM

The model will not be able

to explain a lot of the time
series variation in ∆EOM

Explanatory variables do
not correlate very highly.
Strongest for X1 and X2.

The regression coefficients

themselves are interrelated,
especially X1 and X2.

It is necessary to interpret
the effect of X1 and X2 on
∆EOM of jointly

Example: Mutual Savings Bank Deposits 36

INDE3151: Forecasting Methods ISIK University Dr. Nuri Volkan Kayaçetin

Example: Mutual Savings Bank Deposits 37

INDE3151: Forecasting Methods ISIK University Dr. Nuri Volkan Kayaçetin

Example: Mutual Savings Bank Deposits 38

INDE3151: Forecasting Methods ISIK University Dr. Nuri Volkan Kayaçetin

❑ How to develop the best regression model to forecast a variable?
Step 1: Determine Potential Predictor Variables
❑ Include as many predictors as possible, but costs to obtain/monitor
information on large number of explanatory variables.
Step 2: Screen Out Inappropriate Predictors
❑ Remove the predictors variables which are (a) non-fundamental to
the analysis or redundant in the presence of other predictors and
(b) difficult to measure or subject to large measurement errors.
Step 3: Shorten the List of Predictors
❑ A parsimonious model with few intuitive predictors is a lot more
tractable and clearer compared to a large kitchen-sink type model.

Variable Selection 39

INDE3151: Forecasting Methods ISIK University Dr. Nuri Volkan Kayaçetin

❑ The analyst may start with an equation containing no explanatory
variables and explore each and every possible combination of those
variables in order to select the best of predictors.

❑ Since each explanatory variable will either be or not be in the model,

there would be 2k – 1 possible subsets for k explanatory variables.
This figure equals 3 in the case of two predictors, 1023 in case of ten
potential predictors, and 1,048,576 with twenty potential predictors.

❑ Thus, even with good statistical software at your disposal, it will

become exponentially more time and resource consuming to estimate
and compare all possible regression models through brute force as the
number of potential predictor variables increases.

Variable Selection 40

INDE3151: Forecasting Methods ISIK University Dr. Nuri Volkan Kayaçetin

❑ Nonetheless, trying all possible regression models via brute force is a
possible approach when the number of explanatory variables is not
large, with the goodness of fit being the main criterion of choice.

❑ The models are first divided into subsets according to the number of
parameters to be estimated. In each of these subsets, the model with
the highest R2 would be considered the best and would be recorded
as a possible competitor for crown for the best regression equation.

❑ The ultimate choice would depend on the implied tradeoff between

goodness of fit and simplicity. If the incremental R2 provided by the
addition of a variable to a simpler model is not large enough (what is
enough?), the simpler model should be preferred.

Variable Selection 41

INDE3151: Forecasting Methods ISIK University Dr. Nuri Volkan Kayaçetin

❑ All possible combinations of explanatory variables for a hypothetical
forecasting problem are presented below:

❑ These combinations are divided into six subsets based on the number
of parameters in the equation as shown.

Example: Variable Selection 42

INDE3151: Forecasting Methods ISIK University Dr. Nuri Volkan Kayaçetin

❑ The model with the greatest coefficient of determination is selected
from each subset. These selections are shown below:
Best Regression Equations
Parameters Variables Error DF R2
1 None 29 0000
2 X2 28 .6370
3 X1,X2 27 .8948
4 X1,X2,X5 26 .8953
5 X1,X2,X3,X5 25 .8955
6 X1,X2,X3,X4,X5 24 .8955

❑ Adding more variables after X1 and X2 is not worthwhile since the

resulting increase in the model’s goodness of fit is marginal. Thus,
the final equation will be comprised of these two variables.

Example 1: Variable Selection 43

INDE3151: Forecasting Methods ISIK University Dr. Nuri Volkan Kayaçetin

❑ Stepwise regression is a method which can be used to help sort out
the relevant explanatory variables from a set of candidate explanatory
variables when the number of explanatory variables is too large to
allow all possible regression models to be computed.

❑ The main kinds of stepwise regression in use are:

• Stepwise forward regression
• Stepwise backward regression,
• Stepwise forward-with-a-backward-look regression

❑ The stepwise forward or backward methods are not guaranteed to

produce the optimal combination of explanatory variables. The third
method, which is also imperfect, is superior to either one.

Stepwise Regression 44

INDE3151: Forecasting Methods ISIK University Dr. Nuri Volkan Kayaçetin

❑ The stepwise regression adds one independent variable at a time. The
basic algorithm of the method is as follows:
1) All possible simple regressions are considered. The predictor that
explains the largest proportion of variation in the forecast variable of
interest is the first variable to enter the regression equation.
2) The next predictor to enter the regression equation is the one that
makes the largest significant contribution to the regression sum of
squares, the significance determined by an F test of nested models. F to Enter
3) Once an additional variable has been included in the equation, the
individual contributions to the regression sum of squares of the other
variables already in the equation are checked for significance. F to Exit

4) Repeat 2 and 3 until all possible additions are insignificant.

Stepwise Regression 45

INDE3151: Forecasting Methods ISIK University Dr. Nuri Volkan Kayaçetin

❑ The user of a stepwise regression supplies the values that decide
when a variable is allowed to enter and when a variable is removed.

❑ Since the F statistics used in a stepwise regression is such that F = t2,

F = 4 (corresponding to |t|=2) is a common choice for the critical F
statistic for both F-to-Enter and the F-to-Exit tests.
𝑆𝑆𝐸𝑓 − 𝑆𝑆𝐸𝑟 /1
𝐹𝑛𝑒𝑠𝑡𝑒𝑑 =
𝑆𝑆𝐸𝑓 /(𝑛 − 𝑘 − 1)
❑ Stepwise regression technique is extremely easy to use and misuse.
• If a large number of explanatory variables are analyzed, the final model
may contain predictors not linearly related to the forecast variable. In the
initial selection of predictors, higher order terms (interactions, nonlinear
terms, curvilinear terms,) of importance may be omitted.

Stepwise Regression 46

INDE3151: Forecasting Methods ISIK University Dr. Nuri Volkan Kayaçetin

❑ Based on (i) hunches of experts and other knowledgeable people, (ii)
availability of data, and (iii) practical time and cost constraints, it was
decided that end-of-month balances in a mutual savings bank were
conceivably related to the following 19 economic variables:

Example: Stepwise Regression 47

INDE3151: Forecasting Methods ISIK University Dr. Nuri Volkan Kayaçetin

Example 2: Stepwise Regression 48

INDE3151: Forecasting Methods ISIK University Dr. Nuri Volkan Kayaçetin