Chapter 6 : Multiple Linear Regression Models
Regression models that employ more than one independent variable are called multiple
regression models. Several variables are used to estimate the value of the dependent
variable.
The model
A multiple linear regression model that relates a 𝑌-variable to 𝑝 𝑋-variables is written as
𝑌𝑖 = 𝛽0 + 𝛽1 𝑋1𝑖 + 𝛽2 𝑋2𝑖 + ⋯ + 𝛽𝑝 𝑋𝑝𝑖 + 𝜇𝑖 , 𝑖 = 1,2, … , 𝑛
Assumptions:
• Linear in the parameters
• 𝐸(𝜇𝑖 ) = 0. Zero mean variance of error (residuals) terms or disturbances 𝜇𝑖 .
• 𝑉𝑎𝑟 (𝜇𝑖 ) = 𝜎 2 for all 𝑖. Homoscedasticity or constant variance of 𝜇𝑖 .
• 𝑐𝑜𝑣 (𝜇𝑖 , 𝜇𝑗 ) = 0, 𝑖 ≠ 𝑗 No autocorrelation, or serial correlation, between the error
terms.
• 𝜇𝑖 are normally distributed for all 𝑖.
• 𝜇𝑖 and 𝑋𝑗 are independent for all 𝑖 and 𝑗.
• No Exact collinearity between 𝑋 variables. No exact linear relationship between 𝑋1
and 𝑋2 . i.e. no multicollinearity.
• There is no specification bias. The model is correctly specified.
The model in matrix notation
𝒀 = 𝑿𝜷 + 𝝁
where
1 X11 … X1p
1 X2i … X2p
𝑌1 . 𝜷𝟎 𝝁𝟏
𝒀 = (𝑌.2 ), 𝑿 = . , 𝜷 = (𝜷.𝟏 ) , 𝝁 = (𝝁.𝟐 ).
.. .. ..
. 𝜷𝒑 𝜷𝒏
𝑌𝑛
(1 Xni . . . Xnp )
• ̂ = 𝑯𝒀 where 𝜷
̂ = 𝑿𝜷
Fitted model (values): 𝒀 ̂ = (𝑿′ 𝑿)−𝟏 𝑿′ 𝒀 and
𝑯 = 𝑿(𝑿′ 𝑿)−𝟏 𝑿′
• ̂ =𝒀−𝒀
The residuals are given by: 𝝁 ̂ = 𝒀 − 𝑿𝜷̂ , the difference between the
actual and predicted values of the dependent variable.
Analysis of Variance Table
Analysis of variance approach to testing whether there is a linear association between 𝑌 and
a subset/all of the predictors.
We test the hypotheses
𝐻0 : 𝛽1 = 𝛽2 = ⋯ = 𝛽𝑝 = 0
1
𝐻1 : 𝑎𝑡 𝑙𝑒𝑎𝑠𝑡 𝑜𝑛𝑒 𝑜𝑓 𝑡ℎ𝑒 𝛽𝑖 ≠ 0
It can be shown that:
𝑆𝑆𝑇 = 𝑆𝑆𝑅𝑒𝑔 + 𝑅𝑆𝑆
Total sample variability = Variability explained by the model + Unexplained (or error) variability
If 𝑌𝑖 = 𝛽0 + 𝛽1 𝑋1𝑖 + 𝛽2 𝑋2𝑖 + ⋯ + 𝛽𝑝 𝑋𝑝𝑖 + 𝜇𝑖 and 𝑎𝑡 𝑙𝑒𝑎𝑠𝑡 𝑜𝑛𝑒 𝑜𝑓 𝑡ℎ𝑒 𝛽𝑖 ≠ 0, then RSS should
be “small” and SSReg should be “close” to SST. We test the hypotheses using the test statistic
𝑆𝑆𝑅𝑒𝑔/𝑝
𝐹=
𝑅𝑆𝑆
(𝑛 − 𝑝 − 1)
Analysis of Variance Table
Source of Degrees of Sum of Mean Square (M F
Variation Freedom Squares
(SS)
Regression 𝑝 𝑆𝑆𝑅𝑒𝑔 𝑆𝑆𝑅𝑒𝑔/𝑝 𝑆𝑆𝑅𝑒𝑔/𝑝
𝑅𝑆𝑆 𝐹=
Residual 𝑛−𝑝−1 𝑅𝑆𝑆 𝑅𝑆𝑆
= 𝑠2 (𝑛 − 𝑝 − 1)
(𝑛 − 𝑝 − 1)
Total 𝑛−1 𝑆𝑆𝑇
• 𝐹 −test is used to first test for the existence of a linear association between 𝑌 and any
of the 𝑝 𝑋-variables. If 𝐹 −test is significant, for which of the 𝑝 𝑋-variables is there
evidence of a linear association with 𝑌?. This is also referred to as the global test:
testing whether the multiple regression model is valid.
• 𝑅 2, the coefficient of determination of the regression line is defined as the proportion
of the total sample variability in the 𝑌’𝑠 explained by the regression model.
𝐸𝑥𝑝𝑙𝑎𝑖𝑛𝑒𝑑 𝑣𝑎𝑟𝑖𝑎𝑡𝑖𝑜𝑛 𝑆𝑆𝑅𝑒𝑔 𝑅𝑆𝑆
𝑅2 = = =1−
𝑇𝑜𝑡𝑎𝑙 𝑉𝑎𝑟𝑖𝑎𝑡𝑖𝑜𝑛 𝑆𝑆𝑇 𝑆𝑆𝑇
• Adding irrelevant predictor variables to the regression equation often increases 𝑅 2. To
compensate for this, we use adjusted coefficient of determination.
𝑅𝑆𝑆
2 (𝑛 − 𝑝 − 1)
𝑅𝑎𝑑𝑗 =1−
𝑆𝑆𝑇
𝑛−1
Thus, when comparing models with different numbers of predictors we should use
2
𝑅𝑎𝑑𝑗 not 𝑅 2 .
Testing individual regression coefficients (significance testing of each variable)
We may want know whether a particular 𝑋-variable is making a useful contribution to the
model. That is, given the presence of the 𝑋-variable in the model, does a particular 𝑋-variable
help us predict or explain the 𝑌-variable.
As an example, to determine whether 𝑋1 is a useful predictor variable in the model, we could
test
𝐻0 : 𝛽1 = 0 (insignificant)
𝐻1 : 𝛽1 ≠ 0 (significant)
2
This is a 𝑡-test.
1𝑏 −0 𝑏
The test statistic is 𝑡 = 𝑠𝑒(𝑏 )
= 𝑠𝑒(𝑏1
1 1)
If we fail to reject the null hypothesis, then the variable should be dropped from the model.
After a variable is removed, a new regression model is constructed using the remaining
variables and a new t-test can be conducted for each of the remaining variables.
Checking the Validity of a model
When a valid model has been fit, a plot of standardized residuals against any predictor or any
linear combination of the predictors (such as fitted values) will have the following features:
• A random scatter of point around the horizontal axis, since the mean function of the
residuals is zero when a correct model has been fit.
• Constant variability as we look along the horizontal axis.
An implication of these features is that any pattern in a plot of standardized residuals is
indicative that an invalid model has been fit to the data. If the variability in the standardized
residuals increase or decrease (funnel shaped), the error variance is not equal.
• A normal Q-Q plot of the standardized residuals is used to check for the normality of
the errors. If the resulting plot produces points close to a straight line the data are
consistent that from a normal distribution. Any departures from linearity provides
evidence of non-normality.
Interpretation of the model parameters
• Each 𝛽 parameter represent the change in the mean response, 𝐸(𝑌), per uit increase
in associated predictor variable when all other predictors are held constant.
• For example, 𝛽1 represent the estimated change in the mean response, 𝐸(𝑌), per uit
increase in 𝑋1 variable when all other predictors are held constant.
• The intercept represents the estimated mean response, 𝐸(𝑌), when all the predictors
are all zero. (no practical meaning).
Example: Fuel consumption data
Consider the fuel consumption data and the model
𝑌𝑖 = 𝛽0 + 𝛽1 𝑋1𝑖 + 𝛽2 𝑋2𝑖 + 𝜀𝑖
which relates fuel consumption (𝑌𝑖 ) to average hourly temperature (𝑋1 ) and chill index (𝑋2 ).
Estimate the parameters of this model and comment.
Week, 𝑖 Average Hourly Chill index, 𝑋2𝑖 Fuel Consumption, 𝑌𝑖
Temperature, 𝑋1𝑖
1 𝑋11 = 28.0 𝑋21 = 18 𝑌1 = 12.4
2 𝑋12 = 28.0 𝑋22 = 14 𝑌2 = 11.7
3 𝑋13 = 32.5 𝑋23 = 24 𝑌3 = 12.4
4 𝑋14 = 39.0 𝑋24 = 22 𝑌4 = 10.8
5 𝑋15 = 45.9 𝑋25 = 8 𝑌5 = 9.4
6 𝑋16 = 57.8 𝑋26 = 16 𝑌6 = 9.5
7 𝑋17 = 58.1 𝑋27 = 1 𝑌7 = 8.0
8 𝑋18 = 62.5 𝑋28 = 0 𝑌8 = 7.5
3
Scatter plots of 𝒀 and 𝑿𝟏 and 𝑿𝟐
The Regression model says:
• 𝛽0 + 𝛽1 𝑋1𝑖 + 𝛽2 𝑋2𝑖 is the mean value of 𝑌 when the average hourly temperature is
𝑋1𝑖 and the chill index is 𝑋2𝑖 . For instance, 𝛽0 + 𝛽1 (44.8) + 𝛽2 𝑋2 (8) is the average
fuel consumption for all weeks having an average hourly temperature equal to 44.8
and a chill index equal to 8.
• 𝛽0 , 𝛽1 𝑎𝑛𝑑 𝛽2 are regression parameters relating the mean value of 𝑌 to 𝑋1𝑖 and 𝑋2𝑖
• 𝜀𝑖 is the error term that describes the effects on 𝑌𝑖 of all factors other than 𝑋1𝑖 and
𝑋2𝑖 .
Using SAS
data fuelc;
input x1 x2 y;
cards;
28 18 12.4
28 14 11.7
32.5 24 12.4
39 22 10.8
45.9 8 9.4
57.8 16 9.5
58.1 1 8
62.5 0 7.5
;
proc reg;
model y = x1 x2;
run;
4
proc autoreg ;
model Y=x1 x2;
ods graphics off;
ods listing;
run;
Analysis of Variance
Sum of Mean
Source DF Squares Square F Value Pr > F
Model 2 24.87502 12.43751 92.30 0.0001
Error 5 0.67373 0.13475
Corrected Total 7 25.54875
Root MSE 0.36708 R-Square 0.9736
Dependent Mean 10.21250 Adj R-Sq 0.9631
Coeff Var 3.59440
Parameter Estimates
Parameter Standard
Variable DF Estimate Error t Value Pr > |t|
Intercept 1 13.10874 0.85570 15.32 <.0001
x1 1 -0.09001 0.01408 -6.39 0.0014
x2 1 0.08249 0.02200 3.75 0.0133
The Multiple regression model is 𝑌̂ = 13.10874 − 0.09001 𝑋1 + 0.08249 𝑋2
• We estimate that the mean weekly fuel consumption decreases by 0.09001 when the
average hourly temperature increases by one degree and the chill index does not change.
• We estimate that the mean weekly fuel consumption increases by 0.08249 when there is a
one unit increase of chill index and average hourly temperature does not change.
Table: Predictions and residuals
Week 𝑋1𝑖 𝑋2𝑖 𝑌 𝑌̂ 𝜀 = 𝑌 − 𝑌̂
1 28.0 18 12.4
2 28.0 14 11.7
3 32.5 24 12.4
4 39.0 22 10.8
5 45.9 8 9.4
6 57.8 16 9.5
7 58.1 1 8.0
8 62.5 0 7.5
5
Autocorelation-correlation of successive residuals.
Successive residuals should be independent. This means there is no pattern to the residual,
the residuals are not highly correlated. When successive residuals are correlated we refer to
this as autocorrelation.
We check for first order autocorrelation using Durbin-Watson test.
proc autoreg ;
model Y=x1 x2 /dw=4 dwprob; run;
Heteroscedasticity- variance of residuals is nonconstant.
The variation around the regression equation should be the same regardless of the values of
the independent variables.
We check for heteroscedasticity using the White test.
proc reg;
model y = x1 x2/spec;
run
Multicollinearity-correlation among the independent variables
In practice, it nearly impossible to select a set of independent variables that are not correlated
to some degree. However, if an independent variable is highly correlated with other variables
in the models it should be removed. Multicollinearity makes it difficult to draw inferences about
the individual regression coefficients and their effects on the dependent variable.
Variance Inflation Factor (VIF) is a measure of the degree to which an independent variable
is correlated to the other independent variables in the regression in model. A VIF greater than
10 indicates (multicollinearity) that the independent variable should be removed from the
model
proc reg;
model y = x1 x2/ vif;
run;
_________________________________________________________________________