Multiple R Notes
Multiple R Notes
Econometrics
Contents
4 Goodness of Fit 3
4.1 Decomposition of Variation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
4.2 Coefficient of Multiple Determination, R2 . . . . . . . . . . . . . . . . . . . . . . . . 3
4.3 Adjusted R2 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
9 Multicollinearity 5
10 Example 1 6
12 Example 2 8
Simple linear regression explains Y using a single explanatory variable X. In reality, most outcomes
are influenced by several factors simultaneously. Multiple linear regression extends the model
to two or more explanatory variables:
where:
Y = Xβ + ε
2
Multiple Linear Regression Lecture Notes
β̂ = (X′ X)−1 X′ Y
For the common exam case of two explanatory variables (k = 2), it is far easier to work with
deviations from the mean rather than full matrix inversion. Let x1 = X1 − X̄1 , x2 = X2 − X̄2 ,
y = Y − Ȳ . The slope coefficients solve:
X X X
b1 x21 + b2x1 x2 = x1 y
X X X
b1 x1 x2 + b 2 x22 = x2 y
Solve simultaneously (e.g. by Cramer’s Rule):
b0 = Ȳ − b1 X̄1 − b2 X̄2
This is exactly what the matrix formula β̂ = (X′ X)−1 X′ Y produces for k = 2 — it is simply
written out algebraically instead of via matrix inversion, which is much faster by hand.
4 Goodness of Fit
SSR SSE
R2 = =1− 0 ≤ R2 ≤ 1
SST SST
R2 is the proportion of total variation in Y jointly explained by all the explanatory variables.
4.3 Adjusted R2
Because R2 mechanically rises whenever any variable is added — useful or not — the adjusted
R2 penalises for the number of regressors:
2 2 n−1
R̄ = 1 − (1 − R )
n−k−1
3
Multiple Linear Regression Lecture Notes
R̄2 is the correct criterion for comparing models with different numbers of explanatory variables; a
variable is only ”worth adding” if it raises R̄2 .
The F-test asks: do the explanatory variables, taken together, explain a significant proportion of the
variation in Y ?
H0 : β1 = β2 = · · · = βk = 0 H1 : at least one βj ̸= 0
SSR/k R2 /k
F = = ∼ F(k, n−k−1)
SSE/(n − k − 1) (1 − R2 )/(n − k − 1)
Decision rule: Reject H0 if Fcalc > Fcritical (from F-tables at the chosen significance level,
with k and n − k − 1 degrees of freedom), or if the associated p-value < α.
SSE
σ̂ 2 = M SE =
n−k−1
The divisor is n − k − 1 because
√ k + 1 parameters (β0 , β1 , . . . , βk ) have been estimated, using up
k + 1 degrees of freedom. s = σ̂ 2 is the standard error of the regression.
Each coefficient can be tested separately for significance, holding the others fixed:
H0 : βj = 0 H1 : βj ̸= 0
bj − 0
tcalc = ∼ t(n−k−1)
se(bj )
Reject H0 if |tcalc | > tcritical at (n − k − 1) degrees of freedom, meaning Xj is a statistically
significant predictor of Y given the other regressors in the model.
4
Multiple Linear Regression Lecture Notes
9 Multicollinearity
Multicollinearity occurs when two or more explanatory variables are highly correlated with each
other.
• Coefficient estimates remain unbiased, but their standard errors become large (im-
precise estimates).
• t-statistics shrink, so individually significant variables may appear insignificant.
• Coefficients become very sensitive to small changes in the data or model specification.
• R2 can remain high even though individual coefficients are unreliable — this contradic-
tion (high R2 , insignificant t’s) is a classic diagnostic sign.
Detecting multicollinearity
where Rj2 is the R2 from regressing Xj on all other explanatory variables. A common rule
of thumb: V IFj > 10 indicates serious multicollinearity.
Remedies: drop a redundant variable, collect more data, combine correlated variables into an
index, or use techniques such as ridge regression.
5
Multiple Linear Regression Lecture Notes
10 Example 1
X1 5 6 8 9 10 12
X2 7 4 6 3 8 5
Y 21 19 25 22 31 27
Required: (i) the OLS regression equation; (ii) R2 and adjusted R2 ; (iii) the variance of the
error term; (iv) test the overall significance of the model at 5%; (v) test the significance of each
slope coefficient at 5%.
Step 1: Means.
n = 6, X̄1 = 8.333, X̄2 = 5.5, Ȳ = 24.167
Step 2: Deviation sums (computed from a deviations table, as in simple regression, but now
for three pairs of variables):
X X X
x21 = 33.333 x22 = 17.5 x1 x2 = −1.0
X X X
x1 y = 42.667 x2 y = 24.5 y 2 = 96.833
Step 3: Solve the normal equations for b1 , b2 .
Determinant = (33.333)(17.5) − (−1.0)2 = 583.33 − 1 = 582.33
(42.667)(17.5) − (24.5)(−1.0) 746.67 + 24.5
b1 = = = 1.3243
582.33 582.33
(33.333)(24.5) − (−1.0)(42.667) 816.67 + 42.67
b2 = = = 1.4757
582.33 582.33
b0 = Ȳ − b1 X̄1 − b2 X̄2 = 24.167 − (1.3243)(8.333) − (1.4757)(5.5) = 5.015
6
Multiple Linear Regression Lecture Notes
we obtain:
6 50 33 145
X′ X = 50 450 274 X′ Y = 1251
33 274 199 822
Inverting X′ X and multiplying by X′ Y gives:
5.015
β̂ = (X′ X)−1 X′ Y = 1.3243
1.4757
This exactly matches b0 , b1 , b2 obtained from the deviation-form normal equations in Example 1
above — the two methods are algebraically identical. In an exam, use whichever method you
are more comfortable computing quickly and accurately; for k = 2, the deviation-form method
is almost always faster by hand.
7
Multiple Linear Regression Lecture Notes
12 Example 2
X1 2 3 4 5 6 7 8
X2 70 60 80 65 90 72 85
Y 52 50 61 58 70 66 75
Required: (i) the OLS regression equation; (ii) R2 and adjusted R2 ; (iii) the variance of the
error term; (iv) test the overall significance of the model at 5%; (v) test the significance of each
slope coefficient at 5%.
8
Multiple Linear Regression Lecture Notes
X1 5 6 8 9 10 12
X2 3 4 4 5 6 7
Y 18 21 26 30 34 40
9
Multiple Linear Regression Lecture Notes
Quantity Formula
SST = (Yi − Ȳ )2
P
Total sum of squares
SSR = (Ŷi − Ȳ )2
P
Regression sum of squares
10
Multiple Linear Regression Lecture Notes
X1 2 3 3 4 5 6
X2 8 10 12 11 15 16
Y 30 38 40 44 55 62
Required:
(iv) Test the overall significance of the regression at the 5% level. (3 marks)
(v) Briefly explain one limitation of relying on R2 alone to judge model quality. (2 marks)
(iii) Test, at the 5% significance level, whether income and household size are each individually
significant determinants of consumption. (5 marks)
(iv) Compute SSE and the variance of the error term. (3 marks)
(a) Explain the difference between R2 and adjusted R2 in a multiple regression context, and
state why the adjusted measure is preferred when comparing models with different num-
bers of regressors. (3
marks)
(b) Define multicollinearity and state two of its consequences for OLS coefficient estimates.
(3 marks)
(c) Explain how the Variance Inflation Factor (VIF) is used to detect multicollinearity, in-
cluding the typical threshold value used as a rule of thumb. (2
marks)
(d) Explain why the divisor used to estimate the variance of the error term in a multiple
regression with k explanatory variables is (n − k − 1) rather than (n − 1) or n. (2 marks)
11
Multiple Linear Regression Lecture Notes
Use this table for the individual-coefficient t-test. Find the row matching your degrees of freedom,
df = n − k − 1, and the column matching your (two-tailed) significance level α.
Example use: for Worked Example 1 (n = 6, k = 2), df = 3 and α = 0.05 (two-tailed) ⇒ tcritical =
3.182, matching the value used in the worked solution.
15
Multiple Linear Regression Lecture Notes
Use this table for the overall F-test. Find the column matching df1 = k (numerator, number of
explanatory variables) and the row matching df2 = n − k − 1 (denominator).
df2 \df1 1 2 3 4 5 6 7 8
Example use: for Worked Example 1 (k = 2, df2 = 3), Fcritical = 9.55; for Worked Example 2
(k = 2, df2 = 4), Fcritical = 6.94 — both match the values used in the worked solutions.
16
Multiple Linear Regression Lecture Notes
df2 \df1 1 2 3 4 5 6 7 8
Reading tip: If your exam paper supplies its own statistical tables, always use those values
instead — minor differences can arise between table editions due to rounding. These tables are
provided as a self-contained reference matching the worked examples in these notes.
17