College of Science and Engineering
James Cook University
MA2405:Advanced Statistical Modelling
Variation and model diagnostic in linear model
Dr Carla Ewels
[Link]@[Link]
Aug, 2025
Introduction
1
▶ Last lecture
▶ Structure of linear model
▶ Parameter estimation - the least square method
▶ Variance of parameters, predictions
▶ This lecture
▶ How good is the model?
▶ Source of variations
▶ Model diagnostics
Carla Ewels |
Variation
2
Carla Ewels |
Variation
3
In the absence of any predictor, best prediction of observation is the
overall mean,
yi = ȳ
This is a null model with, βˆ0 = ȳ
Total variation that is not described by the mean is
n
X
(yi − ȳ)2
i
This is called total sum of squares (TSS). Decompose TSS
X X X
(yi − ȳ)2 = (ŷi − ȳ)2 + (yi − ŷi )2
i i i
| {z } | {z } | {z }
TSS SSR SSE
SSR: sum of square due to the regression model. Variation in y
explained by the regression model
SSE: sum of square due to error
Carla Ewels |
Effect on SSR and SSE of adding new predictor
4
In the least square, when adding more predictors to the model,
▶ SSE can only decrease (Loss function)
▶ Because TSS remains unchanged, so SSR can only increases
▶ We can break down SSR by each predictor.
▶ Say we have two predictors, x1 and x2
SSR(x1 , x2 ) = SSR(x1 ) + SSR(x2 |x1 )
X X X
(ŷi12 − ȳ)2 = (ŷi1 − ȳ)2 + (ŷi12 − ŷi1 )2
i i i
| {z } | {z }
SSR(x1 ) SSR(x2 |x1 )
where ŷi1 denote the fitted value when x1 is the only predictor.
And ŷi12 is the fitted when both x1 and x2 are both included.
Carla Ewels |
Sequential SS
5
▶ Sequential sum of squares- Type I sum of squares
▶ Let consider p variable, x1 , . . . , xp .
▶ if we enter these variables into the model one at the time
SSR(x2 |x1 ), SSR(x3 |x1 , x2 ), . . . , SSR(xp |x1 , x2 , . . . , xp−1 )
SSR = SSR(x2 |x1 ) + SSR(x3 |x1 , x2 ) + . . . + SSR(xp |x1 , x2 , . . . , xp−1 )
▶ Contribution of each variable depends on the variables already in
the model
▶ Therefore, when predictors are highly correlated (say xp and
xp−1 ), SSR(xp ) can be much larger than SSR(xp |x1 , . . . , xp−1 ).
Carla Ewels |
Partial SS
6
▶ Partial SS - Type III sum of squares
SSR(x1 |x2 , . . . , xp ), SSR(x2 |x1 , x3 , . . . , xp ), . . . , SSR(xp |x1 , . . . , xp−1 )
▶ Each of these represents the additional variability explained by
the predictor when all other predictors are included.
▶ When predictors are uncorrelated,
Sequential SS= Partial SS
We will discuss how sequential and partial SS impacts the results of
hypothesis testing later
Carla Ewels |
Predictive power
7
▶ We can use TSS, SSR and SSE to investigate the predictive
power of a model
▶ When a model fit the data well, we expect SSR(model) to be
large, therefore
SSR
R2 =
T SS
which measures proportional or variation explained by the model also
known as coefficient of determination
SSR T SS − SSE
=
T SS TPSS
(yˆi − yi )2
=1− P
(yi − ȳ)2
Carla Ewels |
R2 and correlation
8
▶ Another way to measure predictive power is by correlation of yi
and ŷi ,
SSR √ 2
r
corr(yy , ŷŷ) = = R
T SS
Multiple correlation
Carla Ewels |
Predictive power
9
▶ Problem with R2 :
(yˆi − yi )2
P
2
R =1− P
(yi − ȳ)2
as more predictors added to the model, SSR increases
monotonically. Therefore R2 and R also increase.
▶ when n is small and there are several predictors, R2 tends to
overestimate the actual power
▶ Adjusted R2
SSE
n−p n−1
adjusted R2 = 1 − T SS
=1− (1 − R2 )
n−1
n−p
Carla Ewels |
Model diagnostics
10
▶ Now we have the estimates of model parameters and variance
associated with model parameters, coefficient of of determination
(R2 ) and adjusted R2
Carla Ewels |
Model diagnostics
10
▶ Now we have the estimates of model parameters and variance
associated with model parameters, coefficient of of determination
(R2 ) and adjusted R2
▶ Next step is to check if the model is appropriate
Carla Ewels |
Model diagnostics
10
▶ Now we have the estimates of model parameters and variance
associated with model parameters, coefficient of of determination
(R2 ) and adjusted R2
▶ Next step is to check if the model is appropriate
▶ Examining
▶ Residuals
▶ Leverage
▶ Influential points
Carla Ewels |
Model diagnostics- residuals
11
For simple linear model, key assumption
ε ∼ N (0, σ 2 I)
Where I is an identity matrix. Gaussian noise model
Carla Ewels |
Model diagnostics- residuals
11
For simple linear model, key assumption
ε ∼ N (0, σ 2 I)
Where I is an identity matrix. Gaussian noise model
▶ Checking the normality of residuals
Carla Ewels |
Model diagnostics- residuals
11
For simple linear model, key assumption
ε ∼ N (0, σ 2 I)
Where I is an identity matrix. Gaussian noise model
▶ Checking the normality of residuals
▶ Histogram or Q-Q plot
Histogram of y
Normal Q−Q
40
28
3
7
Standardized residuals
30
Frequency
1
20
−1 0
10
−3
0
−2 −1 0 1 2 −10 −5 0 5 10
Theoretical Quantiles y
lm(z1$water ~ z1$weight)
Carla Ewels |
Model diagnostics- residuals
11
For simple linear model, key assumption
ε ∼ N (0, σ 2 I)
Where I is an identity matrix. Gaussian noise model
▶ Checking the normality of residuals
▶ Histogram or Q-Q plot
Histogram of y
Normal Q−Q
40
28
3
7
Standardized residuals
30
Frequency
1
20
−1 0
10
−3
0
−2 −1 0 1 2 −10 −5 0 5 10
Theoretical Quantiles y
lm(z1$water ~ z1$weight)
▶ Shapiro-Wilk test for normality (in conjunction with Q-Q plot)
Carla Ewels |
Model diagnostics- residuals
11
For simple linear model, key assumption
ε ∼ N (0, σ 2 I)
Where I is an identity matrix. Gaussian noise model
▶ Checking the normality of residuals
▶ Histogram or Q-Q plot
Histogram of y
Normal Q−Q
40
28
3
7
Standardized residuals
30
Frequency
1
20
−1 0
10
−3
0
−2 −1 0 1 2 −10 −5 0 5 10
Theoretical Quantiles y
lm(z1$water ~ z1$weight)
▶ Shapiro-Wilk test for normality (in conjunction with Q-Q plot)
▶ When the residuals are NOT normally distributed-
transformation or robust regression instead the least square
method
Carla Ewels |
Model diagnostics- Residuals
12
ε ∼ N (0, σ 2 I)
Variance in residuals is constant for all ŷ
Figure c, example of Heteroscedasticity, variances varies with fitted
value
Possible solution- transform y
Carla Ewels |
Model diagnostics- Residuals
13
ε ∼ N (0, σ 2 I)
Variance in residuals is constant for all ŷ
Figure b, variance is pretty constant, but not centred around zero
Example of relationship between predictor and response variable is
non-linear
Carla Ewels |
Model diagnostics- Standardised residuals
14
▶ Standardised residual
y − ŷi
√i
s 1 − hii
where hii is the diagonal element of the hat matrix,
1 (xi − x̄)2
hii = + Pn 2
n k=1 (xk − x̄)
▶ If the assumption of the linear model hold, all standardised
residuals should be between -3 and 3.
Carla Ewels |
Model diagnostics- Leverage
15
▶ Hat matrix
ŷy = H y
X T β̂
var(X X T X )−1X T σ 2
β ) = X (X
| {z }
hat matrix
▶ Hat matrix gives us some insight into the dependency between ŷ
and y, as well as the variance of predicted values and the true
variance.
▶ 0 ≤ hii ≤ 1
▶ If hii is close to 1, variance of ŷi is similar to the variance of y
▶ If hii is close to 0, variance of ŷi is smaller than the variance of y
Carla Ewels |
Model diagnostics- Leverage
16
What is leverage
▶ Leverage determines the precision on which yˆi estimates
yi
▶ The diagonal elements of the hat matrix, hii .
1 (xi − x̄)2
hii = +P 2
n k (xk − x̄)
▶ 0 ≤ hii ≤ 1
▶ Leverage is the distance between xi and x̄
▶ Leverage increases as xi is farther from x̄.
Carla Ewels |
Model diagnostics
17
6
4
2
y
0
−4 −2
−2 −1 0 1 2 3
▶ With p predictors, the leverage has the mean of p/n. When the
relative leverage of an observation is greater than 3p/n (or 2p/n),
it may influence the fitting of the model
Carla Ewels |
Model diagnostics- Influential points
18
▶ Observations with relative small leverage is not influential
▶ Observations with large leverage is not necessary influential
▶ A point is influential if it is a "regression outlier"
Carla Ewels |
Model diagnostics- Influential points
19
How do we determine if a point is influential (regression outlier)?
Carla Ewels |
Model diagnostics- Influential points
19
How do we determine if a point is influential (regression outlier)?
▶ Studentized residual test
Carla Ewels |
Model diagnostics- Influential points
19
How do we determine if a point is influential (regression outlier)?
▶ Studentized residual test
▶ Cook’s distance
Carla Ewels |
Model diagnostics- Influential points
19
How do we determine if a point is influential (regression outlier)?
▶ Studentized residual test
▶ Cook’s distance
▶ Difference in fits(DFFIT)
Carla Ewels |
Model diagnostics- Influential points
19
How do we determine if a point is influential (regression outlier)?
▶ Studentized residual test
▶ Cook’s distance
▶ Difference in fits(DFFIT)
▶ DFBETA
Carla Ewels |
Model diagnostics- Influential points
20
Studentized residuals (jackknife residuals) to judge of potential
outlier
▶ Slightly different from standardised residuals
▶ To check if a point i is an influential point,
1. fit a model to the data without point i
2. compare the model prediction against the observed value (yi )
using t-statistics
yi − ŷ(i)
ti = √
s(i) 1 − hii
n − p − 1 1/2
= ri ( )
n − p − ri2
where y(i) is the prediction of the model exclude point i, hii is the
leverage of point i, s(i) is the variance estimate with i excluded and
ri is the standardised residuals
ti ∼ tn−p−1
Require correction of multiple testing, e.g. Bonferroni
Carla Ewels |
Model diagnostics- Influential points
21
Cook’s Distance
▶ Similar concept as the Studentized residuals, comparing model
coefficients with and without the point (i)
▶ Cook’s distance can be expressed in terms of the leverage, hii ,
and standardised residual, ri . (p is number of parameters)
(β̂ β )T (X
β (i) − β̂ X T X )(β̂
β (i) − β̂
β)
Di =
ps2
hii
= ri2 [ ]
p(1 − hii )
where β (i) is the least square estimation of coefficient without ith
observations
▶ When Di is large, it suggests that both ri and hii are large
▶ When Di ∼ F(p,n−p),α , and point i is flagged as a influential when
Di exceeds the 50th percentile of the F-distribution
Carla Ewels |
Model diagnostics- Influential points
22
Difference in fits (DFFITS)
▶ DFFITS describes the change in fitted value, yˆi , due to delete
observation i.
▶ The standard DFFIT
p
DFFITi = ri∗ hii /(1 − hii )
where ri∗ is the studentized residual without point i
p p
▶ when |DFFITi | > 3 p/(n − p) (or 2 p/(n − p)), then i is
declared influential
Carla Ewels |
Model diagnostics- Influential points
23
DFBETAS
▶ DFBETAS is the changes in jth coefficient, β̂j , with the removal
of ith observation.
β̂j − β̂j(i)
DFBETAi =
se(β̂j(i) )
√
▶ when |DFBETAi | > 1 (or greater than 2/ n), i is declared
influential
Carla Ewels |
Model diagnostics- Influential points
24
▶ All these methods are guideline only
▶ Measure different aspects of changes when excluding ith
observation is excluded
▶ Hat matrix and standardised residuals
▶ Majority rules
Carla Ewels |
Partial Correlation
25
Partial Correlation
Correlation between two variables (e.g. y and x1 ) after adjust-
ing for other variables (e.g. x2 , . . . , xp )
Carla Ewels |
Why is partial correlation important?
26
Housing Example- example in R
Carla Ewels |
Why is partial correlation important?
27
To avoid Simpson’s Paradox
Simpson’s Paradox
The effect of a predictor changes after adjusting for other pre-
dictors. For example, the marginal effect of a variable is positive
while the partial correlation is negative.
Carla Ewels |
Simpson’s Paradox
28
Portrayal of Simpson’s paradox
Carla Ewels |
Partial Correlation
29
How to find partial correlation between y and x1 while there are also
x2 and x3 in the model?
1. find the residuals for predicting y using x2 and x3
2. find the residuals for predicting x1 using x2 and x3
3. find the ordinary correlation between step 1 and 2
Carla Ewels |
Squared partial correlation
30
▶ The squared partial correlation between y and xj shows the
proportion of variability unexplained when xj is not included over
the variability unexplained with xj is included.
▶ say R02 is the proportion of reduction in error without a variable
and R12 is the proportion of reduction in error with the variable
included, the squared partial correlation is
(R12 − R02 )
(1 − R02 )
Carla Ewels |