ST346 Chapter 1
Contents
Preface 3
Introduction 4
Where did we come from? Where are we going? . . . . . . . . . . . . . . . . . . . . . 4
Assumptions of the normal linear model . . . . . . . . . . . . . . . . . . . . . . . . . 5
Relaxing the assumptions for GLMs . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6
Why learn about GLMs? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6
Structure of the module . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6
1 Weighted linear regression 7
1.1 Relaxing the assumptions of the linear model . . . . . . . . . . . . . . . . . . . . 7
1.2 Weighted estimation - an example . . . . . . . . . . . . . . . . . . . . . . . . . . 7
1.3 Revision - Least squares estimation . . . . . . . . . . . . . . . . . . . . . . . . . 9
1.4 The Gauss-Markov theorem . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9
1.5 Weighted least squares . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9
1.6 Examples of weighted regression . . . . . . . . . . . . . . . . . . . . . . . . . . . 11
1.7 The hat matrix . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14
1.8 Weighted residuals . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15
2
Preface
This slides are a slight adaptation from the original slides developed by Prof Martyn Plummer
for the module.
If you find any typos, please inform the module leader.
These materials are solely for your own use and you must not distribute these in any format.
Do not upload these materials to the internet or any filesharing sites nor provide
them to any third party or forum.
All rights are reserved.
3
Introduction
Where did we come from? Where are we going?
• Prerequisites:
– ST231 Linear Statistical Modelling with R
• Leads to:
– ST332 Medical Statistics
– ST404 Applied Statistical Modelling
• Textbook:
– Generalized linear models with examples in R (2018) by Dunn and Smyth.
• Delivery:
– 3 lectures per week
– 1 computer practical per fortnight (even weeks)
• Assessment:
– Assignment 1 due in Week 6 (10%)
– Assignment 2 due in Week 1 of Term 2 (10%)
– Summer exam: 80%
4
CONTENTS 5
Assumptions of the normal linear model
For i = 1, . . . , n:
Yi = xTi β + ϵi
where
• E(ϵi ) = 0;
• Var(ϵi ) = σ 2 ;
• Cov(ϵi , ϵj ) = 0 for i ̸= j;
• ϵi has a normal distribution.
Rewriting the above in terms of observable quantities:
Y ∼ Nn (µ, σ 2 I n )
where
• E(Yi | xi ) = µi where µi = xTi β;
• Var(Yi | xi ) = σ 2 ;
• Cov(Yi , Yj | xi , xj ) = 0 for i ̸= j;
• Yi | xi ∼ N (µi , σ 2 ),
• Yi is continuous and unbounded.
6 CONTENTS
Relaxing the assumptions for GLMs
Generalized linear models (GLMs) generalise from the normal linear models as follows:
1. The response distribution is not necessarily normal, but a member of the exponential
family of distributions.
2. A link function g allows a non-linear relationship between the parameter vector β and
the mean µi :
g(µi ) = xTi β.
3. A variance function V (µ) allows the variance of the response to depend on the mean:
Var(Yi ) = ϕV (µi ).
Note:
a. We recover the normal linear model by using the normal distribution as the response
distribution and setting g(µ) = µ, V (µ) = 1 and ϕ = σ 2 .
b. Point 3. is a consequence of Point 1.
Why learn about GLMs?
• GLMs greatly expand the range of problems accessible to regression modelling.
• Many methods from normal linear models can be extended or adapted to GLMs.
• Further extensions of normal linear models also apply to GLMs.
Structure of the module
Chapter 1 Weighted linear models
Chapter 2 Binary regression
Chapter 3 Poisson regression
Chapter 4 Exponential dispersion models
Chapter 5 Generalized linear models
Chapter 6 Maximum likelihood estimation for GLMs
Chapter 7 Diagnostics for GLMs
Chapter 8 Model choice and hypothesis tests
Chapter 9 Binary classification
Chapter 10 Some advanced topics
Chapter 1 Weighted linear regression
1.1 Relaxing the assumptions of the linear model
We relax the homoscedasticity assumption in the normal linear model:
• Var(Yi | xi ) = ϕ/wi where wi is the weight of observation i. The weights w1 , . . . , wn are
fixed, known quantities. We are not trying to estimate them. The dispersion parameter ϕ
may be unknown.
• Observations with higher weights have a smaller variance. They contain more information
about the unknown parameters.
1.2 Weighted estimation - an example
Example: Turbines data
Scroby Sands Wind Farm. Rob Faulkner, CC BY 2.0 via Wikimedia Commons.
Each column in the table below represents a group of turbines that were left to run for a certain
time, then the turbines were inspected for fissures (cracks).
Outcome yi represents the proportion of turbines in the ith group that developed a fissure.
What is your estimate of µ = E(Y )?
i 1 2 3 4 5 6 7 8 9 10 11
yi 0 0.08 0.06 0.1 0.17 0.23 0.21 0.46 0.65 0.52 0.58
7
8 CHAPTER 1. WEIGHTED LINEAR REGRESSION
Not all groups had the same number of turbines. The table below gives mi , the number of
turbines for group i.
What is your estimate of µ = E(Y ) now?
i 1 2 3 4 5 6 7 8 9 10 11
yi 0.00 0.08 0.06 0.10 0.17 0.23 0.21 0.46 0.65 0.52 0.58
mi 39 53 33 73 30 39 42 13 34 40 36
The original data recorded Counti , the number of turbines with a fissure in group i.
What is your estimate of µ = E(Y ) now?
i 1 2 3 4 5 6 7 8 9 10 11
yi 0.00 0.08 0.06 0.10 0.17 0.23 0.21 0.46 0.65 0.52 0.58
mi 39 53 33 73 30 39 42 13 34 40 36
Counti 0 4 2 7 5 9 9 6 22 21 21
1.3. REVISION - LEAST SQUARES ESTIMATION 9
1.3 Revision - Least squares estimation
Let Y1 , . . . , Yn ∈ R be independent random variables with
E(Yi | xi ) = xTi β,
Var(Yi | xi ) = ϕ,
where β ∈ Rp . The (ordinary) least squares estimate of β minimises the sum of squares
T
(yi − xTi β)2
X
S(β) = = Y − Xβ Y − Xβ ,
i
where X is the design matrix such that row i of X is xTi .
The (ordinary) least squares estimate can be written in closed form as
−1
β
b = XT X X T y.
The deviance of the fitted model is defined as D = S(β).
b
The (ordinary) least squares estimator is
−1
β
b = XT X XT Y
and satisfies
E(β)
b = β;
−1
Var(β)
b = ϕ XT X .
1.4 The Gauss-Markov theorem
Suppose now that the scalar parameter γ = aT β is the focus of interest.
For example, we might have ai = δik for given k so that γ = βk selects a single element of the
parameter vector.
Let βb be the least squares estimator for β. Then the Gauss-Markov theorem states that
γb = aT β
b is the unique linear unbiased estimator of γ with minimum variance.
1.5 Weighted least squares
Assume a linear model with
E(Yi | xi ) = xTi β,
ϕ
Var(Yi | xi ) = ,
wi
where w1 , . . . , wn are known, non-negative weights.
What is the optimal estimator for β?
10 CHAPTER 1. WEIGHTED LINEAR REGRESSION
We solve the problem by rescaling our predictors xi and outcomes Yi . Let
1/2
Yi∗ = w i Yi ,
1/2
x∗i = wi xi .
Then
E(Yi∗ | x∗i ) = β T x∗i ,
Var(Yi∗ | x∗i ) = ϕ,
and the problem is reduced to a homoscedastic linear model.
From the Gauss-Markov theorem, on the transformed scale (X ∗ , Y ∗ ), the optimal estimator for
β minimizes the deviance 2
Yi∗ − (x∗i )T β
X
D(β) =
i
This can be written on the original scale (X, Y ) as
2
xTi β
X
D(β) = wi Yi −
i
Hence, on the original scale, we minimize a weighted sum of squares.
The estimator β
b that minimizes the deviance D(β) can be written in closed form as
h i−1
β
b = (X ∗ )T X ∗ (X ∗ )T Y ∗
On the original scale (X, Y ) this can be rewritten
h i−1
β
b = XT W X XT W Y
where W = diag(w1 , . . . wn ) is a diagonal matrix of weights.
Note: Observations with a higher weight make a greater contribution to S(β). An observation
with higher weight has a smaller variance. It contains more information about β. This explains
why we divide the variance by the weight.
1.6. EXAMPLES OF WEIGHTED REGRESSION 11
1.6 Examples of weighted regression
1.6.1 Example 1: weighted mean
Suppose Y1 , . . . , Yn are independent with common mean but different variances, that is
E(Yi ) = µ,
Var(Yi ) = ϕ/wi .
Then the optimal linear unbiased estimator is the weighted mean
Pn
i=1wi Yi
µb = Pn
i=1 wi
where the weights are inversely proportional to the variance.
Proof:
This is a weighted linear model with an intercept term and no predictors.
12 CHAPTER 1. WEIGHTED LINEAR REGRESSION
1.6.2 Example 2: Galton’s peas
Galton’s sweet pea data (galtonpeas in alr4 package):
• Parent: mean diameter of the parent (in 0.01 inches),
• Progeny: mean diameter of offspring (in 0.01 inches),
• SD: offspring diameter standard deviation (in 0.01 inches).
The diameter standard deviation tends to be larger for offspring of parents with large diameter.
We compare an ordinary least squares regression of Parent on Progeny against a weighted least
squares regression using SD2 as weights.
lm1 <- lm(Progeny ~ Parent, data = galtonpeas)
lm2 <- lm(Progeny ~ Parent, weights=1/SDˆ2, data=galtonpeas)
OLS coefficients:
(Intercept) Parent
12.70 0.21
Weighted LS coefficients:
(Intercept) Parent
12.8 0.2
OLS
WLS
Progeny mean diameter
17.0
16.5
16.0
15 16 17 18 19 20 21
Parent mean diameter
Exercise 1
Consider an experiment in which it is known that the variance of the errors of the first two
observations is four times as large as the variance of the errors for the next three observations.
The errors are assumed to be independent. You are given the following model:
µ + ϵi
i = 1 or i = 4,
Yi = µ + λ + ϵi i = 2 or i = 5,
µ + 2λ + ϵ3 i = 3.
1.6. EXAMPLES OF WEIGHTED REGRESSION 13
Find the design matrix for this model and then calculate the generalised least squares estimate,
given that y T = (125, 62.5, 12.5, 100, 50).
14 CHAPTER 1. WEIGHTED LINEAR REGRESSION
1.7 The hat matrix
The fitted values of a weighted linear model are given by
−1
Y
c = Xβ
b = X XT W X XT W Y = HY
where H is the hat matrix
−1
H = X XT W X XT W
which projects the observations Y onto the fitted values Y
c
Y
c = HY .
The hat matrix for the weighted linear model is
• idempotent, that is HH = H, but
• not symmetric H T ̸= H.
Some sources use a different definition of the hat matrix
−1
H∗ = W 1/2 X X T W X X T W 1/2
which is both symmetric and idempotent.
This is the hat matrix on the scale of (X ∗ , Y ∗ ), that is the matrix such that
∗
Y
c = H ∗Y ∗.
This disagreement is not important.
• In practice we only ever calculate the diagonal elements of the hat matrix.
• The diagonal elements are the same under both definitions.
The hat-value or leverage hi is the ith diagonal element of the hat matrix
−1
hi = wi xTi X T W X xi .
As we have seen in ST231, the sum of the hat values is equal to the number of parameters, that
is
n
X
hi = p.
i=1
1.8. WEIGHTED RESIDUALS 15
1.8 Weighted residuals
We have two different representations of the model which give us two different sets of residuals.
On the original scale (X, Y ) we have the response residuals defined as the difference between
the observed and fitted values:
ri = yi − xTi β.b
For the transformed scale (X ∗ , Y ∗ ) we have the weighted residuals:
ri∗ = yi∗ − (x∗i )T β
b
√
= wi yi − xTi β)
b
√
= w i ri
Both residuals have zero expectation
E(Ri | X) = 0,
E(Ri∗ | X) = 0,
but different variances
(1 − hi )ϕ
Var(Ri | X) = ,
wi
Var(Ri∗ | X) = (1 − hi )ϕ.
The weighted residuals are an example of what we will later call Pearson residuals when we
look at GLMs.
The variance of the response residual depends on the weight. We do not know if a response
residual is large or small unless we also know the weight. If there are no observations with high
leverage we have approximately
Var(Ri∗ | X) ≈ ϕ.
This makes the weighted residual more useful for model diagnostics.
The weighted residuals retain all the useful properties we associate with residuals in linear
models.
√
• If the model has an intercept term, then ni=1 wi ri∗ = ni=1 wi ri = 0.
P P
• An unbiased estimate of the dispersion parameter ϕ is the sum of squares of the weighted
residuals, divided by the residual degrees of freedom n − p
n
1 X
ϕb = (r∗ )2 .
n − p i=1 i
16 CHAPTER 1. WEIGHTED LINEAR REGRESSION
The standardized residual is the residual divided by an estimate of its standard deviation.
Both residuals lead to the same definition:
√
(s) wi (yi − xTi β)
b
ri = q
(1 − hi )ϕb
The standardized residual has asymptotic variance 1 as n → ∞.
We use standardized residuals to compare observations controlling for
• different weights w1 , . . . , wn , and
• different leverages h1 , . . . , hn .
Exercise 2
Show that the hat matrix on the scale of (X ∗ , Y ∗ ), that is the matrix such that
∗
Y
c = H ∗Y ∗
is given by −1
H ∗ = W 1/2 X X T W X X T W 1/2 .