STATISTICAL METHODS - 2
Regression Analysis - PSTAT 126
NOTES 15
LOGISTIC REGRESSION AND GLMs
Department of Statistics and Applied Probability
University of California, Santa Barbara
Today’s Lecture
Logistic regression / GLMs
I Model framework
I Interpretation
I Estimation
1 of 37
Linear regression
Course started with the model
yi = β0 + β1 xi + i
where
i ∼ (0, σ2 )
In particular, yi has been continuous throughout the course
2 of 37
Binary responses
Binary outcomes are common in practice; usually indicate
some event
Yes vs no
Transplant vs no transplant
Death vs no death
3 of 37
Binary responses
How should we deal with binary (0/1) y’s?
Regression focuses on E(y|x)
For binary outcomes, we want E(y|x) = p(y = 1|x)
Does pi = p(y = 1|x) = β0 + β1 xi work?
4 of 37
Linear regression for binary outcome
1.0
0.5
y
0.0
−10 −5 0 5
x
5 of 37
What we need for binary outcomes
Fitted probabilities should be between 0 and 1
Use a invertible function g : (0, 1) → (−∞, ∞) to link
probabilities to the real line
Build a model for g(pi ) = β0 + β1 xi
6 of 37
Link functions
Lots of possible link functions: logit, probit,
complimentary log-log
By far, most common is the logit link:
pi
g(pi ) = logit(pi ) = log
1 − pi
The inverse link function is also useful:
exp(z)
g−1 (z) =
1 + exp(z)
7 of 37
Logistic regression
Model is now
E(yi |xi ) = pi
pi
g(pi ) = log = β0 + β1 xi
1 − pi
Using the logit link, we have
exp(β0 + β1 xi )
pi = g−1 (β0 + β1 xi ) =
1 + exp(β0 + β1 xi )
8 of 37
Parameter interpretation
Suppose we can estimate β0 , β1 ; what do they mean?
For a binary predictor ...
9 of 37
Parameter interpretation
For a continuous predictor ...
10 of 37
Parameter estimation
For linear regression, we used least squares and found that
this corresponded to ML
Try using maximum likelihood for logistic regression; need
a likelihood ...
11 of 37
ML for logistic regression
Assume that [yi |xi ] ∼ Bern(pi )
y
Density function is p(yi ) = pi i (1 − pi )1−yi
As before, use that logit(pi ) = β0 + β1 xi
Likelihood is
n
y
Y
L(β0 , β1 ; y) = pi i (1 − pi )1−yi
i=1
12 of 37
ML for logistic regression
Log likelihood is easier to work with, but it is typically not
possible to find a closed-form solution
Iterative algorithms are used instead (Newton-Raphson,
Iteratively Reweighted Least Squares)
These are implemented for a variety of link functions in R
13 of 37
♠ Likelihood Functions
• Data: {(yi , xi ), i = 1, 2, . . . , n}, where xi = (1, Xi1 , Xi2 , . . . , Xik )′ .
• Parameters: β = (β0 , β1 , . . . , βk )′ .
• Binary Logistic Regression Models
Let yi be a binary response taking values 0 or 1. Then the likelihood
function is given by
n
Y Yn
exp(yi x′i β)
L(β) = pyi i (1 − pi ) 1−yi
=
i=1 i=1
1 + exp(x ′
i β)
and the log-likelihood function is
n n
X o
′ ′
l(β) = log L(β) = yi xi β − log[1 + exp(xi β)] .
i=1
12-22
• Binomial Logistic Regression Models
Let yi be a binary response taking values 0, 1, . . . , ni . Then the
likelihood function is given by
n
! n
!
Y ni y n −y
Y ni exp(y i x′
i β)
L(β) = pi i (1 − pi ) i i = ni
i=1
yi
i=1
yi [1 + exp(x ′
i β)]
and the log-likelihood function is
n n
!
X ni o
l(β) = log L(β) = yi x′i β − ni log[1 + exp(x′i β)] + log .
i=1
yi
• Poisson Regression Models
Let yi be a count variable taking values 0, 1, . . . . Then the likelihood
function is given by
Yn
exp(yi xi β)
L(β) = exp{− exp(x′i β)}.
i=1
yi !
12-23
and the log-likelihood function is
n n
X o
l(β) = log L(β) = yi x′i β − exp(x′i β) − log(yi !) .
i=1
♠ Maximum Likelihood Estimation (MLE)
For the GLM, there is no analytical form of the MLE β̂ available. The
Newton-Raphson (NR) algorithm is commonly used for computing the
MLE β̂ of β.
12-24