EBPI Epidemiology, Biostatistics and Prevention Institute

STA402: Likelihood & Regression

I
Torsten Hothorn

UZH Master Biostatistics, HS 2025

 Welcome

University of Zurich, EBPI UZH Master Biostatistics, HS 2025 STA402: Likelihood & Regression I Page 2
Schedule and Rooms

– Lecture (TH) Wed 8:15-9:00 and 9:15-10:00 Y03-G-91

– Exercise (Leticia Wüthrich) Wed 10:15-11:00 Y03-G-91
– Lab (TH) Wed 14:15-15:00 HIT-E03 (off Central)

University of Zurich, EBPI UZH Master Biostatistics, HS 2025 STA402: Likelihood & Regression I Page 3
Resources

All communication via OLAT.

– Script and slides are already available
– Exercise assignments are published weekly Wed/Thr
– Reading assignments are published weekly Wed/Thr
– Lab assignments and solutions are published before and
after the lab

University of Zurich, EBPI UZH Master Biostatistics, HS 2025 STA402: Likelihood & Regression I Page 4
Exam

2 hrs open book, in January.

University of Zurich, EBPI UZH Master Biostatistics, HS 2025 STA402: Likelihood & Regression I Page 5
 Introduction

University of Zurich, EBPI UZH Master Biostatistics, HS 2025 STA402: Likelihood & Regression I Page 6
The Central Dogma of Statistics

Everything is in the distribution:

Y ∼ PY

Y is called response (outcome, dependent, endogenous)

variable (actually: “random” variable)

University of Zurich, EBPI UZH Master Biostatistics, HS 2025 STA402: Likelihood & Regression I Page 7
Regression Analysis

Everything is in the conditional distribution:

Y | X = x ∼ PY|X=x

X (typically multivariate) are called explanatory (design,

independent, exogenous, predictor) variables or covariates

How do changes in x propagate to changes in PY|X=x ?

University of Zurich, EBPI UZH Master Biostatistics, HS 2025 STA402: Likelihood & Regression I Page 8
NB: Terminology

“Regression” classically means

– Y∈R
– Y | X = x ∼ N(µ(x), σ 2 ) ⇒ E(Y | X = x) = µ(x)

“Generalised Regression” means

Y | X = x ∼ PY|X=x

without these restrictions.

University of Zurich, EBPI UZH Master Biostatistics, HS 2025 STA402: Likelihood & Regression I Page 9
 Univariate Distributions

University of Zurich, EBPI UZH Master Biostatistics, HS 2025 STA402: Likelihood & Regression I Page 10
Distribution Functions

Sample space: Y ∈ Ξ
σ-algebra: C (in essense the set of “suitable” subsets of Ξ)
Probability measure: PY : C → [0, 1]
Distribution: Y ∼ PY
A ∈ C is called event and PY (A) is a probability

Cumulative distribution function: FY : Ξ → [0, 1] with

FY (y) = PY ({ν ∈ Ξ | ν ≤ y})

FY is monotone non-decreasing

University of Zurich, EBPI UZH Master Biostatistics, HS 2025 STA402: Likelihood & Regression I Page 11
Dichotomous Variables

Y ∈ {y1 , y2 }, FY (y1 ) = π1 , FY (y2 ) = 1

1.0
0.8

π1
0.6
FY

0.4
0.2
0.0

y1 y2

University of Zurich, EBPI UZH Master Biostatistics, HS 2025 STA402: Likelihood & Regression I Page 12
Dichotomous Variables
Density function: fY : Ξ → R+
fY (y1 ) = FY (y1 ) = π1
fY (y2 ) = FY (y2 ) − FY (y1 ) = 1 − π1

1.0
0.8

π1
0.6
FY

0.4

1 − π1
0.2
0.0

y1 y2

University of Zurich, EBPI UZH Master Biostatistics, HS 2025 STA402: Likelihood & Regression I Page 13
Dichotomous Variables

Odds function: OY : Ξ → R+
FY (y)
OY (y) = 1−F Y (y)
π1
OY (y1 ) = 1−π 1

NB: This is equivalent to Y ∼ B(1, π1 )

University of Zurich, EBPI UZH Master Biostatistics, HS 2025 STA402: Likelihood & Regression I Page 14
Polytomous Variables

Y ∈ {y1 , y2 , . . . , yK }, FY (yk ) = πk , FY (yK ) = 1

1.0
0.8

π3
0.6
FY

π2
0.4

π1
0.2
0.0

y1 y2 y3 y4

University of Zurich, EBPI UZH Master Biostatistics, HS 2025 STA402: Likelihood & Regression I Page 15
Polytomous Variables

Density function: fY (y1 ) = π1 ,

fY (yk ) = FY (yk ) − FY (yk−1 ) = πk − πk−1
Pk
NB: FY (yk ) = i=1 fY (yi )

FY (yk ) πk
Odds function: OY (yk ) = 1−FY (yk ) = 1−πk

NB: ordered polytomous means y1 < y2 < · · · < yK

NB: This is equivalent to a multinomial distribution

University of Zurich, EBPI UZH Master Biostatistics, HS 2025 STA402: Likelihood & Regression I Page 16
Count Variables

Ξ = N, FY (i) = πi , FY (∞) = 1

1.0
0.8
0.6
FY(y)

0.4
0.2
0.0

0 5 10 15 20

University of Zurich, EBPI UZH Master Biostatistics, HS 2025 STA402: Likelihood & Regression I Page 17
Count Variables

Density function: fY (0) = π0 , fY (i) = πi − πi−1

0.15
0.10
fY(y)

0.05
0.00

0 5 10 15 20

University of Zurich, EBPI UZH Master Biostatistics, HS 2025 STA402: Likelihood & Regression I Page 18
Count Variables
πi
Odds function: OY (i) = 1−πi

15
10
log(OY(y))

5
0
−5

0 5 10 15 20

University of Zurich, EBPI UZH Master Biostatistics, HS 2025 STA402: Likelihood & Regression I Page 19
Continuous Variables

Bounded: Ξ = (0, 1)

Positive: Ξ = (0, ∞)

Real: Ξ = R

University of Zurich, EBPI UZH Master Biostatistics, HS 2025 STA402: Likelihood & Regression I Page 20
Bounded Continuous Variables

Cumulative distribution function: FY : (0, 1) → [0, 1]

0.8
0.6
FY(y)

0.4
0.2

0.0 0.2 0.4 0.6 0.8 1.0

University of Zurich, EBPI UZH Master Biostatistics, HS 2025 STA402: Likelihood & Regression I Page 21
Bounded Continuous Variables
FY (y)
Odds function: OY (y) = 1−FY (y)

15
10
OY(y)

5
0

0.0 0.2 0.4 0.6 0.8 1.0

University of Zurich, EBPI UZH Master Biostatistics, HS 2025 STA402: Likelihood & Regression I Page 22
Bounded Continuous Variables

Density function: fY (y) = FY′ (y)

3.0
2.5
2.0
fY(y)

1.5
1.0

0.0 0.2 0.4 0.6 0.8 1.0

University of Zurich, EBPI UZH Master Biostatistics, HS 2025 STA402: Likelihood & Regression I Page 23
Bounded Continuous Variables

Quantile function: FY−1 (τ ) with FY−1 : (0, 1) → Ξ

1.0
0.8
0.6
Y (τ)
F−1

0.4
0.2
0.0

0.0 0.2 0.4 0.6 0.8 1.0

University of Zurich, EBPI UZH Master Biostatistics, HS 2025 STA402: Likelihood & Regression I Page 24
NB: Densities
Density wrt dominating measure µ: PY = fY ⊙ µ

Z
y=∞
FY (y) = 1(u ≤ y)fY (u) dµ(u) = 1

Discrete (wrt counting measure):

X
FY (y) = 1(u ≤ y)fY (u)
u∈Ξ

fY (y) is a probability

Continuous (wrt Lebesque measure):

Z
FY (y) = 1(u ≤ y)fY (u) du

fY (y) is NOT a probability (but risk or intensity)

University of Zurich, EBPI UZH Master Biostatistics, HS 2025 STA402: Likelihood & Regression I Page 25
Positive Continuous Variables

Cumulative distribution function: FY : (0, ∞) → [0, 1]

1.0
0.8
0.6
FY(y)

0.4
0.2
0.0

0 2 4 6 8 10

University of Zurich, EBPI UZH Master Biostatistics, HS 2025 STA402: Likelihood & Regression I Page 26
Positive Continuous Variables
FY (y)
Odds function: OY (y) = 1−FY (y)

50
40
30
OY(y)

20
10
0

0 2 4 6 8 10

University of Zurich, EBPI UZH Master Biostatistics, HS 2025 STA402: Likelihood & Regression I Page 27
Positive Continuous Variables

Density function: fY (y) = FY′ (y)

0.25
0.20
0.15
fY(y)

0.10
0.05
0.00

0 2 4 6 8 10

University of Zurich, EBPI UZH Master Biostatistics, HS 2025 STA402: Likelihood & Regression I Page 28
Positive Continuous Variables
−1
Quantile function: FY|X=x (τ )

10
8
Y (τ)

6
F−1

4
2
0

0.0 0.2 0.4 0.6 0.8 1.0

University of Zurich, EBPI UZH Master Biostatistics, HS 2025 STA402: Likelihood & Regression I Page 29
Real Continuous Variables

Cumulative distribution function: FY : R → [0, 1]

1.0
0.8
0.6
FY(y)

0.4
0.2
0.0

−3 −2 −1 0 1 2 3

University of Zurich, EBPI UZH Master Biostatistics, HS 2025 STA402: Likelihood & Regression I Page 30
Real Continuous Variables
FY (y)
Odds function: OY (y) = 1−FY (y)

1.6
1.4
1.2
OY(y)

1.0
0.8
0.6

−0.3 −0.2 −0.1 0.0 0.1 0.2 0.3

University of Zurich, EBPI UZH Master Biostatistics, HS 2025 STA402: Likelihood & Regression I Page 31
Real Continuous Variables

Density function: fY (y) = FY′ (y)

0.4
0.3
fY(y)

0.2
0.1
0.0

−3 −2 −1 0 1 2 3

University of Zurich, EBPI UZH Master Biostatistics, HS 2025 STA402: Likelihood & Regression I Page 32
Real Continuous Variables
−1
Quantile function: FY|X=x (τ )

2
1
Y (τ)

0
F−1

−1
−2

0.0 0.2 0.4 0.6 0.8 1.0

University of Zurich, EBPI UZH Master Biostatistics, HS 2025 STA402: Likelihood & Regression I Page 33
Mixed Discrete / Continuous Variables

Ξ = [0, ∞), FY (0) = π0

1.0
0.8
0.6

π0
FY(y)

0.4
0.2
0.0

0.0 0.5 1.0 1.5 2.0 2.5 3.0

University of Zurich, EBPI UZH Master Biostatistics, HS 2025 STA402: Likelihood & Regression I Page 34
Parameterisations

Dicho/Polytomous: π1 , . . . , πK−1 (binomial or multinomial

distribution); fully parameterised

Sparse parameterisations (“shape” given, few parameters ϑ):

Count: Poisson, Negative-binomial, . . .

Bounded continuous: Beta

Positive continuous: χ2 , F, Weibull, log-normal, . . .

Real: Normal, Logistic, t, . . .

write FY (Y | ϑ)

University of Zurich, EBPI UZH Master Biostatistics, HS 2025 STA402: Likelihood & Regression I Page 35
Estimation (1: “nonparametric”)

Y1 , . . . , YN iid Yi ∼ PY

N
X
F̂Y,N (y) = N−1 1(Yi ≤ y)
i=1

Empirical cumulative distribution function;

| FY (y) − F̂Y,N (y) |→ 0 a.s. when N → ∞ (Glivenko-Cantelli)

University of Zurich, EBPI UZH Master Biostatistics, HS 2025 STA402: Likelihood & Regression I Page 36
Estimation (2: “parametric”)

Y1 , . . . , YN iid Yi ∼ PY with FY (y | ϑ), unknown parameter(s)

ϑ ∈ Θ (parameter space)

N
X
ϑ̂N = arg max log(fY (Yi | ϑ))
ϑ∈Θ i=1

Maximum-likelihood estimation (Fisher, 1922); very general and

some nice properties

Things are a bit more complex, but for the moment we’ve got
everything we need.

University of Zurich, EBPI UZH Master Biostatistics, HS 2025 STA402: Likelihood & Regression I Page 37
 Two-group Comparisons

University of Zurich, EBPI UZH Master Biostatistics, HS 2025 STA402: Likelihood & Regression I Page 38
Two-way Experiments

Simplest design of intervention experiments with design

variable x ∈ {something, something else}. Let’s use “placebo”
and “treatment” from now on.

Simple idea: Estimate FY|X=x (y | placebo) and

FY|X=x (y | treatment) separately and compare the two
conditional distribution functions.

University of Zurich, EBPI UZH Master Biostatistics, HS 2025 STA402: Likelihood & Regression I Page 39
Two-way Experiments

1.0
0.8
0.6
FYN(y)
^

0.4
0.2

placebo
0.0

treatment

0 5 10 15

University of Zurich, EBPI UZH Master Biostatistics, HS 2025 STA402: Likelihood & Regression I Page 40
Two-way Experiments

Can we summarise the difference between FY|X=x (y | placebo)

and FY|X=x (y | treatment) by one number β?

FY|X=x (y | treatment) = FY|X=x (y − βshift | placebo)

FY|X=x (y | treatment) = FY|X=x (y | placebo)βLehmann
1 − FY|X=x (y | treatment) = (1 − FY|X=x (y | placebo))βHR
FY|X=x (y | treatment) FY|X=x (y | placebo)
= βOR
1 − FY|X=x (y | treatment) 1 − FY|X=x (y | placebo)
..
.

University of Zurich, EBPI UZH Master Biostatistics, HS 2025 STA402: Likelihood & Regression I Page 41
Reformulate the Problem

FY|X=x (y | placebo) = FZ (h(y))

with FZ : R → [0, 1] a continuous cdf.

h(y) = FZ−1 (FY|X=x (y | placebo)) always exists

Z = h(Y) is a transformed RV

We can now assume

FY|X=x (y | treatment) = FZ (h(y) − β)

University of Zurich, EBPI UZH Master Biostatistics, HS 2025 STA402: Likelihood & Regression I Page 42
Shift Alternatives

FY|X=x (y | treatment) = Φ(h(y) − β)

⇒ E(h(Y) | treatment) − E(h(Y) | placebo) = β

NB: h(Y) ∼ N(0, 1) in placebo and h(Y) ∼ N(β, 1) in treatment

group

University of Zurich, EBPI UZH Master Biostatistics, HS 2025 STA402: Likelihood & Regression I Page 43
Shift Alternatives

Y | placebo ∼ N(0, 1), βshift = 1

1.0
0.8
0.6
FY(y)

0.4
0.2

placebo
0.0

treatment

−4 −2 0 2 4

University of Zurich, EBPI UZH Master Biostatistics, HS 2025 STA402: Likelihood & Regression I Page 44
Shift Alternatives

Y | placebo ∼ χ23 , βshift = 1

1.0
0.8
0.6
FY(y)

0.4
0.2

placebo
0.0

treatment

0 5 10 15 20

University of Zurich, EBPI UZH Master Biostatistics, HS 2025 STA402: Likelihood & Regression I Page 45
Lehmann Alternatives

FY|X=x (y | treatment) = exp(− exp(−(h(y) − β)))

exp(− exp(−h(y)) exp(β))
exp(− exp(−h(y)))exp(β)

⇒ βLehmann = exp(β)

FZ (z) = exp(− exp(−z)) is cdf of Gumbel (maximum extreme

value) distribution

University of Zurich, EBPI UZH Master Biostatistics, HS 2025 STA402: Likelihood & Regression I Page 46
Lehmann Alternatives

Y | placebo ∼ χ23 , log(βLehmann ) = 1

1.0
0.8
0.6
FY(y)

0.4
0.2

placebo
0.0

treatment

0 5 10 15 20

University of Zurich, EBPI UZH Master Biostatistics, HS 2025 STA402: Likelihood & Regression I Page 47
Hazard Ratios

FY|X=x (y | treatment) = 1 − exp(− exp(h(y) − β))

1 − FY|X=x (y | treatment) = exp(− exp(h(y) − β))
exp(− exp(h(y)))exp(β)

⇒ exp(h(y)) is cumulative hazard function

⇒ βHR = exp(β)

FZ (z) = 1 − exp(− exp(z)) is cdf of Gompertz (minimum extreme

value) distribution

University of Zurich, EBPI UZH Master Biostatistics, HS 2025 STA402: Likelihood & Regression I Page 48
Hazard Ratios

Y | placebo ∼ χ23 , log(βHR ) = 1

1.0
0.8
0.6
FY(y)

0.4
0.2

placebo
treatment

0 5 10 15 20

University of Zurich, EBPI UZH Master Biostatistics, HS 2025 STA402: Likelihood & Regression I Page 49
Odds Ratios

FY|X=x (y | treatment) = expit(h(y) − β)

= (1 + exp(−(h(y) − β))−1
(1 + exp(−(h(y) − β))−1
OY|X=x (y | treatment) =
1 − (1 + exp(−(h(y) − β))−1
1
= exp(β)
exp(−h(y))
= exp(β)OY|X=x (y | placebo)

⇒ βOR = exp(β)

FZ (z) = (1 + exp(−z))−1 is cdf of standard logistic distribution

University of Zurich, EBPI UZH Master Biostatistics, HS 2025 STA402: Likelihood & Regression I Page 50
Odds Ratios

Y | placebo ∼ χ23 , log(βOR ) = 1

1.0
0.8
0.6
FY(y)

0.4
0.2

placebo
treatment

0 5 10 15 20

University of Zurich, EBPI UZH Master Biostatistics, HS 2025 STA402: Likelihood & Regression I Page 51
Swiss Body Mass Indices

2012 survey (N = 16427) in Switzerland

Explain conditional distribution of BMI given
– Sex,
– Smoking status,
– Age,
– Education,
– Physical activity,
– Alcohol intake,
– Fruit and vegetable consumption,
– Region, and
– Nationality.

University of Zurich, EBPI UZH Master Biostatistics, HS 2025 STA402: Likelihood & Regression I Page 52
Swiss Body Mass Indices

Female
Male

1.0

0.8

0.6
FY(BMI)

0.4

0.2

0.0

15 20 25 30 35 40

BMI

University of Zurich, EBPI UZH Master Biostatistics, HS 2025 STA402: Likelihood & Regression I Page 53
Shift Alternative
FY|X=x (y | x) = Φ(h(y) − βshift 1(male))

1.0

0.8

0.6
FY(BMI)

0.4

0.2

0.0

15 20 25 30 35 40

BMI

β̂shift = 0.603
University of Zurich, EBPI UZH Master Biostatistics, HS 2025 STA402: Likelihood & Regression I Page 54
Shift Alternative under Normality
FY|X=x (y | x) = Φ(ξy − α − βshift 1(male))

1.0

0.8

0.6
FY(BMI)

0.4

0.2

0.0

15 20 25 30 35 40

BMI

β̂shift = 1.686 BMI units difference between males and females

University of Zurich, EBPI UZH Master Biostatistics, HS 2025 STA402: Likelihood & Regression I Page 55
Odds Ratio Alternative
FY|X=x (y | x) = expit(h(y) − β1(male))

1.0

0.8

0.6
FY(BMI)

0.4

0.2

0.0

15 20 25 30 35 40

BMI

β̂OR = exp(β̂) = 0.343

University of Zurich, EBPI UZH Master Biostatistics, HS 2025 STA402: Likelihood & Regression I Page 56
Odds Ratio Alternative
h(y) − β1(male) is log-odds function

5
log(OY(BMI))

−5

15 20 25 30 35 40

BMI

β̂ = −1.07 vertical difference

University of Zurich, EBPI UZH Master Biostatistics, HS 2025 STA402: Likelihood & Regression I Page 57
Hazard Ratio Alternative
FY|X=x (y | x) = 1 − exp(− exp((h(y) − β1(male))))

1.0

0.8

0.6
FY(BMI)

0.4

0.2

0.0

15 20 25 30 35 40

BMI

β̂HR = exp(β̂) = 0.69 (Interpretation: STA425)

University of Zurich, EBPI UZH Master Biostatistics, HS 2025 STA402: Likelihood & Regression I Page 58
Lehmann Alternative
FY|X=x (y | x) = exp(− exp(−(h(y) − β1(male))))

1.0

0.8

0.6
FY(BMI)

0.4

0.2

0.0

15 20 25 30 35 40

BMI

β̂Lehmann = exp(β̂) = 2.086

University of Zurich, EBPI UZH Master Biostatistics, HS 2025 STA402: Likelihood & Regression I Page 59
Generalisation

Two groups:

FY|X=x (y | x) = FZ (h(y) − β1(x = treatment))

More general designs:

FY|X=x (y | x) = FZ (h(y) − β(x))

For now, “simple” models with β(x) = βx

University of Zurich, EBPI UZH Master Biostatistics, HS 2025 STA402: Likelihood & Regression I Page 60
 Some Simple Regression Models

University of Zurich, EBPI UZH Master Biostatistics, HS 2025 STA402: Likelihood & Regression I Page 61
Simple Binary Regression

Y ∈ {y1 , y2 }, x ∈ R:

π1 (x) = FY|X=x (y1 | x) = FZ (ϑ1 − βx)

With FZ = expit:

π1 (x)
= exp(ϑ1 − βx)
1 − π1 (x)

and
π1 (x)
1−π1 (x) exp(ϑ1 − βx)
π1 (x+1)
= = exp(β)
exp(ϑ1 − β(x + 1))
1−π1 (x+1)

University of Zurich, EBPI UZH Master Biostatistics, HS 2025 STA402: Likelihood & Regression I Page 62
Simple Binary Regression

1.0
0.8
0.6
π1(x)

0.4
0.2

ϑ1 = 0 and β = 5
ϑ1 = 0 and β = 2
ϑ1 = 1 and β = − 2
0.0

−0.5 0.0 0.5

University of Zurich, EBPI UZH Master Biostatistics, HS 2025 STA402: Likelihood & Regression I Page 63
Simple Ordered Regression

Y ∈ {y1 , y2 , . . . , yK }, x ∈ R:

πk (x) = FY|X=x (yk | x) = FZ (ϑk − βx)

With FZ = expit:

πk (x)
= exp(ϑk − βx) = exp(ϑk ) exp(−βx)
1 − πk (x)

and
πk (x)
1−πk (x) exp(ϑk − βx)
πk (x+1)
= = exp(β)
exp(ϑk − β(x + 1))
1−πk (x+1)

“Proportional odds model”

University of Zurich, EBPI UZH Master Biostatistics, HS 2025 STA402: Likelihood & Regression I Page 64
Simple Ordered Regression

β = 2, ϑ1 = −2, ϑ2 = −1, ϑ3 = 0, ϑ4 = 1, x ∈ [−1, 1]

x x x x x

1.0

0.8

0.6
FY(y)

0.4

0.2

y1 y2 y3 y4 y5 y1 y2 y3 y4 y5 y1 y2 y3 y4 y5 y1 y2 y3 y4 y5 y1 y2 y3 y4 y5

University of Zurich, EBPI UZH Master Biostatistics, HS 2025 STA402: Likelihood & Regression I Page 65
Simple Normal Regression

Y ∈ R, x ∈ R:

FY|X=x (y | x) = Φ(ξy − α − βx), ξ > 0

Y | X = x ∼ N ξ −1 (α + βx), ξ −2

University of Zurich, EBPI UZH Master Biostatistics, HS 2025 STA402: Likelihood & Regression I Page 66
Simple Normal Regression

ξ = 1, α = 0, β = 2

1.0

0.8
0.5

0.6

0.0
x

0.4

−0.5
0.2

0.0

−2 −1 0 1 2

University of Zurich, EBPI UZH Master Biostatistics, HS 2025 STA402: Likelihood & Regression I Page 67
Simple Weibull Regression

Y ∈ R+ , x ∈ R:

FY|X=x (y | x) = 1 − exp(− exp(ξ log(y) − α − βx)), ξ > 0

Y | X = x ∼ W(exp(α + βx), ξ)
NB: Hazard ratio
exp(ξ log(y) − α − βx)
= exp(β)
exp(ξ log(y) − α − β(x + 1))

“Proportional hazards model”

University of Zurich, EBPI UZH Master Biostatistics, HS 2025 STA402: Likelihood & Regression I Page 68
Simple Weibull Regression

ξ = 1, β = 2

1.0

0.8
0.5

0.6

0.0
x

0.4

−0.5
0.2

0.0

0.5 1.0 1.5 2.0 2.5

University of Zurich, EBPI UZH Master Biostatistics, HS 2025 STA402: Likelihood & Regression I Page 69
Simple Transformation Model

FY|X=x (y) = FZ (h(y) − βx)

“Lego” system with bricks



 Φ(z) Normal βshift (x) = βx
 expit(z) Logistic βOR (x) = exp(βx)


FZ (z) = 1 − exp(− exp(z)) Gompertz βHR (x) = exp(βx)
exp(− exp(−z)) Gumbel βLehmann (x) = exp(βx)




...


University of Zurich, EBPI UZH Master Biostatistics, HS 2025 STA402: Likelihood & Regression I Page 70
Simple Transformation Model
P(BMI ≤ 25 | sex) = expit(ϑ1 − β1(Male))

1.0
5

0.8
expit(ϑ1)

h(y1) = ϑ1

0.6
h(y) − βI(Male)

ϑ1 − β
0

FY(y)
expit(ϑ1 − β)

0.4
−5

0.2
0.0
−10

15 20 25 30 35 40 15 20 25 30 35 40

BMI BMI

BMI ≤ 25 means underweight or normal

University of Zurich, EBPI UZH Master Biostatistics, HS 2025 STA402: Likelihood & Regression I Page 71
Simple Transformation Model
P(BMI ≤ yk | sex) = expit(ϑk − β1(Male))

1.0
expit(ϑ3)

expit(ϑ3 − β)
5

0.8
h(y3) = ϑ3
expit(ϑ2)
ϑ3 − β
h(y2) = ϑ2

0.6
h(y) − βI(Male)

ϑ2 − β
0

FY(y)
expit(ϑ2 − β)

0.4
h(y1) = ϑ1
ϑ1 − β
−5

0.2
expit(ϑ1)
expit(ϑ1 − β)
0.0
−10

15 20 25 30 35 40 15 20 25 30 35 40

BMI BMI

BMI ≤ 18.5 means underweight; BMI ≤ 30 mean not obese

University of Zurich, EBPI UZH Master Biostatistics, HS 2025 STA402: Likelihood & Regression I Page 72
Simple Transformation Model

FY|X=x (y) = FZ (h(y) − βx)

“Lego” system with bricks

 h(yk ) = ϑk
 discrete
 ξy − α linear


h(y) = ξ log(y) − α log-linear
 a(y)⊤ ϑ smooth



...


h is monotone nondecreasing! Interpretation (log-odds,

log-cumulative hazard, etc) depends on FZ .

University of Zurich, EBPI UZH Master Biostatistics, HS 2025 STA402: Likelihood & Regression I Page 73
Swiss Body Mass Indices (Males only)
FY|X=x (y | x) = expit(h(y) − βage)

1.0

0.8

0.6 Age
20 yrs
FY(BMI)

30 yrs
40 yrs
50 yrs
60 yrs
0.4 70 yrs
80 yrs

0.2

0.0

15 20 25 30 35 40

BMI

exp(β̂) = 0.967 (odds reduced by this factor per year)

University of Zurich, EBPI UZH Master Biostatistics, HS 2025 STA402: Likelihood & Regression I Page 74
Swiss Body Mass Indices (Males only)
FY|X=x (y | x) = expit(h(y) − βage)

1500

Age
20 yrs
OY(BMI)

1000
30 yrs
40 yrs
50 yrs
60 yrs
70 yrs
80 yrs

500

15 20 25 30 35 40

BMI

exp(β̂) = 0.967 (odds reduced by this factor per year)

University of Zurich, EBPI UZH Master Biostatistics, HS 2025 STA402: Likelihood & Regression I Page 75
Swiss Body Mass Indices (Males only)
FY|X=x (y | x) = expit(h(y) − βage)

Age
log(OY(BMI))

0 20 yrs
30 yrs
40 yrs
50 yrs
60 yrs
70 yrs
80 yrs

−5

−10

15 20 25 30 35 40

BMI

β̂ = −0.033 (log-odds shifted by this amount per year)

University of Zurich, EBPI UZH Master Biostatistics, HS 2025 STA402: Likelihood & Regression I Page 76
 Multiple Regression Models

University of Zurich, EBPI UZH Master Biostatistics, HS 2025 STA402: Likelihood & Regression I Page 77
Linear Predictors

FY|X=x (y) = FZ (h(y) − β(x))

with
β(x) = x̃ ⊤ β

Configuration x with β(x) = 0 is called baseline or reference

configuration. Interpretation of h in this case should make sense!

University of Zurich, EBPI UZH Master Biostatistics, HS 2025 STA402: Likelihood & Regression I Page 78
Model Estimation (Discrete)

Model: y ∈ {y1 , . . . , yK }, x ∈ χ and x̃ ∈ RQ

P(Y ≤ yk | X = x) = FZ (ϑk − x̃ ⊤ β)

Observe datum (y, x) and evaluate density (=probability) for

GIVEN parameters ϑ = (ϑ1 , . . . , ϑK−1 ), β

L((y, x)) = P(Y = y | X = x) = fY (y | x) =

 FZ (ϑk − x̃ ⊤ β)

− 0 k=1
FZ (ϑk − x̃ ⊤ β) ⊤
− FZ (ϑk−1 − x̃ β) 1 < k < K
1 − FZ (ϑk−1 − x̃ ⊤ β) k = K


University of Zurich, EBPI UZH Master Biostatistics, HS 2025 STA402: Likelihood & Regression I Page 79
Model Estimation (Discrete)

Observe data (y, x)i , i = 1, . . . , N

(y, x)i is the ith observation
Assume (y, x)i and (y, x)j are independent ∀i ̸= j

N
Y
L= L((y, x)i )
i=1

is the probability of observing the data GIVEN ϑ, β.

Idea: The larger L, the better the data are explained by the
model GIVEN ϑ, β

University of Zurich, EBPI UZH Master Biostatistics, HS 2025 STA402: Likelihood & Regression I Page 80
Maximum Likelihood (Discrete)

Problem: We don’t know the parameters ϑ, β, we only know the

data
Solution: Make L a function of the parameters and fix the data
N
Y
L(ϑ, β) = L(ϑ, β | (y, x)i )
i=1

and estimate parameters such that

ϑ̂N , β̂ N = arg max L(ϑ, β) st. ϑk < ϑk+1 , 1 ≤ k < K

ϑ∈RK−1 ,β∈RQ

L(ϑ, β) is called the likelihood function

University of Zurich, EBPI UZH Master Biostatistics, HS 2025 STA402: Likelihood & Regression I Page 81
Maximum Likelihood (Discrete)

Computational simplification:

ϑ̂N , β̂ N = arg max log(L(ϑ, β)) st. ϑk < ϑk+1 , 1 ≤ k < K

ϑ∈RK−1 ,β∈RQ

with
N
X
log(L(ϑ, β)) = ℓ(ϑ, β) = ℓi (ϑ, β)
i=1

with ℓi (ϑ, β) = log(L(ϑ, β | (y, x)i ))

ℓi (ϑ, β) is the contribution of the ith observation (the
log-density) to the log-likelihood function ℓ(ϑ, β)

University of Zurich, EBPI UZH Master Biostatistics, HS 2025 STA402: Likelihood & Regression I Page 82
Score Function (Discrete)

N N
∂ℓ(ϑ, β) X X ∂ℓi (ϑ, β)
s(ϑ, β) = = si (ϑ, β) =
∂(ϑ, β) ∂(ϑ, β)
i=1 i=1

Under some regularity conditions we have

s(ϑ̂N , β̂ N ) = 0

University of Zurich, EBPI UZH Master Biostatistics, HS 2025 STA402: Likelihood & Regression I Page 83
Example: OR in 2 × 2 Table

y ∈ {y1 , y2 }, x ∈ {0, 1}

y1 y2
x=0 N01 N02
x=1 N11 N12
N = N01 + N02 + N11 + N12

P(Y = y1 | x = 0) = expit(ϑ1 )
P(Y = y2 | x = 0) = 1 − expit(ϑ1 )
P(Y = y1 | x = 1) = expit(ϑ1 − β)
P(Y = y2 | x = 1) = 1 − expit(ϑ1 − β)

ϑ̂1N , β̂N ?

University of Zurich, EBPI UZH Master Biostatistics, HS 2025 STA402: Likelihood & Regression I Page 84
Likelihood Function (Continuous)

Continuous RVs with Ξ = R.

We always observe intervals (y, ȳ] ⊂ R

¯
For datum ((y, ȳ], x) evaluate probability
¯
P(y < Y ≤ ȳ | X = x) = FZ (h(ȳ) − x̃ ⊤ β) − FZ (h(y) − x̃ ⊤ β)
¯ ¯
Log-likelihood contribution of ((y, ȳ], x)i is
¯
ℓi (h, β) = log(FZ (h(ȳ) − x̃ ⊤ β) − FZ (h(y) − x̃ ⊤ β))
¯

University of Zurich, EBPI UZH Master Biostatistics, HS 2025 STA402: Likelihood & Regression I Page 85
Likelihood Function (Continuous)

University of Zurich, EBPI UZH Master Biostatistics, HS 2025 STA402: Likelihood & Regression I Page 86
Likelihood Function (Continuous)

If measurement is precise (length of (y, ȳ] short), APPROXIMATE

likelihood by density evaluated at y =¯ y + (ȳ − y)/2
¯ ¯

ℓ(h, β) ≈ log(fY (y | X = x))

Problem: What is fY ?

∂FY (y | X = x)
fY (y | X = x) =
∂y

NB: Usually, only y is available (not (y, ȳ])

¯

University of Zurich, EBPI UZH Master Biostatistics, HS 2025 STA402: Likelihood & Regression I Page 87
Likelihood Function (Continuous)

fY|X=x (y | X = x) = fZ (h(y) − β(x))h′ (y)

(yi , x i ) for i = 1, . . . , N with log-likelihood contribution

ℓi (h, β) = log(fZ (h(yi ) − β(x i ))) + log(h′ (yi ))

University of Zurich, EBPI UZH Master Biostatistics, HS 2025 STA402: Likelihood & Regression I Page 88
Model Diagnostics: PIT

Probability Integral Transform (PIT)

Ui = P(Y ≤ yi | X = x i ) ∼ U[0, 1]

Idea: Check

Ûi = P̂(Y ≤ yi | X = x i ) = FZ (ĥ(yi ) − x̃ ⊤

i β̂) ∼ U[0, 1]

or equivalently

Ẑi = ĥ(yi ) − x̃ ⊤
i β̂ ∼ PZ

University of Zurich, EBPI UZH Master Biostatistics, HS 2025 STA402: Likelihood & Regression I Page 89
BMI Quantile-Quantile Plots

University of Zurich, EBPI UZH Master Biostatistics, HS 2025 STA402: Likelihood & Regression I Page 90
Residual Analysis

Model

FY|X=x (y) = FZ (h(y) − x̃ ⊤ β − α), α=0

Idea: If this simple model is correct ⇒ α(x) ≡ 0 in the more

complex model

FY|X=x (y) = FZ (h(y) − x̃ ⊤ β − α(x))

Estimate simple model (α = 0) and look for deviations

University of Zurich, EBPI UZH Master Biostatistics, HS 2025 STA402: Likelihood & Regression I Page 91
Residual Analysis

What is a residual?

Log-Likelihood contribution

ℓi ((h, β, α = 0))

Residual := Score contribution wrt α

∂ℓi ((h, β, α))
ri =
∂α α=0

Any association between ri and x i ?

University of Zurich, EBPI UZH Master Biostatistics, HS 2025 STA402: Likelihood & Regression I Page 92
Residual Analysis: BMI and Age

P(BMI ≤ y | sex) = exp(− exp(−(h(y) − β1(male) − α))), α=0

Look at correlation between residual r and age

University of Zurich, EBPI UZH Master Biostatistics, HS 2025 STA402: Likelihood & Regression I Page 93
Residual Analysis: BMI and Age

8000
6000
4000
2000
r

0
−2000
−4000

20 30 40 50 60 70

age

⇒ Include age in model!

University of Zurich, EBPI UZH Master Biostatistics, HS 2025 STA402: Likelihood & Regression I Page 94
Example: Score Test for Comparing Two Groups

Model (α ≡ 0):

P(Y ≤ y | placebo) = expit(h(y) + α)

P(Y ≤ y | treatment) = expit(h(y) + β + α)

H0 : β = 0 vs. log-odds ratio alternatives

Observe (y, x)i , i = 1, . . . , N (independent etc)

Under H0 (!!!), estimate cumulative distribution function

FY (y) = P(Y ≤ y)

from the whole sample

University of Zurich, EBPI UZH Master Biostatistics, HS 2025 STA402: Likelihood & Regression I Page 95
Example: Score Test for Comparing Two Groups

Maybe very simple by ECDF

N
X
−1
F̂Y,N (yi ) = (N + 1) 1(yj ≤ yi ) = (N + 1)−1 Ri
j=1

where Ri is the rank of the ith response value in the whole

sample

Then: ĥ(yi ) = logit((N + 1)−1 Ri )

University of Zurich, EBPI UZH Master Biostatistics, HS 2025 STA402: Likelihood & Regression I Page 96
Example: Score Test for Comparing Two Groups

Plug-in ĥ(yi ) and compute score wrt α ≡ 0

∂ℓi (ĥ(yi ), α)
Si,α = = 1 − 2Ri /(N + 1)
∂α
α=0

Use “correlation” between score and treatment as test statistic:

N N
Si,α 1(x i = treatment) ∼
X X
= Ri 1(x i = treatment) = W
i=1 i=1

Oups: Wilcoxon-Mann-Whitney-Rank-Sum Test

University of Zurich, EBPI UZH Master Biostatistics, HS 2025 STA402: Likelihood & Regression I Page 97
Example: Score Test for Comparing Two Groups

Plug-in ĥ(yi ) and compute score wrt α ≡ 0

∂ℓi (ĥ(yi ), α)
Si,α = = 1 − 2Ri /(N + 1)
∂α
α=0

Use “correlation” between score and treatment as test statistic:

N N
Si,α 1(x i = treatment) ∼
X X
= Ri 1(x i = treatment) = W
i=1 i=1

Oups: Wilcoxon-Mann-Whitney-Rank-Sum Test

University of Zurich, EBPI UZH Master Biostatistics, HS 2025 STA402: Likelihood & Regression I Page 98
Log-rank Test

Estimate h under the null β = 0 in model

P(Y ≤ y | placebo) = cloglog−1 (h(y) + α)

P(Y ≤ y | treatment) = cloglog−1 (h(y) + β + α)

Use h(yi ) = log(− log(1 − Ri /(N + 1)))) with ranks R1 , . . . , RN

The derivative of the corresponding log-likelihood with respect

to α ≡ 0 is then

Si,α = 1 + log(1 − Ri /(N + 1))

University of Zurich, EBPI UZH Master Biostatistics, HS 2025 STA402: Likelihood & Regression I Page 99
Model Comparison

How can we compare two (or more) models for

P(Y ≤ y | X = x)?

Idea: Compare out-of-sample log-likelihoods

– Estimate model parameters (use maximum likelihood or
whatever) for observations (y, x)i , i = 1, . . . , N, this gives
ĥN , β̂ N
– Obtain Ñ independent new observations
(y, x)i , i = N + 1, . . . , N + Ñ
– Compute out-of-sample log-likelihood

N+
XÑ
ℓ̃(ĥN , β̂ N ) = ℓi (ĥN , β̂ N )
i=N+1

University of Zurich, EBPI UZH Master Biostatistics, HS 2025 STA402: Likelihood & Regression I Page 100
Out-of-sample Log-likelihood

Note
– exp(ℓ̃(ĥN , β̂ N )) is a probability because the parameters
ĥN , β̂ N are fix and the responses in the “evaluation” sample
(y, x)i , i = N + 1, . . . , N + Ñ vary
– thus not restricted to specific models, any model one can
derive ℓi from can be compared
– also known as “log-score” in the scoring rules literature
– ℓ̃ is a “proper scoring rule”: The model assigning the largest
probability to the actual observed response “wins”

University of Zurich, EBPI UZH Master Biostatistics, HS 2025 STA402: Likelihood & Regression I Page 101
 Specific Regression Models

University of Zurich, EBPI UZH Master Biostatistics, HS 2025 STA402: Likelihood & Regression I Page 102
Binary (Logistic) Regression

Y ∈ {y1 , y2 }

P(Y ≤ y1 | x) = P(Y = y1 | x) = π(x) = FZ (ϑ1 + x̃ ⊤ β)

(NOTE: Model with +, not − shift term!)

FZ−1 (E(1(Y = y1 ) | x)) = FZ−1 (π(x)) = ϑ1 + x̃ ⊤ β

FZ−1 is called link function.

Y | x ∼ B(1, π(x))

The three model formulations are equivalent!

University of Zurich, EBPI UZH Master Biostatistics, HS 2025 STA402: Likelihood & Regression I Page 103
Common Binary Regression Models

FZ link FZ−1
expit(z) logit
Φ(z) probit
1 − exp(− exp(z)) cloglog
exp(− exp(−z)) loglog

University of Zurich, EBPI UZH Master Biostatistics, HS 2025 STA402: Likelihood & Regression I Page 104
Models for (Ordered) Polytomous Responses

Y ∈ {y1 , . . . , yK }
Simple Cumulative Model

P(Y ≤ yk | x) = πk (x) = FZ (ϑk + x̃ ⊤ β)

General Cumulative Model

P(Y ≤ yk | x) = πk (x) = FZ (ϑk + x̃ ⊤ β k )

(Note:
Y | X = x ∼ M(K, (π1 (x), π2 (x) − π1 (x), . . . , 1 − πK−1 (x))) is
multinomial model)

University of Zurich, EBPI UZH Master Biostatistics, HS 2025 STA402: Likelihood & Regression I Page 105
Models for Continuous Responses

Y∈R
Shift transformation model

P(Y ≤ y | x) = FZ (h(y) − x̃ ⊤ β)

Distribution regression

P(Y ≤ y | x) = FZ (h(y) − x̃ ⊤ β(y))

University of Zurich, EBPI UZH Master Biostatistics, HS 2025 STA402: Likelihood & Regression I Page 106
Continuous Outcome Logistic Regression

P(Y ≤ y | x) = expit(h(y) − x̃ ⊤ β)

University of Zurich, EBPI UZH Master Biostatistics, HS 2025 STA402: Likelihood & Regression I Page 107
Cox Proportional Hazards Model

P(Y ≤ y | x) = 1 − exp(− exp(h(y) − x̃ ⊤ β))

University of Zurich, EBPI UZH Master Biostatistics, HS 2025 STA402: Likelihood & Regression I Page 108
Anyones Master Thesis?

P(Y ≤ y | x) = exp(− exp(−(h(y) − x̃ ⊤ β)))

University of Zurich, EBPI UZH Master Biostatistics, HS 2025 STA402: Likelihood & Regression I Page 109
Normal Linear Regression Model

P(Y ≤ y | x) = Φ(h(y) − x̃ ⊤ β)

with h(y) = ξy − α is equivalent to

Y | x ∼ N(ξ(α + x̃ ⊤ β), ξ −2 )

University of Zurich, EBPI UZH Master Biostatistics, HS 2025 STA402: Likelihood & Regression I Page 110
Generalised Linear Regression Models

Exponential family model:

Y | x ∼ ExpFam(θ(x), ϕ)

with density (in “canonical form”)

 
yθ(x) − b(θ(x))
fY (y | θ(x), ϕ) = exp c(y, ϕ)
ϕ

University of Zurich, EBPI UZH Master Biostatistics, HS 2025 STA402: Likelihood & Regression I Page 111
Generalised Linear Regression Models

Y | x ∼ ExpFam(θ(x), ϕ)

with

g(E(Y | X = x)) = α + x̃ ⊤ β

is called Generalised Linear Regression Model (GLM) with link

function g : R → R and response function h = g−1 ; h = b′ is
called canonical (with θ(x) = α + x̃ ⊤ β).

University of Zurich, EBPI UZH Master Biostatistics, HS 2025 STA402: Likelihood & Regression I Page 112
Models for Count Data

Y ∈ {0, 1, . . . }

Poisson Generalised Linear Regression Model with canonical

log-link g = log

Y | X = x ∼ Po(λ(x))
λ(x) = exp(α + x̃ ⊤ β)

University of Zurich, EBPI UZH Master Biostatistics, HS 2025 STA402: Likelihood & Regression I Page 113
Models for Contingency Tables

Two variables YA and YB

YA/YB 1 ... J
1 Y11 Y1J
..
. Yij
I YI1 YIJ
Three models
– P(YA = i, YB = j)
– P(YA = i | YB = j) or P(YB = j | YA = i)
– P(Yij = yij | YA = i, yB = j)

University of Zurich, EBPI UZH Master Biostatistics, HS 2025 STA402: Likelihood & Regression I Page 114