STAT 5380

Advanced Mathematical Statistics I

(Lecture Notes  Spring 2024) 1

Alex Trindade
Department of Mathematics & Statistics
Texas Tech University

1Based primarily on TPE (Lehmann & Casella, 1998) and TSH (Lehmann & Romano, 2005).
Abstract. According to Efron & Hastie (2016):
Optimality theories  statements of best possible results  are marks of
maturity in applied mathematics. Classical statistics achieved two such the-
ories: for unbiased or asymptotically unbiased estimation, and for hypothesis
testing.
This course covers the details of this optimality story. You already know the basics
from your introductory mathematical statistics course: it is possible to nd optimal
(uniformly smallest variance) estimators if one restricts attention to the class of unbiased
estimators. For nite samples these are the UMVUEs, and for innte samples the
UMVUEs are MLEs (maximum likelihood estimators). But there are other forms of
optimality (if we do not restrict ourselves to the unbiased class), leading to:
• MREs: minimum risk equivariant estimators, minimize risk under the principle
of equivariance (invariance under location-scale transforms);
• Bayes estimators: minimize the Bayes risk, an integrated risk weighted by the
prior; and
• minimax estimators minimize the maximum risk.
The equivalent optimal hypothesis tests are uniformly most powerful (UMP) and UMP
unbiased (UMPU).
An important sobering message is that this optimal inference is infeasible in most
practical applications, and so one usually settles for the sub-optimal and automatic
MLE, and accompanying Likelihood Ratio, Wald, or Score test. All the details left out
in earlier courses (probability measure-theoretic and otherwise) are covered here.
The term classical statistics refers to the 20th century dominant theme whereby
the number of parameters to estimate is smaller than the available sample size (p ≪ n).
The 21st century bigdata era has reversed this situation, but at the moment there is no
comparable optimality theory when p ≫ n. . . . The course will bring you to this frontier
and provide you with the essential tools and knowledge to go beyond it.
 Contents

Chapter 1. Preliminaries 5
1.1. Conditional Expectation 5
1.2. Suciency 6
1.3. Exponential Families. 17
1.4. Convex Loss Function 28
1.5. Model Selection 28

Chapter 2. Unbiasedness 29
2.1. UMVU estimators. 29
2.2. Non-parametric families 37
2.3. The Information Inequality 40
2.4. Multiparameter Case 47

Chapter 3. Equivariance 56
3.1. Equivariance for Location family 56
3.2. The General Equivariant Framework 64
3.3. Location-Scale Families 68

Chapter 4. Average-Risk Optimality 76

4.1. Bayes Estimation 76
4.2. Minimax Estimation 87
4.3. Minimaxity and Admissibility in Exponential families 91
4.4. Shrinkage Estimators and Bigdata 98
4.5. Discussion (Efron & Hastie, 2016) 103

Chapter 5. Large Sample Theory 104

5.1. Convergence in Probability and Order in Probability 104
5.2. Convergence in Distribution 108
5.3. Asymptotic Comparisons (Pitman Eciency) 114
5.4. M-Estimation Theory 115
5.5. Example: AREs of Mean, Median, Trimmed Mean 118

Chapter 6. Maximum Likelihood Estimation 121

6.1. Consistency 121
6.2. Asymptotic Normality of the MLE 125
6.3. Asymptotic Optimality of the MLE 128
6.4. Asymptotic Eciency of Bayes Estimators 134
6.5. Discussion: MLE vs. Shrinkage (Efron & Hastie, 2016) 137

3
4 CONTENTS

Chapter 7. Optimal Testing Theory 138

7.1. Uniformly Most Powerful (UMP) Tests 138
7.2. The Neyman-Pearson Lemma 140
7.3. P-Values 143
7.4. Monotone Likelihood Ratio 143
7.5. Condence Bounds 148
7.6. Uniformly Most Powerful Unbiased (UMPU) Tests 152
7.7. Likelihood Ratio (LR), Wald, and Score Tests 161
7.8. Discussion 170

Bibliography 171
 CHAPTER 1

Preliminaries

1.1. Conditional Expectation

Denition. Let (X , A, P ) be a probability space. If X ∈ L1 (A, P ) and G is a sub-σ -eld

of A, then E(X|G) is a random variable such that

(i) E(X|G) ∈ G (i.e. is G measurable)

(ii) E(IG X) = E(IG E(X|G)), ∀G ∈ G

Construction. For X ≥ 0, µ(G) = E(IG X) is a measure on G and P (G) = 0 ⇒

µ(G) = 0, so by the Radon-Nikodym theorem there exists a G -measurable function
R
E(X|G) such that µ(G) = G E(X|G)dP , i.e.(ii) is satised. This shows the existence of
E(X + |G) and E(X − |G). Then we dene E(X|G) = E(X + |G) − E(X − |G).

Remark 1.1.1. (ii) generalizes to E(Y X) = E(Y E(X|G)) ∀Y ∈ G such that

E|Y X| < ∞.

The conditional probability of A given G is dened for all A ∈ A as P (A|G) = E(IA |G).

Remark 1.1.2. If X ∈ L2 (A, P ), then E(X|G) is the orthogonal projection in L2 (A, P )

of X onto the closed linear subspace L (G, P ) of L2 (A, P ) since
2

(i) E(X|G) ∈ L2 (G, P ) and

(ii) E(Y (X − E(X|G))) = 0, ∀Y ∈ L2 (G, P ).

Conditioning on a Statistic
Let X be a r.v. dened on (X , A, P ) with E|X| < ∞ and let T be a measurable function
(not necessarily real-valued) from (X , A) into (T , F).

T
→ (T , F, P T )
(X , A, P ) −

Such a T is called a statistic (and is not necessarily real-valued). The σ -eld of subsets
of X induced by T is

σ(T ) = {T −1 S, S ∈ F} = T −1 F

Definition 1.1.3. E(X|T ) ≡ E(X|σ(T ))

5
6 1. PRELIMINARIES

Recall that a real-valued function f on X is σ(T ) measurable ⇔ f = g ◦ T for some

F -measurable g on T, i.e. f (x) = g(T (x)) as shown below.
T g
X −−−→ T −−−→ R
.

This implies that E(X|T ) is expressible as E(X|T ) = h(T ) for some function h ∈ F
which is unique a.e. PT.
T h
X −−−→ T −−−→ R
Definition 1.1.4. E(X|t) ≡ h(t)
Example 1.1.5. Suppose (X, T ) has probability density p(x, t) w.r.t. Lebesgue mea-
2
sure on R and E|X| < ∞. Then E(X|σ(T )) = h(T ) where h(t) = E(X|T = t) =
R
R xp(x,t) dx I T (t), a.s. P T .
p(x,t) dx p (t)>0

PROOF
(i) R.S. is Borel measurable in t (by Fubini)
(ii) G ∈ σ(T ) ⇒ G = T −1 F for some F ∈ F ⇒ IG = IF (T )
Z
∴ E(IG E(X|σ(T ))) = E(IG X) = IG X dP
Z Z Z
= xIF (t)p(x, t) dxdt = IF (t)h(t)pT (t) dt
= E[IF (T )h(T )] = E[IG h(T )]
□

Properties of Conditional Expectation

If T is a statistic, X is the identity function on X and fn , f, g are integrable, then

(i) E[af (X) + bg(X)|T ] = aE[f (X)|T ] + bE[g(X)|T ] a.s.

(ii) a ≤ f (X) ≤ b a.s. ⇒ a ≤ E[f (X)|T ] ≤ b a.s.
(iii) |fn | ≤ g, fn (x) → f (x) a.s. ⇒ E[fn (X)|T ] → E[f (X)|T ] a.s.
(iv) E[E(f (X)|T )] = Ef (X).
(v) If E|h(T )f (X)| < ∞, then E[h(T )f (X)|T ] = h(T )E[f (X)|T ] a.s.
(vi) If G1 and G2 are sub-σ -elds of G with G1 ⊂ G2 , then E[E(X|G1 )|G2 ] = E(X|G2 ).

1.2. Suciency

Set up
X : random observable quantity (the identity function on (X , A, P))
X : sample space, the set of possible values of X
A: σ -algebra of subsets of X
 1.2. SUFFICIENCY 7

P : {Pθ , θ ∈ Ω} is a family of probability measures on A (distributions of X)

T: X → T is an A/F measurable function and T (X) is called a statistic.

X T
probability space (X , A, P) −−−→ sample space (X , A, P) −−−→ (T , F, P T )

We adopt this notation because sometimes we wish to talk about T (X(·)) the random
variable and sometimes about T (X(x)) = T (x), a particular element of T . We shall also
use the notation P (A|T (x)) for P (A|T = T (x)) and P (A|T ) for the random variable
P (A|T (·)) on X .
Definition 1.2.1. The statistic sucient for θ(or P) i the conditional distribution
T is
of X given T = t is independent of θ for all t, i.e. there exists an F measurable P (A|T = ·)
T
such that P (A|T = t) = Pθ (A|T = t) a.s. Pθ for all A ∈ A and all θ ∈ Ω.

Example 1.2.2.

X = (X1 , . . . , Xn ) iid with pdf fθ (x) w.r.t. dx

P = Pθ (dx1 , . . . , dxn ) = fθ (x1 ) · · · fθ (xn ) dx1 · · · dxn
th
T (X) = (X(1) , . . . , X(n) ) where X(i) is the i order statistic.

The probability mass function of X

T = t is
given

X|T =t δt (x(1) ) · · · δtn (x(n) )

pθ (x|t) = 1
n!
1
i.e. it assigns point mass to each x such that x(1) = t1 , · · · , x(n) = tn . This is
n!
independent of θ , indicating that T contains all the information about θ contained in the
sample.

The Factorization Criterion

Definition 1.2.3. A family of probability measure's P = {Pθ : θ ∈ Ω} is equivalent to
a p.m. λ if
λ (A) = 0 ⇐⇒ Pθ (A) = 0 ∀θ ∈ Ω.

We also say that P is dominated by a σ -nite measure µ on (X , A) if

Pθ ≪ µ for all θ ∈ Ω.

It is clear that equivalence to λ implies domination by λ.

Theorem 1.2.4. Let P be dominated by a p.m. λ where
X∞ X
λ= ci Pθ i (ci ≥ 0, ci = 1).
i=0

Then the statistic T (with range (T , F)) is sucient for P ⇐⇒ there exists an F -
measurable function gθ (·) such that
dPθ (x) = gθ (T (x)) dλ(x) ∀θ ∈ Ω.
8 1. PRELIMINARIES

Proof. (⇒) Suppose T is sucient for P. Then

Pθ (A|T (x)) = P (A|T (x)) ∀θ.

Throughout this part of the proof X will denote the indicator function of a subset of X.
The preceding equality then implies that

Eθ (X|T ) = E(X|T ) ∀X ∈ A, ∀θ.

Hence for all θ ∈ Ω, X ∈ A, G ∈ σ(T ), we have

Eθ (IG E(X|T )) = Eθ (Eθ (IG X|T )) = Eθ (IG X).

Set θ = θi , multiply by ci and sum over i = 0, 1, 2, . . . , to get

Eλ (IG E(X|T )) = Eλ (IG X) ∀X ∈ A, ∀G ∈ σ(T ).

This implies that E(X|T ) = Eλ (X|T ) ∀X ∈ A, and hence

Eθ (X|T ) = E(X|T ) = Eλ (X|T ) ∀X ∈ A, ∀θ.

Now dene gθ (T (·)) to be the Radon-Nikodym derivative of Pθ with respect to λ, with

both regarded as measures on σ(T ). We know this exists since λ dominates every Pθ .
We also know it is σ(T ) measurable, so it can be written in the form gθ (T (·)), and we
know that Eθ (X) = Eλ (gθ (T )X) for all X ∈ σ(T ). We need to establish however that
this last relation holds for all X ∈ A. We do this as follows.

X ∈ A ⇒ Eθ (X) = Eθ [E(X|T )]
= Eλ [gθ (T )E(X|T )]
= Eλ [E(gθ (T )X|T )]
= Eλ [Eλ (gθ (T )X|T )]
= Eλ [gθ (T )X].
dPθ
This shows that gθ (T (x)) = dλ
(x) when Pθ and λ are regarded as measures on A.

(⇐) Suppose that for each θ, dP

dλ
θ
(x) = gθ (T (x)) for some gθ . We shall then show that
the conditional probability Pλ (A|t) is a version of Pθ (A|t) ∀θ .

Z Z
A ∈ A, G ∈ σ(T ) ⇒ IA dPθ = Pθ (A|T ) dPθ
G G
Z
= Pθ (A|T )gθ (T ) dλ
G
 1.2. SUFFICIENCY 9

and
Z Z
IA dPθ = IA gθ (T ) dλ
G
ZG
= Eλ [IA gθ (T )|T ] dλ
G
Z
= Eλ [IA |T ]gθ (T ) dλ
G
⇒ Pθ (A|T )gθ (T ) = Eλ (IA |T )gθ (T ) a.s. λ
and hence a.s. Pθ ∀θ. Also gθ (T ) ̸= 0, a.s. Pθ , since dPθ = gθ (T ) dλ. Hence Pθ (A|T ) =
Eλ (IA |T ) = Pλ (A|T ) a.s. Pθ and the R.S. is independent of θ. □
Theorem 1.2.5. (Theorem A.4.2 in appendix of TSH) If P P = {Pθ , θ ∈ Ω} is dominated
by a σ -nite measure µ, then it is equivalent to λ = P ∞ i=0 ci Pθi for some countable
subcollection Pθi ∈ P, i = 0, 1, 2, . . . , with ci ≥ 0 and ci = 1.

Proof. µ is σ−nite, ⇒ ∃An ∈ A with A1 , A2 , . . . disjoint, and ∪Ai = X such that

0 < µ(Ai ) < ∞, i = 1, 2, . . .. Set
∞
X µ(A ∩ Ai )
µ∗ (A) =
i=1
2i µ(Ai )
Then, µ∗ is a probability measure equivalent to µ. Hence we can assume without loss of
generality that the dominating measure µ is a probability measure Let

dPθ
fθ =
dµ
and set
Sθ = {x : fθ (x) > 0}
Then

(1.2.1) Pθ (A) = Pθ (A ∩ Sθ ) = 0 i µ(A ∩ Sθ ) = 0.

(Since Pθ ≪ µ and since µ(A ∩ Sθ ) > 0, fθ > 0 on A ∩ Sθ ⇒ Pθ (A ∩ Sθ ) > 0.) A set
A ∈ A is a kernel if A ⊆ Sθ for some θ; a nite or countable union of kernels is called a
chain. Set
α = sup µ(C)
chains C
∞
Then α = µ(C) for some chain C = ∪n=1 An , An ⊆ Sθn . (since ∃ {Cn } such that µ(Cn ) ↑α
and for this sequence µ(∪Cn ) = α.) P∞ 1
It follows from the following Lemma that P is dominated by λ(·) = n=1 2n Pθn (·). Since

λ(A) = 0 ⇒ Pθn (A) = 0 ∀n

⇒ Pθ (A) = 0 ∀θ (by the Lemma),
it is obvious that
Pθ (A) = 0 ∀ θ ⇒ λ(A) = 0
10 1. PRELIMINARIES

P∞ 1
Hence P is equivalent to λ(·) = n=1 2n Pθn (·). □
Lemma 1.2.6. If {θn } is the sequence used in the construction of C , then {Pθ , θ ∈ Ω} is
dominated by {Pθn , n = 1, 2, . . .}, i.e.
Pθn (A) = 0 ∀n ⇒ Pθ (A) = 0 ∀θ

Proof.

Pθn (A) = 0 ∀n ⇒ µ(A ∩ Sθn ) = 0 ∀n (by 1.2.1)

(C⊆ ∪Sθn )
⇒ µ(A ∩ C) = 0
(Pθ ≪µ)
⇒ Pθ (A ∩ C) = 0 ∀θ
If Pθ (A) > 0 for some θ then, since Pθ (A) = Pθ (A ∩ C) + Pθ (A ∩ C c ),
Pθ (A ∩ C c ) = Pθ (A ∩ C c ∩ Sθ ) > 0
⇒A ∩ C c ∩ Sθ is a kernel disjoint from C

⇒C ∪ (A ∩ C c ∩ Sθ ) is a chain with µ > α, (Pθ (A) > 0 ⇒ µ(A) > 0)

contradicting the denition of α.
Hence, Pθ (A) = 0 ∀ θ. □
Theorem 1.2.7. The Factorization Theorem
Let µ be a σ -nite measure which dominates P = {Pθ : θ ∈ Ω} and let
dPθ
pθ = .
dµ
Then the statistic T is sucient for P if and only if there exists a non negative F -
measurable function gθ : T → R and an A-measurable function h : X → R such that
(1.2.2) pθ (x) = gθ (T (x)) h (x) a.e. µ.

Proof. By theorem 1.2.5, P is equivalent to

X X
λ= ci Pθi , where ci ≥ 0, ci = 1.
i i
If T is sucient for P,
dPθ (x) dPθ (x) dλ (x)
pθ (x) = = ·
dµ (x) dλ (x) dµ (x)
= gθ (T (x)) h (x) by theorem 1.2.4.

On the other hand, if equation (1.2.2) holds,

X X
dλ (x) = ci dPθi (x) = ci pθi (x) dµ(x)
∞
X
= ci gθi (T (x)) h (x) dµ (x)
i=1
(1.2.3) = K (T (x)) h (x) dµ (x) .
 1.2. SUFFICIENCY 11

Thus,

dPθ (x) = pθ (x) dµ (x) by the denition of pθ (x)

gθ (T (x)) h (x)
= dλ (x) by equations (1.2.2) and (1.2.3)
K (T (x)) h (x)
= g̃θ (T (x)) dλ (x) where g̃θ (T (x)) := 0 if K (T (x)) = 0.

Hence T is sucient for P by theorem 1.2.4. □

Remark 1.2.8. If fθ (x) is the density of X with respect to Lebesgue measure then T is
sucient for P i

fθ (x) = gθ (T (x)) h (x)

where h is independent of θ.

Example 1.2.9. Let X1 , X2 , . . . , Xn be iid N (µ, σ 2 ) , µ ∈ R, σ > 0, and write X =

(X1 , X2 , · · · , Xn ). A σ -nite dominating measure on B n is Lebesgue measure with
n
!
1 −1 X 2 µ X nµ2
pµ,σ2 (x) = √
n exp x + xi − 2
σ 2π 2σ 2 1 i σ 2 2σ
X X 
= gµ,σ2 xi , x2i .

T (X) = ( Xi , Xi2 )
P P
Therefore is sucient for P = {Pµ,σ2 } .

T ∗ (X) = X̄, S 2 is also sucient for P = {Pµ,σ2 },

Remark 1.2.10. since

X X 
∗
x2i = gµ,σ 2


gµ,σ2 xi , 2 x̄, S

T and T∗ are equivalent in the following sense.

Definition 1.2.11. Two statistics T and S are equivalent if they induce the same σ-
algebra up to P -null sets. i.e. if there exists a P -null set N and functions f and g such
that

T (x) = f (S (x)) and S (x) = g (T (x)) for all x ∈ N c.

Example 1.2.12. Let X1 , . . . , X n be iid U (0, θ), θ > 0 and X = (X1 , . . . , Xn ).

n
1 Y
pθ (x) = n I[0,∞) (xi )I(−∞,θ] (xi )
θ 1
1
=
I[0,∞) (x(1) )I(−∞,θ] (x(n) )
θn
= gθ (x(n) )h(x)
⇒ T (X) = X(n) is sucient for θ.
12 1. PRELIMINARIES

Example 1.2.13. X1 , . . . , X n iid N (0, σ 2 ), Ω = {σ 2 : σ 2 > 0}. Dene

T1 (X) = (X1 , . . . , Xn )
T2 (X) = (X12 , . . . , Xn2 )
T3 (X) = (X12 + · · · + Xm 2 2
, Xm+1 + · · · + Xn2 )
T4 (X) = X12 + · · · + Xn2
n
1 1 X 2
pθ (x) = √ exp (− 2 Xi )
(σ 2π)n 2σ 1

Each Ti (X) σ(T4 ) ⊆ σ(T3 ) ⊆ σ(T2 ) ⊆ σ(T1 ).

is sucient. However
(since functions of T4 are functions of T3 , functions of T3 are functions of T2 and functions
of T2 are functions of T1 .)

Remark 1.2.14. If T is sucient for θ and T = H (S) where S is some statistic, then S
is also sucient since

pθ (x) = gθ (T (x)) h (x) = gθ (H (S (x)) h (x)

S H
Since σ (T ) = S −1 H −1 BT ⊂ S −1 BS ((X , A) −
→ (S, BS ) −
→ (T , BT )), T provides a greater
reduction of the data than S , strictly greater unless H is one to one, in which case S and
T are equivalent.
Definition 1.2.15. T is a minimal sucient statistic, if for any sucient statistic S,
there exists a measurable function H such that

T = H (S) a.s. P.
Theorem 1.2.16. If P is dominated by a σ -nite measure µ, then the statistic U is
sucient i for every xed θ and θ0 , the ratio of the densities pθ and pθ0 with respect to
µ, dened to be 1 when both densities are zero, satises
pθ (x)
= fθ,θ0 (U (x)) a.s. P for some measurable fθ,θ0 .
pθ0 (x)

Proof. HW problem (TPE Ch 1 Problem 6.6). □

Theorem 1.2.17. Let P be a nite family with densities {p0, p1 , . . . , pk }, all having the
same support (i.e. S = {x : pi (x) > 0} is independent of i). Then
 
p1 (x) p2 (x) pk (x)
T (x) = , ,··· ,
p0 (x) p0 (x) p0 (x)
is minimal sucient. (Also true for a countable collection of densities with no change in
the proof.)

pi (x)
Proof. First T is sucient by theorem (1.2.16) since
pj (x)
is a function of T (x) for

all i and j (need common support here.) If U is a sucient statistic then by theorem
 1.2. SUFFICIENCY 13

(1.2.16),

pi (x)
p0 (x)
is a function of U for each i
⇒ T is a function of U
⇒ T is minimal sucient.

□
Remark 1.2.18. The theorem 1.2.17 extends to uncountable collections under further
conditions. It also extends to countable collections without common support (Prob. 1.6.11).

Theorem 1.2.19. Let P be a family with common support and suppose P0 ⊆ P . If T is

minimal sucient for P0 and sucient for P , then T is minimal sucient for P .

Proof.

U is sucient for P⇒U is sucient for P0 by Definition 1.2.1.

T is minimal sucient for P0 ⇒ T (x) = H(U (x)) a.s. P0 .
But since P has common support, T (x) = H(U (x)) a.s. P.
□

Note the following points.

(1) Minimal sucient statistics for uncountable families P can often be obtained by
combining the above theorems.
(2) Minimal sucient statistics exist under weak assumptions (but not always). In
n n
particular they exist if (X , A) = (R , B ) and P is dominated by a σ -nite
measure.
(3) A generalization of the above results for establishing minimality, which accom-
modates uncountable families without common support, is Theorem 1.2.20.

Theorem 1.2.20. Let P = {pθ (x) : θ ∈ Θ} be a family of densities dominated by a

σ -nite measure. If there exists a measurable function T : X → T such that T (x) = T (y)
if and only if y ∈ D(x), where
D(x) = {y ∈ X : pθ (y) = pθ (x)h(x, y), ∀θ and some h(x, y) > 0} ,
then T (X) is a minimal sucient statistic.

Proof. Schervish (1995), Theorem 2.29. □

Example 1.2.21. P0 : (X1 , . . . , Xn ) iid N (θ, 1), θ ∈ {θ0 , θ1 }.
P : (X1 , . . . , Xn ) iid N (θ, 1), θ ∈ R.
14 1. PRELIMINARIES

 i
pθ1 (x) 1 hX 2
X
2
= exp − (xi − θ1 ) − (xi − θ0 )
pθ0 (x) 2
 i
1 hX 2 2
= exp − 2xi (θ0 − θ1 ) + nθ1 − nθ0
2
This is a function of x̄, hence X̄ is minimal sucient for P0 by Theorem 1.2.17. Since X̄
is sucient for P (by the factorization theorem), X̄ is minimal sucient for P.
Example 1.2.22. P : (X1 , . . . , Xn ) iid U (0, θ), θ > 0.
To show that X(n) is minimal sucient via Theorem 1.2.20, suppose that

pθ (x) θ−n I(0,θ) (x(n) ) I(0,θ) (x(n) )

= −n = = h(x, y), ∀θ.
pθ (y) θ I(0,θ) (y(n) ) I(0,θ) (y(n) )
This is true if and only if x(n) = y(n) , in which case h(x, y) = 1 and D(x) = {y : y(n) =
x(n) }, whence T = X(n) is minimal sucient.

Example 1.2.23. Logistic

We'll show the order statistics are minimal sucient via Theorems 1.2.17 and 1.2.19 (but
this could also be be accomplished via Theorem 1.2.20).

P : (X1 , . . . , Xn ) iid L(θ, 1), θ ∈ R.

P0 : (X1 , . . . , Xn ) iid L(θ, 1), θ ∈ {0, θ1 , . . . , θn }.
P
exp [− (xi − θ)]
pθ (x) = Qn 2,
i=1 {1 + exp [−(xi − θ)]}
so T = (T1 (X), . . . , Tn (X)) is minimal sufficient,
where
n
pθi (x) nθi
Y (1 + e−xj )2
Ti (x) = =e .
p0 (x) j=1
(1 + e−(xj −θi ) )2
We will show that T (X) is equivalent to (X(1) , . . . , X(n) ), by showing that

T (x) = T (y) ⇔ x(1) = y(1) , · · · , x(n) = y(n) .

Proof. (⇐) Obvious from the expression for Ti (x).

(⇒) Suppose that Ti (x) = Ti (y) for i = 1, 2, . . . , n,

n n
Y (1 + e−xj )2 Y (1 + e−yj )2
i.e.
−(xj −θi ) )2
= −(yj −θi ) )2
, i = 1, . . . , n,
j=1
(1 + e j=1
(1 + e
n n
Y 1 + uj ω Y 1 + vj ω
i.e. = , ω = ω1 , . . . , ωn ,
j=1
1 + uj j=1
1 + vj
 1.2. SUFFICIENCY 15

where uj = e−xj , vj = e−yj and ωi = eθi . Here we have two polynomials in ω of degree
n which are equal for n + 1 distinct values, 1, ω1 , . . . , ωn , of ω and hence for all ω .
Yn n
Y
ω=0⇒ (1 + uj ) = (1 + vj )
j=1 j=1
n
Y Yn
∴ (1 + uj ω) = (1 + vj ω) ∀ω
j=1 j=1
∴ the zero sets of both these polynomials are the same

∴ x and y have the same order statistics.

By theorem 1.2.17, the order statistics are therefore minimal sucient for P0 . They are
also sucient for P, so by theorem 1.2.19, the order statistics are minimal sucient for
P. There is not much reduction possible here! This is fairly typical of location families,
the normal, uniform and exponential distributions providing happy exceptions. □

Ancillarity
Definition 1.2.24. A statistic V is said to be ancillary for P if the distribution, PθV , of
V does not depend on θ. It is called rst order ancillary if Eθ V is independent of θ .

Example 1.2.25. In Example 1.2.23, X(2) − X(1) Y1 = X1 − θ, . . . , Yn =

is ancillary since
Xn − θ are iid P0 (the standard member of the family with θ = 0) and X(2) − X(1) =
Y(2) − Y(1) .
Example 1.2.26.

P : (X1 , . . . , Xn ) iid N (θ, 1), θ ∈ R.

X
S2 = (Xi − X̄)2 is ancillary

since
X
S2 = (Yi − Ȳ )2 where Yi = Xi − θ, i = 1, 2, . . . , are iid N (0, 1).
Remark 1.2.27. Ancillary statistics by themselves contain no information about θ, how-
ever minimal sucient statistics may contain ancillary components. For example, in
∗
1.2.23, T = (X(1) , · · · , X(n) ) is equivalent to T = (X(1) , X(2) − X(1) , · · · , X(n) − X(1) ),
whose last (n − 1) components are ancillary. You can't drop them as X(1) is not even
sucient.

Complete Statistic

A sucient statistic should bring about the best reduction of the data if it contains as
little ancillary material as possible. This suggests requiring that no non-constant function
of T be ancillary, or not even rst order ancillary, i.e. that

Eθ f (T ) = c for all θ∈Ω ⇒ f (T ) = c a.s. P

16 1. PRELIMINARIES

or equivalently that

Eθ f (T ) = 0 for all θ∈Ω ⇒ f (T ) = 0 a.s. P.

Definition 1.2.28. A statistic T is complete if

(1.2.4) Eθ f (T ) = 0 for all θ∈Ω ⇒ f (T ) = 0 a.s. P

T is said to be boundedly complete if equation (1.2.4) holds for all bounded measurable
functions f.
Since complete sucient statistics are intended to give a good reduction of the data, it is
not unreasonable to expect them to be minimal. We shall prove a slightly weaker result.

Theorem 1.2.29. Let U be a complete sucient statistic. If there exists a minimal

sucient statistic, then U is minimal sucient.

Proof. Let T be a minimal sucient statistic and let ψ be a bounded measurable

function. We will show that

ψ(U ) ∈ σ(T ) by showing that E(ψ(U )|T ) = ψ(U ) a.s.

Now

E(ψ(U )|T ) = g(U ) for some measurable g since T is minimal and U is sufficient.
Let h(U ) = E(ψ(U )|T ) − ψ(U ), then Eθ h(U ) = 0 ∀ θ so h(U ) = 0 a.s. P since U is
complete. Hence ψ(U ) = E(ψ(U )|T ) ∈ σ(T ). Hence U -measurable bounded functions
are T -measurable, i.e. σ(U ) ⊂ σ(T ), i.e. U is minimal sucient. □
Remark 1.2.30.

(1) If P is dominated by a σ -nite measure and (X , A) = (Rn , B n ), the existence of

a minimal sucient statistic does not need to be assumed.
(2) A minimal sucient statistic is not necessarily complete. See the next example.

Example 1.2.31.

P = {N (θ, θ2 ), θ > 0}
1 1 (x−θ)
2
1 1 x 2
pθ (x) = √ e− 2 θ2 = √ e− 2 ( θ −1)
θ 2π θ 2π
The single observation X is minimal sucient but not complete since

Eθ [I(0,∞) (X) − Φ(1)] = Pθ (X > 0) − Φ(1) = 0 ∀ θ

however Pθ (I(0,∞) (X) − Φ(1) = 0) = 0 ∀θ.
Theorem 1.2.32. (Basu's theorem) If T is complete and sucient for P , then any
ancillary statistic is independent of T .
 1.3. EXPONENTIAL FAMILIES. 17

Proof. If S is ancillary, then Pθ (S ∈ B) = pB , independent of θ.

Suciency of T ⇒ Pθ (S ∈ B|T ) = h(T ), independent of θ.
∴Eθ (h(T ) − pB ) = 0
⇒h(T ) = pB a.s. P by completeness

⇒S is independent of T
□

1.3. Exponential Families.

Definition 1.3.1. A family of probability measure's {Pθ : θ ∈ Ω} is said to be an s-

parameter exponential family if there exists a σ -nite measure µ such that

s
!
dPθ (x) X
pθ (x) = = exp ηi (θ) Ti (x) − B (θ) h (x) ,
dµ (x) 1

where ηi , Ti and B are real-valued.

Remark 1.3.2.

(1) Pθ , θ ∈ Ω {x : pθ (x) > 0} is independent of θ).

are equivalent (since
(2) The factorization theorem implies that T = (T1 , · · · , Ts ) is sucient.
Pn
(3) If we observe X1 , . . . , Xn , iid with marginal distributions Pθ then j=1 T (Xj )
is sucient for θ .

Theorem 1.3.3. If {1, η1 , . . . , ηs } is LI, then T = (T1 , . . . , Ts ) is minimal sucient.

(Linear independence of {1, η1 , . . . , ηs } means c1 η1 (θ) + · · · + cs ηs (θ) + d = 0 ∀θ ⇒ c1 =
· · · = cs = d = 0. Equivalently we can say that {ηi } is anely independent or AI
since the set of points {(η1 (θ), . . . , ηs (θ)), θ ∈ Ω} then lie in a proper ane subspace of
Rs .)

Proof. Fix θ0 ∈ Ω and consider

( s )
dPθ pθ (x) X
(1.3.1) (x) = = exp {B(θ0 ) − B(θ)} exp (ηi (θ) − ηi (θ0 )) Ti (x) .
dPθ0 pθ0 (x) 1

If {1, η1 , . . . , ηs } is LI then so is {1, η1 − η1 (θ0 ), . . . , ηs − ηs (θ0 )}.

Set S = {(η1 (θ) − η1 (θ0 ), . . . , ηs (θ) − ηs (θ0 )), θ ∈ Ω} ⊆ Rs . Then span(S) is a linear
subspace of Rs .

If dim(span(S)) < s, then there exists a non-zero vector v = (v1 , . . . , vs ) s.t.

v1 (η1 (θ) − η1 (θ0 )) + · · · + vs (ηs (θ) − ηs (θ0 )) = 0 ∀θ

18 1. PRELIMINARIES

contradicting the linear independence of {1, ηi − ηi (θ0 )}. Hence

dim(span(S)) = s i.e. ∃ θ1 , . . . , θs ∈ Ω s.t.

(1.3.2) {(η1 (θi ) − η1 (θ0 ), · · · , ηs (θi ) − ηs (θ0 )), i = 1, · · · , s} is LI.

From 1.3.1,

s
X pθ (x)
(ηj (θi ) − ηj (θ0 ))Tj (x) = ln i + (B(θi ) − B(θ0 )), i = 1, . . . , s.
j=1
p θ0 (x)
s
Since the matrix [ηj (θi ) − ηj (θ0 )]i,j=1 is non-singular, Tj (x) can be expressed uniquely in
pθ (x)
terms of ln i , i = 1, . . . , s.
pθ0 (x)
pθi (x)
But
pθ0 (x)
, i = 1, . . . , s is minimal sucient for P0 = {Pθj , j = 0, 1, · · · , s} by theorem
1.2.17. Hence T is minimal sucient by theorem 1.2.19. □
Example 1.3.4.
r
θ 1 θ
pθ (x) = exp{− θx2 + θx − }.
2π 2 2
1
η1 (θ) = − θ, η2 (θ) = θ, T (x) = (x2 , x) is sucient but not minimal
2 r
θ 1
since rewriting the model as pθ (x) = exp{− θ(x − 1)2 }, we see that
2π 2
T ∗ (x) = (x − 1)2 is minimal sucient.
Remark 1.3.5. The exponential family can always be rewritten in such a way that the
functions {Ti } and {ηi } are AI. If there exist constants c1 , . . . , cs , d, not all zero, such
that

c1 T1 (x) + · · · + cs Ts (x) = d a.s. P

then one of the Ti 's can be expressed in terms of the others (or is constant). After
reducing the number of functions Ti as far as possible, the same can be done with their
coecients until the new functions {Ti } and {ηi } are AI.
Definition 1.3.6. ( Order of the exponential family.) If the functions {Ti , i = 1, . . . , s}
on X and {ηi , i = 1, . . . , s} on Ω are both AI, then s is the order of the exponential
family
s
!
dPθ X
pθ (x) = (x) = exp ηi (θ) Ti (x) − B (θ) h (x) .
dµ 1

Proposition 1.3.7. The order is well-dened.

Proof. We shall show that

s + 1 = dim(V )
 1.3. EXPONENTIAL FAMILIES. 19

dPθ
where V is the set of functions on X dened by V = span{1, ln dP θ0
(·), θ ∈ Ω} (inde-

pendent of the dominating measure µ and the choice of {ηi }, {Ti }).

s
dPθ X
ln (x) = (ηi (θ) − ηi (θ0 ))Ti (x) + B(θ0 ) − B(θ)
dPθ0 i=1

so that

V ⊆ span{1, Ti (·), i = 1, . . . , s} ∴ dim(V ) ≤ s + 1

On the other hand, since {1, ηi , i = 1, . . . , s} is LI, each Tj (x) can be expressed as a
dP
linear combination of 1, ln dPθθi (x), i = 1, . . . , s, as in the proof of the previous theorem,
0

∴ span{1, Ti (·), i = 1, . . . , s} ⊆ V
∴ s + 1 ≤ dim(V )
□
Definition 1.3.8. ( Canonical Form) For any s-parameter exponential family (not
necessarily of order s) we can view the vector η(θ) = (η1 (θ), . . . , ηs (θ))′ as the parameter
rather than θ. Then the density with respect to µ can be rewritten as

s
X
p(x, η) = exp[ ηi Ti (x) − A(η)]h(x), η ∈ η(Ω).
i=1

Since p(·, η) is a probability density with respect to µ,

Z Ps
(1.3.3) eA(η) = e 1 ηi Ti (x)
h(x)dµ(x).

Definition 1.3.9. ( The Natural Parameter Set) This is a possibly larger set than
{η(θ), θ ∈ Ω}. It is the set of all s-vectors for which, by suitable choice of A(η), p(·, η)
can be a probability density, i.e.
R Ps
N = {η = (η1 , · · · , ηs ) ⊆ Rs : e 1 ηi Ti (x)
h(x)dµ(x) < ∞}
Theorem 1.3.10. N is a convex set, and A(η) is a convex function.

Proof. Suppose α = (α1 , . . . , αs ) and β = (β1 , . . . , βs ) ∈ N . Then,

Z Ps Ps
ep 1 αi Ti (x)+(1−p)
h(x) dµ(x)
1 βi Ti (x)

Z P p Z P 1−p
αi Ti (x) βi Ti (x)
≤ e h(x) dµ(x) · e h(x) dµ(x) (Holder's Inequality)

<∞
The convexity of A(η) follows similarly (Bickel & Doksum, 2015, Theom 1.6.3). □
20 1. PRELIMINARIES

Theorem 1.3.11. T = (T1 , · · · , Ts ) has density

pη (t) = exp (η · t − A (η))
relative to ν = µ̃ ◦ T −1 where dµ̃ (x) = h (x) dµ (x). (Note: this introduces the Jacobian
term k(t)dν in the density over Euclidean s-space, where k(t) = h(T −1 (t)).)

Proof. If f :T → R is a bounded measurable function,

Z
Ef (T ) = f (T (x))eη·T (x) e−A(η) dµ̃(x)
Z
= f (t)eη·t e−A(η) dµ̃ ◦ T −1 (t)

□
Definition 1.3.12. The family of densities

pη (t) = exp (η · t − A (η)) , η ∈ η(Ω),

n is called an s-dimensional or s-parameter standard exponential family. (Dened
s
on R , not X .)

Theorem 1.3.13. Let {pη (x)} be the s-parameter exponential family,

s
!
X
pη (x) = exp ηi (θ) Ti (x) − B (θ) h (x)), η ∈ η(Ω),
i=1

and suppose
Z Ps
ηj Tj (x)
(1.3.4) ϕ (x) e 1 dµ (x)

exists and is nite for some ϕ and all ηj = aj + ibj such that a ∈ N (=natural parameter
space). Then
Ps
ϕ (x) e 1 ηj Tj (x) dµ (x) is an analytic function of each ηi on {η : ℜ (η) ∈ int (N )}
R
(i)
and Ps
(ii) the derivative of all orders with respect to the ηi 's of ϕ (x) e 1 ηj Tj (x) dµ (x) can
R
be computed by dierentiating under the integral sign.

Proof. Let a0 = (a01 , . . . , a0s ) be in int(N ) and let η10 = a01 + ib01 . Then
Ps
ηj Tj (x)
ϕ(x)e 2 = h1 (x) − h2 (x) + i(h3 (x) − h4 (x))
where h1 and h2 are the positive and negative parts of the real part and h3 and h4 are
the positive and
Ps negative parts of the imaginary part.
R
Then ϕ (x) e 1 ηj Tj (x) dµ (x) can be expressed as
Z Z Z Z
η1 T1 (x) η1 T1 (x) η1 T1 (x)
e dµ1 (x) − e dµ2 (x) + i e dµ3 (x) − i eη1 T1 (x) dµ4 (x),
 1.3. EXPONENTIAL FAMILIES. 21

where dµi (x) = hi (x) dµ(x), i = 1, . . . , 4. Hence it suces to prove (i) and (ii) for
Z
ψ(η1 ) = eη1 T1 (x) dµ(x).

Since a0 ∈ int(N ), there exists δ > 0 s.t. ψ(η1 ) exists and is nite for all η1 with
|a1 − a01 | < δ . Now consider the dierence quotient
(η1 −η10 )T1 (x)
ψ(η1 ) − ψ(η10 ) −1
Z
η10 T1 (x) e
\
(∗) 0
= e 0
µ(dx) with |η1 − η10 | < δ/2.
η1 − η1 η1 − η1
Observe that
∞ ∞
X (zt)j X |zt|j
zt
|e − 1| = | | ≤ = e|zt| − 1
1
j! 1
j!
≤ |zt|e|zt|
ezt − 1
⇒| | ≤ |t|e|zt|
z
0 δ
The integrand in (*) is therefore bounded in absolute value by |T1 (x)|e(a1 + 2 )|T1 (x)| , where
R 0 δ
a01 = Re(η10 ) and |T1 (x)|e(a1 + 2 )|T1 (x)| µ(dx) < ∞ since
 δ 0 3δ )T

 |T1 |e− 4 T1 e|(a1 + 4
{z }
1
if T1 > 0
 | {z }
(a01 + 2δ )|T1 |
|T1 |e = bounded integrable
 z }| { z }| {
δ 0 δ

|T1 |e 4 T1 e(a1 + 4 )T1

if T1 < 0
(independent of η1 ).

Letting η1 → η10 in (*) and using the dominated convergence theorem therefore gives
Z
′ 0
(1.3.5) ϕ (η10 ) = T1 (x)eη1 T1 (x) µ(dx),

where the integral exists and is nite ∀η10 which is the rst component of some η 0 for which Re(η
0
)∈
N.

Applying the same argument to (1.3.5) which we applied to (1.3.4) ⇒ existence of all
derivatives ⇒ (i) and (ii). □

Theorem 1.3.14. For an exponential family of order s in canonical form and η ∈ int (N ),
where N is the natural parameter space,
 T
∂A ∂A ∂A
(i) Eη (T ) =
∂η
= ∂η1
, · · · , ∂ηs
, and
h i s
∂2A 2A
(ii) Covη (T ) =
∂η∂η T
= ∂η∂i ∂η j
.
i,j=1
22 1. PRELIMINARIES

Proof. From theorem 1.3.11

Z Z
A(η) η·t
e = e ν(dt) = eη·T (x) h(x)µ(dx)

so

∂A A(η)
R
(i)
∂ηi
e = Ti (x)eη·T (x) h(x)µ(dx)
∂A
whence Eη Ti = .
∂ηi
∂ 2 A A(η) ∂A ∂A A(η)
R
(ii)
∂ηi ∂ηj
e + ∂ηi ∂ηj e = Ti (x)Tj (x)eη·T (x) h(x)µ(dx)
∂2A
i.e.
∂ηi ∂ηj
= Eη (Ti Tj ) − Eη (Ti )Eη (Tj ) = Covη (Ti , Tj )

Higher order moments of T1 , · · · , Ts are frequently required, e.g.

αr1 ,...,rs = E(T1r1 · · · Tsrs )

µr1 ,...,rs = E[(T1 − E(T1 ))r1 · · · (Ts − E(Ts ))rs ]
etc. These can often be obtained readily from the MGF:

MT (u1 , . . . , us ) := E(eu1 T1 +···+us Ts )

P 2
If MT exists in some neighborhood of 0 ( ui < δ), then all the moments αr1 ,··· ,rs exist
and are the coecients in the power series expansion

∞
X ur11 · · · urss
MT (u1 , . . . , us ) = αr1 ,··· ,rs
r1 ,...,rs
r1 ! · · · rs !

The cumulant generating function, CGF, sometimes more convenient for calculations
(especially for sums of independent random vectors), is dened as

KT (u1 , . . . , us ) := log MT (u1 , . . . , us ).

If MT exists in a neighborhood of 0, then so does KT and

∞
X ur11 · · · urss
KT (u1 , . . . , us ) = κr1 ,...,rs ,
r1 ,...,rs =0
r1 ! · · · rs !

where the coecients κr1 ,...,rs are called the cumulants of T.

The moments and cumulants can be found from each other by formal comparison of the
two series, and can be retrived from their respective generating functions as follows:

∂ r1 +···+rs MT (u1 , . . . , us ) ∂ r1 +···+rs KT (u1 , . . . , us )

αr1 ···rs = , κr1 ···rs = .
∂ur11 · · · ∂urss 0 ∂ur11 · · · ∂urss 0

For an exponential family, computation of these generating functions is particularly easy.

 1.3. EXPONENTIAL FAMILIES. 23

Theorem 1.3.15. If X has the density

s
X
pη (x) = exp [ ηi Ti (x) − A(η)]h(x)
i=1

w.r.t. some σ -nite measure µ, then for any η ∈ int(N ) the MGF and CGF of T exist
in a neighborhood of 0 and
KT (u) = A(η + u) − A(η)
MT (u) = eA(η+u)−A(η)

Proof. HW problem. □

Summary on Exponential Families. The family of probability measures {Pθ } with

densities relative to some σ -nite measure µ,
s
dPθ X
(1.3.6) pθ (x) = (x) = exp{ ηi (θ)Ti (x) − B(θ)}h(x), θ ∈ Ω,
dµ 1

is an s-parameter exponential family

By redening the functions Ti (·) and ηi (·) if necessary, we can always arrange for both
sets of functions to be anely independent. The number of summands in the exponent
is then the order of the exponential family.
If {1, η1 , . . . , ηs } and {1, T1 , . . . , Ts } are both L.I., then the family is said to be minimal
(but this does not imply minimal suciency), and

dpθ
s = dim(span{1, log (·), θ ∈ Ω}) − 1
dpθ0
= order of the exponential family

Remark 1.3.16. Since (1.3.6) is by denition a probability density w.r.t. µ for each
θ ∈ Ω, we have
Z nX o
expηi (θ)Ti (x) − B(θ) h(x)µ(dx) = 1
Z nX o
∴ exp B(θ) = exp ηi (θ)Ti (x) h(x)µ(dx)

which shows that the dependence of B on θ is through η(θ) = (η1 (θ), . . . , ηs (θ)) only, i.e.
B(θ) = A(η(θ)).
Remark 1.3.17. The previous note implies that each member of the family (1.3.6) is a
member of the family.
s
X
(1.3.7) πξ (x) = exp{ ξi Ti (x) − A(ξ)}h(x), ξ = (ξ1 , . . . , ξs ) ∈ η(Ω)
1
24 1. PRELIMINARIES

(in fact pθ (x) = πη(θ) (x)).

The family of densities {πξ , ξ ∈ η(Ω)} dened by (1.3.7) is the canonical family as-
sociated with (1.3.6). It is the same family parameterized by the natural parameter,
ξ =vector of coecients of Ti (x), i = 1, . . . , s.

Remark 1.3.18. Instead of restricting ξ to the set η(Ω), it is natural to extend the family
(1.3.7) to allow all ξ ∈ Rs for which we can choose a value of A(ξ) to make (1.3.7) a
probability density, i.e. for which
Z X
(1.3.8) exp{ ξi Ti (x)}h(x)µ(dx) < ∞

N = {ξ ∈ Rs : (1.3.8) holds} is the natural parameter space of the family (1.3.7).

Remark 1.3.19. N ⊇ η(Ω) since (1.3.7) is by denition a family of probability densities.

Definition 1.3.20. ( Full rank family) As with the original parameterization, we can
always redene ξ to ensure that {T1 , . . . , Ts } is A.I. If η(Ω) contains an s-dimensional
rectangle and {T1 (·), . . . , Ts (·)} is A.I., then T is minimal sucient and we say the
family (1.3.7) is of full rank. (A full rank family is clearly minimal.)

Remark 1.3.21. Since N ⊇ η(Ω), full rank ⇒ int(N ) ̸= ϕ and this is important in view
of the consequence of theorem 1.3.13 that
Z Xs
A(ξ)
e = exp( ξi Ti (x))h(x)µ(dx)
i=1

is analytic in each ξi on the set of s-dimensional complex vectors, ξ : Re(ξ) ∈ int(N ). (So
A(ξ)
derivatives of e w.r.t. ξi , i = 1, . . . , s of all orders can be obtained by dierentiation
under the integral, yielding explicit expressions for the moments of T for all values of the
canonical parameter vector ξ ∈ int(N ).)
Example 1.3.22. Multinomial X ∼ M (θ0 , . . . , θs ; n) = (X0 , . . . , Xs ), where Xi =
number of outcomes of type i in n independent trials where θi , i = 0, . . . , s, is the
probability of an outcome of type i on any one trial.

Ω = {θ : θ0 ≥ 0, · · · , θs ≥ 0, θ0 + · · · + θs = 1}

(1) Probability density with respect to counting measure on Zs+1

+
s
n! x0 xs
Y X
pθ (x) = θ0 · · · θs I[0, n] (xi )I{n} ( xi )
x0 ! · · · xs ! i=0
s
X
= exp{ xi log θi }h(x), θ ∈ Ω.
i=0
 1.3. EXPONENTIAL FAMILIES. 25

This is an (s + 1)-parameter exponential family with Ti (x) = xi , ηi (θ) = log θi .

The vectors η(θ), θ ∈ Ω, are not conned to a proper ane subspace of Rs , so
T is minimal sucient.
(2) {T0 , . . . , Ts } is not A.I. since T0 + · · · + Ts = n. Setting T0 (x) = x0 = n − x1 −
· · · − xn gives
s
X θi
pθ (x) = h(x) exp{n log θ0 + xi log }
i=1
θ0

Redening η(θ) = (log θθ01 , · · · , log θθ0s ), we now have an s-parameter represen-
tation in which {T1 , . . . , Ts } is A.I., since the vectors (x1 , · · · , xs ), x ∈ X , are
Ps
subject only to the constraints xi ≥ 0 and i=1 xi ≤ n.
θ θ
(3) Furthermore the new parameter vectors, η(θ) = (log 1 , · · · , log s ), θ ∈ Ω, are
θ0 θ0
s s
not conned to any proper ane subspace of R , since for any x ∈ R ∃ θ0 , . . . , θs
s
such that η(θ) = x and so η(Ω) = R . Hence T (x) = (x1 , . . . , xs ) is minimal
sucient for P and the order of the family is s.
(4) The canonical representation of the family (2) is

s
X θ1 θs
πξ (x) = exp{ ξi xi − A(ξ)}h(x), ξ ∈ η(Ω) = {(log , · · · , log ) : θ ∈ Ω}
1
θ0 θ0

We know from remark 1.3.16 before that B(θ) = A(η(θ)) for some function A(·).
Although it is not necessary, we can verify this directly in this example, since
from the representation (2) we have

B(θ) = −n log θ0

and

1 θ1 θs
θ0 = 1 − θ1 − · · · − θs ⇒ = 1 + + ··· +
θ0 θ0 θ0
η1 (θ)
=1+e + · · · + eηs (θ)
⇒ B(θ) = n log(1 + eη1 (θ) + · · · + eηs (θ) )
⇒ A(ξ) = n log(1 + eξ1 + · · · + eξs )

A(ξ) is of course also determined by

Z Xs
A(ξ)
e = exp{ ξi xi }h(x)dµ(x)
1
.
s
(5) The natural parameter space in this case is N = R , since we know that N ⊇
η(Ω) and η(Ω) = Rs by (3) above. Clearly N contains an s-dimensional rectangle
and {T1 , . . . , Ts } is A.I., hence {πξ (x), ξ ∈ N } is of full rank.
26 1. PRELIMINARIES

(6) Moments of T (X) = (X1 , . . . , Xs )

∂A
Theorem 1.3.14 ⇒ Eξ Ti = ∀ξ ∈ Rs
∂ξi
neξi
=
1 + eξ1 + · · · + eξs
nθi /θ0
= θ1
1 + θ0 + · · · + θθ0s
= nθi
∂ 2A
and Cov(Ti , Tj ) =
∂ξi ∂ξj
−neξi eξj
(
(1+eξ1 +···+eξs )2
= −nθi θj i ̸= j
= neξi ne2ξi
(1+···+eξs )
− (1+···+eξs )2 = nθi (1 − θi ) i = j
(Moments exist ∀ξ ∈ int(N ) = Rs )

Theorem 1.3.23. (Sucient condition for completeness of T ) If

s
!
X
πξ (x) = exp ξi Ti (x) − A (ξ) h (x) , ξ ∈ η (Ω)
i=1

is a minimal canonical representation of the exponential family P = {pθ : θ ∈ Ω} and

η (Ω) contains an open subset of Rs , then T = (T1 , . . . , Ts )is complete for P.

Proof. Suppose Eξ (f (T )) = 0 ∀ξ ∈ η(Ω). Then,

+ −
(1.3.9) Eξ f (T ) = Eξ f (T ) ∀ξ ∈ η(Ω).
Choose ξ0 ∈ int(η(Ω)) and r>0 such that

N (ξ0 , r) := {ξ : ||ξ − ξ0 || < r} ⊆ η(Ω).

Now dene the probability measures,

R + ξ ·t
f e 0 ν(dt)
+
λ (A) = R A + ξ0 ·t , ν = µ̃ ◦ T −1 , dµ̃(x) = h(x)µ(dx),
J
f e ν(dt)
R − ξ ·t
f e 0 ν(dt)
λ− (A) = R A − ξ0 ·t ,
J
f e ν(dt)
where we have assumed that ν({t : f (t) ̸= 0}) > 0, since otherwise f = 0 a.s. PT and we
are done.

Observe now that

Z Z
(1.3.10) δ·t +
e λ (dt) = eδ·t λ− (dt) ∀δ ∈ Rs with ||δ|| < r
 1.3. EXPONENTIAL FAMILIES. 27

since
Z Z
L.S. = +
f (t)e (ξ0 +δ)·t
ν(dt)/ f + (t)eξ0 ·t ν(dt)
ZJ ZJ
= f − (t)e(ξ0 +δ)·t ν(dt)/ f − (t)eξ0 ·t ν(dt)
J J

by (1.3.9)

Now consider each side of (1.3.10) as a function of the complex argument δ = δ0 + iθ,
θ ∈ Rs . Then
L(δ) = R(δ) ∀δ = δ0 + i · θ
with ||δ0 || < r, since (by Theorem 1.3.13 (i)) both sides are analytic in each component
of δ on the set where Re(ξ0 + δ) ∈ N and they are equal when δ is real. In particular,
Z Z
L(iθ) = e λ (dt) = R(iθ) = eiθ·t λ− (dt)
iθ·t +

Rs . Hence λ+ and λ− have the same characteristic

for all θ ∈ function ⇒ λ+ = λ− ⇒
+ −
f = f a.s., contradicting ν(f ̸= 0) > 0. So f = 0 a.s. ν . □
Example 1.3.24. X1 , . . . , Xn iid N (µ, σ 2 ), with σ2 known.

1 µ X 1 X 2 n
√
pµ (x) = exp{ 2 xi − 2 xi − },
(σ 2π)n σ 2σ 2
µ X
η(µ) = 2 , T (x) = xi
σ
P
Since η(Ω) = R contains a 1-dim rectangle in R, T (x) = xi is complete (and sucient,
or CSS).

Example 1.3.25. X1 , . . . , Xn iid N (σ, σ 2 )

1 1 X 2 1X n
pσ (x) = √ exp{− 2 xi + xi − },
(σ 2π)n 2σ σ 2
1 X
η1 (σ) = , T1 (x) = − x2i
2σ 2
1 X
η2 (σ) = , T2 (x) = xi
σ
2
Since η(Ω)
does not contain a 2-dim rectangle in R , the theorem is silent about com-
P 2 P
pleteness. In fact, we can show that T (x) = ( xi , xi ) is not complete since
X 
2 X 2 2
Eθ x2i − ( xi ) = n(2σ 2 ) − (nσ 2 + n2 σ 2 ) = 0, ∀σ,
n+1 n+1
2
P 2
( xi )2 = 0 on N c .
P
but there exists no P -null set N such that xi − n+1
28 1. PRELIMINARIES

1.4. Convex Loss Function

Lemma 1.4.1. Let ϕ be a convex function on (−∞, ∞) which is bounded below and suppose
that ϕ is not monotone. Then, ϕ takes on its minimum value c and ϕ−1 (c) is a closed
interval and is a singleton when ϕ is strictly convex.

Proof. Since ϕ is convex and not monotone,

lim ϕ (x) = ∞.
x→±∞

Since ϕ is continuous, ϕ attains its minimum value c. ϕ−1 ({c}) is closed by continuity, and
is an interval by convexity. The interval must have zero length if ϕ is strictly convex. □

Theorem 1.4.2. Let ρ be a convex function dened on (−∞, ∞) and X a random variable
such that ϕ (a) = E (ρ (X − a)) is nite for some a. If ρ is not monotone, ϕ (a)takes on
its minimum value and ϕ−1 (a) is a closed set and is a singleton when ρ is strictly convex.

Proof. By the lemma, we only need to show that ϕ is convex and not monotone.
Because limt→±∞ ρ (t) = ∞ and lima→±∞ x − a = ±∞,
lim ϕ (a) = ∞
a→±∞

so that ϕ is not monotone.

The convexity comes from

ϕ (pa + (1 − p) b) = Eρ (p (X − a) + (1 − p) (X − b))
≤ E (pρ (X − a) + (1 − p) ρ (X − b))
= pϕ (a) + (1 − p) ϕ (b) .

1.5. Model Selection

Throughout the course we assume the family P is known a priori, so that the model to
be tted to the data {X1 , . . . , Xn } is correct.
 CHAPTER 2

Unbiasedness

2.1. UMVU estimators.

Notation. P ={Pθ , θ ∈ Ω} is a family of probability measures on A (distributions of

X ).
T:X → R is an A/B measurable function and T (or T (X)) is called a statistic.

g: Ω→R is a function on Ω whose value at θ is to be estimated.

X T
→ R, B, PθT


(X , A, Pθ ) −
→ (X , A, Pθ ) −

Definition 2.1.1. A statistic T (or T (X)) is called an unbiased estimator of g (θ) if

Eθ (T (X)) = g (θ) for all θ ∈ Ω.

Objectives of point estimation. In order to specify what we mean by a good

estimator of g(θ), we need to specify what we mean when we say that T (X) is close to
g(θ). A fairly general way of dening this is to specify a loss function:
L(θ, d) = cost of concluding that g(θ) = d, when the parameter value is θ.
L(θ, d) ⩾ 0 and L(θ, g(θ)) = 0.
Since T (X) is a random variable, we measure the performance of T (X) for estimating
g(θ) in terms of its expected (or long-term average) loss

R(θ, T ) = Eθ L(θ, T (X)),

known as the risk function.

Choice of a loss function will depend on the problem and the purpose of the estimation.
For many estimation problem, the conclusion is not particularly sensitive to the choice
of loss function within a reasonable range of alternatives. Because of this and especially
because of its mathematical convenience, we often choose (and will do so in this chapter)
the squared-error loss function
L(θ, d) = (g(θ) − d)2
29
30 2. UNBIASEDNESS

with corresponding risk function

(2.1.1) R(θ, T ) = Eθ (T (X) − g(θ))2

Ideally we would like to choose T to minimize (2.1.1) uniformly in θ. Unfortunately this

is impossible since the estimator T dened by

(2.1.2) T (x) = g(θ0 ) ∀x ∈ X

(where θ0 is some xed parameter value in Ω) has the risk function,

0 if θ = θ0
R(θ, T ) =
(g(θ) − g(θ0 ))2 if θ ̸= θ0
An estimator which simultaneously minimized R(θ, T ) for all θ ∈ Ω would necessarily

have R(θ, T ) = 0 ∀θ ∈ Ω and this is impossible except in trivial cases.

Why consider the class of unbiased estimators? There is nothing intrinsically

good about unbiased estimators. The only criterion for goodness is that R(θ, T ) should
be small. The hope is that by restricting attention to a class of estimators which ex-
cludes (2.1.2), we may be able to minimize R(θ, T ) uniformly in θ and that the resulting
estimator will give small values of R(θ, T ). This programme is frequently successful if
we attempt to minimize R(θ, T ) with T restricted to the class of unbiased estimators of
g(θ).
Definition 2.1.2. g(θ) is U-estimable, if there exists an unbiased estimator of g(θ).
Example 2.1.3. X1 , . . . , Xn iid Bernoulli(p), p ∈ (0, 1). g(p) = p is U-estimable, since
E X̄n = p ∀p ∈ (0, 1), while h(p) = p1 is not U-estimable, since if
X P
xi
P 1
T (x)p (1 − p)n− xi
= ∀p ∈ (0, 1),
p
limp→0 RS = ∞ and limp→0 LS = T (0). So T (0) = ∞, but this is not possible since then
̸ p1 ∀p ∈ (0, 1).
Ep T (X) = ∞ =
Pn
Xi −1
a.s. Pn n ∀p ∈ (0, 1). n
−→ p a.s.
−→ p
i=1
Remark 2.1.4. and Hence P is a
n i=1 Xi Xi
reasonable estimate of p−1 even though it is not unbiased.

Theorem 2.1.5. If T0 is an unbiased estimator of g (θ) then the totality of unbiased

estimators of g (θ)is given by
{T0 − U : Eθ U = 0 for all θ ∈ Ω} .

Proof. If T is unbiased for g(θ), then T = T0 −(T0 −T ) where Eθ (T0 −T ) = 0 ∀θ ∈ Ω.

Conversely if T = T0 − U where Eθ U = 0 ∀θ ∈ Ω, then Eθ T = Eθ T0 = g(θ) ∀θ ∈ Ω. □
 2.1. UMVU ESTIMATORS. 31

Remark 2.1.6. For squared error loss, L(θ, d) = (d − g(θ))2 , the risk R(θ, T ) is

R(θ, T ) = Eθ ((T (X) − g(θ))2 )

= V arθ (T (X)) if T is unbiased

= V arθ (T0 (X) − U )

= Eθ [(T0 (X) − U )2 ] − g(θ)2
and hence the risk is minimized by minimizing Eθ [(T0 (X) − U )2 ]
with respect to U , i.e.
by taking anyxed unbiased estimator of g(θ) and nding the unbiased estimator of zero
2
which minimizes Eθ [(T0 (X)−U ) ]. Then if U does not depend on θ we shall have found a
uniformly minimum risk estimator of g(θ), while if U depends on θ , there is no uniformly
minimum risk estimator. Note that for unbiased estimators and squared error loss, the
risk is the same as the variance of the estimator, so uniformly minimum risk unbiased is
the same as uniformly minimum variance unbiased in this case.

Example 2.1.7. P (X = −1) = p, P (X = k) = q 2 pk , k = 0, 1, . . ., where q = 1 − p.

T0 (X) = I{−1} (X) is unbiased for p, 0 < p < 1
T1 (X) = I{0} (X) is unbiased for q 2 ,
U is unbiased for 0
∞
X ∞
X
⇔0 = U (k)P (X = k) = pU (−1) + U (k)q 2 pk ∀p
k=−1 k=0
∞
X
= U (0) + (U (k) − 2U (k − 1) + U (k − 2))pk
k=1
⇔U (k) = −kU (−1) = ka for some a
k
(comparing coecients of p , k = 0, 1, 2, . . .)
So an unbiased estimator of p with minimum risk (i.e. variance) is T0 (X) − a∗0 X where
a∗0 is the value of a which minimizes
X
Ep (T0 (X) − aX)2 = Pp (X = k)[T0 (k) − ak]2
2
Similarly an unbiased estimator of q with minimum risk (i.e. variance) is T1 (X) − a∗1 X
∗
where a1 is the value of a which minimizes
X
Ep (T1 (X) − aX)2 = Pp (X = k)[T1 (k) − ak]2
Some straightforward calculations give

−p
a∗0 = 2
P∞ and a∗1 = 0
p+q 1 k 2 pk
∗ 2
Since a1 is independent of p, the estimator T1 (X) of q is minimum variance unbiased for
∗ ∗
all p, i.e. UMVU. However a0 does depend on p and so the estimator T0 (X) = T0 (X) −
∗
a0 X is only locally minimum variance unbiased at p. (We are using estimator
32 2. UNBIASEDNESS

in a generalized sense here since T0∗ (X) depends on p. We shall continue to use this
terminology.) An UMVU estimator of p does not exist in this case.

Definition 2.1.8. Let V (θ) = inf T V arθ (T ) where the inf is over all unbiased estimators
of g(θ). If an unbiased estimator T of g(θ) satises

V arθ (T ) = V (θ) ∀θ ∈ Ω it is called UMVU

If

V arθ0 T = V (θ0 ) for some θ0 ∈ Ω T is called LMVU at θ0

Remark 2.1.9. Let H be the Hilbert space of functions on X which are square integrable
with respect to P (i.e. with respect to every Pθ ∈ P ), and let U be the set of all unbiased
estimators of 0. If T0 is an unbiased estimator of g(θ) in H, then a LMVU estimator in H
at θ0 is T0 − PU (T0 ), where PU denotes orthogonal projection on U in the inner product
2
space L (Pθ0 ), i.e. PU (T0 ) is the unique element of U such that

T0 − PU (T0 ) ⊥ U (in L2 (Pθ0 )).

T0 − PU (T0 ) is LMVU since PU (T0 ) = arg minU ∈U Eθ0 (T0 − U )2 .

Notation 2.1.10. We denote the set of all estimators T with Eθ T 2 < ∞ for all θ∈Ω
by ∆ and the set of all unbiased estimators of 0 in ∆ by U.

Theorem 2.1.11. An unbiased estimator T ∈ ∆ of g (θ) is UMVU i

Eθ (T U ) = 0 for all U ∈ U and for all θ ∈ Ω.

(i.e. Covθ (T, U ) = 0 since Eθ U = 0 for all θ and Eθ T = g (θ) for all θ ∈ Ω.)

Proof. (⇒) Suppose T is UMVU. For U ∈ U, let T ′ = T + λU with λ real. Then

′
T is unbiased and, by denition of T,

V arθ (T ′ ) = V arθ (T ) + λ2 V arθ (U ) + 2λCovθ (T, U ) ⩾ V arθ (T )

therefore, λ2 V arθ (U ) + 2λCovθ (T, U ) ⩾ 0. Setting λ = − Cov θ (T,U )

V arθ (U )
gives a contradiction

to this inequality unless Covθ (T, U ) = 0. Hence Covθ (T, U ) = 0.

(⇐) If Eθ (T U ) = 0 ∀U ∈ U and ∀θ ∈ Ω, let T ′ be any other unbiased estimator. If

V arθ (T ′ ) = ∞, then V arθ (T ) < V arθ (T ′ ), so suppose V arθ (T ′ ) < ∞.
′
Then T = T − U , for some U which is unbiased for 0 (by Theorem 2.1.5).

U = T − T ′ ⇒ Eθ U 2 = Eθ (T ′ − T )2
2
⩽ 2Eθ T ′ + 2Eθ T 2 < ∞
⇒U ∈U
 2.1. UMVU ESTIMATORS. 33

Hence

V arθ (T ′ ) = V arθ (T − U )
= V arθ (T ) + V arθ (U ) − 2Covθ (T, U )
⩾ V arθ (T ) since Covθ (T, U ) = 0,
⇒ T is UMVU.
□

Unbiasedness and suciency. Suppose now that T ∈∆ is unbiased for g(θ) and
S is sucient for P = {Pθ , θ ∈ Ω}. Consider

T ′ = Eθ (T |S) = E(T |S) independent of θ

Then

(a)

Eθ T ′ = Eθ E(T |S) = Eθ (T ) = g(θ) ∀θ.

(b)

V arθ (T ) = Eθ (T − E(T |S) + E(T |S) − g(θ))2

= Eθ ((T − E(T |S))2 ) + V arθ (T ′ ) + 2Eθ [(T − E(T |S))(E(T |S) − g(θ))]
⩾ V arθ (T ′ ).

On the second line we used the fact that T −E(T |S) is orthogonal to σ(S). The inequality
on the third line is strict for all θ ⇔ T = E(T |S) a.s. P .
Theorem 2.1.12. If S is a complete sucient statistic for P , then every U -estimable
function g (θ) has one and only one unbiased estimator which is a function of S .

Proof.

T unbiased ⇒ E(T |S) is unbiased and a function of S

T1 (S), T2 (S) unbiased ⇒ Eθ (T1 (S) − T2 (S)) = 0 ∀θ
⇒ T1 (S) = T2 (S) a.s. P (completeness)
□
Theorem 2.1.13. ( Rao-Blackwell) Suppose S is a complete sucient statistic for P .
Then

(i) If g (θ) is U -estimable, there exists an unbiased estimator which uniformly min-
imizes the risk for any loss function L (θ, d) which is convex in d.
(ii) The UMV U in (i) is the unique unbiased estimator which is a function of S; it
is the unique unbiased estimator with minimum risk provided the risk is nite
and L is strictly convex in d.
34 2. UNBIASEDNESS

Proof. (i) L(θ, d) convex in d means

L(θ, pd1 + (1 − p)d2 ) ⩽ pL(θ, d1 ) + (1 − p)L(θ, d2 ), 0 < p < 1.

Let T be any unbiased estimator of g(θ) and let T ′ = E(T | S), another unbiased
estimator of g(θ). Then

R(θ, T ′ ) = Eθ [L(θ, E(T | S))]

⩽ Eθ [Eθ (L(θ, T ) | S)], by Jensen's inequality for conditional expectation,

= Eθ L(θ, T ) = R(θ, T ) ∀θ.

If T2 is any other unbiased estimator then

T2′ = E(T2 | S) = T ′ a.s. P by Theorem 2.1.12.

Hence starting from any unbiased estimator and conditioning on the CSS S
gives a uniquely dened unbiased estimator which is UMVU and is the unique
function of S which is unbiased for g(θ).
(ii) The rst statement was established at the end of the proof of (i).
′
If T is UMVU then so is T = E(T | S) as shown in (i); We will show that T
is necessarily the uniquely determined unbiased function of S, by showing that
T is a function of S a.s. P.
The proof is by contradiction. Suppose that "T is a function of S a.s. P" is
false. Then there exists θ and a set of positive Pθ measure where

T ′ := E(T | S) ̸= T
But this implies that

R(θ, T ′ ) = Eθ (L(θ, E(T | S)))

< Eθ (Eθ (L(θ, T ) | S))
(Jensen's inequality is strict unless E(T | S) = T a.s. Pθ )
= R(θ, T )
contradicting the UMVU property of T.
□
Theorem 2.1.14. If P is an exponential family of full rank (i.e. {η1 , . . . , ηs } and
{T1 , . . . , Ts } are A.I. and η (Ω) contains an open subset of Rs ) then the Rao-Blackwell
theorem applies to any U -estimable g (θ) with S = T .

Proof. T is complete sucient for P.

[Some obvious U -estimable g(θ)'s are

∂A
Eθ Ti (X) = ξ=η(θ) , {θ : η(θ) ∈ int(N )},
∂ξi
P
ξi Ti (x)−A(ξ)
where πξ (x) = e h(x) is the canonical representation of pθ (x).] □
 2.1. UMVU ESTIMATORS. 35

Two methods for nding UMVU's

Method 1. Search for a function δ(T ), where T is a CSS, such that

Eθ δ(T ) = g(θ), ∀θ ∈ Ω.
Example 2.1.15. X1 , . . . , X n iid N (µ, σ 2 ), µ ∈ R, σ 2 > 0.
T =(X̄, S 2 ) is CSS.

Eµ,σ2 X̄ = µ
X̄ is UMVU for µ.

Method 2. Search for an unbiased δ(X) and a CSS T. Then

S = E(δ(X) | T ) is U M V U
Example 2.1.16. X1 , . . . , Xn iid U (0, θ), θ > 0
θ
g(θ) =
2
δ1 (X) = X1 is unbiased
X(n) is CSS
∴ S = E(X1 | X(n) ) is UMVU

To compute S we note that given X(n) = x,

1
X1 = x w.p.
n
1
X1 ∽ U (0, x) w.p. 1 −
n
x 1 x n+1x
∴ S(x) = + (1 − ) =
n n 2 n 2
1n+1 θ
∴ S(X(n) ) = X(n) is UMVU for
2 n 2
n+1
⇒ X(n) is UMVU for θ
n
Remark 2.1.17.

(a) Convexity of L(θ, ·) is crucial to the Rao-Blackwell theorem.

(b) Large-sample theory tends to support the use of convex L(θ, ·).

Heuristically if X1 , . . . , X n are iid, then as n→∞ the error in estimating g(θ) → 0 for
any reasonable estimates (in some probabilistic sense). Thus only the behavior of L(θ, d)
for d close to g(θ) is relevant for large samples.

A Taylor expansion around d = g(θ) gives

L(θ, d) = a(θ) + b(θ)(d − g(θ)) + c(θ)(d − g(θ))2 + Remainder

36 2. UNBIASEDNESS

But

L(θ, g(θ)) = 0 ⇒ a(θ) = 0

L(θ, d) ≥ 0 ⇒ b(θ) = 0
Hence locally, L(θ, d) ∽ c(θ)(d − g(θ))2 , a convex weighted squared error loss function.

Example 2.1.18. Observe X1 , . . . , X m , iid N (ξ, σ 2 ), and Y1 , . . . , Y n , iid N (η, τ 2 ), inde-

pendent of X1 , . . . , X m .
2
(i) For the 4-parameter family P = {Pξ,η,σ2 ,τ 2 }, (X̄, Ȳ , SX , SY2 ) is a CSS since the
2 2
exponential family is of full rank. Hence X̄ and SX are UMVU for ξ and σ
2 2
respectively and Ȳ and SY are UMVU for η and τ .

(ii) For the 3-parameter family P = {Pξ,η,σ 2 ,σ 2 }, (X̄, Ȳ , SS) is a CSS, where SS :=
2 SS
(m − 1)SX + (n − 1)SY2 . Hence X̄, Ȳ and m+n−2 are UMVU for ξ, η and σ2
respectively.

(iii) For the 3-parameter family with ξ = η, σ 2 ̸= τ 2

(which arises when estimating
2 2
a mean from 2 sets of readings with dierent accuracies), (X̄, Ȳ , SX , SY ) is min-
imal sucient but not complete, since X̄ − Ȳ ̸= 0 a.s. P , but Eθ (X̄ − Ȳ ) = 0 ∀ θ.

σ2
To deal with Case (iii) we shall rst show the following: If
τ2
=r for some xed
r, i.e.
P ∗ = {Pξ,ξ,rτ 2 ,τ 2 }
T ∗ = ( Xi + r Yj , Xi2 + r Yj2 )
P P P P
then is CSS

Proof.

11 1
pξ,τ 2 (x, y) = mm+n n
(2π) (rτ ) 2 (τ 2 ) 2
2 2

mξ 2 nξ 2
 
1 X 2 1 1 X 2 1
× exp − xi + 2 mξ x̄ − − yi + 2 nξ ȳ − 2
2rτ 2 rτ 2rτ 2 2τ 2 τ 2τ
 
 2 1 X
2
X
2 ξ X X
= exp −A(ξ, τ ) ∗ exp − ( xi + r yi ) + 2 ( xi + r yi )
2rτ 2 rτ
□

P P
Xi +r Yi
Since T ∗ is a CSS for P ∗ and since T1 = m+rn
is unbiased for ξ , it is UMVU
for ξ in P ∗ .

T1 is also unbiased for ξ in P = {Pξ,ξ,σ2 ,τ 2 }

σ02 τ02 σ02
∴ V (ξ0 , σ02 , τ02 ) ⩽ V arξ0 ,σ02 ,τ02 (T1 ) = , where = r.
mτ02 + nσ02 τ02
 2.2. NON-PARAMETRIC FAMILIES 37

(V is the smallest variance of all unbiased estimators of ξ for P evaluated at

ξ0 , σ02 , τ02 .)

On the other hand, every T which is unbiased for ξ in P is also unbiased in

∗
P . Hence if T is unbiased for ξ in P , then

σ2
P P
X i + r Yi
V arξ0 ,σ02 ,τ02 (T ) ⩾ V arξ0 ,σ02 ,τ02 ( ), where r = 02 ,
m + rn τ0
and the inequality continues to hold with the left-hand side replaced by V (ξ0 , σ02 , τ02 ).
2 2 σ02 τ02 2 2
So V (ξ0 , σ0 , τ0 ) = and the LMVU estimator at (ξ0 , σ0 , τ0 ) is
mτ02 +nσ02
P σ02 P
Xi + τ02
Yi
σ02
.
m+ τ02
n
σ02
Since this estimate depends on the ratio r= τ02
, an UMVU for ξ does not exist

in P.
A natural estimate for ξ is
P 2
SX P
Xi + SY2
Yi
ξˆ = 2
SX
.
m+ SY2
n
(See Graybill & Deal, 1959, for its properties.)

2.2. Non-parametric families

Consider X = (X1 , . . . , Xn ), where X1 , . . . , Xn are iid F , where F ∈ F , a family of

distribution functions, and P is the corresponding product measure on (Rn , B n ). For
example,

F0 = df 's with density relative to Lebesgue measure,

Z
F1 = df 's with |x|F (dx) < ∞,
Z
F2 = df 's with x2 F (dx) < ∞, etc.

The estimand is g : F → R. For example,

Z
g(F ) = xF (dx) = µF
Z
g(F ) = x2 F (dx)
g(F ) = F (a)
g(F ) = F −1 (p)
38 2. UNBIASEDNESS

Proposition 2.2.1. If F0 is dened as above, then (X(1) , . . . , X(n) ) is complete sucient

for F0 (i.e. for the corresponding family of probability measures P ).

Proof. We know that T (X) = (X(1) , . . . , X(n) ) is sucient for P . It remains to show
(by problem 1.6.32, p.72) that T is complete and sucient for a family P0 ⊆ P such that
n
each member of P0 has positive density on R . Choose P0 to be the set of probability
n
measures on B with densities relative to Lebesgue measure,
X X X
C(θ1 , · · · , θn ) exp{θ1 xi + θ 2 xi xj + · · · + θn x1 · · · xn − x2n
i }}
i<j

This is an exponential family whose natural parameter set N contains an open set (N =
Rn ). So S(x) = ( xi , i<j xi xj , · · · , x1 · · · xn ) is complete. But S is equivalent to T
P P
th
(consider the n degree polynomial whose zeroes are x(1) , · · · , x(n) ), so T is complete for
F0 . □

Measurable functions of the order statistics. If T (x) := (x(1) , . . . , x(n) ) then

δ(X1 , . . . , Xn ) ∈ σ(T ) ⇔ δ(X1 , . . . , Xn ) = δ(Xπ1 , . . . , Xπn )

for every permutation (π1 , . . . , πn ) of (1, . . . , n). Since T is a CSS for F0 , this enables us
to identify UMVU estimators of estimands g for which they exist.

Example 2.2.2. g(F ) = F (a). An obvious unbiased estimator of F (a) is

n
1 X
T1 (X) := I(−∞,a] (Xi )
n i=1

and T1 ∈ σ(T ) so T1 is UMVU for F (a), F ∈ F0 .

R
Example 2.2.3. g(F ) = xdF, F ∈ F0 ∩ F2 . Let
n
1X
T2 (x) = Xi
n i=1
Then T2 ∈ σ(T ) and, since T is also complete for F0 ∩ F2 , it is therefore UMVU for µF .

Example 2.2.4. g(F ) = σF2 , F ∈ F0 ∩ F4 . Let

(x(i) − n1 x(i) )2
P P
(xi − x̄)2
P
2
T3 (x) = S(x) = =
n−1 n−1
2
T3 ∈ σ(T ) and is unbiased for σF . Since T is complete for F0 ∩ F4 , T3 is UMVU for
σF2 .
∗
Remark 2.2.5. T complete for F
does not imply generally that T is complete for F ⊆ F .
∗
In fact the reverse is true. Completeness for F implies completeness for F . However
the same argument used in the proof of Proposition 2.2.1 shows that

T is complete for F0 ∩ F2 (used in example 2.2.3) and

T is complete for F0 ∩ F4 (used in example 2.2.4).

 2.2. NON-PARAMETRIC FAMILIES 39

Example 2.2.6. g(F ) = µ2F , F ∈ F0 ∩ F4

1X 2
T4 (X) = X(i) − S 2 (X) is UMVU for g(F ).
n
This result could also be obtained by observing that µ2F , F ∈
X1 X2 is unbiased for
F0 ∩F4 , therefore E(X1 X2 | X(1) , . . . , X(n) ) is UMVU. But conditioned on X(1) , · · · , X(n) ,
2
X1 X2 = X(i) X(j) w.p. for each subset {i, j} of {1, . . . , n} with i < j
n(n − 1)

1 X
∴ E(X1 X2 | X(1) , · · · , X(n) ) = X(i) X(j)
n(n − 1) i̸=j
1 X X
= (( Xi )2 − Xi2 )
n(n − 1)
1X 2 1 X 2 1 X
= Xi − ( Xi − ( Xi )2 )
n n−1 n
= T4 (X)
More generally suppose g(F ) is U-estimable in F0 . Then

∃ δ(X1 , . . . , Xm ) such that EF δ(X1 , . . . , Xm ) = g(F ) ∀F ∈ F0 .

Suppose also that δ(X1 , . . . , Xm ) has nite second moment for F ∈ F0 ∩ Fk for some
positive integer k. We can assume δ is symmetric in X1 , . . . , Xm , since if not we can
redene δ as
1 X
δ ∗ (X1 , . . . , Xm ) = δ(Xπ1 , . . . , Xπm )
m!
permutations π of (1,...,m)

which is also unbiased and symmetric.

Now we dene the U-statistic (Sering, 1980),
1 X
T = n

δ ∗ (Xi1 , . . . , Xim )
m 1⩽i1 <i2 <···<im ⩽n

This is symmetric in X1 , . . . , X n and unbiased, and therefore UMVU for g(F ), F ∈

F0 ∩ F k .

Questions

(1) Which g(F ) are U-estimable?

(2) If g is U-estimable, what is the smallest value of m for which there exists a
U-statistic for g of the form T? This number is called the degree of g .

Proposition 2.2.7. If g is of degree 1, then for any F1 , F2 ∈ F0 , g(αF1 + (1 − α)F2 ) is

linear in α.
40 2. UNBIASEDNESS

Proof. If g is of degree 1, there exists δ(X1 ) such that

Z
δ(x)F (dx) = g(F )
Z Z
∴ g(αF1 + (1 − α)F2 ) = α δ(x)F1 (dx) + (1 − α) δ(x)F2 (dx)
= αg(F1 ) + (1 − α)g(F2 ).
□

Generalization. Ifg is of degree s, then g(αF1 + (1 − α)F2 ) is a polynomial in α of

degree ⩽s (since dF (x1 , . . . , xs ) = αs dF1 (x1 ) · · · dF1 (xs ) if F1 is replaced by αF1 .)
Example 2.2.8. g(F ) = σF2 is of degree 2 in F0 ∩ F2 .

Proof. Let δ(X1 , X2 ) = 12 (X1 − X2 )2 , then

EF δ = EF X12 − EF (X1 X2 ) = σF2

so deg(g) ⩽ 2. To show deg(g) ̸= 1, consider

2
g(αF1 + (1 − α)F2 ) = σαF +(1−α)F2
Z1 Z
= α x dF1 (x) + (1 − α) x2 dF2 (x) − [αµF1 + (1 − α)µF2 ]2
2

and this is linear in α. ⇔ µF1 = µF2 . But this is not the case for every F1 , F2 ∈ F0 ∩ F2 .
Hence deg(g) = 2. □
Example 2.2.9. g(F ) = σF is not U-estimable in F0 , since g(αF1 + (1 − α)F2 ) is not a
polynomial.

2.3. The Information Inequality

For any estimator T ∈ ∆ of g (θ) and any function ψ (X, θ) such that Eθ |ψ (X, θ)|2 < ∞,
we have the inequality

|Covθ (T, ψ)|2

(2.3.1) Varθ (T ) ≥ .
Varθ (ψ (X, θ))

However, this will not in general provide a useful lower bound for Varθ T since the RHS
depends on T. It can be useful however when the RHS depends on T in a simple way, in
particular when it depends on T only through Eθ T .
Theorem 2.3.1. Covθ (T, ψ) depends on T only through Eθ T i
Covθ (U, ψ) = 0 for all U ∈ U ∩ ∆ (unbiased square-integrable estimators of 0).
 2.3. THE INFORMATION INEQUALITY 41

Proof. ( ⇐) Suppose Covθ (U, ψ) = 0 for all U ∈ U ∩∆ and that T1 , T2 are two
estimators with nite variance and

Eθ T1 = Eθ T2 ∀θ ∈ Ω.
Then T1 − T2 ∈ U , so Covθ (T1 , ψ) = Covθ (T2 , ψ).

(⇒) If Covθ (T, ψ) depends on T only through Eθ T and if U ∈ U, then

Covθ (T + U, ψ) = Covθ (T, ψ)

∴ Covθ (U, ψ) = 0
□

Hammersley-Chapman-Robbins Inequality
Suppose X has probability density p(x, θ) θ ∈ Ω, where p(x, θ) > 0 ∀x and θ. Suppose
∃ θ, θ + δ s.t. g(θ) ̸= g(θ + δ). Then,
p(x, θ + δ)
ψ(x, θ) = −1
p(x, θ)
satises the conditions of the previous theorem, i.e.
R Covθ (U, ψ) = 0 ∀U ∈ U ∩ ∆, since
Eθ ψ(X, θ) = 0 and Eθ (U ψ) = U (x)(p(x, θ + δ) − p(x, θ))dµ(x).

For any statistic S ∈ ∆,

Z
Covθ (S, ψ) = Eθ (Sψ) = S[p(x, θ + δ) − p(x, θ)]µ(dx)
= Eθ+δ S − Eθ S

0 if S ∈ U,
=
g(θ + δ) − g(θ) if S is unbiased for g(θ.)
Hence from (2.3.1), if T ∈∆ is unbiased for g(θ),
(g(θ + δ) − g(θ))2
V arθ (T ) ⩾ ∀δ.
Eθ [( p(X,θ+δ)
p(X,θ)
− 1)2 ]

Hence we obtain the

HCR bound
(g(θ + δ) − g(θ))2
V arθ (T ) ⩾ sup ,
δ V arθ ( p(X,θ+δ)
p(X,θ)
)
if T is unbiased for g(θ).
42 2. UNBIASEDNESS

Letting δ→ 0 in the HCR bound gives

( g(θ+δ)−g(θ)
δ
)2
V arθ T ≥ lim
δ→0 Eθ ( 1δ p(X,θ+δ)−p(X,θ)
p(X,θ)
)2
g ′ (θ)2
=
Eθ ( ∂p
∂θ
/p)2
g ′ (θ)2
=
Eθ ( ∂ log∂θ
p(X,θ) 2
)
provided g is dierentiable and we can dierentiate under the expectation. These steps
are legitimized under the conditions of the following theorem.

Theorem 2.3.2. (Cramer-Rao Lower Bound CRLB) Suppose that the density of
the sample p(x, θ) > 0 and ∂
∂θ
log p(x, θ) exists for all x and θ, and that for each θ there
exists δ such that
Pϕ ≪ Pθ and
(
|ϕ − θ| < δ ⇒ 1 p(x,ϕ)
|ϕ−θ| p(x,θ)
− 1 ≤ G (x, θ)

(where G is independent of ϕ and Eθ G (X, θ)2 < ∞ for all θ). Then for any unbiased
estimator T of g (θ),
g ′ (θ)2
Varθ T ≥ ,
I (θ)
where
  2
 I (θ) = Eθ ∂ log p(x,θ) ,
∂θ
= Fisher Information
 2
 (g ′ (θ))2 = lim sup g(ϕ)−g(θ)
.
ϕ→θ ϕ−θ

Proof. By the HCR bound

g(ϕ) − g(θ) 2
Z
1 p(x, ϕ)
V arθ (T ) 2
( − 1)2 p(x, θ) µ(dx) ≥ ( )
|ϕ − θ| p(x, θ) ϕ−θ
Let {ϕn } be a sequence such that ϕn → θ and

g(ϕn ) − g(θ) 2
( ) → (g ′ (θ))2 .
ϕn − θ
Then setting ϕ = ϕn in the above inequality and letting n → ∞, gives (by DC)

∂
V arθ (T ) Eθ ( log p(X, θ))2 ≥ g ′ (θ)2
∂θ
□
 2.3. THE INFORMATION INEQUALITY 43

Corollary 2.3.3. If X1 , . . . , Xn are iid P1,θ (the marginal distribution of Xi ) and the
corresponding marginal density p1 (x, θ) satises the conditions of the Cramer-Rao Lower
Bound theorem, then for any unbiased estimator T (X1 , · · · , Xn ) of g (θ),
2
g ′ (θ)2

∂ log p1 (X1 , θ)
Varθ (T ) ≥ where I1 (θ) = Eθ .
nI1 (θ) ∂θ

Proof. The sample space is Xn and

n
dPθ Y
(x) = p1 (xi , θ) where µn = µ ⊗ µ ⊗ · · · ⊗ µ.
dµn i=1

The Fisher information for Pθ is

Z n
!2
∂ Y
log p1 (xi , θ) p1 (x1 , θ) · · · p1 (xn , θ)µn (dx)
∂θ i=1
Z X 2
∂ log p1 (xi , θ)
= p1 (x1 , θ) · · · p1 (xn , θ)µn (dx)
∂θ
Z X 2
∂ log p1 (xi , θ)
= p1 (x1 , θ) · · · p1 (xn , θ)µn (dx) = nI(θ)
∂θ
since
   2
∂ log ∂ log ∂ log
Eθ p1 (Xi , θ) p1 (Xj , θ) = Eθ p1 (Xi , θ) (by independence)
∂θ ∂θ ∂θ
and
∂ log
E( p1 (Xi , θ)) = 0.
∂θ
(We can dierentiate under the integral sign by DC and the assumptions on p1 (xi , θ)).

The statement of the Corollary now follows if we can show that the assumptions on
p1 (xi , θ) carry over to p(x, θ), i.e. that |ϕ − θ| < δ ⇒
p1 (x1 , ϕ) · · · p1 (xn , ϕ)
| − 1 | /|ϕ − θ| ≤ G̃(x, θ)
p1 (x1 , θ) · · · p1 (xn , θ)
where Eθ G̃2 (X, θ) < ∞.

Now

p1 (xi , ϕ)
≤ 1 + |ϕ − θ|G(xi , θ)
p1 (xi , θ)
≤ 1 + δG(xi , θ)
44 2. UNBIASEDNESS

and

|a1 · · · an − 1| ≤ |a1 · · · an − a2 · · · an | + |a2 · · · an − a3 · · · an | + · · · by the triangle inequality,

≤ |a1 − 1||a2 · · · an | + |a2 − 1||a3 · · · an | + · · · + |an − 1|.

p1 (xi ,ϕ)
Setting ai = p1 (xi ,θ)
gives

p1 (x1 , ϕ) · · · p1 (xn , ϕ) Y
| − 1 | ≤ |ϕ − θ|G(x1 , θ) (1 + δG(xi , θ))
p1 (x, θ) · · · p1 (xn , θ) i>1
Y
+ |ϕ − θ|G(x2 , θ) (1 + δG(xi , θ))
i>2
+ ···
+ |ϕ − θ|G(xn , θ)
:= |ϕ − θ|G̃(x, θ)
andEθ G̃(X, θ)2 < ∞ since X1 , . . . , Xn are independent and Eθ G(Xi , θ)2 < ∞. (No
G(Xi , θ) is raised to a power greater than 2.) □
Corollary 2.3.4. Suppose p(x, θ) satises the conditions of theorem 2.3.2 and
Eθ T (X) = g(θ) + b(θ)
i.e., T (X) has bias b(θ) for estimating g(θ). Then
M SEθ (T ) = Eθ (T (X) − g(θ))2
= b2 (θ) + Vθ (T )
c(θ)
≥ b2 (θ) +
I(θ)
where  2
g(ϕ) + b(ϕ) − g(θ) + b(θ)
c(θ) = lim sup
ϕ→θ ϕ−θ

Proof. T is unbiased for g(θ) + b(θ), so

c(θ)
Eθ (T (X) − g(θ) − b(θ))2 ≥
I(θ)
c(θ)
Hence M SE = Eθ (T (X) − g(θ))2 = Varθ T (X) + [Eθ (T (X) − g(θ))]2 ≥ b2 (θ) + I(θ)
.
□

Behavior of I(·) under reparameterization. Suppose α = h (θ) reparametrizes

{Pθ : θ ∈ Ω} to {Pα∗ : α ∈ h (Ω)}. Then

p∗ (x, α) = p (x, h← (α)) ,

 2.3. THE INFORMATION INEQUALITY 45

where h← denotes the inverse mapping and, by denition,

 2
∗ ∂ log ∗
I (α) = Eα p (X, α)
∂α
 2
∂ log p (X, θ) d ←
= Eα | h (α)
∂θ |θ=h← (α) dα
 ← 2
← dh (α)
= I (h (α))
dα
I (θ)
= .
(h′ (θ))2 θ=h← (α)

An alternative expression for I(θ). Provided

∂2
∂θ2
log p(x, θ) exists for all x, θ and if

∂2 ∂2
Z Z
(2.3.2) p(x, θ) dµ(x) = p(x, θ) dµ(x) = 0,
∂θ2 ∂θ2
then
∂2
 
I(θ) = −Eθ log p(x, θ) .
∂θ2

Proof.
2
∂2 ∂
∂θ2
p(x, θ) ( ∂p(x,θ)
∂θ
)2
log p(x, θ) = −
∂θ2 p(x, θ) p(x, θ)2
h i
∂2
and Eθ p(X, θ)−1 ∂θ 2 p(X, θ) = 0 by (2.3.2). □

Theorem 2.3.5. One-parameter exponential family. Suppose

p (x, θ) = exp (T (x) η (θ) − B (θ)) · h (x) ,
where θ = Eθ T and η (θ) ∈ Int (N ). Then
1
I (θ) = , and I (η) = Varη T.
Varθ T

Proof.
Z
eηT (x)−A(η) h(x)µ(dx) = 1
′′
and A′ (η) = Eη T , A (η) = V arη T . Hence θ = A′ (η).
T (x)η−A(η)
Now π(x, η) = e h(x) (This is the canonical representation of the density of the
∂ log π(x,η)
exponential family. See (1.3.7)). Hence
∂η
= T (x) − A′ (η), so

I ∗ (η) = Eη (T (X) − A′ (η))2 = V arη T

46 2. UNBIASEDNESS

′′
Since θ = A′ (η) = h−1 (η), and we have I ∗ (η) = I(θ(η))(A (η))2 . So

′′
I(θ) = I ∗ (η(θ))/A (η(θ))2
= V arθ T /(V arθ T )2
1
=
V arθ T

Remark 2.3.6. T attains the CR lower bound in this case. A converse result also holds:
under some regularity conditions, attainment of the CR lower bound implies that T is
the natural sucient statistic of some exponential family {Pθ }.

Example 2.3.7. Poisson family. Suppose that X1 , . . . , X n are iid Poisson. Then

P 1
p(x, θ) = e−nθ θ xi
Qn
1 xi !
P
xi 1
= e−nθ+n log θ n Qn η = n log θ, θ = eη/n
1 xi !
η/n
P
xi 1
π(x, η) = e−ne +η n Qn
1 xi !
P
xi
T (x) = is UMVU for θ
n
A′ (η) = eη/n = θ
′′ 1 η/n θ
A (η) = e =
n n
θ
I ∗ (η) = V arη T =
n
1 n
I(θ) = =
V arθ T θ

The information on n log θ increases with θ. The information on θ decreases with θ.

Theorem 2.3.8. Alternative version of CRLB theorem.

Suppose Ω is an open interval, A = {x : p(x, θ) > 0} is independent of θ, ∂p
∂θ
(x, θ) is nite
for all x ∈ A and for all θ ∈ Ω, Eθ ∂θ log p(x, θ) = 0, ∂θ (Eθ T ) = T (x) ∂θ (x, θ)µ(dx).
∂ ∂ ∂p
R
Then
∂
( ∂θ Eθ T )2
V arθ (T (X)) ≥
I(θ)
 2.4. MULTIPARAMETER CASE 47

∂
Proof. Choose ψ(x, θ) = ∂θ
log p(x, θ) in (2.3.1).

  
∂ ∂
Covθ T, log p(X, θ)
= Eθ T log p(X, θ)
∂θ ∂θ
Z
∂p(X, θ)
= T (x) µ(dx)
∂θ
∂
= (Eθ T )
  ∂θ
∂
V arθ log p(X, θ) = I(θ).
∂θ

2.4. Multiparameter Case

Here we consider the generalization to vector θ = (θ1 , . . . , θs ) ∈ Ω. The estimator T may

be either scalar (when estimating a scalar function of θ) or vector. (In Ch. 4 we will
consider generalizations to estimation of vector g(θ) based on vector T .)

Theorem 2.4.1. For T (X) , ψ1 (X, θ) , · · · , ψs (X, θ) functions with nite 2nd moments
under Pθ , we have the multiparameter analogue of (2.3.1),

Varθ (T ) ≥ γ T C −1 γ,

where γ T = (γ1 , · · · , γs ), γi = Covθ (T, ψi ) and C = [Covθ (ψi , ψj )]si,j=1

Proof. Let Ŷ denotethe minimum

 mean squared error linear predictor, of Y =
ψ1 − Eθ ψ1
T − Eθ T in terms of Ψ=  ··· . Then
ψs − Eθ ψs

Ŷ = aT Ψ
 
where Eθ (Y − Ŷ )(ψj − Eθ ψj ) = 0, j = 1, . . . , s, i.e.

Ca = γ = Covθ (T, Ψ)

These equations have a solution and hence

a = C −1 γ
48 2. UNBIASEDNESS

where C −1 is any generalized inverse of C = Eθ (ΨΨ′ ). So Ŷ = γ T C −1 Ψ.

Also

E(Y − Ŷ )2 = EY 2 − E Ŷ 2 since Ŷ ⊥ Y − Ŷ in L2 (Pθ )

= V arθ T − E(γ T C −1 ΨΨT C −1 γ)
= V arθ T − γ T C −1 γ
∴ E(Y − Ŷ )2 = V arθ T − γ T C −1 γ ≥ 0
∴ V arθ T ≥ γ T C −1 γ

Notice that the right hand side is the same for any generalized inverse C −1 of C. □

Generalization of the Information Inequality

Assume that
Ω is an open interval in Rs


 A = {x : p(x, θ) > 0} is independent of θ

∂p
(x, θ) is nite ∀x ∈ A, ∀θ ∈ Ω, i = 1, . . . , s
 ∂θi ∂


Eθ ∂θi log p(X, θ) = 0, i = 1, . . . , s

Definition 2.4.2. The information matrix.

  s
∂ ∂
I(θ) := Eθ log p(X, θ) log p(X, θ)
∂θi ∂θj i,j=1
  s
∂ ∂
= Covθ log p(X, θ) log p(X, θ)
∂θi ∂θj i,j=1

I(θ) is strictly positive denite if { ∂θ∂ i log p(X, θ), i = 1, . . . , s} is linearly independent
a.s. Pθ .
Theorem 2.4.3. Under the previous assumptions, if I(θ) is strictly positive denite and
if T (X) satises
Eθ T (X)2 < ∞ ∀θ
and Z
∂ ∂
Eθ T (X) = T (x) p(x, θ)µ(dx)
∂θj ∂θj
then
V arθ T (X) ≥ γ T I(θ)−1 γ
∂
 
E T (X)
∂θ1 θ
where γ =  ··· 
∂
E T (X)
∂θs θ
 2.4. MULTIPARAMETER CASE 49

Proof. This is a direct application of Theorem 2.4.1 with the functions ψi dened
by
∂
ψi (x, θ) = log p(x, θ)
∂θi
□

Reparameterization. If θi = fi (α1 , . . . , αs ), i = 1, . . . , s, then we have:

s
∂ log p∗ ∂ log p∗
 
∗
I (α) = E (X, α) (X, α)
∂αi ∂αj i,j=1
"   #s
X X ∂θm ∂ log p ∂ log p ∂θn
= E (X, θ) (X, θ)
n m
∂αi ∂θm ∂θn ∂αj
i,j=1
T
= JI(θ)J
h is
∂θj
where J= ∂αi
.
i,j=1

Corollary 2.4.4. If I(θ) is strictly positive denite, the elements of T = (T1 , . . . , Tn )

are nite variance unbiased for the respective elements of g(θ) = (g1 (θ), . . . , gn (θ)) and
each Ti satises the conditions of theorem 2.4.3, then, Eθ T = g(θ) and,
∂g ∂g
(2.4.1) Covθ T ≥ ( )I −1 (θ)( )T ,
∂θ ∂θ
 ∂g1
· · · ∂g 1

∂θ1 ∂θs
where A ≥ B means aT (A − B)a ≥ 0, ∀a ∈ Rn and ∂g ∂θ
:=  ··· .
∂gn ∂gn
∂θ1
· · · ∂θs

Proof. Since aT (Covθ T )a = V ar(aT T ), (2,4,1) is equivalent to

∂g ∂g
V arθ (aT T ) ≥ aT ( )I −1 (θ)( )T a ∀a ∈ Rn .
∂θ ∂θ
But the above inequality follows at once by applying theorem 2.4.3 to the real-valued
T
statistic a T for which
∂ ∂
E (aT T ) E (T T )
   
∂θ1 θ ∂θ1 θ
γ= ··· = ···  a = ( ∂g )T a.
∂ T ∂ ∂θ
E (a T )
∂θs θ
E (T T )
∂θs θ
□
Corollary 2.4.5. If T1 , · · · , Ts are unbiased for

θ1 , · · · , θs , then
Covθ (T ) ≥ I (θ)−1 .

∂g
Proof. Apply corollary 2.4.4 with gi (θ) = θi , i = 1, . . . , s (where
∂θ
= Is×s ). □
50 2. UNBIASEDNESS

Remark 2.4.6. Suppose we wish to estimate θ1 . If θ2 , . . . , θs are known then the CR

lower bound is "  2 #−1
∂ log p
Eθ (X, θ) .
∂θ1
If θ2 , . . . , θs are not known, then the CR bound for estimating θ1 is the (1,1) component
−1
of I (θ), denoted by I −1 (θ)1,1 (by Corollary 2.4.4 with T = θ1 and g(θ) = θ1 ). Naturally
we expect
"  2 #−1
∂log
Eθ p(X, θ) ≤ I −1 (θ)1,1
∂θ1
By the general formula for the inverse of a partitioned matrix,

−DA12 A−1
 
−1 D 22
A = ,
−A−1
22 A21 D A −1
22 A21 DA 12 A−1
22

where D = (A11 − A12 A−1 −1

22 A21 ) , we nd that

1 1
I −1 (θ)1,1 = ≥ .
a − bT A−1 b a
Theorem 2.4.7. (Order-s exponential family). Suppose that
s
!
X
p (x, θ) = exp ηi (θ) Ti (x) − B (θ) h (x) , θ∈Ω
1
is an order-s exponential family parameterized by
θ = Eθ T
and that η (Ω) contains an open subset of Rs . Then, if C = Cov (T ), we have
I (θ) = C −1 and I ∗ (η) = C.

Proof. By theorem 1.3.14 of chapter 1, we know that for η ∈ int(N )

s
∂ 2A ∂A2 ∂ 2A

= cov(Ti , Tj ), cov(T ) = :=
∂ηi ∂ηj ∂η∂η T ∂ηi ∂ηj i,j=1
We also know that if π(x, η) is the canonical form of the density

∂ log π(x, η) ∂ log π(x, η) ∂A ∂A

= (Ti − )(Tj − )
∂ηi ∂ηj ∂ηi ∂ηj
∂ 2A
∴ I ∗ (η) = cov(T ) =
∂η∂η T
 
∂A/∂η1
∂A
Moreover, θ = Eθ T = ∂η
=  · · · , and hence by the reparameterization formula
∂A/∂ηs
∂ 2A ∂ 2A
I ∗ (η) = I(θ)
∂η∂η T ∂η∂η T
 2.4. MULTIPARAMETER CASE 51

∂2A
But the left hand side is (a symmetric matrix) and hence
∂η∂η T
−1
∂ 2A

I(θ) = = C −1 .
∂η∂η T
□
Remark 2.4.8. Note that for a random sample of size
P P n from p(x, θ), the CSS is
( T1 (Xi ), . . . , Ts (Xi )), and the new A function is nA(η).

Examples of Information Matrices.

Example 2.4.9. X ∼ N (ξ, Σ), ξ ∈ Rs , Σ xed and non-singular. Then
 
1 1 T −1
p(x, ξ) = √ exp − (x − ξ) Σ (x − ξ)
( 2π)s |detΣ|1/2 2
Writing Σ = [σij ]si,j , Σ−1 = [γij ]si,j , we have

s
∂ log p X
(X, ξ) = γik (xk − ξk )
∂ξi k=1
s
∂ log p X
(X, ξ) = γjm (xm − ξm )
∂ξj m=1
  X s X s
∂ log p ∂ log p
∴ E (X, ξ) (X, ξ) = γik E(Xk − ξk )(Xm − ξm )γjm
∂ξi ∂ξj k=1 m=1
s X
X s
= γik σkm γmj
k=1 m=1
−1 −1
= Σ ΣΣ
= Σ−1 .
Example 2.4.10. The order-two exponential family {N (ξ, σ)}, ξ ∈ R; σ > 0}.
1 1 2
p∗ (x, ψ) = √ e− 2σ2 (x−ξ)
σ 2π
1 1 2 ξ ξ2
= √ e− 2σ2 x + σ2 x− 2σ2 −log σ
2π
 
ξ
where ψ= . By Theorem 2.4.7, if we let
σ
     
θ1 ξ X
θ= = = Eθ
θ2 ξ 2 + σ2 X2
52 2. UNBIASEDNESS

then we can rewrite p∗ (x, ψ) as

1
p(x, θ) = √ exp η1 (θ)x + η2 (θ)x2 − B(θ)

2π

   
X X
and since Eθ = θ = Eθ T, T = ,
X2 X2

I(θ) = [Cov(T )]−1

 
σ2 2σ 2 ξ
where Cov(T ) = (check from MGF).
2σ ξ 2σ + 4σ 2 ξ 2
2 4

We can now nd I ∗ (ψ) from the reparameterization formula,

I ∗ (ψ) = JI(θ)J T

∂θ1 ∂θs
 
∂ψ1
··· ∂ψ1    
.
.
.
.
1 2ξ ξ
where J = = ,θ= .
 
. .
∂θ1 ∂θs
0 2σ ξ 2 + σ2
∂ψs
··· ∂ψs

 
∗ −1 −1 T −1 −1 1 −ξ/σ −1
∴ I (ψ) = (J ) I (θ)J where J =
0 1/2σ
  2 2
 
1 0 σ 2σ ξ 1 −ξ/σ
=
−ξ/σ 1/2σ 2σ 2 ξ 2σ 4 + 4σ 2 ξ 2 0 1/2σ
 2 
σ 0
=
0 σ 2 /2
 
∗ 1/σ 2 0 ∗
∴ I (ξ, σ) = (could also be obtained directly from p )
0 2/σ 2

Example 2.4.11. Location-scale families.

Suppose that f is a probability density with respect to Lebesgue measure, that f (x) is
strictly positive for all x and that

1 x−µ
p(x, µ, σ) = f( ), µ ∈ R, σ > 0.
σ σ
 2.4. MULTIPARAMETER CASE 53

Then

∂ log p 1 f ′ ( x−µ
σ
)
(x, µ, σ) = − x−µ
∂µ σ f( σ )
∂ log p 1 x − µ f ′ ( x−µ σ
)
=− − x−µ
∂σ σ σ2 f ( σ )
Z  ′ x−µ 2
1 f( σ ) 1 x−µ
I11 (µ, σ) = 2 x−µ f( )dx
σ f( σ ) σ σ
Z  ′ 2
1 f (x)
= 2 f (x)dx
σ f (x)
2
x − µ f ′ ( x−µ
Z 
1 σ
) 1 x−µ
I22 (µ, σ) = 2 1+ x−µ f( )dx
σ σ f( σ ) σ σ
2
f ′ (x)
Z 
1
= 2 1+x f (x)dx
σ f (x)
Z ′ x−µ 2
x − µ f ′ ( x−µ

f ( σ )1 x−µ ) 1 x−µ
Z
1 1 σ
I12 (µ, σ) = 2 x−µ f( )dx + 2 x−µ f( )dx
σ f( σ ) σ σ σ σ f( σ ) σ σ
Z  ′ 2
1 f (x)
=0+ 2 x f (x)dx
σ f (x)
Thus I is a diagonal matrix if f is symmetric about 0. Example 2.4.9 is a special case of
this.

When the CR bound in theorem 2.4.3 is not sharp, it can be improved by using higher
order derivatives of ψ. This is the content of the following theorem.

Theorem 2.4.12. (Bhattacharya bounds) Suppose p(x, θ) have common support for
any θ. Let T be unbiased for g(θ). Further assume

∂i
Z
T (x) i p(x, θ)µ(dx) = g (i) (θ), i = 1, . . . , k
∂θ
and
∂i
Z
p(x, θ)µ(dx) = 0, i = 1, . . . , k
∂θi
Then
T
V arθ (T ) ≥ g (1) (θ), . . . , g (k) (θ) V −1 g (1) (θ), . . . , g (k) (θ)
  
 T 
∂k
where V = Cov p(x,θ) ∂θ p(x, θ), . . . , p(x,θ) ∂θk p(x, θ)
1 ∂ 1
, and V is assumed to be non-
singular.
54 2. UNBIASEDNESS

Proof. This follows immediately from theorem 2.4.1 by taking

1 ∂i
ψi (x, θ) = p(x, θ), i = 1, . . . , k
p(x, θ) ∂θi
giving

γi = Cov(T, ψi ) = E(T ψi )
∂i
Z
= T (x) i p(x, θ)µ(dx) = g (i) (θ).
∂θ
□
Example 2.4.13. X1 , . . . , Xn iid Poisson(θ ), θ > 0, and g(θ) = e−θ = P (Xi = 0). Then
I0 (X) is unbiased for g(θ). Here
Y 1 P
p(x, θ) = e−nθ+( xi ) log θ
, θ > 0,
xi !
P
is an exponential family of full rank so T (X) = Xi is complete and sucient. So the
UMVU estimator of g(θ is E [I0 (X1 )|T (X)] and

E [I0 (X1 )|T (X) = t] = P (X1 = 0|T (X) = t)

P (X1 = 0 ∩ T (X) = t)
=
P (T (X) = t)
e−θ e−(n−1)θ ((n − 1)θ)t /t!
=
e−nθ (nθ)t /t!
1
= (1 − )t ,
n
so (1 − n1 )T (X) is UMVU.

Noting that T (X) ∼ P (nθ), we can write its probability generating function as Ez T (X) =
e−nθ(1−z) , and hence
1 T (X)
Eθ (1 − ) = e−θ and
n
1 2T (X)
Eθ (1 − ) = exp(−nθ(1 − (1 − 1/n)2 ))
n
θ
= e−2θ+ n .
1 θ
∴ V arθ (1 − )T (X) = e−2θ (e n − 1).
n
g ′ (θ)2
For this problem CRLB = nI1 (θ)
g(θ) = e−θ and
, where
 2
X1 1 1
(2.4.2) I1 (θ) = E −1 + = 2 V arθ X =
θ θ θ
−2θ
θe θ θ 1 θ2
∴ CRLB = < e−2θ (e n − 1) = e−2θ ( + + ···)
n n 2 n2
 2.4. MULTIPARAMETER CASE 55

(Alternatively T is the CSS for a full-rank exponential family with Eθ (T ) = θ, so I(θ) =

1
V arθ T
= nθ = nI1 (θ).)

The Bhattacharya bound with k = 2,

−e−θ
   
g (1) (θ)
=
g (2) (θ) e−θ
P
1 ∂p ∂ xi
ψ1 (x, θ) = = log p = −n +
p ∂θ ∂θ θ
2 2
P P
1∂ p ∂ 1 ∂p 2 xi xi
ψ2 (x, θ) = 2
= 2 log p + ( ) =− 2 +( − n)2
p ∂θ ∂θ p ∂θ θ θ
 
n/θ 0
Covθ ψ(X, θ) =
0 2n2 /θ2
Hence the Bhattacharya bound is

−e−θ
  
n/θ 0
−e−θ e−θ
 
V arθ T ≥
0 2n2 /θ2 e−θ
θ θ2
= e−2θ ( + 2 ) > CRLB
n 2n
but less than V arθ T . By taking more derivatives the bound can be make arbitrarily close
to V arθ T . Extends to the multiparameter case also.
 CHAPTER 3

Equivariance

3.1. Equivariance for Location family

In chapter 2 we introduced unbiasedness as a constraint to eliminate estimators which

may do well at a particular parameter value at the cost of poor performance elsewhere.
Within this limited class we could then sometimes determine uniformly minimum risk
estimators for any convex loss function. Equivariance is a more physically motivated
restriction. We start by considering how equivariance enters as a natural constraint on
statistics used to estimate location parameters.
Suppose X = (X1 , . . . , Xn ) has density

f (x − ξ) = f (x1 − ξ, · · · , xn − ξ)
where f is known, ξ is a location parameter to be estimated and L (ξ, d) is the loss when
ξ is estimated as d. Suppose we have settled on T (x) as a reasonable estimator of ξ as
measured by
R (ξ, T ) = Eξ (L (ξ, T (x)))
Suppose another statistician B wants to measure the data using a dierent origin. So
′ ′
instead of recording X1 , X, · · · , Xn , B records X1 = X1 + 273, · · · , Xn = Xn + 273 say
◦ ◦
(This would be the case if we measured temperatures in C and B measured them in K.)
Then

(3.1.1) X ′ = X + a.
On the new scale the location parameter becomes

(3.1.2) ξ′ = ξ + a
and the joint density of X ′ = (X1′ , · · · , Xn′ ) is
′ ′
f (x − ξ ) = f (x′1 − ξ ′ , · · · , x′n − ξ ′ ) .
The estimated value d on the original scale becomes

(3.1.3) d′ = d + a
on the new one and the loss resulting from its use is L (ξ ′ , d′ ). The problem of es-
timating ξ is said to be invariant under the transformations (3.1.1),(3.1.2),
and (3.1.3) if
(3.1.4) L (ξ + a, d + a) = L (ξ, d) .
This condition on L is equivalent to the assumption that L has the functional form,

56
 3.1. EQUIVARIANCE FOR LOCATION FAMILY 57

L (ξ, d) = ρ (d − ξ)
for some function ρ. This is called an invariant loss function.
Suppose we chose T (X) as a good estimator of ξ in the original scale. Then since
ξ ′ in terms of X1′ , · · · , Xn′ is exactly the same problem, we should use
estimation of

T ′ (X1′ , · · · , Xn′ ) = T (X1 , · · · , Xn ) + a

as our estimator of ξ ′ = ξ + a. If

T (X1 + a, · · · , Xn + a) = T (X1 , · · · , Xn ) + a
then we say that the estimator T is equivariant under the transformations
(3.1.1),(3.1.2), and (3.1.3) or location equivariant . Let E denote the class of all
equivariant estimators.
P
Remark 3.1.1. The mean, median, weighted average of order statistics (with wi = 1)
and the MLE of ξ for the family f (x − ξ) are all location equivariant.

Theorem 3.1.2. If X has density f (x − ξ1) with respect to Lebesgue measure µ and T
is equivariant for ξ with loss
L (ξ, d) = ρ (ξ − d) .
Then the bias, risk and variance of T are independent of ξ .

Proof. We give the proof for the bias. The other proofs are similar.

b (ξ) = Eξ T (x) − ξ
Z
= (T (x) − ξ) f (x − ξ1) µ (dx)
Z
= (T (x + ξ1) − ξ) f (x) µ (dx) by shift-invariance of µ

= E0 T (X ′ ) − ξ
= E0 (T (X) + ξ) − ξ
= E0 T (X) .

□
Remark 3.1.3. Since the risk of an equivariant estimator is independent of θ, the de-
termination of a uniformly minimum risk equivariant estimator reduces to nding the
equivariant estimator with minimum (for every θ) risk - such an estimator typically ex-
ists - and is called a minimum risk equivariant (MRE) estimator. Our rst step is to nd
a representation of all location equivariant estimators (Just as we found a representation
of all unbiased estimators in Chap 2)

Lemma 3.1.4. If T0 is any location-equivariant estimator then

(3.1.5) T is equivariant ⇐⇒ T (x) = T0 (x) − U (x)
58 3. EQUIVARIANCE

where U (x) is any function such that

(3.1.6) U (x + a1) = U (x) for all x and a.

Proof. If T is location-equivariant, set

U (x) = T0 (x) − T (x) .

Then

U (x + a1) = T0 (x + a1) − T (x + a1)

= T0 (x) + a − T (x) − a
= U (x) .
Conversely if equation (3.1.5) and (3.1.6) hold, then

T (x + a1) = T0 (x + a1) − U (x + a1)

= T0 (x) + a − U (x)
= T (x) + a.
□
Lemma 3.1.5. U satises
U (x + a1) = U (x) for all x and a
if and only if
U (x) = v (x1 − xn , · · · , xn−1 − xn ) for some function v.

Proof. ⇐)
U (x + a1) = v ((x1 + a) − (xn + a) , · · · , (xn−1 + a) − (xn + a)) = U (x) .

⇒) Choosing a = −xn , we have

U (x) = U (x + a1) = U (x1 − xn , · · · , xn−1 − xn , 0)

= v (x1 − xn , · · · , xn−1 − xn ) .

Combining these two lemmas gives the following theorem. □

Theorem 3.1.6. If T0 is any location-equivariant estimator, then a necessary and suf-
cient condition for T to be equivariant is that there is a function v of n − 1 variables
such that
T (x) = T0 (x) − v (y) for all x,
where y = (x1 − xn , · · · , xn−1 − xn ).
Example 3.1.7. If n = 1, then only equivariant estimators are X +c for c ∈ R.

Now we can determine the location-equivariant estimator with minimum risk.

 3.1. EQUIVARIANCE FOR LOCATION FAMILY 59

Theorem 3.1.8. Let x have a density function f (x − θ) with respect to Lebesgue measure
and let y = (y1 , · · · , yn−1 )T where yi = xi − xn . Suppose that the loss function is given
by L (θ, d) = ρ (d − θ) and that there exists an equivariant estimator T0 with nite risk.
Assume that for each y there exists a number v (y) = v ∗ (y) which minimizes
(3.1.7) E0 (ρ (T0 (X) − v (Y )) |Y = y)
then there exists an MRE estimator T ∗ of θ given by
T ∗ (x) = T0 (x) − v ∗ (y) .

Proof. If T is equivariant then

T (X) = T0 (X) − v (Y )
for some v. So to nd the MRE, we need to nd v to minimize

R (θ, T ) = Eθ (ρ (T − θ))
and we calculate:

R (θ, T ) = Eθ (ρ (T0 (X) − v (Y ) − θ))

= E0 (ρ (T0 (X) − v (Y ))) by theorem (3.1.2)
Z
= E0 (ρ (T0 (X) − v (Y )) |Y = y) dP0 (y)
Z
≥ E0 (ρ (T0 (X) − v ∗ (Y )) |Y = y) dP0 (y)
= R (0, T ∗ ) .
The risk is nite since R (0, T ∗ ) ≤ R (0, T0 ) < ∞ by assumption. □
Corollary 3.1.9. If ρ is convex and not monotone, then an MRE exists and is unique
if ρ is strictly convex (under the conditions of theorem (3.1.8)).

Proof. Let

ϕ (c) = E (ρ (T0 (X) − c) |Y = y)

and apply theorem (1.4.2). □
Corollary 3.1.10. The following results hold:

(1) ρ (d − θ) = (d − θ)2 ⇒ v ∗ (y) = E0 (T0 (X) |Y = y)

(2) ρ (d − θ) = |d − θ| ⇒ v ∗ (y) =med0 (T0 (X) |Y = y)

E0 (ρ (T0 (X) − c) |Y = y) = E0 (T0 (X) − c)2 |Y = y

Proof. (1) is mini-
mized at c = E0 (T0 (X) |Y = y).
(2) E0 (|T0 (X) − c| |Y = y) is minimized at c = med0 (T0 (X) |Y = y).
□
60 3. EQUIVARIANCE

Example 3.1.11. MRE's can exist also for non-convex ρ. Suppose ρ is given by

1 if |d − θ| > c
ρ (d − θ) =
0 otherwise,

where c is xed. Then for n = 1, using T0 (X) = X , v will minimize

E0 ρ (X − v) = P0 (|X − v| > c)
if and only if it maximizes
P0 (|X − v| ≤ c) .
∗
If f is symmetric and unimodal then v = 0 and hence

T0 (X) − 0 = X is MRE.

On the other hand if f is U-shaped, say f (x) = (x2 + 1) I[−L,L] where c < L, then

P0 (|X − v| ≤ c) = P0 (v − c ≤ X ≤ v + c)
is maximized at v+c = L and v − c = −L. Thus there are two MREE's, X −L+c and
X + L − c.
Example 3.1.12. Let X1 , · · · , Xn be iid N (µ, σ 2 ) where σ 2 is known. Because X is
complete sucient and Y = (X1 − Xn , · · · , Xn−1 − Xn ) is ancillary, T0 (X) = X is
independent of Y. Therefore, by minimizing the expression (3.1.7), we nd that

v ∗ (y) = argmin E0 ρ X − v .





If ρ is convex and even, then ϕ (v) := E0 ρ X − v is convex and even. Therefore,

v ∗ (y) = 0 and X is MRE. (X is also MRE when ρ is the non-convex function of Example
3.1.11.)

Theorem 3.1.13. (Least favorable property of the normal distribution) Let F be the
set of all distributions with pdf relative to Lebesgue measure and with variance 1. Let
X1 , · · · , Xn be iid with pdf f (x − ξ), where ξ = EXi . If rn (F ) is the risk of the MRE
estimator of ξ with squared error loss, then rn (F ) has its maximum value over F when
F is normal.

Proof. By the previous example, the MREE in the normal case is X with corre-
sponding risk,
2 1
rn = E0 X = .
n
However, X is an equivariant estimator of ξ for all F ∈ F , and the risk of X is

2 1
Eξ X − ξ = , for all F ∈ F.
n
Therefore, we must have
1
rn (F ) ≤ , for all F ∈ F.
n
□
 3.1. EQUIVARIANCE FOR LOCATION FAMILY 61

Remark 3.1.14. From corollary (3.1.10), the MRE in general in the previous theorem is


X − E0 X|Y = y .


But for n ≥ 3, E0 X|Y = y = 0 if and only if F is normal (Kagan, Linnik, Rao (1965,

1973)). So the MRE for n ≥ 3 is X if and only if F is normal.

Example 3.1.15. Let X1 , · · · , Xn be iid with


1 − e−(x−θ)/b x ≥ θ
F (x) =
0 x < θ,
where b is known and θ ∈ R. Then T0 = X(1) is equivariant, complete and sucient for
θ (CHECK) and T0 is independent of the ancillary statistic

Y = (X1 − Xn , · · · , Xn−1 − Xn ) = (X1 − θ − (Xn − θ), · · · , Xn−1 − θ − (Xn − θ)) .

Therefore X(1) − v ∗ is MRE if v∗ minimizes


E0 ρ X(1) − v .

(In general we have to minimize E0 (ρ (T0 − v (y)) |Y = y) for each y by Theorem 3.1.8
but here the complete suciency of T0 and the ancillarity of Y implies that v ∗ (y) is
independent of y since the distribution of T0 is the same for all y .)

We now consider some special cases:

ρ (d − θ) = (d − θ)2 , then v = E0 X(1) = b

(1) If

b/n so that MRE is X(1) − n .
(2) If ρ (d − θ) = |d − θ|, then v = med0 X(1) = b log 2/n so that MRE is X(1) −
b log 2 −(x−θ)n/b
n
. (Because FX(1) (x) = 1 − e = 21 implies (x − θ) n/b = log 2.)
(3) If

1 if |d − θ| > c,
ρ (d − θ) =
0 if |d − θ| ≤ c,


then v is the center of the interval I of length 2c which maximizes P0 X(1) ∈ I
so that v=c and the MREE is X(1) − c.
Theorem 3.1.16. ( Pitman Estimator) Under the assumptions of Theorem 3.1.8, if
L (θ, d) = (d − θ) , 2
R
∗ uf (x1 − u, · · · , xn − u) du
T = R
f (x1 − u, · · · , xn − u) du
is the MREE of θ.
Remark 3.1.17. This coincides with the Bayes estimator corresponding to an improper
at prior for the location parameter. (i.e. the conditional expectation of Θ given X=x
under the assumed joint "density" f (x − θ1) of X and Θ.)

Proof. Corollary (3.1.10) implies

T ∗ (X) = T0 (X) − E0 (T0 |Y ) ,

62 3. EQUIVARIANCE

where T0 is any equivariant estimator. Let T0 (X) = Xn . Because

Y1 1 · · · 0 −1 X1
    
 ...   ... . . . ... . 
.
. 
.
.
.

 = ,
 Y   0 · · · 1 −1   X 
n−1 n−1
Xn 0 ··· 0 1 Xn
we have

fY1 ,··· ,Yn−1 ,Xn (y1 , · · · , yn−1 , xn ) = fX1 ,··· ,Xn (y1 + xn , · · · , yn−1 + xn , xn )
and
R
xf (y1 + x, · · · , yn−1 + x, x) dx
E0 (Xn |Y = y) = R
f (y1 + x, · · · , yn−1 + x, x) dx
R
xf (x1 − xn + x, · · · , xn−1 − xn + x, x) dx
= R
f (y1 − xn + x, · · · , xn−1 − xn + x, x) dx
R
uf (x1 − u, · · · , xn−1 − u, xn − u) du
= xn − R .
f (x1 − u, · · · , xn−1 − u, xn − u) du
Substituting in the expression for T∗ completes the proof. □
Example 3.1.18. Suppose f (x) = I(−1/2,1/2) (x) and X1 , · · · , Xn are iid with density
f (x − θ) = I(θ−1/2,θ+1/2) (x). Then

1 1

1 if − ≤ x(1) andx(n) ≤
f (x1 , · · · , xn ) = 2 2
0 otherwise,

and
1 1

1 if u− ≤ x(1) x(n) ≤ u +
and
f (x − u1) = 2 2 .
0 otherwise,

Therefore, since x(n) − x(1) ≤ 1,

R x(1) + 21
x(n) − 12
udu
T ∗ (x) = R x(1) + 12
x(n) − 12
du
 
1 1 2 1 2



2
x(1) + 2
− x(n) − 2
=
x(1) − x(n)
1

= x(1) + x(n) .
2

Remark 3.1.19. ( UMVU vs. MRE)

• MRE estimators often exist for more than just convex loss functions.
• For convex loss functions MRE estimators generally vary with the loss function
(unlike UMVUE's).
 3.1. EQUIVARIANCE FOR LOCATION FAMILY 63

• UMVUE's are frequently inadmissible (i.e., there exists an estimator with uni-
formly smaller risk). The Pitman estimator is admissible under mild assump-
tions.
• The principal application of UMVUE's is to exponential families.
• For location problems, UMVUE's typically do not exist.
• An MRE estimator is not necessarily unbiased. The following lemma examines
this connection.

Lemma 3.1.20. Suppose L (d, θ) = (d − θ)2 , and f (x − θ1) , θ ∈ R, are densities with
respect to Lebesgue measure.

(1) If T is equivariant with bias b, then T − b is equivariant, unbiased, with smaller

risk than T .
(2) The unique MRE estimator is unbiased.
(3) If there exists an UMVU estimator and it is equivariant, then it is MRE.

Proof. (1) It is clear that T −b is equivariant and unbiased. For the smaller
risk part,

R (θ, T − b) = R (0, T − b) = E0 (T − b)2 = VarT ≤ E0 T 2 = R (0, T ) = R (θ, T ) .

(2) The MRE estimator is unique by corollary (3.1.9). It is unbiased by (1), other-
wise its risk could be improved by using the equivariant estimator T − b.
(3) The UMVUE is the unique MR estimator in U. If it falls in E it is the MR
estimator in U ∩ E. But the MREE is the MR estimator in U ∩ E since it is
necessarily unbiased. Hence they are the same.

U = {unbiased estimators of zero}, E = {equivariant estimators}.

□
Definition 3.1.21. An estimator T of g (θ) is risk-unbiased if

Eθ L (θ, T ) ≤ Eθ L (θ′ , T ) for all θ ̸= θ′ .

i.e. T is closer to g (θ) on average than to any false value g (θ′ ).
Example 3.1.22. (mean unbiasedness) If L (θ, d) = (d − g (θ))2 and T is risk-unbiased,
then
2
Eθ (T − g (θ))2 ≤ Eθ (T − g (θ′ )) for all θ ̸= θ′ .
2 ′ 2
This means (assuming that Eθ T < ∞ and Eθ T ∈ g (Ω)) that Eθ (T − g (θ )) is mini-
′
mized by g (θ ) = Eθ T .

Hence g (θ) = Eθ T for all θ i.e. T is unbiased for g in the sense dened in chapter 2.

Example 3.1.23. (Median unbiasedness) If L (θ, d) = |d − g (θ)| and T is risk-unbiased,

then
Eθ |T − g (θ)| ≤ Eθ |T − g (θ′ )| for all θ ̸= θ′ .
′
But the right hand side is minimized when g (θ ) = medθ T .
64 3. EQUIVARIANCE

Hence

(3.1.8) g (θ) = medθ T for all θ.

(assuming that Eθ |T | < ∞ and there exists a medθ T in g (Ω) for all θ.) An estimator
satisfying equation (3.1.8) is said to be median-unbiased for g (θ).
Theorem 3.1.24. Suppose X has density f (x − θ1) with respect to Lebesgue measure.
If T is an MRE estimator with L (θ, d) = ρ (d − θ) then T is risk-unbiased (for θ).

Proof. By Theorem 3.1.2,

Eθ ρ (T − θ) = E0 ρ (T ) .
If θ ̸= θ′ , then T − θ′ is equivariant and by denition of MRE, we have E0 ρ (T ) ≤
E0 ρ (T − θ′ ) for all θ′
⇒ E0 ρ (T ) ≤ Eθ ρ (T − θ′ − θ) for all θ′
′
⇒ E0 ρ (T ) ≤ Eθ ρ (T − θ′ ) for all θ

⇒ Eθ ρ (T − θ) ≤ Eθ ρ (T − θ′ ) for all θ′ .

3.2. The General Equivariant Framework

Notation
X : data.
Ω: parameter space.
P : = {Pθ : θ ∈ Ω}.
G : a group of measurable bijective transformations of X → X.
Remark 3.2.1. The operation associated with the group G is function composition. i.e.

f g (x) = f ◦ g (x) = f (g (x)) .

Definition 3.2.2. We say that g leaves P invariant if for all θ ∈ Ω, there exists θ′ ∈ Ω
such that

X ∼ Pθ ⇒ gX ∼ Pθ′

and there exists θ∗ ∈ Ω such that

X ∼ Pθ∗ ⇒ gX ∼ Pθ .

If C is a class of transformations which leave P invariant then

G (C) = g1±1 g2±1 · · · gm

±1

: gi ∈ C, m ∈ N
is a group (the group generated by C ), each of whose member leaves P invariant.
 3.2. THE GENERAL EQUIVARIANT FRAMEWORK 65

If each member of a group G leaves P invariant we say that G leaves P invariant. If G

leaves P invariant and Pθ ̸= Pθ′ for θ ̸= θ′ then there exists a unique θ′ ∈ Ω such that
X ∼ Pθ ⇒ gX ∼ Pθ′ .
This denes a bijection g : Ω → Ω, via the relation

g (θ) = θ′ .
where Pg(θ) is the distribution of g (X) under θ.
Definition 3.2.3. Under the preceding conditions we dene

G := {g : g ∈ G} .
It is clear that G is also a group.

Remark 3.2.4. For g ∈ G,

Eθ f (gX) = Eg(θ) f (X)
since
Z
Eθ f (gX) = f (g (x)) Pθ (dx)
Z
= f (x) Pθ ◦ g −1 (dx)
Z
= f (x) Pg(θ) (dx)
= Eg(θ) f (X) .

Equivariant Estimation

Let T :X →D be an estimator of h (θ). Instead of observing X, suppose we observe

X ′ = g (X) ,
where X ′ is a sample from Pg(θ) . Suppose that for any g ∈ G , h (gθ) depends on θ only
through h (θ), i.e.

(3.2.1) h (θ1 ) = h (θ2 ) ⇒ h (gθ1 ) = h (gθ2 ) .

Then we denote

g ∗ h (θ) = h (gθ) .

It is clear that

G ∗ := g ∗ : g ∈ G


is a group and each g∗ is a 1 to 1 mapping from H = h (Ω) to H.

66 3. EQUIVARIANCE

Definition 3.2.5. Under the conditions prescribed for existence of the groups of map-
pings G and G ∗, if

(3.2.2) L (gθ, g ∗ d) = L (θ, d) for all g ∈ G,

then we say that L is invariant under G . (g g ∈ G '). If conditions
if we remove 'for all
(3.2.1) and (3.2.2) hold, we say that the problem of estimating h (θ) is invariant
under the group of transformations G (under g if we remove for all g ∈ G ). An
estimator T (X) of h (θ) is said to be equivariant under G if

(3.2.3) g ∗ T (X) = T (gX) for all g ∈ G.

If (3.2.2) and (3.2.3) hold and T (X) is a good estimator of h (θ) based on X then T (gX)
∗
will be a good estimator of g (h (θ)) based on g(X).

Example 3.2.6. (Location parameter) Let h (θ) = θ and g (X) = X + a.

X → X + a1
X + a ∼ Pθ+a θ → gθ = θ + a
h (gθ) = θ + a g ∗ (h(θ)) = h(θ) + a.
Then, the problem of estimating θ is location-invariant if
∗
L(gθ, g d) = L (θ + a, d + a) = L (θ, d) .
An estimator of h(θ) is equivariant if

T (X + a1) = T (X) + a,
−1
Pn
e.g. X1 , median(X), n i=1 Xi , etc.

Theorem 3.2.7. If T is equivariant and g leaves P invariant and

L (θ, d) = L (gθ, g ∗ d) ,
then
R (θ, T ) = R (g (θ) , T ) for all θ,
where
R (θ, T ) := Eθ L (θ, T (X)) .

Proof.

Eθ L (θ, T (X)) = Eθ L (g (θ) , g ∗ T (X))

= Eθ L (g (θ) , T (g (X)))
= Eg(θ) L (g (θ) , T (X))
= R (g (θ) , T ) .

□
Corollary 3.2.8. If G is transitive over Ω (i.e. if θ1 , θ2 ∈ Ω and θ1 ̸= θ2 then there
exists g ∈ G such that g (θ1 ) = θ2 ) then R (θ, T ) is constant for every equivariant T .
 3.2. THE GENERAL EQUIVARIANT FRAMEWORK 67

Proof. Fix θ0 ∈ Ω. If θ ̸= θ0 , there exists g such that g (θ0 ) = θ. Hence

R (θ0 , T ) = R (g (θ0 ) , T ) = R (θ, T ) .

□
Remark 3.2.9. P = {Pθ : θ ∈ Ω} is invariant relative to a transitive group of transfor-
mations if and only if P is a group family (see TPE section 1.4.1). These are families
generated by subjecting a r.v. with xed distribution to a group of transformations. We
can then index P using G. Thus

P = Pg : g ∈ G ,
where θ = g (θ0 ).
Theorem 3.2.10. Suppose G is transitive and G ∗ is commutative. If T is MRE, then T
is risk unbiased.

Proof. Let T be MRE. For θ ̸= θ′ , there exists g g (θ′ ) = θ.

such that Consequently,

Eθ (L (θ′ , T (X))) = Eθ L g −1 (θ) , T (X)

= Eθ L (θ, g ∗ T (X))
≥ Eθ L (θ, T ) .
(In the 2nd equality, note that if T is equivariant then so is g∗T , since letting h∗ ∈ G ∗ ,
we have
h∗ g ∗ T = g ∗ h∗ T = g ∗ T (h) ,
by commutativity.) The inequality then follows because T is MRE. □
Example 3.2.11. Suppose X ∼ N (µ, σ 2 ), θ = (µ, σ 2 ), and we want to estimate h (θ) = µ.

(1) Let G1 = {g : gx = x + c, c ∈ R}, then P is invariant under G1 . T x = x + c,

c ∈ R, are the only equivariant functions since
T (x + a) = T (x) + a for all a
T (x+a)−T (x)
⇒ a
=1 for all a
′
⇒ T (x) = 1
⇒ T (x) = x + c.
(a) Suppose L (d, θ) = (d − µ)2 /σ 2 (squared loss function measured in units of
σ ). Then X is MRE under G1 , i.e.

(X − µ)2
Eθ L (θ, X) = Eθ
σ2
= 1 ( )
(T − µ)2
= min E0 : T equivariant with respect to G1 .
σ2
68 3. EQUIVARIANCE

Because

µ, σ 2 = µ + c, σ 2 ,




g
G1 is not transitive and X is not risk unbiased. To see this, for θ xed
choose θ′ = (µ, 10σ 2 ), whence

′ (X − µ)2 1
Eθ L (θ , X) = Eθ 2
= < Eθ L (θ, X) = 1.
10σ 10
Here, G 1 is not transitive since there exists no g such that g (µ, 10σ 2 ) =
2
g (µ, σ ) .
′ 2
(b) Suppose L (d, θ) = (d − µ) . X is MRE and risk-unbiased since

2
Eθ L′ (θ′ , X) = Eθ (X − µ′ ) ≥ Eθ (X − µ)2 for all θ ̸= θ′ .
G 1 transitive is therefore not necessary in theorem (3.2.10)
2 2 2
(2) Let G2 = {g : gx = ax + c, a > 0, c ∈ R}. Then, G 2 = {g : g (µ, σ ) = (aµ + c, a σ )}
2 2 2 2
since gX ∼ N (aµ + c, a σ ). Also h (θ) = µ so that h (g (θ)) = h (aµ + c, a σ ) =
ah (θ) + c . Thus,
g ∗ h (θ) = aµ + c and G2∗ = {g ∗ : g ∗ (d) = ad + c} .
P is invariant under G2 and T (x) = x is equivariant under G2 since

T gx = T (ax + c) = ax + c = g ∗ (T (x))
and

g ∗ (T x) = g ∗ (x) = ax + c.

X is MRE under G1 ⊂ G2 and is therefore MRE under G2 . (There exist

no G1 -equivariant G2 -estimators
estimators with smaller risk so there exist no
with smaller risk since G2 -equivariance is a more severe restriction than G1 -
2 2
equivariance.) But X is not risk unbiased with respect to L (d, θ) = (d − µ) /σ
∗
. G 2 is transitive in this case but G2 is not commutative. So, theorem (3.2.10)
cannot be applied here.

3.3. Location-Scale Families

Suppose
 
T −n x1 − µ xn − µ
X = (X1 , · · · , Xn ) ∼ σ f ,··· , ,
σ σ
where f is known. The density is with respect to Lebesgue measure and

θ = (µ, σ) ∈ Ω = R × R+ ,
where µ is the location parameter and σ is the scale parameter.
 3.3. LOCATION-SCALE FAMILIES 69

Estimation of µ. Our rst objective is to estimate the location parameter. Suppose

X = Rn , Ω = R × R+ , D = R.
Dene ga,b : Rn → Rn by

ga,b (x) = (ax1 + b, · · · , axn + b) .

Then g a,b : Ω → Ω is dened by

g a,b (θ) = (aµ + b, aσ)

∗
and ga,b :R→R is dened by
∗
ga,b (d) = ad + b.
since g ∗ h (θ) = g ∗ µ = h(gθ) = aµ + b.
Lemma 3.3.1. L (θ, d) is invariant under G if and only if
 
d−µ
L ((µ, σ) , d) = ρ
σ
(i.e. if L is a function of error measured in terms of σ ).

d−µ


Proof. ⇐) If ρ σ
= L ((µ, σ) , d), then

L (gθ, g ∗ d) = L ((aµ + b, aσ) , ad + b)

 
ad + b − aµ − b
= ρ
aσ
= L (θ, d) .

d−µ d′ −µ′
⇒) We need to show if µ, σ, d and µ′ , σ ′ , d′ satisfy
σ
= σ′
and if L is invariant then

L ((µ, σ) , d) = L ((µ′ , σ ′ ) , d′ ) .
But this holds since 
σ ′ σ ′
d = σ ′ d + µ − σ ′ µ

σ = σσ′ σ ′
µ = σ µ ′ + µ − σ µ ′

σ′ σ′
and L is invariant under this transformation by assumption. □

Now observe that P is invariant under each g∈G (check!)

T is equivariant ⇐⇒ g ∗ T (x) = T g (x)

⇐⇒ aT (x) + b = T (ax + b) .
Since G is transitive (given any (µ, σ) we can nd a, b such that (aµ + b, aσ) = (µ′ , σ ′ )),
every equivariant estimator of µ has constant risk by corollary (3.2.8).
70 3. EQUIVARIANCE

Proposition 3.3.2. If T0 is an equivariant estimator of µ and δ1 ̸= 0, and if

δ1 (aX + b) = aδ1 (X) for all a > 0, b ∈ R

then T is equivariant if and only if

T (X) = T0 (X) − W (Z) δ1 (X)

for some W where

 
X 1 − Xn Xn−2 − Xn Xn−1 − Xn
Z = (Z1 , · · · , Zn−2 , Zn−1 ) = ,··· , , .
Xn−1 − Xn Xn−1 − Xn |Xn−1 − Xn |
Note 3.3.3. (1) We have assumed that Xi ̸= Xj for all i ̸= j . This is ok since
{X : Xi = Xj for some i ̸= j} has measure 0.
(2) Z is ancillary for µ so if T is a function of a CSS then T is independent of Z.

Proof. (1) T is equivariant (i.e. T (ax + b) = aT (x) + b) ⇐⇒ U := T −Tδ1

0

satises U (ax + b) = U (x) for all a > 0,b ∈ R. (check the details - simple
algebra)
(2) If U = T −T
δ1
0
= W (Z) for some W then it is easy to see that U (ax + b) = U (x)
and hence by (1), T is equivariant.
(3) If T is equivariant then U (ax + b) = U (x) for all a > 0, b ∈ R. Setting

1 Xn
a= b=− ,
|Xn−1 − Xn | |Xn−1 − Xn |

 
X1 − Xn Xn−1 − Xn
⇒ U (X) = U ,··· , ,0
|Xn−1 − Xn | |Xn−1 − Xn |
= U (Z1 Zn−1 , . . . , Zn−2 Zn−1 , Zn−1 , 0)
=: V (Z1 , . . . , Zn−1 ) .

Theorem 3.3.4. Suppose T0 is equivariant with nite risk and δ1 and Z are as dened
in Proposition 3.3.2. If

E(0,1) (ρ (T0 (X) − W (Z) δ1 (X)) |Z)

is minimized when W (Z) = W ∗ (Z) then

T (X) = T0 (X) − W ∗ (Z) δ1 (X)

is MRE.
 3.3. LOCATION-SCALE FAMILIES 71

Proof. For any equivariant T = T0 − W (Z) δ1 ,

 
T (X) − 0
R ((0, 1) , T ) = E(0,1) ρ
1


= E(0,1) E(0,1) (ρ (T0 (X) − W (Z) δ1 (X)) |Z)
≥ E(0,1) E(0,1) (ρ (T0 (X) − W ∗ (Z) δ1 (X)) |Z)

= E(0,1) ρ (T ∗ )
= R ((0, 1) , T ∗ ) .
Since every equivariant estimator has constant risk, T∗ is MRE (for all θ). □
Example 3.3.5. Suppose X1 , · · · , Xn are iid with common pdf with respect to Lebesgue
measure,
 
1 x−µ 1 − x−µ
f = e σ I(µ,∞) (x)
σ σ σ
and
   2
d−µ d−µ
L ((µ, σ) , d) = ρ = .
σ σ
T0 = X(1) , δ1 (X) = ni=2 X(i) − X(1) , Zi = XXn−1
i −Xn
P

Let (i = 1, · · · , n − 2), Zn−1 =
−Xn
Xn−1 −Xn
.
|Xn−1 −Xn |


We rst show that T0 , δ1 , and Z are independent (see TPE example (2.2.5)).

X (1) , δ1 is
complete sucient for (µ, σ) and Z is already ancillary so that X(1) , δ1 is independent


of Z by Basu's theorem. Also nX(1) , (n − 1) X(2) − X(1) , · · · , X(n) − X(n−1) are iid
E (1) so that X(1) is independent

of


δ1 = (n − 1) X(2) − X(1) +· · ·+ X(n) − X(n−1) . Thus, X(1) , δ1 , and Z are independent.
Consequently,

E(0,1) (T0 (X) − W (Z) δ1 (X))2 |Z

= E(0,1) T02 − 2W (Z) E(0,1) T0 E(0,1) δ1 + W (Z)2 E(0,1) δ12

is minimized with respect to W (Z) if W (Z) = W ∗ (Z) where

E(0,1) T0 E(0,1) δ1
W ∗ (Z) =
E(0,1) δ12
1
n
(n − 1)
=
n − 1 + (n − 1)2
1
=
n2
since δ1 ∼ Γ (n − 1, 1).
72 3. EQUIVARIANCE

Hence the MRE estimator of µ is

n
1 X
T ∗ = T0 − W ∗ δ1 = X(1) −


X (i) − X (1) .
n2 i=2

Note 3.3.6. (1) T∗ is not unbiased and the bias depends on σ since

E(µ,σ) T ∗ (X) = E(0,1) T ∗ (σX + µ)

= µ + σE(0,1) T ∗ (X)
 
1 n−1
= µ+σ −
n n2
σ
= µ+ 2
n
(2) Because
2
T∗ − 0

∗
R (θ, T ) = E(0,1)
1
 2
1
= + Var(0,1) T ∗ ,
n2
where

∗ 1
Var(0,1) T = Var(0,1) X(1) + Var(0,1) δ1
n4
1 n−1
= + ,
n2 n4
we have
n+1
R (θ, T ∗ ) = .
n3
(3) The UMVUE of µ is
n
1 X

T (X) = X(1) − X(i) − X(1)
n (n − 1) 2
with corresponding

R (θ, T ) = E(0,1) T 2
1 n−1 2 1
= + 2 + 0 =
n2 n2 (n − 1) n (n − 1)
n+1
> = R (θ, T ∗ ) .
n3

Estimation of σ r for some constant r. d

Assume L (θ, d) =ρ σr
and P is invariant
under G = {g : gx = ax + b}. Dene ga,b : Rn → Rn by
ga,b (x) = ax + b = (ax1 + b, · · · , axn + b) .
 3.3. LOCATION-SCALE FAMILIES 73

Then g a,b : Ω → Ω is dened by

g a,b (θ) = (aµ + b, aσ)

and
∗
h (θ) = h g a,b (θ) = ar σ r = ar h (θ)


ga,b
,where
h (θ) = h ((µ, σ)) = σ r .
Thus,
∗
ga,b (d) = ar d.
?
We can check invariance of loss function (i.e. L (gθ, g ∗ d) = L (θ, d)) since
 r   
∗ ad d
L (gθ, g d) = ρ r r
=ρ = L (θ, d)
aσ σr
as required. Thus, the condition for equivariant of T is

T (g (x)) = g ∗ T (x)
or
T (ax + b) = ar T (x) .
Proposition 3.3.7. Let T0 be any positive equivariant estimator of σ r . Then T is equi-
variant if and only if
T (X) = W (Z) T0 (X)
for some W , where Z is dened in proposition (3.3.2) .

Proof. If T = W (Z) T0 (X), then

T (aX + b) = W (Z) ar T0 (X)

= ar T (X) .

Conversely, if T is equivariant, then

T (X)
U (X) :=
T0 (X)
satises U (aX + b) = U (X) for all a > 0, b ∈ R and so by part (3) of proof of proposition
(3.3.2),
U (X) = W (Z)
for some W. □
Theorem 3.3.8. Let T0 be a particular equivariant estimator of σ r with nite risk. Sup-
pose
E(0,1) (ρ (W (Z) T0 (X)) |Z)
is minimized when W (Z) = W ∗ (Z). Then
T ∗ (X) = W ∗ (Z) T0 (X)
is MRE.
74 3. EQUIVARIANCE

Proof. If T is any equivariant estimator then

R (θ, T ) = R ((0, 1) , T )
= E(0,1) ρ (W (Z) T0 (X))


= E(0,1) E(0,1) (ρ (W (Z) T0 (X)) |Z)
≥ E(0,1) E(0,1) (ρ (W ∗ (Z) T0 (X)) |Z)

= E(0,1) ρ (T ∗ )
= R (θ, T ∗ ) .
□
1 −(x−µ)/σ
Example 3.3.9. Let X1 , · · · , Xn be e
iid with density
σ
I(µ,∞) (x) and suppose we
wish to estimate σ by minimizing the risk under the (invariant) squared fractional error
loss function,
   2
d d
L((µ, σ), d) = ρ = −1 .
σ σ
Let
n
X n
X

T0 (X) = Xi − X(1) = X(i) − X(1)
i=1 i=2
which is independent of Z (by the argument in example (3.3.5).)

Now, noting that T0 is equivariant (T (aX + b) = aT (X)), by Theorem 3.3.8 we need

to minimize

E(0,1) (ρ (W (Z) T0 (X)) |Z) = E(0,1) (W (Z) T0 (X) − 1)2 |Z

= W (Z)2 E(0,1) T0 (X)2 |Z − 2W (Z) E(0,1) (T0 (X) |Z) + 1

which is minimized for

E(0,1) (T0 (X) |Z) E(0,1) T0 (X) n−1 1

W (Z) = W ∗ (Z) = 2
= 2 = = .
E(0,1) T0 (X) |Z E(0,1) T0 (X) (n − 1) n n
Hence,
n
∗ ∗ 1X

T (X) = W (Z) T0 (X) = X(i) − X(1)
n i=1
is MRE.

Note 3.3.10. (1) Once again we notice that the MRE is biased:

n−1
E(µ,σ) T ∗ (X) = E(0,1) T ∗ (σX + µ) = σE(0,1) T ∗ (X) = σ ̸= σ.
n
(2) The UMVUE is (TPE example (2.2.5)):
n
1 X

T (X) = X(i) − X(1) .
n − 1 i=1
 3.3. LOCATION-SCALE FAMILIES 75

Scale Equivariance
For the special case when only the scale paramter σ is unknown, i.e., the density of the
sample consitutes a scale family:
x xn 
1
X = (X1 , · · · , Xn )T ∼ σ −n f ,··· , ,
σ σ
see the discussion culminating in Theorem 3.3 in TPE. The summary points are as follows.

• Estimand: h(θ) = σ r .
• Invariant loss function: L(θ, d) = ρ(d/σ r ).
• Equivariant T satises: T (aX) = ar T (X).
∗
• MRE T (X) of h(θ) is then found as follows:
 r
let T0 be equivariant for σ with nite risk,
 dene
 
X1 Xn−1 Xn
Z = (Z1 , · · · , Zn−1 , Zn ) = ,··· , , ,
Xn Xn |Xn |
 nd    
∗ T0 (X)
W (Z) = arg min E1 ρ |Z ,
W W (Z)
 and nally
T0 (X)
T ∗ (X) = .
W ∗ (Z)
 CHAPTER 4

Average-Risk Optimality

Thus far we have focused on nding estimators T which minimize the risk R (θ, T ) at
every value of θ. This was possible by restricting the class of estimators to be either
unbiased (Ch. 2) or equivariant (Ch. 3). We now drop these restrictions, thus bringing
all estimators into play, but will therefore have to sacrice uniform minimum risk for
other optimality criteria which make R (θ, T ) small in some overall sense. Two specic
versions of this type of alternative optimality are:

• minimize (weighted) average risk, which leads to Bayes estimates, e.g., hierar-
chical Bayes and Empirical Bayes, and is discussed in detail in TPE Ch. 4.
• minimize maximum risk, which leads to minimax estimates, is discussed in
detail in TPE Ch. 5.

4.1. Bayes Estimation

The main factor contributing to the recent explosion of interest in Bayes estimation is
its ability to handle extremely complicated practical problems. Some other factors which
make Bayes estimation attractive are as follows.

(1) The mathematical structure is very nice.

(2) It permits the incorporation of prior information (although there is a lot of
debate about how this should be done).
(3) It provides a systematic approach to the determination of minimax estimators.

In the Bayesian framework, we consider the parameter and observation vectors to be

jointly distributed on Ω × X . We shall suppose that the parameter vector Θ has the
marginal distribution Λ and that the conditional distribution of the observation vector
X given Θ = θ is Pθ . For any particular value R θ of Θ, we dene the risk of the estimator
T ′ at θ as R(θ, T ′ ) := E[L(Θ, T ′ )|Θ = θ] = X L(θ, T ′ (x))dPθ (x) as before.
Definition 4.1.1. For any estimatorT ′ , the integral
Z Z Z
′
R (θ, T ) dΛ (θ) = L(θ, T ′ (x))dPθ (x)dΛ(θ).
Ω Ω X
is called the Bayes risk of T with respect to the prior distribution Λ.
′
The Bayes
′
risk of T is thus
rΛ := E(R(Θ, T ′ )) = E(L(Θ, T ′ )).
76
 4.1. BAYES ESTIMATION 77

An estimator T is a Bayes estimator with respect to the distribution Λ if

Z Z
R (θ, T ) dΛ (θ) = inf′ R (θ, T ′ ) dΛ (θ) .
Ω T Ω

Theorem 4.1.2. Suppose Θ ∼ Λ and X given Θ = θ has distribution Pθ . Then if there

exists T (·) which minimizes
Z
E (L ((Θ, T (X))) |X = x) = L (θ, T (x)) dP (θ|X = x)
Ω
for each x, where P ( · |X = x) is the conditional (or posterior) distribution of Θ given
X = x, then T (X) is Bayes with respect to Λ.

Proof. For any estimator T ′,

E (L (Θ, T ′ (X)) |X) ≥ E (L (Θ, T (X)) |X)
and so, taking expectations of each side,

EL (Θ, T ′ (X)) ≥ EL (Θ, T (X)) .

□
Example 4.1.3.

(1) Let L (θ, d) = (d − g (θ))2 . The Bayes estimator T of g (Θ) minimizes

2

E (T (X) − g (Θ)) |X = x ∀ x.
Hence
T (x) = E (g (Θ) |X = x) ,
which is the posterior mean.
(2) Let L (θ, d) = |d − g (θ)|. The Bayes estimator T of g (Θ) minimizes

E (|T (X − g (Θ))| |X = x) ∀ x.
Hence
T (x) = med (g (Θ) |X = x) ,
which is the posterior median.
2
(3) Let L (θ, d) = w (θ) (d − g (θ)) . The Bayes estimator T of g (Θ) minimizes

E w (Θ) (T (X) − g (Θ))2 |X = x ∀ x.

It can therefore be obtained by solving

d
(T (x) − g (θ))2 w (θ) dP (θ|X = x)
R
dT (x)
R
= 2 (T (x) − g (θ)) w (θ) dP (θ|X = x) = 0.
Hence
E (w (Θ) g (Θ) |X = x)
T (x) = .
E (w (Θ) |X = x)
78 4. AVERAGE-RISK OPTIMALITY

Example 4.1.4. Suppose X ∼ bin (n, p) and p ∼ B (a, b). Thus,

Γ (a + b) a−1
dΛ (p) = p (1 − p)b−1 dp, 0 < p < 1, a, b > 0.
(Γ (a) Γ (b))
The posterior distribution of p given X=x is B (a + x, b + n − x) since

fX|P (x|p) fP (p)

fP |X (p|x) =
fX (x)
n x
p (1 − p)n−x Kpa−1 (1 − p)b−1


= x
fX (x)
= c (x) pa+x−1 (1 − p)b+n−x−1
which is a beta density. (Without doing any calculations it is clear that c (x) must be
Γ (a + b + n) / (Γ (a + x) Γ (b + n − x)).)
Let L (p, d) = (d − p)2 . Then the Bayes estimator of p is
Z 1
a+x
T (x) = pfP |X (p|x) dp =
0 a+b+n
a+b a n x
= · + · .
a + b + n |a {z
+ b} a+b+n n
|{z}
prior mean the usual etimator

Thus T (X) − X/n → 0 a.s. as n→∞ with a and b xed and as a+b → 0 with n xed.

ClearlyX/n is not a Bayes estimator for any beta prior (i.e. for any a > 0 and b > 0).
However ifΛ is concentrated on the two-point set {0, 1} then X/n is Bayes as the following
argument shows. If P (P = 1) = 1 − π and P (P = 0) = π , then X is either 0 or n with
probability 1 and
(
P (X=n,P =1) 1−π
P (P = 1|X = n) = P (X=n)
= 1−π
=1
.
P (P = 0|X = n) =0
Hence the Bayes estimator satises T (n) = 1 and a similar argument shows that T (0) =
0. Hence T (x) = x/n, x = 0, n. Notice that this two-point distribution is the limit in
distribution of Beta(a, b) asa + b → 0 with a/(a + b) = π .
Theorem 4.1.5. Let L (θ, d) = (d − g (θ))2 . Then no unbiased estimator T of g (θ) can
be Bayes unless
E (T (X) − g (Θ))2 = 0
i.e. unless T (X) = g (Θ) with probability 1 and the Bayes risk of T is zero.

Proof. If T is Bayes with respect to some Λ and unbiased for g (θ) then

E (T (X) |Θ) = g (Θ) , and

E (g (Θ) |X = x) = T (x)
 4.1. BAYES ESTIMATION 79

Hence

E (T (X) g (Θ)) = E (E (T (X) g (Θ)) |X)

= ET (X)2
= E (E (T (X) g (Θ)) |Θ)
= Eg (Θ)2
and so

E (T (X) − g (Θ))2 = ET (X)2 − 2E (T (X) g (Θ)) + E (g (Θ))2

= 0.

□
Example 4.1.6. Given Θ = θ, let X1 , · · · , Xn be iid N (θ, σ 2 ) with σ 2 known and suppose
that Θ ∼ N (µ, τ ) with µ, τ 2 known. Then the joint density of Θ and X is
2

1 1 Pn 2 1 1 2
fΘ,X (θ, x) = √
n e− 2σ2 1 (xi −θ) √ e− 2τ 2 (θ−µ)
σ 2π τ 2π
and the posterior density of Θ is

fΘ,X (θ, x)
fΘ|X (θ|x) =
fX (x)
θ2
 P 
n 2 xi µθ
= c (x) exp − 2 θ + 2 θ − 2 + 2
2σ σ 2τ τ
 nx µ 2 2

2 + τ2 σ τ
= N σn 1 , 2 + σ2
.
σ 2 + τ 2 nτ
For squared error loss, the Bayes estimator of Θ is

nσ −2 τ −2
E (Θ|X) = X + µ = TΛ (X)
nσ −2 + τ −2 nσ −2 + τ −2
and

1 2

Var (Θ|X) = = E (Θ − TΛ (X)) |X
nσ −2 + τ −2
1
rΛ = EL (Θ, TΛ ) = E (Θ − TΛ (X))2 = .
nσ −2 + τ −2

For large n, the Bayes estimator is close to X in the sense that TΛ (X) − X → 0 a.s. as
n → ∞ with τ and σ xed. Also T (X) − X → 0 as τ → ∞ with n and σ xed. However,
X is not Bayes since the prior probability distribution N (µ, τ 2 ) does not converge to a
probability measure as τ → ∞.
80 4. AVERAGE-RISK OPTIMALITY

However one can formally obtainX as a Bayesian estimator with respect to the improper
prior distribution, Λ (dθ) = dθ. Suppose
1 1 Pn 2
p (x|θ) = √
n e− 2σ2 1 (xi −θ)
σ 2π
with σ2 known. Setting Λ (dθ) = dθ we nd that the joint density of (X, Θ) with respect
to Lebesgue measure is

p (x, θ) = p (x|θ) .
The posterior distribution of θ is therefore

n
!!
1 2
X
p (θ|x) = k (x) exp − 2 nθ − 2θ xi
2σ i=1
 n 
= k ∗ (x) exp − 2 (θ − x)2 .
2σ
Hence,
σ2
 
p (θ|x) = N x,
n
and so X is generalized Bayes estimator of Θ with respect to L (θ, d) = (d − θ)2
the
and the improper Lebesgue prior Λ (dθ) = dθ for Θ. The improper prior densities
I(−∞<∞) and I[0,∞) are frequently used to account for total ignorance of parameters with
values in R and R+ respectively.

Note:
( unless otherwise stated, all results from now on apply to Bayes estimators,
not generalized Bayes. )

Conjugate Priors
If there exists a parametric family of prior distributions such that the posterior distribu-
tion also belongs to the same parametric family, then the family is called conjugate.
Example 4.1.7. Conditional on Σ2 = σ 2 , let X1 , · · · , Xn be iid N (0, σ 2 ) and dene
2 −1
Ξ = (2Σ ) .
Pn
x2i n
fX|Ξ (x|ξ) = cξ r e−ξ 1 where r= .
2
Γ g, α1


and let the prior distribution for Ξ be with density

αg g−1 −αξ
λ (ξ) = ξ e , ξ ≥ 0.
Γ (g)
We note that

g
g (g + 1)
E (Ξ) =
, E Ξ2 =
α α2
α α2
E Ξ−1 = E Ξ−2 =



.
g−1 (g − 1) (g − 2)
 4.1. BAYES ESTIMATION 81

Then, the posterior becomes

fΞ|X (ξ|x) = c (x) ξ r+g−1 e−ξ( x2i +α)

P

 
1
= density at ξ of Γ r + g, P 2 ,
xi + α

so that the gamma distribution family is a conjugate family for the normal distribution.

If the loss is squared error then the Bayes estimator of σ 2 = (2ξ)−1 is

α + x2i
Z P
1
T (X) = fΞ|X (ξ|x) dξ =
2ξ 2 (r + g − 1)
α + x2i
P
= .
n + 2g − 2

As α → 0 with g = 1, the prior density/α converges pointwise to the improper prior

density I[0,∞) and the Bayes estimator T (X) satises

T (X) − Σni=1 Xi2 /n → 0 a.s..

s-Parameter Exponential Families

Calculation of Bayes estimators simplies under an s-parameter exponential family, where
for a random sample x = (x1 , . . . , xn ), the density is given by (the canonical form):

s
X
(4.1.1) pη (x) = exp{ ηi Ti (x) − A(η)}h(x).
i=1

Theorem 4.1.8. If X has density (4.1.1) and η has prior density π(η), then for j =
1, . . . , n:
" s
#
X ∂Ti (x) ∂ ∂
(4.1.2) E ηi x = log m(x) − log h(x),
i=1
∂xj ∂xj ∂xj

where
Z
m(x) = pη (x)π(η)dη = marginal of X.

Proof. Letting f (η|x) = pη (x)π(η)/m(x), note that for any integrable function
g(η, X),
Z
E[g(η, X)|x] = g(η, x)f (η|x)dη,
82 4. AVERAGE-RISK OPTIMALITY

g(η, X) = si=1 ηi ∂Ti (x)/∂xj , the LHS of (4.1.2) becomes:

P
whence upon substituting

Z X  P
1 ∂Ti
E[g(η, X)|x] = ηi e ηi Ti −A(η) h(x)π(η)dη
m(x) i
∂x j
Z  
1 ∂ P ηi Ti −A(η)
= e e h(x)π(η)dη
m(x) ∂xj
Z  
1 ∂ P P
ηi Ti ∂h(x)
ηi Ti
e−A(η) π(η)dη,


= e h(x) − e
m(x) ∂xj ∂xj

where the last equality follows by bringing h(x) inside the round brackets and using the
chain rule. Finally, switching integration and dierentiation gives,
Z P
1 ∂
E[g(η, X)|x] = e ηi Ti −A(η) h(x)π(η)dη
m(x) ∂xj
Z P
∂h(x)/∂xj 1
− e ηi Ti −A(η) h(x)π(η)dη
h(x) m(x)
∂m(x)/∂xj ∂h(x)/∂xj
= −
m(x) m(x)
∂ ∂
= log m(x) − log h(x).
∂xj ∂xj
□

Corollary 4.1.9. If in Theorem 4.1.8 X = (X1 , . . . , Xs ) has theP density pη (x) with
Ti (x) = xi , then the Bayes estimator of η under the loss L(η, δ) = (ηi − δi )2 is given
by:
" s #
X ∂Ti (x) ∂ ∂
E(ηj |x) = E ηi x = log m(x) − log h(x),
i=1
∂x j ∂x j ∂x j

for j = 1, . . . , s.

Proof. Problem 4.3.3 using Theorem 4.1.8 with Ti = Xi . □

Example 4.1.10 (Multiple Normal Model). Xi |θi ∼ indep. N (θi , σ 2 ) and Θi ∼ N (µ, τ 2 ),
where σ 2 , µ, τ 2 , are all known, and i = 1, . . . , s.
(xi − θi )2
 P 
2 −s/2
pη (x) = (2πσ ) exp −
2σ 2
 
 
X θ2 

 X θ 
2 −s/2 − 2σ12
P 2
i i xi
= exp − 2
x i − 2
(2πσ ) e .

 σ
|{z} | 2σ }
{z  | {z }
  h(x)
ηi A(η)
 4.1. BAYES ESTIMATION 83

By Corollary 4.1.9 the Bayes estimator of θi is

 
2 2 ∂ log m(x) ∂ log h(x)
E(Θi |x) = σ E(ηi |x) = σ −
∂xi ∂xi
τ2 σ2
= x i + µ, (Claim).
τ 2 + σ2 τ 2 + σ2
To prove the Claim, note that one term is easily handled:
 X 2 
∂ ∂ xj xi
log h(x) = − = − .
∂xi ∂xi 2σ 2 σ2
For the other term, we compute m(x) to within multiplicative constants (free of x):
Z
m(x) = pη (x)π(θ)dθ
Rs
(xi − θi )2 (θi − µ)2
Z  P P 
2 −s/2
= (2πσ ) exp − − (2πτ 2 )−s/2 dθ
2σ 2 2τ 2
Z  
2 2 2 −s/2 − 2σ12
P 2
xi 1 X
2
X  xi µ
= (4π σ τ ) e exp − θi + + θi dθ.
2(σ 2 + τ 2 ) σ2 τ 2
Completing the square on the exp term in the integrand:

(θi − ci )2 (xi − µ)2 τ 2 xi + µσ 2 2

P P
exp = − − , ci = , λ = σ2 + τ 2.
2(σ 2 τ 2 /λ2 ) 2λ2 λ2
Thus,
 Z  
1 X 2 1 X
2
m(x) ∝ exp − 2 (xi − µ) exp − (θi − ci ) dθ ,
2λ 2(σ 2 + τ 2 )
| {z }
constant (free of x)

from which we conclude that, marginally, Xi ∼ iid N(µ, λ2 ), whence

∂ log m(x) ∂ X (xj − µ)2
=− = −(xi − µ)/λ2 ,
∂xi ∂xi 2λ2
and therefore

τ2 σ2
   
2 ∂ log m(x) ∂ log h(x) 2 xi − µ x i
σ − =σ − + = x i + µ.
∂xi ∂xi λ2 σ2 τ 2 + σ2 τ 2 + σ2

Having obtained the Bayes estimator by simply dierentiating apropriate functions, the
next obvious question concerns the computation of its risk.

Theorem 4.1.11. The risk of the Bayes estimator in Corollory 4.1.9 is:
s
" 2 #
∂ 2 log m(X)
  X 
∂ log h(X) ∂ log m(X)
R(η, E(η|X)) = R η, − + E 2 + .
∂X i=1
∂Xi2 ∂Xi
84 4. AVERAGE-RISK OPTIMALITY

Proof. Stein's Identity is applicable to an exponential family in canonical form (as

in Theorem 4.1.8), and states that for any dierentiable real-valued function g with
E|g ′ (X)| < ∞, we have
(" s
# )
∂ log h(X) X ∂Ti (X) ∂g(X)
E + ηi g(X) = −E , for j = 1, . . . , n.
∂Xj i=1
∂X j ∂X j

provided the support of each

P Xj is all of R. (If the support is a bounded interval, then
the above holds if exp{ ηi Ti (x)}h(x) → 0, as x approaches the boundaries.) Applying
the Identity with g(x) = 1, and noting that in our case Ti = Xi , we get:
 
 ∂h(X)/∂X s
j
X ∂Xi 
+ ηi  = −E(0) = 0,
 
Eη 
 h(X) i=1
∂Xj 
|{z}
δij

which leads to
   
∂ log h(X) ∂ log h(X)
−Eη = ηj , ∀j ⇐⇒ −Eη = η.
∂Xj ∂X
Thus −∂ log h(X)/∂X is an unbiased estimate of η with risk:
  s  2
∂ log h(X) X ∂ log h(X)
R η, − = Eη ηi + .
∂X i=1
∂X i

Now, the risk of the Bayes estimator is given by:

s
X
R(η, E(η|X)) = E [ηi − E(ηi |X)]2
i=1
s   2
X ∂ log m(X) ∂ log h(X)
= E ηi − −
i=1
∂X i ∂Xi
s  2 s   s  2
X ∂ log h(X) X ∂ log h(X) ∂ log m(X) X ∂ log m(X)
= E ηi + −2 E ηi + + E
i=1
∂Xi i=1
∂Xi ∂Xi i=1
∂Xi
| {z } | 
{z }
∂ log h(X) ∂ 2 log m(X)
R(η,− ∂X ) −E
∂X 2
, g(x)=∂ log m/∂xi
Stein's Id. with
i
s
" 2 #
2
  X 
∂ log h(X) ∂ log m(X) ∂ log m(X)
= R η, − + E 2 2
+ .
∂X i=1
∂Xi ∂Xi

□
Example 4.1.12 (Multiple Normal Model (continued)). From before, −∂ log h(x)/∂xi =
xi /σ 2 , which as we saw in the above proof, is unbiased for ηi = θi /σ 2 . Thus, and since
X = (X1 , . . . , Xs ) is CSS, we have that −∂ log h(x)/∂X = (X1 /σ 2 , . . . , Xs /σ 2 ) is UMVU
 4.1. BAYES ESTIMATION 85

for η = (θ1 /σ 2 , . . . , θs /σ 2 ). Now, and from the above proof, the risk of the UMVUE is:
  s  2 X s  2
∂ log h(X) X ∂ log h(X) Xi − θi s
R η, − = Eη ηi + = Eθi 2
= 2,
∂X i=1
∂Xi i=1
σ σ

since Xi |θi ∼ N (θi , σ 2 ) implies Eθi (Xi − θi )2 = σ 2 . Thus, the risk of the Bayes estimator
is given by:

s
" 2 #
∂ 2 log m(X)
  X 
∂ log h(X) ∂ log m(X)
R(η, E(η|X)) = R η, − + Eη 2 +
∂X i=1
∂Xi2 ∂Xi
s
(xi − µ)2
 
s X 2
= 2
+ Eη 2 2 2
− 2
σ i=1
(σ + τ ) σ + τ2
2 Xs 
sτ 4 σ2

µ 2
= + η i − , (by Problem 4.3.6)
σ 2 (σ 2 + τ 2 )2 σ2 + τ 2 i=1 | {zσ 2
}
a2i

s τ4 s X
= 2 2 2 2
< 2, if a2i = 0.
σ (σ + τ ) σ
Summary:

• UMVUE of η is (X1 /σ 2 , . . . , Xs /σ 2 ), with risk R (η, −∂ log h(X)/∂X) = s/σ 2 .

• Bayes estimator of η is

τ 2 /σ 2 τ 2 /σ 2
 
µ µ
2 2
X 1 + 2 2
, . . . , 2 2
Xs + 2 ,
σ +τ σ +τ σ +τ σ + τ2
with risk

sτ 4 σ 4 a2i
P
R (η, E(η|X)) = 2 2 + ,
σ (σ + τ 2 )2 (σ 2 + τ 2 )2
which is smaller than the risk of the UMVUE if, e.g., ηi = µ/σ 2 , ∀i.

Empirical Bayes

General Bayes setup thus far has been:

(
Xi |θ ∼ f (x|θ), i = 1, . . . , n
Θ|γ ∼ π(θ|γ), γ is known.
The Empirical Bayes idea is to treat γ as unknown, and use frequentist methods to
estimate it based on the marginal distribution of X:
n
Z Y
m(x|γ) = f (xi |θ)π(θ|γ)dθ.
i=1
86 4. AVERAGE-RISK OPTIMALITY

For example, one can use m(x|γ) to produce the MLE γ̂(x). Then, the empirical Bayes
estimator, T̂ (x), minimizes the (empirical) posterior risk:
Z
L(θ, T̂ (x))π(θ|x, γ̂(x))dθ.

The empirical Bayes estimator is best suited to situations in which there are many prob-
lems that can be modeled simultaneously in a common way.

Example 4.1.13 (Multiple Binomial Model). For the k -th treatment group, k = 1, . . . , K ,
we measure Xk , the number of successes out of n trials, and model it as

Xk ∼ Bin(n, pk ).
We attach the same prior to each pk (this is appropriate since the treatments all corre-
spond to the same disease):
pk ∼ Beta(a, b).
Now, from Example 4.1.4, the Bayes estimator of pk is:

a + xk
T (x) = .
a+b+n
These were easy calculations for xed (a, b) and conjugacy. To compute the empirical
Bayes estimator, we rst need the marginal of X = (X1 , . . . , XK ):
Z 1 Z 1YK  
n xk Γ(a + b) a−1
m(x|a, b) = ··· pk (1 − pk )n−xk pk (1 − pk )b−1 dpk
0 0 k=1 x k Γ(a)Γ(b)
K  
Y n Γ(a + b)Γ(a + xk )Γ(n − xk + b)
= .
k=1
xk Γ(a)Γ(b)Γ(a + b + n)

There is no closed-form solution to this, but one can compute the MLEs (â, b̂) numerically,
leading to the empirical Bayes estimator:

â + xk
T̂ (x) = .
â + b̂ + n
It turns out that the Bayes risk of the empirical Bayes estimator is often just slightly
higher than that of the Bayes estimator (which therefore enjoys a certain degree of
robustness ).

For estimation of the canonical parameter in exponential families, the empirical Bayes
estimator can be expressed in the same form as the Bayes estimator.

Theorem 4.1.14. , For the situation of Corollary 4.1.9 with prior π(η|λ) where λ is a
hyperparameter, the empirical Bayes estimator of ηi is:

 ∂ log m(x|λ)
∂xi
− ∂ log∂xh(x)
i
, using the MLE λ̂(x),
E(ηi |x, λ̂) = λ= λ̂(x)
 ∂ log m(x|λ̂(x)) − ∂ log h(x) ,
∂xi ∂xi
using general estimate λ̂(x).
 4.2. MINIMAX ESTIMATION 87

Proof. Straightforward from Corollary 4.1.9 with with m(x) 7→ m(x|λ̂). For the
MLE-based simplication, apply the chain-rule to the general estimate:

∂ log m(x|λ̂(x)) ∂ log m(x|λ) ∂ λ̂(x) ∂ log m(x|λ)

= · + .
∂xi ∂λ λ=λ̂(x) ∂xi ∂xi λ=λ̂(x)
| {z }
=0 since λ̂(x) is the MLE

□
Example 4.1.15 (Multiple Normal Model (continued)).

Xi |θi ∼ indep. N (θi , σ 2 ), Θi ∼ indep. N (µ, τ 2 ), i = 1, . . . , s,

with µ unknown, {σ 2 , τ 2 } X from before:
known. Note the marginal of

(xi − µ)2
 P 
2 2 −s/2
m(x|µ) = [2π(σ + τ )] exp − ,
2(σ 2 + τ 2 )
which implies µ̂ = X̄ is the MLE of µ. By Problem 4.6.10(a) and Example 4.1.10, we
thus obtain:

τ2 σ2
empirical Bayes estimator of θi = x i + µ̂
σ2 + τ 2 σ2 + τ 2
= Bayes estimator of θi under a N (µ̂, σ 2 ) prior.

Note 4.1.16. Using Theorem 4.1.11, we can express risk of Bayes estimator in Theo-
rem 4.1.14 as:
 !2 
s
∂ 2 log m(X|λ̂)
  X
  ∂ log h(X) ∂ log m(X|λ̂)
R η, E(η|X, λ̂) = R η, − + Eη 2 2
+ .
∂X i=1
∂X i ∂Xi

For the Multiple Normal Model we get the result in Problem 4.6.10(b).

4.2. Minimax Estimation

Definition 4.2.1. A statistic T is said to be minimax if T satises

min
′
sup R (θ, T ′ ) = sup R (θ, T ) .
T θ∈Ω θ

We have seen that many estimation problems allow the determination of UMVU, MRE
or Bayes estimators. Minimax estimators however are usually much harder to nd.

A minimax estimator minimizes the maximum risk. i.e., a minimax estimator minimizes
the risk in the worst case. This suggests a possible connection with Bayes estimation
under the worst possible prior.

Given Λ on Ω, let rΛ denote the Bayes risk of the Bayes estimator TΛ , i.e.
Z
rΛ := R (θ, TΛ ) dΛ (θ) .
88 4. AVERAGE-RISK OPTIMALITY

Definition 4.2.2. A prior distribution Λ is least favorable if

rΛ ≥ rΛ′ for all other priors Λ′ .
Theorem 4.2.3. Suppose a prior Λ satises
rΛ = sup R (θ, TΛ ) .
θ

Then

(1) TΛ is minimax.
(2) If TΛ is the unique Bayes estimator under Λ, then it is the unique minimax
estimator.
(3) Λ is least favorable.

Proof. (1) For any estimator T,

Z
sup R (θ, T ) ≥ R (θ, T ) dΛ (θ)
θ∈Ω
Z
≥ R (θ, TΛ ) dΛ (θ) (since TΛ is Bayes for Λ.)
= sup R (θ, TΛ ) by hypothesis.
θ∈Ω

(2) If TΛ is the unique Bayes solution then the second inequality in (1) becomes
strict. (i.e., ≥→>.)
(3) If Λ′ is another prior distribution then
Z
rΛ′ = R (θ, TΛ′ ) Λ′ (dθ)
Z
≤ R (θ, TΛ ) dΛ′ (θ) since TΛ′ is Bayes for Λ′
≤ sup R (θ, TΛ )
θ
= rΛ by hypothesis.

□
Corollary 4.2.4. If TΛ has constant risk then it is minimax.

Proof. If R (θ, TΛ ) is independent of θ then

Z
R (θ, TΛ ) dΛ (θ) = sup R (θ, TΛ ) .
θ

□
 4.2. MINIMAX ESTIMATION 89

Example 4.2.5. Let X ∼ bin (n, p), L (θ, d) = (d − θ)2 , and θ = p. We shall show below
that X/n is not minimax. Let us use Corollary 4.2.4 to derive a minimax estimator.
Suppose P ∼ B (a, b). Then we have from example (4.1.4) that

fP |X (p|x) = c (x) pa+x−1 (1 − p)b+n−x−1 and

a+x
TΛ = .
a+b+n
Thus we obtain
1 2
R (p, TΛ ) = 2 Ep (a + X − p (a + b + n))
(a + b + n)
npq + (qa − pb)2
= .
(a + b + n)2
Now, choose a and b to make the risk independent of p. Since the coecient of p2 is
2
−n + (a + b) and the coecient of p n − 2a (a + b),
is setting these equal to zero gives
√
n
a=b= .
2
Hence √
x + 2n
TΛ0 = √
n+ n
is Bayes with constant risk and is therefore minimax.
√ √ 
n n
Since TΛ0 is the unique Bayes solution with respect to Λ0 ∼ B 2
, 2
, it follows from

theorem 4.2.3 part (2) that TΛ0 is the unique minimax estimator of p with respect to
squared error loss and the risk is
Z
2
rΛ0 = E (TΛ0 − p) = R (p, TΛ0 ) Λ0 (dp)
Z
1 n
= √ 2 Λ0 (dp)
(n + n) 4
1 √
2
= = R (p, TΛ0 ) , where βn = 4 1 + n .
βn
The risk of the usual estimator with squared error loss, T (X) = X/n, is
 2
X p (1 − p)
R (p, T ) = Ep −p = .
n n
1
p
Thus, since the risks of T and TΛ0 coincide at
2
± c n , where cn = βn2 − 4nβn /(2βn ), we
have:

1
R (p, TΛ0 ) < R (p, T ) , for p− < cn ,
2
1
R (p, TΛ0 ) > R (p, T ) , for p− > cn .
2
90 4. AVERAGE-RISK OPTIMALITY

Now, cn → 0 as n → ∞, but cn is larger for small n. For n = 1,

(
1
4
if x=0
TΛ0 (x) = 3
4
if x = 1.
Remark 4.2.6. Squared error loss might not be best for estimating p. The penalty
should perhaps be larger at the endpoints p=0 and p = 1. If we use

(d − p)2 (d − p)2
L (p, d) = = ,
pq p (1 − p)
then X/n has constant risk and is Bayes with respect to U (0, 1) prior (check!).

Definition 4.2.7. Let Λk be a sequence of priors, and suppose

Z
rΛk = R (θ, Tk ) dΛk (θ) → r as k → ∞,

where Tk is Bayes with respect to Λk for each k. We say that the sequence {Λk } is least
favorable if
rΛ ≤ r for all Λ,
i.e. if the limit of the minimum Bayes risk is at least as bad as the minimum Bayes risk
for any prior. (Compare Denition 4.2.2.)

Theorem 4.2.8. If there exists an estimator T and a sequence of prior distributions {Λk }
such that
sup R (θ, T ) = lim rΛk
θ k→∞
then

(1) T is minimax (but not necessarily unique.)

(2) {Λk } is least favorable.

Proof. (1) If T′ is any estimator, then

Z
′
sup R (θ, T ) ≥ R (θ, T ′ ) dΛk
θ
≥ rΛk , ∀k
and hence
sup R (θ, T ′ ) ≥ lim rΛk = sup R (θ, T ) .
θ θ
(2) If Λ is any prior, then
Z
rΛ = R (θ, TΛ ) dΛ (θ)
Z
≤ R (θ, T ) dΛ (θ)
≤ sup R (θ, T )
θ
= lim rΛk .
k→∞
 4.3. MINIMAXITY AND ADMISSIBILITY IN EXPONENTIAL FAMILIES 91

□
Example 4.2.9. Suppose that X1 , · · · , Xn ∼ iid N (θ, σ 2 ).

(1) σ 2 known:
Θ ∼ N (µ, k 2 ) = Λk . In example (4.1.6), we saw that the Bayes estimator for
squared error loss is

nσ −2 k −2
E (Θ|X) = X + µ = TΛk (X)
nσ −2 + k −2 nσ −2 + k −2
with

rΛk = E (TΛk − Θ)2

= E E (Θ − E (Θ|X))2 |X

= E Var (Θ|X)
1
= .
n/σ + 1/k 2
2

As k → ∞,
σ2

rΛ → = R θ, X .
n
Hence, X is minimax (for squared error loss) by theorem (4.2.8).
(2) σ unknown:
2
2
Since supθ,σ 2 R ((θ, σ ) , T ) = ∞ for T (X) = X and X is minimax for each xed
σ , we restrict σ to satisfy σ 2 ≤ m < ∞. If T is minimax on θ ∈ R, σ 2 ≤ m
2

then

θ, σ 2 , T ≤ sup R θ, σ 2 , X = sup R θ, σ 2 , X .







sup R
θ,σ 2 =m θ,σ 2 ≤m θ,σ 2 =m

But X is minimax on σ 2 = m,
hence this is an equality. Hence X is minimax
on θ ∈ R, σ 2 ≤ m. 2
Although the restriction σ ≤ m was necessary to make the
minimax problem meaningful, the minimax estimator X does not depend on m.

4.3. Minimaxity and Admissibility in Exponential families

Given two estimators T, T ′ , such that

R (θ, T ) ≤ R (θ, T ′ ) for all θ

then T is preferable to T ′ on the basis of risk.
Definition 4.3.1. T ′ is inadmissible (with respect to the loss function L) if there exists
T such that
R (θ, T ) ≤ R (θ, T ′ ) for all θ
with strict inequality for some θ . (In this case, we also say that T dominates T ′ .)
An estimator is admissible if it is not inadmissible.
92 4. AVERAGE-RISK OPTIMALITY

In general, it is dicult to determine whether or not an estimator is admissible. . .

Note:
( We will naturally restrict our attention to non-constant estimators, since any
T = θ0 , where θ0 ∈ Ω, is trivially admissible.)

Theorem 4.3.2. If T is a unique Bayes estimator with respect to some Λ, then T is

admissible. (Uniqueness means that any two Bayes estimators T and T ′ dier only a set
D where Pθ (D) = 0, ∀θ.)

Proof. If T is not admissible then there exists T′ such that

R (θ, T ) ≥ R (θ, T ′ ) for all θ

and

R (θ, T ) > R (θ, T ′ ) for some θ.

R (θ, T ) dΛ ≥ R (θ, T ′ ) dΛ implies that T ′ is Bayes with respect to Λ.
R R
Now Hence by
′
uniqueness R (θ, T ) = R (θ, T ) for all θ and we obtain the contradiction. □
Example 4.3.3. Suppose that X1 , · · · , Xn ∼ iid N (θ, σ 2 ) with σ2 known. Let Θ ∼
2 2
N (µ, τ ) and L (θ, d)=(d − θ) . Then

nτ 2 σ2
T = X + µ (∗)
σ 2 + nτ 2 σ 2 + nτ 2
is the unique Bayes estimator of Θ and is therefore admissible.

This example shows that aX + b is admissible for all a ∈ (0, 1) and b ∈ R since any
a ∈ (0, 1) and b ∈ R can be obtained in (*) by suitable choice of µ and τ 2 and hence
aX + b is unique Bayes for some Λ.
Theorem 4.3.4. If X ∼ N (θ, σ 2 ), σ 2 is known and L (θ, d) = (d − θ)2 then aX + b is
inadmissible for θ if

(1) a > 1,
(2) a < 0, or
(3) a = 1 and b ̸= 0.

Proof. We rst calculate

R (θ, aX + b) = Eθ (aX + b − θ)2

= Eθ (a (X − θ) + θ (a − 1) + b)2
= a2 σ 2 + ((a − 1) θ + b)2 .

(1) If a > 1,
R (θ, aX + b) > R (θ, X) = σ 2 for all θ.
 4.3. MINIMAXITY AND ADMISSIBILITY IN EXPONENTIAL FAMILIES 93

(2) If a < 0, then (a − 1)2 > 1 and

R (θ, aX + b) ≥ ((a − 1) θ + b)2

 2
2 b
= (a − 1) θ +
a−1
 
b
> R θ, 0 · X − .
a−1
(3) If a=1 and b ̸= 0, then

R (θ, X + b) = σ 2 + b2 > σ 2 = R (θ, X) .

□
Corollary 4.3.5. Suppose that X1 , · · · , Xn ∼ N (θ, σ 2 ) with σ 2 known. Then aX + b
is inadmissible if

(1) a > 1,
(2) a < 0, or
(3) a = 1 and b ̸= 0,

and admissible if 0 ≤ a < 1.

Proof. It remains only to establish admissibility when a = 0. This is trivial since

for the estimator T (X) = b corresponding to a = 0,
the risk at θ = b is R (b, b) = 0 and
′
every other estimator has positive risk at b since if Pθ (T (X) ̸= b) > 0 for some θ then
T ({b} ) has positive Lebesque measure and this implies that Pb (T ′ (X) ̸= b) > 0 and
′−1 c
′ 2
hence that Eb (T − b) > 0. □
Proposition 4.3.6. Suppose that X1 , · · · , Xn ∼ iid N (θ, σ 2 ) with σ 2 known. Then, X
is admissible.

Proof. We give two proofs.

2
(1) (Limiting Bayes method) Assume without loss of generality that σ = 1. If
∗ ∗ 1
X is inadmissible then there exists T such that R (θ, T ) ≤ n for all θ and
R (θ0 , T ∗ ) < n1 for some θ0 .
n 1 2
e− 2 (xi −θ)
Z Y
∗ ∗ 2 ∗ 2
R (θ, T ) = Eθ (T − θ) = (T (x) − θ) √ dx
i=1
2π
is continuous in a neighborhood of θ0 . Hence there exist a and b such that
a < θ0 < b and c>0 such that

1
R (θ, T ∗ ) < −c for all θ ∈ (a, b) .
n
94 4. AVERAGE-RISK OPTIMALITY

Suppose that Θ ∼ N (0, τ 2 ) =: Λ. Then we shall obtain a contradiction by

showing that
Z
1 θ2
rΛ∗ ∗
:= ER (Θ, T ) = √ R (θ, T ∗ ) e− 2τ 2 dθ
τ 2π
is smaller than the minimum Bayes risk for Λ, i.e.

1 τ2
rΛ = ER (Θ, TΛ ) = = .
nσ −2 + τ −2 nτ 2 + 1
(As in example (4.1.6) ,

τ2 2

E (Θ − TΛ (X)) |X = 2 = Var (Θ|X)
nτ + 1
so that

τ2
rΛ = E (Θ − TΛ (X))2 = E E (Θ − TΛ (X))2 |X



= .)
nτ 2 + 1
Now
−θ 2
√1 1
− R (θ, T ∗ )
R

1
n
−rΛ∗ τ 2π n
e 2τ 2
1 = 1 2 dθ
n
− rΛ n
− nττ2 +1
Z b
n (nτ 2 + 1)

1 θ2
≥ √ − R (θ, T ) e− 2τ 2 dθ
∗
τ 2π a n
n (nτ 2 + 1) b − θ22
Z
> c √ e 2τ dθ
τ 2π a

where c > 0 is independent of τ . Since the integral in the last line converges to
b − a as τ → ∞ by DCT, the ratio goes to innity as τ → ∞. Thus, for all
suciently large τ0 ,
∗
rΛ(µ,τ 0)
< rΛ(µ,τ0 ) ,
which contradicts the fact that rΛ(µ,τ0 ) is minimum Bayes risk.
(2) (Via the information inequality) If T is any estimator of θ with nite second
moment under each Pθ , then Eθ T = b (θ) + θ and

R (θ, T ) = Varθ T + b2 (θ)

(1 + b′ (θ))2
≥ + b2 (θ)
I (θ)
(1 + b′ (θ))2
= + b2 (θ)
n
′
(b(θ) exists by theorem (1.3.13) and we assume without loss of generality σ = 1.)
Hence if T is risk-preferable to X ,

1
R (θ, T ) ≤ for all θ, (∗)
n
 4.3. MINIMAXITY AND ADMISSIBILITY IN EXPONENTIAL FAMILIES 95

i.e.
(1 + b′ (θ))2 1
b2 (θ) + ≤ for all θ (∗∗)
n n
and so
1
|b (θ)| ≤ √ for all θ (∗ ∗ ∗)
n
and

(1 + b′ (θ))2 ≤ 1
⇒ −2 ≤ b′ (θ) ≤ 0
⇒ b is non-increasing.
′
Now, we claim b (θk ) → 0 for some sequence θk → ∞. If this is not the case,
′
then limθ→∞ b (θ) < 0 and there exists θ0 and ϵ > 0 such that b′ (θ) < −ε for all
θ > θ0 . Then,
Z θ
b (θ) = b′ (y) dy + b (θ0 )
θ0
≤ (θ − θ0 ) (−ε) + b (θ0 ) → −∞ as θ → ∞,
∗
contradicting (***) and thus proving the claim. Similarly, there exists θj → −∞
′ ∗


such that b θj → 0.
∗


Now, (**) implies that b θj → 0 and b (θk ) → 0. But since b is non-increasing,
′
this implies that b (θ) = 0 for all θ and hence that b (θ) = 0 for all θ . Hence, by
1
the information inequality, R (θ, T ) ≥ , and so by (*),
n
1

R (θ, T ) = = R θ, X
n
so that X is admissible.

The above argument also shows that X is minimax since there is no estimator whose
maximum risk is less than 1/n. In fact, X is the unique minimax estimator by the
following theorem.

Proposition 4.3.7. Suppose that T has constant risk and is admissible. Then T is
minimax. If in addition L(θ, ·) is strictly convex, then T is the unique minimax estimator.

Proof. By the admissibility of T , if there is another estimator T ′ with supθ R(θ, T ′ ) ≤

R(θ, T ) then R(θ, T ) = R(θ, T ) for all θ, since the risk of T ′ can't go strictly below that
′

of T for any θ . This proves that T is minimax. If the loss function is strictly convex and
T ′ is a minimax estimator such that Pθ (T ′ ̸= T ) > 0, then if T ∗ = 21 (T + T ′ ),
1
R (θ, T ∗ ) < (R (θ, T ) + R (θ, T ′ )) = R (θ, T ) ,
2
which contradicts the admissibility of T . □
96 4. AVERAGE-RISK OPTIMALITY

Exponential Families (with s = 1). Suppose that the probability density of X

with respect to the σ -nite measure µ is

eθT (x)−φ(θ) h (x)

where Z
eθT (x) h (x) dµ (x) = eφ(θ)
Then
Eθ T (X) = φ′ (θ) = g (θ) .
Suppose the natural parameter space Ω is an interval with end-points θL , θU , −∞ ≤
θL ≤ θU ≤ ∞ and
L (θ, d) = (d − g (θ))2 .
The same argument used in the proof of theorem (4.3.4) shows that aT +b is inadmissible
for (1) a < 0, (2) a > 1, or (3) a=1 and b ̸= 0.
 
If a = 0, then aT + b is admissible since g θ̂ = b is the only estimator with zero risk at

b. To deal with the remaining cases consider

1 rλ
T+ , 0 ≤ λ < ∞, r ∈ R (i.e. 0 < a ≤ 1).
1+λ 1+λ
Theorem 4.3.8 (Karlin's Theorem). The estimator
1 rλ
T+ , 0 ≤ λ < ∞, r ∈ R,
1+λ 1+λ
is admissible for g (θ) = φ′ (θ) = Eθ T if for some (and hence for all) θ0 ∈ (θL , θU )
Z θ0
e−rλθ+λφ(θ) dθ = ∞
θL

and Z θU
e−rλθ+λφ(θ) dθ = ∞.
θ0

Proof. Recall that

φ′ (θ) = Eθ T
φ′′ (θ) = Varθ (T )
 2
∂ log p (x, θ)
I (θ) = Eθ
∂θ
2
= Eθ (T − φ′ (θ)) = φ′′ (θ) .
Suppose there exists δ (X) such that
 2
′ 2 T + rλ ′
Eθ (δ (X) − φ (θ)) ≤ Eθ − φ (θ) for all θ. (∗)
1+λ
 4.3. MINIMAXITY AND ADMISSIBILITY IN EXPONENTIAL FAMILIES 97

We have that
2
Eθ (δ (X) − φ′ (θ)) = Varθ δ (X) + b2 (θ)

2
d
(b (θ) + φ′ (θ))
≥ dθ
+ b2 (θ)
I (θ)
2 (b′ (θ) + I (θ))2
= b (θ) + .
I (θ)
′
(b (θ) exists since Eθ |δ (X)| < ∞.) So by (*),

I (θ) λ2 (r − φ′ (θ))2 2 (b′ (θ) + I (θ))2

+ ≥ b (θ) + (∗∗).
(1 + λ)2 (1 + λ)2 I (θ)
Letting
 
λ T rλ
h (θ) = b (θ) − (r − φ′ (θ)) = b (θ) − bias + ,
1+λ 1+λ 1+λ
λ
h′ (θ) = b′ (θ) + φ′′ (θ) .
1+λ
(**) is exactly equivalent to

′ 1 ′′

2
λ h (θ) + 1+λ
φ (θ) ϕ′′ (θ)
0 ≥ h2 (θ) + 2h (θ) (r − φ′ (θ)) + − (∗ ∗ ∗)
1+λ φ′′ (θ) (1 + λ)2
!
λ 2h′ (θ) h′ (θ)2
= h2 (θ) − 2h (θ) (φ′ (θ) − r) + + ′′ .
1+λ 1+λ φ (θ)
Letting k (θ) = h (θ) erλθ−λφ(θ) , (***) becomes

2
k 2 (θ) erλθ−λφ(θ) + k ′ (θ) ≤ 0 (∗ ∗ ∗∗)
1+λ
⇒ k ′ (θ) ≤ 0 for all θ.
Hence, k (θ) is decreasing. To prove

k (θ) ≥ 0 for all θ,

suppose k (θ0 ) < [Link] k (θ) < 0 for all θ > θ0 . From (****), we also have

k ′ (θ)
 
d 1 1 + λ rλθ−λφ(θ)
=− 2 ≥ e for all θ > θ0 (∗ ∗ ∗ ∗ ∗).
dθ k (θ) k (θ) 2
Integrating both sides of (*****) from θ0θ1 > θ0 ,
to

1 + λ θ1 rλθ−λφ(θ)
Z
1 1
− ≥ e dθ.
k (θ1 ) k (θ0 ) 2 θ0

As θ1 → θU , this integral converges to ∞ by assumption. But the left hand side is less
than − k(θ10 ) so that we obtain the contradiction. Hence
k (θ) ≥ 0 for all θ.
98 4. AVERAGE-RISK OPTIMALITY

Similarly,
Z θ0
erλθ−λφ(θ) dθ = ∞ ⇒ k (θ) ≤ 0 for all θ.
θL

Hence,

k (θ) = 0 ⇒ h (θ) = 0 for all θ ⇒ h′ (θ) = 0 for all θ.

Thus, h′ (θ) = 0 and h (θ) = 0 implies equality in (***), equality in (**), and nally
equality in (*). (RS of (*)=LS of (**) ≥ LS of (*) ≥RS of (**)). Since T1+λ +rλ
has the
T +rλ
same risk as δ , is admissible. □
1+λ

Note [Link] case λ = 0 is of particular interest, i.e. T is admissible for Eθ T

provided θL = −∞ and θU = ∞.
p
Example 4.3.10. Suppose X ∼ b (n, p) and θ := log 1−p , −∞ < θ < ∞. Then

   
n x n−x n θx−n log(1+eθ )
p (x, θ) = p (1 − p) = e
x x
and T (X) = X is admissible for

neθ
ϕ′ (θ) = = np
1 + eθ
since θL = −∞ and θU = ∞.

Example 4.3.11. Suppose X1 , · · · , Xn ∼iid N (θ, σ 2 ) with σ 2 known.

 P 2
xi nθ2
 P 
1 xi
p (x, θ) = √
n exp − exp θ 2 − 2
σ 2π 2σ 2 σ 2σ
P
Xi
T (X) =
σ2
nθ
ϕ′ (θ) = .
σ2
T (X) is admissible for nθ/σ 2 since θL = −∞ and θU = ∞.

4.4. Shrinkage Estimators and Bigdata

The idea behind shrinkage is to deliberately introduce bias in unbiased (or nearly un-
biased) estimators (UMVUE, MLE) in order to reduce their risk. James & Stein (1961)
were the rst to do this. While this was seen as unusual and irrelevant to the applications
of the time, it has become crucially important in the bigdata era. Before telling this story
we discuss extensions to our earlier esults.
 4.4. SHRINKAGE ESTIMATORS AND BIGDATA 99

Simultaneous estimation and extensions of earlier results. So far our esti-

mand, g(θ), has been scalar. In general we want to estimate an r-dimensional function
g(θ) = (g1 (θ), . . . , gr (θ))T , based on an r-dimensional estimator T = (T1 , . . . , Tr )T . The
rst issue is how to dene risk with multivariate arguments. The two most common
denitions under squared error loss are as follows.

(a) Risk under sum of squared errors loss:

r
X
T
R(θ, T ) = Eθ [T − g(θ)] [T − g(θ)] = [Ti − gi (θ)]2 = scalar.
i=1
′
This gives a complete ordering of estimators, i.e., for T and T , exactly one of
R(θ, T ) < R(θ, T ′ ), R(θ, T ) = R(θ, T ′ ), or R(θ, T ) > R(θ, T ′ ), holds. (It will
be our default risk function.)
(b) Risk under concentration matrix loss:
R(θ, T ) = Eθ [T − g(θ)][T − g(θ)]T = matrix (r × r)
= Varθ T , if T is unbiased.

We say that T is more concentrated about g(θ) than T′ if

(4.4.1) R(θ, T ′ ) − R(θ, T ) ≥ 0, (p.s.d.)

This gives only a partial ordering of estimators, because the matrix in (4.4.1)
may be neither psd nor nsd (e.g., when it has both negative and positve eigen-
values).

Note 4.4.1. It can be shown that if R(θ, T ) ≤ R(θ, T ′ ) for every convex loss function
L(θ, d), then (4.4.1) holds (TPE Lemma 5.4.1).

With the (obvious) denition that T is unbiased for g(θ) if and only if Eθ T = g(θ), we
have the following extensions of earlier results.

(1) Rao-Blackwell Theorem (Ch. 2). The multivariate version is essentially the same;
if T0 is unbiased for g(θ) and S is complete & sucient, then E(T0 |S), has uni-
formly minimum risk among all unbiased estimators (is UMVU), and is thus
more concentrated about g(θ) than any other unbiased estimator.
(2) Equivariant Estimation (Ch. 3). All denitions and results apply without change.
(3) Bayes, Minimaxity, Admissibility (Ch. 4).
 The denition of Bayes estimator remains unchanged, but one can often
compute these componentwise by marginalizing both the likelihood and the
prior (e.g., TPE Problem 5.4.3). This marginalization trick always holds
true under sq. error loss (TPE Lemma 5.4.3).
 The denition of minimaxity remains unchanged, but results have to be
derived individually for each situation.
 For admissibility the story is quite dierent, as we will see in the remainder
of this section.
100 4. AVERAGE-RISK OPTIMALITY

James-Stein Estimator. Xi ∼ N (θi , 1) ,i = 1, · · · , s

Let be independent random
variables and let X
L (θ, d) = ∥d − θ∥2 = (di − θi )2 .
The usual (MLE) estimator of θ
T (X) = (X1 , · · · , Xs )
will be shown to be inadmissible if s > 2.
Let  
s−2
δi,c = 1−c 2 Xi , i = 1, · · · , s,
S
Ps
where S2 = i=1 Xi2 . Here
δ1,c
 
.
δc :=  . .
.
δs,c
Theorem 4.4.2.
2c − c2
 
2
R (θ, δc ) = s − (s − 2) Eθ .
S2

To prove the theorem, we need the following two lemmas.

Lemma 4.4.3. If X ∼ N (0, 1) and g : R → R is absolutely continuous with derivative g ′

then
E |g ′ (X)| < ∞ ⇒ Eg ′ (X) = E (Xg (X)) .
1
Proof. Let ϕ (x) = (2π)− 2 exp (−x2 /2). Then

xϕ (x) = −ϕ′ (x) (∗).

Then
Z
′
Eg (X) = g ′ (x) ϕ (x) dx
Z ∞ Z 0
= + g ′ (x) ϕ (x) dx
0 −∞
Z ∞ Z ∞  Z 0 Z x 
′ ′
= g (x) zϕ (z) dz dx − g (x) zϕ (z) dz dx by (*)
0 x −∞ −∞
Z ∞ Z z  Z 0 Z 0 
′ ′
= g (x) dx zϕ (z) dz − g (x) dx zϕ (z) dz by Fubini
0 0 −∞ z
Z ∞ Z 0
= (g (z) − g (0)) zϕ (z) dz − (g (0) − g (z)) zϕ (z) dz
0 −∞
= E (Xg (X)) .

□
 4.4. SHRINKAGE ESTIMATORS AND BIGDATA 101

Lemma 4.4.4. Suppose X = (X1 , · · · , Xs ) where X1 , · · · , Xs are independent and

Xi ∼ N (µi , vi ) .
Suppose f : R → R is absolutely continuous in xi for almost all (x1 , · · · , xi−1 , xi+1 , · · · , xs ).
s

Then if
∂
E f (X) < ∞,
∂xi
∂
vi E f (X) = E (Xi − µi ) f (X) .
∂xi

Proof. Let
Xi − µi
Z= √ .
vi
then from lemma (4.4.3),
∂ √


E ∂z
f x1 , · · · , xi−1 , µi +
vi ζ, xi+1 , · · · , xs |ζ=Z
√


= E Zf x1 , · · · , xi−1 , µi + vi Z, xi+1 , · · · , xs .
Thus,
√ 
∂f

vi E ∂x i
(X) |X 1 , · · · , Xi−1 , Xi+1 , · · · , Xs
 
Xi −µi
= E √vi f (X) |X1 , · · · , Xi−1 , Xi+1 , · · · , Xs .
Taking expectation of each side, we obtain the desired result. □

Now, we are ready to prove the theorem.

Proof. (of theorem) Let

f1
 
c (s − 2) .
fi (X) = Xi and f = . .
.
S2
fs
Then
δc = X − f
and
X 
R (θ, δc ) = E (Xi − fi − θi )2
X X X 
2
= E (Xi − θi ) − 2 (Xi − θi ) fi + fi2
s X X2 
X ∂
= s−2 E fi (X) + E i
4
c2 (s − 2)2 .
1
∂xi S
Since
S 2 − Xi · 2Xi 2Xi2
 
∂ 1
fi (X) = c (s − 2) = c (s − 2) − ,
∂xi S4 S2 S4
102 4. AVERAGE-RISK OPTIMALITY


s−2 1
R (θ, δc ) = s − 2c (s − 2) E 2
+ c2 (s − 2)2 E 2
S S
2
 
2c − c
= s − (s − 2)2 E .
S2

□
Corollary 4.4.5. For 0 < c < 2 and s > 2,
R (θ, δc ) < s for all θ
and δ1 dominates all the other δc 's.

Proof. 2c − c2 > 0 for all c ∈ (0, 2) and has a maximum value at c = 1. □

James-Stein estimator 1 − s−2
X has risk R (θ, δ1 ) = s−(s − 2)2 E 1



Remark 4.4.6. The .
S2 S2
The risk of X is
X
R (θ, X) = E (Xi − θi )2 = s.
Therefore, X is not admissible for θ. In fact, the James-Stein estimator is not admissible
either. A strictly risk-preferable estimator can be arrived at as follows.

Empirical Bayes Interpretation of the James-Stein Estimator. Suppose Θ1 , · · · , Θs

2 2
are iid N (0, τ ), i.e. this is the prior on each Θi . If τ were known, the Bayes estimator
with respect to squared error loss would be
 
Xi 1
θ̂i = = 1− Xi , i = 1, · · · , s.
1 + τ −2 1 + τ2
However, if τ2 is unknown it must be replaced by some estimate. Write

Xi = Θi + Zi , {Zi } ∼ iid N (0, 1) with {Zi } independent of {Θi },

so that Xi |Θi = θi ∼ N (θi , 1). Then,

{Xi } ∼ iid N 0, τ 2 + 1 .


Ps s−2 1
Since S2 = 2
i=1 Xi is complete and sucient for τ 2, S2
is UMVU for
1+τ 2
. So a natural
(empirical Bayes) estimator of θi is
 
s−2
δ1 = 1− X.
S2
Moreover since
1
< 1,
1 + τ2
1 s−2


a better estimate of is min , 1 which suggests using
1+τ 2 S2
    
∗ s−2 s−2
δ1 = 1 − min ,1 X = max 1 − , 1 X.
S2 S2
 4.5. DISCUSSION (EFRON & HASTIE, 2016) 103

∗
δ1∗ is strictly risk-preferable to δ1 (so δ1 δ1
is also inadmissible. It
is inadmissible). But
was a dicult problem to nd an estimator which is strictly risk-preferable toδ1∗ , in fact
it took twenty years. It is now known that there are many admissible minimax estimators
(TPE p. 357).

4.5. Discussion (Efron & Hastie, 2016)

• Although we gave a Bayesian interpretation of the JSE (James-Stein Estimator),
it does not rely on any Bayesian assumptions!
• It's hard to construct JSE-like estimators; have to be done case-by-case (like
UMVUEs). MLE on the other hand provides automatically asymptotically
UMVU estimates (Ch. 6).
• The shrinkage in the above examples was toward zero, but in general it is
toward a common central value like a mean (usually representing a null of no
dierence).
• Classical vs. Bigdata (to shrink or not to shrink): let n denote the sample size
and p the number of parameters to estimate.
Classical Data (n ≫ p) Big Data (n ≈p or n ≪ p)
Shrink? no (generally) yes (generally)
MLE penalized/regularized likelihood
Methods least-squares penalized/regularized least-squares
max. posterior probability (MAP)

• Shrinkage tends to produce better results in general (on average), but this comes
at the expense of extreme cases (outliers). E.g., if most of the θi ≈ 0 in JSE, but
there are a very few large |θi |, the result will be heavy shrinkage of the latter
(toward overall mean ≈ 0). This situation is not uncommon in contemporary
bigdata where the outliers are precisely the interesting cases swimming in a
sea of uninterestingness. . . (see TPE p. 364365).
 CHAPTER 5

Large Sample Theory

This chapter introduces denitions, tools, and techniques for establishing asymptotic
results. Before detailing the dierent modes of convergence, we recall the following basic
result which is often used in proofs (along with the Triangle Inequality).

Proposition 5.0.1 (Chebychev's Inequality). If Eg(X) < ∞, where g(·) is a nonnega-

tive function and ε > 0, then
P (g(X) ≥ ε) ≤ Eg(X)/ε.

Proof. TPE Problem 1.8.1. □

5.1. Convergence in Probability and Order in Probability

Definition 5.1.1. A sequence of random variables Xn is said to converge to 0 in prob-

ability if for any ε > 0,
P (|Xn | > ε) → 0 as n → ∞,
in which case we write
p
Xn →
− 0
or equivalently
Xn = op (1) .
Definition 5.1.2. {Xn } is bounded in probability (or tight ) if for any ε > 0, there exists
M (ε) < ∞ such that

P (|Xn | > M ) < ε for all n, (or equivalently for all n suciently large),

in which case we write

Xn = Op (1) .
Definition 5.1.3.
p p
Xn →
− X ⇐⇒ Xn − X → − 0 ⇐⇒ Xn − X = op (1)
Xn
Xn = op (an ) ⇐⇒ = op (1)
an
Xn
X = Op (an ) ⇐⇒ = Op (1) .
an
Proposition 5.1.4. Let {Xn } and {Yn } be sequences of r.v's and suppose an > 0 and
bn > 0. Then the following results hold:
104
 5.1. CONVERGENCE IN PROBABILITY AND ORDER IN PROBABILITY 105

(1) If Xn = op (an ) and Yn = op (bn ) , then

Xn Yn = op (an bn )
Xn + Yn = op (max (an , bn ))
|Xn |r = op (arn ) , r > 0.
(2) If Xn = op (an ) and Yn = Op (bn ) , then
Xn Yn = op (an bn ) .
(3) If Xn = Op (an ) and Yn = Op (bn ) , then
Xn Yn = Op (an bn )
Xn + Yn = Op (max (an , bn ))
|Xn |r = Op (arn ) , r > 0.

Proof. We only prove the rst part and leave the remaining parts as exercises.

Xn Yn
If
an bn
> ε, then

Yn Xn
either ≤1 and >ε
bn an
Yn Xn Yn
or >1 and > ε.
bn an b n
Thus, if Xn = op (an ) and Yn = op (bn ), then
     
|Xn Yn | Xn Yn
P >ε ≤P >ε +P >1 →0 as n → ∞.
an b n an bn
|Xn +Yn |
If
max(an ,bn )
> ε, then since |Xn + Yn | ≤ |Xn | + |Yn | ,
|Xn | ε |Yn | ε
> or > .
an 2 bn 2
Thus, as in the previous part,
 
Xn + Yn
P >ε →0
max (an , bn )
|Xn |r |Xn | 1
If
arn
> ε, then
an
> εr . Thus,

|Xn |r
 
P > ε → 0.
arn

□
Definition 5.1.5. For a sequence of random vectors Xn = (Xn1 , · · · , Xnm ), we dene
Op and op as follows:
106 5. LARGE SAMPLE THEORY

Xn = op (an ) ⇐⇒ Xnj = op (an ) , j = 1, · · · , m.

Xn = Op (an ) ⇐⇒ Xnj = Op (an ) , j = 1, · · · m.
p p
Xn →
− X ⇐⇒ Xn − X = op (1) ⇐⇒ Xnj →
− Xj , j = 1, · · · , m.
Definition 5.1.6.
m
X
2
∥Xn − X∥ := |Xnj − Xj |2 .
j=1

Proposition 5.1.7.

Xn − X = op (1) ⇐⇒ ∥Xn − X∥ = op (1) .

Proof. ⇒)
m
!
X
P ∥Xn − X∥2 > ε = P |Xnj − Xj |2 > ϵ


1
m
!
[ n ε o
≤ P |Xnj − Xj |2 >
j=1
m
m
X 
2 ε
≤ P |Xnj − Xj | > → 0.
j=1
m

⇐)
|Xni − Xi |2 ≤ ∥Xn − X∥2 ⇒ Xni − Xi = op (1) .

□
p p
Proposition 5.1.8. If Xn − Yn →
− 0 and Yn − Y →
− 0, then
p
Xn − Y →
− 0.

Proof. By the triangle inequality:

∥Xn − Y∥ ≤ ∥Xn − Yn ∥ + ∥Yn − Y∥ = op (1) .

□
p
Proposition 5.1.9 (Continuous Mapping). If Xn →
− X and g : Rm → Rs is continuous,
then
p
g (Xn ) →
− g (X) .

p 
Proof. We note that
n o Xn →
− X if and only if every subsequence Xnj has a subse-

quence Xnjk such that Xnjk → X a.s. as k → ∞. Hence if Xnj is any subsequence
 5.1. CONVERGENCE IN PROBABILITY AND ORDER IN PROBABILITY 107

 
a.s. a.s.
of {Xn }, then there exists a subsequence Xnjk −−→ X whence g Xnjk −−→ g (X). But
p
this implies that g (Xn ) →
− g (X). □
d
Example 5.1.10. If Xn = X , then Xn = Op (1) and Xn = op (an ) for any sequence {an }
such that an → ∞.
Example 5.1.11. Suppose X1 , X2 , · · · iid. Then Xn = Op (1) and Xn = op (an ) if an →
∞. Also, by WLLN and CLT,

n
X op (n)
 if EX1 = 0
Xi ≡ Sn = Op (n) if EX1 ̸= 0
1 O (√n)

if EX1 = 0 and Var (X1 ) < ∞.
p

Taylor Expansions in Probability

Proposition 5.1.12. Suppose Xn = a + Op (rn ) with rn → 0 and rn > 0. If g has s
derivatives at a then
s
X g (j) (a)
g (Xn ) = (Xn − a)j + op (rns ) .
j=0
j!

Proof. Let
(j)

 g(x)− sj=0 g j!(a) (x−a)j
P

(x−a)s if x ̸= a
h (x) := s!
0 if x = a.

Xn −a
Then h is continuous since g has s derivatives at a. Since
rn
= Op (1),
Xn − a = op (1) .
Thus,
p
h (Xn ) →
− h (a) = 0
, i.e.
h (Xn ) = op (1) .
Thus,
(Xn − a)s
h (Xn ) = op (rns ) .
s!
□
Example 5.1.13. Suppose {Xn } ∼ (µ, σ 2 ), µ > 0. By
iid Chebychev,
 1
X n = µ + Op n− 2 .
Thus,
1  1
X − µ + o p n− 2


log X n = log µ +
µ
108 5. LARGE SAMPLE THEORY

and √
√
n

n log X n − log µ = X n − µ + op (1) .
µ

We end this section with a multivariate version of Proposition 5.1.12.

Proposition 5.1.14. Suppose Xn = a+Op (rn ) with a ∈ Rm and rn → 0. If g : Rm 7→ R

with continuous derivatives ∂g/∂xj in a neighborhood of a, then:
m
X ∂g
g (Xn ) = g (a) + (a) (Xnj − aj ) + op (rn ) .
j=1
∂xj

Proof. Brockwell and Davis (1991, Proposition 6.1.6). □

5.2. Convergence in Distribution

Definition 5.2.1. We say that a sequence of random vector Xn converges to X and

denote
d
Xn →
− X
if FXn (x) → FX (x) for all x ∈ C = {x : FX is continuous at x}.
Remark 5.2.2. Convergence for all x is too stringent a requirement as illustrated by
1 d
Xn ≡ n
. We would like to say Xn →
− X ≡ 0 even though FXn (0) = 0 ̸→ FX (0) = 1.
Theorem 5.2.3. Suppose Xn ∼ Fn and X ∼ F0 . Then the following are equivalent:
d
(1)
Rn→
X − X.
g (x) dFn (x) → g (x) dF0 (x) for all bounded continuous function g .
R
(2)

dFn (x) → eit XdF0 (x)for all t ∈ Rm

R itT x R T
(3) e
(i.e. ϕn (t) := E eit =: ϕ0 (t) for all t ∈ Rm .)
TX TX
n
→ E eit

Proof. See Billingsley pp.378-383. □

Note: This theorem enables us to prove the Cramer-Wold device:

d d
Xn →
− X ⇐⇒ λT Xn →
− λT X, for all λ ∈ Rm .
Tx
Proof. ( ⇒) Apply Theorem 5.2.3 (#2) with g(x) = eitλ to get

d
ϕλT Xn (t) → ϕλT X (t) =⇒ λT Xn →
− λT X.

(⇐) Apply Theorem 5.2.3 (#3) to get

ϕXn (λ) = ϕλT Xn (1) → ϕλT X (1) = ϕX (λ) .

□
 5.2. CONVERGENCE IN DISTRIBUTION 109

p
Proposition 5.2.4. If Xn →
− X, then

→ 0 for all t ∈ Rm and

TX TX
(1) E eit n
− eit
d
(2) − X.
Xn →

Proof. (1) Since

T (X −X) T (X −X)
E 1 − eit n
≤ E 1 − eit n
I∥Xn −X∥≤δ + 2P (∥Xn − X∥ > δ) ,
given any ε > 0, we can choose δ to make the rst term less than ε/2. Then
choose n to make the second term less than ε/2 . Hence the left hand side
converges to 0 as n → ∞.
(2) Since |·| is convex, by Jensen's inequality,
TX TX TX TX
Eeit n
− Eeit ≤ E eit n
− eit → 0.
Thus, by theorem (5.2.3)
d
Xn →
− X.

□
d
Proposition 5.2.5. (Slutzky's theorem) If Xn − Yn = op (1) and Xn →
− X, then
d
Yn →
− X.

Proof. For a random vector Z, dene ϕZ (θ) by

Z
T
ϕZ (θ) := eiθ z dFZ (z)

Then, we have

|ϕYn (t) − ϕX (t)| ≤ |ϕYn (t) − ϕXn (t)| + |ϕXn (t) − ϕX (t)|
By Jensen's inequality and the same argument as in proposition (5.2.4)-(1),
T (X −Y )
|ϕYn (t) − ϕXn (t)| ≤ E 1 − e−it n n
→ 0.
d
Also, since Xn →
− X, the second term converges to 0. □
d
Proposition 5.2.6 (Continuous Mapping). If Xn →
− X and h : Rm → Rs is continuous,
then
d
h (Xn ) →
− h (X) .

Proof. Since exp itT h (·) is a bounded continuous function,
T h(X ) T h(X)
Eeit n
→ Eeit .
□
110 5. LARGE SAMPLE THEORY

d p
Proposition 5.2.7. (Generalized Slutzky's theorem) If Xn →
− X and Yn →
− b, with
dim(Yn ) = m1 and dim(Xn ) = m2 , then

   
Xn d X
→
− .
Yn b
In particular, we have the following special cases:
d d
(1) If m1 = m2 , then Xn + Yn → − bT X.
− X + b and YnT Xn →
(2) If g : R ×R → R is continuous at every point where the vector (b, X) takes
m1 m2 s
d
its values, then g(Yn , Xn ) →
− g(b, X).
   
Xn Xn p
Proof. Let Zn = . Then, Zn − →
− 0, since each component converges
b Yn
 
d X
to 0 in probability. Also Zn →
− since the sequence of characteristic functions
b
converges. Hence by Slutzky,
   
Xn d X
→
− .
Yn b
Because the mappings (x, y) → x + y and (x, y) → x′ y are continuous from R2m → Rm
when m1 = m2 = m, we obtain result (1) from proposition (5.2.6). Result (2) follows
similarly. □

Asymptotic Normality
Definition 5.2.8. A sequence of random variables {Xn } is said to be asymptotically
normal with mean µn and standard deviation σn if σn > 0 for all suciently large n and
if
d
σn−1 (Xn − µn ) →
− Z
, where Z ∼ N (0, 1). We write

Xn is AN ( µn , σn2 ).
|{z} |{z}
asymptotic asymptotic
mean variance

Example 5.2.9. The classical CLT states that if

  X1 , · · · , Xn are iid with mean µ and
2 σ2
variance σ , then Xn is AN µ, n
i.e.

√
n
d
Xn − µ →− Z.
σ
Proposition 5.2.10 (Delta Method). If Xn is
AN (µ, σn2 )with σn → 0 and g is dieren-
tiable at µ, then g (Xn ) is AN g (µ) , g ′ (µ)2 σn2 .
 5.2. CONVERGENCE IN DISTRIBUTION 111

Proof. Because
Xn − µ d
Zn := →
− Z ∼ N (0, 1) ,
σn
Zn = Op (1) .
(Choose a pair of continuity points of Fz such that F (z1 ) < ε/4 and F (z2 ) > 1 − ε/4.
Then for all n > N (ε),
ε ε
FZn (z1 ) < and FZn (z2 ) > 1 − .
3 3
For n = 1, · · · , N (ε), there exists v (ε) such that
P (|Zn | > v (ε)) < ε n = 1, · · · , N (ε) .
Choose M (ε) ≥ max (v (ε) , |z1 | , |z2 |). Then

P (|Zn | > M (ε)) < ε for all n = 1, 2, · · · .)

Thus,
Zn = Op (1) ⇒ Xn = µ + Op (σn ) .
′
By proposition (5.1.12) (g (X) = g (µ) + g (µ) (X − µ) + op (σn )),

g (Xn ) − g (µ) g ′ (µ) (Xn − µ) d

− N 0, g ′ (µ)2 .


= + op (1) →
σn σn

□
 
1 σ2
Example 5.2.11. Let {Xn } ∼iid (µ, σ 2 ). Then (X1 + · · · + Xn ) is AN µ, n .
X n := n
  2 2 
1 1
Suppose µ ̸= 0. Then g (x) := x
has derivative at µ so 1/X n is AN
µ
1
, − µ2 σn , i.e.
√
nµ2
 
1 1 d
− →
− N (0, 1) .
σ Xn µ
p
Moreover since Xn →
− µ (by the WLLN), proposition (5.2.10) implies
√ 2  
nX n 1 1 d
− →
− N (0, 1) .
σ Xn µ
1
Note 5.2.12. Although
µ
is the asymptotic mean of 1/X n in the above example, it is

not


the limit as n→∞ of E 1/X n . In fact, E 1/X n = ∞ if X1 ∼ N (µ, σ 2 ).
Example 5.2.13. Suppose X1 , · · · , Xn ∼ iid (0, σ 2 ). Then
√
nX n d
σ
→
− Z ∼N (0, 1) by CLT
2 d
⇒ nX n → − σ2Z 2,
where Z 2 ∼ χ2 (1) 2
since g (x) = x is continuous.

Multivariate Asymptotic Normality

112 5. LARGE SAMPLE THEORY

Definition 5.2.14. Xn is AN (µn , Σn ) if

(1) Σn has no zero diagonal elements

for all large enough n.
(2) λ Xn is AN λ µn , λ Σn λ for all λ ∈ Rm such that λT Σn λ > 0
T T T
for all large
enough n.

Recalling the Cramer-Wold device, i.e.

d d
Xn →
− X ⇐⇒ λT X n →
− λT X for all λ ∈ Rm ,
we see that Xn is AN (µn , Σn )where Σn satises (1) if and only if

λT (Xn − µn ) d
√ →
− N (0, 1) ,
λT Σn λ
for all λ such that λT Σn λ > 0 for large enough n.
Proposition 5.2.15 (Multivariate Delta Method). Suppose Xn is AN (µ, c2n Σ) with
cn → 0. If g : R → R is continuously dierentiable in a neighborhood of µ and
m k

DΣDT has all diagonal elements greater than 0 where D = [∂gi /∂xj ]X=µ , then g (Xn ) is
AN g (µ) , c2n DΣDT .

Definition 5.2.16. A sequence of estimates Tn = Tn (X1 , · · · , Xn ) of g (θ) is said to be

(weakly) consistent if
p
Tn →
− g (θ) for all θ (where p is under the measure Pθ ),
and strongly consistent if

Tn → g (θ) a.s. Pθ for all θ.

Example 5.2.17. (Moment Estimation) Let {XN } ∼iid Pθ such that Eθ |X|r < ∞ for
all θ. Suppose mj (θ) = Eθ X1j for 1 ≤ j ≤ r and g (θ) = ϕ (m1 (θ) , · · · , mr (θ)) where ϕ
is continuous. Then

Tn (X1 , · · · , Xn ) = ϕ (m̂1 , · · · , m̂r ) → g (θ) a.s. Pθ ,

where
n
1X j
m̂j = X .
n k=1 k

Definition 5.2.18. A sequence of estimators is said to be asymptotically normal if there

exists µn (θ) and σn (θ) > 0 such that Tn is AN (µn (θ) , σn2 (θ)) for all θ. i.e.
 
Tn − µn (θ)
Pθ ≤ x → Φ (x) for all θ.
σn (θ)
Remark 5.2.19. Suppose Tn is AN (µ′n (θ) , σn′2 (θ)). If

σn (θ) µn (θ) − µ′n

→1 and → 0,
σn′ (θ) σn
 5.2. CONVERGENCE IN DISTRIBUTION 113

then Tn is AN (µn (θ) , σn2 (θ)) since

Tn − µn σ ′ Tn − µ′n µ′n − µn
= n· +
σn σn σn′ σn
d↓ ↓ d↓ d↓
N (0,1) 1 N (0,1) 0

Definition 5.2.20. A sequence of asymptotically normal estimators {Tn } is said to be

asymptotically unbiased for g (θ) if

µn (θ) − g (θ)
→ 0,
σn (θ)
in which case
Tn − g (θ) d
→ N (0, 1)
σn (θ)
and
µn (θ)−g(θ)
σn (θ)
is called the (standardized) asymptotic bias.

It follows that if both the bias and variance of Tn go to zero, then Tn is consistent for its
mean.
p
Proposition 5.2.21. If ETn → µ and Var(Tn ) → 0, then Tn →
− µ.

Proof. Chebychev's Inequality and Triangle Inequality. □

Theorem 5.2.22 (Multivariate CLT). If {Xn } ∼ iid (µ, Σ), then
√
 
1
X n ∼ AN µ, Σ , whence X n = µ + Op (1/ n).
n
Example 5.2.23. Let {Xn } ∼ iid Pθ such that Eθ |X|2r < ∞ for all θ. Suppose mj (θ) =
Eθ X1j for1 ≤ j ≤ r, g (θ) = ϕ (m1 (θ) , · · · , mr (θ)) for somePcontinuously dierentiable
1 n j
function ϕ, and dene Tn = ϕ (m̂1 , · · · , m̂r ), where m̂j := j=1 Xi . From the CLT,
n

m̂1 − m1 (θ)
 
√ . d i j r



n .
.
→ N (0, Σ) , where Σ = (Σij ) = Cov X1 , X1 .
i,j=1
m̂r − mr (θ)
Then, using Proposition 5.1.14, we have
X ∂ϕ  
1
Tn = ϕ (m1 (θ) , · · · , mr (θ)) + (m (θ)) (m̂j − mj (θ)) + op √ .
j
∂xj n
Finally, applying Proposition 5.2.15 gives:

Tn − ϕ (m1 (θ) , · · · , mr (θ)) d

→ N (0, 1) ,
σn (θ)
where (assuming Σ > 0)
∂ϕ/∂m1
 
 
1 ∂ϕ ∂ϕ .
σn2 (θ) = ,··· , Σ .
.
.
n ∂m1 ∂mr
∂ϕ/∂mr
114 5. LARGE SAMPLE THEORY

Example 5.2.24. Let X1 , · · · , Xn ∼ Γ (α, β). Then EX1 = αβ , EX12 = αβ 2P+ α2 β 2 =

1
m2 (θ)−m21 (θ) Xi2 −X̄ 2
β 2 (α2 + α), and
2
VarX1 = αβ . ϕ(m1 (θ), m2 (θ)) = β =
m1 (θ)
. Tn = n
X̄
=
m̂2 −m̂21
m̂1
. Then Tn AN (β, σn2 (θ)),where
is

m2
1 −2m21 − (m2 − m21 ) 1 X −1 −
   
2 m21
σn (θ) = , 1
n m21 m1 m1

and
 
Cov(X1 , X1 ) Cov(X1 , X12 )
Σ=
Cov(X1 , X12 ) Cov(X12 , X22 )
 
αβ 2 α(α + 1)(α + 2)β 3 − αβαβ 2
=
α(α + 1)(α + 2)β 3 − αβαβ 2 α(α + 1)(α + 2)(α + 3)β 4

5.3. Asymptotic Comparisons (Pitman Eciency)

√ d √  ′ 
d
Definition 5.3.1. If − N (0, σ02 (θ)) and
n (Tn − g(θ)) → n Tn′ (n) (X1 , . . . , Xn′ ) − g(θ) →−
N (0, σ02 (θ)), then the Pitman asymptotic relative eciency (ARE) of {Tn } relative to
{Tn′ } is
n′ (n)
eT, T′ (θ) = lim
n→∞ n
provided the limit exists and is independent of the sequence n′ (n) chosen to satisfy

√ d
n Tn′ ′ (n) − g(θ) →
− N (0, σ02 (θ))

Roughly speaking, eT, T ′ is the ratio of the number of observations required for the two
estimators to achieve the same precision.
   
σ 2 (θ) σ 2 (θ)
Theorem 5.3.2. if Tn ∼ AN g(θ), 0n and Tn′ ∼ AN g(θ), 1n , then eT,T ′ (θ) =
σ12 (θ)
σ02 (θ)
.

h 2 i
′ σ (θ)
Proof. Let n = n σ12 (θ) , where [x] is the integer part of x. Then
0
s
√ σ12 (θ)

n′ (Tn′ ′ − g(θ)) = n 2 (Tn′ ′ − g(θ))
σ0 (θ)
σ1 (θ) √
 
1
= n (Tn′ ′ − g(θ)) + Op
σ0 (θ) n
d
and since LHS − N (0, σ12 (θ)), therefore
→
√
n (Tn′ ′ − g(θ)) → N 0, σ02 (θ)


 5.4. M-ESTIMATION THEORY 115

Hence
n′ (n) σ 2 (θ)
eT, T ′ (θ) = lim = 12
n→∞ n σ0 (θ)
′ ′
provided the limit is the same for all n (n) such that lim n (n)/n exists and
√
n (Tn′ ′ − g(θ)) → N 0, σ02 (θ)

Suppose n′ (n) is any such sequence, then the LHS of the previous line is equal to
√
n√ ′ ′ n σ1 σ0 √ ′ ′
r  
√ n (Tn′ − g(θ)) = n (Tn′ − g(θ))
n′ n′ σ 0 σ 1
d↓
( )
2 (θ)
N 0, σ0

2 σ12
n σ1 n′
therefore
n′ σ02
→ 1, i.e.
n
→ σ02
. □

5.4. M-Estimation Theory

M-estimation is a very general method to derive consistency and asymptotic normality

results for a lot of classical estimators that are obtained by solving a system of equation(s).
This includes e.g., method of moments estimators and maximum likelihood estimators.
The main idea is to write the system as an empirical average, to which the CLT is applied
after a Taylor series expansion. See Sering (1980, Ch 7) for a classical treatment, and
Van der Vaart (1998, Ch 5) for a more recent coverage.

Background
• X1 , . . . , Xn ∼ iid Fθ (x), fθ (x), with θ = (θ1 , . . . , θd ) ∈ Ω ⊆ Rd , let θ0 denote its
true value, and θ̂n = θ̂n (X1 , . . . , Xn ) its estimator.
• Dene θ̂n as an M-estimator :
n
1X
θ̂n = arg max Mn (θ), Mn (θ) = m(xi , θ),
θ∈Ω n i=1
where m(x, θ) is a scalar concave function of θ (which ensures the maximum
exists, but can be relaxed).
• Often θ̂n is found as a root of the equation that results from dierentiating
Mn (θ):
n
∂ 1X
Ψn (θ) = Mn (θ) = ψ(xi , θ) = 0,
∂θ n i=1
where ψ(x, θ) is the vector-valued map

ψ1 (x, θ)
 
. ∂m(x, θ)
ψ(x, θ) =  . = := ṁ(x, θ).
.
∂θ
ψd (x, θ)
116 5. LARGE SAMPLE THEORY

• In both cases, the true value θ0 satises:

θ0 = arg max Eθ0 m(X, θ), and Eθ0 ψ(X, θ) = 0.

θ∈Ω

(Note: nding the appropriate m(.) and/or ψ(.) functions may be challenging.)

Example 5.4.1 (Estimation of a location parameter). For estimating a measure of loca-

tion like the mean (µ), median (ξ0.5 ), and a quantile in general (ξp ), proceed as follows:

• For µ use m(x, θ) = −(x − θ)2 and ψ(x, θ) = (x − θ). Then note that µ̂ = x̄
solves:
n
1X
(xi − θ) = 0.
n i=1
• For ξ0.5 use m(x, θ) = −|x − θ| and ψ(x, θ) = 1(x > θ) − 1(x < θ). Then note
that ξˆ0.5 solves:
n
1X
[1(xi > θ) − 1(xi < θ)] = 0.
n i=1

Example 5.4.2 (Maximum likelihood estimation). Using m(x, θ) = log fθ (x) leads to
the log-likelihood and score functions:
n n
1X 1X ∂
Mn (θ) = log f (xi , θ), Ψn (θ) = log f (xi , θ) = 0.
n i=1 n i=1 ∂θ

Consistency. Consistency of θ̂n follows if the functions {m(x, θ) : θ ∈ Ω} or {ψj (x, θ) :

θ ∈ Ω, j = 1, . . . , d} are Glivenko-Cantelli (Van der Vaart, 1998, Ch 19). A simple set
of sucient conditions for this is:

(i) Ω is a compact set; and

(ii) the maps θ 7→ m(x, θ) or θ 7→ ψj (x, θ) are continuous ∀ x and ∀ j , and are
dominated by an integrable function H(x) in the vicinity of θ0 ; i.e.,
m(x, θ) ≤ H(x), ψj (x, θ) ≤ H(x), with EH(X) < ∞.
(Note: H(x) can depend on θ0 but not θ.)

Asymptotic Normality (AN). Under mild regularity assumptions θ̂n is AN:

√  
d
n θ̂n − θ0 −→ N 0, Vθ−1 Wθ0 Vθ−T


(5.4.1)
0 0
,

where the d×d constituent matrices in the asymptotic covariance can (under appropriate
conditions) be expressed either in terms of derivatives of the m(.) or ψ(.) functions
evaluated at θ0 , as follows:
  T
T ∂m(X, θ) ∂m(X, θ)
Wθ0 = Eθ0 ψ(X, θ)ψ(X, θ) = Eθ0 ,
∂θ ∂θ
 5.4. M-ESTIMATION THEORY 117

and
∂ 2 Eθ0 m(X, θ)
   
∂ψ(X, θ)
Vθ0 = Eθ0 = .
∂θT ∂θ∂θT
The regularity conditions needed here are not easy to state; most importantly they require
continuity (in probability) of the map θ 7→ ψ(·, θ), that Vθ0 be nonsingular, and that the
collection of maps x 7→ ψ(x, θ) form a Donsker Class (Van der Vaart, 1998, Ch 19).

Note: If changing the order of dierentiation (w.r.t. θ) and integration (w.r.t. x) are
permissible, then the two denitions of Vθ0 above coincide. (This fails e.g., in the case of
quantiles, see below.)

Proof. A sketch of the AN proof in the d = 1 case is as follows. Taylor expand

ψ(x, θ) θ0 :
about
X X X
ψ(xi , θ) = ψ(xi , θ0 ) + (θ − θ0 ) ψ̇(xi , θ0 ) + remainder.
P
Now substitute θ = θ̂n , whence the LHS is zero since θ̂n ψ(xi , θ̂n ) = 0, and
solves
since it can be shown that the remainder is
√ op (1), we obtain, after rearranging terms and
multiplying by n:
− √1n ψ(xi , θ0 )
P
√ A
n(θ − θ0 ) = 1 P := .
n
ψ̇(xi , θ0 ) B
Now analyze the numerator and denominator terms separately. For the rst we obtain
that
√ √
 
1X

A= n − ψ(Xi , θ0 ) = n Ȳn − 0 ,
n
where Yi = −ψ(Xi , θ0 ) ∼ iid (µY , σY2 ) with µY = −Eθ0 ψ(X, θ0 ) = 0 and σY2 = V arθ0 ψ(X, θ0 ) =
d
Eθ0 ψ 2 (X, θ0 ). Thus by applying the CLT we obtain: A → N (0, σY2 ). For the denomina-
2
tor, apply the WLLN to Zi = ψ̇(Xi , θ0 ) ∼ iid (µZ , σZ ), where µZ = Eθ0 ψ̇(X, θ0 ), to see
p
that: B → µZ . Putting it all together using Slutzky gives:

√ 2
σY2 Eθ0 ψ 2 (X, θ0 )
 
d N (0, σY )
n(θ − θ0 ) → ∼ N 0, 2 = .
µZ µZ [Eθ0 ψ̇(Xi , θ0 )]2
□
Example 5.4.3 (Estimation of mean). For X1 , . . . , Xn ∼ iid (µ0 , σ02 ), use ψ(x, θ) =
(x − µ) for estimation of µ0 via the sample mean X̄ , implying ψ̇(x, µ) = −1, so that
Eµ0 ψ̇(X, µ) = −1. Now, Eµ0 ψ 2 (X, µ) = Eµ0 (X − µ0 )2 = σ02 , and thus we obtain the
classical CLT result:
!
√ d Eµ0 ψ 2 (X, µ0 )
n(X̄ − µ0 ) → N 0, = σ02 .
[Eµ0 ψ̇(X, µ0 )]2
118 5. LARGE SAMPLE THEORY

5.5. Example: AREs of Mean, Median, Trimmed Mean

Proposition 5.5.1 (AN for a Quantile). Let F (x) be a cdf such that F (x) is dierentiable
at ξp = F (p), the p-quantile of F , and that f (ξp ) = dFdx(x) x=ξp > 0. Let X1 , . . . , Xn
−1

be iid F and let Y1 ≤ · · · ≤ Yn be the order statistics. Then, if [np] denotes any of the
integers on either side of np, we have for any 0 < p < 1:
 
p(1 − p)
Y[np] ∼ AN ξp , .
nf 2 (ξp )

Proof. The p-quantile θ0 = ξp is an M-estimator with m(x, θ) = (1 − p)(x − θ)1(x <

θ) − p(x − θ)1(x > θ). To see why, consider:
Z θ Z ∞
Eθ m(X, θ) = (1 − p) (x − θ)dF (x) − p (x − θ)dF (x) := g(θ).
−∞ θ

Dierentiating using Leibnitz's Rule:

" #
Z b(θ) Z b(θ)
d ∂
h(x, θ)dx = h(b, θ)b′ (θ) − h(a, θ)a′ (θ) + h(x, θ)dx
dθ a(θ) a(θ) ∂θ

gives,

g ′ (θ) = −(1 − p)[F (θ) − 0] + [1 − F (θ)]p = p − F (θ), g ′′ (θ) = −f (θ) = Vθ ,

whence we note that θ0 = ξp = arg max Eθ0 m(X, θ). To obtain Wθ :
∂m(x, θ)
= ψ(x, θ) = p1(x > θ) − (1 − p)1(x < θ)
∂θ
ψ 2 (x, θ) = p2 1(x > θ) + (1 − p)2 1(x < θ)
Z ∞ Z θ0
2 2 2
Wθ0 = Eθ0 ψ (X, θ) = p dF (x) + (1 − p) dF (x)
θ0 −∞
2 2
= p (1 − p) + (1 − p) p = p(1 − p).

Thus invoking the result in (5.4.1):

Eθ0 ψ 2 (X, θ) p(1 − p)

Vθ−1 Wθ0 Vθ−T = ′′ 2
= 2 .
0 0
[g (θ0 )] f (ξp )
□
Proposition 5.5.2. Suppose F (x) is continuous at the quantiles ξp1 and ξp2 , and that
f (ξp1 ) > 0 and f (ξp2 ) > 0. Then, for 0 < p1 < p2 < 1, we have that
     
Y[np1 ] ξp1 1 p1 (1 − p1 )/f 2 (ξp1 ) p1 (1 − p2 )/(f (ξp1 )f (ξp2 ))
∼ AN , .
Y[np2 ] ξp2 n p1 (1 − p2 )/(f (ξp1 )f (ξp2 )) p2 (1 − p2 )/f 2 (ξp2 )
 5.5. EXAMPLE: ARES OF MEAN, MEDIAN, TRIMMED MEAN 119

Proof. This is a d = 2 version of the previous proposition, θ = (θ1 , θ2 ), where

θ0 = (ξp1 , ξp2 ), and we take

2
X
m(x, θ) = [(1 − pj )(x − θj )1(x < θj ) − pj (x − θj )1(x > θj )] .
j=1

Then, Eθ m(X, θ) = g1 (θ1 ) + g2 (θ2 ) := g(θ), with

Z θj Z ∞
gj (θ) = (1 − pj ) (x − θj )dF (x) − pj (x − θj )dF (x),
−∞ θj

which implies

∂ 2 g(θ)
 
−f (ξp1 ) 0
Vθ0 = = .
∂θ∂θT θ=θ0
0 −f (ξp2 )
It's now easy to see that θ0 is a root of the Jacobian of g(θ), and that its Hessian Vθ0
is negative denite (hence nonsingular), whence Eθ0 m(X, θ) is maximized at θ0 . The
elements of Wθ0 follow similarly to the d=1 case:
   
∂m(x, θ) p1 1(x > θ1 ) − (1 − p1 )1(x < θ1 ) ψ1 (x, θ1 )
= = .
∂θ p2 1(x > θ2 ) − (1 − p2 )1(x < θ2 ) ψ2 (x, θ2 )
so that, upon noting that Eθ0 ψj2 (X, θj ) = pj (1 − pj ) and Eθ0 ψ1 (X, θ1 )ψ2 (X, θ2 ) = p1 (1 −
p2 ) , leads to
   
ψ12 (X, θ1 ) ψ1 (X, θ1 )ψ2 (X, θ2 ) p1 (1 − p1 ) p1 (1 − p2 )
Wθ0 = Eθ0 = .
ψ1 (X, θ1 )ψ2 (X, θ2 ) ψ22 (X, θ2 ) p1 (1 − p2 ) p2 (1 − p2 )

Computing Vθ−1
0
Wθ0 Vθ−T
0
then leads to the stated asymptotic covariance matrix. □
Proposition 5.5.3 (AN for Trimmed Mean). Let F (x) be symmetric about 0 and suppose
∃ 0 < c ≤ ∞ such that F (−c) = 0, F (c) = 1, and that f (x) is strictly positive and
continuous on (−c, c). If X1 , . . . , Xn are iid F (x − θ), then for any 0 < α < 1/2
σα2
 
X̄α ∼ AN 0, ,
n
where
n−[nα] Z ξ1−α 
1 X 2
X̄α = Yi , σα2 = 2
t f (t)dt + 2
αξ1−α ,
n − 2[nα] (1 − 2α)2 0
i=[nα]+1

Y1 ≤ · · · ≤ Yn are the order statistics, and ξα = F −1 (α).

Proof. Omitted, but can be proved similarly via M-estimation. □

These results can now be used to eect asymptotic relative eciency calculations.
120 5. LARGE SAMPLE THEORY

Remark 5.5.4. Suppose X1 , . . . , Xn ∼ iid F (x − θ), F (0) = 1/2, and f (0) > 0. Then
 
1
X̃ := median(X1 , · · · , Xn ) = X([n/2]) ∼ AN θ, .
4nf 2 (0)
If EX1 = θ and σ 2 = V arθ X1 , then from Example 5.4.3,X̄ ∼ AN (θ, σ 2 /n). Therefore,
eX̃, X̄ = 4σ 2 f 2 (0). So if 2σf (0) < 1 then X̄ is more ecient, whereas if 2σf (0) > 1 then

X̃ is more ecient.
Remark 5.5.5. Again assume X1 , . . . , Xn ∼ iid F (x−θ), F (−c) = 0, and f is symmetric
continuous and positive. Then, since X̄α → X̃ as α ↑ 1/2 and X̄α → X̄ as α ↓ 0:

1
lim1 σα2 = 2 , and lim σα2 = σ 2 ,
α↑ 2 4f (0) α↓0

so that the ARE's of X̃ and X̄α relative to X̄ are:

2 2
eX̃, X̄ (f ) = 4σ f (0), and eX̄α , X̄ (f ) = σ 2 /σα2 .
Dening the following mixture of two normals

1 2 1 2
T (ϵ, τ ) = (1 − ϵ) √ e−x /2 + ϵ √ e−x /2τ ,
2π τ 2π
some numerical computations for dierent f 's and α's then lead to the following AREs
of X̄α relative to X̄ :

f (x)\α .125 .25 .5

2
√1 e−x /2 .94 .84
2
=.64
2π π
1
π(1+x2 )
∞ ∞ ∞
1 −|λ|/2
2
e 1.40 1.63 2
T (.01, 3) .98 .89 .68
T (.05, 3) 1.19 1.09 .83
t3 1.91 1.97 1.62
t5 1.24 1.21 .96

Remark 5.5.6. X̄ is inecient for heavy tails. This is because it is sensitive to one or
two extreme observations.

Remark 5.5.7. The optimal α depends on the distribution sampled. For large n the
distribution can be estimated and α chosen accordingly as α̂.
 CHAPTER 6

Maximum Likelihood Estimation

Assessing the performance of dierent types of estimators (UMVUE, MRE, Bayes, MLE)
is usually dicult in nite samples. This task is made considerably simpler as n → ∞,
and especially for the Maximum Likelihood Estimator (MLE). This chapter discusses
the various notions of asymptotic assessment and optimality, and establishes the relevant
classical results for the MLE.

Efron & Hastie (2016) poignantly summarize the success of maximum likelihood estima-
tion in the following quote:

If Fisher had lived in the era of apps, maximum likelihood estimation

might have made him a billionarie. Arguably the 20th century's most
inuential piece of applied mathematics, maximum likelihood continues
to be a prime method of choice in the statistician's toolkit. Roughly
speaking, maximum likelihood provides nearly unbiased estimates of
nearly minimum variance, and does so in an automatic way.

6.1. Consistency

Suppose that X1 , X2 , . . . are iid Pθ , θ = (θ1 , . . . , θd ) ∈ Ω ⊆ Rd , and make the following

assumptions.

(A0 ) Pθ ̸= Pθ′ if θ ̸= θ′ .
(A1 ) Pθ , θ ∈ Ω, have common support.
d
(A2 ) P (x) = f (x, θ).
dµ θ
(A3 ) The true parameter θ0 ∈ int(Ω).

1 f (xi , θ) is called the likelihood, and ℓ(x, θ) =

Qn Pn
Definition 6.1.1. L(x, θ) = 1 log f (xi , θ)
is the log likelihood. An estimator θ̂ of θ is called a (global) MLE if

ℓ(x, θ̂(x)) = sup ℓ(x, θ).

θ∈Ω

The MLE of g(θ) is dened to be g(θ̂).

121
122 6. MAXIMUM LIKELIHOOD ESTIMATION

Likelihood equations. If θ∈Ω and Ω is an open subsect of Rd and ℓ is dierentiable

on Ω, then θ̂ (if it exists) satises

∂ℓ  
x, θ̂(x) = 0, 1 ≤ j ≤ d.
∂θj
In general
▽θ ℓ(x, θ) = 0
may not have a unique solution. Sometimes the likelihood is unbounded and the MLE
does not exist.

Example 6.1.2.

1 − e−ax

0≤x<τ
F (x, θ) =
1 − e−aτ −b(x−τ ) x≥τ
θ ∈ Ω = (a, b, τ ) ∈ (0, ∞)3


f (x, θ) = ae−ax I[0,τ ) (x) + be−aτ −b(x−τ ) I[τ,∞) (x)


f (x, θ) a x < τ,
Hazard rate = =
1 − F (x, θ) b x ≥ τ.
k
X n
  X  
ℓ(x, a, b, τ ) = −ax(i) + log a + −aτ − b(x(i) − τ ) + log b ,
1 k+1

where x(i) is the ith order statistic, and k = #{xi : xi < τ }.

For any xed a, letting b= 1
x(n) −τ
and τ ↑ x(n) , we see that ℓ(x, a, x(n)1−τ , τ ) → ∞ as τ ↑
x(n) , and hence a global MLE does not exist. However if we restrict τ < x(n−1) the
constrained MLE exists and is consistent.

Lemma 6.1.3. Let

f (X, θj )
I(θj |θi ) = −Eθi log
f (X, θi )
Z
f (x, θj )
= − log f (x, θi )dµ(x)
f (x, θi )
= Kullback-Leibler discrepancy of f (·, θj ) relative to f (·, θi )
Then I(θj |θi ) ≥ 0 with equality holding if and only if θj = θi .

Proof. Jensen's inequality gives

f (X, θj ) f (X, θj )
−Eθi log ≥ − log Eθi
f (X, θi ) f (X, θi )
Z
≥ − log f (x, θj )dµ(x)
{x:f (x, θi )>0}
≥ − log 1 = 0
 6.1. CONSISTENCY 123

f (X,θj ) R
Equality holds if and only if is constant a.s. Pθi , and f (x, θj )dµ(x) = 1.
f (X,θi ) {x:f (x, θi )>0}
The latter equality implies Pθj ≪ Pθi and the former equality implies Pθj = Pθi , since
R
cf (x, θi )dµ = 1 implies that c = 1. □

Note: All theorems in this section apply only to the one-dimensional case of Ω ⊂ R,
but they are the easiest to check! For multidimensional versions, use the M-estimation
method in Ÿ5.4.

Theorem 6.1.4. Suppose Ω = {θ0 , . . . , θk } is composed of nitely many elements, and

conditions A0 − A2 hold. Then the MLE θ̂n = θ̂(X1 , · · · , Xn ) is unique for suciently
a.s.
large n and θ̂n −−→ θ.

Proof. Suppose that θ0 is the true parameter value. Then lemma 6.1.3 and the
SLLN imply that
n
1X f (Xi , θj )
− log → I(θj |θ0 ) a.s. Pθ0 , j = 1, . . . , k.
n 1 f (Xi , θ0 )
Hence for n suciently large
n
1X f (Xi , θj ) 1
− log > I(θj |θ0 )
n 1 f (Xi , θ0 ) 2
n
(6.1.1) i.e. ℓ (X, θ0 ) − ℓ (X, θj ) > I(θj |θ0 ) > 0 if θj ̸= θ0 .
2
So for all n suciently large ℓ(X, θj ) has a unique maximum at θj = θ0 . Hence θ̂M L →
θ0 a.s. Pθ0 , i.e. θ̂M L is strongly consistent. □
Remark 6.1.5. Theorem 6.1.4 may not hold if Ω is countably innite (see example on
page 410 of TPE).

Theorem 6.1.6. Suppose conditions A0 − A3 hold and for almost all x, f (x, θ) is dier-
entiable with respect to θ ∈ N with continuous derivative f ′ (x, θ), where N is an open
subset of Ω containing θ0 and Ω ⊆ R. Then with Pθ0 probability 1, for n large
X f ′ (Xi , θ)
ℓ′ (θ, X) = =0
f (Xi , θ)
has a root θ̂n and θ̂n → θ0 a.s. Pθ0 .

Proof. Choose ϵ>0 small so that (θ0 − ϵ, θ0 + ϵ) ⊆ N and dene

Sn = {x : ℓ(θ0 , x ) > ℓ(θ0 − ϵ, x ) and ℓ(θ0 , x ) > ℓ(θ0 + ϵ, x )} .

From eqtn (6.1.1), a.s. Pθ0 , ℓ (θ0 , X) − ℓ (θ0 ± ϵ, X) > 0 for all n large. Hence there exists
θ̂n (X) ∈ (θ0 −ϵ, θ0 +ϵ) such that ℓ′ (θ̂n ) = 0. If there exist more than one θ̂n ∈ (θ0 −ϵ, θ0 +ϵ),
′
choose the one closest to θ0 (the set of roots is closed since f (x, θ) is continuous in θ ).
∗
Call this root θ̂n . (Note that there could be 2 closest roots in which case choose the
124 6. MAXIMUM LIKELIHOOD ESTIMATION

larger.) Let Aϵ = {X = (X1 , X2 , . . .) : ∃ θ̂n∗ ∈ (θ0 − ϵ, θ0 + ϵ) s.t. ℓ′ (θ̂n∗ ) = 0, ∀ sucient

∗
large n and θ̂n is the closest to θ0 }. Then Pθ0 (Aϵ ) = 1. Dene A0 = limk→∞ A1/k . Clearly

Pθ0 (A0 ) = 1 and on A0 , θ̂n∗ → θ0 . □

Remark 6.1.7. Theorem 6.1.6 says there exists a sequence of local maxima which con-
verges a.s. Pθ 0 to θ0 . However since we don't know θ0 , we can't determine the sequence
unless ℓ has a unique local maximum for each n.
Corollary 6.1.8. If ℓ′ (θ) = 0 has a unique root θ̂n for all X and for all suciently
large n, then
θ̂n → θ0 a.s. Pθ0
If in addition Ω is the open interval (θL , θU ) and ℓ′ (θ) is continuous on Ω for all X ,
then θ̂n maximizes the likelihood (globally), i.e. θ̂n is the MLE and hence the MLE is
consistent.

Proof. The rst statement follows straight from theorem 6.1.6.

If θ̂n is not the MLE, then

ℓ(θ) → sup ℓ(α) as θ ↓ θL or θ ↑ θU

α

But θ̂n is a local max by the proof of theorem 6.1.6 and hence ℓ must also have a local
min and ℓ′ (θ) = 0 for some θ ̸= θ̂n , a contradiction. So θ̂n is the MLE. □

As we might expect, under mild conditions the MLE exists, is unique, and consistent for
an exponential family.
Theorem 6.1.9. Consider a full-rank s-parameter exponential family in canonical form
where the density can be written as
( s )
X
p(x, η) = exp ηi Ti (x) − A(η) h(x), η ∈ η(Ω),
i=1
and let the natural parameter space N be an open set. Let x is the observed data vector
from a sample from this model, and t be the oberved value of the complete and sucient
statistic T = (T1 (x), . . . , Ts (x)). Then:

(i) The MLE exists with probability tending to 1 as n → ∞.

(ii) The MLE is consistent.
(iii) If the density function is continuous, then the MLE η̂ exists almost surely, and
satises the equation:
∂A(η)
= t.
∂η η̂

Proof. This is a combination of Theorems 2.3.1, 2.3.2, and 5.2.2 in Bickell & Doksum
(2015). □
 6.2. ASYMPTOTIC NORMALITY OF THE MLE 125

Example 6.1.10. f (x, θ) = θe−θx , 0 < x < ∞, 0 < θ < ∞. Obviously θ̂n = 1/x̄ → θ
a.s. by SLLN (and continuous mapping), since EX = 1/θ . However let us show it by
applying Theorem 6.1.9. For a random sample x the density is:

f (x) = exp {−θnx̄ − (−n log θ)} ,

from which we identify η = −θ, t = nx̄, and A(η) = −n log(−η), whence

∂A(η) n
= − = nx̄, =⇒ η̂ = −1/x̄, θ̂n = 1/x̄.
∂η η
Since Ω = (0, ∞) is open we have immediate consistency of the MLE.

6.2. Asymptotic Normality of the MLE

Theorem 6.2.1. Suppose that X1 , . . . , Xn are iid Pθ0 , θ̂n = θ̂n (X1 , . . . , Xn ) is the MLE
of the true parameter θ0 ∈ Rd , and the following conditions hold:

(i) θ0 ∈ int(Ω) ⊆ Ω, and the model density f (x, θ) is 3 times dierentiable w.r.t. θ
in some open neighborhood of θ0 .
Pθ
(ii)θ̂n −−→0
θ0 .
(iii) Eθ0
∂
∂θj
log f (X, θ0 ) = 0, 1 ≤ j ≤ d.
(iv) The Fisher information matrix per observation, dened as
" T #
∂2
  
∂ ∂
I(θ0 ) = Eθ0 log f (X, θ0 ) log f (X, θ0 ) = −Eθ0 log f (X, θ0 ) ,
∂θ ∂θ ∂θ∂θT

is non-singular. (Note: this is the I1 (θ0 ) dened in Ch. 2.)

(v) There exists δ > 0 such that Eθ0 Wδ (X) < ∞ where
X ∂2 ∂2
Wδ (X) = sup log f (x, θ) − log f (x, θ0 )
qθ−θ0 q<δ
1≤α,β≤d
∂θα ∂θβ ∂θα ∂θβ

∂2
(vi)
∂θα θβ
log f (x, θ) is continuous in θ for all x.

Then
√  
d
− N 0 , I −1 (θ0 )


n θ̂n − θ0 →

Note 6.2.2. If ℓ(θ) is the log-likelihood for X1 , . . . , X n under the above iid model, then

1
I(θ0 ) = − Eθ0 ℓ′′ (θ0 ).
n

Proof. From (vi), Wϵ (X) ↓ 0 as ϵ ↓ 0. Hence from (v), Eθ0 Wϵ (X) ↓ 0 as ϵ ↓ 0.

126 6. MAXIMUM LIKELIHOOD ESTIMATION

Now for some ϕ between θ0 and θ̂n ,

n n
1X ∂ 1X ∂
(6.2.1) log f (xi , θ0 ) = log f (xi , θ̂n )
n 1 ∂θα n 1 ∂θα
n d
1 X X ∂2
+ log f (xi , ϕ)(θβ0 − θ̂nβ )
n i=1 β=1 ∂θα ∂θβ

(The rst term equals 0 since θ̂n = M LE .)

We will show that

1 X ∂2
(6.2.2) log f (xi , ϕ) → −Iαβ (θ0 )
n i ∂θα ∂θβ

Write

n n
1 X ∂2 1 X ∂2
log f (xi , ϕ) = log f (xi , θ0 )
n i=1 ∂θα ∂θβ n i=1 ∂θα ∂θβ
∂2 ∂2
 
1X
+ log f (xi , ϕ) − log f (xi , θ0 )
n i ∂θα ∂θβ ∂θα ∂θβ

The rst term on the right hand side goes to −Iαβ (θ0 ) in probability by the WLLN. For
any c > 0,

Pθ0 (|2nd term| > c) =Pθ0 (∥ ϕ − θ0 ∥> ϵ, |2nd term| > c) + Pθ0 (∥ ϕ − θ0 ∥≤ ϵ, |2nd term| > c)
≤Pθ0 (∥ ϕ − θ0 ∥> ϵ)
!
1X ∂2 ∂2
+ Pθ 0 sup log f (xi , θ) − log f (xi , θ0 ) > c
n i ∥θ−θ0 ∥<ϵ ∂θα ∂θβ ∂θα ∂θβ
=I + II
 
1
Wϵ (xi ) > c ≤ 1c Eθ0 Wϵ (X) → 0
P

where I ≤ Pθ0 ∥ θ̂n − θ0 ∥> ϵ → 0, and II = Pθ0 n i
as ϵ ↓ 0. This proves (6.2.2).

Now write (6.2.1) in matrix form

n  
1X ∂  
log f (Xi , θ0 ) = (−I(θ0 ) + op (1)) θ0 − θ̂n
n i=1 ∂θα α=1,...,d

By CLT,
n  
1 X ∂ d
√ log f (Xi , θ0 ) →
− N (0 , I(θ0 ))
n i=1 ∂θα α=1,...,d
 6.2. ASYMPTOTIC NORMALITY OF THE MLE 127

It follows that
n
√  
−1 1
X ∂
n θ̂n − θ0 = (I(θ0 ) + op (1)) √ log f (xi , θ0 )
n 1 ∂θ
d
− N 0 , I −1 (θ0 )I(θ0 )I −1 (θ0 ) .


→
□
Note 6.2.3. Condition (v) can be replaced by: ∃δ>0 s.t. ∀x in the support of f (x, θ)
∂3
log f (x, θ) ≤ Mα,β,γ (x),
∂θα θβ θγ
for any θ with ∥ θ − θ0 ∥< δ , and where Eθ0 Mα,β,γ (X) < ∞. Condition (vi) can be
replaced by:
d  2
X ∂ log f (x, θ)
Eθ0 < ∞.
j=1
∂θj

(See Van der Vaart, 1998, Theom 5.41.)

Example 6.2.4. One parameter exponential family

f (xi , η) = eηT (xi )−A(η) h(xi )

′ 1
T (xi ) = A′ (η) = Eη T (X1 ). Since A′′ (η) =
P
The likelihood equation ℓ (η) = 0 implies
n
I(η) = V arη T > 0, A (η) is strictly increasing so that ℓ′ (η) = 0 has at most one solu-
′
d3
tion. By Theorem 6.1.6 and its Corollary, η̂ → η a.s. Also
dη 3
log f (x, η) = A′′′ (η) is
independent of x and continuous. Hence by Theorem 6.2.1,

√
 
d −1

1
n (η̂ − η) →
− N 0, I (η) = N 0, .
V arη T

In fact, this is a special case of a more general result, which complements Theorem 6.1.9.

Theorem 6.2.5. Consider a full-rank s-parameter exponential family in canonical form:

( s )
X
p(x, η) = exp ηi Ti (x) − A(η) h(x), η ∈ η(Ω),
i=1

and let the natural parameter space N be an open set, with T = (T1 (x), . . . , Ts (x)) the
complete and sucient statistic. Then, if η̂ is dened to be the MLE, if it exists, and
some xed value otheriwse, we have that
√ d ∂A(η)
− N 0, I −1 (η) , where I(η) = Var(T ) =


n (η̂ − η) → .
∂η∂η ′

Proof. This is Theorem 5.3.5 in Bickell & Doksum (2015). □

128 6. MAXIMUM LIKELIHOOD ESTIMATION

Example 6.2.6. (Censoring). Suppose X1 , · · · , Xn ∼ iid E( 1θ ) (i.e. EXi = θ), and

suppose we observe the censored data Yi = min(Xi , T ) with T xed. Let µ be the
measure on [0, T ] dened by
Z
µ(A) = dx + IA (T )
A

(Lebesgue measure plus unit mass at T ).

Then the density of Y with respect to µ is

1 −y/θ

θ
e 0≤y<T
p(y, θ) = −T /θ
e y=T (= Pθ (Xi ≥ T ))
∂2 ∂ ∂
θ + yθ 0 ≤ y < T


log
∴ − 2 log p(y, θ) = ∂θ
∂
∂θ
∂ T


∂θ ∂θ ∂θ θ
y=T
− θ12 + 2y

θ3
0≤y<T
= 2T
θ3
y=T
2
∂
∴ I(θ) = −E 2 log p(y, θ)
∂θ
Z T 
1 2y 1 −y/θ 2T
= − 2+ 3 e dy + 3 e−T /θ
0 θ θ θ θ
1
= 2 1 − e−T /θ


θ
3
− θ23 + 6y

∂ θ4
0≤y<T
log p(y, θ) = 6T
∂θ 3
θ4
y=T

∂3 θ0 3θ0
Since θ0 ∈ (0, ∞), ∂θ3
log f (y, θ) ≤ Ay + B for
2
<θ< 2
, and Eθ0 (Ay + B) < ∞.
P
Check that θ̂n −
→ θ0 under Pθ 0 . Then by theorem (6.2.1)

√  θ2
  
d
n θ̂n − θ0 →− N 0,
1 − e−T /θ

6.3. Asymptotic Optimality of the MLE

Under the conditions of Theorem 6.2.1, the MLE satises

√  
d
− N 0, I −1 (θ0 )


n θ̂n − θ0 →

We will show now that I −1 (θ0 ) is the minimal attainable covariance matrix for a class
of asymptotically normal estimates. (Note. As in section 6.2, I(θ) denotes the Fisher
information matrix per observation.)
 6.3. ASYMPTOTIC OPTIMALITY OF THE MLE 129

Theorem 6.3.1. Suppose {Tn } is a sequence of asymptotically normal estimators of θ

with V arθ (Tn ) < ∞ for all n, and dene
 
∂ ∂
△n (θ) = Eθ (Tn ) = Eθ (Tnj ) .
∂θ ∂θi i,j=1,...,d

If all the following conditions hold:

√ d
(i) n (Tn − θ) →− N (0, Σ(θ)),
(ii) n Covθ (Tn ) ≥ △Tn I −1 (θ)△n√, (A ≥ B ⇐⇒ A − B psd, as dened earlier),
(iii) ∃δ(θ) such that supn Eθ ∥ n (Tn − θ) ∥2+δ < ∞,
(iv) △n → Id×d ,

then
Σ(θ) ≥ I −1 (θ).
√
Proof. (iii) implies that { n(Tn − θ)} and {n(Tn,α − θα )(Tn,β − θβ )} are uniformly
integrable (Billingsley, p. 338). Hence from (i), if Z ∼ N (0, Σ(θ)),
√
Eθ n(Tn − θ) → EZ = 0
and

n Cov(Tn ) → Cov(Z) = Σ(θ)

(Billingsley, p.338). As n → ∞, the LHS of (ii) → Σ(θ), and the RHS of (ii) → I −1 (θ)
by (iv). Therefore, Σ(θ) ≥ I −1 (θ). □
Note 6.3.2. Assumption (ii) will be satised under the conditions of corollary (2.4.4)(in
the UMVU section).

Definition 6.3.3. If {Tn } satises (i) with Σ(θ) = I −1 (θ), then it is said to be asymp-
totically ecient.
Corollary 6.3.4. Let g = (g1 , · · · , gr )T : Ω → Rr have continuous derivatives ∂gi
∂θα
,
1 ≤ α ≤ d, 1 ≤ i ≤ r. Dene
 
∂g ∂gj
△(θ) = =
∂θ ∂θi i=1,··· ,d, j=1,··· ,r
and assume that the sequence {Tn } of estimators satises
√ d
(i) − N (0, Σ(θ)),
n (Tn − g(θ)) →
(ii) n Covθ (Tn ) ≥ △Tn (θ)I −1 (θ)△n (θ), where △n (θ) = ∂θ
∂
Eθ Tn ,
√
(iii) ∃δ(θ) > 0 such that supn Eθ ∥ n (Tn − g(θ)) ∥ < ∞,2+δ

(iv) △n → △,

then
Σ(θ) ≥ △T I −1 (θ)△
130 6. MAXIMUM LIKELIHOOD ESTIMATION

Proof. The same as the proof of Theorem 6.3.1. ((ii) again holds under the condi-
tions of Corollary 2.4.4.) □

Next we show that g(θ̂n ), where θ̂n = M LE(θ), achieves the lower bound in Corollary
(6.3.4) provided the conditions of Theorem 6.2.1 hold.

√  
d
Corollary 6.3.5. Suppose n θ̂n − θ →− N (0, I −1 (θ)) and ∂gj /∂θi is continuous for
1 ≤ j ≤ r, 1 ≤ i ≤ d, then
 
1 T −1
g(θ̂n ) is AN g(θ), △ I (θ)△ .
n

Proof. Since
 
1
θ̂n − θ = Op √
n
 
T
  1
g(θ̂n ) − g(θ) = △ θ̂n − θ + op √
n
√  
d
− N 0, △T I −1 (θ)△ by Slutzky.


∴ n g(θ̂n ) − g(θ) →

□
Example 6.3.6. Suppose X1 , · · · , Xn are iid lognormal(µ, σ 2 ), (i.e. Yi = log Xi ∼
n
N (µ, σ 2 )) with σ 2 = 1. The MLE of µ is µ̂n = n1 i=1 log Xi . Suppose g(µ) = EX1 =
P
2 2 2
eµ+1/2 (Set λ = 1 in EeλY1 = eλµ+λ σ /2 ⇒ EX1 = eµ+σ /2 ).

Consider the two estimators of θ := g(µ)

θ̂n = g(µ̂n ) = eµ̂n +1/2 (MLE)

and
n
1X
θ̃n = Xi (Moment estimator).
n 1
The Fisher information for µ in Xi = the Fisher information for
 µ in
2 Yi since the trans-
∂
formations Yi = log Xi is one to one. So it equals Eµ ∂µ
log f (Yi , µ) = Eµ (Yi − µ)2 = 1
−µ)2
(notice log f (Y, µ) = − 21 ln 2π − (Y 2 ). Therefore
√  
d
− N 0, △T I −1 (µ)△


n θ̂n − θ →
d
where △= dµ
g(µ) = eµ+1/2 , i.e.

√  
d
− N 0, e2µ+1


n θ̂n − θ →
 6.3. ASYMPTOTIC OPTIMALITY OF THE MLE 131

1
P
On the other hand, for the moment estimator θ̃n = n
Xi , we have

√  
d
− N (0, V ar(Xi )) = N 0, e2µ+1 (e − 1)


n θ̃n − θ →

2
(EXi = Ee2Y = (MGF of N (µ, 1) evaluated at λ = 2) = e2µ+2 , therefore V ar(Xi ) =

2
e2µ+2 − eµ+1/2 = e2µ+1 (e − 1).)

The ARE of θ̃n relative to θ̂n is thus

ARE(θ̃n , θ̂n ) = 1/(e − 1) = .582

The sample mean thus has poor ARE as an estimator of the mean of a log normal
distribution. (The log normal distribution has heavy tails so this is not too surprising.)

Example 6.3.7. (Hodges, TPE, p.440) X1 , · · · , Xn ∼ iid N (θ, 1), I(θ) = 1. Dene

X̄n if |X̄n | > n−1/4


Tn =
0 if |X̄n | < n−1/4

Then, from a homework problem, Tn is AN (θ, v(θ)

n
), where v(θ) = 1 if θ ̸= 0; and 0
if θ = 0. So v(θ) < I(θ) at θ = 0 and the parameter value 0 is called a point of
supereciency. Note the following:

• Theorem 6.3.1 does not apply to this example, since condition (ii) fails.
• Tn is not uniformly better than X̄n for nite n, for example θn = n−1/4 ⇒
Eθn n(Tn − θn )2 → ∞ > 1 = limn→∞ Eθn n(X̄n − θn )2 .
Remark 6.3.8. LeCam (1953) showed that for any estimator satisfying

√  
d
n θ̂n − θ0 →− N (0, v(θ))

the set of θ's where v(θ) < I(θ)−1 has Lebesgue measure zero. (See Theorem 2.6 in TPE.)

Iterative Methods Suppose we have a sequence of estimators θ̃n such that

1
θ̃n = θ0 + Op ( √ )
n
Set

ℓ′ (θ̃n )
Tn = θ̃n −
ℓ′′ (θ̃n )
Then

ℓ′ (θ̃n ) = ℓ′ (θ0 ) + (θ̃n − θ0 )ℓ′′ (θn∗ )

132 6. MAXIMUM LIKELIHOOD ESTIMATION

where θn∗ is between θ0 and θ̃n . Thus

!
√ √ ℓ′ (θ̃n )
n (Tn − θ0 ) = n θ̃n − θ0 −
ℓ′′ (θ̃n )
√ ℓ′ (θ0 ) ℓ′′ (θn∗ )
 
= n θ̃n − θ0 − − (θ̃n − θ0 )
ℓ′′ (θ̃n ) ℓ′′ (θ̃n )
− √1n ℓ′ (θ0 ) √ ℓ′′ (θn∗ )
 
= 1 + n(θ̃n − θ0 ) 1 −
n
ℓ′′ (θ̃n ) ℓ′′ (θ̃n )
Under the conditions of Theorem 6.2.1

1 d
√ ℓ′ (θ0 ) →− N (0, I(θ0 )) ,
n
1 ′′ P
ℓ (θ̃n ) −
→ −I(θ0 ) and
n
1 ′′ ∗ P
ℓ (θn ) −→ −I(θ0 ).
n
Hence the term in square brackets is op (1).

Thus

− √1n ℓ′ (θ0 ) d
− N 0, I −1 (θ0 )


RHS = 1 ′′
+ op (1) →
n
ℓ (θ̃n )
and so we have proved the following theorem.

Theorem 6.3.9. Suppose that (A0 )(A2 ) and all conditions of Theorem 6.2.1 hold, with
√
the possible exception of (ii), and that θ̃n is n-consistent estimator of θ, i.e.
1
θ̃n = θ + Op ( √ ).
n
Then,
ℓ′ (θ̃n )
Tn := θ̃n −
ℓ′′ (θ̃n )
is asymptotically ecient.

Corollary 6.3.10. If I(·) is continuous then the estimator

ℓ′ (θ̃n )
Tn′ = θ̃n +
nI(θ̃n )
is asymptotically ecient.
 6.3. ASYMPTOTIC OPTIMALITY OF THE MLE 133

Proof.
!
√ √ ′ ′
ℓ (θ̃n ) ℓ ( θ̃n )
n (Tn′ − Tn ) = n +
′′
ℓ (θ̃n ) nI(θ̃n )
 √ 
′ n 1
= ℓ (θ̃n ) +
ℓ′′ (θ̃n ) nI(θ̃n )
ℓ′′ (θ̃ )
!
ℓ′ (θ̃n ) I(θ̃n ) + n n
= √ ′′
n I(θ̃n ) ℓ (θ̃n )
n
= op (1).
(The rst factor is Op (1), the numerator of the second factor is op (1) and the denominator
P
→ I(θ0 )2 .) □
Example 6.3.11. Location family. Suppose that X1 , X2 , · · · are iid f (x − θ) where
f is dierentiable and symmetric, f (x) > 0 for all x and f ′ is continuous. Then the
conditions of Theorem 6.1.6 hold

A0 : Pθ ̸= Pθ0 , θ ̸= θ0
A1 : common support
A2 : dP
dλ
θ
(x) = f (x − θ)
A3 : θ ∈ int(Ω) = (−∞, ∞)

Hence
n
′
X f ′ (xi − θ)
(6.3.1) ℓ (θ, X) = =0
1
f (xi − θ)

has a sequence of roots θ̂n for n large such that θ̂n → θ0 a.s. Pθ 0 . Since ℓ(θ, X) → 0 as
θ → ±∞, ℓ(θ, X) must have a max, however there may be several solutions of (6.3.1).

Provided f (x − θ) satises conditions (v) and (vi) of Theorem 6.2.1, i.e.

" #
∂2 ∂2
Eθ0 sup log f (x − θ) − log f (x − θ0 ) <∞
|θ−θ0 |<δ ∂θ2 ∂θ2
∂2
and
∂θ2
log f (x − θ) is continuous in θ for all x, then all the conditions of theorem (6.2.1)
σ2
(apart from (ii)) are satised. Also X̄ is AN (θ, ), and hence
n
1
X̄ = θ + Op ( √ ).
n
The corollary of Theorem 6.3.9 therefore implies the asymptotic eciency of
Pn f ′ (Xi −X̄n )
i=1 f (Xi −X̄n )
Tn = X̄n +
nI(X̄n )
134 6. MAXIMUM LIKELIHOOD ESTIMATION

where
2
f ′ (y)
Z 
I(θ) = f (y)dy.
f (y)
Note 6.3.12. The signicance of this result is that, e.g. in Example 6.3.6 (lognormal),
if the likelihood has multiple roots (so one doesn't know which to take), Theorem 6.3.9
and Corollary 6.3.10 say that one gets just as good an estimator (asymptotically), by
√
starting with a n-consistent one (e.g., the MOME X̄ ), and using Tn obtained from one
iteration of Newton-Raphson.

6.4. Asymptotic Eciency of Bayes Estimators

Example 6.4.1. Suppose X ∼ Bin(n, p), with p ∼ B(a, b), then from Example 4.1.4, the
Bayes estimator of p
Tn = (a + x)/(a + b + n), and hence:
is
√
√ √
   
X n X
n(Tn − p) = n −p + a − (a + b) ≡ S1 + S2 .
n a+b+n n
d p
Note that S1 →
− N(0, p(1 − p)) by CLT, and since X/n →
− p by WLLN, then we have that
S2 → 0 as n → ∞. Thus both the Bayes estimator Tn and the MLE X/n have the same
limiting asymptotic distribution.

Questions:
• Does this limiting result also hold for an arbitrary prior?
• And does it extend to more general models (not just the Binomial)?

The answer to both questions is YES, but requires some regularity conditions. (In the
ensuing, let θ∈Ω denote the d-dimensional parameter vector, and θ0 its true value.)

Regularity Conditions:
(B1): The log-likelihood function ℓ(θ) satises all the statements and assumptions
of Theorem 6.2.1 (asymptotic normality of MLE).
(B2): Given ϵ > 0, ∃δ > 0 such that

P (sup{|Rn (θ)/n| : |θ − θ0 | ≤ δ} ≥ ϵ) −→ 0, as n → ∞,
where Rn is the remainder term in a Taylor-series expansion of ℓ(θ) about θ0
1
ℓ(θ) = ℓ(θ0 ) + (θ − θ0 )ℓ′ (θ0 ) − (θ − θ0 )2 [nI(θ0 ) + Rn (θ)],
2
p
Rn (θ)/n →
which satises − 0 as n → ∞.
(B3): Given δ > 0, ∃ϵ > 0 such that
   
ℓ(θ) − ℓ(θ0 )
P sup : |θ − θ0 | ≥ δ ≤ −ϵ −→ 1, as n → ∞.
n
 6.4. ASYMPTOTIC EFFICIENCY OF BAYES ESTIMATORS 135

(Controls the behavior of ℓ(θ) at a distance from θ0 ; important since Bayes

estimators involve integration over entire range of θ values.)
(B4): The priorR density π(θ) on θ is continuous and positive for all θ ∈ Ω.
(B5): Eπ |Θ| = |θ|π(θ)dθ < ∞.
The following lemma establishes that under these conditions the posterior is AN:

ℓ′ (θ0 )
 
2 1
π(θ|x) ∼ AN µn = θ0 + , σ = .
nI(θ0 ) n nI(θ0 )
√ ℓ′ (θ0 )
Lemma 6.4.2. If π (t|x) is the posterior density of t ≡ n(θ−Tn ), where Tn = θ0 + nI(θ ,
∗
0)
we have the following two results, where ϕ(·) is the pdf of a N(0, 1).

(i) If (B1)(B4) hold:

Z p p p
π ∗ (t|x) − I(θ0 )ϕ(t I(θ0 )) dt →
− 0.

(ii) If (B1)(B5) hold:

Z p p p
(1 + |t|) π ∗ (t|x) − I(θ0 )ϕ(t I(θ0 )) dt →
− 0.
p p p
|t| π ∗ (t|x) −
R
Note 6.4.3. (i) and (ii) imply I(θ0 )ϕ(t I(θ0 )) dt →
− 0.

Proof. TPE Theorem 8.2. □

Theorem 6.4.4 (Asymptotic Eciency of Bayes Estimators). If (B1)(B5) hold, and if
θ̃n is the Bayes estimator under squared error loss with prior pdf π(θ), then:
√ d
− N 0, I −1 (θ0 ) ,


n(θ̃n − θ0 ) →
so that θ̃n is consistent and asymptotically ecient.

Proof. Note the following relation, with Tn as dened in Lemma 6.4.2:

√ √ √
n(θ̃n − θ0 ) = n(θ̃n − Tn ) + n(Tn − θ0 ).
Now, from the proof of Theorem 6.2.1:

ℓ′ (θ ) d √ d 1
p 0 ∼ N 0, I −1 (θ0 ) ,


→
− N (0, 1) , =⇒ n(Tn − θ0 ) →
− p N (0, 1)
nI(θ0 ) I(θ0 )
√ p
whence we only need to show that n(θ̃n − Tn ) →
− 0. Now, under squared error loss:
Z
θ̃n = θπ(θ|x)dθ
√ √ √
Z
= (Tn + t/ n)π ∗ (t|x)dt, (transforming θ 7→ t = nθ − nTn ),
Z
1
= Tn + √ tπ ∗ (t|x)dt,
n
136 6. MAXIMUM LIKELIHOOD ESTIMATION

√
n(θ̃n − Tn ) = tπ ∗ (t|x)dt.
R R
which implies Finally, and noting that yϕ(y)dy = 0
(integral of an odd function), we have:

√
Z Z p p
∗
n|θ̃n − Tn | = tπ (t|x)dt − t I(θ0 )ϕ(t I(θ0 ))dt
Z n p p o
∗
= t π (t|x) − I(θ0 )ϕ(t I(θ0 )) dt
Z n p p o
≤ |t| · π ∗ (t|x) − I(θ0 )ϕ(t I(θ0 )) dt, (|xy| ≤ |x|.|y|),
p
→
− 0, (by Note 6.4.3).

As we would expect, Bayes estimators in the context of exponential family models are
asymptotically ecient.

Example 6.4.5 (One parameter exponential family).

f (xi , θ) = eθT (xi )−A(θ) h(xi ) = canonical form

From Theorems 2.4.7 and 1.3.4, A′ (θ) = ET (X), A′′ (θ) = VarT (X) = I(θ). Now check
that (B1)(B5) hold:

• (B1) holds since this is exponential family.

• (B4)-(B5) are conditions on the prior of choice.
P
• ℓ(θ) = θT (xi ) − nA(θ), we have
For (B2), since
X
ℓ(θ) − ℓ(θ0 ) = (θ − θ0 ) T (xi ) − n[A(θ) − A(θ0 )]
X
= (θ − θ0 ) [T (xi ) − A′ (θ0 )] −n {[A(θ) − A(θ0 )] − [(θ − θ0 )A′ (θ0 )]},
| {z } | {z }
=ℓ′ (θ0 ) = 12 (θ−θ0 )2 A′′ (θ∗ )

where the 2nd underbrace follows from the T-series expansion:

1
A(θ) = A(θ0 ) + (θ − θ0 )A′ (θ0 ) (θ − θ0 )2 A′′ (θ∗ ),
2
∗ ∗
which holds for some θ between θ and θ0 , with θ → θ0 as n → ∞. Thus,

1 A′′ (θ∗ )
ℓ(θ) = ℓ(θ0 ) + (θ − θ0 )ℓ′ (θ0 ) − (θ − θ0 )2 .
2 n
Matching this up with the T-series expansion of ℓ(θ) in the statement of Condi-
tion (B2) implies that

A′′ (θ∗ ) 1
= nI(θ0 ) + Rn (θ) =⇒ Rn (θ) = A′′ (θ∗ ) − I(θ0 ).
n n
Therefore we must show that given ϵ > 0, ∃δ > 0 such that
P (sup {|A′′ (θ∗ ) − I(θ0 )| : |θ∗ − θ0 | ≤ δ} ≥ ϵ) −→ 0,
 6.5. DISCUSSION: MLE VS. SHRINKAGE (EFRON & HASTIE, 2016) 137

which is satised because A′′ (θ) = I(θ) is continuous (Assumption (vi) in The-
∗
orem 6.2.1) and θ → θ0 .

Finally (B3). From the proof of (B2) we can write:

 X  
1 1 ′ A(θ) − A(θ0 ) ′
(†) [ℓ(θ) − ℓ(θ0 )] = (θ − θ0 ) [T (xi ) − A (θ0 )] − − A (θ0 ) .
n n θ − θ0
Now, since A′′ (θ) = I(θ) > 0, we have that A(θ) is strictly convex, whence assuming
w.l.o.g. that θ > θ0 , implies (from the def. of convexity) that

A(θ) − A(θ0 )
> A′ (θ0 ).
θ − θ0
Since
1X ℓ′ (θ0 ) 1 ℓ′ (θ0 ) p
[T (xi ) − A′ (θ0 )] = = √ √ →
− 0,
n n n n
|{z} | {z }
p d
→
−0 → − N (0,I(θ0 ))
′
√
where the distributional convergence of ℓ (θ0 )/ n follows from the Iterative Methods
discussion of section 6.3, it therefore follows from (†) that

1
[ℓ(θ) − ℓ(θ0 )] < 0, w.p. 1 as n → ∞,
n
whence for given δ > 0, let θ − θ0 ≥ δ , so that
   X  
ℓ(θ) − ℓ(θ0 ) 1 ′ A(θ) − A(θ0 ) ′
sup ≤ δ [T (xi ) − A (θ0 )] − inf − A (θ0 )
n n θ − θ0
≤ −ϵ, w.p. 1 as n → ∞.

6.5. Discussion: MLE vs. Shrinkage (Efron & Hastie, 2016)

• Although MLE and its accompanying asymptotic optimality theory is one of the
crowning achievements of classical statistical inference, it has proved to be an
inadequate and dangerous tool in many 21st century applications (bigdata). To
quote Efron & Hastie (2016):
Unbiasedness can be an unaordable luxury when there are 100's or
1000's of parameters to estimate at the same time.
• As we saw in Ch. 4, deliberate introduction of bias via shrinkage in order to
improve overall performance (at a possible danger to some individual estimates)
is usually preferable in such (bigdata) cases.
• However, whereas MLE comes equipped with an elegant theory for optimal un-
at present there is no equivalent optimality theory for
biased estimation,
shrinkage estimation.
 CHAPTER 7

Optimal Testing Theory

Whereas we can view point estimation as a primary level type of inference, tests (or
equivalently, condence regions), are a second level type of inference; one usually rst
desires the former before embarking on a quest for the latter.

In this chapter we will see that the UMVU notion of optimal estimation translates into
UMP and UMPU tests. The former are rather restrictive in that they typically do not
exist for two-sided situations; the notion of unbiasedness helps to remedy this situation,
so that one can derive UMPU two-sided tests for a large class of nice problems, including
the s-parameter exponential family.

After battling in this ground of provably-optimal procedures, we end with feasible and
practical guidance. In the failure of identifying an optimal procedure (almost always
the case), one settles for the near-optimal likelihood ratio, Wald, or Score test. This
general approach parallels our point estimation story, where in the failure of identifying
an optimal UMVU, MRE, or minimax estimator, we settled for the near-optimal MLE
(asymptotically UMVU), or the Bayes estimator (admissible).

The UMP procedures (Ÿ7.17.5) apply only to the one-dimensional parameter θ. In

Ÿ7.6 we see how UMPU optimality accommodates the case when there is additionally a
vector of nuisance parameters. Finally, Ÿ7.7 deals with the most general case when θ is
partitioned into two vectors, only one of which is the parameter of interest.

7.1. Uniformly Most Powerful (UMP) Tests

Our basic decision problem is to either accept or reject a given hypothesis about θ based
on an observation of a r.v. X when the underlying p.m.

P = {Pθ , θ ∈ Ω}.
Suppose that Ω = ΩK ∪ ΩH , where ΩK ∩ ΩH = ∅.
(
H : θ ∈ ΩH (null)
Hypotheses
K : θ ∈ ΩK (alternative)

Non-random test
Divide sample space S as: S = S0 ∪ S1 , where S0 ∩ S1 = ∅.
Accept H if X ∈ S0 .
138
 7.1. UNIFORMLY MOST POWERFUL (UMP) TESTS 139

Reject H if X ∈ S1 .
S1 is called the critical region (or the rejection region).
The power of the test is dened (for all θ) as:
β(θ) = Pθ (X ∈ S1 ) = Pθ (reject H).
The test is said to have signicance level α if
β(θ) ≤ α, ∀θ ∈ ΩH .
In contrast to level, the test is said to have size α if this is the maximum power over the
null space:
sup β(θ) = α.
θ∈ΩH

(In continuous settings size and level are synonymous  it's only in discrete situations
that we make a distinction.)

Ideally we would like

Pθ (X ∈ S1 ) = 0, ∀θ ∈ ΩH (probability of Type I error),

Pθ (X ∈ S0 ) = 0, ∀θ ∈ ΩK (probability of Type II error).

However, in general such an ideal test is impossible to construct, and so we search instead
for a Uniformly Most Powerful (UMP) test.

Randomized test
IfX = x is observed, we toss a coin with P (Head) = ϕ(x) ∈ [0, 1]. If the coin lands Head
we reject H , otherwise we accept H . Note therefore that Head|X ∼ Bern(ϕ(X)), where:

ϕ(x) = Pθ (Head | X = x),

is called the critical function. If ϕ(x) ∈ {0, 1}, then we are back in the non-random
case with:

S1 = {x : ϕ(x) = 1}, and S0 = {x : ϕ(x) = 0}.

The probability of rejection (of H) by the randomized test is thus:

Pθ (Head) = Eθ (Head) = Eθ [Eθ (Head | X)] = Eθ ϕ(X) = β(θ).

Problem: Choose ϕ(·) to maximize the power

βϕ (θ) = Eθ ϕ(X), ∀θ ∈ ΩK ,
subject to the level α test constraint:

βϕ (θ) ≤ α, ∀θ ∈ ΩH .
Definition 7.1.1 (UMP test). A test ϕ is UMP of level α if the following two conditions
are satised.

(i) βϕ (θ) ≤ α, ∀θ ∈ ΩH . (The test has level α.)

140 7. OPTIMAL TESTING THEORY

(ii) βϕ (θ) ≥ βϕ′ (θ), ∀θ ∈ ΩK , and for every critical function ϕ′ such that βϕ′ (θ) ≤
α, ∀θ ∈ ΩH . (The power of the test is at least as large as that of any other level
α test.)

7.2. The Neyman-Pearson Lemma

A class of distributions is called simple if it contains a single distribution; otherwise it

is said to be composite. The solutionϕ to the problem stated above of maximizing the
power subject to being of level α, is given by the Neyman-Pearson (NP) Lemma if K is
simple.

Theorem 7.2.1 (Neyman-Pearson Lemma). Suppose Ω0 and Ω1 are simple, consisting of

the probabilty measures P0 and P1 , respectively, with corresponding densities p0 = dP0 /dµ
and p1 = dP1 /dµ, with respect to dominating measure µ (e.g., take µ = P0 + P1 ). Then,
dening Aµ := µ{x : p1 (x) = kp0 (x)}, we have the following results.

Existence & Suciency: For 0 ≤ α ≤ 1, there exists a test ϕ and a constant k

such that:
(i) E0 ϕ = α, (i.e., the test has size α).
(ii) The test is a likelihood ratio test given by

1, p1 (x) > kp0 (x),

ϕ(x) = 0, p1 (x) < kp0 (x),
γ, p (x) = kp (x) and A ̸= 0,

1 0 µ

where 0 ≤ γ ≤ 1 is an arbitrary constant.

(iii) E1 ϕ ≥ E1 ϕ , for every test ϕ satisfying E0 ϕ ≤ α.
′ ′ ′

Necessity: If ϕ∗ is a UMP level α test, then ϕ∗ satises (ii) for some k, a.e. µ.
It also satises (i) unless there is a test of size less than α with a power of 1.

Proof. Existence & Suciency. If α = {0, 1},

k = {∞, 0}, respectively. If
choose
P0 and P1 µ-measure zero),
are mutually singular (the intersection of their supports has
then taking k = 0 gives ϕ(x) = 1 if p1 (x) > 0, and we can set ϕ(x) = α where p0 (x) > 0.
Thus the result of the Lemma follows since: (i) E0 ϕ = E0 α = α; (ii) ϕ(x) = 1 if p1 (x) > 0
′ ′
and ϕ(x) = 0 if p1 (x) < 0; (iii) E1 ϕ = 1 ≥ E1 ϕ for any other ϕ .

It remains to consider the case 0 < α < 1 and µ(x : p0 (x)p1 (x) > 0) > 0. Let
    
′ p1 (X) ′ p1 (X) ′
k = inf k : P0 ≥ k ≥ α ≥ P0 >k
p0 (X) p0 (X)
    
′ p1 (X) ′ p1 (X) ′
= inf k : P0 < k ≤ 1 − α ≤ P0 ≤k ,
p0 (X) p0 (X)
 7.2. THE NEYMAN-PEARSON LEMMA 141

and note that 0 ≤ k < ∞. Analogously to Aµ , let Aj := Pj ({x : p1 (x) = kp0 (x)}) for
j = 0, 1. Dene


1, p1 (x) > kp0 (x),

0, p1 (x) < kp0 (x),
ϕ(x) = α−P0 (p1 >kp0 )

 A0
, p1 (x) = kp0 (x) and A0 ̸= 0,

0, p1 (x) = kp0 (x) and A0 = 0.


Observe now that (i) follows because, from the above def. of k,
    
P0 p1 (X) > k = P0 p1 (X) ≥ k = α, A0 = 0
E0 ϕ(X) =  p0 (X)  h p0 (X)
i
P0 p1 (X) > k + α−P0 (p1 >kp0 ) A0 = α, A0 =
̸ 0.
p0 (X) A0

Since the test is clearly of the form given by (ii), it remains to show (iii). To this end,
′ ′ ′
suppose ϕ is such that E0 ϕ ≤ α. Since 0 ≤ ϕ ≤ 1, we have

ϕ − ϕ′ > 0 =⇒ ϕ>0 =⇒ p1 − kp0 ≥ 0, [by 1st and 3rd branches of ϕ],

ϕ − ϕ′ < 0 =⇒ ϕ ̸= 1 =⇒ p1 − kp0 ≤ 0, [by 2nd and 3rd branches of ϕ],
′
Thus (ϕ − ϕ )(p1 − kp0 ) ≥ 0, whence
Z
(7.2.1) 0 ≤ (ϕ − ϕ′ )(p1 − kp0 )dµ = E1 ϕ − E1 ϕ′ − k(E0 ϕ − E0 ϕ′ ) ≤ E1 ϕ − E1 ϕ′ ,

since E0 ϕ = α and E0 ϕ′ ≤ α implies k(E0 ϕ − E0 ϕ′ ) ≥ 0. Therefore E1 ϕ ≥ E1 ϕ′ .

Necessity. If ϕ∗ is a UMP level α test, then from (7.2.1)
Z
0≤ (ϕ − ϕ∗ )(p1 − kp0 )dµ,
∗
with equality holding only if (ϕ − ϕ )(p1 − kp0 ) = 0 µ-a.e., since the integrand is non-
∗ ∗
negative. This implies ϕ must satisfy (ii) µ-a.e. (Note that if ϕ has size smaller than
∗
α, then ϕ can be increased until either the size equals α or the power equals 1.) □
Remark 7.2.2. UMP tests are determined uniquely up to sets of µ-measure 0 by (i) and
(ii), provided Aµ = 0. If Aµ = 0, then the UMP test is non-random. If Aµ > 0, then the
UMP test can be randomized by choosing ϕ to be constant (= γ ) on the boundary set
Aµ . However, any ϕ will do provided the test has size α.
Corollary 7.2.3. Let β be the power of a UMP level α test for testing P0 vs. P1 , with
0 < α < 1. Then, α < β unless P0 = P1 .

Proof. Take ϕ∗ (x) ≡ α.

E1 ϕ∗ = α ≤ β , by def. of UMP. If α = β < 1, then ϕ∗
Then
is UMP and must satisfy (ii). Therefore p0 (x) = kp1 (x) for every x, i.e., P0 = P1 (must
have k = 1, otherwise p0 will not integrate to 1). □

Geometric interpretation
For testing P0 vs. P1 via the NP Lemma, dene N = (α, β) such that ∃ a test ϕ with
142 7. OPTIMAL TESTING THEORY

α = E0 ϕ(X) and β = E1 ϕ(X). Then, obviously N ⊂ [0, 1] × [0, 1], and it can be shown
that:

(i) N is convex,
(ii) both (0, 0) and (1, 1) are inN,
(iii) N is symmetric about (1/2, 1/2), so that (α, β) ∈ N =⇒ (1 − α, 1 − β) ∈ N ,
(iv) N is a closed set.

Plotting β , we see that N describes a convex set extending from (0, 0) to (1, 1),
α vs.
centered at (1/2, 1/2). For a given level α0 , the level α0 tests are represented by the
portion of N to the left of the vertical line α = α0 (shaded region). The UMP test
(tests) is (are) the single point (line) with largest β value in the shaded region at α = α0 .

Geometric proof of Corollary

Clearly β ≥ α since x is an α level test with power α. If ∃α0 for
ϕ(x) = α for every
which the level α0 UMP test has power α0 , then by convexity and symmetry, N is the
R R
line segment joining (0, 0) and (1, 1). Therefore ϕdP0 = ϕdP1 for every test function
ϕ, which implies P0 = P1 .
The NP Lemma can usually be invoked to nd a general one-sided UMP test for composite
hypotheses, as the next (classical) example shows.

Example 7.2.4 (UMP one-sided test for normal mean). Consider the single obs X ∼
2 2
N(µ, σ ), σ is known. To nd the UMP test of H : µ = 0 vs.
where K : µ = µ2 > 0,
note that Aµ = 0, so that the UMP is non-random, and is given by the NP Lemma as
 n o
( p2 (x) µ2 x µ22 (
1, p0 (x) > k  1, exp σ2
− 2σ 2
>k 1, x > k ′
ϕ(x) = = n o =
0, pp20 (x)
2
(x)
<k 0, exp µ22x − µ22 < k 0, x < k ′
σ 2σ

where the last equality follows by the equivalence of events

 
p2 (x)
>k ⇐⇒ {x > k ′ }.
p0 (x)
The cuto k′ is found by requiring the test to have size α:
k′ − 0
 
α = P0 (X > k ′ ) = P Z> , =⇒ k ′ = σz1−α ,
σ
where Z ∼ N(0, 1) and z1−α is its 1−α quantile, i.e., Φ(z1−α ) = 1 − α. Noting that the
test did not require specic knowledge of µ2 , µ2 > 0, we can in fact conclude
only that
that the level α UMP test for H :µ=0 vs. K2 : µ > 0, rejects when x > σz1−α . Its
power function is

β2 (µ) = P2 (X > σz1−α ) = 1 − Φ(z1−α − µ/σ).

Similarly, the test that rejects for x < σzα is UMP level α for H:µ=0 vs. K1 : µ < 0.
Its power function is

β1 (µ) = P1 (X < σzα ) = Φ(zα − µ/σ).

 7.4. MONOTONE LIKELIHOOD RATIO 143

Remark 7.2.5 (Nonexistence of two-sided UMP). UMP tests typically do not exist for
two-sided alternatives. E.g., consider testing H : µ = 0 vs. K : µ ̸= 0, a pair of hypotheses
with a simple null, in the previous example. Sketching the power function β1 (µ) over all
µ ∈ R, note that
lim β1 (µ) = 1, and lim β1 (µ) = 0,
µ↓−∞ µ↑∞
and is monotone decreasing betwen these two endpoints. Similarly, β2 (µ) is monotone
increasing between the endpoints:

lim β2 (µ) = 0, and lim β2 (µ) = 1.

µ↓−∞ µ↑∞

By the necessity part of the NP Lemma, a UMP test for K would therefore have to
coincide with β1 (µ) for µ < 0 and β2 (µ) for µ > 0, but neither of these two is UMP over
R. (The power function of the obvious test that rejects when either x < σzα/2 or
all of
x > σz1−α/2 is below each of these over their respective optimal regions.) Thus, no UMP
test exists here.

7.3. P-Values

See Ÿ3.3 of TSH.

7.4. Monotone Likelihood Ratio

We saw in Example 7.2.4 that we can sometimes extend the NP simple hypotheses results
to a composite one, which hold for all θ ∈ K. This will now be seen to be an instance of a
general result that holds whenever the family of measures {Pθ } has a monotone likelihood
ratio (MLR).

Definition 7.4.1 (MLR). The family P = {pθ := dPθ /dµ : θ ∈ Ω ⊂ R} has MLR in
T (·) (usually a sucient statistic) if ∀θ1 < θ2 there exists a non-decreasing function hθ1 ,θ2
of T (·) such that

pθ2 (x)
= hθ1 ,θ2 (T (x)), on the set A+ (θ1 , θ2 ) = {x : pθ2 (x)pθ1 (x) > 0}.
pθ1 (x)
(Families with non-increasing MLR may be treated by symmetry by reparametrizing,
ϕ := −θ, which has the eect of reversing the inequalities in the next theorem; see
Remark 7.4.5.)

Example 7.4.2 (Uniform). X = (X1 , . . . , Xn ), where X1 , . . . , Xn ∼ iid U(0, θ). Then,

1 1
pθ (x) = n
I(0,θ) (x(1) )I(0,θ) (x(n) ) = n I(0,x(n) ) (x(1) )I(0,θ) (x(n) )
θ θ
whence we see that T (x) = x(n) is sucient. Hence for θ1 < θ2 ,
( n
θ1
pθ2 (x) θ2
, 0 < x(n) < θ1 ,
=
pθ1 (x) +∞, θ1 < x(n) < θ2 .
144 7. OPTIMAL TESTING THEORY

Since this ratio is constant over A+ (θ1 , θ2 ) = {x : 0 < x(n) < θ1 }, {pθ } has MLR in x(n) .
Example 7.4.3 (One-parameter exponential family).

pθ (x) = exp{θT (x) − A(θ)}h(x).

For θ1 < θ2 , we have that

pθ2 (x)
= exp{(θ2 − θ1 )T (x) − [A(θ1 ) − A(θ2 )]},
pθ1 (x)
which is increasing in T (x), and thus the family has MLR in T (x).

The most important result under MLR is the following theorem, which states that there
is a one-sided UMP composite hypotheses test.

Theorem 7.4.4 (One-sided UMP test under MLR). Suppose {pθ } has MLR in T . Then
we have the following results.

(i) For testing H : θ ≤ θ0 vs. K : θ > θ0 , there exists a UMP level α test, given by

1, T (x) > c,

ϕ(x) = γ, T (x) = c

0, T (x) < c,

where −∞ ≤ c ≤ ∞ and 0 ≤ γ ≤ 1 are determined by the level α constraint:

Eθ0 ϕ(X) = α.
(ii) The power function β(θ) = Eθ ϕ(X) is strictly increasing on the set
{θ : 0 < β(θ) < 1}.
(iii) For any θ < θ0 , the test ϕ minimizes the Type I error, i.e., it minimizes β(θ)
among all tests ϕ′ satisfying
Eθ0 ϕ′ (X) = α. (i.e., Eθ ϕ ≤ Eθ ϕ′ , ∀θ < θ0 .)

Proof. We will consider only the case 0 < α < 1.

(i) Letting c = inf{c′ : Pθ0 (T > c′ ) < α}, it is clear that

Pθ0 (T > c) ≤ α ≤ Pθ0 (T ≥ c) .

Let
( α−P
θ0 (T >c)
Pθ0 (T =c)
, if Pθ0 (T = c) > 0,
γ=
0, otherwise.

Using similar arguments to the proof of the NP Lemma, we have, for this (c, γ)
pair, that Eθ0 ϕ = α. Now consider H : θ = θ0 vs. K : θ = θ1 , where θ1 > θ0 . We
 7.4. MONOTONE LIKELIHOOD RATIO 145

know from the NP Lemma that a UMP size α test is of the form

1, pθ1 (x) > kpθ0 (x),

∗
ϕ (x) = γ, pθ1 (x) = kpθ0 (x)

0, p (x) < kp (x),
θ1 θ0

where k was dened in the proof. But ϕ is UMP for H : θ = θ0 vs. K : θ = θ1 ,

since Eθ0 ϕ(x) = α and,
pθ1 (x)
> hθ1 ,θ2 (T (x)) ≡ hθ1 ,θ2 (c) ⇐⇒ T (x) > c,
pθ0 (x)
pθ1 (x)
< hθ1 ,θ2 (T (x)) ≡ hθ1 ,θ2 (c) ⇐⇒ T (x) < c,
pθ0 (x)
which follows from the fact that hθ1 ,θ2 (·) is monotone increasing in T (x) ≡ c.
Thus,
pθ1 (x)


 1, pθ0 (x)
> hθ1 ,θ2 (c),

pθ1 (x)
ϕ(x) = γ, pθ0 (x)
= hθ1 ,θ2 (c),

 pθ1 (x)
0, < hθ1 ,θ2 (c),
pθ0 (x)

whence ϕ = ϕ∗ and hθ1 ,θ2 (c) = k . But, since c depends only on θ0 , ϕ is indepen-
dent of θ1 and is therefore UMP for H : θ = θ0 vs. K : θ > θ0 . We next extend
this to K : θ ≤ θ0 . By the NP Lemma, note that ϕ is UMP for H : θ = θ1
′
vs. K : θ = θ2 , for any θ2 > θ1 , al level α = Eθ1 ϕ = β(θ1 ). By Corollary 7.2.3,
we then have that

(7.4.1) β(θ2 ) > α′ = β(θ1 ), provided β(θ1 ) < 1,

whence β(θ) ≤ β(θ0 ) = α, for every θ ≤ θ0 . Finally, since ϕ maximizes β(θ) for
each θ > θ0 subject to Eθ0 ϕ ≤ α, it also does so subject to the more stringent
condition: Eθ ϕ ≤ α for every θ ≤ θ0 . Hence ϕ is UMP level α for H : θ ≤ θ0
vs. K : θ > θ0 .
(ii) This was shown in (7.4.1) above.
′ ′
(iii) For θ < θ0 , 1 − ϕ is a UMP test of H : θ = θ0 vs. K : θ = θ . Consequently, if
′ ′
Eθ0 ϕ = α, then Eθ0 (1 − ϕ ) = 1 − α = Eθ0 (1 − ϕ), which implies, since 1 − ϕ is
′ ′
UMP, that Eθ ′ (1 − ϕ) ≥ Eθ ′ (1 − ϕ ), whence Eθ ′ ϕ ≤ Eθ ′ ϕ .

□
Remark 7.4.5. We can make the following remarks concerning this theorem.

• To test H : θ ≥ θ0 vs. K : θ < θ0 , just reverse the inequalitites in the denition

of ϕ(x) in (i).
• For testing H : θ ≤ θ0 vs. K : θ > θ0 as in the theorem, analogous results hold
if the likelihood ratio pθ2 /pθ1 in the denition of MLR is instead non-increasing;
again we just reverse the inequalitites in the denition of ϕ(x). Thus to have a
one-sided UMP test, we just need the likelihood ratio to be monotone.
146 7. OPTIMAL TESTING THEORY

Under a one-parameter exponential family, we have the following consequence of this

theorem.

Corollary 7.4.6. Let θ ∈ R and suppose X has density (w.r.t. a dominating measure)
that is a one-parameter exponential family of the form:
pθ (x) = exp{Q(θ)T (x) − B(θ)}h(x),
where Q(θ) is strictly monotone. Then, the UMP test of H : θ ≤ θ0 vs. K : θ > θ0 is
given by one of the following two cases.

Case Q(θ) ↑ (increasing):


1, T (x) > c,

ϕ(x) = γ, T (x) = c

0, T (x) < c.

Case Q(θ) ↓ (decreasing):


1, T (x) < c,

ϕ(x) = γ, T (x) = c

0, T (x) > c.

In each case, c and γ are determined by the level α constraint:

Eθ0 ϕ(X) = α.
For a UMP test of H : θ ≥ θ0 vs. K : θ < θ0 , just reverse all the above inequalitites.
Example 7.4.7. X1 , . . . , Xn ∼ iid Gamma(θ, λ), where θ > 0 and λ > 0 are respectively,
the shape and rate (inverse of scale) parameters, with density

λθ θ−1 −λx θ
f (x; θ, λ) = x e I(x > 0), E(X) = .
Γ(θ) λ
If λ is known, the objective is to nd the UMP test of H :θ ≥1 vs. K : θ < 1. The
density of the sample (likelihood) is seen to be a one-parameter exponential family:
X
L(θ) = exp {θt(x) − n[log Γ(θ) − θ log λ]} h(x), t(x) = log xi .
Since Q(θ) = θ is increasing, the corollary identies the UMP level α test as:
(
1, t(x) < c,
ϕ(x) =
0, t(x) > c,
where c solves
 X 
α = Eθ0 ϕ(X) = Pθ=1 (T < c) = Pθ=1 − λ log(λXi ) > c′ .
Now, when θ = 1, f (x; θ = 1) = λe−λx I(x > 0) ∼ Exp(λ), and it can be shown that

Y = −λ log(λX) ∼ Gumbel(µ = 0 , |σ {z
= λ}),
| {z }
location scale
 7.4. MONOTONE LIKELIHOOD RATIO 147

c′
P
so that the appropriate quantile can be found from the cdf of Yi (which seems to be
non-standard, but at least the mgf can be computed and quantiles obtained by inverting
it via a saddlepoint approximation), or c can be found by Monte Carlo simulation directly
from T.

A Decision-Theoretic Formulation
We can place the hypothesis testing problem on a decision-theoretic formulation, akin to
the point estimation problem. For ϕ which tests

H : θ ≤ θ0 vs. K : θ > θ0 ,

there are two possible decisions: d0 = {accept H}, or d1 = {reject H}. We can therefore
dene corresponding loss functions:

L0 (θ) := L0 (θ, d0 ) = loss incurred when θ is the truth and we accept H,

L1 (θ) := L1 (θ, d0 ) = loss incurred when θ is the truth and we reject H,

so that the (total) loss is

L(θ, ϕ) = L0 (θ)(1 − ϕ) + L1 (θ)ϕ.

We can now dene the risk in the usual way as expected loss:

R(θ, ϕ) = EL(θ, ϕ) = L0 (θ)(1 − Eθ ϕ) + L1 (θ)Eθ ϕ.

Definition 7.4.8 (Inadmissible test). A test ϕ is inadmissible if ∃ϕ′ such that

R(θ, ϕ′ ) ≤ R(θ, ϕ), ∀θ

R(θ, ϕ′ ) < R(θ, ϕ), for some θ.

That is, ϕ is inadmissible if ∃ϕ′ which dominates ϕ. A test ϕ is admissible if its not
inadmissible.

Definition 7.4.9 (Complete classes). A class C of tests is complete if ∀ϕ ̸∈ C , ∃ϕ′ ∈ C

′
such that ϕ is dominated by ϕ.

A complete class is minimal if it does not contain a proper complete subclass. (If a
minimal complete class exists, it consists of precisely the admissible tests.)

A class C is essentially complete if ∀ϕ ̸∈ C , ∃ϕ′ ∈ C which is at least as good as ϕ, i.e.,

R(θ, ϕ ) ≤ R(θ, ϕ), ∀θ. Such a class is minimal if it does not contain a proper essentially
′

complete subclass.

The point is that if there is a (minimal) essentially complete class, then one need not
bother with considering tests outside of this class.
148 7. OPTIMAL TESTING THEORY

Theorem 7.4.10. Under the setting and assumptions of Theorem 7.4.4, let C be the class
consisting of all tests of the form given by (i) of that theorem. If
L1 (θ) − L0 (θ) > 0, for θ < θ0 ,
L1 (θ) − L0 (θ) < 0, for θ > θ0 ,

then we have the following results.

(i) C is essentially complete.

(ii) If additionally the set {x : pθ (x) > 0} is independent of θ, C is minimal essen-
tially complete.

Proof. For any given test ϕ′ , let α = Eθ0 ϕ′ . Choose ϕ as in Theorem 7.4.4, i.e.,
Eθ0 ϕ = α with

1, T (x) > c

ϕ(x) = γ, T (x) = c .

0 T (x) < c

Then, Eθ ϕ ≤ Eθ ϕ′ , ∀θ < θ0 (has smaller Type I error), and Eθ ϕ ≥ Eθ ϕ′ , ∀θ > θ0 (is

UMP). Hence

EL(θ) := R(θ, ϕ) = L1 (θ)P (reject H) + L0 (θ)P (accept H)

= L1 (θ)Eθ ϕ + L0 (θ)(1 − Eθ ϕ)
= L0 (θ) + Eθ ϕ (L1 (θ) − L0 (θ))
≤ L0 (θ) + Eθ ϕ′ (L1 (θ) − L0 (θ))
= R(θ, ϕ′ ),
where the ≤ part follows by the assumptions on L1 and L0 in the statement of the
theorem. □

7.5. Condence Bounds

UMP one-sided tests can be used to derive upper and lower condence bounds (CBs).
As we will see, inverting a UMP test leads to UMA condence CBs (dened next). Since
lower and upper bounds are analogous, it suces to focus our attention, say, on lower
bounds, θ.
Definition 7.5.1 (UMA lower CB). We dene the following based on sample data X.

(i) θ(X) is a (1 − α) lower condence bound for θ if:

Pθ (θ(X) ≤ θ) ≥ 1 − α, ∀θ.
(The idea is that θ falls below θ with a specied high probability of at least
1 − α.)
 7.5. CONFIDENCE BOUNDS 149

(ii) The condence coecient or condence level for θ(X) is dened to be

inf Pθ (θ(X) ≤ θ).

θ

(This usually turns out to be 1 − α.)

(iii) θ is a uniformly most accurate (UMA) lower CB for θ with condence level
(1 − α), if, in addition to (i), we have
Pθ (θ(X) ≤ θ′ ) ≤ Pθ (θ∗ (X) ≤ θ′ ), ∀θ′ < θ,
and for every other lower CB θ∗ (X) satisfying (i). (The idea is that we want to
underestimate θ by as little as possible.)

Our aim is to nd a lower CB for θ which falls below θ with high probability (≥ 1 − α),
but not too far below. Excessive underestimation can be assessed via a loss function.
Suppose the following conditions hold

L(θ, θ) =0, if θ > θ,

(7.5.1) L(θ, θ) ≥0, ∀ θ ≤ θ,
′
L(θ, θ) ≥L(θ, θ ), if θ ≤ θ′ ≤ θ.

Problem: Minimize the risk Eθ L(θ, θ), subject to

(7.5.2) Pθ (θ(X) ≤ θ) ≥ 1 − α.

Solution: An UMA lower CB minimizes the risk subject to (7.5.2).(See Problem 3.44 in
TSH.)

Thus the determination of an UMA lower CB also solves the more general problem for-
mulated in terms of any loss function satisfying (7.5.1). Finding UMA CBs is faciltated
by introducing the following concept.

Definition 7.5.2 (Condence Sets). A family of subsets S(x) of Ω, where x ∈ X , is said

to be a family of condence sets at condence level (1 − α), if
Pθ (θ ∈ S(X)) ≥ 1 − α, ∀θ ∈ Ω.

Thus, the random set S(X) covers the true parameter with probability at least (1 − α).
Example 7.5.3. If θ(X) is dened as in (i) of Denition 7.5.1, then the sets S(x) =
[θ(x), ∞) constitute a family of (1 − α)-level condence sets for θ.

The next theorem shows that inverting a UMP test leads to an UMA condence set.

Theorem 7.5.4. For all θ0 ∈ Ω, let A(θ0 ) be the acceptance region of a (non-random)
level α test of H(θ0 ) : θ = θ0 , and let
S(x) = {θ : x ∈ A(θ) and θ ∈ Ω}.
Then, we have the following results.
150 7. OPTIMAL TESTING THEORY

(i) For x ∈ X , S(x) is a family of level (1 − α) condence sets for θ.

(ii) If, for all θ0 , A(θ0 ) is the acceptance region of a level α UMP test of H(θ0 )
vs. the alternative K(θ0 ), then the corresponding condence set S(x), minimizes
Pθ (θ0 ∈ S(X)), ∀θ ∈ K(θ0 ),
among all (1 − α) level families of condence sets for θ.

Proof. (i) By def., θ ∈ S(x) if and only if x ∈ A(θ), and therefore

Pθ (θ ∈ S(X)) = Pθ (X ∈ A(θ)) ≥ 1 − α.
(ii) If S ∗ (x) is any other family of level (1 − α) condence sets, then A∗ (θ) = {x :
θ ∈ S ∗ (x)} denes an α level test of H(θ0 ) vs. K(θ0 ), since
Pθ0 (X ∈ A∗ (θ0 )) = Pθ0 (θ0 ∈ S ∗ (X)) ≥ 1 − α.
However, A(θ0 ) is UMP, and hence

Pθ (X ∈ A (θ0 )) = Pθ (θ0 ∈ S ∗ (X)) ≥ Pθ (X ∈ A(θ0 )) = Pθ (θ0 ∈ S(X)).

∗

□
Corollary 7.5.5. Suppose {pθ (x), θ ∈ Ω} has MLR in T (x), and that the cdf Fθ (t) of
T is marginally continuous in each t and θ (when the other is xed). Then, we have the
following results.

(i) For each level (1 − α), there exists a UMA lower condence bound θ for θ.
(ii) If Fθ (T (x)) = 1 − α has a solution θ = θ̂ for each x, then the UMA (lower)
bound is unique, and θ = θ̂.

Proof. (i) For each θ0 there exists a c(θ0 ) such that

(7.5.3) Pθ0 (T > c(θ0 )) = α.

Moreover, from Theom 7.4.4,
(
1, if T (x) > c(θ0 )
ϕ(x) =
0, if T (x) < c(θ0 )
is a UMP level α test for H(θ0 ) vs. K : θ > θ0 , with

β(θ) = Eθ ϕ > α, ∀θ > θ0 ,

and consequently Pθ (T > c(θ0 )) > α for all θ > θ0 . By def., (7.5.3) holds
also for everyθ > θ0 , whence c(θ) > c(θ0 ), i.e., c(·) is strictly increasing (and
continuous by the continuity of Fθ (t) in θ ). Now, set A(θ) = {x : T (x) ≤ c(θ)},
S(x) = {θ : x ∈ A(θ)}, and dene θ(x) = inf{θ : T (x) ≤ c(θ)}. Then,
θ ≥ θ(x) ⇐⇒ c(θ) ≥ T (x) ⇐⇒ x ∈ A(θ).
θ, Pθ (θ(X) ≤ θ) = Pθ (T (X) ≤
Consequently, it follows from (7.5.3) that for every
c(θ)) = 1−α. Since by Theom 7.5.4 [θ(x), ∞) minimizes Pθ (θ(X) ≤ θ′ ) for every
θ′ < θ, it follows that θ is a UMA lower bound for θ.
 7.5. CONFIDENCE BOUNDS 151

(ii) Suppose 0 < Fθ0 (t) < 1. Then, setting

(
1, if T >t
ϕ=
0, if T ≤t
implies Eθ0 (1 − ϕ) = Pθ0 (T ≤ t) = 1 − α, which holds also for every θ > θ0 (by
the corollary to the Neyman-Pearson Lemma), and this means that Fθ (t) is a
strictly decreasing function of θ at each θ such that 0 < Fθ (t) < 1. Consequently,
Fθ (t) = 1 − α can have at most one solution: θ = θ̂ . Then Fθ̂ (t) = 1 − α, and

(by def.) c(θ̂) = t, so that

t ≤ c(θ) ⇐⇒ c(θ̂) ≤ c(θ) ⇐⇒ θ̂ ≤ θ.

(Follows by part (i) where it was shown c(·) is continuous and strictly increasing.)
Setting t = T (x) gives

θ ≥ θ̂(x) ⇐⇒ T (x) ≤ c(θ) ⇐⇒ θ ≥ θ(x),

whence it follows that θ̂ = θ.
□
Example 7.5.6 (Exponential waiting times). If X1 , . . . , Xn ∼ iid E(λ), we wish to derive
UMA lower and upper CBs for λ. Since we have a one-parameter exponential family,
this is most easily done by invoking rst Corollary 7.4.6 to derive a corresponding UMP
one-sided test, followed by Corollary 7.5.5 which guarantees an UMA lower/upper CB
upon inversion of the (UMP) test. From the pdf of the sample
Y X
pλ (x) = λn exp {−λT (x)} I(xi > 0), T (x) = xi ∼ Γ(n, λ),
i i

we note that Q(λ) = −λ is monotone decreasing in λ, and the cdf of T is continuous in

both t and λ. (For ease of quantile calculation, we also note that 2λT ∼ χ2 (2n).) Thus
the UMP test of H : λ ≥ λ0 accepts for T ≤ c, where

χ21−α (2n)
1 − α = Pλ0 (T ≤ c) = P (χ2 (2n) ≤ 2λ0 c), =⇒ c= .
2λ0
Thus,
X χ21−α (2n) χ21−α (2n)
x ∈ A(λ0 ) ⇐⇒ xi ≤ ⇐⇒ λ0 ≤ P ,
i
2λ0 2 xi
2
so that χ1−α (2n)/(2t) is a (1 − α) UMA upper CB for λ.
Similarly, χ2α (2n)/(2t) is a (1 − α) UMA lower CB for λ, obtained by inverting the UMP
test of H : λ ≤ λ0 , which accepts for T ≥ c.

UMA Condence Intervals (CIs)

Lower and upper CBs can be used to construct the more common CI, dened as follows.

Definition 7.5.7 (Condence Interval). Suppose all of the following hold:

152 7. OPTIMAL TESTING THEORY

• θ is a lower CB with condence level 1 − α1 ,

• θ is a upper CB with condence level 1 − α2 ,
• θ<θ for every sample point x (occurs if, e.g., α1 + α2 < 1).

Then, the interval (θ, θ) is called a condence interval for θ with level (1 − α1 − α2 ), i.e.,

Pθ (θ ≤ θ ≤ θ) = 1 − α1 − α2 , ∀θ ∈ Ω.

Ifθ and θ are UMA, then they minimize the risks under their respective alternatives,
Eθ L1 (θ, θ) and Eθ L2 (θ, θ), at their respective levels. This is so for any L1 that is non-
increasing in θ forθ < θ and 0 for θ ≥ θ, and for any L2 that is nondecreasing in θ for
θ>θ and 0 for θ ≤ θ. Letting
L(θ; θ, θ) = L1 (θ, θ) + L2 (θ, θ),
the CI (θ, θ) thus minimizes the risk under the alternative, Eθ L(θ; θ, θ), subject to having
condence level (1 − α1 − α2 ):
Pθ (θ > θ) ≤ α1 , and Pθ (θ < θ) ≤ α2 .

Examples of loss functions satisfying these propertis are as follows.

Natural measure:

θ − θ,
 if θ ≤ θ ≤ θ,
L(θ; θ, θ) = θ − θ, if θ < θ,

θ − θ, if θ > θ.

Coverage: L(θ; θ, θ) = θ − θ.
Weighted distance from ends: L(θ; θ, θ) = a(θ − θ)2 + b(θ − θ)2 .
Example 7.5.8 (Exponential waiting times (continued)). From the lower and upper
(1 − α) CBs we obtained in Example 7.5.6, it is easy to see that the interval

χ2α (2n) χ21−α (2n)

 
P , P ,
2 xi 2 xi
is a CI with condence level (1 − 2α).

7.6. Uniformly Most Powerful Unbiased (UMPU) Tests

In Remark 7.2.5 the real reason there was no UMP two-sided test for the normal mean, is
that the power function of the one-sided UMP tests dips below the size of the test in the
null space. If this region is to be in the alternative space K for the two-sided test, then
we should introduce a constraint that the power function over K must not dip below the
size of the test. This is the concept of an unbiased test.
 7.6. UNIFORMLY MOST POWERFUL UNBIASED (UMPU) TESTS 153

Definition 7.6.1 (Unbiased test). A level α test ϕ of H : θ ∈ ΩH vs. K : θ ∈ ΩK is

unbiased if

(7.6.1) Eθ ϕ ≥ α, ∀θ ∈ ΩK .
(And since ϕ is level α we also have Eθ ϕ ≤ α, ∀θ ∈ ΩH .)

Clearly any UMP level α test is unbiased, since ϕ′ (x) = α ∀x is a level α test, and so a
UMP test ϕ (which has a power function at least as large), must satisfy (7.6.1).

Now, let ω denote the set of parameter points that are on the boundary of H and K , i.e.,
the set of points θ that are points or limit points of both ΩH and ΩK . If βϕ (θ) = Eθ ϕ is
a continuous function of θ, then for θ ∈ ω , we must have βϕ (θ) = α. The reason for this
′
is that if θ is a limit point of values θn ∈ ΩH and θn ∈ ΩK , then

βϕ (θ) = lim βϕ (θn ) ≤ α, and βϕ (θ) = lim βϕ (θn′ ) ≥ α.

n→∞ n→∞

This embodies the concept of an α-similar test.

Definition 7.6.2 ( α-similar test). A test ϕ is α-similar on the parameter points ω that
are on the boundary of H and K , if

(7.6.2) βϕ (θ) = α, ∀θ ∈ ω.

The importance behind this denition, is that it allows us to establish unbiasedness

through the more tractable concept of α-similarity, as the next result shows.

Lemma 7.6.3 (UMPU test). Suppose ΩH ∩ ΩK ⊂ Rk and βϕ (θ) is continuous in θ for

every test ϕ. If ϕ is UMP α-similar of level α, then it is UMP unbiased (UMPU) of
′

level α.

Proof. Since ϕ ≡ α is α-similar, Eθ ϕ′ ≥ Eθ ϕ = α, for every θ ∈ ΩK , and hence ϕ′ is

unbiased. Now let ϕ be any unbiased level-α test. Since:

Eθ ϕ ≥ α ∀θ ∈ ΩK (unbiased),

and

Eθ ϕ ≤ α ∀θ ∈ ΩH (level α),
we must have

Eθ ϕ = α ∀θ ∈ (∂ΩH ∩ ∂ΩK )
(since for θ on the boundary ∂ΩH , there exists a sequence {θn } ∈ ΩH such that θn → θ;
likewise there exists a sequence {θn′ } ∈ ΩK such that θn′ → θ; and Eθ ϕ is continuous in
θ). Hence ϕ is α-similar, and consequently

Eθ ϕ′ ≥ Eθ ϕ, ∀θ ∈ ΩK ,
that is, ϕ′ is UMPU of level α. □
154 7. OPTIMAL TESTING THEORY

One-parameter Exponential Families and Two-sided Tests

Suppose X = (X1 , . . . , Xn ) has a density belonging to the one-parameter exponential
family

pθ (x) = exp{θT (x) − A(θ)}h(x).

Then, letting θ2 > θ1 , we have the following results concerning the existence of UMP
tests.

(i) A UMP test exists for H : θ ≤ θ0 vs. K : θ > θ0 , and the reverse situation, by
Corollary 7.4.6.
(ii) A UMP test exists for H : θ ≤ θ1 or θ ≥ θ2 vs. K : θ ∈ (θ1 , θ2 ), by TSH Theorem
3.7.1.
(iii) A UMP test does NOT exist for H : θ1 ≤ θ ≤ θ2 vs. K : θ < θ1 or θ > θ2 , by
TSH Problem 3.54. In this case, consider the test

1, T (x) < c1 or T (x) > c2 ,

ϕ(x) = γi , T (x) = ci , i = 1, 2,

0, c < T (x) < c ,
1 2

where c1 , c2 ∈ R and 0≤γ≤1 are determined by the level α constraint:

Eθ1 ϕ(X) = Eθ2 ϕ(X) = α.

From the results on exponential families in Ch. 1 (Theorem 1.3.13), we know
that βϕ (θ) = Eθ ϕ is continuous in θ on int(N ), ω = {θ1 , θ2 }, and Eθ1 ϕ(X) =
Eθ2 ϕ(X) = α, whence it follows that ϕ is α-similar. Now, by TSH Theorem
′
3.7.1, it follows that 1 − ϕ is UMP level (1 − α) for H : θ < θ1 or θ > θ2
′ ′ ′ ′
vs. K : θ1 ≤ θ ≤ θ2 , and also that ∀ϕ such that Eθ1 ϕ = Eθ2 ϕ = 1 − α,

Eθ (1 − ϕ) ≤ Eθ (1 − ϕ′ ), ∀θ < θ1 or θ > θ2 ,
whence

Eθ ϕ′ ≤ Eθ ϕ, ∀ ̸∈ [θ1 , θ2 ].
Hence, the test ϕ(x) dened above is UMPU level α by Lemma 7.6.3.
(iv) A UMP test does NOT exist for H : θ = θ0 vs. K : θ ̸= θ0 , by TSH Problem
3.54. In this case, consider the test

1, T (x) < c1 or T (x) > c2 ,

ϕ(x) = γi , T (x) = ci , i = 1, 2,

0, c < T (x) < c ,
1 2

where c1 , c2 ∈ R and 0≤γ≤1 are determined by the level α constraint

Eθ0 ϕ(X) = α, and Eθ0 ϕ(X)T (X) = αEθ0 T (X).

Then, ϕ(x) is UMPU by the argument on pp. 111-113 of TSH.
 7.6. UNIFORMLY MOST POWERFUL UNBIASED (UMPU) TESTS 155

Example 7.6.4 (UMPU two-sided test for normal mean). Let X1 , . . . , Xn ∼ iid N(θ, σ 2 ),
2
where σ is known. By the result from case (iv) above, the UMPU test of H : θ = θ0
̸ θ0 , is given by
vs. K : θ =
( (
1, X̄n < c1 or X̄n > c2 1, Z < z1 or Z > z2
ϕ(X) = ⇐⇒ ϕ(Z) =
0, otherwise, 0, otherwise,
√
where Z = (X̄n − θ0 )/(σ/ n) ∼ N(0, 1) with density f (z) under H , and z1 , z2 satisfy
Z z2
Eθ0 (ϕ) = P (Z < z1 ) + P (Z > z2 ) = α ⇐⇒ f (z)dz = 1 − α,
z1

and
Z z2
Eθ0 (ϕZ) = αEθ0 (Z) ⇐⇒ Eθ0 [(1 − ϕ)Z] = (1 − α)Eθ0 (Z) ⇐⇒ zf (z)dz = 0.
z1

The rst condition states that the interval [z1 , z2 ] must enclose an area of (1 − α), while
the second stipulates that z1 < 0, z2 > 0, with |z1 | = z2 (the integral of an odd function
can only be zero if the limits are the same distance apart and on opposite sides of zero).
The only values that satisfy these are z1 = −z1−α/2 and z2 = z1−α/2 . Thus the UMPU
level α test rejects for
σ σ
X̄n < θ0 − z1−α/2 √ or X̄n > θ0 + z1−α/2 √ .
n n
Example 7.6.5 (UMPU two-sided test for normal std. deviation). Let X1 , . . . , X n ∼
2
iid N(0, σ ). The density of the sample is

1 1 X
pσ (X) = √ exp{θT (X)}, θ=− , T = Xi2 ,
2πσ 2 2σ 2
so that from case (iv) above, the UMPU test of H : σ = σ0 vs. K : σ ̸= σ0 , which is
equivalent to H : θ = θ0 vs. K : θ ̸= θ0 , accepts for

T
c1 ≤ T ≤ c2 ⇐⇒ t1 ≤ ≤ t2
σ2
where T /σ02 ∼ χ2 (n) with density fn (t) ∝ tn/2−1 e−t/2 under H , and t1 , t2 satisfy
Z t2
(a) Eσ0 (1 − ϕ) = fn (t)dt = 1 − α,
t1

and Z t2
Eσ0 [(1 − ϕ)T ] = (1 − α)Eσ0 (T ) ⇐⇒ tfn (t)dt = (1 − α)n.
t1
This 2nd condition is equivalent to (TSH Problem 4.5)

n/2 n/2
(b) t1 e−t1 /2 = t2 e−t2 /2 .
The system (a) and (b) of two equations in two unknowns can now be solved numerically
2 2
for t1 , t2 . Alternatively, the equal-tails test with t1 = χα/2 (n) and t2 = χ1−α/2 (n) provides
(by the CLT since T is an empirical sum) a good approximation for large n.
156 7. OPTIMAL TESTING THEORY

UMPU Tests for Multi-parameter Exponential Families

Here we generalize the above to the situation when only one parameter (θ ∈ R) is of
k
interest in an exponential family, while the remaining parameters (ξ ∈R ) comprise a
vector of nuisance parameters. We suppose X = (X1 , . . . , Xn ) has a density of the form

dPθ,ξ
(x) = C(θ, ξ) exp{θU (x) + ξ · T (x)},
dµ
where (θ, ξ) ∈ Ω, where the parameter space Ω is convex and contains an open set of
Rk+1 . (The density is in canonical form, but we combine the A(·) and h(·) functions into
C(·) for a compact representation.)
We will assume that Ω contains points for which θ < θ0 , θ1 , θ2 , and points for which
θ > θ0 , θ1 , θ2 . We are interested in tests of the following types:

(1) H1 : θ ≤ θ0 vs. K1 : θ > θ0 .

(2) H2 : θ ≤ θ1 or θ ≥ θ2 vs. K2 : θ1 < θ < θ2 .
(3) H3 : θ1 ≤ θ ≤ θ2 vs. K3 : θ < θ1 or θ > θ2 .
(4) H4 : θ = θ0 vs. K4 : θ ̸= θ0 .

Since the sucient statistics U and T contain all the information in the sample regarding
(θ, ξ), we can restrict attention to tests based on them.

Now, by Theorem 1.3.11, note that (U, T ) have the joint density

U,T
dPθ,ξ
(u, t) = C(θ, ξ)eθu+ξ·t ,
dν
with respect to the measure

ν(B) = µ {x : (U (x), T (x)) ∈ B} , ∀ B ∈ B(Rk+1 ).

The next result shows that the conditional distributions of U given T =t constitute a
one-parameter exponential family.

Lemma 7.6.6 (Distribution of U |T ). For any xed (θ0 , ξ0 ) ∈ Ω, dene

Z
U |T =t 1
dνt′ (u) = dPθ0 ,ξ0 (u), dνt (u) = e −θ0 u ′
dνt (u), = e(θ−θ0 )u dνt′ (u).
Ct (θ)
Then, the distribution of U |T = t constitutes a one-parameter exponential family of the
form
U |t
dPθ (u) = Ct (θ)eθu dνt (u),
which therefore does not depend on the nuisance parameter ξ .

Proof. Note that since we can write

U,T C(θ, ξ) (θ−θ0 )u+(ξ−ξ0 )·t U,T

dPθ,ξ (u, t) = e dPθ0 ,ξ0 (u, t),
C(θ0 , ξ0 )
 7.6. UNIFORMLY MOST POWERFUL UNBIASED (UMPU) TESTS 157

we obtain the density of U |t as the joint divided by the marginal of T , using the measures
dened above:

U |t dPθU,T
0 ,ξ0
(u, t) e(θ−θ0 )u dPθU,T
0 ,ξ0
(u, t) e(θ−θ0 )u dνt′ (u)
dPθ (u) = R =R =R
dPθU,T
0 ,ξ0
(du, t) e(θ−θ0 )u dPθU,T
0 ,ξ0
(du, t) e(θ−θ0 )u dνt′ (du)
= Ct (θ)eθu dνt (u).
□

We now tackle each of the four cases described above.

Case (1): For every α ∈ (0, 1), there exist constants c(t) and γ(t) such that the
test 
1,
 u > c(t),
ϕ1 (u, t) = γ(t), u = c(t),

0, u < c(t),
satises

(⋆) Eθ0 [ϕ1 | T ] = α.

Moreover, ϕ1 is UMP level α conditional on T = t, i.e., for every ϕ satisfying
(⋆), we have

(7.6.3) Eθ [ϕ1 | T = t] ≥ Eθ [ϕ | T = t], ∀θ > θ0 ,

since the distribution of U |T = t is a one-parameter exponential family.

Note 7.6.7. The following points should be noted.

• We use the notationEθ [ϕ|T ] instead of Eθ,ξ [ϕ|T ], since the distribution of
U |T is independent of ξ (Lemma 7.6.6.)
• Tests satisfying (⋆) are said to have Neyman structure (see TSH Ÿ4.3).
• Reverse the inequalities in the denition of ϕ1 if one wishes to test instead
H1 : θ ≥ θ0 vs. K1 : θ < θ0 .
Case (2): Similarly, the test

1,
 c1 (t) ≤ u ≤ c2 (t),
ϕ2 (u, t) = γi (t), u = ci (t), i = 1, 2,

0, u < c1 (t) or u > c2 (t),
satisfying
Eθi [ϕ2 | T ] = α, i = 1, 2,
is UMP level α conditional on T = t.
Case (3): Similarly, the test

1,
 u < c1 (t) or u > c2 (t),
ϕ3 (u, t) = γi (t), u = ci (t), i = 1, 2,

0, c1 (t) ≤ u ≤ c2 (t),
158 7. OPTIMAL TESTING THEORY

satisfying
Eθi [ϕ3 | T ] = α, i = 1, 2,
is UMP level α conditional on T = t.
Case (4): Similarly, the test

1,
 u < c1 (t) or u > c2 (t),
ϕ4 (u, t) = γi (t), u = ci (t), i = 1, 2,

0, c1 (t) ≤ u ≤ c2 (t),
satisfying

Eθ0 [ϕ4 | T ] = α, and Eθ0 [ϕ4 U | T ] = αEθ0 [U | T ],

is UMP level α conditional on T = t.

The conditional UMP property of these tests is of limited usefulness; in practice we

almost never want the value of T to be xed/given. What one wishes for instead is some
sort of unconditional optimality, and this is established by the following theorem.

Theorem 7.6.8 (Unconditional UMPU). The tests ϕ1 , . . . , ϕ4 dened for Cases (1)(4)
above are unconditionally UMPU level α.

Proof. We prove only Case (1); the remainder being similar. Suppose ϕ is an
unbiased level α test. Then, because it's α-similar, Eθ0 ,ξ ϕ(U, T ) = α for every ξ . Now
let g(T ) = Eθ0 (ϕ|T ) − α, and note that
(7.6.4) Eθ0 ,ξ g(T ) = Eθ0 ,ξ ϕ(U, T ) − α = α − α = 0.
For θ = θ0 the dist. of T belongs to the k -parameter exp. family with parameter set
Ω0 = {ξ : (θ0 , ξ) ∈ Ω}. By our assumptions on Ω, Ω0 contains an open subset of Rk , and
therefore T is complete for {Pθ0 ,ξ : ξ ∈ Ω0 }. Now, because of (7.6.4), we deduce (from
the def. of completeness) that g(T ) = 0 a.s., whence it follows immediately that

(7.6.5) Eθ0 (ϕ|T ) = α a.s.

(This shows that an unconditional unbiased level α test is also level α conditionally.)
Finally, from (7.6.3) it follows that for θ > θ0 ,
(7.6.6) Eθ,ξ (ϕ1 ) = Eθ,ξ [Eθ (ϕ1 | T )] ≥ Eθ,ξ [Eθ (ϕ | T )] = Eθ,ξ (ϕ),
whence ϕ1 is UMPU level α (UMPU by (7.6.6), and level α by (7.6.5)). □
Example 7.6.9 (Comparison of Poisson means). For X ∼ Poi(λ) and Y ∼ Poi(µ), with
X and Y independent, we see that the joint density is the 2-parameter exp. family:
 
  e−(λ+µ)
fX,Y (x, y) = exp y log(µ/λ) + (x + y) log(λ) = eθu+ξt C(θ, ξ),
|{z} | {z } | {z } | {z } x!y!
u θ t ξ
2
with (λ, ξ) ∈ Ω = R clearly convex. Now note that tests about θ = log(µ/λ) ⇒ µ/λ =
θ
e , correspond to comparing µ and λ. E.g., suppose we wish to test H : θ ≤ 0 (⇔ µ ≤ λ)
 7.6. UNIFORMLY MOST POWERFUL UNBIASED (UMPU) TESTS 159

vs. K : θ > 0 (⇔ µ > λ). Then, from Case (1), the conditional UMP level α test is
given by

1,
 u > c(t),
ϕ1 (u, t) = γ(t), u = c(t),

0, u < c(t),
with c(t) and γ(t) satisfying Eθ=0 [ϕ1 |T ] = α.
To compute the power function we need
θ θ
the distribution of U |t, which will be seen to be Bin(n = t, p = e /(1 + e )). Noting that
ξ ξ+θ
λ = e and µ = e , the joint of (U, T ) is

U,T exp{−eξ (1 + eθ )}
dPθ,ξ (u, t) = eθu+ξt I{0,...,t} (u)I{0,...,∞} (t),
u!(t − u)!
where the fact that the support is the lattice region on the upper portion of the rst
quadrant of the u vs. t plane separated by the line u = t, stems form the fact that
y=u≥0 and x = t − u ≥ 0. Summing over u yields the marginal of T :
t
T ξt ξ θ
X eθu
dPθ,ξ (t) = exp{e − e (1 + e )}I{0,...,∞} (t) .
u!(t − u)!
|u=0 {z }
t!/(1+eθ )t

where in the summation we used the following identity for Z ∼ Bin(n, p):
n  z
X 1 p 1
= .
z=0
z!(n − z)! 1−p n!(1 − p)n

The density of U |t now follows straightforwardly by dividing the joint of (U, T ) by the
marginal of T. To nd the cutto points c(t) ∈ R and 0 ≤ γ(t) ≤ 1, we solve:

α = Eθ=0 [ϕ1 |t] = P (Zt > c(t)) + γ(t)P (Zt = c(t)), Zt ∼ Bin(n = t, p = 1/2).
This can be solved exactly via tables, or approximated via the CLT (normal approxima-
tion to the Binomial).

By Theorem 7.6.8, the (unconditional) UMPU level α test is identical. So what is the
dierence? In both situations we observe the value of t = x + y , so t is known. The
distinction is that if we don't x anything, then ϕ1 is only UMPU, whereas if we want
the optimal test among all those with the same value of t (x and y vary but their sum is
xed), then we obtain the stronger result that ϕ1 is in fact UMP.

2
Example 7.6.10 (Testing a normal std. deviation). For X1 , . . . , Xn ∼ iid N(µ, σ ), TSH
Ÿ5.2 investigates the following 4 tests:

• H1 : σ ≤ σ0 vs. K1 : σ > σ 0 .
• H2 : σ ≥ σ0 vs. K2 : σ < σ0 .
• H3 : µ ≤ µ0 vs. K3 : µ > µ0 .
• H4 : µ ≥ µ0 vs. K4 : µ < µ0 .
160 7. OPTIMAL TESTING THEORY

The diculty of these situations is that both parameters are unknown. TSH Ÿ3.9 shows
P
that H1 is the only one for which there exists a UMP test (which rejects for large (xi −
2
x̄) ). Here we will investigate optimal tests for H2 . Treating µ as a nuisance parameter,
we have the 2-parameter exp. family:
 

 

 1 X 2 µ X 
fσ,µ (x) = exp − 2 xi + 2 xi C(θ, ξ).

 2σ | {z } σ | {z }

| {z } |{z} 
θ u ξ t

Thus, testing H2 : σ ≥ σ0 is equivalent to H2 : θ ≥ θ0 . By Theorem 7.6.8, the UMPU

level α test ϕ2 rejects for u < c(t), and accepts if u > c(t), which is equivalent to rejecting
P 2 P
if xi < c(x̄), since t = xi = nx̄. To nd the cuto c(x̄), we solve
X 
α = Pσ0 Xi2 < c(X̄) | X̄
X 
2 2 ′
= Pσ0 Xi − nX̄ < c (X̄) | X̄ subtract a constant
X X
= Pσ0 (n − 1)S 2 < c′ (X̄) | X̄ , (n − 1)s2 = (xi − x̄)2 = x2i − nx̄2 ,

= Pσ0 (n − 1)S 2 < c′ , 2

since S is independent of X̄,

(n − 1)S 2 c′
 
= Pσ0 < 2 , divide by a constant,
σ02 σ0
= P χ2 (n − 1) < c2 , =⇒ c2 = χ2α (n − 1).

Thus the rejection rule is:

X X t2
(xi − x̄)2 < σ02 χ2α (n − 1) ⇐⇒ x2i < + σ02 χ2α (n − 1) ≡ c(t).
n
The power function of this test is:
 Z σ2 χ2α (n−1)/σ2
σ02 2

0
2
β2 (σ) = P χ (n − 1) < 2 χα (n − 1) = fn−1 (y)dy,
σ 0

where fk (y) is the density of a χ2 (k).

It is interesting to compare this test to the UMP test ϕ∗2 of H2 discussed in TSH Example
P
3.9.1, for the (much) simpler situation when µ is known, and which rejects for (xi −
µ)2 < c. Using similar arguments, we nd c by solving
!
X  X  Xi − µ 2 c
α = Pσ0 ,µ (Xi − µ)2 < c = Pσ0 ,µ < 2
σ0 σ0
 
2 c
= P χ (n) < 2 , =⇒ c = σ02 χ2α (n),
σ0
whence the power function is given by:
!
X  Xi − µ  2 σ 2 χ2 (n)

σ 2 χ2 (n)

β2∗ (σ) = Pσ,µ < 0 α2 =P χ (n) < 0 α2
2
.
σ σ σ
 7.7. LIKELIHOOD RATIO (LR), WALD, AND SCORE TESTS 161

Plotting these two power functions, we might expect their dierence to be small, with
∗ ∗
perhaps β2 (σ) slightly larger than β2 (σ) over most values of σ , since ϕ2 uses more infor-
∗
mation (note that ϕ2 cannot be implemented without knowledge of µ).

7.7. Likelihood Ratio (LR), Wald, and Score Tests

The Neyman-Pearson Lemma naturally suggests the LR as a good test. In the absence
of an optimal test (UMP, UMPU, etc.), we fall back on LR, Wald, and Score tests. A
complete coverage of this subject can be found in Severini (2000), Chs 3 & 4, and we
follow Severini's compact notation here.

We continue to let ℓ(θ) denote the log likelihood based on a sample of size n, where
θ = (θ1 , . . . , θd ) ∈ Ω ⊂ Rd . When needed, we partition θ = (ψ, λ), where ψ = (ψ1 , . . . , ψq )
d−q
denotes the parameter of interest, while λ ∈ R is a nuisance parameter. Derivatives
of ℓ(θ) w.r.t. θ are denoted as:

∂ℓ(θ)
ℓθ (θ) = , Jacobian vector (a tensor of dim=1)
∂θ
∂ 2 ℓ(θ)
ℓθθ (θ) = , Hessian matrix (a tensor of dim=2)
∂θ∂θT

etc. We assume the following (loosely stated) regularity conditions, satised by all nice
(henceforth called regular) models:
R1. ℓ(θ) can be approximated by a 4th order polynomial in θ around the true value
θ0 ∈ Ω,
1
ℓ(θ) = ℓ(θ0 ) + ℓθ (θ0 )(θ − θ0 ) + · · · + ℓθθθθ (θ0 )(θ − θ0 )4 + Rn (θ),
4!
with the remainder term satisfying the following condition over some neighbor-
hood N0 of θ0 :
supθ∈N0 |Rn (θ)|
= Op (1).
n||θ − θ0 ||5
R2. The rst 4 derivatives of ℓ(θ), {ℓθ , . . . , ℓθθθθ√
}, have joint cumulants which are
O(n), and the vector of sample averages ℓθ / n obeys the CLT:
√ d
ℓθ / n →− N(0, I(θ0 )),
where now, and throughout this chapter, I(θ) is as dened in (7.7.4).
R3. For non-negative integers {i1 , . . . , i4 } with i1 + · · · + i4 ≤ 4, and for {j, k, l, m} ∈
{0, 1, . . . , d}, one is able to interchange up to 4th order derivatives with integrals
as follows:
( )
∂ i1 +···+i4 ∂ i1 +···+i4
i4 Eθ0 exp{ℓ(θ) − ℓ(θ0 )}|θ=θ0 = Eθ0 i4 exp{ℓ(θ) − ℓ(θ0 )}|θ=θ0 .
∂θji1 ∂θki2 ∂θli3 ∂θm ∂θji1 ∂θki2 ∂θli3 ∂θm
162 7. OPTIMAL TESTING THEORY

Properties R1R3 hold in most models of practical interest (and most, if not all, models
covered in this course). Examples include models where the observations are independent
but not identical (e.g., regression and GLM), and models where the observations are
dependent (e.g., some types of stochastic processes).

Bartlett Identities
Property R3 leads in particular to the so-called Bartlett identities. The key to this is the
following equation, which in the scalar θ case is:

∂j ∂j
 
(7.7.1) Eθ exp{ℓ(θ) − ℓ(θ0 )}|θ=θ0 = Eθ0 exp{ℓ(θ) − ℓ(θ0 )}|θ=θ0 ,
∂θj 0 ∂θj
Now, since
Z Z
L(θ)
Eθ0 exp{ℓ(θ) − ℓ(θ0 )} = L(θ0 )dx = L(θ)dx = 1,
L(θ0 )
it implies that, in particular, by (7.7.1) with j = 1,
 
∂ ∂ ∂
Eθ0 exp{ℓ(θ) − ℓ(θ0 )}|θ=θ0 = Eθ0 exp{ℓ(θ) − ℓ(θ0 )}|θ=θ0 = (1) = 0,
∂θ ∂θ ∂θ
and for general j (and ∀θ0 ):
∂j ∂j
 
Eθ0 exp{ℓ(θ) − ℓ(θ0 )}|θ=θ0 = 0 = j Eθ0 exp{ℓ(θ) − ℓ(θ0 )}|θ=θ0 .
∂θj ∂θ
Thus, in the j=1 case,
 
∂ ℓ(θ)−ℓ(θ0 )
= E ℓθ (θ0 )eℓ(θ0 )−ℓ(θ0 ) = Eℓθ (θ0 ),

0=E e θ=θ0
∂θ
and since this holds for every θ0 , we obtain the 1st Bartlett Identity: Eℓθ (θ) = 0.
Similarly, in the j = 2 case,
 2 
∂
eℓ(θ)−ℓ(θ0 ) = E ℓθθ eℓ(θ0 )−ℓ(θ0 ) + ℓ2θ eℓ(θ0 )−ℓ(θ0 ) ,

0=E θ=θ0
∂θ2
which leads to the 2nd Bartlett Identity: Eℓθθ (θ) + Eℓθ (θ)2 = 0. This generalizes to the
vector θ case, and for every integer j there is a corresponding identity. The rst two
Bartlett identities are:
(7.7.2) Eℓθ (θ) = 0,
(7.7.3) Eℓθθ (θ) + Eℓθ (θ)ℓθ (θ)T = 0.

Dierent types of Information

For regular models we have the following types of information-related quantites and
results.
 7.7. LIKELIHOOD RATIO (LR), WALD, AND SCORE TESTS 163

Score Function: ℓθ (θ). The rst two Bartlett identities imply that the score
vector has mean zero and its variance is equal to the (expected) Information
matrix:

Eℓθ (θ) = 0,
T
Var[ℓθ (θ)] = Eℓθ (θ)ℓθ (θ) := I(θ), ( Expected Information).

Observed Information: J (θ) := −ℓθθ (θ). The 2nd Bartlett identity implies
that:
EJ (θ) = I(θ).
Partial Information: Invoking the θ = (ψ, λ) partition, partition the Informa-
tion matrix accordingly as:
 
Iψψ (θ) Iψλ (θ)
I(θ) = ,
Iλψ (θ) Iλλ (θ)
where, using obvious notation, we have e.g.,

Iψλ (θ) = Eℓψ (θ)ℓλ (θ)T = −Eℓψλ (θ),

and

∂ℓ(θ) ∂ℓ(θ) ∂ 2 ℓ(θ)

ℓψ (θ) = , ℓλ (θ) = , ℓψλ (θ) = .
∂ψ ∂λ ∂ψ∂λT
Definition 7.7.1 (Average Information per observation). For regular mod-
els, we dene the average (expected) Information per observation as

1
(7.7.4) I(θ) := lim I(θ).
n→∞ n

The asymptotic normality of the MLE result for regular models is more general
than those in Ch. 6, and allows us to break free from the iid assumption (e.g.,
regression). If θ̂ denotes the MLE of θ based on a sample of size n from a regular
model, then
√ d −1
(7.7.5) n(θ̂ − θ0 ) →
− N(0, I (θ0 )),
with I(θ) as dened in (7.7.4). In particular, if the MLE is based on an iid
sample of size n, then I(θ) = nI(θ), whence I(θ) coincides with the (expected)
Information per observation of Ch. 6.

Definition 7.7.2 (Partial Information). The partial (expected) Information

for ψ, dened as
−1
(7.7.6) Iψ (θ) := Iψψ (θ) − Iψλ (θ)Iλλ (θ)Iλψ (θ),
plays the same role for inference on ψ that I(θ) plays for inference on the entire
θ, and can be derived as the appropriate CRLB by generalizing the argument in
Remark 2.4.6. If λ is known, then Iψ (θ) = Iψψ (θ), so that

Iψ (θ) − Iψψ (θ)

164 7. OPTIMAL TESTING THEORY

represents the loss of information about ψ due to the fact that λ is unknown.
(Similar results hold for the denition of partial observed Information,Jψ (θ).)
Note that we can dene partial Information per observation by replacing I(θ) 7→
I(θ) everywhere in (7.7.6), whence for an iid sample, Iψ (θ) = nIψ (θ).

Let θ̂ denote the MLE of θ based on a sample of size n from a regular model. We describe
the three tests for testing the two-sided hypothesis

H : θ = θ0 vs. K : θ ̸= θ0 .
The tests reject H for large values of the corresponding statistic (W , Ww , Ws ), and as
we will show next, the asymptotic distribution of each of these under H is χ2 (d).

(i) LR Test. The test statistic is:

(7.7.7) W = W (θ0 ) := 2[ℓ(θ̂) − ℓ(θ0 )].

To derive the asymptotic distribution of W under H , Taylor-series expand ℓ(θ)−
ℓ(θ0 ) around θ̂ = θ0 , so that

1 1 √
W = ℓθ (θ0 )T (θ̂ − θ0 ) + (θ̂ − θ0 )T ℓθθ (θ0 )(θ̂ − θ0 ) + Op (1/ n).
2 2
Now, from Ch 6 results, and with Id denoting the identity matrix of rank d, we
have:
√
θ̂ − θ0 = I −1 (θ0 )ℓθ (θ0 )) + Op (1/ n),
d
I −1/2 (θ0 )ℓθ (θ0 ) →
− N(0, Id ),
√
ℓθθ (θ0 ) = −I(θ0 ) + Op ( n).
Substituting these results into the above expression for W/2, we obtain
 −1/2 T  −1/2  √
(7.7.8) W = I (θ0 )ℓθ (θ0 ) I (θ0 )ℓθ (θ0 ) + Op (1/ n),
so that we have an asymptotic chi-square distribution for the LR statistic under
the null hypothesis:
d
− χ2 (d).
W →
(ii) Wald Test. The test statistic is:

(7.7.9) Ww = Ww (θ0 ) := (θ̂ − θ0 )T I(θ̂)(θ̂ − θ0 ).

To derive the asymptotic distribution of Ww under H, Taylor-series expand the
LR test statistic W around θ0 = θ̂ θ̂ = θ0 ),
(before we expanded around so that

1 1 √
− W = ℓθ (θ̂)T (θ0 − θ̂) + (θ0 − θ̂)T ℓθθ (θ̂)(θ0 − θ̂) + Op (1/ n).
2 2
Now, since ℓθ (θ̂) = 0 and
√ √
−ℓθθ (θ̂) = I(θ0 ) + Op ( n) = I(θ̂) + Op ( n),
 7.7. LIKELIHOOD RATIO (LR), WALD, AND SCORE TESTS 165

it follows that
√ √ √
W = Ww + (θ0 − θ̂)T Op ( n) (θ0 − θ̂) +Op (1/ n) = Ww + Op (1/ n),
| {z√ } | {z√ }
Op (1/ n) Op (1/ n)

so that Ww has the same limiting distribution as W.

(ii) Score Test. The test statistic is:

(7.7.10) Ws = Ws (θ0 ) := ℓθ (θ0 )T I −1 (θ0 )ℓθ (θ0 ).

The fact that Ws has the same limiting distribution as W follows straightfor-
wardly from (7.7.8). (The Score test is also known as the Rao Score Test, or
Lagrange Multiplier Test.)

Note 7.7.3. One can use any of the four versions of Information (expected or observed

evaluated at θ0 or θ̂) in the denition of Ww and Ws , namely

{I(θ0 ), I(θ̂), J (θ0 ), J (θ̂)},

without aecting the asymptotics.

Testing only a subset of parameters

Partition θ = (ψ, λ), where ψ ∈ Rq is the parameter of interest, and λ ∈ Rd−q is a
nuisance parameter. The (unrestricted) MLE is θ̂ = (ψ̂, λ̂), and let θ̂ψ = (ψ, λ̂ψ ) denote
the (prole or restricted) MLE of θ when ψ is held xed, which just involves maximizing
ℓ(θ) over λ, i.e., λ̂ψ = arg maxλ ℓ(ψ, λ). We now wish to test

H : ψ = ψ0 vs. K : ψ ̸= ψ0 .
The analogous versions of the LR, Wald, and Score Tests are now as follows:

(7.7.11) W = W (ψ0 ) = 2[ℓ(θ̂) − ℓ(θ̂ψ0 )],

(7.7.12) Ww = Ww (ψ0 ) = (ψ̂ − ψ0 )T Iψ (θ̂)(ψ̂ − ψ0 ),
(7.7.13) Ws = Ws (ψ0 ) = ℓψ (θ̂ψ0 )T Iψ−1 (θ̂ψ0 )ℓψ (θ̂ψ0 ).
Using similar arguments as before, it can be shown that now

d
− χ2 (q),
W →
which is also the limiting distribution of Ww and Ws .
One-sided tests
Partition θ = (ψ, λ) as above, but ψ is a scalar (q = 1). To test, e.g.,

H : ψ ≤ ψ0 vs. K : ψ > ψ0 ,
use the signed square root of the statistics in (7.7.11)(7.7.13):
p
R = R(ψ0 ) = − ψ0 ) W (ψ0 ),
sgn(ψ̂
p
Rw = Rw (ψ0 ) = sgn(ψ̂ − ψ0 ) Ww (ψ0 ),
p
Rs = Rs (ψ0 ) = sgn(ψ̂ − ψ0 ) Ws (ψ0 ).
166 7. OPTIMAL TESTING THEORY

To derive the asymptotics, one can show that (under H)

√
q
R(ψ0 ) = Iψ (θ̂)(ψ̂ − ψ0 ) + Op (1/ n),
and thus, since the rst term in the above summand converges to a standard normal, we
obtain
d
R →
− N(0, 1),

with identical conclusions for Rw and Rs .

Condence Regions
Construction of condence regions by inverting each of the tests is immmediate. E.g., if
W (θ0 ) denotes any of the three test (7.7.7), (7.7.9), or (7.7.10), inversion of the two-sided
test leads to the (1 − α) acceptance region
A(θ0 ) = θ0 | W (θ0 ) ≤ χ21−α (d) .


Likewise, inversion of the two-sided subset case tests (7.7.11)(7.7.13), leads to the (1−α)
acceptance region

A(ψ0 ) = ψ0 | W (ψ0 ) ≤ χ21−α (q) .



Example 7.7.4 (Inference for Weibull shape). Consider the following shape-scale parametriza-
tion for the density of a Weibull distribution

fθ (x) = ψλ(λx)ψ−1 exp −(λx)ψ I(x > 0),


θ = (ψ, λ),
where the parameter of interest ψ > 0 controls the shape, while the inverse of the nuisance
parameter λ > 0 controls the scale. Since this is not an exponential family, we have little
hope of deducing any kind of optimal test. Tedious computations lead to the following
expression for the Information matrix (per oservation):
" #
π 2 /6+γ 2 −2γ 1−γ  
ψ2 λ Iψψ (θ) Iψλ (θ)
I(θ) = ψ2
= , γ = 0.5772 . . . ( Euler's constant).
1−γ Iλψ (θ) Iλλ (θ)
λ λ2

From this we obtain the partial Information

π2
 
−1 n
Iψ (θ) = Iψψ (θ) − Iψλ (θ)Iλλ (θ)Iλψ (θ), =⇒ Iψ (θ) = nIψ (θ) = 2 −1 .
ψ 6
The log likelihood based on a random sample of size n is
X X
ℓ(θ) = ℓ(ψ, λ) = nψ log(λ) + n log ψ + (ψ − 1)t − λψ sψ , t= log xi , sψ = xψi .

It is possible to obtain the prole MLE of λ as λ̂ψ = (n/sψ )1/ψ , which upon substitution
leads to the prole log-likelihood
 
n
ℓ(θ̂ψ ) = ℓ(ψ, λ̂ψ ) = n log + n log ψ + (ψ − 1)t − n.
sψ
 7.7. LIKELIHOOD RATIO (LR), WALD, AND SCORE TESTS 167

This can now be maximized (numerically) for the MLE of ψ which is then substituted
into the above formulas, yielding the following cascade of results:
!1/ψ̂
X n
ψ̂ = arg max ℓ(θ̂ψ ), sψ̂ = xψ̂i , λ̂ = λ̂ψ̂ = , ℓ(θ̂) = ℓ(ψ̂, λ̂).
ψ sψ̂
Straightforward substitution into (7.7.11)(7.7.13) then leads to:
!
sψ0 ψ̂  
W (ψ0 ) = 2n log + 2 ψ̂ − ψ0 t.
sψ̂ ψ0
 2  2 
ψ0 π
Ww (ψ0 ) = −1 n −1 .
ψ̂ 6
!2   
P ψ0 3
π2

ψ0 xi log xi ψ0 t n
Ws (ψ0 ) = − −1 −1 .
sψ0 n ψ0 6
A two-sided level α test of H : ψ = ψ0 then rejects W (ψ0 ) > χ21−α (1), whereas the
for
p
one-sided test of H : ψ ≤ ψ0 rejects for R(ψ0 ) = sgn(ψ̂ − ψ0 ) W (ψ0 ) > z1−α , etc.
To illustrate construction of condence intervals, inversion of Wald leads to the (1 − α)
acceptance region
A(ψ0 ) = ψ0 | Ww (ψ0 ) ≤ χ21−α (1) .


Example 7.7.5 (Neyman Smooth Test). Suppose we have a family of densities from a
full-rank k -parameter exponential family

k
X Xk
fθ (x) = c(θ) exp{ θj tj (x)} = exp{ θj tj (x) − log c(θ)−1 },
j=1 j=1

where θ ∈ Ω ⊂ Rk is the natural parameter set (of which θ = 0 is an interior point), and
where the tj (x) are a set of orthonormal functions satisfying:
(
1, i = j,
E0 (tj (X)) = 0, Cov0 (ti (X), tj (X)) = δij =
0, i ̸= j.
(The notation E0 and Cov0 here means that expectations are taken with respect to the
measure for the case θ = 0, which is a Unif(0, 1).) By Theom 2.4.7 in canonical form, we
identify A(θ) = − log c(θ), whence the rst two moments for the vector t = (t1 , . . . , tk )′
are:

∂A(θ) 1 ∂c(θ)
(7.7.14) 0 = E0 (t) = =− ,
∂θ θ=0 c(θ) ∂θ θ=0
∂ 2 A(θ)
 
∂ 1 ∂c(θ)
(7.7.15) Ik = Cov0 (t) = = − .
∂θ∂θ′ θ=0 ∂θ′ c(θ) ∂θ θ=0

For a random sample x1 , . . . , xn from fθ (x), and in the context of goodness-of-t, Neyman
(1937) proposed (what is now known to be) a Score test for H : θ = 0 vs. K : θ ̸= 0.
168 7. OPTIMAL TESTING THEORY

T = ( t1 (xi ), . . . , tk (xi ))′ ,

P P
Note that now the CSS is and the new A function is
An (θ) = nA(θ), so that the log-likelihood and its derivative are:

∂A(θ)
ℓ(θ) = θ′ T − nA(θ), ℓθ (θ) = T − n .
∂θ
Thus, under H, we have from (7.7.14) that ℓθ (0) = T , and (7.7.15) implies that

∂ 2 A(θ)
I(0) = n = nIk .
∂θ∂θ′ θ=0
which leads to
1 ′ d
WS = ℓθ (0)′ I(0)−1 ℓθ (0) = − χ2 (k),
TT → under H.
n
Example 7.7.6 (Poisson GLM). In this log-linear regression model for counts, we observe
{(x1 , y1 ), . . . , (xn , yn )}, where the yi are independent Poisson with means µi =
the pairs
exp{λ + ψxi }, and the xi are known covariates. The joint density of the yi is therefore
seen to be the 2-parameter exponential family,
( )
X Y 1 X X
f (y) = exp ψt + λs − eλ eψxi I{0,1,...} (yi ), s= yi , t = xi y i ,
i i
yi ! i i

and the goal is to test if there is an eect from the covariates, i.e. H : ψ = 0 vs. K : ψ ̸= 0.
Since this is in canonical form for θ = (ψ, λ) with (s, t) the CSS, we make the identication
A(θ) = eλ u(ψ), where u(ψ) =
P ψxi
e , whence the Information matrix for the model is
obtained straightforwardly as:

∂ 2 A(θ)
 
λ u(ψ) v(ψ)
X X
I(θ) = =e , v(ψ) = xi eψxi , w(ψ) = x2i eψxi ,
∂θ∂θT v(ψ) w(ψ)
and since this is a regular model, we have the asymptotic distribution for the MLE of θ
as in (7.7.5). The partial information is:
2
Iψ (θ) = Iψψ (θ) − Iψλ (θ)/Iλλ (θ) = eλ [w(ψ) − v 2 (ψ)/u(ψ)].
Now, since ℓ(θ) = ψt + λs − A(θ) + constant, we have from the score equations

ℓψ (θ) = t − eλ v(ψ) = 0,
ℓλ (θ) = s − eλ u(ψ) = 0, =⇒ λ̂ψ = log(s/u(ψ)),

but we must then solve for the MLE of ψ numerically to obtain: θ̂ = (ψ̂, λ̂ψ̂ ), leading to,

ℓ(θ̂ψ ) = ℓ(ψ, λ̂ψ ) = ψt + s log(s/u(ψ)) − s,

ℓ(θ̂) = ℓ(ψ̂, λ̂ψ̂ ) = ψ̂t + s log(s/u(ψ̂)) − s.

and,
v 2 (ψ) v̂ 2
   
s s
Iψ (θ̂ψ ) = w(ψ) − , Iψ (θ̂) = ŵ − ,
u(ψ) u(ψ) û û
 7.7. LIKELIHOOD RATIO (LR), WALD, AND SCORE TESTS 169

where we use the shorthand û ≡ u(ψ̂), etc. In addition, we will need the MLEs at the
null value of ψ0 = 0: θ̂ψ0 ≡ θ̂0 = (0, λ̂0 ), where λ̂0 ≡ λ̂ψ0 = log(s/n) = log(ȳ), which
leads to:
hX i
ℓψ (θ̂0 ) = t − ȳx̄, and Iψ (θ̂0 ) = ȳ x2i − nx̄ 2
.
From the above results, we can now calculate the triad of statistics, all of which are χ2 (1)
under H:

• LR: h i
W (0) = 2[ℓ(θ̂) − ℓ(θ̂0 )] = 2 tψ̂ + s log(n/û) .
• Wald:
v̂ 2
 
2 s
WW (0) = (ψ̂ − ψ0 ) Iψ (θ̂) = ŵ − ψ̂ 2 .
û û
• Score:
(t − ȳx̄)2
WS (0) = ℓψ (θ̂0 )2 /Iψ (θ̂0 ) = P 2 .
ȳ[ xi − nx̄2 ]

It would be interesting to compare this triad of tests with the UMPU, Case (4) of the
multiparameter EF, with critical function:


 1, t < c1 (s) or t > c2 (s),

γ (s), t = c (s),
1 1
ϕ=

 γ2 (s), t = c2 (s),

0, otherwise,


where the cuto points are determined from Eψ=0 (ϕ|S) = α and Eψ=0 (ϕT |S) = αEψ=0 (T |S).
−1
Q
However, the i (yi !) term in the expression for f (y) appears to make the calculation
of the joint distribution of (T, S) intractable!
170 7. OPTIMAL TESTING THEORY

7.8. Discussion
• The LR, Wald, and Score statistics can be shown to have a non-central chi-square
asymptotic distribution under (local) alternatives (Severini, 2000, Ch 4).
• The asymptotic normality result for the MLE in (7.7.5) holds quite generally
beyond iid data. Two common instances that considerably extend our range of
applications include: (i) regression models where the data are independent but
not identically distributed, and (ii) stationary time series models where the data
are not independent but are identically (marginally) distributed.
• Example 7.7.6 is at the threshold of tractability in terms of what can feasibly be
analytically computed for the sub-optimal LR, Wald, and Score tests. In prac-
tice (implemented in software packages) the process is automated by computing
the MLEs numerically, and substituting the expected by the observed Informa-
tion throughout, I(θ) 7→ J (θ), which requires only numerical evaluation of the
Hessian of ℓ(θ).
• For small n it may be necessary to compute the null distribution of the sub-
optimal LR, Wald, and Score test statistics via Monte Carlo simulation, since
2
the asymptotic χ may be unreliable.
• Hypothesis testing in the Big Data era (Efron, 2010). The 21st century has
ushered in the era of high-dimensional testing, where dim(θ) = d ≫ n. One
successful way forward here has been the formulation of this problem into a
large-scale testing framework, where one constructs many simultaneous tests and
tries to control the error rate. E.g., Efron (2010) describes a typical microarray
study on the eect of d = 6, 033 genes on n = 102 subjects (52 with disease and
50 without, serving as controls). The eect of each gene is then investigated
individually by carrying out d two-sample t-tests. The struggle here has been
twofold: (i) adapting existing multiple comparison procedures (e.g., Tukey's
MCP) to cope with a number of comparisons far in excess of what they were
designed for, and (ii) devising new types or denitions of error rate. The current
best recommendations from Efron (2010) are usage of: (i) adapted FamilyWise
Error Rate (FEWR) control procedures, and (ii) the False Discovery Rate (FDR)
paradigm proposed by Benjamini & Hochberg (1995).
 Bibliography

[1] Y. Benjamini and Y. Hochberg (1995). Controlling the false discovery rate: A practical and powerful
approach to multiple testing. J. Roy. Statist. Soc. Ser. B, 57, 289300.
[2] P. Billingsley (1995). Probability and Measure, 3rd Edition. Wiley.
[3] P.J. Bickell and K.A. Doksum (2015). Mathematical Statistics (Vol. 1), 2nd Edition. CRC Press.
[4] P.J. Brockwell and R.A. Davis (1991). Time Series: Theory and Methods, 2nd Edition. Springer.
[5] B. Efron (2010). Large-Scale Inference: Empirical Bayes Methods for Estimation, Testing and Pre-
diction. Cambridge.
[6] B. Efron and T. Hastie (2016). Computer Age Statistical Inference: Algorithms, Evidence, and Data
Science. Cambridge.
[7] F.A. Graybill and R.B. Deal (1959). Combining unbiased estimators. Biometrics, 15, 543550.
[8] W. James and C. Stein (1961). Estimation with quadratic loss. Proc. 4th Berkeley
Symp. Math. Statist. Prob., 1. Berkeley: University of California Press.
[9] E.L. Lehmann and G. Casella (1998). Theory of Point Estimation, 2nd Edition. Springer.
[10] E.L. Lehmann and J.P. Romano (2005). Testing Statistical Hypotheses, 3rd Edition. Springer.
[11] J. Neyman (1937). Smooth test for goodness of t. Scandinavian Actuarial Journal, 3, 149199.
[12] M.J. Schervish (1995). Theory of Statistics. Springer.
[13] R.J. Sering (1980). Approximation Theorems of Mathematical Statistics. Wiley.
[14] T.A. Severini (2000). Likelihood Methods in Statistics. Oxford.
[15] A. Van der Vaart (1998). Asymptotic Statistics. Cambridge.

171