Introduction to Statistical Inference

Complete Theory & Problem Compendium

MATH F315 — BITS Pilani Goa, Sem 1 2026–27

Instructor: Prof. Shilpa Gondhali Textbook: Rohatgi & Saleh (3rd ed., Wiley 2015)

Compiled for Exam Preparation

2026–27
 Preface

This compendium is designed to be the only document you need for MATH F315 (Introduction
to Statistical Inference). Every major theorem is stated precisely and proved in full, with every
algebraic step written out. Before each formal statement, there is conceptual motivation explaining
why the result matters and what problem it solves. After each theorem, worked examples show
you exactly how to apply it — including the thought process: “how do I know to do this?”
How to use this document. Read the motivation first to build intuition. Then work through
the proof carefully — cover it and try to reconstruct each step. Finally, attempt the worked
examples before reading the solutions.
Reference texts:
• T1: Rohatgi & Saleh, Introduction to Probability & Statistics (3rd ed., Wiley 2015)
• R1: Casella & Berger, Statistical Inference
• R2: Hogg, McKean & Craig (8th ed.)
Contents

I Parametric Point Estimation 1

1 Statistical Models and Estimators 3

1.1 Parametric Statistical Models . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
1.2 Random Samples and Statistics . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4

2 Properties of Estimators: Unbiasedness, MSE, Consistency, BLUE 5

2.1 Unbiasedness . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5
2.2 Mean Square Error and the Bias-Variance Decomposition . . . . . . . . . . . . . 7
2.3 Consistency . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8
2.4 Best Linear Unbiased Estimators (BLUE) . . . . . . . . . . . . . . . . . . . . . . 9
2.5 Method of Moments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10

3 Sufficiency, Completeness, and Ancillarity 11

3.1 Sufficiency . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11
3.1.1 The Neyman-Fisher Factorization Theorem . . . . . . . . . . . . . . . . . 12
3.2 Exponential Families . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14
3.3 Completeness . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15
3.4 Ancillary Statistics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16
3.5 Minimal Sufficient Statistics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17
3.6 Basu’s Theorem . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17

4 UMVUE Theory: Rao-Blackwell and Lehmann-Scheffé 19

4.1 The UMVUE . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19
4.2 The Rao-Blackwell Theorem . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20
4.3 The Lehmann-Scheffé Theorem . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21

5 Fisher Information and the Cramér-Rao Lower Bound 25

5.1 The Score Function . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25
5.2 Fisher Information . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26
5.3 The Cramér-Rao Lower Bound . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28

6 Maximum Likelihood Estimation 31

6.1 The Likelihood Function and MLE . . . . . . . . . . . . . . . . . . . . . . . . . . 31
6.2 Finding MLEs: Full Worked Examples . . . . . . . . . . . . . . . . . . . . . . . . 32
6.3 Properties of the MLE . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33
6.3.1 Invariance Property . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33
6.3.2 Asymptotic Properties . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34
6.4 MLEs in Exponential Families . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34

iii
iv CONTENTS

7 Bayes and Minimax Estimation 37

7.1 The Bayesian Framework . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37
7.2 Conjugate Priors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38
7.3 Minimax Estimation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39

II Interval Estimation 41

8 Confidence Intervals 43
8.1 Pivotal Quantity Method . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43
8.2 Confidence Interval for Normal Mean — σ 2 Known . . . . . . . . . . . . . . . . . 43
8.3 Confidence Interval for Normal Mean — σ 2 Unknown: The t-Interval . . . . . . . 44
8.4 Confidence Interval for Variance: χ2 -Interval . . . . . . . . . . . . . . . . . . . . 45
8.5 Large-Sample Confidence Intervals . . . . . . . . . . . . . . . . . . . . . . . . . . 45

III Hypothesis Testing 47

9 Foundations of Hypothesis Testing 49

9.1 The Basic Setup . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 49

10 The Neyman-Pearson Theory: Optimal Tests 51

10.1 The Neyman-Pearson Lemma . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51
10.2 Uniformly Most Powerful Tests . . . . . . . . . . . . . . . . . . . . . . . . . . . . 53
10.3 Locally Most Powerful Tests . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 54

11 Generalized Likelihood Ratio Tests and Standard Tests 55

11.1 The Generalized Likelihood Ratio Test . . . . . . . . . . . . . . . . . . . . . . . . 55
11.2 The z-Test via GLRT . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 56
11.3 The t-Test via GLRT . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 56
11.4 The χ2 -Test for Variance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 56

IV Tests of Independence and Association 59

12 Pearson’s Chi-Squared Test and Kendall’s Tau 61

12.1 Pearson’s Chi-Squared Goodness-of-Fit Test . . . . . . . . . . . . . . . . . . . . . 61
12.2 Pearson’s Chi-Squared Test of Independence . . . . . . . . . . . . . . . . . . . . . 62
12.3 Kendall’s Tau . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 63

V Comprehensive Worked Problem Bank 67

13 Point Estimation: Problems with Full Solutions 69

14 Testing: Problems with Full Solutions 73

A Key Distributions and Reference Formulas 75

A.1 Important Distributions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 75
A.2 Key Relationships . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 75
A.3 Critical Inequality Facts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 76
A.4 Summary: UMVUE Finding Strategy . . . . . . . . . . . . . . . . . . . . . . . . 76
A.5 Summary: Hypothesis Testing Decision Tree . . . . . . . . . . . . . . . . . . . . . 77
 Part I

Parametric Point Estimation

1
Chapter 1

Statistical Models and Estimators

How This Chapter Fits Into the Course

Statistical inference begins with a deceptively simple question: we observe data — what can we
learn about the process that generated it? We cannot directly observe the true population; we
can only observe a random sample from it. The goal is to use that sample to draw conclusions
about unknown parameters that characterize the population.
This chapter sets up the language we need. We define what a parametric model is, what an
estimator is, and what properties we might want an estimator to have. Everything in the rest of
the course — sufficiency, UMVUE theory, the Cramér–Rao bound, MLE, confidence intervals,
and hypothesis testing — is an answer to the question: among all possible estimators, which one
is best, and in what sense?

1.1 Parametric Statistical Models

Parametric Family / Statistical Model

A parametric family of distributions is a set

P = {Pθ : θ ∈ Θ}

indexed by a parameter θ taking values in a parameter space Θ ⊆ Rk . Each Pθ is a

probability distribution on the same sample space X . If θ ∈ R we say the model is
one-parameter; if θ = (θ1 , . . . , θk ) ∈ Rk it is k-parameter.
When each Pθ has a density (PDF or PMF) fθ , we write X ∼ fθ (x) to mean X has
distribution Pθ .

Standard Parametric Families

• X ∼ N (µ, σ 2 ): θ = (µ, σ 2 ), Θ = R × (0, ∞).

• X ∼ Poisson(λ): θ = λ, Θ = (0, ∞).

• X ∼ Bernoulli(p): θ = p, Θ = (0, 1).

• X ∼ Gamma(α, β): θ = (α, β), Θ = (0, ∞)2 .

• X ∼ Uniform(0, θ): θ ∈ (0, ∞).

The unknown θ is what we want to estimate from data.

3
4 CHAPTER 1. STATISTICAL MODELS AND ESTIMATORS

1.2 Random Samples and Statistics

Random Sample and Statistic

iid
X1 , . . . , Xn ∼ fθ is a random sample of size n from the model fθ . The joint density of
(X1 , . . . , Xn ) is
n
Y
fθ (x) = fθ (xi ).
i=1

A statistic is any measurable function T = T (X1 , . . . , Xn ) that does not depend on θ.

An estimator of ψ(θ) is a statistic T used to estimate the quantity ψ(θ).

Estimator vs. Estimate

The word estimator refers to the random variable T (X1 , . . . , Xn ) (before data are observed).
Once we plug in observed values x1 , . . . , xn , we get the estimate T (x1 , . . . , xn ), a fixed
number. The distinction matters: statistical theory (bias, variance, MSE) refers to the
distribution of the estimator over all possible samples.
Chapter 2

Properties of Estimators:
Unbiasedness, MSE, Consistency,
BLUE

How This Chapter Fits Into the Course

Given infinitely many possible estimators of θ, we need criteria to compare them. This chapter
introduces the four most fundamental:

1. Unbiasedness: on average, does the estimator hit the target?

2. MSE: how spread around the target is the estimator?
3. Consistency: as n → ∞, does the estimator converge to θ?
4. BLUE: among all linear unbiased estimators, which has smallest variance?

These criteria are used constantly throughout the rest of the course.

2.1 Unbiasedness
Unbiased Estimator

An estimator T = T (X) is unbiased for ψ(θ) if

Eθ [T (X)] = ψ(θ) for all θ ∈ Θ.

The bias of T is defined as

Bias(T, θ) = Eθ [T ] − ψ(θ).

A parametric function ψ(θ) is called estimable if there exists an unbiased estimator for
it.

Why unbiasedness matters

Unbiasedness means that if we could repeat the experiment infinitely many times and
average our estimates, we would converge to the true value. This is a desirable long-run
frequency property. However, it is not the only criterion — sometimes a slightly biased
estimator with much lower variance is preferable (the bias-variance tradeoff, Section 2.2).

5
6 CHAPTER 2. PROPERTIES OF ESTIMATORS: UNBIASEDNESS, MSE, CONSISTENCY, BLUE

Sample Mean and Sample Variance Are Unbiased

Let X1 , . . . , Xn be a random sample with E[Xi ] = µ and Var(Xi ) = σ 2 .

1 Pn
Claim 1: X̄ = n i=1 Xi is unbiased for µ.
Proof:
n n
" #
1X 1X 1
E[X̄] = E Xi = E[Xi ] = · nµ = µ. ✓
n i=1 n i=1 n

1 Pn
Claim 2: S 2 = (X − X̄)2 is unbiased for σ 2 .
n−1 h i
i=1 i
(Xi − X̄)2 . Use the identity:
P
Proof: First compute E

n
X n
X
(Xi − X̄)2 = Xi2 − nX̄ 2 .
i=1 i=1

Taking expectations:
" n # n
X X
Xi2 E[Xi2 ] = n σ 2 + µ2 ,


E =
i=1 i=1
!
h i     σ2
E nX̄ 2 = n E[X̄ 2 ] = n Var X̄ + (E[X̄])2 =n + µ2 = σ 2 + nµ2 .
n

Therefore " n #
X
2
E (Xi − X̄) = n(σ 2 + µ2 ) − (σ 2 + nµ2 ) = (n − 1)σ 2 .
i=1
(n−1)σ 2
Hence E[S 2 ] = n−1 = σ2. ✓
Why divide by n − 1 and not n? If we used σ̂ 2 = n1 (Xi − X̄)2 , then E[σ̂ 2 ] = n−1 2
n σ ̸=
P
2 2
σ — it underestimates σ because we’ve used the data to estimate µ with X̄, which “uses
up” one degree of freedom.

Non-existence of Unbiased Estimators

Let X ∼ Bernoulli(p) and ψ(p) = p2 . We claim there is no unbiased estimator of p2 based

on a single observation X.
Any estimator T (X) satisfies T (0) = a and T (1) = b for some constants. For unbiasedness:

Ep [T (X)] = a(1 − p) + bp = p2 for all p ∈ (0, 1).

But a(1 − p) + bp = a + (b − a)p is linear in p, while p2 is quadratic. A linear function

cannot equal a quadratic for all p — contradiction. Hence no unbiased estimator of p2
based on n = 1 exists.
However, with n = 2 observations, T = X1 X2 is unbiased: E[X1 X2 ] = E[X1 ]E[X2 ] =
p · p = p2 . ✓
2.2. MEAN SQUARE ERROR AND THE BIAS-VARIANCE DECOMPOSITION 7

2.2 Mean Square Error and the Bias-Variance Decomposition

Mean Square Error

The mean square error (MSE) of an estimator T for ψ(θ) is

h i
MSE(T, θ) = Eθ (T − ψ(θ))2 .

Bias-Variance Decomposition

For any estimator T and target ψ(θ):

 2
MSE(T, θ) = Var(T ) + Bias(T, θ) .

Proof of Bias-Variance Decomposition

Let µT = Eθ [T ] and b = µT − ψ(θ) (the bias). Then:

h i
MSE(T, θ) = E (T − ψ(θ))2
h i
= E (T − µT + µT − ψ(θ))2
h i
= E (T − µT )2 + 2(T − µT )(µT − ψ(θ)) + (µT − ψ(θ))2
h i
= E (T − µT )2 + 2(µT − ψ(θ)) E[T − µT ] +(µT − ψ(θ))2
| {z }
=0
= Var(T ) + b2 .

The Bias-Variance Tradeoff

For an unbiased estimator, b = 0 so MSE = Var(T ). We can sometimes reduce MSE by

introducing a small bias while greatly reducing variance. Ridge regression is a famous
example. The Cramér–Rao bound (Chapter 5) gives a lower bound on Var(T ) for unbiased
T , so for unbiased estimators MSE = variance = (bounded below by) CR bound.

Comparing Two Estimators via MSE

iid
Let X1 , . . . , Xn ∼ N (µ, σ 2 ). Compare:

• T1 = X̄: unbiased, Var(T1 ) = σ 2 /n, MSE(T1 ) = σ 2 /n.

• T2 = cX̄ for constant c ̸= 1: E[T2 ] = cµ, bias = (c − 1)µ, Var(T2 ) = c2 σ 2 /n, so

c2 σ 2
MSE(T2 ) = + (c − 1)2 µ2 .
n

For T2 to beat T1 in MSE:

c2 σ 2 σ2
+ (c − 1)2 µ2 < .
n n
√
This can hold for certain c when |µ| is large relative to σ/ n, showing that biased
estimators can outperform unbiased ones.
8 CHAPTER 2. PROPERTIES OF ESTIMATORS: UNBIASEDNESS, MSE, CONSISTENCY, BLUE

2.3 Consistency

Consistent Estimator

A sequence of estimators Tn = Tn (X1 , . . . , Xn ) is (weakly) consistent for ψ(θ) if for

every ε > 0:
lim Pθ (|Tn − ψ(θ)| > ε) = 0 for all θ ∈ Θ.
n→∞
P
In other words, Tn −
→ ψ(θ) as n → ∞.

Sufficient Condition for Consistency via MSE

If MSE(Tn , θ) → 0 as n → ∞ for all θ ∈ Θ, then Tn is consistent for ψ(θ).

Proof

By Markov’s inequality applied to (Tn − ψ(θ))2 :

  E[(Tn − ψ(θ))2 ] MSE(Tn , θ)
P (|Tn − ψ(θ)| > ε) = P (Tn − ψ(θ))2 > ε2 ≤ 2
= → 0.
ε ε2

Corollary

 n ) → 0, then Tn is consistent.
In particular, if Tn is unbiased (so MSE = Var(Tn )) and Var(T
This is why X̄ is consistent for µ: it is unbiased and Var X̄ = σ 2 /n → 0.

Consistency of the Sample Mean

iid
Let X1 , . . . , Xn ∼ with mean µ and variance σ 2 <∞. Then X̄ is consistent for µ.
Proof 1 (via Theorem 2.3): MSE(X̄, µ) = Var X̄ = σ 2 /n → 0. ✓
Proof 2 (via Chebyshev): By Chebyshev’s inequality:
 
Var X̄ σ2
P (|X̄ − µ| > ε) ≤ = → 0. ✓
ε2 nε2
Proof 3 (Weak Law of Large Numbers): This is precisely the statement of the
WLLN.

Consistency of S 2

1
S2 = (Xi − X̄)2 is consistent for σ 2 .
P
n−1  
E[S 2 ] = σ 2 (unbiased). For the variance: one can show Var S 2 = n1 µ4 − n−3 4 where


n−1 σ
µ4 = E[(X − µ)4 ]. Assuming µ4 < ∞, Var S 2 → 0, so S 2 is consistent for σ 2 .


2.4. BEST LINEAR UNBIASED ESTIMATORS (BLUE) 9

2.4 Best Linear Unbiased Estimators (BLUE)

Linear Estimator and BLUE

A linear estimator of µ is any T = ni=1 ai Xi . It is unbiased for µ iff

P P
ai = 1 (so
P
E[T ] = µ ai = µ).
The Best Linear Unbiased Estimator (BLUE) of µ is the linear unbiased estimator
with smallest variance among all linear unbiased estimators.

BLUE for iid Case

iid 1
If X1 , . . . , Xn ∼ with mean µ and variance σ 2 , then X̄ =
P
n Xi is the BLUE of µ.

Proof
P P
Among all linear unbiased estimators T = ai Xi with ai = 1:
n
X n
X
Var(T ) = a2i σ 2 =σ 2
a2i .
i=1 i=1

We minimize a2i subject to ai = 1. By the Cauchy–Schwarz inequality (or Lagrange

P P

multipliers):
n
!2 n
!2 n
! n
! n
X X X X X
1= ai = ai · 1 ≤ a2i 1 2
=n a2i ,
i=1 i=1 i=1 i=1 i=1

a2i ≥ 1/n. Equality

  holds iff a1 = a2 = · · · = an = 1/n, giving T = X̄. Hence
P
so
2
Var(T ) ≥ σ /n = Var X̄ .

Gauss-Markov Theorem — Non-iid Case

If X1 , . . . , Xn are independent (not necessarily identically distributed) with E[Xi ] = µ and

Var(Xi ) = σi2 , then the BLUE of µ is
Pn
i=1 wi Xi 1
µ̂ = P n , wi = .
i=1 wi σi2

(Weighted average, giving more weight to more precise observations.)

Proof

We minimize Var(T ) = a2i σi2 subject to ai = 1. Using the Lagrangian L =

P 2 2
ai σi −
P P

λ( ai − 1):
P
∂L λ 1
= 2ai σi2 − λ = 0 =⇒ ai = 2 ∝ 2 = wi .
∂ai 2σi σi
P P
The constraint ai = 1 gives ai = wi / j wj .
10CHAPTER 2. PROPERTIES OF ESTIMATORS: UNBIASEDNESS, MSE, CONSISTENCY, BLUE

2.5 Method of Moments

Method of Moments Estimator

The k-th population moment is µ′k = Eθ [X k ], which is a function of θ. The k-th

sample moment is m′k = n1 ni=1 Xik .
P

The Method of Moments (MoM) estimator is obtained by:

1. Express the first k population moments in terms of θ: µ′j = gj (θ), j = 1, . . . , k.

2. Set population moments equal to sample moments: gj (θ) = m′j , j = 1, . . . , k.

3. Solve for θ.

MoM for Normal Distribution

iid
X1 , . . . , Xn ∼ N (µ, σ 2 ). Parameters: θ = (µ, σ 2 ).
Population moments:

µ′1 = E[X] = µ, µ′2 = E[X 2 ] = σ 2 + µ2 .

Set equal to sample moments:

µ = m′1 = X̄,
1X 2
σ 2 + µ2 = m′2 = Xi .
n
2 = n1 Xi2 − X̄ 2 = n1 (Xi − X̄)2 .
P P
Solving: µ̂MoM = X̄ and σ̂MoM
Note this is the biased estimator (divides by n, not n − 1).

MoM for Gamma Distribution

α−1 −x/β
X ∼ Γ(α, β) with PDF f (x) = x β α Γ(α)
e
.
Population moments: E[X] = αβ, Var(X) = αβ 2 , so E[X 2 ] = αβ 2 + α2 β 2 .
Setting X̄ = αβ and n1 Xi2 = αβ 2 + α2 β 2 :
P

m′2 − (m′1 )2 Sn2 X̄ X̄ 2

β̂ = = , α̂ = = ,
m′1 X̄ β̂ Sn2
1
where Sn2 = (Xi − X̄)2 .
P
n

Properties of MoM

MoM estimators are:

P
• Consistent (by LLN: m′k −
→ µ′k , and by continuous mapping theorem).

• Easy to compute, even for complex distributions.

• Not always unbiased, not always efficient.

• Sometimes inadmissible (beaten by MLE in terms of MSE).

Chapter 3

Sufficiency, Completeness, and

Ancillarity

How This Chapter Fits Into the Course

Suppose you observe n = 100 Bernoulli(p) random variables and record all 100 values (X1 , . . . , X100 ).
P
You immediately see that the full data contains Xi = 60 ones. Is there any reason to retain the
P
full sequence (X1 , . . . , X100 ), or does T = Xi = 60 capture everything the data can possibly
tell you about p?
P
The answer is: T = Xi is sufficient for p — conditioning on T , the distribution of the full
data does not depend on p. This is the concept of sufficiency: a statistic T is sufficient if it
contains all the information about θ in the sample.
This chapter develops:

• The formal definition of sufficiency and the Neyman–Fisher Factorization Theorem

(the practical tool for identifying sufficient statistics).

• Completeness: a stronger property that kills all “irrelevant” unbiased estimators of zero.

• Ancillarity: statistics whose distribution is free of θ and which contain no information

about θ.

• Basu’s theorem: complete sufficient statistics are independent of every ancillary statistic.

• Minimal sufficiency: the most informative data reduction.

3.1 Sufficiency
Sufficient Statistic

Let {Pθ : θ ∈ Θ} be a family of distributions for X = (X1 , . . . , Xn ). A statistic T = T (X)

is sufficient for θ (or for the family {Pθ }) if the conditional distribution of X given T = t
does not depend on θ for any t.
That is: Pθ (X ∈ A | T = t) is the same for all θ ∈ Θ.

Intuition

Once you know T = t, knowing θ gives you no additional information about the individual
observations. All the information about θ has been “compressed” into T .

11
12 CHAPTER 3. SUFFICIENCY, COMPLETENESS, AND ANCILLARITY

Sufficiency from First Principles: Binomial

iid
X1 , . . . , Xn ∼ Bernoulli(p). Let T = ni=1 Xi .
P
P
For any sequence (x1 , . . . , xn ) with xi = t:

Pp (X = x, T = t) Pp (X = x) X
Pp (X = x | T = t) = = (since T = xi = t given X = x)
Pp (T = t) Pp (T = t)
Qn x 1−x
p i (1 − p) i
= i=1n
t n−t
t p (1 − p)
pt (1 − p)n−t 1
= n
t n−t
= n
.
t p (1 − p) t
P
This does not depend on p! So T = Xi is sufficient for p.
Meaning: Once you know the total number of successes, the arrangement of those
successes among the n trials contains no additional information about p.

3.1.1 The Neyman-Fisher Factorization Theorem

Checking sufficiency from the definition (as above) can be tedious. The Factorization Theorem
provides a far more practical criterion.
Neyman-Fisher Factorization Theorem

Let fθ (x) be the joint PMF or PDF of X = (X1 , . . . , Xn ). Then T (X) is sufficient for θ if
and only if there exist functions g(t, θ) and h(x) such that

fθ (x) = g(T (x), θ) · h(x)

for all x and all θ.

Proof (Discrete case)

(⇐) Suppose the factorization holds. For any x with T (x) = t:

Pθ (X = x)
Pθ (X = x | T = t) =
Pθ (T = t)
g(t, θ)h(x)
=P ′
x′ :T (x′ )=t g(t, θ)h(x )
g(t, θ)h(x)
= P ′)
g(t, θ) x′ :T (x′ )=t h(x
h(x)
=P ′)
.
x′ :T (x′ )=t h(x

This is free of θ, so T is sufficient. ✓

(⇒) Suppose T is sufficient. Let q(t) = Pθ (T = t) (this depends on θ, so write qθ (t)).
Then:

Pθ (X = x) = Pθ (X = x | T = t) · Pθ (T = t) = P (X = x | T = t) · qθ (t) .
| {z } | {z }
free of θ, = h(x) = g(t,θ)

Setting h(x) = P (X = x | T = T (x)) and g(t, θ) = qθ (t) gives the factorization. ✓

3.1. SUFFICIENCY 13

Factorization Theorem: Bernoulli/Binomial

iid
X1 , . . . , Xn ∼ Bernoulli(p). The joint PMF is:
n
Y P P
fp (x) = pxi (1 − p)1−xi = p xi
(1 − p)n− xi

i=1
= pt (1 − p)n−t · |{z}
1 .
| {z }
xi h(x)
P
g(t, p) with t=
P
By the Factorization Theorem, T = Xi is sufficient for p. ✓

Factorization Theorem: Poisson

iid
X1 , . . . , Xn ∼ Poisson(λ). Joint PMF:
n
e−λ λxi P 1
= e−nλ λ
Y
xi
fλ (x) = ·Q .
i=1
xi ! xi !
P
Setting T = Xi :
1
fλ (x) = e| −nλ T
{zλ } · Q xi ! .
g(T,λ) | {z }
h(x)
P
So T = Xi is sufficient for λ.

Factorization Theorem: Normal (both parameters unknown)

iid
X1 , . . . , Xn ∼ N (µ, σ 2 ). Joint PDF:
n
!
1 X
fµ,σ2 (x) = (2πσ 2 )−n/2 exp − 2 (xi − µ)2 .
2σ i=1

(xi − µ)2 =
P 2
xi − 2µ xi + nµ2 . So:
P P
Expand:
P 2 !
xi − 2µ xi + nµ2
P
2 −n/2
fµ,σ2 (x) = (2πσ ) exp − · |{z}
1 .
2σ 2
| P P{z 2 } h(x)
g(( xi , xi ), (µ,σ 2 ) )

Xi2 is a sufficient statistic for (µ, σ 2 ). Equivalently, T = (X̄, S 2 ) is

P P

So T = Xi ,
sufficient.
Special case σ 2 known: T = Xi = nX̄ is sufficient for µ.
P

Factorization Theorem: Uniform(0, θ)

iid
X1 , . . . , Xn ∼ U (0, θ). Joint PDF:
n
Y 1 1   1
fθ (x) = 1(0 < xi < θ) = 1 0 < x(1) , x (n) < θ = n 1(x(n) < θ) · 1(x(1) > 0).
i=1
θ θn θ
14 CHAPTER 3. SUFFICIENCY, COMPLETENESS, AND ANCILLARITY

Here X(n) = maxi Xi . Setting T = X(n) :

1
fθ (x) = n
1(T < θ) · 1(x(1) > 0) .
θ
| {z } | {z }
g(T,θ) h(x)

So X(n) is sufficient for θ.

Intuition: Knowing the maximum tells you the range of θ (it must exceed the max), and
all the data fits uniformly within (0, X(n) ) regardless of θ.

Factorization Theorem: Exponential

iid
X1 , . . . , Xn ∼ Exp(λ) with PDF fλ (x) = λe−λx , x > 0.
n P
λe−λxi = λn e−λ n −λT
Y
xi
fλ (x) = | e{z } · |{z}
=λ 1 ,
i=1 g(T,λ) h(x)

P P
where T = Xi . So Xi (equivalently X̄) is sufficient for λ.

3.2 Exponential Families

Most standard parametric distributions belong to the exponential family, which provides a unified
framework for sufficiency.
k-Parameter Exponential Family

A family {fθ : θ ∈ Θ} is a k-parameter exponential family if the density/PMF can be

written as  
k
X 
fθ (x) = exp Qj (θ)Tj (x) + D(θ) + S(x) ,
 
j=1

where Q1 , . . . , Qk are functions of θ only, T1 , . . . , Tk and S are functions of x only, and

D(θ) is a normalizing function. For a sample of size n, the joint density is
 
Xk n
X n
X 
fθ (x) = exp Qj (θ) Tj (Xi ) + nD(θ) + S(Xi ) .
 
j=1 i=1 i=1

P P
By the Factorization Theorem, T(X) = ( T1 (Xi ), . . . , Tk (Xi )) is a sufficient statistic for
θ.
Exponential Family Examples

Distribution θ Q(θ) T (x) Sufficient stat

p P
Bernoulli(p) p log 1−p x Xi
P
Poisson(λ) λ log λ x X
P i P
N (µ, σ 2 ) (µ, σ 2 ) (µ/σ 2 , −1/2σ 2 ) (x, x2 ) ( Xi , Xi2 )
(α − 1, −1/β)
Q P
Γ(α, β) (α, β) (log x, x) ( Xi , Xi )
(α − 1, β − 1) (log x, log(1 − x)) log(1 − Xi ))
P P
Beta(α, β) (α, β) ( log Xi ,
3.3. COMPLETENESS 15

Non-example: U (0, θ) is NOT an exponential family (support depends on θ).

3.3 Completeness
Completeness is a property of a family of distributions, not of a particular distribution. Intuitively,
a statistic T is complete for a family if the only function of T that is unbiased for zero (for all θ)
is the zero function.
Complete Statistic

A statistic T is complete for the family {Pθ : θ ∈ Θ} if, for every measurable function g:

Eθ [g(T )] = 0 for all θ ∈ Θ =⇒ g(T ) = 0 Pθ -a.s. for all θ.

Equivalently: the only unbiased estimator of 0 that is a function of T is identically 0.

Why completeness matters

Completeness, combined with sufficiency, enables us to uniquely identify the UMVUE

(Chapter 4). The Lehmann-Scheffé theorem says: if T is complete and sufficient, and h(T )
is unbiased for ψ(θ), then h(T ) is the unique UMVUE of ψ(θ).

Completeness of Exponential Families

Let {fθ : θ ∈ Θ} be a k-parameter exponential family with sufficient statistic T =

(T1 (X), . . . , Tk (X)). If the parameter space Θ contains an open set in Rk (the family is of
“full rank”), then T is complete.

Proof Sketch (1-parameter case)

Suppose Eθ [g(T )] = 0 for all θ ∈ Θ = (α, β).

P
In the one-parameter exponential family, the distribution of T = T (Xi ) (the natural
sufficient statistic) has density

fQ (t) = exp{Q · t + C(Q)} h∗ (t),

where Q = Q(θ) ranges over an interval. Then:

Z Z
∗
0 = Eθ [g(T )] = g(t) exp{Qt + C(Q)}h (t) dt = e C(Q)
g(t)h∗ (t)eQt dt.

Since eC(Q) ̸= 0, the function Q 7→ g(t)h∗ (t)eQt dt (the Laplace transform of g(t)h∗ (t))
R

is identically zero on an interval. By the uniqueness of the Laplace transform,

g(t)h∗ (t) = 0 a.e., hence g(T ) = 0 a.s. ✓

Completeness: Poisson

iid
X1 , . . . , Xn ∼ Poisson(λ), λ > 0. The family is one-parameter exponential with T =
P
Xi .
Since λ ranges over (0, ∞), by Theorem 3.3, T = Xi is complete for λ.
P
16 CHAPTER 3. SUFFICIENCY, COMPLETENESS, AND ANCILLARITY

Direct verification: Suppose Eλ [g(T )] = 0 for all λ > 0. T ∼ Poisson(nλ), so:

∞ ∞
(nλ)t g(t)nt t
0 = Eλ [g(T )] = e−nλ = e−nλ
X X
g(t) λ.
t=0
t! t=0
t!

This is a power series in λ that is identically zero, so every coefficient must be zero:
g(t)nt /t! = 0 =⇒ g(t) = 0 for all t = 0, 1, 2, . . . . Hence T is complete.

Incomplete Statistic: U (0, θ) with two-element range

iid
Let X1 , . . . , Xn ∼ U (0, θ) and consider the pair T = (X(1) , X(n) ). Note Eθ [X(n) − X(1) ] =
n−1
n+1 θ and this is not zero for θ > 0.
n−1 X +X
Now let g(X(1) , X(n) ) = X(n) − X(1) − n+1 · (n) 1 (1) ... Actually the classic example:
Eθ [X(n) − X(1) ] = n+1 θ and Eθ [X(n) ] = n+1 θ, so g = (X(n) − X(1) ) − n−1
n−1 n
n · X(n) ...
n−1 n−1 n−1 n
Simpler: Eθ [X(n) − X(1) ] = n+1 θ and Eθ [ n+1 X(n) ] = n+1 · n+1 θ. These are not equal to
zero identically, so the incomplete example must be constructed differently.
The clearest example is this: consider X ∼ U (θ − 12 , θ + 12 ). Then X itself is sufficient and
R θ+1/2
minimal sufficient for θ. But g(x) = sin(2πx) satisfies Eθ [g(X)] = θ−1/2 sin(2πx) dx = 0
for all θ, yet g(x) ̸≡ 0. So X is not complete for this family.

3.4 Ancillary Statistics

Ancillary Statistic

A statistic A(X) is ancillary for θ if its distribution does not depend on θ:

Pθ (A(X) ∈ B) is the same for all θ ∈ Θ, for all B.

Intuition

An ancillary statistic is “parameter-free”: it contains no information about θ in a distribu-

tional sense. However (Basu’s theorem), it can still be useful — if it’s independent of a
complete sufficient statistic, it can be used to improve estimation efficiency.

Ancillary Statistics

iid
1. X1 , . . . , Xn ∼ N (µ, 1): A = (n − 1)S 2 = (Xi − X̄)2 ∼ χ2 (n − 1), distribution free
P

of µ. So S 2 is ancillary for µ.
iid
2. X1 , . . . , Xn ∼ N (0, σ 2 ): A = X̄/S ∼ t(n − 1), free of σ 2 .
iid
3. X1 , . . . , Xn ∼ f (x−θ) (location family): A = (X1 −Xn , . . . , Xn−1 −Xn ) (differences)
is ancillary.
iid
4. X1 , . . . , Xn ∼ from PDF f (x/θ)/θ (scale family): A = X1 /Xn , X2 /Xn , . . . (ratios)
are ancillary.
3.5. MINIMAL SUFFICIENT STATISTICS 17

3.5 Minimal Sufficient Statistics

A sufficient statistic compresses the data without losing information about θ. But some sufficient
statistics are more parsimonious than others.

Minimal Sufficient Statistic

A sufficient statistic T is minimal sufficient if it is a function of every other sufficient

statistic. That is, for any sufficient statistic T ′ , there exists a function g such that T = g(T ′ )
a.s.
Equivalently, T achieves the maximum possible data reduction (to the coarsest partition
of the sample space that is still sufficient).

Lehmann-Scheffé Criterion for Minimal Sufficiency

Suppose the family has PDF/PMF fθ . Define x ∼ y (likelihood equivalent) if the ratio
fθ (x)/fθ (y) does not depend on θ. Then any statistic T that is constant on equivalence
classes of ∼ (and takes different values on different classes) is minimal sufficient.
Practical criterion: T (X) is minimal sufficient if T (x) = T (y) if and only if fθ (x)/fθ (y)
is free of θ.

Minimal Sufficient Statistic: Normal with Known Variance

iid
X1 , . . . , Xn ∼ N (µ, 1).

fµ (x) 1X 2
 X 
= exp − (xi − yi2 ) + µ (xi − yi ) .
fµ (y) 2
P P P
This is free of µ iff xi = yi iff x̄ = ȳ. So T = X̄ (equivalently Xi ) is minimal
sufficient for µ.

Minimal Sufficient Statistic: Normal with Both Parameters Unknown

iid
X1 , . . . , Xn ∼ N (µ, σ 2 ).
P 2 P 2 !
fµ,σ2 (x) xi − yi xi −
P P
µ( yi )
= exp − + .
fµ,σ2 (y) 2σ 2 σ2

This is free of (µ, σ 2 ) iff x2i = yi2 AND Xi2 ), equiva-

P P P P P P
xi = yi . So T = ( Xi ,
lently (X̄, S 2 ), is minimal sufficient.

3.6 Basu’s Theorem

Basu’s Theorem

If S(X) is a complete sufficient statistic for θ, then S is independent of every ancillary

statistic A(X).
18 CHAPTER 3. SUFFICIENCY, COMPLETENESS, AND ANCILLARITY

Proof

Let A be ancillary, so Pθ (A ∈ B) = p(B) for some function p free of θ. Define g(S) =

Pθ (A ∈ B | S) − p(B).
We claim Eθ [g(S)] = 0 for all θ:

Eθ [g(S)] = Eθ [Pθ (A ∈ B | S)] − p(B)

= Pθ (A ∈ B) − p(B) (tower property)
= p(B) − p(B) = 0.

Since S is complete, Eθ [g(S)] = 0 for all θ implies g(S) = 0 a.s., i.e.,

Pθ (A ∈ B | S) = p(B) = Pθ (A ∈ B) a.s.

This is the definition of independence: A ⊥⊥ S.

Basu’s Theorem in Action: Normal

iid
X1 , . . . , Xn ∼ N (µ, σ 2 ) with σ 2 known.

• X̄ is complete sufficient for µ (exponential family, range is all of R).

1
• S2 = (Xi − X̄)2 is ancillary for µ ((n − 1)S 2 /σ 2 ∼ χ2 (n − 1), free of µ).
P
n−1

By Basu’s theorem: X̄ ⊥ ⊥ S2.

This is the key fact behind the t-test! The test statistic t = X̄−µ √ 0 has X̄ in the
S/ n
numerator and S in the denominator. Since they’re independent (Basu) and each has a
known distribution (X̄ ∼ N (µ, σ 2 /n), S 2 ∼ σ 2 χ2 (n − 1)/(n − 1)), the ratio has a t(n − 1)
distribution.

Basu’s Theorem: Exponential Distribution

iid
X1 , . . . , Xn ∼ Exp(θ).
P
• T = Xi is complete sufficient for θ.

Xj is ancillary (it has a Beta(1, n − 1) distribution, free of θ).

P
• A = X1 /

Xi ⊥
⊥ X1 /
P P
By Basu’s theorem: Xj .
Chapter 4

UMVUE Theory: Rao-Blackwell and

Lehmann-Scheffé

How This Chapter Fits Into the Course

We have established what it means to be unbiased. But among all unbiased estimators, how
do we find the best one (minimum variance)? This chapter answers that question completely
through two landmark theorems:

1. Rao-Blackwell: Given any unbiased estimator, we can systematically improve it (or at

least not worsen it) by conditioning on a sufficient statistic.

2. Lehmann-Scheffé: If a complete sufficient statistic exists, then conditioning on it gives

the unique UMVUE.

The strategy for finding UMVUEs is therefore:

Find a complete sufficient statistic T −→ Find any unbiased estimator h(X) −→ Compute E[h(X) | T ] =

4.1 The UMVUE

UMVUE

An unbiased estimator T0 of ψ(θ) is a Uniformly Minimum Variance Unbiased

Estimator (UMVUE) if

Var([) θ]T0 ≤ Var([) θ]T for all θ ∈ Θ

for every unbiased estimator T of ψ(θ).

Uniqueness of UMVUE

The UMVUE, if it exists, is unique (almost surely).

Proof

T0 +T1
Suppose T0 and T1 are both UMVUEs of ψ(θ). Then 2 is also unbiased, and by the

19
20 CHAPTER 4. UMVUE THEORY: RAO-BLACKWELL AND LEHMANN-SCHEFFÉ

parallelogram law:
T0 + T1 1
 
Var = Var(T0 + T1 )
2 4
1
= (Var(T0 ) + 2Cov(T0 , T1 ) + Var(T1 ))
4
1 q 
≤ Var(T0 ) + 2 Var(T0 ) Var(T1 ) + Var(T1 ) (Cauchy-Schwarz)
4
2
1 q
 q
= Var(T0 ) + Var(T1 ) .
4

Since T0 is UMVUE and (T0 + T1 )/2 is unbiased: Var(T0 ) ≤ Var((T0 + T1 )/2). Since T1 is
UMVUE: Var(T1 ) ≤ Var(T0 ), and by symmetry Var(T p 0 ) ≤ Var(T1 ), so Var(T0 ) = Var(T1 ).
The Cauchy-Schwarz inequality Cov(T0 , T1 ) ≤ Var(T0 ) Var(T1 ) holds with equality iff
T1 = aT0 + b a.s. for some constants. Since both are unbiased for ψ(θ): b = (1 − a)ψ(θ)
for all θ. If ψ is non-constant, this forces a = 1, b = 0, so T0 = T1 a.s.

4.2 The Rao-Blackwell Theorem

Rao-Blackwell Theorem

Let T be a sufficient statistic for θ and let h(X) be any unbiased estimator of ψ(θ) with
Eθ [h2 (X)] < ∞. Define
ϕ(T ) = Eθ [h(X) | T ].
(Note: ϕ(T ) is free of θ by sufficiency, so this is well-defined.) Then:

1. ϕ(T ) is unbiased for ψ(θ).

2. Var([) θ]ϕ(T ) ≤ Var([) θ]h(X) for all θ ∈ Θ.

3. Equality holds iff h(X) = ϕ(T ) a.s.

Proof

(1) Unbiasedness: By the tower property of conditional expectation:

Eθ [ϕ(T )] = Eθ [Eθ [h(X) | T ]] = Eθ [h(X)] = ψ(θ). ✓

(2) Variance reduction: By the law of total variance:

Var(h(X)) = Eθ [Var(h(X) | T )] + Var([) θ]Eθ [h(X) | T ] = Eθ [Var(h | T )] +Var(ϕ(T )) .

| {z }
≥0

Therefore Var(ϕ(T )) ≤ Var(h(X)). ✓

(3) Equality holds iff Eθ [Var(h(X) | T )] = 0, i.e., Var(h(X) | T ) = 0 a.s., meaning h is a
function of T a.s., i.e., h(X) = ϕ(T ) a.s.
4.3. THE LEHMANN-SCHEFFÉ THEOREM 21

What Rao-Blackwell Says Practically

The theorem tells you: any unbiased estimator that is not already a function of the sufficient
statistic can be improved by conditioning on that statistic. After Rao-Blackwellization,
you need only consider estimators that are functions of the sufficient statistic.

Rao-Blackwell: Poisson
iid
X1 , . . . , Xn ∼ Poisson(λ). Estimate ψ(λ) = e−λ = P (X = 0).
Naive estimator: h(X) = 1(X1 = 0). Then E[h] = P (X1 = 0) = e−λ , so h is unbiased.
But Var(h) = e−λ (1 − e−λ ) is large.
Sufficient statistic: T = Xi ∼ Poisson(nλ).
P

Rao-Blackwell improvement:
Pn
ϕ(T ) = E[h(X) | T ] = P (X1 = 0 | i=1 Xi = T ).

Given Xi = t, we need P (X1 = 0 | Xi = t).

P P

Since X1 ∼ Poisson(λ) and ni=2 Xi ∼ Poisson((n − 1)λ) are independent:

P

P (X1 = 0, ni=2 Xi = t)
P
Pn
P (X1 = 0 | i=1 Xi = t) =
P ( ni=1 Xi = t)
P

t
e−λ · e−(n−1)λ ((n−1)λ)
t!
= t
e−nλ (nλ)
t!
t
(n − 1)t λt n−1

= = .
nt λt n
 T  P Xi
1 n−1
So ϕ(T ) = 1 − n = n .
 T 
n−1
Verification: Eλ [ϕ(T )] = E n where T ∼ Poisson(nλ). Using the MGF of the
s −1)
Poisson: E[esT ] = enλ(e , so with s = log n−1
n :
" T #
n−1 n−1
E = enλ( n
−1 ) = enλ·(−1/n) = e−λ . ✓
n

This is the UMVUE of e−λ (we’ll confirm via Lehmann-Scheffé).

4.3 The Lehmann-Scheffé Theorem

Lehmann-Scheffé Theorem

Suppose T is a complete sufficient statistic for θ, and suppose ϕ(T ) is an unbiased

estimator of ψ(θ) that is a function of T . Then ϕ(T ) is the unique UMVUE of ψ(θ).

Proof

Step 1: ϕ(T ) has minimum variance among all unbiased estimators.

Let h(X) be any unbiased estimator of ψ(θ). By Rao-Blackwell, ϕ∗ (T ) = E[h(X) | T ] is
unbiased and Var(ϕ∗ ) ≤ Var(h).
22 CHAPTER 4. UMVUE THEORY: RAO-BLACKWELL AND LEHMANN-SCHEFFÉ

Now ϕ∗ (T ) − ϕ(T ) is a function of T and:

Eθ [ϕ∗ (T ) − ϕ(T )] = ψ(θ) − ψ(θ) = 0 for all θ.

Since T is complete, ϕ∗ (T ) − ϕ(T ) = 0 a.s., i.e., ϕ∗ (T ) = ϕ(T ) a.s.

Therefore Var(ϕ(T )) = Var(ϕ∗ (T )) ≤ Var(h(X)) for all unbiased h. So ϕ(T ) is UMVUE.
✓
Step 2: Uniqueness. If ϕ1 (T ) and ϕ2 (T ) are both unbiased functions of T , then
ϕ1 (T ) − ϕ2 (T ) is a function of T with Eθ [ϕ1 − ϕ2 ] = 0. By completeness, ϕ1 = ϕ2 a.s.

The Complete Workflow

To find the UMVUE of ψ(θ):

1. Find a complete sufficient statistic T (usually via exponential family).

2. Find any function ϕ(T ) that is unbiased for ψ(θ). — Option A: Guess directly. —
Option B: Take a simple unbiased estimator h and compute E[h | T ].

3. By Lehmann-Scheffé, ϕ(T ) is the UMVUE.

UMVUE: Normal Distribution

iid
X1 , . . . , Xn ∼ N (µ, σ 2 ), both parameters unknown.
Complete sufficient statistic: T = (X̄, S 2 ) (from exponential family structure).
UMVUE of µ: X̄ is unbiased and a function of T . By Lehmann-Scheffé, X̄ is the
UMVUE of µ.
UMVUE of σ 2 : S 2 is 2 2 2
√ unbiased for σ and a function of T . So S is the UMVUE of σ .
2 2
UMVUE of σ: S = S is not unbiased for σ. Since (n − 1)S /σ ∼ χ (n − 1): 2 2

s
2 Γ(n/2)
E[S] = σ · .
n − 1 Γ((n − 1)/2)
q
2 Γ(n/2)
So the UMVUE of σ is S/c(n) where c(n) = n−1 · Γ((n−1)/2) .  
UMVUE of µ2 : We need an unbiased function of (X̄, S 2 ). E[X̄ 2 ] = Var X̄ + µ2 =
σ 2 /n + µ2 , so X̄ 2 − S 2 /n is unbiased for µ2 . By Lehmann-Scheffé: UMVUE of µ2 is
X̄ 2 − S 2 /n.

UMVUE: Bernoulli/Binomial

iid
X1 , . . . , Xn ∼ Bernoulli(p). CSS: T = Xi ∼ Binomial(n, p).
P

UMVUE of p: X̄ = T /n is unbiased. By Lehmann-Scheffé: X̄ is UMVUE.

 
UMVUE of p(1 − p): E[X̄(1 − X̄)] — let’s compute: E[X̄ 2 ] = Var X̄ + p2 = p(1 −
p)/n + p2 . So E[nX̄(1 − X̄)/(n − 1)] = ? Actually:

T (n − T ) 1 1
     
E = n · np − E[T 2 ] = n2 p − np(1 − p) − n2 p2 = p(1−p).
n(n − 1) n(n − 1) n(n − 1)
T (n−T )
So the UMVUE of p(1 − p) is n(n−1) .
4.3. THE LEHMANN-SCHEFFÉ THEOREM 23

UMVUE: Poisson — Estimating e−λ

From the Rao-Blackwell example above:

 T
Xi ∼ Poisson(nλ) is CSS. We found ϕ(T ) = n−1
is unbiased for e−λ .
P
T = n
By Lehmann-Scheffé: UMVUE of e−λ = P (X = 0) is
Pn Xi
1

i=1
P̂ (X = 0) = 1 − .
n

UMVUE: Exponential Distribution

iid
X1 , . . . , Xn ∼ Exp(θ) (mean θ). CSS: T = Xi ∼ Γ(n, θ).
P

UMVUE of θ (the mean): X̄ = T /n is unbiased. UMVUE: X̄.

UMVUE of the CDF F (θ, x0 ) = P (X > x0 ) = e−x0 /θ for fixed x0 :
Naive estimator: h(X) = 1(X1 > x0 ). Then E[h] = P (X1 > x0 ) = e−x0 /θ . Rao-
Blackwellize:

ϕ(T ) = E[1(X1 > x0 ) |

P
Xi = T ] .

Since X1 |
P
Xi = t has distribution Beta-related, one can show:
x0
n−1
(
1− T if T > x0
ϕ(T ) =
0 if T ≤ x0 .

This is the UMVUE of e−x0 /θ .

24 CHAPTER 4. UMVUE THEORY: RAO-BLACKWELL AND LEHMANN-SCHEFFÉ
Chapter 5

Fisher Information and the

Cramér-Rao Lower Bound

How This Chapter Fits Into the Course

The Rao-Blackwell + Lehmann-Scheffé approach finds the UMVUE but requires a complete
sufficient statistic. What if none exists? More fundamentally, what is the smallest variance any
unbiased estimator can possibly have?
The Cramér-Rao lower bound (CRLB) answers this. It provides a universal lower bound
for the variance of any unbiased estimator, in terms of the Fisher information — a quantity
that measures how much “information” the data contains about θ.
If an unbiased estimator achieves the CRLB, it is called efficient. Often the UMVUE is also
efficient, but not always.

5.1 The Score Function

Score Function

For a family with log-density ℓ(θ; x) = log fθ (x), the score function is

∂ f ′ (x)
s(θ; x) = log fθ (x) = θ .
∂θ fθ (x)
∂ Pn
For a sample X, the score is s(θ; X) = ∂θ log fθ (X) = i=1 s(θ; Xi ) (for iid data).

Score Has Mean Zero

Under regularity conditions (that allow differentiation under the integral):

Eθ [s(θ; X)] = 0 for all θ.

Proof
R
We use the fact that fθ (x) dx = 1 for all θ. Differentiating both sides with respect to θ

25
26 CHAPTER 5. FISHER INFORMATION AND THE CRAMÉR-RAO LOWER BOUND

(justifying the interchange):

d ∂fθ (x)
Z Z
0= fθ (x) dx = dx
dθ ∂θ
∂ log fθ (x)
Z
= fθ (x) · dx
∂θ
∂ log fθ (X)
 
= Eθ = Eθ [s(θ; X)]. ✓
∂θ

5.2 Fisher Information

Fisher Information

The Fisher information in a single observation X is

" 2 #
∂ log fθ (X)
I(θ) = Eθ = Eθ [s(θ; X)2 ] = Var([) θ]s(θ; X),
∂θ

(the last equality holds since E[s] = 0, so Var(s) = E[s2 ]).

Alternative Formula for Fisher Information

Under regularity conditions allowing differentiation under the integral twice:

" #
∂ 2 log fθ (X)
I(θ) = −Eθ .
∂θ2

Proof

Differentiate Eθ [s(θ; X)] = 0 again with respect to θ:

d
Z
0= s(θ; x)fθ (x) dx
dθ
∂s(θ; x) ∂fθ (x)
Z Z
= fθ (x) dx + s(θ; x) dx
"
∂θ #
∂θ
∂ 2 log fθ (X) h
2
i
= Eθ + Eθ s(θ; X)
∂θ2
" #
∂ 2 log fθ (X)
= Eθ + I(θ).
∂θ2

∂ 2 log fθ (X)
h i
Therefore I(θ) = −Eθ ∂θ2
.

Fisher Information is Additive for iid Samples

iid
If X1 , . . . , Xn ∼ fθ , the Fisher information in the entire sample X is

In (θ) = n I(θ).
5.2. FISHER INFORMATION 27

Proof
Pn Pn
The log-likelihood is log fθ (X) = i=1 log fθ (Xi ). The score is sn (θ; X) = i=1 s(θ; Xi ).
Since the Xi are iid:
n n
!
X X
In (θ) = Var(sn ) = Var s(θ; Xi ) = Var(s(θ; Xi )) = nI(θ).
i=1 i=1

(Independence gives additive variances.) ✓

Fisher Information: Bernoulli(p)

fp (x) = px (1 − p)1−x , so log fp (x) = x log p + (1 − x) log(1 − p).

∂ log fp x 1−x x−p

= − = .
∂p p 1−p p(1 − p)
" 2 #
X −p E[(X − p)2 ] p(1 − p) 1
I(p) = Ep = 2 2
= 2 2
= .
p(1 − p) p (1 − p) p (1 − p) p(1 − p)

Fisher Information: Poisson(λ)

log fλ (x) = x log λ − λ − log(x!).

∂ log fλ x x−λ
= −1= .
∂λ λ λ
∂ 2 log fλ
Using the alternative formula: ∂λ2
= − λx2 .

X E[X] λ 1
 
I(λ) = −E − 2
= 2
= 2 = .
λ λ λ λ

Fisher Information: N (µ, σ 2 ), σ 2 known

2
log fµ (x) = − (x−µ)
2σ 2
+ const.

∂ log fµ x−µ ∂ 2 log fµ 1

= , = − 2.
∂µ σ2 ∂µ 2 σ

1 1
 
I(µ) = −E − 2 = 2 .
σ σ
Fisher information for n observations: In (µ) = n/σ 2 .
28 CHAPTER 5. FISHER INFORMATION AND THE CRAMÉR-RAO LOWER BOUND

5.3 The Cramér-Rao Lower Bound

Cramér-Rao Lower Bound (CRLB)

iid
Let X1 , . . . , Xn ∼ fθ satisfying regularity conditions. Let T (X) be any unbiased estimator
of ψ(θ) with Eθ [T 2 ] < ∞. Then

[ψ ′ (θ)]2
Var([) θ]T (X) ≥ .
n I(θ)

When ψ(θ) = θ (estimating θ itself):

1
Var([) θ]T (X) ≥ .
n I(θ)

Full Proof of the CRLB

Key idea: We will use the Cauchy-Schwarz inequality applied to the covariance of T (X)
and the score function sn (θ; X).
Step 1: Compute Covθ (T (X), sn (θ; X)).
Since Eθ [sn ] = 0 and Eθ [T ] = ψ(θ):

Cov(T, sn ) = Eθ [T (X) · sn (θ; X)] − Eθ [T ] · Eθ [sn ]

∂
 
= Eθ T (X) · log fθ (X) .
∂θ
∂ ∂fθ (X)/∂θ
Now ∂θ log fθ (X) = fθ (X) , so:

∂fθ (x)/∂θ
Z
Cov(T, sn ) = T (x) · · fθ (x) dx
fθ (x)
∂fθ (x)
Z
= T (x) · dx
∂θ
∂
Z Z
= T (x)fθ (x) dx (interchange ∂/∂θ and )
∂θ
d d
= Eθ [T (X)] = ψ(θ) = ψ ′ (θ).
dθ dθ
Step 2: Apply Cauchy-Schwarz.
The Cauchy-Schwarz inequality states: [Cov(T, sn )]2 ≤ Var(T ) · Var(sn ).
We know:

• Cov(T, sn ) = ψ ′ (θ), so [Cov(T, sn )]2 = [ψ ′ (θ)]2 .

• Var(sn ) = In (θ) = nI(θ).

Therefore:
[ψ ′ (θ)]2 ≤ Var(T ) · nI(θ),
which gives
[ψ ′ (θ)]2
Var(T ) ≥ . ✓
nI(θ)
5.3. THE CRAMÉR-RAO LOWER BOUND 29

Efficient Estimator
′ 2
An unbiased estimator T is efficient if it attains the CRLB: Var(T ) = [ψnI(θ)
(θ)]
.
An efficient estimator is automatically the UMVUE (since no unbiased estimator can have
smaller variance).

Condition for Attaining the CRLB

The Cauchy-Schwarz bound is attained (with equality) if and only if the score function is
a linear function of T :
∂ log fθ (X)

sn (θ; X) = = c(θ) T (X) − ψ(θ)
∂θ
for some function c(θ) not depending on X.

Proof

Cauchy-Schwarz holds with equality iff T − E[T ] and sn − E[sn ] = sn are proportional:
T − ψ(θ) = k · sn for some k possibly depending on θ. Solving for sn gives the result.

CRLB for Normal Distribution

iid
X1 , . . . , Xn ∼ N (µ, σ 2 ), σ 2 known, estimate µ.
1 σ2
I(µ) = 1/σ 2 , so CRLB= n/σ 2 = n .
 
X̄ achieves this: Var X̄ = σ 2 /n. So X̄ is efficient.
Check the condition:
n(X̄ − µ) n
sn (µ; X) = 2
= 2 (X̄ − µ).
σ σ
Indeed sn = c(µ)(X̄ − µ) with c(µ) = n/σ 2 . ✓

CRLB for Poisson

iid
X1 , . . . , Xn ∼ Poisson(λ), estimate λ.
I(λ) = 1/λ, so CRLB= n/λ 1
= nλ .
 
X̄ has E[X̄] = λ, Var X̄ = λ/n. So X̄ is efficient.

CRLB for Bernoulli

iid
X1 , . . . , Xn ∼ Bernoulli(p), estimate p.
1 p(1−p)
I(p) = 1/[p(1 − p)], so CRLB= n/[p(1−p)] = n .
X̄ has variance p(1 − p)/n. So X̄ is efficient.

CRLB Not Attained: Uniform(0, θ)

iid
X1 , . . . , Xn ∼ U (0, θ).
The Uniform(0, θ) family does not satisfy regularity conditions (the support depends on
30 CHAPTER 5. FISHER INFORMATION AND THE CRAMÉR-RAO LOWER BOUND

θ), so the CRLB does not apply!

The MLE is X(n) with bias −θ/(n + 1). The UMVUE is n+1 n X(n) (we’ll verify this later).
This is an important example: always check regularity conditions before applying the
CRLB.

CRLB for a Function of the Parameter

iid
X1 , . . . , Xn ∼ Poisson(λ), estimate ψ(λ) = e−λ .
I(λ) = 1/λ, ψ ′ (λ) = −e−λ , so

[−e−λ ]2 λe−2λ
CRLB = = .
n/λ n

The UMVUE was ϕ(T ) = (1 − 1/n)T . Can compute Var(ϕ(T )) and check whether it
equals the CRLB. In general, for this problem, the UMVUE does not achieve the CRLB —
the CRLB is not always tight.
Chapter 6

Maximum Likelihood Estimation

How This Chapter Fits Into the Course

The Method of Moments (Chapter 2) gives estimators by matching moments. But is there a more
principled approach? Maximum Likelihood Estimation (MLE) says: choose the parameter
value that makes the observed data most probable. This is the most widely used estimation
method in statistics.
MLEs have three remarkable properties:
• Consistency: as n → ∞, the MLE converges in probability to θ.
√ d
• Asymptotic normality: n(θ̂M LE − θ) → N (0, 1/I(θ)).
• Invariance: the MLE of g(θ) is g(θ̂M LE ).
The asymptotic variance 1/I(θ) is exactly the CRLB — so MLEs are asymptotically efficient.

6.1 The Likelihood Function and MLE

Likelihood Function and MLE

Given observed data x = (x1 , . . . , xn ), the likelihood function is

n
Y
L(θ; x) = fθ (x) = fθ (xi ).
i=1

(It is the joint density viewed as a function of θ with x fixed.)

The log-likelihood is ℓ(θ; x) = log L(θ; x) = ni=1 log fθ (xi ).
P

The Maximum Likelihood Estimator (MLE) is

θ̂M LE = arg max L(θ; X) = arg max ℓ(θ; X).

θ∈Θ θ∈Θ

Practical Strategy for Finding MLEs

P
1. Write the log-likelihood ℓ(θ) = log fθ (xi ).
2. If the domain doesn’t depend on θ: differentiate, set to zero, solve. Check the second
derivative (or boundary) to confirm it’s a maximum.
3. If the domain depends on θ (like Uniform): maximize directly by inspecting where
L is maximized — often at a boundary.

31
32 CHAPTER 6. MAXIMUM LIKELIHOOD ESTIMATION

6.2 Finding MLEs: Full Worked Examples

MLE: Bernoulli(p)

iid
X1 , . . . , Xn ∼ Bernoulli(p).
n
X
ℓ(p) = [xi log p + (1 − xi ) log(1 − p)] = t log p + (n − t) log(1 − p),
i=1
P
where t = xi .
Differentiating:
dℓ t n−t t
= − = 0 =⇒ t(1 − p) = (n − t)p =⇒ t = np =⇒ p̂ = = x̄.
dp p 1−p n
d ℓ 2t n−t
Second derivative: dp 2 = − p2 − (1−p)2 < 0 (when 0 < t < n). So p̂ = X̄ is the MLE.

Note: MLE = MoM = UMVUE for Bernoulli.

MLE: Poisson(λ)

iid
X1 , . . . , Xn ∼ Poisson(λ).
n
X
ℓ(λ) = [xi log λ − λ − log(xi !)] = t log λ − nλ − C,
i=1
P
where t = xi and C is constant in λ.

dℓ t t
= − n = 0 =⇒ λ̂ = = x̄.
dλ λ n
d2 ℓ
dλ2
= −t/λ2 < 0. So λ̂ = X̄ is the MLE.

MLE: Normal( µ, σ 2 ) — Both Unknown

iid
X1 , . . . , Xn ∼ N (µ, σ 2 ).
n
n n 1 X
ℓ(µ, σ 2 ) = − log(2π) − log σ 2 − 2 (xi − µ)2 .
2 2 2σ i=1

Differentiating with respect to µ:

∂ℓ 1 X
= 2 (xi − µ) = 0 =⇒ µ̂ = x̄. (for any σ 2 )
∂µ σ
Differentiating with respect to σ 2 (let v = σ 2 ):
(xi − µ)2 (xi − µ)2
P P
∂ℓ n
=− + = 0 =⇒ v̂ = .
∂v 2v 2v 2 n
Plugging in µ̂ = x̄:
n
2 1X
σ̂M LE = (xi − x̄)2 .
n i=1
2
Note: σ̂M LE is biased (divides by n not n − 1). The MLE is not always unbiased! The
bias is −σ 2 /n → 0 as n → ∞ (asymptotically unbiased).
6.3. PROPERTIES OF THE MLE 33

MLE: Uniform(0, θ) — Boundary Case

iid
X1 , . . . , Xn ∼ U (0, θ).
L(θ; x) = θ1n ni=1 1(0 < xi < θ) = 1
Q
θn 1(θ > x(n) ).
So for fixed x: (
θ−n if θ > x(n)
L(θ) =
0 if θ ≤ x(n) .
The function θ−n is decreasing in θ for θ > 0. So L(θ) is maximized at the smallest possible
θ, which is θ = x(n) = maxi xi .

θ̂M LE = X(n) = max(X1 , . . . , Xn ).

Properties:

nxn−1
PDF of X(n) : fX(n) (x; θ) = , 0 < x < θ.
θn
n
E[X(n) ] = θ,
n+1
n θ
Bias: = θ−θ =− < 0. (biased)
n+1 n+1
n+1
The UMVUE is n X(n) (unbiased), but the MLE is X(n) (biased but consistent).

MLE: Exponential(λ)

iid
X1 , . . . , Xn ∼ Exp(λ) with PDF λe−λx .
X
ℓ(λ) = n log λ − λ xi .

dℓ n X n 1
= − xi = 0 =⇒ λ̂ = P = .
dλ λ Xi X̄
Note: E[λ̂] = E[1/X̄] ̸= 1/θ in general (MLE can be biased). The MLE of the mean
θ = 1/λ is X̄ by invariance.

6.3 Properties of the MLE

6.3.1 Invariance Property

Invariance of the MLE

If θ̂ is the MLE of θ, then for any function g, the MLE of g(θ) is g(θ̂).

Proof

Define the induced likelihood: L∗ (g(θ)) = supθ:g(θ)=ϕ L(θ). The MLE of ϕ = g(θ) maximizes
L∗ (ϕ) over all ϕ.
Since g(θ̂) achieves this maximum (L∗ (g(θ̂)) = L(θ̂) ≥ L(θ) for all θ), the MLE of g(θ) is
g(θ̂).
34 CHAPTER 6. MAXIMUM LIKELIHOOD ESTIMATION

Invariance: Normal MLE

1
From the Normal example: µ̂ = X̄, σ̂ 2 = (Xi − X̄)2 .
P
n
By invariance:
√
• MLE of σ is σ̂ = σ̂ 2 .

• MLE of µ2 is X̄ 2 .
   
c−µ c−X̄
• MLE of P (X ≤ c) = Φ σ is Φ σ̂ .

6.3.2 Asymptotic Properties

Asymptotic Normality of MLE

Under regularity conditions, the MLE satisfies:

√ 1
 
d
n(θ̂M LE − θ) −→ N 0, .
I(θ)
 
1
Equivalently: θ̂M LE ≈ N θ, nI(θ) for large n.

Sketch

The MLE solves the score equation ℓ′ (θ̂) = 0. Taylor-expand ℓ′ around the true θ0 :

0 = ℓ′ (θ̂) ≈ ℓ′ (θ0 ) + (θ̂ − θ0 )ℓ′′ (θ0 ).

′
Solving: θ̂ − θ0 ≈ − ℓℓ′′(θ0)
(θ0 ) .
d
√1 ℓ′ (θ0 ) = √1 s(θ0 ; Xi ) → N (0, I(θ0 )).
P
By the CLT: n n
1 ′′ P
By LLN: n ℓ (θ0 ) −→ E[ℓ′′ (θ0 )/n] = −I(θ0 ).
Therefore:

√ √1 ℓ′ (θ0 )
N (0, I(θ0 )) 1
 
n d
n(θ̂ − θ0 ) ≈ − 1 ′′ → − = N 0, . ✓
n ℓ (θ0 )
−I(θ0 ) I(θ0 )

Why This Matters

• The asymptotic variance 1/(nI(θ)) equals the CRLB. So the MLE is asymptotically
efficient — no unbiased estimator can have lower asymptotic variance.
q
• This gives approximate confidence intervals: θ̂ ± zα/2 nI(θ̂).

6.4 MLEs in Exponential Families

In the exponential family, the MLE has a particularly clean form.
6.4. MLES IN EXPONENTIAL FAMILIES 35

MLE in Exponential Family

For a one-parameter exponential family fθ (x) = exp{Q(θ)T (x) + D(θ) + S(x)}, the MLE
θ̂ satisfies: hX i X
Eθ̂ T (Xi ) = T (xi ),

i.e., the expected value of the sufficient statistic under θ̂ equals its observed value. The
MLE is a function of the sufficient statistic.

MLE Summary Table

Distribution θ̂M LE Unbiased? Sufficient stat

P
Bernoulli(p) X̄ Yes X
P i
Poisson(λ) X̄ Yes X
P i
N (µ, σ 2 ) known σ 2 X̄ Yes Xi
N (µ, σ 2 ) unknown σ 2 X̄, n1 (Xi − X̄)2 µ:Yes, σ 2 :No ( Xi , Xi2 )
P P P
P
Exponential(1/θ) (mean θ) X̄ Yes Xi
Uniform(0, θ) X(n) No X(n)
36 CHAPTER 6. MAXIMUM LIKELIHOOD ESTIMATION
Chapter 7

Bayes and Minimax Estimation

How This Chapter Fits Into the Course

Frequentist estimation (MoM, MLE, UMVUE) treats θ as an unknown fixed constant. Bayesian
estimation treats θ as a random variable with a prior distribution π(θ) reflecting our beliefs
before observing data. After observing X = x, we update to the posterior π(θ|x).
This Bayesian framework gives us estimators that minimize expected loss (risk), and connects
to minimax estimation which seeks to minimize the worst-case risk.

7.1 The Bayesian Framework

Prior, Likelihood, Posterior

• Prior: π(θ) — our distribution for θ before data.

f (xi |θ) — the model.

Q
• Likelihood: f (x|θ) =

• Posterior: π(θ|x) ∝ f (x|θ) · π(θ) — updated distribution of θ given data.

By Bayes’ theorem:

f (x|θ) π(θ)
Z
π(θ|x) = , m(x) = f (x|θ)π(θ) dθ.
m(x)

Loss, Risk, and Bayes Estimator

A loss function L(θ, δ) measures the cost of using estimate δ when the true value is θ.
The risk (frequentist) is R(θ, δ) = Eθ [L(θ, δ(X))].
The Bayes risk is r(π, δ) = Eπ [R(θ, δ)] = Eπ Eθ [L(θ, δ)].
The Bayes estimator minimizes the Bayes risk, equivalently minimizes the posterior
expected loss Eθ [L(θ, δ)|X = x] for each x.

Bayes Estimator under Squared Error Loss

Under L(θ, δ) = (θ − δ)2 (squared error loss), the Bayes estimator is the posterior mean:

θ̂Bayes = E[θ|X].

37
38 CHAPTER 7. BAYES AND MINIMAX ESTIMATION

Proof

We minimize the posterior expected loss:

h i h i
Eθ (θ − δ)2 | X = x = E θ2 |x − 2δ E[θ|x] + δ 2 .

Differentiating with respect to δ and setting to zero:

−2E[θ|x] + 2δ = 0 =⇒ δ = E[θ|x]. ✓

The second derivative is +2 > 0, confirming this is a minimum.

Other Loss Functions

• Absolute error L = |θ − δ|: Bayes estimator = posterior median.

• 0-1 loss L = 1(θ ̸= δ): Bayes estimator = posterior mode.

7.2 Conjugate Priors

A prior π(θ) is conjugate to the likelihood f (x|θ) if the posterior π(θ|x) belongs to the same
family as the prior. Conjugate priors make Bayes computation tractable.
Beta-Binomial Conjugacy

iid
X1 , . . . , Xn ∼ Bernoulli(p), prior p ∼ Beta(α, β).
Prior: π(p) ∝ pα−1 (1 −P p)β−1 . P
Likelihood: f (x|p) = p xi (1 − p)n− xi = pt (1 − p)n−t .
Posterior:
π(p|x) ∝ pα+t−1 (1 − p)β+n−t−1 ,
which is Beta(α + t, β + n − t).
Bayes estimator (posterior mean):
α+t
p̂Bayes = E[p|x] = .
α+β+n
Interpretation: Prior contributes α “pseudo-successes” and β “pseudo-failures”. As n → ∞,
p̂Bayes → x̄ = p̂M LE .

Gamma-Poisson Conjugacy

iid
X1 , . . . , Xn ∼ Poisson(λ), prior λ ∼ Γ(α, β) (mean αβ).
Prior: π(λ) ∝ λα−1 e−λ/β . P
Likelihood: f (x|λ) = e−nλ λ xi .
Posterior: P
π(λ|x) ∝ λα+ xi −1 e−λ(n+1/β) ,
   
P 1 β
which is Γ α + xi , n+1/β = Γ α + t, nβ+1 .
Bayes estimator (posterior mean):
(α + t)β αβ + nx̄β
λ̂Bayes = = → x̄ as n → ∞.
nβ + 1 nβ + 1
7.3. MINIMAX ESTIMATION 39

Normal-Normal Conjugacy

iid
X1 , . . . , Xn ∼ N (θ, σ 2 ) (σ 2 known), prior θ ∼ N (µ0 , τ 2 ).
Posterior: !
µ0 /τ 2 + nX̄/σ 2 1
θ|X ∼ N , .
1/τ 2 + n/σ 2 1/τ 2 + n/σ 2
Bayes estimator (posterior mean) — the precision-weighted average:

(1/τ 2 )µ0 + (n/σ 2 )X̄

θ̂Bayes = .
1/τ 2 + n/σ 2

As n → ∞ or τ 2 → ∞ (diffuse prior): θ̂Bayes → X̄.

7.3 Minimax Estimation

Minimax Estimator

An estimator δ ∗ is minimax if it minimizes the maximum (worst-case) risk:

sup R(θ, δ ∗ ) = inf sup R(θ, δ).

θ δ θ

Bayes Estimator with Constant Risk is Minimax

If δπ is the Bayes estimator for prior π and its risk is constant in θ (i.e., R(θ, δπ ) = c for
all θ), then δπ is minimax.

Proof

For any other estimator δ:

Z
sup R(θ, δ) ≥ R(θ, δ) π(θ) dθ = r(π, δ)
θ
≥ r(π, δπ ) (δπ minimizes Bayes risk)
Z
= R(θ, δπ ) π(θ) dθ = c = sup R(θ, δπ ).
θ

So supθ R(θ, δπ ) ≤ supθ R(θ, δ) for all δ. ✓

Minimax Estimation: Bernoulli

iid
X1 , . . . , Xn ∼ Bernoulli(p), squared error loss. p̂ = X̄ has risk R(p, X̄) = p(1 − p)/n,
which varies with p.
√ √
The minimax estimator uses the Beta( n/2, n/2) prior. This gives:
P √ √
Xi + n/2 nX̄ + 1/2
p̂∗ = √ = √ ,
n+ n n+1

and one can verify R(p, p̂∗ ) = 1√

4(1+ n)2
is constant in p. So p̂∗ is minimax.
40 CHAPTER 7. BAYES AND MINIMAX ESTIMATION
 Part II

Interval Estimation

41
Chapter 8

Confidence Intervals

How This Chapter Fits Into the Course

Point estimation gives a single number as our best guess for θ. But how certain are we? A
confidence interval (CI) gives a random interval [L(X), U (X)] such that Pθ (L(X) ≤ θ ≤
U (X)) ≥ 1 − α.
The key insight: the interval is random (it varies with the sample), while θ is fixed. A 95% CI
means: if we repeated the experiment many times, 95% of the resulting intervals would contain
the true θ.

8.1 Pivotal Quantity Method

Pivotal Quantity

A function Q(X, θ) is a pivot (or pivotal quantity) if its distribution is completely known
(free of all unknown parameters).

Confidence Interval

An interval [L(X), U (X)] is a 100(1 − α)% confidence interval for θ if

Pθ (L(X) ≤ θ ≤ U (X)) ≥ 1 − α for all θ ∈ Θ.

Constructing CIs from a Pivot

1. Find a pivot Q(X, θ) with known distribution.

2. Find constants a, b such that P (a ≤ Q ≤ b) = 1 − α.

3. Invert the inequality to isolate θ: solve for θ in a ≤ Q ≤ b.

8.2 Confidence Interval for Normal Mean — σ 2 Known

iid
X1 , . . . , Xn ∼ N (µ, σ 2 ) with σ 2 known.
X̄−µ
√ ∼ N (0, 1) (exactly).
Pivot: Z = σ/ n
For level 1 − α: P (−zα/2 ≤ Z ≤ zα/2 ) = 1 − α.

43
44 CHAPTER 8. CONFIDENCE INTERVALS

!
X̄ − µ σ σ
 
P −zα/2 ≤ √ ≤ zα/2 = P X̄ − zα/2 √ ≤ µ ≤ X̄ + zα/2 √ = 1 − α.
σ/ n n n

z-interval for µ (σ known)

σ
X̄ ± zα/2 · √
n
where zα/2 = Φ−1 (1 − α/2). E.g., z0.025 = 1.96 for 95% CI.

8.3 Confidence Interval for Normal Mean — σ 2 Unknown: The

t-Interval
iid
X1 , . . . , Xn ∼ N (µ, σ 2 ), both parameters unknown.
X̄−µ
√ is no longer N (0, 1).
If we use S in place of σ, the pivot S/ n

t-Distribution of Studentized Mean

iid
Let X1 , . . . , Xn ∼ N (µ, σ 2 ). Then:

1. X̄ ∼ N (µ, σ 2 /n).

2. (n − 1)S 2 /σ 2 ∼ χ2 (n − 1).

3. X̄ and S 2 are independent (by Basu’s theorem).

4. Therefore: √
X̄ − µ (X̄ − µ)/(σ/ n)
T = √ =p ∼ t(n − 1).
S/ n (n − 1)S 2 /σ 2 /(n − 1)

Proof of (3): Independence via Basu

The complete sufficient statistic for (µ, σ 2 ) is (X̄, S 2 ). But wait — we need to be more
careful. When σ 2 is fixed: X̄ is CSS for µ, and (n − 1)S 2 /σ 2 ∼ χ2 (n − 1) is ancillary for
µ. By Basu’s theorem, X̄ ⊥ S 2 .
Alternatively, use the explicit computation: since Xi ∼ N (µ, σ 2 ), write Xi = µ + σZi
iid
where Zi ∼ N (0, 1). X̄ = µ + σ Z̄ and (n − 1)S 2 /σ 2 = (Zi − Z̄)2 . These are independent
P

because Z̄ and (Zi − Z̄) are uncorrelated (hence independent for normal) jointly normal
random variables.

Proof of (4): T ∼ t(n − 1)

By definition, t(k) = √Z where Z ∼ N (0, 1), V ∼ χ2 (k), and Z ⊥ V .

V /k
Here: √
X̄ − µ (X̄ − µ) n/σ
T = √ =p .
S/ n (n − 1)S 2 /σ 2 / (n − 1)
p
The numerator is Z ∼ N (0, 1) and the denominator is V /(n − 1) where V = (n −
1)S 2 /σ 2 ∼ χ2 (n − 1), and they’re independent. So T ∼ t(n − 1). ✓
8.4. CONFIDENCE INTERVAL FOR VARIANCE: χ2 -INTERVAL 45

t-interval for µ (σ unknown)

S
X̄ ± tn−1, α/2 · √
n
where tn−1,α/2 is the upper α/2 quantile of t(n − 1).

8.4 Confidence Interval for Variance: χ2 -Interval

iid
X1 , . . . , Xn ∼ N (µ, σ 2 ).
2
Pivot: Q = (n−1)S σ2
∼ χ2 (n − 1).
Find χL = χn−1,1−α/2 (lower) and χ2U = χ2n−1,α/2 (upper) such that P (χ2L ≤ Q ≤ χ2U ) = 1 − α.
2 2

Then: ! !
2 (n − 1)S 2 2 (n − 1)S 2 2 (n − 1)S 2
P χL ≤ ≤ χU = P ≤σ ≤ = 1 − α.
σ2 χ2U χ2L

χ2 -interval for σ 2
" #
(n − 1)S 2 (n − 1)S 2
, .
χ2n−1, α/2 χ2n−1, 1−α/2

CI Worked Example

Data: n = 16, x̄ = 10.5, s2 = 4.2. Assume normality.

95% CI for µ (σ unknown): t15,0.025 = 2.131.
√
4.2 2.049
10.5 ± 2.131 · √ = 10.5 ± 2.131 · = 10.5 ± 1.092 = [9.408, 11.592].
16 4

95% CI for σ 2 : χ215,0.025 = 27.488, χ215,0.975 = 6.262.

15 × 4.2 15 × 4.2
 
, = [2.292, 10.061].
27.488 6.262

8.5 Large-Sample Confidence Intervals

X̄−µ
When normality cannot be assumed but n is large, the CLT gives: √
σ/ n
≈ N (0, 1).
X̄−µ
If σ is unknown, substitute the consistent estimator S: √
S/ n
≈ N (0, 1).

Large-sample CI for proportion p

p
X ∼ Binomial(n, p). p̂ = X/n, SE
c = p̂(1 − p̂)/n.
s
p̂(1 − p̂)
p̂ ± zα/2 .
n
q
For MLE: θ̂M LE ± zα/2 / nI(θ̂) is a large-sample CI.
46 CHAPTER 8. CONFIDENCE INTERVALS
 Part III

Hypothesis Testing

47
Chapter 9

Foundations of Hypothesis Testing

How This Chapter Fits Into the Course

Confidence intervals quantify uncertainty about θ. Hypothesis testing addresses a different
question: is there enough evidence in the data to reject a specific claim about θ?
We formulate a null hypothesis H0 (the default/no-effect claim) and an alternative
hypothesis H1 . Based on data, we either reject H0 or fail to reject it.
The theory culminates in the Neyman-Pearson Lemma (Chapter 10), which characterizes
the optimal test for simple hypotheses. For composite hypotheses, the concepts of UMP tests
and GLRT provide powerful tools.

9.1 The Basic Setup

Hypothesis Testing Problem

A hypothesis is a statement about θ.

• A simple hypothesis specifies θ exactly: H0 : θ = θ0 .

• A composite hypothesis specifies a set: H1 : θ > θ0 .

A test is a decision rule based on data: given X, decide H0 or H1 . The critical region
(rejection region) C is the set of values of X for which we reject H0 :

Reject H0 ⇐⇒ X ∈ C.

Types of Errors and Power

Reject H0 Don’t reject H0

H0 true Type I error Correct
H1 true Correct Type II error

α(θ) = Pθ (Reject H0 ) = Pθ (X ∈ C), θ ∈ Θ0 (Type I probability)

β(θ) = Pθ (Not reject H0 ) = 1 − Pθ (X ∈ C), θ ∈ Θ1 (Type II probability)
The power function is
π(θ) = Pθ (X ∈ C), θ ∈ Θ.
We want π(θ) small for θ ∈ Θ0 and large for θ ∈ Θ1 .

49
50 CHAPTER 9. FOUNDATIONS OF HYPOTHESIS TESTING

The size of the test is α = supθ∈Θ0 π(θ).

A test has level α if its size is ≤ α.

The Tradeoff Between Type I and Type II Errors

Reducing the critical region C reduces α (Type I error) but increases β (Type II error).
They cannot both be minimized simultaneously (for fixed n). The Neyman-Pearson
approach: fix α at an acceptable level (say 0.05), then find the most powerful test
(minimize β).

Power Function: z-test for Normal Mean

iid
X1 , . . . , Xn ∼ N (µ, 1). Test H0 : µ = 0 vs. H1 : µ > 0.
√
Reject H0 if X̄ > zα / n (one-sided z-test).
Power function:
zα √
√
 
π(µ) = Pµ X̄ > √ = P Z > zα − µ n = 1 − Φ(zα − µ n).
n

• At µ = 0: π(0) = 1 − Φ(zα ) = α. ✓

• As µ → ∞: π(µ) → 1 (perfect power).

• Power increases as n increases (for fixed µ > 0).

Chapter 10

The Neyman-Pearson Theory:

Optimal Tests

How This Chapter Fits Into the Course

For a simple vs. simple test (H0 : θ = θ0 vs. H1 : θ = θ1 ), the Neyman-Pearson Lemma gives
the optimal test — the one with maximum power for a given size. This is the foundation of all
of testing theory.
For composite alternatives, the Monotone Likelihood Ratio property extends NP theory
to give UMP (Uniformly Most Powerful) tests.

10.1 The Neyman-Pearson Lemma

Neyman-Pearson Lemma

Consider testing H0 : θ = θ0 vs. H1 : θ = θ1 at size α. Let f0 = fθ0 and f1 = fθ1 .

Define the Likelihood Ratio test (LRT) with critical region:

f1 (x)
 
∗
C = x: >k ,
f0 (x)

where k ≥ 0 is chosen so that Pθ0 (X ∈ C ∗ ) = α.

Then:

1. (Sufficiency) The LRT C ∗ is the most powerful (MP) test at level α: for any other
level-α test with critical region C,

Pθ1 (X ∈ C ∗ ) ≥ Pθ1 (X ∈ C).

2. (Necessity) Any MP test at size α must be of the form C ∗ (with possible random-
ization on the boundary).

Full Proof of NP Lemma

Notation: For any set A, let πi (A) = Pθi (X ∈ A).

Key Observation: For any set A and any k > 0:
(
> 0 if x ∈ C ∗
(f1 (x) − kf0 (x))
/ C ∗.
< 0 if x ∈

51
52 CHAPTER 10. THE NEYMAN-PEARSON THEORY: OPTIMAL TESTS

This is because C ∗ = {f1 /f0 > k} = {f1 > kf0 }.

Proof of Sufficiency: Let C be any level-α test region (π0 (C) ≤ α). We must show
π1 (C ∗ ) ≥ π1 (C).
Consider the difference π1 (C ∗ ) − π1 (C):
Z Z
π1 (C ∗ ) − π1 (C) = f1 dµ − f1 dµ
∗
ZC CZ

= f1 dµ − f1 dµ.
C ∗ \C C\C ∗

On C ∗ \ C ⊆ C ∗ : f1 ≥ kf0 . On C \ C ∗ ⊆ (C ∗ )c : f1 ≤ kf0 .
Therefore:
Z Z
∗
π1 (C ) − π1 (C) ≥ kf0 dµ − kf0 dµ
C ∗ \C C\C ∗
= k (π0 (C ∗ \ C) − π0 (C \ C ∗ ))
= k (π0 (C ∗ ) − π0 (C ∗ ∩ C) − π0 (C) + π0 (C ∗ ∩ C))
= k (π0 (C ∗ ) − π0 (C))
= k(α − π0 (C)) ≥ 0, since π0 (C) ≤ α.

So π1 (C ∗ ) ≥ π1 (C). ✓
Proof of Necessity: If π1 (C) = π1 (C ∗ ) (both MP), then all inequalities above are
equalities. The equality k(π0 (C ∗ ) − π (C)) = 0 with k > 0 gives π (C) = π (C ∗ ) = α.
0 0 0
And equality in C ∗ \C f1 ≥ k C ∗ \C f0 requires f1 = kf0 on C ∗ \ C (a.e.), i.e., C differs
R R

from C ∗ only on the boundary {f1 /f0 = k} (which has measure zero for continuous
distributions).

NP Lemma: Simple vs. Simple Normal Test

iid
X1 , . . . , Xn ∼ N (µ, 1). Test H0 : µ = 0 vs. H1 : µ = 1 at size α.
Likelihood ratio:

(2π)−n/2 exp(− 12 (xi − 1)2 )

P
f1 (x)
=
f0 (x) (2π)−n/2 exp(− 21 x2i )
P

1 hX
 X i
= exp − (xi − 1)2 − x2i
2
1h X
 i
= exp − −2 xi + n
2
n
X 
= exp xi −
2
nx̄−n/2
=e .

The LRT rejects when enx̄−n/2 > k, i.e., when x̄ > c for some c.
√
Under H0 : X̄ ∼ N (0, 1/n), so nX̄ ∼ N (0, 1). The size condition: P0 (X̄ > c) = α =⇒
√
c = zα / n.
√ √
Result: Reject H0 ifX̄ > zα / n,i.e., if Z = nX̄ > zα . This is the one-sided z-test.
√ √ √
Power at µ = 1: P1 X̄ > zα / n = P (Z > zα − n) = 1 − Φ(zα − n).
10.2. UNIFORMLY MOST POWERFUL TESTS 53

NP Lemma: Exponential Distribution

iid
X1 , . . . , Xn ∼ Exp(θ) (mean θ). Test H0 : θ = θ0 vs. H1 : θ = θ1 where θ1 > θ0 .
Likelihood ratio:
P
θ−n e− xi /θ1
n
fθ (x) θ0 1 1
  X  
Λ(x) = 1 = 1−n P = exp − xi − .
fθ0 (x) θ0 e− xi /θ0 θ1 θ1 θ 0

Since θ1 > θ0 , we have 1/θ1 − 1/θ0 < 0, so Λ is increasing in xi .

P
P
The NP test rejects when Λ > k, equivalently when Xi > c (equivalently, X̄ > c/n).
Xi ∼ Γ(n, θ0 ). Determine c from Pθ0 ( Xi > c) = α.
P P
Under H0 :

10.2 Uniformly Most Powerful Tests

For composite hypotheses H1 : θ > θ0 , we want a test that is MP for every specific θ1 > θ0 . Such
a test is called UMP.

UMP Test

A level-α test with power function π(θ) is Uniformly Most Powerful (UMP) if for
every level-α test with power function π ′ (θ):

π(θ) ≥ π ′ (θ) for all θ ∈ Θ1 .

Monotone Likelihood Ratio (MLR)

A family {fθ } has Monotone Likelihood Ratio in statistic T if for all θ1 > θ0 , the
ratio fθ1 (x)/fθ0 (x) is a monotone non-decreasing function of T (x).

Karlin-Rubin: UMP via MLR

Suppose {fθ } has MLR in T (X) (non-decreasing in T ). For testing H0 : θ ≤ θ0 vs.

H1 : θ > θ0 :
The test “Reject H0 if T (X) > c” (where c satisfies Pθ0 (T > c) = α) is UMP at level α.

Sketch

Step 1: For fixed θ1 > θ0 , the NP lemma gives MP test against θ1 . The NP test rejects
when fθ1 /fθ0 > k, which (by MLR) is equivalent to T > c for some c. So the rejection
region is {T > c} regardless of θ1 .
Step 2: Since {T > c} is MP against every specific θ1 > θ0 , it is UMP for the composite
H1 : θ > θ0 .
Step 3: For θ ≤ θ0 : Power is Pθ (T > c) ≤ Pθ0 (T > c) = α (power is increasing in θ for
MLR families), so the test has level α.
54 CHAPTER 10. THE NEYMAN-PEARSON THEORY: OPTIMAL TESTS

MLR and UMP: One-Parameter Exponential Family

For a one-parameter exponential family with Q(θ) increasing in θ and T (x) as the sufficient
statistic:
fθ1 (x) n X o
= exp (Q(θ1 ) − Q(θ0 )) T (Xi ) + n(D(θ1 ) − D(θ0 )) .
fθ0 (x)
P
If Q is increasing, then Q(θ1 ) > Q(θ0 ), so the ratio is increasing in T (Xi ). This shows
P
all one-parameter exponential families with increasing Q have MLR in T (Xi ).
Consequences:
P
• Bernoulli(p), test H1 : p > p0 : UMP rejects when Xi > c (i.e., X̄ > c/n).
P
• Poisson(λ), test H1 : λ > λ0 : UMP rejects when Xi > c.

• N (µ, σ 2 ) (σ 2 known), test H1 : µ > µ0 : UMP rejects when X̄ > c.

No UMP for Two-Sided Alternatives

For testing H0 : µ = 0 vs. H1 : µ ̸= 0 in N (µ, 1), there is in general no UMP test.

Reason: The test optimal against µ = 1 rejects when X̄ > c1 (one-sided). The test
optimal against µ = −1 rejects when X̄ < c2 (other side). These cannot be the same test.
√
The standard solution is the two-sided z-test {|X̄| > zα/2 / n}, which is the UMP
unbiased (UMPU) test.

10.3 Locally Most Powerful Tests

When no UMP exists, we can seek a test that is most powerful for alternatives near θ0 .
LMP Test

A level-α test is Locally Most Powerful (LMP) at θ0 if there exists δ > 0 such that it
maximizes power for all θ1 with |θ1 − θ0 | < δ (or for all θ1 in a neighborhood).

Score Test is LMP

For testing H0 : θ = θ0 vs. H1 : θ > θ0 , the LMP test rejects for large values of the score
statistic:
∂
s(θ0 ; X) = log fθ (X) .
∂θ θ=θ0

Sketch

The NP test for H0 : θ = θ0 vs. H1 : θ = θ0 + ε rejects when

fθ0 +ε (x)
> k.
fθ0 (x)
For small ε > 0, Taylor-expand the log-ratio:
fθ0 +ε ∂ log fθ
log ≈ε· = ε · s(θ0 ; X).
fθ0 ∂θ θ0
So the LR is large iff s(θ0 ; X) is large. For one-parameter exponential families, s(θ0 ; X) ∝
T (X) − Eθ0 [T ], so the score test is equivalent to rejecting for large T (X).
Chapter 11

Generalized Likelihood Ratio Tests

and Standard Tests

How This Chapter Fits Into the Course

The NP lemma handles simple vs. simple hypotheses. For composite hypotheses (e.g., H0 : µ = µ0
vs. H1 : µ ̸= µ0 ), we need a more general approach.
The Generalized Likelihood Ratio Test (GLRT) compares the maximized likelihood
under H0 to the maximized likelihood over all θ. Wilks’ theorem gives the asymptotic distribution.
This framework derives the standard z, t, and χ2 tests from first principles.

11.1 The Generalized Likelihood Ratio Test

GLRT

For testing H0 : θ ∈ Θ0 vs. H1 : θ ∈ Θ1 = Θ \ Θ0 , the generalized likelihood ratio is

supθ∈Θ0 L(θ; X)
Λ(X) = .
supθ∈Θ L(θ; X)

The GLRT rejects H0 when Λ(X) < c (small ratio = data inconsistent with H0 ).
Equivalently, many texts write −2 log Λ(X) and reject for large values.

Wilks’ Theorem

Under H0 and regularity conditions, as n → ∞:

d
−2 log Λ(X) −→ χ2 (r),

where r = dim(Θ) − dim(Θ0 ) is the number of restrictions imposed by H0 .

Sketch

Let θ̂0 = arg maxθ∈Θ0 L(θ) and θ̂ = arg max L(θ). Taylor-expand:

−2 log Λ = 2[ℓ(θ̂) − ℓ(θ̂0 )].

A second-order Taylor expansion of ℓ(θ̂0 ) around θ̂, combined with the asymptotic normality
√
of n(θ̂ − θ0 ), gives that −2 log Λ ≈ a sum of squares of r asymptotically standard normal
variables — a χ2 (r) distribution.

55
56 CHAPTER 11. GENERALIZED LIKELIHOOD RATIO TESTS AND STANDARD TESTS

11.2 The z-Test via GLRT

iid
X1 , . . . , Xn ∼ N (µ, σ02 ), σ02 known. Test H0 : µ = µ0 vs. H1 : µ ̸= µ0 .
Under H0 : supµ=µ0 L = L(µ0 ). Over all µ: sup L = L(X̄).
!2
L(µ0 ) n(X̄ − µ0 )2 X̄ − µ0
−2 log Λ = −2 log = 2[ℓ(X̄) − ℓ(µ0 )] = = √ = Z 2,
L(X̄) σ02 σ0 / n

where Z ∼ N (0, 1) under H0 . So −2 log Λ = Z 2 ∼ χ2 (1).

The GLRT rejects when |Z| > zα/2 — the standard two-sided z-test.

11.3 The t-Test via GLRT

iid
X1 , . . . , Xn ∼ N (µ, σ 2 ), both unknown. Test H0 : µ = µ0 vs. H1 : µ ̸= µ0 .
Under H0 (µ = µ0 , σ 2 free): σ̂02 = n1 (Xi −µ0 )2 . Over all (µ, σ 2 ): µ̂ = X̄, σ̂ 2 = 1
(Xi − X̄)2 .
P P
n
After algebra:
!
t2 X̄ − µ0
−2 log Λ = n log 1 + where t = √ .
n−1 S/ n

Reject H0 for large | − 2 log Λ|, equivalently for large |t|.

Under H0 : t ∼ t(n − 1) exactly.
The GLRT is equivalent to the t-test: reject when |t| > tn−1,α/2 .

The One-Sample t-Test

X̄ − µ0
t= √ ∼ t(n − 1) under H0 : µ = µ0 .
S/ n
Reject H0 : µ = µ0 vs. H1 : µ ̸= µ0 at size α if |t| > tn−1,α/2 .
For one-sided H1 : µ > µ0 : reject if t > tn−1,α .

Full t-Test Worked Example

A company claims its batteries last 100 hours (mean). A sample of n = 25 batteries has
x̄ = 97.2 and s = 8.4. Test at α = 0.05.
H0 : µ = 100 vs. H1 : µ ̸= 100.
Step 1: Test statistic:
97.2 − 100 −2.8
t= √ = = −1.667.
8.4/ 25 1.68
Step 2: Critical value: t24,0.025 = 2.064.
Step 3: Decision: |t| = 1.667 < 2.064, so fail to reject H0 .
Conclusion: Insufficient evidence that the mean battery life differs from 100 hours.
p-value: p = 2 · P (t24 > 1.667) ≈ 0.109 > 0.05.

11.4 The χ2 -Test for Variance

iid
X1 , . . . , Xn ∼ N (µ, σ 2 ). Test H0 : σ 2 = σ02 .
(n−1)S 2
Pivot: Q = σ02
∼ χ2 (n − 1) under H0 .
11.4. THE χ2 -TEST FOR VARIANCE 57

χ2 -Test for Variance

Reject H0 : σ 2 = σ02 vs. H1 : σ 2 ̸= σ02 if

(n − 1)S 2
Q= < χ2n−1, 1−α/2 or Q > χ2n−1, α/2 .
σ02

For H1 : σ 2 > σ02 : reject if Q > χ2n−1,α .

58 CHAPTER 11. GENERALIZED LIKELIHOOD RATIO TESTS AND STANDARD TESTS
 Part IV

Tests of Independence and

Association

59
Chapter 12

Pearson’s Chi-Squared Test and

Kendall’s Tau

How This Chapter Fits Into the Course

So far we’ve tested hypotheses about parameters of a known distribution. Two fundamental
questions remain:

1. Goodness-of-fit: Does the data come from a specific distribution?

2. Independence: Are two categorical (or ordinal) variables independent?

Pearson’s χ2 test answers both. Kendall’s τ measures rank correlation and tests association
between two ordinal or continuous variables, without assuming a distributional form (non-
parametric).

12.1 Pearson’s Chi-Squared Goodness-of-Fit Test

Chi-Squared Statistic
P
Suppose we have k categories with observed counts O1 , . . . , Ok (with Oi = n) and
expected counts Ei = nπi under H0 (where πi is the hypothesized probability for category
P
i, πi = 1).
The Pearson χ2 statistic is:
k
X (Oi − Ei )2
χ2 = .
i=1
Ei

Asymptotic Distribution of χ2

Under H0 : π = π0 (the probabilities are fully specified), as n → ∞:

k
X (Oi − Ei )2 d
χ2 = −→ χ2 (k − 1).
i=1
Ei

If m parameters are estimated from the data to compute Ei , then the limiting distribution
is χ2 (k − 1 − m).

61
62 CHAPTER 12. PEARSON’S CHI-SQUARED TEST AND KENDALL’S TAU

Derivation Sketch via Multinomial CLT

Let Oi = nπ̂i where π̂i = Oi /n is the sample proportion for category i. By the multinomial
√ d
CLT: n(π̂ − π0 ) → N (0, Σ), where Σii = πi (1 − πi ) and Σij = −πi πj (i ̸= j).
Write Oi − Ei = n(π̂i − πi ). Then:
k k
X (Oi − Ei )2 X n(π̂i − πi )2
χ2 = = = Z T Z,
i=1
Ei i=1
πi
√ √
where Zi = n(π̂i − πi )/ πi . The vector Z is asymptotically multivariate normal with a
rank-(k − 1) covariance matrix (since π̂i = 1 imposes one linear constraint). A quadratic
P

form in a multivariate normal with rank-(k − 1) covariance is χ2 (k − 1). ✓

Goodness-of-Fit Test: Die Fairness

Roll a die 120 times, observe: 1:18, 2:22, 3:17, 4:21, 5:20, 6:22. Test H0 : die is fair
(πi = 1/6 for all i).
Ei = 120/6 = 20 for all i.

(18 − 20)2 (22 − 20)2 (17 − 20)2 (21 − 20)2 (20 − 20)2 (22 − 20)2
χ2 = + + + + +
20 20 20 20 20 20
4 4 9 1 4 22
= + + + +0+ = = 1.1.
20 20 20 20 20 20
Under H0 : χ2 ∼ χ2 (5). Critical value at α = 0.05: χ25,0.05 = 11.07.
Since 1.1 < 11.07, fail to reject H0 . No evidence the die is unfair.

12.2 Pearson’s Chi-Squared Test of Independence

Consider a two-way contingency table with r rows (levels of variable A) and c columns (levels of
P
variable B). Let Oij be the observed count in cell (i, j) and n = i,j Oij the total sample size.
Contingency Table Notation

• Oij = observed count in row i, column j.

P
• Ri = j Oij = row i total.
P
• Cj = i Oij = column j total.

• H0 : Variables A and B are independent, i.e., pij = pi· · p·j where pi· = P (A = i),
p·j = P (B = j).
Ri ·Cj
• Under H0 , estimated expected count: Eij = n .

Chi-Squared Test of Independence

The test statistic r X

c
X (Oij − Eij )2
χ2 =
i=1 j=1
Eij
12.3. KENDALL’S TAU 63

has the asymptotic null distribution χ2 ((r − 1)(c − 1)) under H0 . The degrees of freedom
is (r − 1)(c − 1) because we estimated r − 1 row marginal probabilities and c − 1 column
marginal probabilities (each losing one df to the sum-to-1 constraint), from the rc − 1 free
parameters total.

Why (r − 1)(c − 1) Degrees of Freedom

Under the general model: rc − 1 free parameters pij (minus 1 for sum-to-1). Under H0
(independence): (r − 1) + (c − 1) free parameters. Number of restrictions: (rc − 1) − [(r −
1) + (c − 1)] = rc − r − c + 1 = (r − 1)(c − 1). By Wilks’ theorem, the limiting χ2 df equals
the number of restrictions.

Independence Test: Gender and Product Preference

Survey 200 people on gender (M/F) and preferred product (A/B/C):

A B C Total
Male 40 35 25 100
Female 20 50 30 100
Total 60 85 55 200

Step 1: Expected counts Eij = Ri · Cj /n:

E11 = 100 · 60/200 = 30, E12 = 100 · 85/200 = 42.5, E13 = 100 · 55/200 = 27.5,
E21 = 30, E22 = 42.5, E23 = 27.5.
Step 2: Compute χ2 :

(40 − 30)2 (35 − 42.5)2 (25 − 27.5)2

χ2 = + +
30 42.5 27.5
(20 − 30)2 (50 − 42.5)2 (30 − 27.5)2
+ + +
30 42.5 27.5
100 56.25 6.25 100 56.25 6.25
= + + + + +
30 42.5 27.5 30 42.5 27.5
= 3.333 + 1.324 + 0.227 + 3.333 + 1.324 + 0.227
= 9.769.

Step 3: Critical value: df = (2 − 1)(3 − 1) = 2, χ22,0.05 = 5.991.

Step 4: Decision: 9.769 > 5.991, so reject H0 .
Conclusion: Gender and product preference are not independent (p < 0.05).

12.3 Kendall’s Tau

While Pearson’s χ2 tests independence for categorical data, Kendall’s τ measures ordinal
association between two variables.
Concordant and Discordant Pairs

Given n observations (X1 , Y1 ), . . . , (Xn , Yn ), a pair (i, j) with i < j is:

• Concordant if (Xi − Xj )(Yi − Yj ) > 0 (both increase or both decrease together).

64 CHAPTER 12. PEARSON’S CHI-SQUARED TEST AND KENDALL’S TAU

• Discordant if (Xi − Xj )(Yi − Yj ) < 0 (one increases while the other decreases).

• Tied if (Xi − Xj )(Yi − Yj ) = 0.

Let P = #{concordant pairs} and Q = #{discordant pairs}.

Kendall’s τ

Kendall’s τ is:
P −Q P −Q
τ= n
= ,
2 n(n − 1)/2
ranging from −1 (perfectly discordant) to +1 (perfectly concordant). τ = 0 suggests no
monotone association.
Population version: τ = P (concordant) − P (discordant) = E[sgn((Xi − Xj )(Yi − Yj ))].

Computing Kendall’s τ : Step by Step

Data: (X, Y ): (1, 2), (2, 4), (3, 3), (4, 5), (5, 1).
Sort by X (already sorted). For each pair (i, j) with i < j, compare Yi and Yj :

Pair (Xi , Yi ) (Xj , Yj ) Sign of (Yi − Yj ) C or D

(1, 2) (1, 2) (2, 4) 2−4<0 D
(1, 3) (1, 2) (3, 3) 2−3<0 D
(1, 4) (1, 2) (4, 5) 2−5<0 D
(1, 5) (1, 2) (5, 1) 2−1>0 C
(2, 3) (2, 4) (3, 3) 4−3>0 C
(2, 4) (2, 4) (4, 5) 4−5<0 D
(2, 5) (2, 4) (5, 1) 4−1>0 C
(3, 4) (3, 3) (4, 5) 3−5<0 D
(3, 5) (3, 3) (5, 1) 3−1>0 C
(4, 5) (4, 5) (5, 1) 5−1>0 C

5

P = 5 concordant, Q = 5 discordant, 2 = 10.
5−5
τ= = 0.
10
Kendall’s τ = 0 suggests no monotone association — consistent with the data not being
monotonically related (it goes up then comes back down).

Asymptotic Distribution of Kendall’s τ

Under H0 : X and Y are independent (i.e., τ = 0), for large n:

τ d
Z=p −→ N (0, 1).
2(2n + 5)/9n(n − 1)
2(2n+5)
Equivalently, Var(τ ) ≈ 9n(n−1) under H0 .
12.3. KENDALL’S TAU 65

Sketch
2
i<j h(Xi , Xj , Yi , Yj ) where h = sgn((Xi − Xj )(Yi − Yj )). Under H0 ,
P
Write τ = n(n−1)
each h has mean 0 and each pair is uncorrelated with non-overlapping pairs. The variance
follows from computing E[h2 ] = 1 and accounting for pairs sharing one observation
(Cov(hij , hik ) ̸= 0). The asymptotic normality follows from the theory of U -statistics (the
Hoeffding projection / CLT for U -statistics).
2(2n+5)
The exact variance formula 9n(n−1) is derived by careful enumeration of covariances
between overlapping pairs.

Hypothesis Test for Kendall’s τ

A researcher observes rankings (X, Y ) for n = 15 subjects and computes τ = 0.43. Test
H0 : τ = 0 vs. H1 : τ > 0 at α = 0.05.
Step 1: Compute standard deviation under H0 :
s
√
r r
2(2 · 15 + 5) 2 · 35 70
SD(τ̂ ) = = = = 0.03704 = 0.1925.
9 · 15 · 14 1890 1890

Step 2: Test statistic:

0.43
Z= = 2.234.
0.1925
Step 3: Critical value: z0.05 = 1.645.
Step 4: Decision: 2.234 > 1.645, so reject H0 .
Conclusion: Significant positive monotone association between X and Y . p-value
= P (Z > 2.234) = 1 − Φ(2.234) ≈ 0.013.

Kendall’s τ vs. Spearman’s ρ

Both are rank-based correlation measures.

• Spearman’s ρ = Pearson correlation of the ranks.

• Kendall’s τ counts concordant minus discordant pairs directly.

2
• In general: |τ | ≤ |ρ|. For bivariate normal, τ ≈ π arcsin(ρ).

• Kendall’s τ has a cleaner probabilistic interpretation: τ = P (concordant) −

P (discordant).

• Both have asymptotically normal distributions under independence.

66 CHAPTER 12. PEARSON’S CHI-SQUARED TEST AND KENDALL’S TAU
 Part V

Comprehensive Worked Problem

Bank

67
Chapter 13

Point Estimation: Problems with Full

Solutions

Problem 1: Finding the UMVUE for Exponential

iid
Let X1 , . . . , Xn ∼ Exp(θ) with mean θ > 0. Find the UMVUE of ψ(θ) = θ2 .

Solution

Step 1: Complete Sufficient Statistic.

The Exponential(θ) family is a 1-parameter exponential family. The sufficient statistic is
T = ni=1 Xi ∼ Γ(n, θ).
P

Since θ ranges over (0, ∞) (an open set), T is complete sufficient.

Step 2: Find an unbiased estimator of θ2 that is a function of T .
We know E[T 2 2 2 2 2 2
h ] =2 nθi and E[T ] = Var(T ) + (E[T ]) = nθ + n θ = n(n + 1)θ .
T
Therefore E n(n+1) = θ2 .
Step 3: Apply Lehmann-Scheffé.
P
T 2 ( X )2
ϕ(T ) = n(n+1) i
= n(n+1) is unbiased for θ2 and is a function of the CSS T .
By Lehmann-Scheffé, the UMVUE of θ2 is:
Pn 2
( i=1 Xi )
.
n(n + 1)

Problem 2: Cramér-Rao Bound for Gamma Distribution

iid
Let X1 , . . . , Xn ∼ Γ(α, β) with α known and β unknown. Compute the Fisher information
I(β) and the CRLB for estimating β. Determine if X̄/α is efficient.

Solution

Step 1: Log-likelihood of a single observation.

α−1 e−x/β
f (x; β) = x β α Γ(α) , x > 0.
x
log f = (α − 1) log x − − α log β − log Γ(α).
β
Step 2: Score function.

69
70 CHAPTER 13. POINT ESTIMATION: PROBLEMS WITH FULL SOLUTIONS

∂ log f x α
s(β; x) = = 2− .
∂β β β
Verify E[s] = 0: E[s] = E[X]/β 2 − α/β = αβ/β 2 − α/β = 0. ✓
Step 3: Fisher information via second derivative.

∂ 2 log f 2x α
2
= − 3 + 2.
∂β β β
2X α 2E[X] α 2αβ α 2α α α
 
I(β) = −E − 3 + 2 = 3
− 2 = 3 − 2 = 2 − 2 = 2.
β β β β β β β β β
Step 4: CRLB for unbiased estimators of β.
ψ(β) = β, ψ ′ (β) = 1.

1 1 β2
CRLB = = = .
nI(β) n · α/β 2 nα
Step 5: Check if X̄/α is efficient.
E[X̄]
 = αβ,
 so E[X̄/α] =2 β (unbiased). ✓
Var(X) αβ β2
Var X̄/α = α2 n = α2 n = αn .
2
 
β
Since Var X̄/α = αn = CRLB, the estimator X̄/α attains the CRLB and is therefore
efficient (and is the UMVUE).

Problem 3: MLE and Sufficiency for a Two-Parameter Family

iid
Let X1 , . . . , Xn ∼ f (x; µ, σ) = σ1 e−(x−µ)/σ for x > µ, σ > 0 (two-parameter exponential,
shifted). Find the MLE of (µ, σ) and identify the sufficient statistic.

Solution

Step 1: Log-likelihood.
log f = − log σ − (x − µ)/σ for x > µ.
n
1X
ℓ(µ, σ) = −n log σ − (xi − µ),
σ i=1
valid only when µ < mini xi = x(1) (since all xi > µ).
Step 2: MLE of µ.
For fixed σ > 0: ℓ decreases as µ decreases, increases as µ increases, but is only defined
for µ < x(1) . Taking the limit as µ ↗ x(1) :
µ̂ = X(1) = min(X1 , . . . , Xn ).
(The likelihood is increasing in µ over its domain, so MLE is at the supremum of the
domain.)
Step 3: MLE of σ.
With µ̂ = x(1) : ℓ(x(1) , σ) = −n log σ − σ1 (xi − x(1) ).
P

Differentiating:
Pn
(xi − x(1) ) − X(1) )
P
∂ℓ n i=1 (Xi
=− + = 0 =⇒ σ̂ = = X̄ − X(1) .
∂σ σ σ2 n
MLEs: µ̂ = X(1) , σ̂ = X̄ − X(1) .
 71

Step 4: Sufficient Statistic.

Joint PDF:
xi − nµ
 P 
fµ,σ (x) = σ −n exp − 1(x(1) > µ).
σ
P
By Factorization Theorem (set T = (X(1) , Xi )):

nT2 /n − nµ
 
−n
f =σ exp − 1(T1 > µ) · |{z}
1 .
σ
| {z } h(x)
g(T,µ,σ)

P
(X(1) , Xi ) (equivalently (X(1) , X̄)) is sufficient for (µ, σ).

Problem 4: Bayes Estimation with Non-Conjugate Prior

X ∼ Binomial(n, p). Use prior p ∼ Uniform(0, 1). Find the Bayes estimator of p under
squared error loss. Show it lies between the MLE p̂ = X/n and the prior mean 1/2.

Solution

Step 1: Posterior distribution.

Uniform(0, 1) = Beta(1, 1), so by Beta-Binomial conjugacy:

p|X = x ∼ Beta(x + 1, n − x + 1).

Step 2: Posterior mean (Bayes estimator).

x+1 X +1
p̂Bayes = E[p|X] = = .
(x + 1) + (n − x + 1) n+2
Step 3: Show it lies between X/n and 1/2.
Write:
X +1 X n 1 2 n 2 1
= · + · = · p̂M LE + · .
n+2 n n+2 2 n+2 n+2 n+2 2
This is a convex combination (weights n/(n + 2) and 2/(n + 2) sum to 1) of p̂M LE = X/n
̸ n/2).
and the prior mean 1/2. So p̂Bayes lies strictly between p̂M LE and 1/2 (whenever X =
Intuition: The Bayes estimator shrinks the MLE toward the prior mean. As n → ∞, the
data overwhelms the prior and p̂Bayes → p̂M LE .
72 CHAPTER 13. POINT ESTIMATION: PROBLEMS WITH FULL SOLUTIONS
Chapter 14

Testing: Problems with Full Solutions

Problem 5: UMP Test via NP Lemma

iid
X1 , . . . , Xn ∼ N (µ, σ 2 ) with σ 2 known. Find the UMP test for H0 : µ ≤ µ0 vs. H1 : µ > µ0
at size α.

Solution

Step 1: Show MLR in X̄.

For any µ1 > µ0 :
!
(xi − µ1 )2 − (xi − µ0 )2
P P
fµ1 (x)
= exp −
fµ0 (x) 2σ 2
!
(µ1 − µ0 ) xi − n(µ21 − µ20 )/2
P
= exp .
σ2

Since µ1 > µ0 : (µ1 − µ0 ) > 0, so the ratio is strictly increasing in

P
xi = nx̄. This shows
MLR in X̄.
Step 2: Apply Karlin-Rubin.
By the Karlin-Rubin theorem, the UMP test rejects when X̄ > c.
Step 3: Determine the critical value.
The size condition at µ = µ0 (the boundary):

c − µ0 σ
 
Pµ0 (X̄ > c) = α =⇒ P Z > √ = α =⇒ c = µ0 + zα √ .
σ/ n n

Note: for µ < µ0 , Pµ (X̄ > c) < α (power is increasing in µ for the normal with MLR), so
the size is indeed α.
UMP test: Reject H0 if
X̄ − µ0
Z= √ > zα .
σ/ n

Problem 6: GLRT for Testing Poisson Rate

iid
X1 , . . . , Xn ∼ Poisson(λ). Derive the GLRT for H0 : λ = λ0 vs. H1 : λ ̸= λ0 . What is the
asymptotic distribution of the test statistic?

73
74 CHAPTER 14. TESTING: PROBLEMS WITH FULL SOLUTIONS

Solution

Step 1: Compute the restricted and unrestricted MLEs.

Unrestricted MLE: λ̂ = X̄. Restricted MLE (under H0 : λ = λ0 ): λ̂0 = λ0 .
Step 2: Log-likelihood.
ℓ(λ) = ( xi ) log λ − nλ − log(xi !) = nt log λ − nλ + C, where t = x̄.
P P

Step 3: Compute −2 log Λ.

−2 log Λ = −2[ℓ(λ0 ) − ℓ(λ̂)]

= −2[nx̄ log λ0 − nλ0 − nx̄ log x̄ + nx̄]
x̄
 
= 2n x̄ log − (x̄ − λ0 ) .
λ0
Step 4: Asymptotic distribution.
d
By Wilks’ theorem: H0 imposes 1 restriction (λ = λ0 vs. free λ), so −2 log Λ → χ2 (1)
under H0 .
Step 5: Reject H0 if −2 log Λ > χ21,α .
Consistency check at x̄ = λ0 : −2 log Λ = 0 (no evidence against H0 ). For large |x̄ − λ0 |:
−2 log Λ is large (strong evidence against H0 ). ✓

Problem 7: Neyman-Pearson and Power Calculation

X ∼ Poisson(λ) (single observation). Test H0 : λ = 1 vs. H1 : λ = 4 at size α = 0.05.

Find the NP test and compute its power.

Solution

Step 1: Likelihood ratio.

P (X = x; λ = 4) e−4 4x /x!
Λ(x) = = −1 x = e−3 · 4x .
P (X = x; λ = 1) e 1 /x!
This is increasing in x. NP test: reject when x > c.
Step 2: Find c for size α = 0.05.
Under H0 : X ∼ Poisson(1). P (X ≥ c) ≤ 0.05. Compute cumulative probabilities:
P (X ≤ 3) = e−1 (1+1+1/2+1/6) = e−1 (8/3) ≈ 0.9810. P (X ≥ 4) = 1−0.9810 = 0.0190 <
0.05. P (X ≥ 3) = 1 − P (X ≤ 2) = 1 − e−1 (1 + 1 + 1/2) ≈ 1 − 0.9197 = 0.0803 > 0.05.
So we take c = 4: reject when X ≥ 4.
Actual size: P1 (X ≥ 4) = 0.0190 < 0.05. ✓
Step 3: Power.
Power = P4 (X ≥ 4) = 1 − P4 (X ≤ 3).
P4 (X ≤ 3) = e−4 (1 + 4 + 8 + 32/6) = e−4 (1 + 4 + 8 + 32/6).
k
More carefully: P4 (X ≤ 3) = e−4 3k=0 4k! = e−4 (1 + 4 + 8 + 32/3) ≈ 0.01832 × 18.33 ≈
P

0.2381.
Power = 1 − 0.2381 = 0.762.
Interpretation: The test detects the difference λ = 4 vs. λ = 1 with probability ≈ 76%
when based on a single observation.
Appendix A

Key Distributions and Reference

Formulas

A.1 Important Distributions

Distribution PDF/PMF Mean Variance

x 1−x
Bernoulli(p) p
(1 − p) , x ∈ {0, 1} p p(1 − p)
n x n−x
Binomial(n, p) x p (1 − p) np np(1 − p)
Poisson(λ) e−λ λx /x! λ λ
Geometric(p) (1 − p)x−1 p, x = 1, 2, . . . 1/p (1 − p)/p2
2 2
N (µ, σ 2 ) (2πσ 2 )−1/2 e−(x−µ) /2σ µ σ2
Uniform(a, b) 1/(b − a) on (a, b) (a + b)/2 (b − a)2 /12
Exp(λ) λe−λx , x > 0 1/λ 1/λ2
xα−1 e−x/β
Γ(α, β) β α Γ(α) αβ αβ 2
χ2 (k) = Γ(k/2, 2) k 2k
t(k) complex 0 (k > 1) k/(k − 2) (k > 2)
F (m, n) complex n/(n − 2) —
xα−1 (1−x)β−1 α αβ
Beta(α, β) B(α,β) α+β (α+β)2 (α+β+1)

A.2 Key Relationships

Chi-Squared, t, and F Relationships

Z
Z ∼ N (0, 1), V ∼ χ2 (k), Z ⊥ V =⇒ p ∼ t(k).
V /k
U/m
U ∼ χ2 (m), V ∼ χ2 (n), U ⊥ V =⇒ ∼ F (m, n).
V /n
iid (n − 1)S 2
X1 , . . . , Xn ∼ N (µ, σ 2 ) =⇒ ∼ χ2 (n − 1), X̄ ⊥ S 2 .
σ2
t(k)2 ∼ F (1, k).

75
76 APPENDIX A. KEY DISTRIBUTIONS AND REFERENCE FORMULAS

A.3 Critical Inequality Facts

Inequalities Used in Proofs

Cauchy-Schwarz: [E(XY )]2 ≤ E(X 2 )E(Y 2 ).

Markov: P (X ≥ a) ≤ E(X)/a for X ≥ 0, a > 0.
Chebyshev: P (|X − µ| ≥ kσ) ≤ 1/k 2 .
Jensen: E[g(X)] ≥ g(E[X]) for convex g.
Law of Total Variance: Var(X) = E[Var(X|Y )] + Var(E[X|Y ]) .
Tower Property: E[E[X|Y ]] = E[X].

A.4 Summary: UMVUE Finding Strategy

UMVUE Roadmap

1. Identify the family. Is it exponential? If yes, the natural sufficient statistic

P
T = ( Tj (Xi )) is complete sufficient (CSS).

2. If not exponential family: Use Neyman-Fisher factorization to find sufficient

statistic. Check completeness by definition.

3. Find an unbiased function of the CSS:

• Guess directly (e.g., X̄ for mean of exponential family).

• Take a simple unbiased estimator h(X) and compute E[h(X)|T ].

4. Invoke Lehmann-Scheffé: The unbiased function of the CSS is the unique

UMVUE.

5. Check: Compare variance to CRLB. If equal, the UMVUE is also efficient.

A.5. SUMMARY: HYPOTHESIS TESTING DECISION TREE 77

A.5 Summary: Hypothesis Testing Decision Tree

Choosing the Right Test

• Simple H0 vs. Simple H1 : Neyman-Pearson lemma ⇒ LRT.

• One-sided H1 , exponential family: Check MLR ⇒ UMP test via Karlin-Rubin.

• Two-sided H1 , general composite: GLRT with Wilks’ χ2 asymptotics.

• Normal mean (σ known): z-test.

• Normal mean (σ unknown): t-test with t(n − 1).

• Normal variance: χ2 -test.

• Categorical independence: Pearson χ2 with (r − 1)(c − 1) df.

• Monotone association (ordinal/continuous): Kendall’s τ with N (0, 1) asymptotics.