Statistics — Exam Revision Notes 2025–2026

Revision Notes
Fisher Information · Confidence Intervals · Bayesian Inference · Hypothesis Testing
All key formulas, tricks, and pitfalls in one place

Fisher Information

Definition
Given a model f (x; θ), the score function is

∂
s(θ; x) = log f (x; θ).
∂θ
The Fisher information is the variance of the score:
" #
∂ log f (X; θ) 2
 2 
∂ log f (X; θ)
I(θ) = E = −E .
∂θ ∂θ2

The second equality holds under the regularity conditions (support must not depend on θ, and
differentiation under the integral sign is valid).
Astuce — two formulas, one answer

Always compute with both formulas as a cross-check. The score-variance formula is safer when
the second derivative is messy. The second-derivative formula is faster when log f is quadratic
in x.

Properties
ˆ E[s(θ; X)] = 0 always (regularity conditions required).
ˆ Additivity for iid data: In (θ) = n · I1 (θ).
ˆ Reparametrisation: if ϕ = g(θ) with g differentiable, then Iϕ (ϕ) = Iθ (θ) / [g ′ (θ)]2 .
ˆ Fisher information matrix (vector parameter θ ∈ Rp ): I(θ)ij = −E ∂ 2 log f /∂θi ∂θj .
 

Cramér–Rao Lower Bound (CRLB)

For any unbiased estimator θ̂ of θ:

1 1
Var(θ̂) ≥ = .
In (θ) n I1 (θ)

An estimator that achieves the CRLB is called efficient. The MLE is asymptotically efficient.

Pitfall: The CRLB applies only to unbiased estimators. Never compare a biased estimator’s
variance directly to the CRLB.

Data Processing Inequality (DPI)

For any statistic T = T (X):

1
Statistics — Exam Revision Notes 2025–2026

IT (θ) ≤ IX (θ),
with equality if and only if T is a sufficient statistic for θ.

Interpretation: Processing data can only lose information, never create it. A sufficient statistic
retains all the Fisher information in the sample.
Classic example. X ∼ N (θ, 1), T = sign(X):
[φ(θ)]2
IX (θ) = 1, IT (θ) = < 1 ∀ θ.
Φ(θ)(1 − Φ(θ))
At θ = 0: IT (0) = 2/π ≈ 0.637. Only 63.7% of the information survives.

Key examples of I(θ)

Model Parameter I(θ) CRLB (n obs)

1 θ(1 − θ)
Bernoulli(θ) θ
θ(1 − θ) n
1 σ2
N (θ, σ 2 ), σ 2 known µ=θ
σ2 n
1 2σ 4
N (µ, θ), µ known σ2 = θ
2θ2 n
1 θ
Poisson(θ) λ=θ
θ n
1 θ2
Exp(θ) λ=θ
θ2 n

Confidence Intervals

General method: find a pivot — a function of the data and θ whose distribution is fully known
— then invert the inequalities to isolate θ.

CI for µ — variance known (Z interval)

X̄ − µ
Pivot: Z = √ ∼ N (0, 1).
σ/ n

σ
CI1−α : X̄ ± zα/2 √ ,
n
where zα/2 = Φ−1 (1 − α/2).

Common critical values: z0.025 = 1.96, z0.005 = 2.576.

CI for µ — variance unknown (t interval)

Replace σ with S; the distribution inflates to a t.
X̄ − µ
Pivot: T = √ ∼ tn−1 .
S/ n

S
CI1−α : X̄ ± tα/2, n−1 √ .
n

2
Statistics — Exam Revision Notes 2025–2026

Astuce

As n → ∞, tn−1 → N (0, 1), so tα/2,n−1 → zα/2 . For n ≥ 30 in practice the two intervals are
almost identical.

CI for σ 2 (χ2 interval)

(n − 1)S 2
Pivot: Q = ∼ χ2n−1 .
σ2
" #
(n − 1)S 2 (n − 1)S 2
CI1−α : , .
χ21−α/2, n−1 χ2α/2, n−1

Pitfall: The χ2 distribution is asymmetric, so the interval is not symmetric around S 2 . Note
the positions of χ2α/2 and χ21−α/2 : the smaller quantile goes in the denominator of the upper
bound and vice versa (because σ 2 is in the denominator of the pivot).

CI for a proportion p (large sample)

p̂ − p
Pivot (CLT): Z = p ≈ N (0, 1).
p(1 − p)/n
r
p̂(1 − p̂)
CI1−α : p̂ ± zα/2 .
n

Valid when np̂ ≥ 5 and n(1 − p̂) ≥ 5.

Summary table

Target Pivot Distribution Symmetric?

√
µ (σ 2 known) (X̄ − µ)/(σ/ n) N (0, 1) Yes
√
µ (σ 2 unknown) (X̄ − µ)/(S/ n) tn−1 Yes
σ2 (n − 1)S 2 /σ 2 χ2n−1 No
p
p (proportion) (p̂ − p)/ p(1 − p)/n N (0, 1) approx Yes

Bayesian Inference

Bayes’ theorem

f (x | θ) π(θ)
π(θ | x) = ∝ f (x | θ) π(θ),
m(x)

R where π(θ) is the prior, f (x | θ) is the likelihood, π(θ | x) is the posterior, and m(x) =
f (x | θ)π(θ) dθ is the marginal (normalising constant).

3
Statistics — Exam Revision Notes 2025–2026

Astuce — the kernel trick

You almost never need to compute m(x) explicitly. Write

π(θ | x) ∝ f (x | θ) × π(θ),
| {z } |{z}
likelihood prior

identify the functional form of θ, and read off the posterior family and its updated parameters.

Point estimators from the posterior

ˆ Posterior mean = E[θ | x]: minimises posterior expected squared loss.
ˆ Posterior median: minimises posterior expected absolute loss.
ˆ MAP (Maximum A Posteriori) = arg maxθ π(θ | x): minimises 0–1 loss.

Conjugate priors (must-know table)

Likelihood Conjugate prior Posterior

P
Bernoulli(θ) Beta(α, β) Beta(α +
P xi , β + n −
xi )
P
Poisson(λ) Gamma(a, b) Gamma(a + xi , b + n)

N (µ, σ 2 ), σ 2 known N (µ0 , τ02 ) N (µn , τn2 ) (see below)

N (µ, σ 2 ), µ known InvGamma(a, b) InvGamma(a′ , b′ )

P
Exp(λ) Gamma(a, b) Gamma(a + n, b + xi )

Normal–Normal update (known σ 2 , prior µ ∼ N (µ0 , τ02 )):

 −1  
1 n µ0 nX̄
τn2 = 2 + 2 , µn = τn2 + 2 .
τ0 σ τ02 σ

The posterior mean is a weighted average of the prior mean and the data mean, with weights
proportional to their respective precisions 1/τ02 and n/σ 2 .

Credible interval
A (1 − α) credible interval [a, b] satisfies P (a ≤ θ ≤ b | x) = 1 − α.
Astuce — credible vs confidence interval

A Bayesian credible interval has a direct probability statement about θ: “There is 95% proba-
bility that θ lies in [a, b] given the data.” A frequentist CI does not say this — θ is fixed; the
interval is random.

Prior choices
ˆ Informative prior: encodes genuine prior knowledge.
ˆ Non-informative (flat) prior: π(θ) ∝ 1 (may be improper).
ˆ Jeffreys prior: π(θ) ∝ I(θ), invariant under reparametrisation.
p

4
Statistics — Exam Revision Notes 2025–2026

Hypothesis Testing

Setup
A hypothesis test decides between:

H0 : θ ∈ Θ0 (null) vs H1 : θ ∈ Θ1 (alternative).

H0 true H1 true

Reject H0 Type I error (α) Correct (power 1 − β)

Fail to reject Correct Type II error (β)

ˆ α = P (reject H0 | H0 true) — significance level (controlled by the analyst).

ˆ β = P (fail to reject H0 | H1 true) — Type II error rate.
ˆ Power = 1 − β = P (reject H0 | H1 true).

The p-value
p-value = P (T ≥ tobs | H0 ) ,
where T is the test statistic and tobs is its observed value. Reject H0 when p-value < α.

Pitfall: The p-value is not P (H0 is true). It is the probability of observing data at least as
extreme as what we observed, assuming H0 is true.

Common tests

Test Statistic Null dist. Reject H0 if

X̄ − µ0
One-sample Z (σ Z= √ N (0, 1) |Z| > zα/2
σ/ n
known)
X̄ − µ0
One-sample t (σ un- T = √ tn−1 |T | > tα/2,n−1
S/ n
known)
Two-sample t (equal T = tn1 +n2 −2 |T | > tα/2
var.) X̄ − X̄2
p 1
Sp 1/n1 + 1/n2
Chi-squared GoF χ2 = χ2k−1 χ2 > χ2α,k−1
P (Oi − Ei )2
Ei
F -test (variances) F = S1 /S22
2 Fn1 −1,n2 −1 F > Fα/2

(n1 − 1)S12 + (n2 − 1)S22

where Sp2 = is the pooled variance.
n1 + n2 − 2

Neyman–Pearson lemma
For testing a simple H0 : θ = θ0 vs simple H1 : θ = θ1 , the most powerful test at level α is the
likelihood ratio test:

5
Statistics — Exam Revision Notes 2025–2026

f (x; θ1 )
Reject H0 when Λ(x) = > k,
f (x; θ0 )
where k is chosen so that P (Λ > k | H0 ) = α.

Likelihood Ratio Test (composite)

For testing H0 : θ ∈ Θ0 vs H1 : θ ∈ Θ1 :

supθ∈Θ0 f (x; θ) d
Λ(x) = , → χ2r under H0 ,
−2 log Λ(x) −
supθ∈Θ f (x; θ)

where r = dim Θ − dim Θ0 is the number of restrictions.

Useful tricks

Astuce — duality between CIs and tests

A two-sided test at level α rejects H0 : θ = θ0 if and only if θ0 falls outside the (1−α) confidence
interval for θ. You can always convert one into the other.

Astuce — power increases when. . .

ˆ n increases (more data ⇒ more power).

ˆ α increases (looser threshold ⇒ easier to reject).
ˆ |θ1 − θ0 | increases (larger effect is easier to detect).
ˆ σ 2 decreases (less noise ⇒ cleaner signal).

Keep calm, read the question twice, and check your degrees of freedom. Good luck!