Comprehensive Guide to Statistical

Inference
Concepts, Formulas, Strategies & Exam Problems

Based on Course Materials (2025)

1
Contents

1 Introduction 4
1.1 Key Concepts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4

2 Mathematical Preliminaries & Useful Identities 5

2.1 Multivariate Normal Distribution . . . . . . . . . . . . . . . . . . . . . . 5
2.2 Binomial expansion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6
2.3 The Gamma Integral . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6
2.4 The Beta Integral . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6
2.5 Gaussian Integral . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6

3 Polar Jacobian & Sampling Distributions 8

3.1 Key Concepts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8
3.2 Formula Sheet . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8

4 Review of Order Statistics 9

4.1 Key Concepts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9
4.2 Probability Density Function of the 1st Order Statistic (Minimum) . . . 9
4.3 Probability Density Function of the n-th Order Statistic (Maximum) . . 9
4.4 Joint Probability Density Function of the 1st and n-th Order Statistics . 9
4.5 Formula Sheet . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10

5 Sufficiency 11
5.1 Intuition . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11
5.2 Formal Definition . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11
5.3 Example: Bernoulli Trials . . . . . . . . . . . . . . . . . . . . . . . . . . 11
5.4 Sufficiency in Hypergeometric Model – Complete Clarification . . . . . . 11
5.5 Neyman-Fisher Factorization Theorem (FNFT) . . . . . . . . . . . . . . 13
5.6 Formula Sheet . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14

6 Completeness & Complete Sufficiency 15

6.1 Key Concepts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15
6.2 Completeness in family . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15

7 Integral Transforms & Uniqueness (Tools for Completeness) 16

7.1 Definition . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16
7.2 Common Transforms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16
7.3 The Uniqueness Property (The Main Result) . . . . . . . . . . . . . . . . 16
7.4 Applications to Normal Families . . . . . . . . . . . . . . . . . . . . . . . 16

8 Point Estimation I: Unbiasedness & UMVUE 18

8.1 Key Concepts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18

9 Theoretical Results on UMVUE 19

9.1 1. Uniqueness of UMVUE . . . . . . . . . . . . . . . . . . . . . . . . . . 19
9.2 2. The Covariance Method (Necessary & Sufficient) . . . . . . . . . . . . 19
9.3 3. Algebra of UMVUEs . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19
9.4 4. Correlation and Efficiency . . . . . . . . . . . . . . . . . . . . . . . . . 19

2
 9.5 5. Symmetry Principle . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20

10 Point Estimation II: MLE & Invariance 21

10.1 Maximum Likelihood Estimation (MLE) . . . . . . . . . . . . . . . . . . 21
10.2 The Invariance Property . . . . . . . . . . . . . . . . . . . . . . . . . . . 21

11 Method of Moments Estimation (MME) 22

11.1 Key Concepts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22
11.2 Definitions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22
11.3 The Procedure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22
11.4 Examples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22

12 Cramér-Rao Lower Bound (CRLB) 24

12.1 C-R Regularity Conditions . . . . . . . . . . . . . . . . . . . . . . . . . . 24
12.2 The Score Function & Fisher Information . . . . . . . . . . . . . . . . . . 24
12.3 The Inequality . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24
12.4 Step-by-Step Calculation Guide . . . . . . . . . . . . . . . . . . . . . . . 24
12.5 Efficiency & Attainment . . . . . . . . . . . . . . . . . . . . . . . . . . . 25
12.6 Limitations & Failure Cases . . . . . . . . . . . . . . . . . . . . . . . . . 25

13 Selected Assignment & Exam Problems (With Solutions) 26

13.1 Assignment 2: Problem 1 Solution . . . . . . . . . . . . . . . . . . . . . . 26
13.1.1 The Model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26
13.1.2 Solution 1: Using the Definition . . . . . . . . . . . . . . . . . . . 26
13.1.3 Solution 2: Using Factorization Theorem (FNFT) . . . . . . . . . 27
13.2 The joint distribution of X(1) &X(n) in U (θ1 , θ2 ) . . . . . . . . . . . . . . 27
13.3 Distribution and Completeness of T = X(n) for Uniform(0, θ) . . . . . . . 28
13.4 Problem 1: Poisson Estimation (Mid-Sem 2025) . . . . . . . . . . . . . . 30
13.5 Problem 2: Bernoulli CRLB (Assignment 6) . . . . . . . . . . . . . . . . 30
13.6 Problem 3: Normal Distribution & MSE (Mid-Sem 2025) . . . . . . . . . 31
13.7 Problem 4: UMVUE for Uniform Family (Assignment 4) . . . . . . . . . 31
13.8 Problem 5: CRLB Not Attained (Assignment 6) . . . . . . . . . . . . . . 31

3
1 Introduction
1.1 Key Concepts
Statistical inference is the process of learning from uncertainty using inductive reason-
ing—moving from specific observed data to general conclusions about a population.

• Random Sample: Variables X1 , . . . , Xn are i.i.d. with distribution F .

• Sample Space (W): The set of all possible values of the data.

• Parameter Space (Ω): The set of all possible values for the unknown parameter
θ.

How to Think & Solve

Strategy:

1. Identify the Randomness: Distinguish between the random variable X

(the process) and the realization x (the number you see).

2. Define the Spaces:

• If the problem asks for the Sample Space, look at the support of X (e.g.,
x > 0 for Exponential).
• If it asks for the Parameter Space, look at valid values for constants
(e.g., σ 2 > 0).

4
2 Mathematical Preliminaries & Useful Identities
2.1 Multivariate Normal Distribution
If we have a p x 1 random vector X that is distributed according to a multivariate normal
distribution with a population mean vector µ and population variance-covariance matrix
Σ, then this random vector, X, will have the joint density function as shown in the
expression below:
1 p/2
|Σ|−1/2 exp{− 21 (x − µ)′ Σ−1 (x − µ)}


ϕ(x) = 2π
|Σ| denotes the determinant of the variance-covariance matrix Σ and Σ−1 is just the
inverse of the variance-covariance matrix Σ. Again, this distribution will take maximum
values when the vector X is equal to the mean vector µ, and decrease around that max-
imum.

Example: Check Assignment-2 Q2: We are given a single n-dimensional random

vector X = (X1 , X2 , . . . , Xn ) drawn from a multivariate normal distribution, Nn (µ, Σ).
 
nθ1
 0 
µ =  .. 
 
 . 
0
n − 1 + θ22 −ϵ′
 
Σ=
−ϵ In−1

ϵ′ = (1, ...., 1)
| {z }
n−1times

n − 1 + θ22 (−1 −1 ... −1)#n − 1times

 

 −1 (1 0 ... 0)#n − 1times  
 −1 (0 1 ... 0)#n − 1times 
Σn×n = 
 

 
 . (... ... ... ...)#n − 1times 
−1 (0 0 ... 1)#n − 1times
treating Σ as 2-D matrix:
determinant(Σ) = ad−bc = ((n−1+θ22 )×|In−1 |)−(−ϵ′ ×−ϵ) = (n−1+θ22 )−(n−1) = θ22
On calculating, |Σ| = θ22
n
!
1 n2 2 1X 2
f (X|θ) = exp − (X − θ 1 ) − X
(2π)n/2 θ2 2θ22 2 i=2 i
We can separate this into two functions:
 " n
!#
n2
 
1 1 1 X
f (X|θ) = √ exp − 2 (X − θ1 )2 · (n−1)/2
exp − Xi2
θ 2π 2θ (2π) 2
| 2 {z 2 } | {z i=2
}
g(T (X)|θ) h(X)

By the Fisher-Neyman Factorization Theorem, T (X) = X is a sufficient statistic for

θ = (θ1 , θ2 ).

5
2.2 Binomial expansion
Here is the negative binomial expansion for (1−x)−k .For any real number x where |x| < 1
and any positive integer k, the expansion is given by the infinite series:
∞  
−k
X n+k−1
(1 − x) = xn
n=0
n

Because of the symmetry of combinations, n+k−1 n+k−1

n
is exactly the same as k−1
. So,
you will often see it written interchangeably as:
∞  
−k
X n+k−1
(1 − x) = xn
n=0
k−1

2.3 The Gamma Integral

This integral is essential for calculating expectations (moments) for Exponential, Gamma,
Chi-Square, and Normal distributions.
General Formula:
Z ∞
Γ(n + 1)
xn e−ax dx = , for n > −1, a > 0
0 an+1

Note: Γ(n) = (n − 1)! if n is an integer.

Examples from Course Context:

• Standard Gamma: Z ∞
Γ(α)
xα−1 e−βx dx =
0 βα

• Moment Calculation (shifted power):

Z ∞
Γ(α + r)
xα+r−1 e−βx dx =
0 β α+r

This is often used when calculating E[X r ] for X ∼ Gamma(α, β).

2.4 The Beta Integral

Essential for Bayesian inference and order statistics distributions.
Z 1
Γ(α)Γ(β)
xα−1 (1 − x)β−1 dx = B(α, β) =
0 Γ(α + β)

2.5 Gaussian Integral

Essential for Normal distribution normalization.
Z ∞ r
2 π
e−ax dx =
−∞ a

6
How to Think & Solve
Strategy (The ”Kernel” Method): Instead of integrating by parts multiple
times:

1. Identify the Kernel: Look at the integrand. Does it look like xpower e−const·x ?
This is a Gamma kernel.

2. Match Parameters: Set the exponent of e to be −βx and the exponent of

x to be α − 1.
Γ(α)
3. Use the Identity: The integral equals βα
.
R∞
4. Example: To find E[X 2 ] for X ∼ Exp(λ), you need 0
x2 λe−λx dx. Kernel
is x2 e−λx . Here a = λ, n = 2. Result: λ · Γ(3)
λ3
= λ22 .

7
3 Polar Jacobian & Sampling Distributions
3.1 Key Concepts
To derive the distribution of sums of squares (like χ2 ), we transform independent nor-
mal variables into polar coordinates (r, θ1 , . . . ). The Jacobian of this transformation is
recursive.

3.2 Formula Sheet

• Jacobian (n dimensions): Jn = rn−1 sinn−2 (θ1 ) . . . sin(θn−2 ).

• Chi-Square (χ2n ): If Xi ∼ N (0, 1), then

P 2
Xi ∼ χ2n .

• Student’s t: t = √Z where Z ∼ N (0, 1), V ∼ χ2n .

V /n

• Fisher’s t: used for comparing two means (X̄1 − X̄2 ).

How to Think & Solve

Strategy:

1. DegreesPof Freedom (df ): When asking for the distribution of a sum of

squares (Xi − X̄)2 :

• Count the variables (n).

• Subtract linear constraints (e.g.,
P
(Xi − X̄) = 0 is 1 constraint).
• Resulting df = n − 1.

2. Identifying Distributions:

• Sum of squared Normals → χ2 .

• Normal / χ2 → t-distribution.
p

• Ratio of χ2 s → F -distribution (often implied in variance ratios).

8
4 Review of Order Statistics
4.1 Key Concepts
Order statistics are the sorted observations: X(1) ≤ X(2) ≤ · · · ≤ X(n) .
Let X1 , X2 , . . . , Xn be an independent and identically distributed (i.i.d.) random
sample of size n from a continuous population with probability density function f (x) and
cumulative distribution function F (x).

4.2 Probability Density Function of the 1st Order Statistic (Min-

imum)
Let X(1) = min(X1 , X2 , . . . , Xn ). The PDF is given by:
fX(1) (x) = n[1 − F (x)]n−1 f (x)
Intuition: For the minimum to be exactly at x, one observation must be at x—giving
the f (x) term—and the remaining n − 1 observations must be strictly greater than x—
giving the [1 − F (x)]n−1 term. The multiplier n accounts for the fact that any of the n
observations could be the minimum.

4.3 Probability Density Function of the n-th Order Statistic

(Maximum)
Let X(n) = max(X1 , X2 , . . . , Xn ). The PDF is given by:
fX(n) (x) = n[F (x)]n−1 f (x)
Intuition: For the maximum to be exactly at x, one observation must be at x—giving
the f (x) term—and the remaining n − 1 observations must be less than or equal to x—
giving the [F (x)]n−1 term. Again, n accounts for the n possible variables that could act
as the maximum.

4.4 Joint Probability Density Function of the 1st and n-th Or-
der Statistics
Let X(1) be the minimum and X(n) be the maximum of an i.i.d. sample of size n. By
substituting i = 1 and j = n into the general joint distribution formula, the expression
simplifies significantly. For x < y, the joint PDF is given by:
fX(1) ,X(n) (x, y) = n(n − 1)[F (y) − F (x)]n−2 f (x)f (y)
Intuition: We are again dividing the observations into buckets relative to x and y:
• Exactly 1 observation is exactly at the minimum x (giving f (x)).
• Exactly 1 observation is exactly at the maximum y (giving f (y)).
• The remaining n − 2 observations must fall strictly between x and y (giving [F (y) −
F (x)]n−2 ).
The leading coefficient n(n − 1) simply represents the number of ways to choose which of
the n observations will be the minimum, and which of the remaining n − 1 observations
will be the maximum.

9
4.5 Formula Sheet
• PDF of Max (X(n) ): fn (x) = n[F (x)]n−1 f (x).

• Expectation of max = x · fX(n) (x)dx : similarly for min also

R

• PDF of Min (X(1) ): f1 (x) = n[1 − F (x)]n−1 f (x).

• Joint PDF: n! f (xi ) for x1 < · · · < xn .

Q

How to Think & Solve

Strategy:

1. Don’t Memorize, Derive:

• For Max ≤ x, ALL items must be ≤ x. Probability is F (x)n .

• For Min > x, ALL items must be > x. Probability is (1 − F (x))n .

2. Differentiation: Once you have the CDF from step 1, differentiate with
respect to x to get the PDF.

3. Support: Always remember the support limits. If X ∼ U (0, θ), then 0 <
X(1) < X(n) < θ.

10
5 Sufficiency
5.1 Intuition
A statistic T (X) is sufficient if it compresses the data X without losing any information
about the unknown parameter θ.
• Conceptually: Once you know T , knowing the individual data points X adds no
new knowledge about θ.

• Prediction View: You do not need the original data to estimate θ; the statistic
T is enough.

5.2 Formal Definition

Let X be a random sample from a distribution family P = {pθ (x) : θ ∈ Ω}. A statistic
T (X) is said to be sufficient for θ if the conditional distribution of X given T = t is
independent of θ for all values of t.

Pθ (X = x | T (X) = t) is independent of θ.

5.3 Example: Bernoulli Trials

P
Let X1 , . . . , Xn ∼ Bernoulli(θ). Let T (X) = Xi .
• Numerator (Sample PDF): fX (x|θ) = θ
P P
xi
(1 − θ)n− xi
= θt (1 − θ)n−t .

• Denominator (Statistic PDF): T ∼ Binomial(n, θ).

 
n t
fT (X) (t|θ) = θ (1 − θ)n−t
t

• The Ratio:
fX (x|θ) θt (1 − θ)n−t 1
= n t

= n


fT (X) (t|θ) t
θ (1 − θ)n−t t

1
Since does not contain θ, T is mathematically sufficient.
(nt)

5.4 Sufficiency in Hypergeometric Model – Complete Clarifica-

tion
Problem Setup
• Total objects: N (known)

• Defective objects: D (unknown)

• Sample size: n (without replacement)

(
1 if i-th draw is defective
• Xi =
0 otherwise

11
 Define the statistic: n
X
T = Xi
i=1

which counts the number of defectives in the sample.

Step 1: Distribution of T
Since sampling is without replacement,

T ∼ Hypergeometric(N, D, n).

The probability of observing exactly t defectives is:

D N −D



t n−t
Pθ (T = t) = PD (T = t) = P (T = t | D) = N

.
n
Explanation:

• Choose t defectives from D: D

t

• Choose n − t non-defectives from N − D: N −D

n−t

• Total samples of size n from N : Nn

Important Clarification
n


1. Why no division by t
here?
Because the hypergeometric model selects an unordered subset of size n.
Each subset corresponds to exactly one vector x. Therefore,

P (X = x | D) = P (T = t | D).

No extra division is needed.

2. When does nt appear?

It appears when computing:

P (X = x | T = t, D).
n


Given that T = t, all t
arrangements of t ones are equally likely.
Hence:

1
P (X = x | T = t, D) = n .


t

This does NOT depend on D.

12
Definition of Sufficient Statistic
A statistic T (X) is sufficient for parameter θ if:

f (x | T (x) = t, θ)
does not depend on θ.

Factorization Theorem
T (X) is sufficient for θ if:

f (x | θ) = g(T (x), θ) h(x),

where:

• g depends on data only through T (x)

• h does not depend on θ

Applying Factorization
We have:
D N −D



t n−t
P (X = x | D) = N

= g(t, D) · 1.
n
Thus it factors as required.

Final Conclusion
n
X
T = Xi
i=1

is a sufficient statistic for D.

Key Insight: All information about D is contained in the number of defectives observed,
not in their order.

5.5 Neyman-Fisher Factorization Theorem (FNFT)

While the definition is useful for checking sufficiency, it is difficult to use for finding a
statistic. The Factorization Theorem is the operational tool.
Theorem: A statistic T (X) is sufficient for θ if and only if the joint PDF/PMF pθ (x)
can be factored as:
pθ (x) = gθ (T (x)) · h(x)
where:

• gθ (T (x)) depends on the data only through the statistic T (and involves θ).

• h(x) depends only on the data x (and is independent of θ).

13
5.6 Formula Sheet
• Neyman-Fisher Factorization: T is sufficient iff pθ (x) = gθ (T (x)) · h(x).

• Minimal Sufficiency: T is minimal if pθ (x)

pθ (y)
is independent of θ ⇐⇒ T (x) = T (y).

How to Think & Solve

Strategy:

1. Factorization is Key:

• Write the joint Likelihood

Q
f (xi ).
• Use algebra (exp(a + b) = exp(a) exp(b)) to separate terms.
• Group everything with θ into one block. The data x inside that block is
your T .
• Group everything else into h(x).

2. Minimal Sufficiency Test: Calculate the ratio of likelihoods

P forPtwo sam-
ples
P x and y. If the ratio is constant free of θ only when xi = yi , then
Xi is Minimal Sufficient.

14
6 Completeness & Complete Sufficiency
6.1 Key Concepts
• Completeness: Eθ [g(T )] = 0 =⇒ g(T ) = 0 almost everywhere ∀θ. No non-zero
function of T has 0 expectation.

• Complete Sufficient Statistic (CSS): A statistic that is both Complete and

Sufficient.

• Exponential Family: If pθ (x) = K(θ)eQ(θ)T (x) h(x), then T is CSS.

6.2 Completeness in family

Let X be a random variable (or random vector - not a statistic) with a probability
distribution Pθ belonging to a family of distributions P = {Pθ : θ ∈ Θ}, where Θ is the
parameter space.
The family of distributions P is said to be complete if, for any measurable function
g(x), the condition:
Eθ [g(X)] = 0 for all θ ∈ Θ
implies that:
Pθ (g(X) = 0) = 1 for all θ ∈ Θ
See example in assignment 3 question 3

How to Think & Solve

Strategy:

1. The Shortcut: Check if the distribution is an **Exponential Family** (Nor-

mal, Poisson, Binomial, Gamma). If yes, the ”natural” sufficient statistic
X 2 ) is automatically Complete Sufficient.
P P
(e.g., X or

2. The Integral Definition: If it’s not exponential family (like Uniform), then
check for integral transformations.

3. Via differentiation: If integral transformation also not present, differentiate

w.r.t. the parameter (ex - see checking the completeness of X(n) for U (0, θ)
in problems section)

15
7 Integral Transforms & Uniqueness (Tools for Com-
pleteness)
7.1 Definition
An integral transform of a function f (x) with kernel g(x, t) is defined as:
Z
ϕ(t) = g(x, t)f (x)dx

These transforms are crucial for proving the completeness of statistics in continuous
families.

7.2 Common Transforms

• Unilateral Laplace Transform:
Z ∞
ϕ(t) = e−tx f (x)dx, t>0
0

• Bilateral Laplace Transform:

Z ∞
ϕ(t) = e−tx f (x)dx
−∞

• Mellin Transform: Z ∞
ϕ(t) = xt−1 f (x)dx
0

• Stieltjes Transform: Z ∞
f (x)
ϕ(t) = dx
0 t+x

7.3 The Uniqueness Property (The Main Result)

The utility of these transforms in statistical inference comes from their **Uniqueness
Property**: If ϕ1 (t) and ϕ2 (t) are integral transforms of functions f1 (x) and f2 (x) re-
spectively, then:

ϕ1 (t) = ϕ2 (t) ∀t ⇐⇒ f1 (x) = f2 (x) almost everywhere

ImplicationR for Completeness: If an integral transform of a function f (x) is identically

zero (i.e., g(x, t)f (x)dx = 0 for all t), then f (x) = 0 almost everywhere.

7.4 Applications to Normal Families

2 /2
Example 1: Normal Mean N (θ, 1) The pdf is f (x) ∝ e−(x−θ) . To check complete-
ness, set Eθ [h(X)] = 0: Z ∞
2 2
h(x)e−(x −2xθ+θ )/2 dx = 0
−∞

16
 Z ∞  
−θ2 /2 −x2 /2
e h(x)e eθx dx = 0
−∞
2
This is a Bilateral Laplace Transform of the function g(x) = h(x)e−x /2 with param-
eter t = −θ (or similar). By uniqueness, g(x) = 0 =⇒ h(x) = 0 a.e. Thus, the family is
complete.
Example 2: Normal Variance N (0, θ) Here, Eθ [X] = 0 for all θ, so X itself is not
complete. However, T = X 2 is complete. The expectation Eθ [h(T )] = 0 leads to:
Z ∞
h(t)t−1/2 e−t/2θ dt = 0
0

This is a Unilateral Laplace Transform (with parameter 1/2θ). By uniqueness, h(t) =

0 a.e..

How to Think & Solve

Strategy for Using Integral Transforms:

1. RWrite the Expectation: Start with definition of completeness:

h(x)f (x; θ)dx = 0.

2. Isolate the Kernel: Use algebra to group terms involving x and θ into the
exponent (e.g., eθx ). Move terms depending only on θ outside the integral.

3. Identify the Transform: Look at the structure.

• Does it look like e−tx (. . . )dx? It’s Laplace.

R

• Does it look like xt−1 (. . . )dx? It’s Mellin.

R

4. Invoke Uniqueness: State ”By the uniqueness property of the [Name]

Transform, the function inside the integral must be zero.”

17
8 Point Estimation I: Unbiasedness & UMVUE
8.1 Key Concepts
• Unbiased: E[θ̂] = θ.

• UMVUE: The unbiased estimator with minimum variance.

• Lehmann-Scheffé Theorem: If T is Complete Sufficient and U is an unbiased

estimator based on T , then U is the UMVUE.

How to Think & Solve

Strategy (Lehmann-Scheffé) [Your Exam Recipe]: Instead of checking your
estimator against every other possible estimator in the universe to prove it has the
minimum variance, you just follow these steps:

1. Step 1: Find a Complete Sufficient Statistic T (often using the Exponential

Family form).

2. Step 2: Create some function of it, say h(T ).

3. Step 3: Tweak h(T ) so that its expected value equals the parameter you
want to estimate (E[h(T )] = g(θ)).

4. Step 4: Stop. Lehmann-Scheffé guarantees that the h(T ) you just built is
the UMVUE.

5. Harder Cases: If simple algebra fails, find any simple unbiased estimator
(e.g., I(X1 = 1)) and compute E[Simple | T ].

18
9 Theoretical Results on UMVUE
Derived from the method of covariance and orthogonality properties.

9.1 1. Uniqueness of UMVUE

If a UMVUE exists, it is unique (almost everywhere).

• Proof Concept: If T1 and T2 were both UMVUEs, they would have the same vari-
ance. The correlation coefficient ρ(T1 , T2 ) must be 1. This implies a linear relation-
ship T1 = aT2 + b. Consistency requires T1 = T2 .

9.2 2. The Covariance Method (Necessary & Sufficient)

Let U0 be the class of all unbiased estimators of zero (i.e., E[h(X)] = 0).

U0 = {h(X) : E [h(X)] = 0, V ar (h(X)) < ∞, ∀θ ∈ Ω},

θ θ

Let
Ug = {T (X) : E [T (X)] = g(θ), V ar (T (X)) < ∞}
θ θ

be the class of unbiased estimators of g(θ).

An unbiased estimator T of g(θ) is UMVUE if and only if :

Covθ (T, h) = 0 for all h ∈ U0 and for all θ ∈ Ω

• Intuition: The UMVUE must be uncorrelated with ”noise” (estimators of zero). If

it were correlated with some error h, we could subtract a multiple of h from T to
reduce the variance.

9.3 3. Algebra of UMVUEs

If T1 , . . . , Tk are UMVUEs for functions g1 (θ), . . . , gk (θ), then:

1. Linearity: ki=1 ai Ti is the UMVUE for ki=1 ai gi (θ).

P P

2. Product Rule: If T1 and T2 are UMVUEs of their expectations (E[T1 ] and E[T2 ]),
then the product T1 T2 is the UMVUE for E[T1 T2 ] (provided second-order moments
exist).

3. Power Rule: If T is UMVUE of E[T ], then T k is the UMVUE of E[T k ].

9.4 4. Correlation and Efficiency

Let T be the UMVUE and T ′ be any other unbiased estimator with efficiency e =
V ar(T )/V ar(T ′ ). The correlation between the UMVUE and T ′ is determined by the
efficiency: √
ρ(T, T ′ ) = e
Since efficiency is positive, the correlation is always non-negative (ρ ≥ 0).

19
9.5 5. Symmetry Principle
If the sample X1 , . . . , Xn is i.i.d., the UMVUE must be a symmetric function of the
observations.

• Strategy: If you find an unbiased estimator that uses only X1 (not symmetric),
you can improve it by symmetrizing (averaging over all permutations), effectively
applying Rao-Blackwellization.

How to Think & Solve

Strategy for UMVUE Properties:

1. Check for Zero-Estimators: If asked to prove T is UMVUE without find-

ing a sufficient statistic, show Cov(T, U ) = 0 for any U where E[U ] = 0.

2. Use Linearity: If you know X̄ is UMVUE for µ, then 3X̄ +2 is automatically

UMVUE for 3µ + 2. You do not need to re-derive it.

3. Use Symmetry: If estimating a mean and someone suggests T = X1 , reject

it. The UMVUE must treat all Xi equally (like X̄).

20
10 Point Estimation II: MLE & Invariance
10.1 Maximum Likelihood Estimation (MLE)
Maximize the Likelihood function L(θ). The value θ̂ that maximizes L(θ) is the MLE.

10.2 The Invariance Property

If θ̂ is the MLE of θ, and g(θ) is a function that is one-to-one and differentiable, then the
MLE of g(θ) is simply g(θ̂).

How to Think & Solve

Strategy:

1. Regular Case (Normal/Poisson):

• Take log (ℓ = ln L).

• Differentiate w.r.t θ, set to 0.
• Check 2nd derivative < 0.

2. Using Invariance: If you need the MLE of e−2θ and know θ̂, the answer is
simply e−2θ̂ .

3. Boundary Case (Uniform):

• If the domain depends on θ (e.g., 0 < x < θ), differentiation usually

fails.
• Logic: Choose the value of θ (usually X(n) ) that makes the likelihood
non-zero and as large as possible.

21
11 Method of Moments Estimation (MME)
11.1 Key Concepts
MME is an alternative to MLE that is often mathematically simpler, though it may
be less efficient. The core idea is to match the theoretical population moments to the
observed sample moments.

11.2 Definitions
• Population Moment (k-th): µ′k (θ) = E[X k ]. The first moment is the mean µ,
the second is E[X 2 ] = Var(X) + µ2 .

• Sample Moment (k-th): Mk = n1 ni=1 Xik . The first sample moment is the
P
sample mean X̄.

11.3 The Procedure

If we need to estimate k parameters θ = (θ1 , . . . , θk ):

1. Calculate the first k true moments E[X], E[X 2 ], . . . , E[X k ] in terms of θ.

2. Calculate the first k sample moments from the data (M1 , . . . , Mk ).

3. Set up a system of k simultaneous equations:

n
1X j
E[X j ] = X for j = 1, . . . , k
n i=1 i

4. Solve this system for θ1 , . . . , θk .

11.4 Examples
• Uniform Distribution (U (0, θ)):

– True Moment: E[X] = θ/2.

– Equation: θ/2 = X̄.
– Result: θ̂M oM = 2X̄. (Note: This differs from the MLE X(n) ).

• Normal Distribution (N (θ1 , θ2 )):

– Parameters: θ1 = µ, θ2 = σ 2 .
– 1st Moment Eq: θ1 = n1
P
Xi = X̄.
– 2nd Moment Eq: E[X 2 ] = Var(X) + (E[X])2 = θ2 + θ12 .
– Set θ2 + θ12 = n1
P 2
Xi .
– Solve: θ̂2 = n1 Xi − (X̄)2 = n1 (Xi − X̄)2 .
P 2 P

– Result: MME matches the MLE for the Normal distribution.

22
How to Think & Solve
Strategy:

1. Count the Unknowns: Determine how many unknown parameters (k)

exist.

• If k = 1 (e.g., Poisson λ), you only need E[X].

• If k = 2 (e.g., Gamma α, λ or Normal µ, σ 2 ), you need E[X] and E[X 2 ].

2. Derive Moments: Write down the expectations in terms of parameters.

• Remember: E[X 2 ] = V ar(X) + (E[X])2 .

3. Equate and Solve:

• Set E[X] = X̄.

• Set E[X 2 ] = 1
Xi2 .
P
n
• Solve the algebra. For 2 parameters, solve for the first parameter using
the first equation, then substitute it into the second equation.

23
12 Cramér-Rao Lower Bound (CRLB)
12.1 C-R Regularity Conditions
The Cramér-Rao bound is valid only if the family of distributions P = {pθ (x) : θ ∈ Ω}
satisfies specific conditions:

1. Ω is an open interval (not a single point).

∂
2. The partial derivative p (x)
∂θ θ
exists for all θ.

3. Crucial: The support of X (where pθ (x) > 0) must not depend on θ. (e.g., CRLB
fails for U (0, θ)).
∂
R R ∂
4. Differentiation under the integral sign is permitted: ∂θ pθ (x)dx = ∂θ pθ (x)dx.

5. The Fisher Information I(θ) exists and is positive.

12.2 The Score Function & Fisher Information

The Score Function (S):
∂
S(X, θ) = ln pθ (X)
∂θ
Properties: Eθ [S] = 0 and V arθ (S) = I(θ).
Fisher Information (I(θ)): Measures the amount of information a single observa-
tion carries about θ.
" 2 #  2 
∂ ∂
I(θ) = Eθ ln pθ (X) = −Eθ ln pθ (X)
∂θ ∂θ2

(Note: The second form using the second derivative is usually easier to calculate).

12.3 The Inequality

Let T be an unbiased estimator of a differentiable function g(θ). Then:

[g ′ (θ)]2
V arθ (T ) ≥
nI(θ)
where n is the sample size.

12.4 Step-by-Step Calculation Guide

Based on the simplified method:

1. Start with the pdf f (x; θ) for a single observation.

2. Take the natural log: ln f (x; θ).

∂
3. Score: Differentiate w.r.t θ: ∂θ
ln f .
∂2
4. Differentiate again: ∂θ2
ln f .

24
 ∂ 2
5. Find Expected Value: E[ ∂θ 2 ln f ]. (This removes the x’s).

6. Fisher Info: Fisher Information is negative of the expected value of the single-
∂2
sample second derivative. Ie: I(θ) = −E[ ∂θ 2 ln f ].

[g ′ (θ)]2
7. CRLB: The bound is nI(θ)
.

12.5 Efficiency & Attainment

• Efficiency (e): The ratio of the CRLB to the actual variance of the estimator.
e = VCRLB
ar(T )
. If e = 1, the estimator is **Efficient** (or Minimum Variance Bound
Estimator).

• Attainment Condition: The variance equals the CRLB **if and only if** the
distribution belongs to the **Exponential Family** and T is the natural sufficient
statistic.

• Corollary: If an estimator attains the CRLB, it is a Complete Sufficient Statistic.

12.6 Limitations & Failure Cases

• Support Depends on θ: For f (x) = e−(x−θ) , x > θ, the CRLB does not apply.
In this case, estimators like X(1) − 1/n can have variance 1/n2 , which is actually
*lower* than the ”theoretical” CRLB of 1/n.

• Non-Attainment: Sometimes UMVUEs exist but do not attain the bound. Ex-
ample: Estimating θ2 in N (θ, 1). The UMVUE exists (X 2 − 1/n), but its variance
is strictly larger than the CRLB.

How to Think & Solve

Strategy:

1. Regularity Check: First, look at the range of x. If it involves θ (e.g.,

0 < x < θ), write ”Regularity conditions violated, CRLB does not apply.”

2. Calculus: Use the second derivative method for I(θ)—it’s usually faster.

3. Transformations: If estimating g(θ) = e2θ , remember to multiply the base

bound by [g ′ (θ)]2 = (2e2θ )2 .

4. Sample Size: Don’t forget to divide by n (or multiply I(θ) by n).

25
13 Selected Assignment & Exam Problems (With
Solutions)
13.1 Assignment 2: Problem 1 Solution
Problem Statement
Suppose there are N objects of which D are defective. N is known, but D is
unknown. Let n objects be drawn without replacement. Let X = (X1 , . . . , Xn )
where: (
1 if i-th draw is defective
Xi =
0 otherwise
Find a sufficient statistic for D using: 1. The Definition. 2. The Neyman-Fisher
Factorization Theorem (FNFT).

13.1.1 The Model

This describes
Pn sampling without replacement, which follows a Hypergeometric setup.
Let T = i=1 Xi be the number of defective items in the sample. The probability of
observing
P a specific sequence x = (x1 , . . . , xn ) containing exactly t defectives (where
t = xi ) is:
D N −D



1
PD (X = x) = N
· t nn−t

n t

−D

N
Reasoning: There are n total samples. The term Dt Nn−t


is the number of groups
n


with t defectives. We divide by t because all specific orderings (sequences) of those t
defectives are equally likely.

13.1.2 Solution 1: Using the Definition

We compute the conditional probability P (X = x | T = t).

P (X = x, T = t)
P (X = x | T = t) =
P (T = t)
P P
Note that if xi ̸= t, this probability is 0. If xi = t, then the event {X = x} implies
{T = t}, so P (X = x, T = t) = P (X = x).
The denominator P (T = t) is the standard Hypergeometric PMF (probability of
getting t defectives regardless of order):
D N −D



t n−t
P (T = t) = N


n

Substituting the expression for P (X = x) derived in the Model section:

−D
(Dt)(Nn−t
1 )
n N
( ) (n)t 1
P (X = x | T = t) = D N −D
= n

( t )( n−t ) t
(Nn )

26
 1
Conclusion: The result depends only on the sample size n and the value t. It is
(nt)
independent of the parameter D. Therefore, by definition, T = ni=1 Xi is a sufficient
P
statistic for D.

13.1.3 Solution 2: Using Factorization Theorem (FNFT)

We write the joint PMF of the sample x:
PD −D
NP



xi n− xi
PD (x) = N Pn



n xi

We attempt to factor this into gD (T (x)) · h(x).

Let:   
D N −D
gD (T (x)) = P P
xi n − xi
(This
P part contains the parameter D and depends on data only through the sum T =
xi ).
Let:
1
h(x) = N
n
· I(xi ∈ {0, 1})
P
n xi

(This part contains N and n, which are known constants, but does not depend on the
unknown parameter D). P
Conclusion: Since the PMF factors into a part depending on P
D only through Xi
n
and a part independent of D, by the Factorization Theorem, T = i=1 Xi is a sufficient
statistic for D.

13.2 The joint distribution of X(1) &X(n) in U (θ1 , θ2 )

Let X1 , . . . , Xn be i.i.d. continuous random variables with pdf f (x) and CDF F (x). Define

U = X(1) = min(X1 , . . . , Xn ), V = X(n) = max(X1 , . . . , Xn ).

By definition of joint density,

P (u < U < u + ∆u, v < V < v + ∆v)
fU,V (u, v) = lim .
∆u,∆v→0 ∆u∆v
For this event to occur:

1. One observation must fall in (u, u + ∆u),

2. One observation must fall in (v, v + ∆v),
3. The remaining n − 2 observations must lie in (u, v).

The number of ways to choose the minimum and maximum observations is

n(n − 1).

Using independence,

P (Xi ∈ (u, u + ∆u)) ≈ f (u)∆u,

27
 P (Xj ∈ (v, v + ∆v)) ≈ f (v)∆v,
P (u < Xk < v) = F (v) − F (u).
Thus,
P ≈ n(n − 1) f (u)∆u f (v)∆v [F (v) − F (u)]n−2 .
Dividing by ∆u∆v and taking limits gives

fX(1) ,X(n) (u, v) = n(n − 1) f (u) f (v) [F (v) − F (u)]n−2 , u < v.

Suppose X1 , . . . , Xn are i.i.d. Uniform(θ1 , θ2 ). Then

1 x − θ1
f (x) = , F (x) = , θ1 < x < θ2 .
θ2 − θ1 θ2 − θ1
Using the general result

fX(1) ,X(n) (u, v) = n(n − 1) f (u) f (v) [F (v) − F (u)]n−2 ,

we compute
v−u
F (v) − F (u) = .
θ2 − θ1
Substituting into the formula,
 2  n−2
1 v−u
fX(1) ,X(n) (u, v) = n(n − 1) .
θ2 − θ1 θ2 − θ1
Simplifying,

n(n − 1)
fX(1) ,X(n) (u, v) = n
(v − u) n−2 , θ1 < u < v < θ2 .
(θ2 − θ1 )
The density is zero otherwise.

13.3 Distribution and Completeness of T = X(n) for Uniform(0, θ)

Let X1 , . . . , Xn ∼ Uniform(0, θ), θ > 0, and define

T = X(n) = max(X1 , . . . , Xn ).

Step 1: Understand what T ≤ t means

If the maximum is less than or equal to t, then every observation must be less than
or equal to t. Thus,

T ≤t ⇐⇒ X1 ≤ t, . . . , Xn ≤ t.
Therefore,

P (T ≤ t) = P (X1 ≤ t, . . . , Xn ≤ t).

Step 2: Use independence

28
 Since X1 , . . . , Xn are independent,
n
Y
P (X1 ≤ t, . . . , Xn ≤ t) = P (Xi ≤ t).
i=1

Step 3: Compute P (Xi ≤ t)

For a Uniform(0, θ) random variable,
t
P (Xi ≤ t) = , 0 < t < θ.
θ

Step 4: Raise to the power n

Hence,
 n
t
P (T ≤ t) = , 0 < t < θ.
θ
This is the CDF of T .

Step 5: Differentiate to obtain the pdf

tn
FT (t) = .
θn
Differentiating with respect to t,

d tn ntn−1
 
fT (t) = = , 0 < t < θ.
dt θn θn

Step 6: Completeness
Assume that for some function g,

Eθ [g(T )] = 0 ∀θ > 0.
Then
θ
ntn−1
Z
g(t) dt = 0.
0 θn
n
Multiplying both sides by θ ,
Z θ
g(t)ntn−1 dt = 0 ∀θ.
0
Define
Z θ
F (θ) = g(t)ntn−1 dt.
0

Since F (θ) = 0 for all θ, differentiating w.r.t θ gives (note: here we are able to
differentiate because it is function of θ, whereas in laplace/bilaplace etc cases - we
cannot differentiate like this because there the function is of variable ’t’ and also the
ranges are in ∞ & not a function of t: so when we differentiate - the inner function
of t is only differentiated & the integral sign remains as it is)

29
 F ′ (θ) = g(θ)nθn−1 = 0.
Because nθn−1 ̸= 0 for θ > 0, we conclude

g(θ) = 0.
Hence, T = X(n) is complete.

T = X(n) is sufficient and complete for θ.

13.4 Problem 1: Poisson Estimation (Mid-Sem 2025)

Source: 2025 Mid Semester Examination, Q1.
Problem: Let X1 , . . . , Xn be a random sample from Poisson(λ).

(a) Show that there does not exist any unbiased estimator of λ−1 .

(b) Find the Method of Moments estimator (MME) of λ.

(c) Find the CRLB for the variance of an unbiased estimator of λ. Prove MME is
UMVUE.

Solution:
e−λ λx
(a) Suppose d(X) is an unbiased estimator for 1/λ. Then E[d(X)] = ∞
P
x=0 d(x) x! =
−1 λ
P∞ d(x) x −1 λ −1 λ2 1
λ . Multiply by e : x=0 x! λ = λ e = λ (1+λ+ 2! +. . . ) = λ +1+. . . The
LHS is a power series with non-negative powers of λ (x ≥ 0). The RHS contains
λ−1 . Coefficients cannot match. Thus, no such estimator exists.

(b) MME: Population mean E[X] = λ. Sample mean is X̄. Equating them: λ̂M M E =
X̄.

(c) CRLB: f (x) = e−λ λx /x!. ln f = −λ + x ln λ − ln x!. ∂ ∂λ

ln f
= −1 + λx . I(λ) =
X 1 1 1 1
2
E[( λ − 1) ] = λ2 V ar(X) = λ2 (λ) = λ . CRLB for sample size n is nI(λ) = nλ .
V ar(λ̂M M E ) = V ar(X̄) = nλ . Since variance equals CRLB, X̄ is UMVUE.

13.5 Problem 2: Bernoulli CRLB (Assignment 6)

Source: Assignment 6, Q1.
Problem: Xi ∼ Bernoulli(θ). Find Fisher Information and CRLB.

(a) Fisher Information: f (x) = θx (1 − θ)1−x . ln f = x ln θ + (1 − x) ln(1 − θ).

∂ ln f 2 n(1−θ)
∂θ
= xθ − 1−x
1−θ
. ∂ ∂θln2 f = − θx2 − (1−θ)
1−x nθ
2 . I(θ) = −nE[2nd Deriv] = θ 2 + (1−θ)2 =
n n n
θ
+ 1−θ = θ(1−θ) .

1 θ(1−θ)
(b) CRLB: CRLB = I(θ)
= n
.
θ(1−θ)
(c) Attainment: V ar(X̄) = n
. Since Variance = CRLB, the UMVUE X̄ attains
the bound.

30
13.6 Problem 3: Normal Distribution & MSE (Mid-Sem 2025)
Source: 2025 Mid Semester Examination, Q2.
Problem: Xi ∼ N (µ, σ 2 ).

(a) Derive MLEs for µ and σ 2 .

(b) Compare Mean Squared Errors (MSE) for the MLE of σ 2 and the unbiased estimator
T1 = S 2 . Which is smaller?

Solution:
2 1
(Xi − X̄)2 .
P
(a) MLEs: µ̂ = X̄ and σ̂M LE = n

(b) MSE Comparison:

(n−1)S 2
• For S 2 (Unbiased): M SE(S 2 ) = V ar(S 2 ). We know σ2
∼ χ2n−1 . V ar(S 2 ) =
σ4 σ4 2σ 4
(n−1)2
V ar(χ2n−1 ) = (n−1)2 (2(n − 1)) = n−1 .

• For MLE σ̂ 2 : It is n−1

n
S 2 . It is biased. M SE(σ̂ 2 ) = E[( n−1
n
S 2 − σ 2 )2 ]. Calcu-
2n−1 4
lation yields M SE = n2 σ .
• Conclusion: 2n−1
n2
< 2
n−1
for all n > 1. The MLE has smaller MSE despite
being biased.

13.7 Problem 4: UMVUE for Uniform Family (Assignment 4)

Source: Assignment 4, Q1.
Problem: Xi ∼ U (θ1 , θ2 ) where both are unknown.

(a) Identify a complete sufficient statistic (CSS).

(b) Find the UMVUE of θ1 .

Solution:

(a) CSS: The pair of order statistics T = (X(1) , X(n) ) is Complete Sufficient for (θ1 , θ2 ).

(b) UMVUE: We need a function of T such that its expectation is θ1 . From Source
2 −θ1 2 −θ1
95 (Example 5), we have derived: E[X(1) ] = θ1 + θn+1 and E[X(n) ] = θ2 − θn+1 .
Solving this linear system for θ1 , we get the UMVUE:
nX(1) − X(n)
θ̂1 =
n−1

13.8 Problem 5: CRLB Not Attained (Assignment 6)

Source: Assignment 6, Problem 4.
Problem: Xi ∼ N (θ, 1). Estimate g(θ) = θ2 .

(a) Find CRLB for unbiased estimators of θ2 .

(b) Find UMVUE.

(c) Show UMVUE does not attain CRLB.

31
 Solution:
(2θ)2
(a) CRLB: Fisher Info for N (θ, 1) is n. g(θ) = θ2 =⇒ g ′ (θ) = 2θ. CRLB = n·1
=
4θ2
n
.

(b) UMVUE: We found earlier (Problem 1, Ex 2 in notes) that X 2 − 1/n is UMVUE

for θ2 .

(c) Check: V ar(X̄ 2 − 1/n) = V ar(X̄ 2 ). Using normal moments (E[Z 4 ] = 3), variance
2 2
is 4θn + n22 . This is strictly greater than the CRLB ( 4θn ). Thus, the CRLB is not
attained.

32