Statistical Inference Notes

March 11, 2026

1 Introduction to Inference

1.1 The Inference Cycle

The fundamental process of statistics flows between the Population and the Sample. We
use the partial information derived from the sample to understand the total information of
the population.

Population Sampling Sample Data

(The Truth) (Observation)
g( x ) Probability X1 , . . . , X n

Analysis

Statistics
Estimation
(Calculation)
Inference x̄, s2

Definition 1.1: Statistical Inference

The method of inferring properties about the population based on the sample is called
Statistical Inference.

Definition 1.2: Random Variable

Mathematically, the "data" comes from a function that maps the sample space (Ω) to
real numbers (R):
f :Ω→R
This function is called a Random Variable.
S TATISTICAL I NFERENCE L ECTURE N OTES

1.2 Example Calculation

Given a sample dataset: S = {11, 2, 139}.

Example: Basic Statistics
• Sample Mean (x̄):

11 + 2 + 139 152
x̄ = = ≈ 50.67
3 3

• Sample Range:
Max − Min = 139 − 2 = 137

2 Frequency and Distributions

2.1 Definitions
• Frequency: The count of occurrences of an event.

• Relative Frequency:

Frequency
≈ Probability (Ratio between 0 and 1)
Total Count

2.2 Population Example: Heights

Consider the population of all 18-year-old boys in India in 2025. Let their height be repre-
sented by a variable X. We can characterize this population using a function g( x ), known
as the Probability Density Function (PDF).
Properties of g( x ):

1. g( x ) ≥ 0 (Non-negative).
R∞
2. −∞ g( x ) dx = 1 (Total probability is 1).

To find the proportion of boys with height between 4 and 4.25 units:
Z 4.25
P(4 ≤ X ≤ 4.25) = g( x ) dx
4

2
S TATISTICAL I NFERENCE L ECTURE N OTES

g( x ) Area = Probability

4 .25 (x)
Height

3 Properties of Distribution Functions

3.1 Connection between Population and PDF

If g( x ) is the PDF, we define key population parameters:

• Median (m̃): The point that splits the area in half.

Z m̃
g( x ) dx = 0.5
−∞

• Mean (µ): The weighted average (center of mass).

Z ∞
µ= x · g( x ) dx
−∞

3.2 Key Question

If we know only the function of the graph (the theoretical model), can we derive properties of the data?
Will these match the calculations from the observed data?

Answer
Yes, we can derive theoretical properties (Parameters like µ, σ2 ) from the function
g( x ). However, these will not be exactly the same as the values calculated from a
finite sample.

• Values from g( x ) are Parameters (Fixed, True values).

• Values from Data are Statistics (Estimates).

3
S TATISTICAL I NFERENCE L ECTURE N OTES

4 Sampling Distributions

4.1 Population (P) vs. Sample

Population Characteristics:

1. A complete collection of measurements.

2. Non-Random (The population truth is fixed).

3. Characterized by g( x ) (PDF or PMF).

Sample Definition: A sample of size n consists of random variables X1 , X2 , . . . , Xn that are

i.i.d. (Independent and Identically Distributed).

Xi ∼ g ( x )

5 Inference on Parameters

In real-world scenarios, the population function is usually unknown. We assume a "shape"

(a parametric family) but do not know the specific values.
Example: Assume Population P ∼ N (µ, σ2 ).

• The shape is Normal (Bell curve).

• The parameters µ (mean) and σ2 (variance) are unknown.

5.1 The Goal

iid
We have a sample X1 , . . . , Xn ∼ N (µ, σ2 ). We need to infer (µ, σ2 ).

Estimators
To find these unknown parameters, we use Estimators derived from our sample data:
1. Estimator for Mean (µ): We use the Sample Mean (X̄):

1 n
n i∑
µ̂ = X̄ = Xi
=1

2. Estimator for Variance (σ2 ): We use the Sample Variance (S2 ). Note the division by
n − 1 to make it an unbiased estimator:
n
1
n − 1 i∑
σ̂2 = S2 = ( Xi − X̄ )2
=1

4
S TATISTICAL I NFERENCE L ECTURE N OTES

6 Parametric Inference

Definition 6.1: Parametric Inference

When it is assumed that samples X1 , X2 , . . . , Xn are i.i.d. according to some distribu-
tion G (θ ), where:

• G (·) is the known form of the CDF.

• θ is the vector of unknown parameters.

The task of inferring θ based on the sample is called Parametric Inference.

Definition 6.2: Modelling

The task of identifying the form of G (θ ) (the distribution family) is called Modelling.

7 Example 1: Agricultural Experiment

Scenario: A 25-acre land is split into 10,000 plots. A seed is planted in each. We want to
find the proportion of germination.

7.1 Methodology

We examine a small sample of n = 15 plots.

• Notation: 1 = Germinated, 0 = Not Germinated.

Data Collection:

Plot # 1 2 ... 15
Variable X1 X2 ... X15
Outcome 0 1 ... 1

7.2 The Model (Bernoulli)

Let θ be the true proportion of germinated seeds.

Xi ∼ Bernoulli(θ ); θ ∈ [0, 1]

P( Xi = 1) = θ, P ( Xi = 0 ) = 1 − θ

5
S TATISTICAL I NFERENCE L ECTURE N OTES

Bernoulli Model (Example: θ = 0.6)

1

0.8

Probability
0.6
0.6
0.4
0.4

0.2

0
Fail (0) Success (1)
Outcome

7.3 Estimation

1 n
n i∑
Estimate of θ ≈ X̄n = Xi
=1

8 Random Samples vs. Realizations

! Important: CRITICAL DISTINCTION

1. Random Sample (X1 , . . . , Xn ): These are discussed BEFORE the experiment.
They are random variables. We talk about their probability distributions.

2. Realizations (x1 , . . . , xn ): These are obtained AFTER the experiment. These are
fixed numbers. We talk about their specific values.

9 Example 2: Physics Experiment

Objective: Estimate the gravitational constant ’g’.

2l
Physical Law: g ≈ 4πT2
(where l = 0.75m).

9.1 Data

We take 4 repeated measurements of period T.

Data x = {1.82, 1.68, 1.61, 1.76} ( n = 4)

6
S TATISTICAL I NFERENCE L ECTURE N OTES

9.2 Modelling Assumptions

We assume the measurement error is random and normally distributed.

iid
Xi ∼ N (µ, σ2 )
Here, µ represents the true period Ttrue .

Model: N (µ, σ2 )
4 Mean
Theoretical Model
Observed xi
3
Density

0
1.6 µ̂ 1.85
Time (s)

9.3 Calculations (Inference)

Step 1: Estimate the Period (T̂) We estimate the true period µ using the sample mean x̄:
1.82 + 1.68 + 1.61 + 1.76 6.87
T̂ = x̄ = = = 1.7175 sec
4 4

Step 2: Estimate the Uncertainty (Variance)

4
1
4 − 1 i∑
s2 = ( xi − 1.7175)2 ≈ 0.0082
=1

Step 3: Estimate Gravity ( ĝ) Using l = 0.75 meters and T̂ = 1.7175 s:

4π 2 (0.75) 29.6088
ĝ = 2
= ≈ 10.03 m/s2
(1.7175) 2.9498

Conclusion

Based on our sample of 4 measurements, our point estimate for gravity is 10.03 m/s2 .
The difference from the standard 9.81 m/s2 is likely due to measurement error (vari-
ance) or systematic bias in the pendulum length.

7
S TATISTICAL I NFERENCE L ECTURE N OTES

10 Statistical Models and Statistics

Model:
X1 , X2 , . . . , X n ∼ G ( θ )
where θ is a parameter (Unknown Constant).
Goal: Need to infer θ based on X = ( X1 , X2 , . . . , Xn ).
Example: Bernoulli Statistic Example
• θ: Population proportion of germinated seeds.
# 1s in X1 ...Xn 1
• Statistic Tn (X) = n = n ∑ Xi (Sample Proportion).

Definition 10.1: Statistic

Any function of the samples (only) { X1 , X2 , . . . , Xn } is called a Statistic.

T:X →R

It must not depend on any unknown parameters.

Examples:

1. T1 (X) = X̄n (Valid)

2. T2 (X) = maxi Xi − mini Xi (Valid)

3. [Critical Example] T3 (X; θ ) = ∑in=1 ( Xi − θ )

• This is NOT a statistic.

• Reason: It depends on the unknown parameter θ.

11 Sample Space and Properties

Sample Space (X ): All possible values X can take.

• Example (Germination): X = {(0, . . . , 0), . . . , (1, . . . , 1)}.

• X has 2n components.

Properties of Statistic T (X): 1. T (X) is random. 2. T (X) has a probability distribution.

Example:

T (X) = 0 with prob. P( T (X) = 0) = P( X1 = 0, X2 = 0, . . . , Xn = 0)

8
S TATISTICAL I NFERENCE L ECTURE N OTES

12 Sampling Distribution

The distribution of T (X) is called the "Sampling Distribution".

Example 1: Bernoulli Model
iid
Xi ∼ Ber(θ ), P( Xi = 1) = θ, P ( Xi = 0 ) = 1 − θ

Consider the histograms of T (X) (Sample Proportion) for different θ:

θ is small (Left skew) θ is large (Right skew)

T (X) T (X)

12.1 Uniform Distribution Example (n = 2)

iid
Let Xi ∼ Uniform(0, θ ) for i = 1, 2. We visualize the sample space as a square of side θ:

x2
(θ, θ )
θ
x2 > x1
x2
=
x1

x1 > x2
x1
θ

Define Statistic T (X) based on order:

(
1 if X1 ≥ X2 (Blue Region)
T (X) =
2 if X2 > X1 (Green Region)

Definition 12.1: Estimator

A statistic which is used to estimate g(θ ) is called an estimator of g(θ ).

9
S TATISTICAL I NFERENCE L ECTURE N OTES

13 Important Results (MGF Technique)

iid
Example: Oscillation Time Xi ∼ N ( T0 , σ2 ).

σ2
 
T (X) = X̄n ∼ N T0 ,
n

Result: Linear Combination of Normals

iid
If Xi ∼ N (µi , σi2 ) for i = 1, . . . , n, then the linear combination T (X) = a1 X1 + · · · +
an Xn is distributed as: !
n n
T (X) ∼ N ∑ ai µi , ∑ a2i σi2
i =1 i =1

Proof (MGF Technique): The Moment Generating Function (MGF) of X is

2 t2 /2
MX (t) = E[etX ] = eµt+σ .

Let T = ∑in=1 ai Xi . h i
MT (t) = E[etT ] = E et ∑ ai Xi

Product Law of Expectation

Since Xi are independent:

h i
E ∏e tai Xi
= ∏ E[etai Xi ]

M T ( t ) = M X1 ( a 1 t ) · M X2 ( a 2 t ) . . . M X n ( a n t )
2 2 2 /2 2 2 2 /2
= e a1 µ1 t+a1 σ1 t . . . e an µn t+an σn t
t2 2 2
= et(∑ ai µi )+ 2 (∑ ai σi )

By recognizing the form of the Normal MGF, this implies:

 
T ∼ N ∑ ai µi , ∑ a2i σi2

14 Order Statistics

More Summation Results:

iid
• Xi ∼ Bin(mi , θ ) =⇒ ∑ Xi ∼ Bin(∑ mi , θ )

10
S TATISTICAL I NFERENCE L ECTURE N OTES

ind
• Xi ∼ Poisson(λi ) =⇒ ∑ Xi ∼ Poisson(∑ λi )
ind
• Xi ∼ Gamma(αi , β) =⇒ ∑ Xi ∼ Gamma(∑ αi , β)

iid
Order Statistics Definitions: Let X1 , . . . , Xn ∼ G (θ ).

• T (X) = X(n) = max{ X1 , . . . , Xn } (Highest order statistic).

• S(X) = X(1) = min{ X1 , . . . , Xn } (Lowest order statistic).

Derivation of CDF of T (X) (The Max):

GT ( t ) = P( T (X) ≤ t) = P(max{ X1 . . . Xn } ≤ t)
= P( X1 ≤ t, X2 ≤ t, . . . , Xn ≤ t)
= P( X1 ≤ t) · P( X2 ≤ t) . . . P( Xn ≤ t) (by independence)
= [ FX (t)]n

iid t
Example: Uniform Max Let Xi ∼ U (0, θ ). The CDF is FX (t) = θ for 0 ≤ t ≤ θ.
 n
t
GT ( t ) =
θ

PDF of T:
d tn ntn−1
 
d
g T ( t ) = GT ( t ) = = , 0≤t≤θ
dt dt θ n θn

15 Supplementary Notes: Populations and Modeling

Definition: Population
A Population is a collection of measures. It is non-random and represents the com-
plete set of data we are interested in.

The population is mathematically represented by the Cumulative Distribution Function

(CDF), denoted as F ( x ).

• F (−∞) = 0

• F (+∞) = 1

• F is a non-decreasing function.

11
S TATISTICAL I NFERENCE L ECTURE N OTES

1 F(x)

Non-decreasing
0.5

x
−1 1 2 3 4 5

Example: Let the population P follow a Gamma distribution: P = Gamma(2, 5). The
relative frequency of P in the interval (4, 4.5) is given by the integral of the PDF:
Z 4.5
constant × x2−1 e−5x dx
4

16 Sampling and Identical Distributions (Recap)

Since the entire population is rarely known, we rely on samples.

Concept: The Sample

A Sample consists of representative candidates of P to be investigated. Before observ-
ing the data point Xi , its value is unknown.

While we don’t know the exact value, we calculate the probability of the value falling in a
range using the CDF:
P( Xi ∈ [ a, b]) = F (b) − F ( a)

The I.I.D. Assumption

We typically assume the sample X1 , . . . , Xn are:

1. Identically Distributed: They follow the same distribution (e.g., X ∼ N (0, 1)).

2. Independent: One observation does not influence another.

17 Advanced Statistical Modeling

Often, the population function F is unknown. We use domain knowledge to guess the
shape. This "art of deriving a shape" is called Modeling.

12
S TATISTICAL I NFERENCE L ECTURE N OTES

Model: Mixture Model Example: Income

Consider "Income per hour". There is a probability p of zero income (unemployment)
and a probability (1 − p) of positive income following a Gamma distribution.

X ∼ p · δ(·) + (1 − p) f (· | Gamma)

This implies a "spike" at 0 and a continuous tail for x > 0.

A Parametric Model assumes a form f ( x; θ ) with a finite number of unknown constants

(parameters), such as α, β, p.

18 Summary: Parametric Inference

Definition: Inference
The task of parametric inference is to estimate the parameters (or functions of param-
eters) from the sample X1 , . . . , Xn .

Statistics and Sampling Distributions

A function of the sample X1 , . . . , Xn is called a Statistic (e.g., the sample mean θ̂).

θ̂n : Sample Space → R

Crucial Note: Since the samples are random, a statistic is also a random variable. The
distribution of this statistic is called the Sampling Distribution.

19 Finding Sampling Distributions

The distribution of a statistic Tn (X) is called its sampling distribution. Since the distribu-
tion of the data X depends on parameters θ, the sampling distribution of Tn also typically
depends on θ.

13
S TATISTICAL I NFERENCE L ECTURE N OTES

Example: Visualizing Parameter Dependence

Consider X ∼ N (µ, σ2 ). The shape is determined by θ = (µ, σ2 ).

0.4 µ = 0, σ2 = 1
µ = 2, σ2 = 2.2
f (x)
0.2

0
−4 −2 0 2 4 6
x

Just as the PDF changes shape, the distribution of any statistic derived from X will
change.

We generally use three main techniques to find sampling distributions:

1. CDF Technique (Distribution Function Method)

2. MGF Technique (Moment Generating Function)

3. Transformation Technique (Jacobian Method)

19.1 Technique I: The CDF Technique

This is a fundamental method involving first finding the Cumulative Distribution Function
(CDF) and then differentiating it to find the PDF.
Steps:

1. Define the statistic T = g( X ).

2. Express the CDF: FT (t) = P( T ≤ t) = P( g( X ) ≤ t).

3. Invert the inequality to express the probability in terms of X.

4. Integrate f X ( x ) over the region to find FT (t).

5. Differentiate: f T (t) = FT′ (t).

14
S TATISTICAL I NFERENCE L ECTURE N OTES

Example: Ex 3: Cauchy to Beta Transformation

Let X ∼ Cauchy(0, 1), i.e., f X ( x ) = 1

π (1+ x 2 )
, x ∈ R.
1
Consider the transformation T = of T.
1+ X 2
. Find the PDF
2
1. Support: Since x ≥ 0, we have 0 < T ≤ 1.
2. CDF Derivation: For 0 < t < 1:
   
1 2 1
FT (t) = P ≤ t = P 1+X ≥
1 + X2 t
 
1
= P X2 ≥ − 1
t
 
2 1
= 1−P X < −1
t

By symmetry of the Cauchy distribution around 0:

r r !
1 1
FT (t) = 1 − P − −1 < X < −1
t t
Z √1/t−1
1
= 1−2 dx
0 π (1 + x 2 )
2 h i√1/t−1
−1
= 1− tan ( x )
π 0
r !
2 1−t
= 1− tan−1
π t

3. PDF Derivation: Differentiating FT (t) w.r.t t:

d 1
f T (t) = FT (t) = p , 0<t<1
dt π t (1 − t )
 
Observation: This is the PDF of the Beta 12 , 12 distribution (Arcsine distribution).

Resulting Distribution: Beta(0.5, 0.5)

3
f T (t)

2
1
0
0 0.2 0.4 0.6 0.8 1
t

15
S TATISTICAL I NFERENCE L ECTURE N OTES

Result: Example 4: Probability Integral Transform (PIT)

Let X be a continuous random variable with CDF FX ( x ).

Define T = FX ( X ). Then:
T ∼ Uniform(0, 1)
Proof:
P( T ≤ t) = P( FX ( X ) ≤ t) = P( X ≤ FX−1 (t)) = FX ( FX−1 (t)) = t
Since the CDF is t, the variable is Uniform(0,1).

Example: Ex 5: Minimum of Exponentials

iid
Let Xi ∼ Exp(λ) for i = 1, . . . , n. ( f ( x ) = λe−λx ).
Let Tn = min{ X1 , . . . , Xn }.
Using the CDF technique, it is often easier to work with the Survival Function
P( Tn > t):

P( Tn > t) = P(min( Xi ) > t)

= P ( X1 > t ∩ X2 > t · · · ∩ X n > t )
n
= ∏ P( Xi > t) (by independence)
i =1
n  n
= ∏ e−λt = e−λt
i =1
−nλt
=e

The CDF is FTn (t) = 1 − e−(nλ)t .

Differentiating gives the PDF: f Tn (t) = nλe−(nλ)t .
Conclusion: The minimum of n i.i.d. exponentials is Exp(nλ).

! Important: Next Steps

The CDF technique is powerful, but for sums of random variables (like convolution),
other methods are often faster. We will next explore:

• MGF Technique: Useful for sums of independent variables (T = ∑ Xi ).

• Transformation Technique: Using the Jacobian Matrix for one-to-one map-

pings.

19.2 Technique II: The Transformation (Jacobian) Technique

While the CDF technique is universal, it can be cumbersome. For continuous random
variables with smooth, one-to-one mappings, we use the Transformation Technique (also
known as the Change of Variable method).

16
S TATISTICAL I NFERENCE L ECTURE N OTES

Definition 19.1: The Jacobian Formula

Let X be a continuous random variable with PDF f X ( x ).
Let Y = h( X ), where h is a one-to-one (monotonic) and differentiable function.
The PDF of Y is given by:
dx
f Y (y) = f X ( x ) ·
dy
where x = h−1 (y) is expressed in terms of y, and J = dx
dy is the Jacobian of the trans-
formation.

Mapping Probability Mass: f X ( x )dx = f Y (y)dy

y

y = h( x )
dy

x
dx

17
S TATISTICAL I NFERENCE L ECTURE N OTES

Example: Ex 6: Sine Transformation

Let X ∼ Uniform(−π/2, π/2). The PDF is f X ( x ) = π1 .

Consider the transformation Y = sin( X ).
1. Support: Since x ∈ [− π2 , π2 ], we have y ∈ [−1, 1].
2. Jacobian:
y = sin x =⇒ x = sin−1 y
dx d 1
= (sin−1 y) = p
dy dy 1 − y2
3. Apply Formula:

dx 1 1
f Y (y) = f X ( x ) · = ·p , −1 < y < 1
dy π 1 − y2

Note: This is the Arcsine Distribution.

4. Further Transformation (Connection to Beta): Let Z = 1+2 Y . This maps [−1, 1] to
[0, 1]. Using the linear transformation rule (z = 1+2 y =⇒ dy
dz = 2):

f Z (z) = f Y (2z − 1) · |2|

1
= p ·2
π 1 − (2z − 1)2
2 2
= p = √
π 1 − (4z2 − 4z + 1) π 4z − 4z2
2 1
= p = p
2π z(1 − z) π z (1 − z )

Conclusion: Z ∼ Beta(1/2, 1/2). This confirms the result we saw earlier with the
Cauchy distribution!

20 Expectation of Functions

Often we do not need the full distribution of Y = h( X ), but only its expected value E[Y ].

! Important: Law of the Unconscious Statistician (LOTUS)

There are two ways to calculate E[h( X )]:
Method 1 (Via derived distribution): Find f Y (y) first, then calculate:
Z ∞
E [Y ] = y · f Y (y) dy
−∞

Method 2 (Directly via X):

Z ∞
E[h( X )] = h( x ) · f X ( x ) dx
−∞

18
S TATISTICAL I NFERENCE L ECTURE N OTES

Example: Ex 7: Expectation of sin( X )

Let X ∼ U (−π/2, π/2). Find E[sin X ].

Using Method 2 (LOTUS):

1
Z π/2
E[sin X ] = sin( x ) · dx
−π/2 π

Since sin( x ) is an odd function and the domain is symmetric around 0, the integral
is exactly 0.

21 Introduction to Point Estimation

We often have multiple statistics (estimators) that could serve as a guess for a parameter θ.

• Example: To estimate center θ, we could use Sample Mean X̄ or Sample Median X̃.

How do we choose? We need a metric for "accuracy".

Definition 21.1: Mean Squared Error (MSE)

The Mean Squared Error of an estimator Tn for a parameter θ is defined as:

MSE( Tn ) = E[( Tn − θ )2 ]

It measures the average squared distance between the estimator and the truth.
True θ
Density

T1
T2

Value

22 Important Sampling Distributions

When sampling from Normal populations, three derived distributions appear repeatedly.
These form the backbone of hypothesis testing and interval estimation.

19
S TATISTICAL I NFERENCE L ECTURE N OTES

22.1 The Chi-Square Distribution (χ2 )

Definition 22.1: Definition

iid
Let Z1 , Z2 , . . . , Zn ∼ N (0, 1). The sum of their squares is distributed as Chi-Square
with n degrees of freedom:
n
Y= ∑ Zi2 ∼ χ2(n)
i =1

Properties:
 
n 1
• Relation to Gamma: χ2(n) ≡ Gamma 2, 2 .

• PDF: f ( x ) = 1
2n/2 Γ(n/2)
x n/2−1 e− x/2 , x > 0.

• Moments:
E [Y ] = n and Var(Y ) = 2n

22.2 The F-Distribution

Used primarily for comparing variances (ANOVA, F-tests).

Definition 22.2: Definition

Let U ∼ χ2(n) and V ∼ χ2(m) be independent. The random variable

U/n
F= ∼ Fn,m
V/m
follows an F-distribution with degrees of freedom (n, m).

1
Property: If Z ∼ Fn,m , then Z ∼ Fm,n .

22.3 The Student’s t-Distribution

Used for inference on the mean when variance is unknown.

Definition 22.3: Definition

Let Z ∼ N (0, 1) and U ∼ χ2(n) be independent.

Z
T= √ ∼ t(n)
U/n

Key Relationships:

20
S TATISTICAL I NFERENCE L ECTURE N OTES

• The PDF is symmetric around 0 (E[ T ] = 0 for n > 1).

• T 2 ∼ F1,n .

• As n → ∞, t(n) → N (0, 1).

0.4 N (0, 1)
t (2) (Fatter Tails)
Density

0.2

0
−4 −3 −2 −1 0 1 2 3 4
x

23 Case Study: Estimating Variance

iid
Let X1 , . . . , Xn ∼ N (µ, σ2 ). We wish to estimate the population variance g(θ ) = σ2 .

23.1 Two Competing Estimators

1. The MLE (Biased):
1 n
n i∑
Sn2 = ( Xi − X̄n )2
=1

2. Sample Variance (Unbiased):

n
1
Sn∗2 = ∑ ( X − X̄n )2
n − 1 i =1 i

Result: Sampling Distribution of Sum of Squares

∑( Xi − X̄n )2
∼ χ2(n−1)
σ2
Let Y ∼ χ2(n−1) . Then ∑( Xi − X̄n )2 = σ2 Y.

23.2 Expectation and Bias

Using E[Y ] = n − 1:

21
S TATISTICAL I NFERENCE L ECTURE N OTES

h i
σ2 Y σ2 n −1 2
• E[Sn2 ] = E n = n E [Y ] = n σ .
h i
σ2 Y σ2
• E[Sn∗2 ] = E n −1 = n − 1 E [Y ] = σ2 .

Conclusion: Sn2 is biased (underestimates variability), while Sn∗2 is unbiased.

24 The Bias-Variance Decomposition

Which estimator is "better"? We use the Mean Squared Error (MSE).

Definition 24.1: MSE Decomposition

MSE(θ̂ ) = E[(θ̂ − θ )2 ] = Var(θ̂ ) + [Bias(θ̂ )]2

24.1 Comparing MSEs

1. MSE of Unbiased Estimator (Sn∗2 ): Since bias is 0, MSE is just variance. Using Var(Y ) =
2( n − 1):
 2 
∗2 σ Y σ4 σ4 2σ4
MSE(Sn ) = Var = Var ( Y ) = · 2 ( n − 1 ) =
n−1 ( n − 1)2 ( n − 1)2 n−1

n −1 2 2
2. MSE of Biased Estimator (Sn2 ): Here, Bias = E[Sn2 ] − σ2 = n σ − σ2 = − σn .

MSE(Sn2 ) = Var(Sn2 ) + (Bias)2

 2 
σ Y σ4
= Var + 2
n n
σ 4 σ4
= 2 (2(n − 1)) + 2
n n
σ 4 2n − 1 4
= 2 (2n − 2 + 1) = σ
n n2

The Verdict
2n−1 2
We compare n2
vs n −1 .

2n − 1 2
2
< ⇐⇒ (2n − 1)(n − 1) < 2n2
n n−1

2n2 − 3n + 1 < 2n2 ⇐⇒ 1 < 3n

This is true for all n ≥ 1.

22
S TATISTICAL I NFERENCE L ECTURE N OTES

Conclusion: The Biased Estimator (Sn2 ) has a smaller MSE than the Unbiased Estimator.
In the context of MSE, minimizing variance is often more valuable than eliminating bias
perfectly.

Class of All Estimators

Sn2 (Lower MSE)

Unbiased Estimators
Sn∗2

The best estimator often lies outside the unbiased class.

25 Evaluating Estimators: The Quest for the Best

25.1 Estimators vs. Estimates

When dealing with parameters θ, we often wish to estimate a specific function of the pa-
rameter, denoted as g(θ ).

• Estimator (Tn ): A function of the random sample T ( X1 , X2 , . . . , Xn ) defined before the

experiment. It is a random variable and has a sampling distribution that depends on
θ.
• Estimate (tn ): A specific numerical value obtained by applying the estimator to a spe-
cific realization of the data T ( x1 , x2 , . . . , xn ).

True g(θ )
PDF of Tn

Distribution of Tn

Tn

25.2 Error and Mean Squared Error (MSE)

iid
Let X1 , . . . , Xn ∼ Bernoulli(θ ). We use Tn = ∑in=1 Xi ∼ Binomial(n, θ ).
We define the Error as Ψ T (θ ) = Tn − θ, and the Squared Error as Ψ2T (θ ) = ( Tn − θ )2 . The
distributions can be tabulated as follows:

23
S TATISTICAL I NFERENCE L ECTURE N OTES

Value of Tn 0 1 ... n

Error ( Tn − θ ) −θ 1−θ ... n−θ

Squared Error ( Tn − θ )2 θ2 (1 − θ )2 ... ( n − θ )2

Probability P( Tn = t) (1 − θ ) n nθ (1 − θ )n−1 ... θn

Definition 25.1: Mean Squared Error

The expectation of the squared error is called the Mean Squared Error (MSE) of the
estimator Tn in estimating g(θ ):

MSETn ( g(θ )) = E[( Tn − g(θ ))2 ]

The Selection Problem: An estimator with a smaller MSE is preferred. The one with the
smallest MSE in a given class is the "best" estimator. However, the class of all estimators
is infinitely large, making it impossible to find a global minimum for all θ. Therefore, we
restrict our focus to a smaller, sensible sub-class: Unbiased Estimators.

26 Unbiased Estimators and UMVUE

Class of All Estimators of g(θ )

Class of
Unbiased
Estimators

Definition 26.1: Unbiased Estimator

An estimator Tn = T ( X1 , . . . , Xn ) is called unbiased for g(θ ) if its expected value
matches the parameter for all possible values of θ:

Eθ [ Tn ] = g(θ ) for all θ

24
S TATISTICAL I NFERENCE L ECTURE N OTES

Example: Checking Unbiasedness

Let X1 , . . . , Xn ∼ N (θ, 1). Consider two estimators for θ:

1. T1n = X̄n = 1
n ∑in=1 Xi .

1 1
E[ T1n ] =
n ∑ E[ Xi ] = (nθ ) = θ
n
=⇒ T1n is unbiased.
2. T2n = 2
n ∑in=2 ( Xi − Xi−1 ).

2 n 2 n
n i∑ ∑ (θ − θ ) = 0
E[ T2n ] = ( E [ X i ] − E [ X i −1 ]) =
=2
n i =2

=⇒ T2n is NOT unbiased (it only equals θ if θ = 0, not for all θ).

26.1 The UMVUE

Recall the MSE decomposition: MSETn ( g(θ )) = Varθ ( Tn ) + [BiasTn (θ )]2 .

If Tn is unbiased, Bias = 0, so MSETn (g(`)) = Var` (Tn ). The best unbiased estimator is
simply the one with the minimum variance.

Definition 26.2: UMVUE

An estimator Tn is called the Uniformly Minimum Variance Unbiased Estimator
(UMVUE) of g(θ ) if:

1. Tn is unbiased for g(θ ).

2. For any other unbiased estimator Sn of g(θ ), we have:

Varθ ( Tn ) ≤ Varθ (Sn ) for all θ

25
S TATISTICAL I NFERENCE L ECTURE N OTES

27 Comparing Unbiased Estimators

Example: Variance Showdown

Let X1 , . . . , Xn ∼ Bernoulli(θ ), where sample size n = 2m is even.
We have two unbiased estimators for θ:

1. T1n = X̄n = 1
n ∑in=1 Xi (Uses all data)
X2 + X4 +···+ X2m
2. T2n = m (Uses only even-indexed data)

Both have Bias = 0. Let’s compare their variances (noting Xi are independent, so
covariance terms are 0):

∑ Xi nθ (1 − θ ) `(1 − `)
 
1
Varθ ( T1n ) = Var = 2 ∑ Var( Xi ) = 2
=
n n n n

∑even Xi mθ (1 − θ ) θ (1 − θ )
 
1
Varθ ( T2n ) = Var = 2 ∑ Var( Xi ) = 2
=
m m even m m
1
Since n = 2m, m = n2 , so:
2`(1 − `)
Varθ ( T2n ) =
n
Conclusion: Var( T1n ) < Var( T2n ). Using all the data (T1n ) provides a lower vari-
ance, making it a better estimator than T2n .

Definition 27.1: Relative Efficiency

Let T1n and T2n be two unbiased estimators of g(θ ). The Relative Efficiency of T2n
compared to T1n is:
Varθ ( T2n )
RE( T1n , T2n ) =
Varθ ( T1n )
• If RE > 1, T1n is the better estimator.

• If RE < 1, T2n is the better estimator.

28 Finding the UMVUE via Sufficiency

Comparing two estimators is straightforward, but proving one is better than all possible
unbiased estimators requires advanced theory.

26
S TATISTICAL I NFERENCE L ECTURE N OTES

Result: The Sufficiency Approach (Lehmann-Scheffé)

To obtain the UMVUE, we focus exclusively on Complete Sufficient Statistics (CSS).

Suppose Tn∗ is a Complete Sufficient Statistic for θ. If we can find a function h( Tn∗ )
such that:
Eθ [h( Tn∗ )] = g(θ )
Then h( Tn∗ ) must be the unique UMVUE for g(θ ).

iid
Example: Let Xi ∼ Bernoulli(θ ). We know that the sum Y = ∑in=1 Xi is a Complete Suffi-
cient Statistic (CSS) for θ.
What is the UMVUE for θ? We know E[Y ] = nθ. If we define h(Y ) = Yn = X̄n , then
E[h(Y )] = θ.
Because X̄n is an unbiased function of the CSS, X̄n is the UMVUE for θ.

29 Finding the UMVUE: Advanced Theory

Previously, we found that the UMVUE is the unbiased estimator with the smallest possible
variance. However, checking the variance against an infinite number of other estimators is
impractical. We need a constructive method to find it.

29.1 Sufficiency and Completeness

Definition 29.1: Sufficient Statistic

A statistic T ∗ (X) is sufficient for θ if the conditional distribution of the sample X given
T ∗ is free of θ. Intuitively, T ∗ captures all the information in the sample regarding the
parameter.

Definition 29.2: Complete Statistic

A statistic T ∗ (X) is complete for a family of distributions if, for any measurable func-
tion h:

Eθ [h( T ∗ )] = 0 for all θ =⇒ Pθ (h( T ∗ ) = 0) = 1 (almost everywhere)

Intuition: Completeness ensures there are no "redundant" unbiased estimators. If T ∗

is complete, there is exactly one unique unbiased function that can be formed from it.

If T ∗ satisfies both properties, it is called a Complete Sufficient Statistic (CSS). Important

property: Any one-to-one function of a CSS is also a CSS.

27
S TATISTICAL I NFERENCE L ECTURE N OTES

Result: The Lehmann-Scheffé Theorem

Let T ∗ be a Complete Sufficient Statistic (CSS) for θ. If there exists a function h( T ∗ )
such that it is unbiased for g(θ ), i.e.,

Eθ [h( T ∗ )] = g(θ )

Then h( T ∗ ) is the unique UMVUE of g(θ ).

Example: Example: Uniform Distribution (Irregular Family)

iid
Let X1 , . . . , Xn ∼ U (0, θ ). We are given that the maximum order statistic T ∗ = X(n)
is a CSS for θ.
The PDF of T ∗ is f T ∗ (t) = θnn tn−1 for 0 < t < θ.
Let’s find its expectation to check for bias:
θ
n t n +1

n n −1 n
Z θ
∗
Eθ [ T ] = t · nt dt = n = θ
0 θ θ n+1 0 n+1

Since E[ X(n) ] ̸= θ, it is biased. We correct the bias by multiplying by the reciprocal:

 
n+1
Eθ X( n ) = θ
n
n +1
Because n X( n ) is an unbiased function of the CSS, it is the UMVUE for θ.

30 The Exponential Family

Proving Completeness and Sufficiency from scratch is mathematically rigorous. The Expo-
nential Family provides a structural template. If a distribution fits this template, we can
identify the CSS instantly.

Definition 30.1: The Exponential Family Form

A distribution f θ ( x ) belongs to the k-parameter Exponential Family if the joint
PDF/PMF of a sample X1 , . . . , Xn can be expressed as:
( )
k
f X (x; θ) = exp A(θ) + H (x) + ∑ Cj (θ) Tj (x)
j =1

where:

• A(θ) and Cj (θ) are functions of the parameters only.

• H (x) and Tj (x) are functions of the data only.

28
S TATISTICAL I NFERENCE L ECTURE N OTES

! Important: Crucial Regularity Conditions

1. Support Restriction: The support (range of valid X values) must not depend
on the parameter θ. Distributions like Uniform or Pareto fail this condition.

2. Open Set Condition: The parameter space Θ ⊆ Rk must contain an open set
(e.g., an open 2D region, not just a line). This ensures the analytical properties
needed for completeness.

Open Set Condition in R2

β
Contains Open Set
α=β

Result: CSS from the Exponential Family

If a distribution meets the regularity conditions above, then the statistics:

{ T1 (X), T2 (X), . . . , Tk (X)}

are Jointly Complete Sufficient Statistics (CSS).

30.1 Examples and Non-Examples

1. Distributions NOT in the Exponential Family:

• Uniform(0, θ ): Support depends on θ.

• Cauchy(θ, 1): f ( x ) ∝ 1+(x1−θ )2 . Cannot be factored into the required exponential form.
(Note: No CSS exists other than the full order statistics).

• Student’s t-distribution: Cannot be factored.

2. Binomial Distribution (Xi ∼ Bin(m, θ )):

 

 

     
 m θ 
f X (x) = exp ∑ log
1−θ ∑
+ mn log(1 − θ ) + log x i

 xi | {z } | {z } 
A(θ )

T (x) 
 | {z } | {z } 
H (x) C (θ )

CSS: ∑in=1 Xi .

29
S TATISTICAL I NFERENCE L ECTURE N OTES

3. Poisson Distribution (Xi ∼ Poisson(λ)):

n
e − λ λ xi
∏ xi ! = exp −nλ − ∑ log(xi !) + log(λ) ∑ xi

f X (x) =
i =1

CSS: ∑ Xi .
UMVUE for λ: Since E[ X̄n ] = λ, X̄n is the UMVUE.
4. Gamma Distribution (Xi ∼ Gamma(α, β)): Support x > 0 is independent of θ = (α, β).

f X (x) = exp nα log β − n log Γ(α) − ∑ log xi + α ∑ log xi − β ∑ xi



Joint CSS: {∑in=1 log Xi , ∑in=1 Xi }.

5. Beta Distribution (Xi ∼ Beta(α, β)):

f X (x) = exp −n log B(α, β) + (α − 1) ∑ log xi + ( β − 1) ∑ log(1 − xi )



Joint CSS: {∑ log Xi , ∑ log(1 − Xi )}.

6. Inverse Gaussian (Xi ∼ IG (µ, λ)):
 
n nλ 1 λ λ 1
f X (x) = exp log λ − − ∑ log(2πxi3 ) − 2 ∑ xi − 2 ∑ xi
2 µ 2 2µ
n o
Joint CSS: ∑ Xi , ∑ X1 .
i

30.2 Finding UMVUEs for the Normal Distribution

iid
Let X1 , . . . , Xn ∼ N (µ, σ2 ), with parameters µ ∈ R, σ2 > 0. Taking the logarithm of the joint
PDF and moving it to the exponent:
( )
n
n 1
f X (x) = exp − log(2πσ2 ) − 2 ∑ ( xi2 − 2µxi + µ2 )
2 2σ i=1
 

 

2

 n   
nµ  µ  1 
= exp − log(2πσ ) − 2 + 2 ∑ xi + − 2 ∑ xi
2 2
| 2
 {z 2σ } | σ 2σ | {z }
{z } | {z } | {z 
T } T 


A(θ) C 1 2 
1 C2

From this, the statistics T1 = ∑ Xi and T2 = ∑ Xi2 are Jointly CSS.

Because any 1-to-1 transformation of a CSS is also a CSS, we can apply algebraic manipula-
tions to find more convenient sufficient statistics:
( ) ( )
n n n
1
∑ Xi , ∑ Xi2 ≡ X̄n , Sn2 = n − 1 ∑ (Xi − X̄n )2
i =1 i =1 i =1

The Sample Mean and Sample Variance are jointly CSS!

30
S TATISTICAL I NFERENCE L ECTURE N OTES

Conclusion: UMVUEs for Normal

By the Lehmann-Scheffé Theorem:

1. We want to estimate g1 (θ ) = µ. We know E[ X̄n ] = µ. Since X̄n is an unbiased

function of the CSS, X̄n is the UMVUE for µ.

2. We want to estimate g2 (θ ) = σ2 . We proved earlier that E[Sn2 ] = σ2 . Since Sn2 is

an unbiased function of the CSS, Sn2 is the UMVUE for σ2 .

31
S TATISTICAL I NFERENCE L ECTURE N OTES

31 Methods of Point Estimation

31.1 Recap: The Estimation Problem

Recall our fundamental setup: We have a random sample X1 , X2 , . . . , Xn ∼ Fθ, where θ is

the unknown population parameter. Our job is to estimate θ (or a function of it, g(θ)) to
understand the population.
We define an estimator Tn = T ( X1 , X2 , . . . , Xn ).

• Unbiasedness: If E[ Tn ] = g(θ), the estimator is unbiased.

• Mean Squared Error (MSE): E[( Tn − g(θ))2 ] = Bias2 + Var( Tn ). For unbiased estima-
tors, MSE = Var( Tn ).

Example: Recall: Exponential Family & UMVUE

If X1 , . . . , Xn ∼ Gamma(α, β), let θ = ( αβ). We want to estimate g(θ) = α. The PDF
is:
βα α−1 − βx
f (x) = x e
Γ(α)
From the Exponential Family form, the Joint Complete Sufficient Statistics (CSS) are:
( )
n n
∑ log Xi , ∑ Xi
i =1 i =1

Concept: The Next Step

What if Fθ does not belong to the exponential family (like the Uniform distribution
U (0, θ ))? Or what if finding a function of the CSS is algebraically too difficult? We
need general, systematic methods to obtain estimators for g(θ).
The two primary methods are:

1. Method of Moments (MoM)

2. Maximum Likelihood Estimation (MLE)

32 The Method of Moments (MoM)

The Method of Moments is an intuitive approach: we estimate population parameters by

equating theoretical population moments to the observed sample moments.

32
S TATISTICAL I NFERENCE L ECTURE N OTES

32.1 Defining Moments

Population Moments (X ∼ Fθ) Sample Moments (X1 , . . . , Xn )

• rth order Raw Moment: • rth order Sample Raw Moment:

µr′ (θ) = E[ X r ] for r = 0, 1, 2, . . . mr′ = n1 ∑in=1 Xir for r = 0, 1, 2, . . .

• rth order Central Moment: • rth order Sample Central Moment:

µr (θ) = E[( X − µ1′ )r ] mr = n1 ∑in=1 ( Xi − m1′ )r
Note: µ1′ = µ (Mean), µ2 = σ2 (Vari-
ance).

Definition 32.1: The MoM Principle

To obtain the Method of Moments estimator for a parameter vector θ ∈ Rk , we solve
the system of k equations created by setting the sample raw moments equal to the
population raw moments:

mr′ = µr′ (θ) for r = 1, 2, . . . , k

! Important: The Plug-in Property

If we want to estimate a function of the parameters Ψ(θ) = Ψ(α, β), the MoM estima-
tor is simply Ψ(θ̂MoM ).
Example: If we want to estimate (α + β), the MoM estimator is α̂MoM + β̂ MoM .

33
S TATISTICAL I NFERENCE L ECTURE N OTES

33 Examples of MoM Estimators

Example: Example 1: Gamma Distribution

iid
Let Xi ∼ Gamma(α, β). We have two parameters θ = ( αβ). We need two equations
(r = 1, 2).
Equation 1 (First Moment):
α
m1′ = µ1′ (θ) =⇒ X̄n =
β

Equation 2 (Second Moment): Using the variance identity E[ X 2 ] = Var( X ) +

( E[ X ])2 :
 2
1 n 2 α α
′ ′
m2 = µ2 (θ) =⇒ ∑
n i =1
Xi = 2 +
β β
 2
Solving the System: Notice that αβ = (m1′ )2 . Substituting this into Equation 2:
 
α 1 α
m2′ = 2 + (m1′ )2 =⇒ m2′ − (m1′ )2 =
β β β

Substitute α
β = m1′ :

m1′ m1′
m2′ − (m1′ )2 = =⇒ β̂ MoM = ′
β m2 − (m1′ )2

Now find α̂MoM using Equation 1 (α = βm1′ ):

(m1′ )2
α̂MoM = m1′ β̂ MoM =
m2′ − (m1′ )2

Example: Example 2: Normal Distribution

iid
Let Xi ∼ N (µ, σ2 ). Find MoM estimators for µ and σ2 .
1. Estimate for µ:

m1′ = µ1′ =⇒ X̄n = µ =⇒ µ̂MoM = X̄n

2. Estimate for σ2 :
1
n∑ i
m2′ = σ2 + µ2 =⇒ X 2 = σ2 + ( X̄n )2

1 1 n
n∑ i n i∑
2 2 2
σ̂MoM = X − X̄ n = ( Xi − X̄n )2 = s2n
=1
Conclusion: The MoM estimator yields the sample mean and the biased sample vari-
ance. (Valid/Logical Estimator)

34
S TATISTICAL I NFERENCE L ECTURE N OTES

34 The Flaw of MoM: The Uniform Case

Example: Example 3: Uniform Distribution U (0, θ )

iid
Let Xi ∼ U (0, θ ) for i = 1, 2, . . . , n.
The MoM Equation:
θ
m1′ = µ1′ =⇒ X̄n =
2
θ̂MoM = 2X̄n
Comparing to UMVUE: Recall from earlier  that  the UMVUE for θ is derived from
n +1
the maximum order statistic: θ̂UMVUE = n X(n) . The MSE of 2X̄n is significantly
 
higher than the MSE of n+ n
1
X( n ) .
The Logical Contradiction: Because θ is the absolute maximum possible value of
the distribution, it must be true that θ ≥ X(n) (the true parameter cannot be smaller
than the largest observed data point). However, the MoM estimator 2X̄n can some-
times yield a value strictly less than X(n) .

X̄n X(n) (Max Observation)

Value

θ̂× Logical
MoM = 2X̄Impossibility!
n
MoM Estimate Domain 2X̄n < X(n) is possible.

! Important: Why MoM is imperfect

The Method of Moments does not utilize the joint distribution or the sufficient statis-
tics (like X(n) ). It merely forces summary averages to match. Because of this, it can
produce parameter estimates that are logically impossible given the observed sample
space. This leads us to a more robust methodology: Maximum Likelihood Estimation.

35 Introduction to Maximum Likelihood Estimation (MLE)

If MoM is prone to logical errors, how do we find a better estimate? We ask a different
question: Which parameter value makes our observed data the most mathematically probable?

35
S TATISTICAL I NFERENCE L ECTURE N OTES

Example: Intuitive Example: Exponential Likelihood

Let Xi ∼ Exp(θ ), where E[ Xi ] = θ. Suppose we know that the true parameter θ must
be one of three discrete values: θ ∈ {1, 3, 6}.
We take a sample of n = 4 observations:

x1 = 4.5, x2 = 2.5, x3 = 3.2, x4 = 4.0

The sum of our observations is ∑ xi = 14.2.

The joint PDF (which we call the Likelihood Function when evaluating it for vary-
ing θ) is:
4
1 1 1
f θ ( x1 , x2 , x3 , x4 ) = ∏ e− xi /θ = 4 e− θ ∑ xi
i =1
θ θ
Let’s plug in our data (∑ xi = 14.2) and evaluate the likelihood of each possible
parameter:

1. If θ = 1:
1 −14.2/1
f 1 (x) = 4
e ≈ 6.8 × 10−7 (Rank 3: Least Likely)
1

2. If θ = 3:
1 −14.2/3 1
f 3 (x) = 4
e = e−4.733 ≈ 0.000108 (Rank 1: Most Likely)
3 81

3. If θ = 6:
1 −14.2/6 1 −2.366
f 6 (x) = 4
e = e ≈ 0.000072 (Rank 2)
6 1296

Conclusion: The parameter θ = 3 yields the highest probability density for the
specific sample we observed. Therefore, the Maximum Likelihood Estimate (MLE)
among the choices provided is 3.

36 Formalizing Maximum Likelihood Estimation (MLE)

In our previous intuitive example, we saw that MLE asks a fundamental question: "Out of
all possible parameter values, which one makes the data we actually observed the most probable?"
Let’s formalize this powerful concept.

36
S TATISTICAL I NFERENCE L ECTURE N OTES

Definition 36.1: The Likelihood Function

iid
Let X1 , X2 , . . . , Xn ∼ Fθ, having a Joint PDF (or PMF) denoted by f θ (x). When we
view this joint density not as a function of the data x (which is fixed after the experi-
ment), but rather as a function of the unknown parameter θ, we call it the Likelihood
Function. It is denoted by L(θ; x):
n
L(θ; x) = ∏ f θ ( xi )
i =1

Definition 36.2: Maximum Likelihood Estimator (MLE)

Let Θ be the parameter space (the set of all valid values for θ). The Maximum Likeli-
hood Estimator, denoted as θ̂ML , is the value of θ ∈ Θ that maximizes the likelihood
function:
θ̂ML = argmax L(θ; x)
θ∈Θ

36.1 The Log-Likelihood Trick

Products (∏) are notoriously difficult to differentiate when applying calculus. To find the
maximum of L(θ; x), we rely on a brilliant mathematical convenience: the natural logarithm.

Concept: Log-Likelihood Function

We define the Log-Likelihood Function, denoted by ℓ(θ; x), as:

n
ℓ(θ; x) = log L(θ; x) = ∑ log f θ(xi )
i =1

Crucial Remarks on MLE:

[1] It is a true statistic: As θ̂ML is the argmax of the likelihood, it depends purely on the
observed sample x and not on any unknown parameters.

[2] Validity: Because the search is restricted to θ ∈ Θ, the resulting estimator is guar-
anteed to lie within the valid parameter space (unlike MoM, which sometimes falls
outside it).

[3] Maximization Equivalence: Because the natural logarithm (log x) is a monotonically

strictly increasing function, the θ that maximizes the log-likelihood ℓ(θ) is exactly the
same θ that maximizes the original likelihood L(θ).

37
S TATISTICAL I NFERENCE L ECTURE N OTES

Original Likelihood (Product)

L(θ; x)

θ
θ̂ ML
Argmax
Log-Likelihood (Sum)
is preserved
ℓ(θ; x) = log L θ

θ̂ ML

37 Finding MLE via Calculus (F.O.C.)

If the likelihood function is differentiable and the maximum does not lie on the boundary of
the parameter space, we can find the MLE by finding the roots of the First Order Condition
(F.O.C.) and verifying concavity using the Second Order Condition.

38
S TATISTICAL I NFERENCE L ECTURE N OTES

Example: Example: Generalized Exponential MLE

Let X1 , . . . , Xn ∼ Exp(θ ), where f ( x ) = 1θ e− x/θ and θ ∈ (0, ∞). Let’s find θ̂ ML .

1. Formulate the Log-Likelihood:
( )
n
1 − xi /θ 1 1 n
L(θ; x) = ∏ e = n exp − ∑ xi
i =1
θ θ θ i =1

1 n
ℓ(θ; x) = log θ −n − ∑ xi
θ i =1
= −n log θ − θ −1 ∑ xi

2. First Order Condition (F.O.C.): Take the derivative with respect to θ and set it to
0.
∂ℓ n
= − + θ −2 ∑ x i = 0
∂θ θ
n ∑x
= 2i
θ θ
∑ xi
nθ = ∑ xi =⇒ θ̂ ML =
n
= x̄n

3. Verify the Maximum (Second Derivative): We must ensure this critical point is a
peak, not a valley.
∂2 ℓ n
2
= 2 − 2θ −3 ∑ xi
∂θ θ
Evaluate at θ̂ = x̄:

∂2 ℓ n 2n x̄ n 2n n
= − 3 = 2− 2 =− 2
∂θ 2 x̄ 2 x̄ x̄ x̄ x̄
θ = x̄

Since n > 0 and x̄2 > 0, the second derivative is strictly negative (< 0). Therefore,
θ̂ ML = x̄n is indeed the global maximum!

38 Multi-Parameter MLE: The Normal Distribution

When estimating a vector of parameters θ = [θ1 , θ2 ] T , we must solve a system of partial

derivatives. To verify it is a maximum, we cannot just check a single second derivative; we
must verify that the multi-dimensional curvature forms a peak. We do this by checking if
the Hessian Matrix is Negative Definite.

39
S TATISTICAL I NFERENCE L ECTURE N OTES

Example: MLE for N (µ, σ2 )

iid
Let X1 , . . . , Xn ∼ N (µ, σ2 ). Here, θ = (σ2 ).
µ

1. The Likelihood and Log-Likelihood:

n
( x i − µ )2
 
1
L(µ, σ ; x) = ∏ √
2
exp −
i =1 2πσ2 2σ2
( )
n
1
n = (2πσ2 )−n/2 exp − 2 ∑ ( xi − µ)2
2σ i=1

n n 1 n
ℓ(µ, σ2 ; x) = − log(2π ) − log(σ2 ) − 2 ∑ ( xi − µ)2
2 2 2σ i=1
2. First Order Conditions (Set Partial Derivatives to 0): For µ:

∂ℓ 1 1
= − 2 × 2 ∑( xi − µ)(−1) = 2 ∑ ( xi − µ ) = 0 =⇒ ∑ xi = nµ =⇒ ˆ¯ = x̄
∂µ 2σ σ

For σ2 (Treat σ2 as a single variable block):

 
∂ℓ n 1 n 1
∂σ 2
=− 2− −
2σ 2
2( σ ) 2 ∑ ( x i − µ )2 = − 2 + 4
2σ 2σ ∑ ( x i − µ )2 = 0
Multiply by 2σ4 :

1
−nσ2 + ∑( xi − µ)2 = 0 =⇒ œ̂2 =
n∑
(xi − ˆ¯)2

Note: As seen previously with MoM, the MLE for variance is the biased sample
variance s2n .

38.1 Checking the Hessian Matrix

To confirm (µ̂, σ̂2 ) is a maximum, we calculate the Hessian matrix H, which holds all second-
order partial derivatives:  
∂2 ℓ ∂2 ℓ
H(µ, σ2 ) =  22
∂µ ∂µ∂σ2 
∂ ℓ ∂2 ℓ
∂σ2 ∂µ ∂ ( σ 2 )2

Evaluating the second derivatives:

∂2 ℓ n
2
=− 2
∂µ σ
2
∂ ℓ n 1
2 2
= 4 − 6 ∑ ( x i − µ )2
∂(σ ) 2σ σ
2
∂ ℓ 1
2
= − 4 ∑ ( xi − µ )
∂µ∂σ σ

40
S TATISTICAL I NFERENCE L ECTURE N OTES

Now, we evaluate the Hessian exactly at our critical point (µ̂, σ̂2 ). Since ∑( xi − µ̂) = 0, the
cross-diagonal terms vanish. Furthermore, substituting ∑( xi − µ̂)2 = nσ̂2 into the bottom
2
right term gives 2nσ̂4 − nσ̂σ̂6 = − 2nσ̂4 .

− σ̂n2
 
2 0
H(µ̂, σ̂ ) =
0 − 2nσ̂4

Result: Negative Definite Condition

 
a 0
For a 2 × 2 diagonal matrix M = to be Negative Definite (which guarantees a
0 d
local maximum), we require:

1. a < 0

2. The Determinant | M | = ad − bc > 0

Here, a = −n/σ̂2 which is strictly < 0.

2
The determinant is (−n/σ̂2 ) × (−n/2σ̂4 ) = 2nσ̂6 , which is strictly > 0.
Thus, the Hessian is Negative Definite, proving that (µ̂, σ̂2 ) is indeed the global MLE.

39 MLE at the Boundaries (When Calculus Fails)

The calculus-based approach only works when the maximum lies in the interior of the pa-
rameter space Θ. But what if the support of the distribution depends entirely on the param-
eter itself?

! Important: The Trap of the Uniform Distribution

If you blindly apply calculus to U (0, θ ), you will take the derivative of the likelihood,
set it to zero, and find that it has no roots. The calculus approach fundamentally fails
here. Let’s see why, and how to logically deduce the MLE.

41
S TATISTICAL I NFERENCE L ECTURE N OTES

Example: MLE for the Uniform Distribution

iid
Let X1 , . . . , Xn ∼ U (0, θ ). The parameter is θ ∈ (0, ∞). The PDF is f ( x ) = 1θ if
x ∈ [0, θ ], and 0 otherwise.
1. The Indicator Function Likelihood: The likelihood is the product of the PDFs.
It is only non-zero if every single observation xi is ≤ θ. If even a single data point is
greater than θ, the entire product collapses to 0 (because the data would be impossi-
ble under that model). (
1
n if xi ≤ θ ∀i
L(θ; x) = θ
0 otherwise
2. The Logical Constraint: Saying "every xi ≤ θ" is mathematically identical to
saying "the maximum observed value is ≤ θ". Therefore, θ ≥ X(n) .
3. Maximizing the Likelihood: We want to maximize L(θ ) = θ1n subject to the
constraint θ ≥ X(n) . Because θ is in the denominator, the function θ1n is a strictly
decreasing function. To make a fraction as large as possible, we must make the
denominator (θ) as small as possible.
What is the absolute smallest value θ can take without violating our constraint? It is
X( n ) .
=⇒ ˆ`ML = X(n)

Likelihood Function for U (0, θ )

L(θ; x) Max → ˆ`ML = X(n)

The function strictly
decreases. The peak is
exactly at the bound-
ary.

θ (Parameter space)

x1 x2 x n −1
X( n )
Generalizing the Uniform Bounds
iid
Consider X1 , . . . , Xn ∼ U (α, β) where α < β. We must simultaneously estimate both
bounds. The likelihood is ( β−1α)n provided that all data falls within the bounds.
Constraints: 1. All xi ≥ α =⇒ α ≤ X(1) (The minimum) 2. All xi ≤ β =⇒ β ≥ X(n)
(The maximum)
To maximize ( β−1α)n , we must make the interval ( β − α) as small as possible.

ˆ ML = X(1) .
• We make α as large as possible: ff

• We make β as small as possible: fiˆ ML = X(n) .

42
S TATISTICAL I NFERENCE L ECTURE N OTES

40 Advanced MLE: Two-Parameter Uniform Distribution

In our previous discussion, we looked at U (0, θ ). Let us elevate this and look at the general
Uniform distribution where both boundaries are unknown.

Example: MLE for U (α, β)

iid
Let X1 , X2 , . . . , Xn ∼ U (α, β) where α < β. Our parameter vector is θ = ( αβ).
The PDF for a single observation is:
(
1
if α ≤ x ≤ β
f θ ( x ) = β−α
0 otherwise

The Joint Likelihood Function demands that every single observation falls within the
interval [α, β]. If even one observation falls outside, the entire product becomes
zero. We can express this using the order statistics X(1) (the minimum) and X(n) (the
maximum):
(
1
n if α ≤ X(1) ≤ X(n) ≤ β
L(α, β; x) = ( β−α)
0 otherwise

To truly understand how to maximize this, let’s map out the likelihood space based on the
constraints.

β condition
β < X(n) β ≥ X(n)
α condition
1
α ≤ X (1) 0 ( β−α)n

α > X (1) 0 0

1
The Optimization Logic: To maximize ( β−α)n
, we must make the denominator ( β − α) as
small as legally possible.

• Fixing β: If we hold β constant, we need to make α as large as possible. The upper

bound for α is the smallest data point, X(1) . Thus, α̂ ML = X(1) .

• Fixing α: If we hold α constant, we need to make β as small as possible. The lower

bound for β is the largest data point, X(n) . Thus, β̂ ML = X(n) .

43
S TATISTICAL I NFERENCE L ECTURE N OTES

Fixing β: Maximizing w.r.t α Fixing α: Maximizing w.r.t β

L(α, β; x) α̂ ML L(α, β; x) β̂ ML

α→ β→
Valid α X (1) X( n ) X (1) X( n ) Valid β

41 MLE for the Double Exponential (Laplace) Distribution

Now, let’s examine a distribution where the calculus works, but requires careful handling
of absolute values. This is a classic example that proves why the median is statistically
significant.
iid
Let X1 , . . . , Xn ∼ Double Exponential(µ, β). The parameter vector is θ = (µβ), where µ ∈ R
and β > 0.
| x − µ|
 
1
f θ(x) = exp −
2β β

1. Formulate the Log-Likelihood:

n
| xi − µ |
 
1
L(µ, β; x) = ∏ exp −
i =1
2β β
1 n
ℓ(µ, β; x) = −n log 2 − n log β − ∑ | xi − µ|
β i =1

2. Maximize with respect to β: Taking the partial derivative with respect to β and setting it
to 0 (First Order Condition):
n
∂ℓ n 1
∂β
=− + 2
β β ∑ | xi − µ | = 0
i =1
1 n
n i∑
=⇒ β̂ ML = | xi − µ | [Solution to F.O.C.]
=1

Check Second Derivative (S.O.C.):

∂2 ℓ n 2
∂β 2
= 2− 3
β β ∑ | xi − µ |

Plugging in β̂ = 1
n ∑ | xi − µ|, we get n
β̂2
− 2n
β̂3
β̂
= − β̂n2 < 0. This confirms it is a maximum.

3. Maximize with respect to µ: Observe the log-likelihood function. To maximize ℓ with

respect to µ (holding β fixed), we must minimize the sum of absolute deviations: ∑in=1 | xi −
µ |.

44
S TATISTICAL I NFERENCE L ECTURE N OTES

Concept: Why the Median?

What value of µ minimizes the total absolute distance to all points? Calculus tells us
that the derivative of | xi − µ| is −sgn( xi − µ). If we set the sum of derivatives to zero:
n
∑ sgn(xi − µ) = 0
i =1

This equation implies we need an equal number of positive and negative signs. In
other words, an equal number of data points must lie to the left of µ as to the right of
µ. This is the exact definition of the Median!

µ̂ ML = Median

| x (5) − µ |
| x (1) − µ |

x (1) x (2) x (3) x (4) x (5)

Moving µ left or right increases the total distance
because you move away from a denser cluster of points.

Therefore, the MLEs for the Double Exponential are:

µ̂ ML = Median{ X1 , X2 , . . . , Xn }
1 n
β̂ ML = ∑ | xi − µ̂ ML |
n i =1

41.1 Constrained Parameter Spaces

Suppose we have prior knowledge that the dispersion parameter is bounded, e.g., fi ≥ 1.
How does this modify our result?
If our calculated β̂ unconstrained = n1 ∑ | xi − µ̂| happens to be less than 1, we cannot use it.
Because the likelihood function is concave (shaped like a downward parabola), if the un-
constrained peak is outside our valid region, the maximum within the valid region occurs
exactly at the boundary.

( )
1 n
n i∑
β̂ ML = max 1, | xi − µ̂ ML |
=1

42 Critical Remarks on MLE Properties

As powerful as MLE is, it is not flawless. We must recognize its limitations mathematically:

45
S TATISTICAL I NFERENCE L ECTURE N OTES

! Important: 1. MLE is not necessarily unbiased.

1
• Example: For Xi ∼ N (µ, σ2 ), we proved earlier that σ̂ML
2 = s2n = n ∑( xi − x̄ )2 ,
which is biased.
n
• Example: For Xi ∼ U (0, θ ), we proved θ̂ ML = X(n) . However, E[ X(n) ] = n +1 θ ̸=
θ, making it biased.

! Important: 2. MLE may not be unique.

• Example: Consider the Double Exponential distribution with an even sample
size (e.g., n = 4). Any value between X(2) and X(3) serves as the median and
perfectly minimizes the absolute distance. Thus, there are infinitely many valid
MLEs for µ in this specific case.

! Important: 3. MLE may not exist.

• Example: Let Xi ∼ U ([0, θ )). Notice the interval is open at θ. Because the data
can never literally equal θ, the parameter θ must be strictly strictly greater than
X(n) . The likelihood function increases as θ → X(n) , but since θ cannot equal
X(n) , the supremum is never reached within the valid parameter space. Thus, a
strict maximum does not exist.

43 The Invariance Property of MLE

One of the greatest algebraic advantages of Maximum Likelihood Estimation is its invari-
ance to transformations. This makes calculating estimates for complex functions of param-
eters incredibly easy.

Result: Invariance Property

Suppose θ is a parameter and g(θ) is a function of θ. Let ψ = g(θ).

If the MLE of θ is θ̂ML , then the MLE of ψ is simply:

ψ̂ ML = g(θ̂ML )

46
S TATISTICAL I NFERENCE L ECTURE N OTES

Example: Application: Estimating a Probability in Poisson

iid e−θ θ x
Let X1 , . . . , Xn ∼ Poisson(θ ) for i = 1, . . . , n. We know the PDF is f ( x ) = x! .
Step 1: Find MLE of θ
n
e − θ θ xi
L(θ; x) = ∏ xi !
i =1
ℓ(θ; x) = −nθ + ∑ xi log θ − ∑ log( xi !)

∂ℓ ∑ xi
= −n + = 0 =⇒ θ̂ ML = X̄n
∂θ θ
∂ ℓ 2 ∑ xi
*(Checking S.O.C: ∂θ 2 = − θ 2 < 0, so it is a maximum).*
Step 2: Estimate a Function Suppose we want to estimate the probability that ex-
actly 1 event occurs:
e−θ θ 1
ψ = Pθ ( X = 1) = = θe−θ
1!
By the Invariance Property, the MLE for this probability is trivially:

ψ̂ ML = θ̂ ML e−θ̂ ML = X̄n e−X̄n

Professor’s Note: While easy to calculate, X̄n e−X̄n is not an unbiased estimator for
ψ! If we wanted an unbiased estimator, we could define an indicator function T =
I ({ X1 = 1}). Since E[ T ] = 1 · P( X1 = 1) + 0 = ψ, T is unbiased.

44 Fundamental Probability Inequalities

As we move toward understanding the asymptotic (large sample) behaviors of these esti-
mators, we must build a bridge using mathematical inequalities. These inequalities allow
us to bound probabilities even when we do not know the exact distribution!

44.1 1. Markov’s Inequality

This inequality gives an upper bound on the probability that a non-negative random vari-
able is exceptionally large.

Result: Markov’s Inequality

If X is a non-negative random variable (X ≥ 0), then for any t > 0:

E( X )
P( X ≥ t) ≤
t

Proof (for Continuous Case): Since X is non-negative, its PDF f ( x ) = 0 for x < 0. By

47
S TATISTICAL I NFERENCE L ECTURE N OTES

definition of Expectation:
Z ∞
E( X ) = x f ( x ) dx
0
Z t Z ∞
= x f ( x ) dx + x f ( x ) dx
0 t
Rt
Since X is non-negative, the first integral 0 x f ( x ) dx ≥ 0. Therefore, dropping it makes the
right side smaller:
Z ∞
E( X ) ≥ x f ( x ) dx
t
Inside this integral, the smallest value x can take is t. If we replace x with the constant t, we
make the integral smaller still:
Z ∞
E( X ) ≥ t f ( x ) dx
t
Z ∞
E( X ) ≥ t f ( x ) dx
t
E(X)
E( X ) ≥ t · P( X ≥ t) =⇒ P(X ≥ t) ≤ ■
t

44.2 2. Chebyshev’s Inequality

While Markov relies only on the mean, Chebyshev uses the variance to provide a much
tighter bound regarding how far data strays from the center.

Result: Chebyshev’s Inequality

If X is a random variable with finite expectation E( X ) = µ and finite variance

Var( X ) = σ2 , then for any t > 0:

σ2
P(| X − µ| > t) ≤
t2

Proof (Using Markov’s): Let Y = ( X − µ)2 . Because it is squared, Y is a non-negative

random variable. Note that the event {| X − µ| > t} is mathematically identical to {( X −
µ)2 > t2 }. Apply Markov’s Inequality to Y:
E (Y )
P (Y > t 2 ) ≤
t2
E[( X − µ)2 ]
P(( X − µ)2 > t2 ) ≤
t2
By definition, E[( X − µ)2 ] = Var( X ) = σ2 . Therefore:
σ2
P(| X − µ| > t) ≤ ■
t2

E[| X −µ|r ]
Extension: This can be extended to higher moments. P(| X − µ| > t) ≤ tr for r =
1, 2, 3 . . . provided the r th moment exists.

48
S TATISTICAL I NFERENCE L ECTURE N OTES

45 Applying Chebyshev: Sample Size Calculation

Inequalities are immensely practical. We can use them to guarantee estimation precision
without knowing the underlying distribution.
Example: Ensuring Precision via Sample Size
iid
Let X1 , . . . , Xn ∼ Fθ. Assume E( Xi ) = µ and Var( Xi ) = 1. We are using the sample
mean X̄n to estimate µ.
Question: What is the minimum sample size (n) that ensures our estimate is within
1 unit of the true mean with at least 95% probability? Mathematically: Find n such
that P(| X̄n − µ| ≤ 1) ≥ 0.95.

Solution: First, we analyze the sampling distribution of our estimator X̄n :

h i
∑ Xi nµ
• E[ X̄n ] = E n = n =µ
h i
∑ Xi 1 n ·1 1
• Var( X̄n ) = Var n = n2 ∑ Var( Xi ) = n2
= n

Using Chebyshev’s Inequality on X̄n with t = 1:

Var( X̄n ) 1
P(| X̄n − µ| > 1) ≤ 2
=
1 n

We want the probability of being close (≤ 1) to be ≥ 0.95. This means the probability of being
far (> 1) must be ≤ 0.05. Since Chebyshev guarantees the "far" probability is bounded by n1 ,
we simply set:
1
= 1 − 0.95 = 0.05
n
1
n= = 20
0.05

To be safe, we always take the upper integer bound. Thus, a sample size of n = 20 mathe-
matically guarantees our desired precision, regardless of the shape of the population distribution.

Distribution of X̄n for n = 5 Distribution of X̄n for n = 25

µ
µ Variance n1
shrinks, mass
centers on µ.

µ−1 µ+1 µ−1 µ+1

49