S AMPLING D ISTRIBUTIONS OF E STIMATORS
Rohini Somanathan
2025
D ISTRIBUTIONS OF E STIMATORS
DEFINING A SAMPLING DISTRIBUTION
Estimators are random variables with their own probability distributions
We call these sampling distributions because they are induced by the sample
The sampling distribution of estimator based on a random sample is determined by
the distribution F of the observations
the sample size n
the functional form of the statistic
If F is unknown, then we cannot know the exact sampling distribution
Sometimes, even if F is known, but complication so we use an asymptotic distribution of the
estimator (large-sample approximation)
1 / 21
D ISTRIBUTIONS OF E STIMATORS
THE VALUE OF A SAMPLING DISTRIBUTION
Given a sampling distribution, we can
make appropriate trade-offs between sample size and precision of our estimator since
sampling distributions depend on sample size.
obtain interval estimates rather than point estimates after we have a sample
– an interval estimate is a random interval such that the true parameter lies within this
interval with a given probability (say 95%).
choose between to estimators
for example, we can use the distribution of θ̂ to obtain the MSE Eθ [(θ̂ − θ)2 ]
2 / 21
D ISTRIBUTIONS OF E STIMATORS
SAMPLE SIZE AND PRECISION : EXAMPLE 1
Suppose Xi ∼ N(θ, 4), and we want E(X̄n − θ)2 ≤ .1?
E(X̄n − θ)2 is simply the variance of X̄n
We know the sampling distribution of X̄n
X̄n ∼ N(θ, 4/n)
Using this we have
4
≤ .1 if n ≥ 40
n
3 / 21
D ISTRIBUTIONS OF E STIMATORS
SAMPLE SIZE AND PRECISION : EXAMPLE 2
We are choosing a random sample of size n from a Uniform distribution on [0, θ]
Our estimator is the MLE
U = max{X1 , . . . , Xn }
The CDF of U is given by
0 n
if u ≤ 0
F(u) = u
θ if 0 < u < θ
1 if u ≥ θ
How big should n be if we want the probability that U lies within .1θ to be bigger than 0.95?
Pr(|U − θ| ≤ .1θ) = Pr(θ − U ≤ .1θ) = Pr(U ≥ .9θ) = 1 − F(.9θ) = 1 − 0.9n
log(.05)
For 1 − 0.9n ≥ 0.95, 0.9n ≤ 0.05. With the LHS decreasing in n, n ≥ log(.9) = 28.43. Our
minimum sample size is therefore 29.
4 / 21
S PECIAL SAMPLING DISTRIBUTIONS
A few distributions are essential for statistical inference. We will now define these
Chi-square: The Chi-square with n degrees of freedom (χ2n ) is a type of Gamma distribution.
Student’s t: This is a ratio of a Normal and χ2n
F-distribution: This is ratio of two chi-square distributions, (χ2m and χ2m )
Normal: We are already familiar with this
5 / 21
T HE G AMMA FAMILY
T HE GAMMA FUNCTION
R∞
The gamma function of α is defined as Γ(α) = yα−1 e−y dy
0
R∞ ∞
If α = 1, Γ(α) = e−y dy = −e−y = 1 so Γ(1) = 1
0 0
R∞ ∞ R∞
For α > 1, use the formula for integration by parts: f (y)g′ (y) = f (y)g(y) − f ′ (y)g(y)dy
0 0 0
−yα−1 ∞ R∞
Set f (y) = yα−1 and g′ (y) = e−y and integrate to get Γ(α) = ey + (α − 1) yα−2 e−y dy
0 0
The exponential function goes to zero faster than any polynomial so the first term goes to zero
so we get Γ(α) = (α − 1)Γ(α − 1)
For any integer α > 1, we therefore have
Γ(α) = (α − 1)(α − 2)(α − 3) . . . (3)(2)(1)Γ(1) = (α − 1)!
6 / 21
T HE G AMMA FAMILY
T HE GAMMA DENSITY
x
Define x by y= β , where β > 0. Then dy = β1 dx and can rewrite Γ(α) as
Z∞
x α−1 − βx 1
Γ(α) = e dx
β β
0
Rearranging terms, we see that the RHS is a valid density
Z∞
1 −x
1= α
xα−1 e β dx
Γ(α)β
0
This is the gamma probability density function
1 −x
f (x; α, β) = α
xα−1 e β I(0,∞) (x)
Γ(α)β
7 / 21
T HE G AMMA FAMILY
T HE SHAPE AND SCALE OF A GAMMA DENSITY
1 x
α−1 − β
f (x; α, β) = x e I(0,∞) (x)
Γ(α)β α
α is the shape parameter, higher values
make the density more symmetric
β is the scale parameter, higher values β
compress the density
8 / 21
T HE G AMMA FAMILY
MOMENTS AND OTHER PROPERTIES
Selected Moments: The mean and variance are given by µ = αβ , σ 2 = αβ 2
Gamma Additivity: Let X1 , . . . Xn be independently distributed random variables with
respective gamma densities Gamma(αi , β). Then
n
X Xn
Y= Xi ∼ Gamma( αi , β)
i=1 i=1
Scaling Gamma RVs: Let X be distributed with gamma density Gamma(α, β) and let c > 0.
Then
Y = cX ∼ Gamma(α, βc)
9 / 21
T HE G AMMA FAMILY
ial Distributions
E XPONENTIAL AND C HI - SQUARE DISTRIBUTIONS
the
a ! 0.1, b ! 0.1
ent 1.2 a ! 1, b ! 1
ith a ! 2, b ! 2
1.0 a ! 3, b ! 3
An exponential distribution is a gamma
Gamma p.d.f.
0.8 distribution with α = 1. The density
decreases throughout
0.6
0.4 A Chi-square distribution is a gamma
distribution with α = v2 and β = 2. It’s shape
0.2 depends on α
0 1 2 3 4 5 x
10 / 21
T HE G AMMA FAMILY
R ESULTS ON F UNCTIONS OF χ2 RANDOM VARIABLES
The mean and variance of a χ2v distribution are given by µ = v, σ 2 = 2v
Result 1: If X1 , . . . Xk are independent random variables, and each Xi has a χ2 distribution
with ni degrees of freedom, the sum X1 + · · · + Xk has a χ2 distribution with n1 + . . . nk
degrees of freedom
Result 2: If X1 , . . . Xk are independent random variables, and each Xi has a standard normal
distribution, the sum X12 + · · · + Xk2 has a χ2 distribution with k degrees of freedom
11 / 21
J OINT DISTRIBUTION OF Xn AND SAMPLE VARIANCE
n
1
For a normal random sample we know that the MLEs are µ̂ = X̄n and σ̂ 2 = (Xi − X̄n )2
P
n
i=1
Theorem: If X1 , . . . Xn form a random sample where Xi ∼ N(µ, σ 2 ), then the sample mean X̄n
n
and the sample variance n1 (Xi − X̄n )2 are independent random variables and
P
i=1
n
(Xi − X̄n )2
P
σ2 i=1
X̄n ∼ N(µ, ) and ∼ χ2n−1
n σ2
Notes:
n
(Xi −µ)2
P
1. t i=1
σ2
∼ χ2n since this is the square of a standard normal random variable. We lose a
degree of freedom, when we replace the population mean by the sample mean.
2. We would like to use σ̂ 2 since σ 2 is generally unknown.
3. This result is only true for normal samples
12 / 21
T HE T- DISTRIBUTION
DEFINITION
Let Z ∼ N(0, 1) , let Y ∼ χ2v , and let Z and Y be independent random variables. Then
Z
X = q ∼ tv
Y
v
Features of the t-distribution:
The t-density is symmetric with a maximum value at x = 0.
It is similar to the standard normal but with fatter tails.
13 / 21
T HE T- DISTRIBUTION
SOME USEFUL RESULTS
n
RESULT 1: Define S2n = (Xi − X̄n )2 . The random variable
P
i=1
√
n(Xn − µ)
U= q
2
∼ tn−1
Sn
n−1
√ 2
Proof: We know that n(Xσn −µ) ∼ N(0, 1) and that σSn2 ∼ χ2n−1 . Dividing the first random variable
by the square root of the second, divided by its degrees of freedom, the σ in the numerator and
denominator cancels to obtain U.
Implication: We cannot make statements about |X̄n − µ| using the normal distribution if σ 2 is
n
(Xn −µ)
unknown. This result allows us to use its estimate σ̂ 2 = (Xi − X̄n )2 /n since σ̂/
P
√
n−1
∼ tn−1
i=1
14 / 21
T HE T- DISTRIBUTION
SOME USEFUL RESULTS
RESULT 2 As n → ∞, U −→ Z ∼ N(0, 1)
q √
n−1 n(Xn −µ)
To see why: U can be written as n σ̂ ∼ tn−1 . As n gets large σ̂ gets very close to σ and
n−1
n is close to 1.
F−1 (.55) = .129 for t10 , .127 for t20 and .126 for the standard normal distribution. The differences
between these values increases for higher values of their distribution functions (why?)
15 / 21
AN INTERVAL ESTIMATOR FOR Xn
Given σ 2 , let us see how we can obtain an interval estimator for µ, i.e. an interval which is likely to
contain µ with a pre-specified probability.
n −µ) (Xn −µ)
Since (X √
σ/ n
∼ N(0, 1) , Pr −2 < √
σ/ n
< 2 = .955
2σ 2σ 2σ 2σ
But this event is equivalent to the events − √ n
< Xn − µ < √
n
and Xn − √
n
< µ < Xn + √
n
2σ 2σ
With known σ, each of the random variablesXn − √ n
and Xn + √ n
are statistics. Therefore, we
have derived a random interval within which the population parameter lies with probability
.955, i.e.
2σ 2σ
Pr Xn − √ < µ < Xn + √ = .955 = γ
n n
Notice that there are many intervals for the same γ, this is the shortest one.
16 / 21
CONFIDENCE INTERVALS
DEFINITION
Now, given our sample, our statistics take particular values and the resulting interval either
contains or does not contain µ.
We can therefore no longer talk about the probability that it contains µ because the experiment
has already been performed.
We say that
2σ 2σ
(xn − √ < µ < xn + √ )
n n
is a 95.5% confidence interval for µ.
Alternatively, we may say that µ lies in the above interval with confidence γ or that the above
interval is a confidence interval for µ with confidence coefficient γ
17 / 21
CONFIDENCE INTERVALS
C ONFIDENCE I NTERVALS .. EXAMPLE 1
Using the exact distribution of Xn
X1 , . . . , Xn forms a random sample from a normal distribution with unknown µ and σ 2 = 10.
xn is found to be 7.164 with n = 40.
An 80% confidence interval for the mean µ is given by
r r
10 10
(7.164 − 1.282 ), 7.164 + 1.282 )
40 40
This is (6.523, 7.805). The confidence coefficient is .8
stata command: display invnormal(.9)
18 / 21
CONFIDENCE INTERVALS
C ONFIDENCE I NTERVALS .. EXAMPLE 2
Using the CLT approximation for the distribution of Xn :
Let Xn denote the sample mean of a random sample of size n = 25 from a distribution with
variance 100 and mean µ. In this case, √σn = 2 and, making use of the central limit theorem the
following statement is approximately true:
(Xn − µ)
Pr −1.96 < < 1.96 = .95 or Pr Xn − 3.92 < µ < Xn + 3.92 = .95
2
If the sample mean is given by xn = 67.53, an approximate 95% confidence interval for the
sample mean is given by (63.61, 71.45).
19 / 21
CONFIDENCE INTERVALS
C ONFIDENCE I NTERVALS .. EXAMPLE 3
Using the MLE for the variance and the t-distribution:
We are interested in finding a confidence interval for the mean of a normal distribution but σ 2
is unknown.
We know that
(Xn − µ)
√ ∼ tn−1
σ̂/ n − 1
so can use the t-distribution with (n − 1) degrees of freedom to construct our interval estimate.
With n = 10, xn = 3.22, σ̂ = 1.17, a 95% confidence interval is given by
√ √
(3.22 − (2.262)(1.17)/ 9, 3.22 + (2.262)(1.17)/ 9) = (2.34, 4.10)
(display invt(9,.975) gives you 2.262)
20 / 21