MTL390: Statistical Methods

Instructure: Dr. Biplab Paul

January 16, 2025

Lecture 6

Sample Moments and Their Distributions (Cont.)

Finite Population
Let {c1 , c2 , . . . , cN } be a finite population. Then the population mean is given by
N
1 X
µ= ci
N i=1

and the population variance is

N
2 1 X
σ = (ci − µ)2
N i=1

The following theorem for the sample mean and variance is stated without proof.

Theorem 1. If X1 , X2 , . . . , Xn is a sample of size n (chosen without replacement) from

a population {c1 , c2 , . . . , cN }, then

σ2 N − n
 


E X̄ = µ, and Var X̄ =
n N −1
We remark here that the sample in the theorem is not a random sample, and the
Xi ’s are not independent and identically distributed (iid) random variables. In sampling
without replacement from a finite population, the independence condition is not satisfied.
Suppose that a sample of size 2 is taken from a finite population {c1 , c2 , . . . , cN } without
replacement. Let X1 be the outcome on the first draw and X2 be the outcome on the
second draw. Then

1 1
P [X1 = c1 ] = , P [X2 = c2 | X1 = c1 ] = , P [X2 = c2 | X1 = c2 ] = 0.
N N −1

1
 Thus, the PMF of X2 depends on the outcome of the first draw (that is, on the value
of X1 ), and X1 and X2 are not independent. Note, however, that
N
X
P [X2 = c2 ] = P (X1 = ci )P (X2 = c2 | X1 = ci )
i=1
N
X
= P (X1 = ci )P (X2 = c2 | X1 = ci )
i̸=2
1
= .
N
Thus, X1 and X2 have same distribution. A similar argument can be used to show that
X1 , X2 , . . . , Xn all have the same distribution, but they are not independent.
Remark Sampling from a probability distribution is sometimes referred to as sampling
from an infinite population since one can obtain samples of any size one desires even if
the population is finite (by sampling with replacement).
Example Let the population consist of the numbers {1, 2, 3, 4, 5}. Consider all possible
samples consisting of three numbers randomly chosen without replacement from this
population. Obtain the distribution of the sample mean.

Solution Disregarding the order, it is clear that there are 53 = 10 equally likely possible
samples of size 3. They are:

(1, 2, 3), (1, 2, 4), (1, 2, 5), (1, 3, 4), (1, 3, 5), (1, 4, 5), (2, 3, 4), (2, 3, 5), (2, 4, 5), and (3, 4, 5).

Calculating the mean x̄ for each of the samples, we will get the sampling distribution of
x̄ as
x̄ p(x̄)
2 1
1 10
7 1
3 10
8 2
3 10
3 2
1 10
10 2
3 10
11 1
3 10
4 1
1 10

For example, in the table, P X̄ = 38 = 10 2

because the two samples (1, 2, 5) and (1, 3, 4)
8
both give x̄ = 3 , which is an estimate of the population mean µ.

Order Statistics
The function X(k) of n-dimensional (X1 , X2 , . . . , Xn ) that takes on the value x(k) in each
possible sequence (x1 , x2 , . . . , xn ) of values assumed by (X1 , X2 , . . . , Xn ) is known as the
k-th order statistic or statistic of order k. {X(1) , X(2) , . . . , X(n) } is called the set of order
statistics for (X1 , X2 , . . . , Xn ).
Theorem 2. Let (X1 , X2 , . . . , Xn ) be an n-dimensional random vector. Let X(k) , 1 ≤
k ≤ n, be the statistic of order k. Then X(k) is also a random variable.
In the following, we assume that X1 , X2 , . . . , Xn are independent and identically dis-
tributed (iid) random variables. In the discrete case, there is no straightforward formula

2
for computing the distribution of any order statistic X(k) , nor for any of the joint distri-
butions. In such cases, direct computation is the most reliable approach.
Let X1 , X2 , . . . , Xn be a random sample from a continuous distribution with pdf f (x).
Let X(1) , . . . , X(n) be a permutation of X1 , X2 , . . . , Xn such that

X(1) ≤ · · · ≤ X(n) .

Then the ordered random variables X(1) , . . . , X(n) are called the order statistics of the
random sample X1 , X2 , . . . , Xn . Here, X(k) is called the kth order statistic. Because the
underlying distribution is continuous, the equality sign in the ordering could be ignored.
Remark. Although Xi ’s are iid random variables, the random variables X(i) ’s (order
statistics) are neither independent nor identically distributed.
The following result is useful in determining the distribution of functions of more than
one order statistic.

Theorem 3. Let X1 , . . . , Xn be a random sample from a population with pdf f (x). Then
the joint pdf of the order statistics X(1) , . . . , X(n) is given by:
(
n!f (x(1) )f (x(2) ) · · · f (x(n) ), if x(1) < x(2) < · · · < x(n) ,
fX(1) ,...,X(n) (x1 , . . . , xn ) =
0, otherwise.

The pdf of the kth order statistic is provided by the following result.

Theorem 4. The pdf of the kth order statistic X(k) is:

n!  k−1  n−k
fk (x) = fX(k) (x) = f (x) F (x) 1 − F (x) ,
(k − 1)!(n − k)!

for −∞ < x < ∞, where F (x) = P (Xi ≤ x) is the cdf of Xi .

The joint PDF of the order statistics is given by the following result:

Theorem 5. Let X1 , X2 , . . . , Xn be a random sample with a continuous probability density

function f (x) and a distribution function F (x). Let X(1) , X(2) , . . . , X(n) denote the order
statistics. Then, for any 1 ≤ i < k ≤ n and −∞ < x ≤ y < ∞, the joint PDF of X(i)
and X(k) is given by:

n!  i−1  k−i−1  n−k

fX(i) ,X(k) (x, y) = F (x) F (y)−F (x) 1−F (y) f (x)f (y),
(i − 1)!(k − i − 1)!(n − k)!

where f (x) and F (x) are the PDF and CDF of the original distribution, respectively.

Exercise Let X1 , X2 , . . . , Xn be a random sample from the uniform distribution. Find

the followings

(a) The probability density function (pdf) of the k-th order statistic X(k) .

(b) The joint pdf of the second and fifth order statistics, X(2) and X(5) .

3
Sample Characteristics
Let X1 , X2 , . . . , Xn be a sample from a population with distribution function (DF) F .
Empirical distribution function Let
n
1X
Fn∗ (x) = ϵ(x − Xi ),
n i=1

where ϵ(x − Xi ) is an indicator function that is 1 if Xi ≤ x and 0 otherwise. Then nFn∗ (x)
is the number of Xi ’s (1 ≤ i ≤ n) that are less than or equal to x. The function Fn∗ (x) is
called the sample (or empirical) distribution function.
Properties We note the following properties of Fn∗ (x):
• 0 ≤ Fn∗ (x) ≤ 1 for all x,
• Fn∗ (x) is right continuous,
• Fn∗ (x) is non-decreasing,
• Fn∗ (−∞) = 0 and Fn∗ (∞) = 1.
Thus, Fn∗ (x) satisfies the properties of a distribution function.

Order Statistics and emirical distribution If X(1) , X(2) , . . . , X(n) are the order statis-
tics of X1 , X2 , . . . , Xn , then the empirical distribution function can be expressed as

0 if x < X(1) ,

∗
Fn (x) = nk if X(k) ≤ x < X(k+1) , k = 1, 2, . . . , n − 1,

1 if x ≥ X(n) .


Fn∗ (x) itself is a random variable (RV) of the discrete type. The following result is
immediate.
Theorem 6. The random variable Fn∗ (x) has the probability function:
   
∗ i n
P Fn (x) = = [F (x)]i [1 − F (x)]n−i , i = 0, 1, . . . , n,
n i
with mean:
E[Fn∗ (x)] = F (x),
and variance:
F (x)[1 − F (x)]
Var(Fn∗ (x)) = .
n
nFn∗ (x) ∼ b(n, p),
where p = F (x).
Proof. Since ϵ(x − Xi ), i = 1, 2, . . . , n, are iid random variables (RVs), each with the
probability mass function (PMF):
P {ϵ(x − Xi ) = 1} = P (x − Xi ≥ 0) = F (x),
and
P {ϵ(x − Xj ) = 0} = 1 − F (x),
their sum nFn∗ (x) is a b(n, p) random variable, where p = F (x).

4
Corollary 6.1. For each x ∈ R,
P
Fn∗ (x) −
→ F (x) as n → ∞.
Proof. Exercise. Use Law of Large number.
Corollary 6.2. For each x ∈ R,
√
n [F ∗ (x) − F (x)] d
p n →
− Z as n → ∞,
F (x)[1 − F (x)]
where Z ∼ N (0, 1).
Proof. Exercise. Central Limit Theorem (CLT).
We next consider some typical values of the distribution function Fn∗ (x), referred to
as sample statistics.
• sample moment of order k, ak = n1 nj=1 Xjk .
P

• sample central moment of order k, bk = n1 nj=1 (Xj − X̄)k .

P

Clearly, the first sample central moment is:

b1 = 0,

and the second sample central moment is:

n−1 2
b2 = S .
n
As mentioned earlier, we do not call b2 the sample variance. S 2 will be referred to
as the sample variance, for reasons that will subsequently become clear.
• moment-generating function (MGF):

n
∗ 1 X tXi
M (t) = e .
n i=1

• The sample quantiles are defined as follows:

If 0 < p < 1, the sample quantile of order p, denoted by Zp , is the order statistic
X(r) , where:
(
np if np is an integer,
r=
⌊np⌋ + 1 if np is not an integer.
Here, ⌊x⌋ denotes the largest integer less than or equal to x. Note that if np is
an integer, the p-th sample quantile can take any value between X(np) and X(np+1) .
For p = 1/2, and n is even, we can take any value between X(n/2) and X(n/2+1) , the
two middle values, as the median. It is customary to take the average. Thus the
sample median is defined as
(
X((n+1)/2) if n is odd,
Z1/2 = X(n/2) +X((n/2)+1)
2
if n is even.

5
Exercise A random sample of 25 observations is taken from the interval (0, 1). The
observations are:
0.50 0.06 0.88 0.24 0.21 0.61 0.89 0.58 0.35 0.54
0.07 0.06 0.34 0.56 0.90 0.48 0.23 0.72 0.33 0.41
0.16 0.95 0.29 0.12 0.67

Calculate Empirical CDF and all sample characteristics.