09

31
TX
Chapter 2

Point estimation

2.1 Introduction
-M
Let X1 , X2 , · · · Xn be a random sample from a population with pdf f (x, θ), where θ is a parameter(or
hs
vector of parameters). The functional form of the pdf is assumed to be known but θ is unknown. Let
x1 , x2 , · · · xn be an observed value of the sample. Our objective is to use the information in the sample
to estimate θ or a function of θ.

Let us recall the following definition of a statistic

at

Definition 2.1.1. Statistic

A statistic T (X1 , X2 , · · · , Xn ) is a real valued function of a random sample independent of θ.
-M

Definition 2.1.2. Estimator

If T = T (X1 , X2 , · · · , Xn ) is used to provide an estimate about or information about θ, then T is called
an estimator.

Definition 2.1.3. Estimate

A real value of T = T (x1 , x2 , · · · , xn ) is called an estimate.

Note that T = T (X1 , X2 , · · · , Xn ) is a random variable but an estimate is not a random variable. It is a
AK

value obtained from the sample.

2.2 Types of estimates

M

There are two types of estimates, namely

1. Point estimates

29
 09
M AKERERE U NIVERSITY MTX3109 - STATISTICAL INFERENCE

2. Interval estimates

31
A point estimate of θ is a numerical value derived from the sample.
An interval estimate of θ is a pair of two values (numerical) derived from the sample between which θ is
likely to lie.

TX
2.3 Point estimation

Point estimation of an unknown parameter θ is done in the following way: The observed values x1 , x2 , . . . , xn

-M
of a random sample X1 , X2 , . . . , Xn are used to suggest an approximation of θ of the form t(x1 , x2 , . . . , xn ),
where t is such that t(X1 , X2 , . . . , Xn ) is a statistic.

Note: t does not depend on θ, or any other unknown parameter. Then the random variable Θ̂ =
t(X1 , X2 , . . . , Xn ) is called an estimator of θ and a numerical value θ̂ = t(x1 , x2 , . . . , xn ) an estimate of
θ.

Example 2.3.1. Let the mean µ of the population distribution be unknown. Let (X1 , X2 , . . . , Xn ) be a
hs
sample drawn from the population, then the following could be used as estimates of µ:
x +x
Sample mean x̄, sample median x̂, (n) 2 (1) , x̄+x̂
2
. etc

Note that each of those estimates is a function of (x1 , x2 , . . . , xn ). We shall use θ̂ to denote an estimate
at

of the parameter θ and the random variable Θ̂ which takes on θ̂ as its value as an estimator of θ.

Θ̂ = t(X1 , X2 , . . . , Xn )
-M

Thus an estimator of a parameter is a statistic.

Two problems immediately arise:

1. What are the methods used to construct estimates for a given parameter?

2. How to evaluate the quality of an estimator?

AK

2.3.1 Methods of Parameter estimation

We concern ourselves with only two methods, namely,

• Method of moments
M

• Method of maximum likelihood

J ULIET NAKAKAWA N SUMBA - M ATHS D EPT PAGE 30 OF 110

 09
M AKERERE U NIVERSITY MTX3109 - STATISTICAL INFERENCE

[Link] The method of Moments

31
Suppose θ, the parameter in the density f (x; θ) has k components (k unknown parameters). E.g., if
f (x; θ) is the pdf of N (µ, σ 2 ), then θ = (µ, σ 2 ) and thus k = 2. If f (x; θ) is exponential, then k = 1. In
this method, we equate the k th sample moments about the origin

TX
1X k
Mk∗ = Xi
n
to the k th population moments about the origin
µ0k = E(X k ),
i.e.,

-M Mk∗ = µ0k
and then solve the system of equations for the parameters θi (i = 1, 2, . . . , k).
Example 2.3.2. Let X1 , X2 , · · · , Xn be a random sample drawn from N (µ, σ 2 ). The first two raw
sample moments are
1X
M1∗ = Xi = X̄
hs
n
1X 2
M2∗ = Xi
n
The first two raw population moments are
at

µ01 = E(X) = µ
µ02 = E(X 2 ) = σ 2 + µ2
So the moment estimators µ̂ and σ̂ 2 are solutions to
-M

X̄ = µ̂
1X 2
Xi = σ̂ 2 + µ̂2
n
From which we find
1X 2
σ̂ 2 = Xi − X̄ 2
n
P 2
Xi − nX̄
AK

=
P n
(Xi − X̄)2
= .
n
(Xi −X̄)2
P
Thus µ̂ = X̄ and σ̂ 2 = n
.
M

Example 2.3.3. Suppose the following data were values of a random sample of students giving their
age (in years). If it is assumed that the age of students follows a normal distribution with mean µ and
variance σ 2 . Find the moment estimates of these parameters. The data are 20.2, 19.8, 19.5, 21.6, 22,
21.3.

J ULIET NAKAKAWA N SUMBA - M ATHS D EPT PAGE 31 OF 110

 09
M AKERERE U NIVERSITY MTX3109 - STATISTICAL INFERENCE

Solution

31
(20.2 + 19.8 + 19.5 + 21.6 + 22 + 21.3)
X̄ =
6
= 20.73
(Xi − X̄)2
P
2
σ̂ =

TX
n
1  X  P x 2
2 i
= xi −
n n
1
= 2584.58 − (124.4/6)2
6

-M
= 0.89

So σ̂ = 0.944.
Example 2.3.4. Let X1 , X2 , · · · , Xn be a random sample from a population with pdf

f (x) = (θ + 1)xθ , 0 < x < 1.

Find an estimator of θ using the method of moments.

hs

Solution

Since there is only one parameter µ, the moment estimator is a solution to

at

X̄ = E(X),

where Z 1
θ+1
-M

E(X) = (θ + 1) xθ+1 dx = .
0 θ+2
So the moment estimator is a solution to
θ+1 2x̄ − 1
= X̄ ⇔ θ̂ = .
θ+2 1 − X̄

Exercise
AK

Based on a random sample of size n, find the moment estimator of θ for the following populations

(i) θx (1 − θ)1−x
e−θ θx
M

(ii) , x = 0, 1, 2, · · ·
x!
Γ(2θ) 1−θ
(iii) x (1 − x)θ−1 , 0 < x < 1
(Γ(θ))2

J ULIET NAKAKAWA N SUMBA - M ATHS D EPT PAGE 32 OF 110

 09
M AKERERE U NIVERSITY MTX3109 - STATISTICAL INFERENCE

[Link] The method of maximum likelihood (ML)

31
Maximum likelihood is a relatively simple method of constructing an estimator for an un- known param-
eter θ. It was introduced by R. A. Fisher, a great English mathematical statistician, in 1912. Maximum
likelihood estimation (MLE) can be applied in most problems, it has a strong intuitive appeal, and often

TX
yields a reasonable estimator of θ. Furthermore, if the sample is large, the method will yield an excellent
estimator of . For these reasons, the method of maximum likelihood is probably the most widely used
method of estimation in statistics.

Definition 2.3.5. Likelihood function

Let X1 , X2 , . . . , Xn ∼ X. Suppose X has density f (x; θ) and θ ∈ × ⊂ R. The Likelihood function of
X1 , X2 , . . . Xn is a function

-M
L(θ, x) = L(θ, x1 , x2 , . . . , xn ) =

Example 2.3.6. X ∼ Bernoulli(θ), θ ∈ (0, 1), then

n
Y

i=1
f (xi ; θ)

f (x; θ) = θx (1 − θ)1−x , x = 0, 1
hs
and Pn Pn
xi
L(θ, x) = θ i=1 (1 − θ)n− i=1 xi
xi = 0, 1

Definition 2.3.7. A value θ̂, (where θ̂ = t(x1 , x2 , . . . , xn )) which maximizes the likelihood function
at

L(θ, x) over all θ ∈ Θ is called the maximum likelihood estimate, i.e., if

L(t(x1 , x2 , . . . , xn ), x1 , x2 , . . . , xn ) ≥ L(θ, x1 , x2 , . . . , xn ), for all θ ∈ Θ,

-M

then
θ̂ = t(x1 , x2 , . . . , xn )
is called the ML estimate of θ and the corresponding variable

Θ̂ = t(X1 , X2 , . . . , Xn )

is called a maximum likelihood estimator.

AK

Remark 2.3.8.

1. The ML estimator is not necessarily unique.

2. The ML estimator is not necessarily unbiased.

M

3. In many cases, but not always, the maximum of L can be found by differentiation methods.

4. Since L is a product, it is always usually more convenient to maximize lnL, (which is a sum of
terms). Since ln L is monotone, any value of θ that maximizes L also maximizes ln L.

J ULIET NAKAKAWA N SUMBA - M ATHS D EPT PAGE 33 OF 110

 09
M AKERERE U NIVERSITY MTX3109 - STATISTICAL INFERENCE

Definition 2.3.9. Log-likelihood function

31
The function l(θ, x) = ln L(θ, x) = ln l(θ, x1 , x2 , . . . , xn ) is called the log likelihood function of
X1 , X2 , . . . , Xn . Here, x = (x1 , x2 , . . . , xn ).
Definition 2.3.10. Score function
∂
The function S(θ, x) = S(θ, x1 , x2 , . . . , xn ) = ∂θ
l(θ, x) is called the score function.

TX
Definition 2.3.11. Information function
The function
∂ ∂2
I(θ, x) = I(θ, x1 , x2 , . . . , xn ) = − S(θ, x) = − 2 l(θ, x)
∂θ ∂θ
is called the information function of X1 , X2 , . . . , Xn .

and checking that I(θ̂, x) > 0.

S(θ, x) = 0
-M
Hence, in many cases, Θ̂ can be obtained by solving the maximum likelihood equation

i.e.,
∂
∂θ
l(θ, x) = 0

Example 2.3.12. Consider X1 , X2 , . . . , Xn is a random sample from X ∼Ber(θ), θ ∈ (0, 1) i,e f (x; θ) =
hs
θx (1 − θ)1−x , x = 0, 1 Then.
n
Y Pn Pn
L(θ, x) = θx (1 − θ)1−x = θ i=1 xi
(1 − θ)n− i=1 xi
(2.1)
i=1
n n
at

X X
l(θ, x) = ln L(θ, x) = xi ln θ + (n − xi ) ln(1 − θ) (2.2)
i=1
P n Pni=1
∂ x i n − i=1 xi
S(θ, x) = l(θ, x) = i=1 − (2.3)
∂θ θ (1 − θ)
-M

Pn
n − ni=1 xi
P
∂ i=1 xi
I(θ, x) = − S(θ, x) = − (2.4)
∂θ θ2 (1 − θ)2
Solving S(θ, x) = 0 yields
Pn
n − ni=1 xi
P
i=1 xi
− =0
θ (1 − θ)
n
X Xn
⇒ (1 − θ) xi − θ(n −
AK

xi ) = 0
i=1
Pi=1
n
i=1 xi
⇒ θ̂ = = x̄
n

and
M

Pn
n − ni=1 xi
P
i=1 xi
I(θ̂, x) = + > 0.
x̄2 (1 − x̄)2
Since I(θ̂, x) > 0, we conclude that θ̂ = x̄ is a ML estimate of θ and Θ̂ = X̄ is a ML estimator (MLE).

J ULIET NAKAKAWA N SUMBA - M ATHS D EPT PAGE 34 OF 110

 09
M AKERERE U NIVERSITY MTX3109 - STATISTICAL INFERENCE

Example 2.3.13. Let X1 , X2 , . . . , Xn be a random sample from N (µ, σ 2 ). Find the MLE of µ if σ 2 is

31
known. Put θ = µ.

Solution

The pdf of the population is

TX
1 1 x−θ 2
f (x, θ) = √ e− 2 ( σ )
2πσ

n
1 1 xi −θ 2 1 1 Pn xi −θ 2
e− 2 ( σ ) = √ e− 2 i=1 ( σ )
Y
L(θ, x) = √ (2.5)
2πσ ( 2πσ)n

-M
i=1
n  2
n 2 1 X xi − θ
l(θ, x) = ln L(θ, x) = − ln(2πσ ) − (2.6)
2 2 i=1 σ
n  
∂ X xi − θ
S(θ, x) = l(θ, x) = (2.7)
∂θ i=1
σ2
∂ n
I(θ, x) = − S(θ, x) = 2 (2.8)
∂θ σ
hs
Solving S(θ, x) = 0 yields
n  
X xi − θ
=0
σ2
at

i=1
n
!
1 X
⇒ 2 xi − nθ =0
σ i=1
Pn
i=1 xi
-M

⇒ θ̂ = = x̄
n

Since I(θ̂, x) > 0, we conclude that θ̂ = x̄ is a ML estimate of θ = µ and Θ̂ = X̄ is a ML estimator

(MLE) of µ.

Example 2.3.14. Let X1 , X2 , . . . , Xn be a random sample from N (µ, σ 2 ). Find the MLE of σ 2 if µ is
known. Put θ = σ 2 .
AK

Solution

The pdf of the population is

1 1 2
f (x, θ) = √ e− 2θ (x−µ)
2πθ
M

J ULIET NAKAKAWA N SUMBA - M ATHS D EPT PAGE 35 OF 110

 09
M AKERERE U NIVERSITY MTX3109 - STATISTICAL INFERENCE

n
1 1

31
Y 1 2 1 Pn 2
L(θ, x) = √ e− 2θ (xi −µ) = n e
− 2θ i=1 (xi −µ) (2.9)
i=1
2πθ (2πθ) 2
n
n 1 X
l(θ, x) = ln L(θ, x) = − ln(2πθ) − (xi − µ)2 (2.10)
2 2θ i=1
n
∂ n 1 X

TX
S(θ, x) = l(θ, x) = − + 2 (xi − µ)2 (2.11)
∂θ 2θ 2θ i=1
(2.12)
Solving S(θ, x) = 0 yields
n

-M
n 1 X
= 2 (xi − µ)2
2θ 2θ i=1
Pn
(xi − µ)2
⇒ θ̂ = i=1
n

Next, we compute
∂
hs
I(θ, x) = − S(θ, x)
∂θ
n
∂ hn 1 X 2
i
= − (xi − µ)
∂θ 2θ 2θ2 i=1
n
n 1 X
at

=− 2 + 3 (xi − µ)2
2θ θ i=1
Check
n 1 n
-M

I(θ̂, x) = − + nθ̂ =
2θ̂2 θ̂3 2θ̂2
Pn 2
Pn 2
i=1 (xi −µ) i=1 (Xi −µ)
Since I(θ̂, x) > 0, we conclude that θ̂ = n
is a ML estimate of θ = σ 2 and Θ̂ = n
is
a ML estimator (MLE) of σ 2 .
Example 2.3.15. Suppose that X1 , X2 , · · · Xn form a random sample from a uniform distribution on the
interval (0, θ), where of the parameter θ > 0 but is unknown. Please find MLE of θ.
AK

Solution

The pdf of each observation has the following form:

 1
θ
for 0 ≤ x ≤ θ
f (x) = (2.13)
0 otherwise
M

Therefore, the likelihood function has the form

 1
θn
for 0 ≤ xi ≤ θ
L(θ) =
0 otherwise

J ULIET NAKAKAWA N SUMBA - M ATHS D EPT PAGE 36 OF 110

 09
M AKERERE U NIVERSITY MTX3109 - STATISTICAL INFERENCE

Equivalently, we can write the likelihood function as

31
L(θ) = 0 if max{x1 , x2 , · · · , xn } > θ

and
1
L(θ) = if max{x1 , x2 , · · · , xn } ≤ θ
θn

TX
Therefore, looking at the figure 2.1, we see that Θ̂ = max{X1 , X2 , · · · , Xn } is the MLE of θ.

-M
hs
at

Figure 2.1: MLE for the uniform distribution

Remark 2.3.16.
-M

1. Sometimes it is not so easy to find the maximum of the likelihood function as in the examples above
and one might have to do it numerically.

2. Also, MLE does not always exist. Here is an example: let us consider uniform distribution U (0, θ)
and define the density by
AK

 1
θ
for 0 < x < θ
f (x) = (2.14)
0 otherwise
The only difference between (2.13) and (2.14) is that the value of the pdf at the two endpoints 0 and
θ has been changed by replacing the weak inequalities in (2.13) with strict inequalities in (2.14).
Either equation could be used as the pdf of the uniform distribution.
M

However, if (2.14) is used as the pdf, then an MLE of θ will be a value of θ for which θ > xi for
i = 1, · · · , n and which maximizes θ1n among all such values. It should be noted that the possible
values of θ no longer include the value θ = max(x1 , · · · , xn ), since θ must be, strictly greater

J ULIET NAKAKAWA N SUMBA - M ATHS D EPT PAGE 37 OF 110

 09
M AKERERE U NIVERSITY MTX3109 - STATISTICAL INFERENCE

than each observed value xi for i = 1, · · · , n. Since θ can be chosen arbitrarily close to the value

31
max(x1 , · · · , xn ) but cannot be chosen equal to this value, it follows that the MLE of θ does not
exist in this case.

Example 2.3.17. Suppose that X1 , X2 , · · · Xn form a random sample from a uniform distribution on the
interval (θ; θ + 1), where the value of the parameter θ is unknown (−∞ < θ < ∞). Clearly, the density

TX
function is

1 for θ < x < θ + 1
f (x) = (2.15)
0 otherwise

In this example, the likelihood function is

M

1 for θ < xi < θ + 1 (i = 1, · · · n)
L(θ) = (2.16)
0 otherwise

The condition that θ ≤ xi for i = 1, · · · , n is equivalent to the condition that θ ≤ min(x1 , · · · , xn ).

Similarly, the condition that xi ≤ θ + 1 for i = 1, · · · , n is equivalent to the condition that θ ≥
s-
max(x1 , · · · , xn ) − 1. Therefore, we can rewrite the the likelihood function as

1 for max(x1 , · · · , xn ) − 1 ≤ θ ≤ min(x1 , · · · , xn )
L(θ) = (2.17)
0 otherwise
h

Thus, we can select any value in the interval [max(x1 , · · · , xn ) − 1, min(x1 , · · · , xn )] as the MLE for θ.
Therefore, the MLE is not uniquely specified in this example.
at

Exercise 2.3.18.

1. Let X1 , · · · , Xn be an i.i.d sample from a Poisson distribution with parameter λ, i.e.,

-M

λx e−λ
f (x) = , x = 0, 1, . . .
x!
Find the MLE of the parameter λ.

2. Let X1 , · · · , Xn be an i.i.d sample from an exponential distribution with density function

1 − β1 x
f (x) = e , x≥0
β
AK

Find the MLE of the parameter β.

3. The Gamma distribution has a density function as

1 α α−1 −λx
f (x) = λ x e , x≥0
M

Γ(α)
Suppose that α is known, find the MLE of λ based on an i.i.d sample X1 , · · · , Xn .

J ULIET NAKAKAWA N SUMBA - M ATHS D EPT PAGE 38 OF 110

 09
M AKERERE U NIVERSITY MTX3109 - STATISTICAL INFERENCE

4. Let X1 , · · · , Xn be an i.i.d sample from a population with density function

31
 θ−1
θx for 0 < x < 1
f (x) =
0 x≤0

Find the MLE of the parameter θ.

TX
5. Let X1 , · · · , Xn be an i.i.d sample from a population with density function
1
f (x) = e|x−θ| x ∈ (−∞, ∞)
2
Find the MLE of the parameter θ.

-M
6. Let X1 , · · · , Xn be an i.i.d sample from a population with density function

f (x) =
 θ−x
e
0
for x > θ
x≤θ

• Show that the MLE of the parameter θ does not exist.

• Determine another version of the pdf of this same distribution for which the MLE of θ will
hs
exist, and find this estimate.

7. Let X1 , · · · , Xn be an i.i.d sample from a uniform distribution on the interval [θ1 , θ2 ] with density
function  1
θ2 −θ1
for θ1 ≤ x ≤ θ2
f (x) =
at

0 otherwise
Also suppose that the values of θ1 and θ2 are unknown (∞ < θ1 < θ2 < ∞). Find the MLE’s of
θ1 and θ2 .
-M

2.4 Properties of estimators

If Θ̂1 and Θ̂2 are estimators of θ, then for some values of a and b, aΘ̂1 + bΘ̂2 is also an estimator of θ.
Thus, there is an infinite number of estimators if we require that the estimators satisfy certain conditions.
We will now introduce some general properties of estimators.
AK

2.4.1 The Invariance property of MLE

If θ̂ is a MLE of θ based on a sample of size n from a population with pdf f (x; θ), then the MLE of g(θ)
is g(Θ̂).
M

J ULIET NAKAKAWA N SUMBA - M ATHS D EPT PAGE 39 OF 110

 09
M AKERERE U NIVERSITY MTX3109 - STATISTICAL INFERENCE

2.4.2 Unbiased estimators

31
Definition 2.4.1. An Estimator Θ̂ = t(X1 , X2 , . . . , Xn ) is said to be unbiased estimator of θ ∈ Θ if
E(Θ̂) = θ, for all θ ∈ Θ,
where Θ is the set of all possible values of the parameter θ.

TX
Example 2.4.2. Let X1 , X2 , . . . , Xn be a random sample from X, let
E(X) = µ and V ar(X) = σ 2 .
Then X̄ is an unbiased estimator of µ and S 2 is an unbiased estimator of σ 2 .

Proof.
E(X̄) = E
n
X Xi
i=1
n
!
1
= E
n -M n
X

i=1
Xi
!

⇒ E(X̄) = µ ⇒ X̄ is an unbiased estimator of µ

=
1X
n

n i=1
1
E(Xi ) = .nµ = µ
n

Also
n
" # n
1 X 1 X
E(S 2 ) = E (Xi − X̄)2 = E (Xi − X̄)2
hs
n − 1 i=1 n − 1 i=1
n
X n
X n
X
2 2
E (Xi − x̄) = E(Xi − X̄) = E (Xi2 − 2X̄Xi + X̄ 2 )
i=1 i=1 i=1
n n
!
at

X X
= E Xi2 − 2X̄ Xi + nX̄
i=1 i=1
n
! n
!
X X
= E Xi2 − nX̄ 2 =E Xi2 − nE(X̄ 2 )
-M

i=1 i=1
n
X
= E(Xi2 ) − nE(X̄ 2 )
i=1

Recall V ar(X) = E(X ) − [E(X)] ⇒ E(X 2 ) = V ar(X) + [E(X)]2 = σ 2 + µ2 Thus

2 2

n
! n  2 
X
2
X
2 2 σ
E (Xi − X̄) = (σ + µ ) − n + µ = nσ 2 + nµ2 − σ 2 − nµ2 = σ 2 (n − 1).
2

i=1 i=1
n
AK

1
Thus E(S 2 ) = n−1
(n − 1)σ 2 = σ 2 ⇒ S 2 is an unbiased estimator of σ 2

Biased Estimator
M

Definition 2.4.3. If E(Θ̂) 6= θ, then Θ̂ is said to be a biased estimator of θ and

E(Θ̂) − θ
is called the bias of Θ̂.

J ULIET NAKAKAWA N SUMBA - M ATHS D EPT PAGE 40 OF 110

 09
M AKERERE U NIVERSITY MTX3109 - STATISTICAL INFERENCE

Notation

31
The bias of Θ̂ is denoted by Bias(Θ̂) and is given by

Bias(Θ̂) = E(Θ̂) − θ.

TX
Thus, Θ̂ is unbiased for θ if Bias(Θ̂) = 0 for all θ ∈ Θ.

Definition 2.4.4. If Bias(Θ̂) 6= 0 but tends to 0 as n → +∞ for all θ ∈ Θ i.e.,

lim Bias(Θ̂) = 0, for all θ ∈ Θ,

n→+∞

-M
then Θ̂ is an asymptotically unbiased estimator of θ.

Theorem 2.4.5. M2 = n1 ni=1 (Xi − X̄)2 is asymptotically unbiased estimator of σ 2 , where V ar(X) =
σ2
P

Proof.
hs
Bias(M2 ) = E(M2 ) − σ 2
n
! n
1X 2 1 X n−1 2
E(M2 ) = E( (Xi − X̄) = E (Xi − X̄)2 = σ .
n i=1 n i=1 n
 
n−1 1
⇒ Bias(M2 ) = σ2 − σ2 = − σ2.
at

n n
1
lim − σ 2 = 0. Therefore Bias(M2 ) → 0 as n → ∞
n→+∞ n
-M

Theorem 2.4.6. Given that Θ̂1 and Θ̂2 are unbiased estimators of θ we have also

aΘ̂1 + bΘ̂2 ( for a + b = 1 )

is unbiased estimator of θ.
AK

Proof.

E(aΘ̂1 + bΘ̂2 ) = aE(Θ̂1 ) + bE(Θ̂2 ) = aθ + bθ = (a + b)θ = θ since a + b = 1

M

So even the class of unbiased estimators is infinite.

J ULIET NAKAKAWA N SUMBA - M ATHS D EPT PAGE 41 OF 110

 09
M AKERERE U NIVERSITY MTX3109 - STATISTICAL INFERENCE

2.4.3 Efficient estimators

31
Definition 2.4.7 (Minimum variance estimator). An estimator Θ̂ is said to be minimum variance esti-
mator of θ if for any sample size, the variance of Θ̂ is the smallest among all unbiased estimators of
θ.

TX
Definition 2.4.8 (Efficient Estimator). An estimator Θ̂ is said to be efficient if

i) Θ̂ is Unbiased.

ii) Θ̂ has minimum variance.

M
An efficient estimator is also known as a minimum variance unbiased estimator (MVUE).

Question
Is it possible to find an estimator of θ which has minimum variance among all unbiased estimators Θ̂?
s-
Answer
There is no simple, general procedure for finding the MVUE. In the next several lectures we will discuss
several approaches.
h

1. Determine the so-called Cramer-Rao Lower Bound (CRLB) and verify that the estimator achieves
it.
at

2. Find a sufficient statistic and apply the Rao-Blackwell theorem

3. Further restrict the estimator to a class of estimators (e.g., linear or polynomial functions of the
data)
-M

2.4.4 The Cramer-Rao’s lower bound

Theorem 2.4.9. Let X1 , X2 , . . . , Xn be a random sample from X with density f (x; θ), θ ∈ Θ ⊂ R. Let
Θ̂ = t(X1 , X2 , . . . , Xn ) be an unbiased estimator of θ. Then, under smoothness assumptions on f (x; θ),
1
V ar(Θ̂) ≥ , for all θ ∈ Θ̂,
AK

ni(θ)
where  
∂ ln f (xi , θ) 2
i(θ) = E
∂θ
is the fisher information number and
M

1
ni(θ)
is the Cramer-Rao (CR) lower bound.

J ULIET NAKAKAWA N SUMBA - M ATHS D EPT PAGE 42 OF 110

 09
M AKERERE U NIVERSITY MTX3109 - STATISTICAL INFERENCE

1
Theorem 2.4.10. If an unbiased estimator Θ̂ exists that attains Cramer-Rao’s lower bound for the

31
ni(θ)
variance of unbiased estimators, then the estimator is efficient, i.e., If Θ̂ satisfies

• E(Θ̂) = θ
1
• V ar(Θ̂) =

TX
ni(θ)

then Θ̂ is efficient.
Note 2.4.11. Note that if Cramer-Rao’s lower bound is not attained it does not follow that there’s no
unbiased estimator with minimum variance.

-M
Example 2.4.12. Let {X1 , X2 , . . . , Xn } be a random sample from Bernoulli distribution with pmf

f (x, θ) = θx (1 − θ)1−x , x = 0, 1

Is X̄, the sample mean, efficient for θ.

Solution.
hs
First, we check whether or not, X̄ is unbiased.
n n n
1 X 1X 1X
E(X̄) = E( Xi ) = E(Xi ) = θ=θ
n i=1 n i=1 n i=1
at

Next, we find the variance of X̄.

n n n
1 X 1 X 1 X
V ar(X̄) = var( X i ) = var(X i ) = {E(Xi2 ) − [E(Xi )]2 }
n2 n2 n 2
-M

i=1 i=1 i=1

θ(1 − θ)
=
n
Next, we calculate the Cramer-Rao’s lower bound

ln f (x, θ) = x ln θ + (1 − x) ln(1 − θ)
∂ x 1−x
ln f (x, θ) = − (2.18)
∂θ θ 1−θ
AK

 2  2
∂ x 1−x
⇒ ln f (x, θ) = − (2.19)
∂θ θ 1−θ
Thus
" 2 #
∂ E(X 2 ) E(1 − X)2 2E(X(1 − X))
E ln f (x, θ) = + −
M

∂θ θ2 (1 − θ)2 θ(1 − θ)
E(X 2 ) E(1 − 2X + X 2 ) 2E(X) − E(X 2 )
= + − .
θ2 (1 − θ)2 θ(1 − θ)

J ULIET NAKAKAWA N SUMBA - M ATHS D EPT PAGE 43 OF 110

 09
M AKERERE U NIVERSITY MTX3109 - STATISTICAL INFERENCE

Recall E(X k ) = E(X) = θfor all k > 0, for X ∼ Bernoulli(θ). Thus,

31
" 2 #
∂ θ 1 − 2θ + θ θ−θ 1 1 1
E ln f (x, θ) = 2+ − 2 = + = .
∂θ θ (1 − θ)2 θ(1 − θ) θ 1−θ θ(1 − θ)

So the Cramer-Rao lower bound is

TX
θ(1 − θ)
n
which is V ar(X̄). Therefore, X̄ is efficient.
Example 2.4.13. Consider a random sample from a normal distribution with mean µ and Variance σ 2 ,
with a pdf given by

Show that X̄ is an efficient estimator of µ.

Solution
We compute the CR lower bound ni(θ) 1
where
-M
f (x, µ, σ 2 ) = √
1
2πσ 2
1 x−µ 2
e2( σ )

" 2 #
∂
i(θ) = E ln f (x, µ, σ 2 )
hs
∂µ

Note that
1 1
ln f (x, µ, σ 2 ) = − ln(2πσ 2 ) 2 −
(x − µ)2
at

2σ 2
∂ 2 1 x−µ
⇒ ln f (x, µ, σ ) = − 2 2(x − µ)(−1) = .
∂µ 2σ σ2
Thus
-M

" 2 # 2
σ2

∂ 2 x−µ 1 2 1
E ln f (x, µ, σ ) = E 2
= 4
E(X − µ) = 4
= 2
∂µ σ σ σ σ

So the Cramer-Rao lower bound is

1 1 σ2
= =
n σ12


ni(θ) n
σ2
which is V ar(X̄) = . Therefore, X̄ is efficient.
AK

Note: When differentiating and integration are interchangeable,

" 2 #  2 
∂ ∂
E ln f (x, θ) = −E ln f (x; θ) .
∂θ ∂θ2
M

Example 2.4.14. Let {X1 , X2 , . . . , Xn } be a random sample from Bernoulli distribution with pmf

f (x, θ) = θx (1 − θ)1−x , x = 0, 1

J ULIET NAKAKAWA N SUMBA - M ATHS D EPT PAGE 44 OF 110

 09
M AKERERE U NIVERSITY MTX3109 - STATISTICAL INFERENCE

ln f (x, θ) = x ln θ + (1 − x) ln(1 − θ)

31
∂ x 1−x x 1−x
ln f (x, θ) = + (−1) = −
∂θ θ 1−θ θ 1−θ
∂2 x 1−x
ln f (x, θ) = − 2−
∂θ2 θ (1 − θ)2

TX
 2 
∂ E(X) E(1 − X)
E ln f (x, θ) = − −
∂θ2 θ2 (1 − θ)2
 2 
∂ E(X) 1 − E(X) θ 1−θ 1
−E ln f (x, θ)= + = 2+ = (2.20)
∂θ2 θ 2 (1 − θ)2 θ (1 − θ)2 θ(1 − θ)

-M
 2 2 ( 2 )
X2
  
∂ X 1−X 1−X 1−X
E ln f (x; θ) = E − =E + − 2X
∂θ θ 1−θ θ2 1−θ θ(1 − θ)
E(X 2 ) E(1 − X)2 2E(X(1 − X))
= + −
θ2 (1 − θ)2 θ(1 − θ)
2
E(X ) E(1 − 2X + X ) 2E(X) − E(X 2 )
2
= + −
θ2 (1 − θ)2 θ(1 − θ)
k
Recall E(X ) = E(X) = θ for all k > 0, for X ∼ Bernoulli(θ)
hs
θ 1 − 2θ + θ θ−θ 1 1 1
= 2+ 2
−2 = + = (2.21)
θ (1 − θ) θ(1 − θ) θ 1−θ θ(1 − θ)
From equations (2.20) and (2.21) we observe that
 2  2 
at

∂ ∂
E ln f (x, θ) = −E ln f (x; θ) .
∂θ ∂θ
Exercise 2.4.15.
-M

(i). Consider X1 , X2 , . . . , Xn random sample from X ∼ Poisson(λ), where λ > 0. Show that X̄ is an
efficient estimator for λ
(ii). Consider a random sample X1 , X2 , . . . Xn from a Bernoulli with parameter θ. Calculate i(θ) and
hence show that X̄ is an unbiased estimator of θ that attains the CR lower bound.
(iii). Consider X1 , X2 , . . . Xn be a random sample from X ∼ N (µ, σ 2 ) with µ known but σ 2 unknown.
1 n
Taking θ = σ 2 , calculate i(θ) and CR lower bound. Check that n i=1 (Xi − µ)2 an unbiased
P
estimator of θ, attains CR lower bound.
AK

2.4.5 Relative Efficiency

If Θ̂1 and Θ̂2 are both unbiased estimators of a parameter θ, then we say that Θ̂1 is relatively more
M

efficient if var(Θ̂1 ) < var(Θ̂2 ). We use the ratio

var(Θ̂1 )
var(Θ̂2 )

J ULIET NAKAKAWA N SUMBA - M ATHS D EPT PAGE 45 OF 110

 09
M AKERERE U NIVERSITY MTX3109 - STATISTICAL INFERENCE

as a measure of the relative efficiency of Θ̂2 with respect to Θ̂1 .

31
Example 2.4.16. Suppose X1 , X2 , · · · Xn is an i.i.d random sample from a Poisson distribution with
parameter λ. Let λ̂1 = X̄ and λ̂2 = X1 +X
2
2
be two unbiased estimators of λ. Find the relative efficiency
of λ̂2 w.r.t. λ̂1 .

TX
Remark 2.4.17. The search for a “best” estimator (in the MVUE sense) involves a comparison with all
other unbiased estimators, which is not feasible explicitly. Hence, other concepts and techniques are
necessary for finding out whether an MVUE exists and whether it is unique. An important concept in
this context is sufficiency.

2.5 Sufficiency
-M
In making inference about an unknown parameter θ, we collapse the random vector X1 , X2 , . . . , Xn of
the sample into one single random variable T = t(X1 , X2 , . . . , Xn ). A desirable property now is that T
should tell us as much information about θ as the sample X1 , X2 , . . . , Xn itself. This property is called
sufficiency.
hs
Definition 2.5.1. A statistic T = t(X1 , X2 , . . . , Xn ) is called a sufficient estimator for a parameter θ if
the conditional distribution of X1 , X2 , . . . , Xn given T = t does not depend on θ for all values of T .

Recall that the joint pdf of the sample is

at

Y
f (x1 , x2 , · · · , xn ) = f (xi , θ).

If fT (t, θ) is the pdf of the sufficient statistic, then

-M

Qn
i=1 f (xi , θ)
f (x1 , x2 , · · · , xn |T = t) = ,
fT (t, θ)
does not depend on θ.

Example 2.5.2. Consider X1 , X2 , . . . Xn a random sample from a Bernoulli population with pmf

f (x, θ) = θx (1 − θ)1−x , x = 0, 1.
AK

Pn
Show that T = i=1 Xi is a sufficient for θ.

Solution
Note that in this example Qn
M

i=1 f (xi , θ)
f (x1 , x2 , · · · , xn |T = t) = ,
fT (t, θ)

J ULIET NAKAKAWA N SUMBA - M ATHS D EPT PAGE 46 OF 110

 09
M AKERERE U NIVERSITY MTX3109 - STATISTICAL INFERENCE

where f (xi , θ) = θxi (1 − θ)1−xi , xi = 0, 1 and

31
fT (t, θ) = P (T = t)
Xn
= P( Xi = t).
i=1

TX
Pn
If we let Y = i=1 Xi , then it can be easily shown that MY (t) = [MX (t)]n since the Xi ’s form a random
sample. Since MX (t) = θet + (1 − θ), we have that
MY (t) = [θet + (1 − θ)]n
which is a moment generating function of binomial distribution with parameters n and θ. Thus Y ∼
Bin(n, θ) and

Hence,
fT (t, θ) =
 
n t
t -M
θ (1 − θ)n−t .

Qn
i=1
(θxi (1 − θ)1−xi )
hs
f (x1 , x2 , · · · , xn |T = t) = n t
t
θ (1 − θ)n−t
Pn
n− n
P
θ i=1 xi (1 − θ) i=1 xi
= n


t
θt (1 − θ)n−t
at

n
θt (1 − θ)n−t X
= n t

(since xi = t)
t
θ (1 − θ)n−t i=1

1
-M

= n


t

which is independent of θ.
Consider X1 , X2 , . . . , Xn a random sample from X ∼ Poisson with parameter θ > 0.
Example 2.5.3.P
Show that T = ni=1 Xi is a sufficient statistic for θ.

Solution
AK

Note that in this example Qn

i=1 f (xi , θ)
f (x1 , x2 , · · · , xn |T = t) = ,
fT (t, θ)
e−θ θxi
where f (xi , θ) = , xi = 0, 1, . . . and
xi !
M

fT (t, θ) = P (T = t)
Xn
= P( Xi = t).
i=1

J ULIET NAKAKAWA N SUMBA - M ATHS D EPT PAGE 47 OF 110

 09
M AKERERE U NIVERSITY MTX3109 - STATISTICAL INFERENCE

If we let Y = ni=1 Xi , then it can be easily shown that MY (t) = [MX (t)]n since the Xi ’s form a random
P

31
sample. Since
∞
tX
X e−θ etx θx
MX (t) = E(e ) = ,
x=0
x!
∞ t x

TX
−θ
X (θe )
=e ,
x=0
x!
−θ θet t
=e e = eθ(e −1) ,

we have that

M
t
MY (t) = enθ(e −1)
which is a moment generating function of a poisson distribution with parameters nθ. Thus Y ∼
Poisson(nθ) and

e−nθ (nθ)t
fT (t, θ) = .
s-
t!
Hence,
Qn  e−θ θxi 
i=1 xi !
h

f (x1 , x2 , · · · , xn |T = t) = e−nθ (nθ)t

t!
Pn ,
at

e−nθ θ i=1 xi
e−nθ (nθ)t
= Qn
i=1 xi ! t!
,
e−nθ θt e−nθ (nθ)t
= Qn
-M

i=1 xi ! t!
t!
= Qn
nt i=1 xi !
which is independent of θ.

2.5.4. Consider X1 , X2 , . . . , Xn a random sample from X ∼ N (µ, σ 2 ), σ 2 known. Show that

Task P
Tn = ni=1 Xi is a sufficient statistic for θ.
AK

2.5.1 Factorization Theorem (Single sufficient statistic)

The definition of sufficiency is often tedious to use in checking whether or not a given statistic is suffi-
M

cient. The Fisher-Neyman factorization theorem gives an equivalent definition which is much simpler to
use.

J ULIET NAKAKAWA N SUMBA - M ATHS D EPT PAGE 48 OF 110

 09
M AKERERE U NIVERSITY MTX3109 - STATISTICAL INFERENCE

Theorem 2.5.5. Let X1 , X2 , . . . , Xn be a random sample from the density f (x, θ), where θ could be a

31
vector. A statistic T = t(X1 , X2 , . . . , Xn ) is sufficient for a parameter θ if and only if the joint density
factors as
f (x1 , x2 , · · · xn ; θ) = g(t(x1 , x2 , . . . , xn ); θ).h(x1 , x2 , . . . , xn )
where g is a non-negative function depending on θ and x1 , x2 , . . . , xn only through t(x1 , x2 , . . . , xn ) and

TX
h is non-negative function independent of θ.

Example 2.5.6. Consider X1 , X2 , . . . , Xn a random sample from X ∼Ber(θ), 0 < θ ≤ 1. Then

(
θx (1 − θ)1−x , x = 0, 1
f (x; θ) =
0 otherwise

-M
n
( n Pn
θ i=1 xi (1 − θ)n− i=1 xi xi = 0, 1 = g( ni=1 xi ; θ)h(x1 , x2 , . . . , xn )
P P
Y
f (x; θ) =
i=1
0 otherwise
(
Pn Pn Pn 1 if x1 , x2 , . . . , xn = 0 or 1
where g( i=1 xi ; θ) = θ i=1 xi (1 − θ)n− i=1 xi and h = .
0 otherwise
Pn
Hence T = i=1 Xi is sufficient for θ.
hs
Example 2.5.7. Consider X1 , X2 , . . . , Xn a random sample from X ∼ N (µ, σ 2 ) with µ known. Let
1 1
2 2


θ = σ . Then f (xi ; θ) = √2πθ exp − 2θ (x − µ) and
n  1 Pn n
!
Y exp − 2θ (xi − µ)2 X
f (xi ; θ) = √ i=1 =g (xi − µ)2 ; θ h(x1 , x2 , . . . , xn ),
at

i=1
( 2πθ)n i=1
Pn
where h(x1 , x2 , . . . , xn ) = 1 for −∞ < x < ∞. Thus, if µ known, then i=1 (xi − µ)2 is a sufficient
statistic for σ 2 .
-M

Exercise 2.5.8. Find a sufficient statistic for the parameter in the following cases.

1. X1 , X2 , . . . , Xn a random sample from X ∼Poisson(λ)

2. X1 , X2 , . . . , Xn a random sample from X ∼ N (µ, σ 2 ), σ 2 is known.

3. X1 , X2 , . . . , Xn a random sample from X ∼Geometric(θ)

AK

(
θxθ−1 x>0
4. X1 , X2 , . . . , Xn a random sample from X with pdf f (x; θ) =
0 otherwise

2.5.2 Joint sufficiency

M

Let X1 , X2 , . . . , Xn be a random sample from the density f (x, θ). A set of statistics

T1 = t1 (X1 , X2 , . . . , Xn ), . . . Tr = tr (X1 , X2 , . . . , Xn )

J ULIET NAKAKAWA N SUMBA - M ATHS D EPT PAGE 49 OF 110

 09
M AKERERE U NIVERSITY MTX3109 - STATISTICAL INFERENCE

is jointly sufficient for θ iff the joint density of the sample can be factored as

31
f (x1 , x2 , · · · xn ; θ) = g(t1 (x1 , x2 , . . . , xn ) . . . tr (x1 , x2 , . . . , xn ))h(x1 , x2 , . . . , xn )
where g is a non-negative function depending on θ and on x1 , x2 , . . . , xn only through
t1 (x1 , x2 , . . . , xn ), . . . , tr (x1 , x2 , . . . , xn ) and h is a non-negative function independent of θ.

TX
Example 2.5.9. Consider X1 , X2 , . . . , Xn a random sample from X ∼ N (µ, σ 2 ) with µ and σ 2 un-
known. Here, θ = (θ1 , θ2 ), where θ1 = µ and θ2 = σ 2 . Thus
n
( n
)
Y 1 1 X
f (xi ; θ) = √ exp − (xi − θ1 )2
i=1
( 2πθ2 ) n 2θ 2 i=1

M
( " n n
#)
1 1 X 2 X
= √ exp − x − 2θ1 xi + nθ12 .1
( 2πθ2 )n 2θ2 i=1 i i=1
n n
!
X X
= g xi , x2i ; θ .h(x1 , x2 , . . . , xn ).
i=1 i=1
s-
Pn Pn
Hence, i=1 xi and i=1 x2i are jointly sufficient for µ and σ 2 .
Theorem 2.5.10. Theorem of Rao and Blackwell
Let θ̂ be an unbiased estimator of θ. T be a sufficient statistic for θ and define Ψ(t) = E(θ̂|T ) = t, then
h

a). Ψ(t) does not depend on θ ( hence Ψ(T ) is an estimator of θ ).

at

(b). E(Ψ(T )) = θ ( hence Ψ(T ) is an unbiased estimator θ).

(c). V ar(Ψ(T )) ≤ V ar(θ̂) ( hence Ψ(T ) has a variance which is not larger than that of θ̂)
-M

2.5.3 Consistency of an estimator

The next property to be considered is consistency. Consistency is a property of a sequence of estimators

rather than a single estimator. However, it is rather common to refer to an estimator as being consistent.
Let Θ̂n = t(X1 , X2 , · · · , Xn ) be an estimator of θ. So Θ̂n is a sequence of random variables that depend
on the size of the random sample.
AK

Definition 2.5.11. A sequence of estimators Θ̂n is said to be a consistent estimator of the parameter θ if
Θ̂n converges in probability to θ as n approaches infinity; i.e.,
lim P(|Θ̂n − θ| ≥ ) = 0 for any  > 0 (2.22)
n→∞
or lim P(|Θ̂n − θ| < ) = 1 for any  > 0. (2.23)
n→∞
M

It is unfortunately not possible to verify consistency of an estimator by applying the unbiasedness and
efficiency approach. It is however possible to verify consistence of an estimator using Chebyshev’s
inequality.

J ULIET NAKAKAWA N SUMBA - M ATHS D EPT PAGE 50 OF 110

 09
M AKERERE U NIVERSITY MTX3109 - STATISTICAL INFERENCE

Theorem 2.5.12 (Chebyshev’s inequality). Let X be a random variable with mean µ and variance σ 2 .

31
Then for any  > 0,
σ2 σ2
Pr(|X − µ| < ) > 1 − ⇔ Pr(|X − µ| ≥ ) ≤ (2.24)
2 2

TX
Using Chebyshev’s inequality, we find
E(Θ̂n − θ)2
P(|Θ̂n − θ| ≥ ) ≤ (2.25)
2
The quantity on the right hand side in (2.25) is referred to as the mean square error of Θ̂n , i.e.,

-M
Definition 2.5.13 (Mean square error (MSE)). The mean square error of Θ̂n is given by
M SE(Θ̂n ) = E(Θ̂n − θ)2 .
Theorem 2.5.14. The mean square error of Θ̂n can be expressed as
M SE(Θ̂n ) = V ar(Θ̂n ) + (Bias(Θ̂n ))2 .

Proof.
hs
n o2
2
M SE(Θ̂n ) = E(Θ̂n − θ) = E Θ̂n − E(Θ̂n ) + E(Θ̂n ) − θ
n o
= E [Θ̂n − E(Θ̂n )]2 + [E(Θ̂n ) − θ]2 + 2[Θ̂n − E(Θ̂n )][E(Θ̂n ) − θ]
at

Note that Bias(Θ̂n ) = E(Θ̂n ) − θ is a constant. Further, E(E(Θ̂n )) = E(Θ̂n ). Consequently,

n o
2E [Θ̂n − E(Θ̂n )][E(Θ̂n ) − θ] = 0,
-M

which implies
n o
2 2
M SE(Θ̂n ) = E [Θ̂n − E(Θ̂n )] + [E(Θ̂n ) − θ]
h i
= E[Θ̂n − E(Θ̂n )]2 + E [E(Θ̂n ) − θ]2
= E[Θ̂n − E(Θ̂n )]2 + [E(Θ̂n ) − θ]2
= V ar(Θ̂n ) + (Bias(Θ̂n ))2
AK

Definition 2.5.15 (Mean square error consistency of an estimator). An estimator Θ̂n is said to be MSE
consistent if
M SE(Θ̂n ) → 0 as n → ∞.
If Θ̂n is an unbiased estimator of θ, then Θ̂n is MSE consistency if
M

lim V ar(Θ̂n ) = 0
n→∞

since for an unbiased estimator Θ̂n , E(Θ̂n ) − θ = 0

J ULIET NAKAKAWA N SUMBA - M ATHS D EPT PAGE 51 OF 110

 09
M AKERERE U NIVERSITY MTX3109 - STATISTICAL INFERENCE

Example 2.5.16. Consider the sample mean X̄ as an estimator of the population mean µ.

31
Then X̄ is an unbiased estimator of µ i.e.,

E(X̄) = µ ⇒ Bias(X̄) = 0

and

TX
σ2
V ar(X̄) = →0
n
as n → ∞.
Therefore X̄ is MSE consistent estimator of µ.

Example 2.5.17. Let {X1 , X2 , . . . , Xn } be a random sample from N (µ, σ 2 ). Show that

is MSE consistent for σ 2 .

M2 =
-M 1X
n

n i=1
(Xi − X̄)2

Solution:
hs
n
1X n−1 2
M2 = (Xi − X̄)2 = S . S 2 is the sample variance
n i=1 n
n−1 2 n−1 n−1 2
=⇒ E(M2 ) = E( S )= E(S 2 ) = σ
at

n n n
σ2
Thus Bias(M2 ) = E(M̂2 ) − σ 2 = −
 n 2
(n − 1)2 2σ 4
  
n−1 2 n−1 n−1
-M

2 4
V ar(M2 ) = V ar S = V ar(S ) = = 2σ
n n n2 n − 1 n2
   2
n−1 −σ
Thus M SE(M2 ) = 2σ 4 2
+ → 0 as n → ∞.
n n
M SE
Hence M2 −−−→ σ 2
M SE
Task 2.5.18. Show that X̄ −−−→ θ, where
AK

1
1. f (x; θ) = 1θ e− θ x , x > 0
M SE
2. Show that X̄ −−−→ αθ , where X ∼ Gamma(α, θ)
M

J ULIET NAKAKAWA N SUMBA - M ATHS D EPT PAGE 52 OF 110