7.

4 CONSTRUCTING PARAMETER ESTIMATES 321

As an example of the implementation of this estimation method, suppose that x 1 , . . . , x n

are a set of Bernoulli observations, with each taking the value 1 with probability p and the
value 0 with probability 1 − p. The expectation of the Bernoulli distribution is E(X ) = p, so
that the method of moments point estimate of p is found from the equation
x̄ = p
This simply provides the usual point estimate p̂ = x/n, where x is the number of data
observations that take the value 1.

Method of Moments Point Estimates for Two Parameters

If a data set consists of observations x1 , . . . , x n from a probability distribution that
depends upon two unknown parameters, the method of moments point estimates of the
parameters are found by solving the equations

x̄ = E(X ) and s 2 = Var(X )

This is an intuitively reasonable method of estimation since it simply sets the

population mean and variance equal to the sample mean and variance. Some
practitioners may use n instead of n − 1 in the denominator of s 2 here, but it generally
makes little difference to the point estimates.

Normally distributed data provide a simple example of estimating two parameters by the
method of moments. Since E(X ) = µ and Var(X ) = σ 2 for a N (µ, σ 2 ) distribution, the
method of moments immediately gives the usual point estimates
x̄ = µ̂ and s 2 = σ̂ 2
In general, the method of moments is a simple, easy-to-use method for obtaining sensible
point estimates. However, it is not foolproof. Suppose that the data observations
2.0 2.4 3.1 3.9 4.5 4.8 5.7 9.9
are obtained from a U (0, θ) distribution. In this case the upper endpoint of the uniform dis-
tribution is the unknown parameter to be estimated. Since the expectation of a U (0, θ) distri-
bution is
θ
E(X ) =
2
and the sample mean is x̄ = 4.5375, the method of moments point estimate of θ is obtained
from the equation
θ
4.5375 =
2
This gives
θ̂ = 2 × 4.5375 = 9.075
The problem with this point estimate is that it is clearly impossible! One of the data
observations 9.9 exceeds the value θ̂ , whereas the true value of θ must necessarily be larger
than all the data values. Nevertheless, even though this example shows that point estimation
using the method of moments may be unsuitable in certain cases, in general it is a simple and
sensible method.
322 CHAPTER 7 STATISTICAL ESTIMATION AND SAMPLING DISTRIBUTIONS

The method of moments can be generalized to problems with three or more unknown
parameters by equating additional population moments
E(X − µ)k
where k ≥ 3, with the corresponding sample moments. However, examples of this kind are
rare.

7.4.2 Maximum Likelihood Estimates

Maximum likelihood estimation is a more technical method of obtaining point estimates, yet it
is a very powerful method with a great deal of theoretical justification behind its use. Consider
a set of data values x 1 , . . . , xn that are taken to be observations with a probability density
function f (x, θ ) depending on one unknown parameter θ . The joint density function of the
data observations is therefore
f (x 1 , . . . , xn , θ) = f (x 1 , θ) × · · · × f (x n , θ )
which can be thought of as the “likelihood” of observing the data values x1 , . . . , x n for a given
value of θ .

Maximum Likelihood Estimate for One Parameter

If a data set consists of observations x 1 , . . . , xn from a probability distribution f (x, θ)
depending upon one unknown parameter θ , the maximum likelihood estimate θ̂ of
the parameter is found by maximizing the likelihood function

L(x 1 , . . . , x n , θ ) = f (x 1 , θ ) × · · · × f (xn , θ)

This method of estimation has an intuitive appeal to it, since it asks the question

For what parameter value is the observed data “most likely” to have arisen?

In practice, the maximization of the likelihood function is usually performed by taking the
derivative of the likelihood function with respect to the parameter value. Often, however, it is
convenient to take the natural log of the likelihood function before differentiating. Since the
natural log is a monotonic function, maximizing the log-likelihood is equivalent to maximizing
the likelihood.
To illustrate this estimation method, suppose again that x 1 , . . . , x n are a set of Bernoulli
observations, with each taking the value 1 with probability p and the value 0 with probability
1 − p. In this case, the probability distribution (actually a probability mass function) is
f (1, p) = p and f (0, p) = 1 − p

A succinct way of writing this is

f (x i , p) = p xi (1 − p)1−xi

The likelihood function is therefore

n
L(x 1 , . . . , x n , p) = p xi (1 − p)1−xi = p x (1 − p)n−x
i=1
 7.4 CONSTRUCTING PARAMETER ESTIMATES 323

where x = x 1 + · · · + x n , and the maximum likelihood estimate p̂ is the value that maximizes
this. The log-likelihood is
ln(L) = x ln( p) + (n − x) ln(1 − p)
and
d ln(L) x n−x
= −
dp p 1− p
Setting this expression equal to 0 and solving for p produce
x
p̂ =
n
which can be checked to be a true maximum of the likelihood function. Consequently, the
method of maximum likelihood estimation is seen to produce the usual estimate of the success
probability p, which is the proportion of the sample that are successes.

Maximum Likelihood Estimate for Two Parameters

If a data set consists of observations x1 , . . . , x n from a probability distribution
f (x, θ1 , θ2 ) depending upon two unknown parameters, the maximum likelihood
estimates θ̂1 and θ̂2 are the values of the parameters that jointly maximize the
likelihood function
L(x 1 , . . . , x n , θ1 , θ2 ) = f (x 1 , θ1 , θ2 ) × · · · × f (x n , θ1 , θ2 )

Again, the best way to perform the joint maximization is usually to take derivatives of the
log-likelihood with respect to θ1 and θ2 and to set the two resulting expressions equal to 0.
The normal distribution is an example of a distribution with two parameters, with a prob-
ability density function
1 2
/2σ 2
f (x, µ, σ 2 ) = √ e−(x−µ)
2π σ
The likelihood of a set of normal observations is therefore
n
L(x 1 , . . . , x n , µ, σ 2 ) = f (xi , µ, σ 2 )
i=1
n/2 n
1
= exp − (xi − µ)2 /2σ 2
2πσ 2
i=1

so that the log-likelihood is

n
n i=1 (xi − µ)2
ln(L) = − ln(2πσ 2 ) −
2 2σ 2
Taking derivatives with respect to the parameter values µ and σ 2 gives
n
d ln(L) i =1 (xi − µ)
=
dµ σ2
and
n
d ln(L) n i=1 (x i − µ)2
2
=− 2 +
dσ 2σ 2σ 4
324 CHAPTER 7 STATISTICAL ESTIMATION AND SAMPLING DISTRIBUTIONS

Setting d ln(L)/dµ = 0 gives

µ̂ = x̄
and setting d ln(L)/dσ 2 = 0 then gives
n n
i=1 (xi − µ̂)2 i=1 (xi − x̄)2
σ̂ 2 = =
n n
which are consequently the maximum likelihood estimates of the parameters. It is interesting
to notice that these point estimates come out to be the usual estimates that have been discussed
in this chapter, except that the variance estimate uses n rather than n − 1 in the denominator.
As with point estimates produced by the method of moments, maximum likelihood es-
timates are generally sensible point estimates, and theoretical results show that they have
very good properties when the sample size n is reasonably large. If there are three or more
unknown parameters to be estimated, then the method of maximum likelihood estimation can
be generalized in the obvious manner.
In most cases the two methods of estimation produce identical point estimates, although
in certain cases the estimates may differ slightly. In certain cases the point estimates obtained
from these methods may not be unbiased, as was seen with the maximum likelihood estimate
of the normal variance, but any bias is usually small and decreases as the sample size n
increases.

7.4.3 Examples
Example 27 Suppose that the quality inspector at the glass manufacturing company inspects 30 randomly
Glass Sheet Flaws selected sheets of glass and records the number of flaws found in each sheet. These data values
are shown in Figure 7.15. If the distribution of the number of flaws per sheet is taken to have
a Poisson distribution, how should the parameter λ of the Poisson distribution be estimated?
If the random variable X has a Poisson distribution with parameter λ, then
E(X ) = λ
Consequently, the method of moments immediately suggests that the parameter estimate
should be
λ̂ = x̄
This is also the maximum likelihood estimate, which can be shown as follows. The probability
mass function of a data observation x i is
e−λ λxi
f (x i , λ) =
xi !
so that the likelihood is
n
e−nλ λ(x1 +···+xn )
L(x 1 , . . . , x n , λ) = f (x i , λ) =
(x1 ! × · · · × xn !)
i=1

The log-likelihood is therefore

ln(L) = −nλ + (x 1 + · · · + xn ) ln(λ) − ln(x 1 ! × · · · × x n !)

FIGURE 7.15 0 1 1 1 0 0 0 2 0 1 0 1 0 0 0
0 0 1 0 2 0 0 3 1 2 0 0 1 0 0
Glass sheet flaws data set
 7.4 CONSTRUCTING PARAMETER ESTIMATES 325

so that
d ln(L) (x 1 + · · · + xn )
= −n +
dλ λ
Setting this expression equal to 0 gives λ̂ = x̄.
The sample average of the 30 data observations in Figure 7.15 is 0.567, so that the quality
inspector should use the point estimate
λ̂ = 0.567
In addition, since each data observation has a variance of λ,
λ
Var( X̄ ) =
n
so that the standard error of the estimate of a Poisson parameter can be calculated as

λ̂
s.e.(λ̂) =
n
The quality inspector’s point estimate λ̂ = 0.567 consequently has a standard error of
0.567
s.e.(λ̂) = = 0.137
30

Example 26 Fish tagging and recapture present a way to estimate the size of a fish population. Suppose that
Fish Tagging and a fisherman wants to estimate the fish stock N of a lake and that 34 fish have been tagged and
Recapture released back into the lake. If, over a period of time, the fisherman catches 50 fish (without
release) and 9 of them are tagged, an intuitive point estimate of the total number of fish in the
lake is
34 × 50
N̂ = 189
9
This point estimate is based upon the reasoning that the proportion of fish in the lake that are
tagged should be roughly equal to the proportion of the fisherman’s catch that is tagged.
This point estimate is also the method of moments point estimate. Under the assumption
that all the fish are equally likely to be caught, the distribution of the number of tagged fish X
in the fisherman’s catch of 50 fish is a hypergeometric distribution with r = 34, n = 50, and
N unknown. The expectation of X is therefore
nr 50 × 34
E(X ) = =
N N
and the method of moments point estimate of N is found by equating this to the observed
value x = 9. Notice that here there is only one data observation x, which is therefore x̄.
A similar point estimate is arrived at if the binomial approximation to the hypergeometric
distribution is employed. In this case the success probability p = r/N is estimated to be
x 9
p̂ = =
n 50
with
r
N̂ =
p̂