Prof.
VanDerwerken SM239 Notes
Probability vs Statistics
So far, we have dealt with families of distributions (e.g., Hgeom, Expo, Normal) with one or more
parameters, assuming these parameters to be known. In real-world applications, we know
(or assume) the family of the distribution and want to estimate the parameter values using
observed data.
Method of Moments Estimation
The Method of Moments estimation technique consists of the following steps:
1. Find µk = E[X k ] for as many k’s as there are parameters in the distribution.
2. Given observations x1 , . . . , xn , estimate µk by µ̂k = n1 (xk1 + . . . + xkn ).
3. Estimate the desired parameters using the appropriate moment estimates; that is, solve the
system given by µk = µ̂k .
Example Assume these data are from a Pois(θ) dist’n: 8, 11, 7, 4. Estimate θ using MoM.
Example Assume these data are from a Unif(a,b) dist’n: 1.0, 7.7, 2.1, 3.9, 0.2. Estimate a and b
using MoM.
Is there anything that doesn’t seem right about these estimates for a and b and the data that
we actually observed??
Method of Moments & Maximum Likelihood & Conf. Intervals Page 1/4
Prof. VanDerwerken SM239 Notes
Maximum Likelihood Estimation
Definition Let θ be a vector of parameters and let ~x = (x1 , . . . , xn ) be a vector of independent
observations drawn from a common distribution. The likelihood function is the function of θ given
by:
Interpretation: When the distribution is discretea , the likelihood is the probability of observing
the data set, considered as a function of the distribution’s parameters.
Maximum likelihood estimation finds the θ that maximizes L~x (θ).
For convenience, we’ll often work with the natural log of the likelihoodb , denoted `~x :
n
X
`~x (θ) = ln L~x (θ) = ln f (xi ; θ).
i=1
Maximum likelihood estimation consists of finds the maximizer of L~x :
1. Given a vector of data values x = (x1 , . . . , xn ), find L~x .
2. Find θ̂, the θ that maximizes L~x (θ), usually by taking the nautral log, finding its derivative
(w.r.t. θ), setting the equation equal to zero, and solving for θ̂.
Example Suppose we observe the following observations from an Expo(λ) distribution: x1 , . . . , xn .
Find the likelihood, log-likelihood, and the maximum likelihood estimate (MLE) λ̂.
How would you find the MLE if our parameter vector had length greater than 1, like θ = (µ, σ)
for Normal data?
a
For a continuous distribution, the likelihood is proportional to the probability of observing a data set very similar
to the observed one (see Intuition 5.1.8 on pg. 217 of online text).
b
We take the natural log because it turns the product into a sum and is easier to work with. Since ln is a monotone
function, what maximizes ln L~x also maximizes L~x .
Method of Moments & Maximum Likelihood & Conf. Intervals Page 2/4
Prof. VanDerwerken SM239 Notes
Confidence Interval: An interval we believe contains µ
The estimation techniques described so far generate point estimates for unknown parameters.
Often, we also want a range of plausible values for the parameter. We will discuss confidence
interval (CI) estimation for a population mean.
The CI formula we will use is:
Example 1. Suppose you’d like to estimate the average weight of Midshipmen currently in the
Brigade. You don’t have the time to weigh every Mid, but you manage to weigh 100 Mids that you
are satisfied constitute a random sample from the Brigade. The sample average weight is 168 lbs.
Suppose we know that the standard deviation of all Midshipmen weights is 15 lbs.
(a) Find a 95% CI for the average weight of Midshipmen currently in the Brigade.
(b) Find a 99.7% CI for the average weight of Midshipmen currently in the Brigade.
(c) How large would our sample need to be so that we are 99.7% confident our estimate of µ is
off by less than 2.5 lbs. In other words, what sample size do we need so that the margin of
error is less than 2.5 lbs.?
(d) How could you use the standard normal percentiles to obtain a confidence interval with con-
fidence level other than 68, 98, or 99.7? For example, can you find a 98% CI for the average
weight of Midshipmen currently in the Brigade?
Method of Moments & Maximum Likelihood & Conf. Intervals Page 3/4
Prof. VanDerwerken SM239 Notes
What if we don’t know the population variance σ 2 ?
Estimate it! When n is large (20 or 30 is often sufficient), we can estimate σ 2 with the sample
variance (denoted s2 ):
1
s2 = {(x1 − x̄)2 + (x2 − x̄)2 + · · · + (xn − x̄)2 }.
n−1
We can calculate s2 in Excel using =VAR(. . . )).
To recap: the formula for a 100(1 − α)% CI for µ, when σ 2 is known, is:
And the formula 100(1 − α)% CI for µ, when σ 2 is unknown, is:
(In SM339 we will examine other approaches for when σ 2 is unknown.)
Where does the CI formula come from? How do we interpret a calculated interval?
After presenting a 95% confidence interval estimate, it’s appropriate to say that we
are 95% confident that µ is in this interval. What does this mean?
• The procedure we used to calculate this confidence interval works 95% of the time; or
• There is a 95% probability that we obtain a sample for which µ falls in the associated
random interval ; or
• If we obtain many samples and calculate interval estimates based on each, approximately
95% of the fixed intervals will contain µ.
For an illustration of this last interpretation, see [Link]
Do NOT say there is a 95% probability/chance/likelihod that µ falls in a particular calcu-
lated interval. Why? Neither µ nor the interval is random!
Do NOT define confidence by using the word “confident” or a colloquial synonym like
“sure”. Why? This is circular!
Method of Moments & Maximum Likelihood & Conf. Intervals Page 4/4