REV. FR.

MOSES ORSHIO ADASU UNIVERSITY, MAKURDI, BENUE STATE,

NIGERIA.

STA 321: STAISTICAL INFERENCE III (3 units)

INTRODUCTION AND OVERVIEW

Statistical inference is concerned with drawing conclusions about an unknown population using
sample data. In practice, we rarely observe entire populations; instead, we collect samples and
use probability theory to quantify uncertainty about population characteristics, known as
parameters.

In this course we will focus on three tightly connected tasks as follows

a. Estimation – constructing numerical values (estimators) for unknown parameters.

b. Evaluation of estimators – assessing how good those estimators are.
c. Statistical decision-making – using confidence intervals and hypothesis tests to make
informed conclusions.

Throughout this course, we assume that data are generated from a known family of probability
distributions, called a statistical model.

2. Statistical Models, Parameters, and Estimators

2.1 Statistical Model

A statistical model is a collection of probability distributions indexed by a parameter.

Ƒ= { f ( x ; θ ) :θϵΘ }

Here:

x repsent observed data ,

θ an unknown parameter ( scalar∨vector )

Θ represent the parameter space

1
Example: Consider a normal population,

X N ( μ , δ 2 ) , θ=( μ ,δ 2 )

Parameter: A parameter in statistical inference refers to a fixed but unknown numerical

characteristic of the population e.g. (( μ , δ 2 , p ) .

Statistic: A statistic is any function of the sample data only and contains no unknown
parameters.

If X 1 , X 2 , … , X n is a random sample, then

n
1
X= ∑X
n i=1 i

is a statistic.

Estimators: An estimator is a statistic used to approximate a population parameter.

Let X 1 , X 2 , .. . , X n be a random sample from a population with μ=E ¿)

Show that ¿) is an estimator of μ

Solution:

We first show that X is a statistic, from the definition of a statistic, we note:

n
1
X = ∑ Xi
n i=n

X is computed entirely from X 1 , X 2 , .. . , X n and once the sample is observed, X can be

fully computed.

We know that

μ=E (X i )

To show that X targets μ

n
1
E ( X )=E( ∑ X i)
n i=n

2
 n
1
¿ ∑ E (X i )
n I=1

But E ( X i )=μ

therefore
n
1
E ( X )= ∑ E( X i)
n i=1

∑ E (X ¿¿ i)=nμ ¿
I=n

1
E ( X )= nμ
n

E( X )=μ

Therefore, X is an estimator of μ

Why is an estimator not equal to an estimate?

Let X 1 , X 2 , .. . , X n be a random sample from a population with mean μ and va variance

δ . We define the sample variance by
2

n
1
S= 2
∑ ¿¿
n−1 i=1

Then S2 is an estimator of the population variance δ 2

To show that S2 is an estimator of δ 2, we must show that S2 is a statistic and is

constructed to estimate δ 2.

Recall
n
1
S2 = ∑ ¿¿
n−1 i=1

We can note that

X 1 , X 2 , .. . , X n are observed sample values, also, X is computed from this same sample
and there is no unknown parameter in the formula.

3
We conclude that S2 is a statistic.

We define the population variance as

2
δ =Var ( X i ) =E ¿

We show that S2 targets δ 2.

Algebraically,
n

∑ ¿¿
i=1

This separates the population variance from the sampling error.

We take expectations

E¿

Since each observations is i .i . d

E¿

Variance of the sample mean is given as

E¿

Substituting back

E¿

But
n
1
S2 = ∑ ¿¿
n−1 i=1

Taking the expectation of S2

1
E ( S )=
2
E¿
n−1

1
¿ ( n−1 ) δ 2=δ 2
n−1

4
 E ( S2 ) =δ 2

ESTIMATION IN STATISTICAL INFERENCE

Estimation is a fundamental aspect of statistical inference concerned with estimating

unknown population parameters using sample data. In practice, parameters such as the
population mean (μ), variance (σ²), or proportion (p) are fixed but unknown, while the
sample observations are random. At this level, we shall be considering two basic but
broad categories of estimation as follows:

a. Point Estimation
b. Interval Estimation

Point Estimators

Let X 1 , X 2 , .. . , X n be a random sample from a population whose probability distribution

depends on an unknown parameter θ . Any statistic of the form
^
θ=g (X 1 , X 2 , … , X n)

Is called the point estimator of θ.Once data is observed, the numerical value obtained
is called a point estimate.

Properties of point estimators include unbiasedness, consistency, efficiency (Minimum

Variance), and sufficiency. We shall look at these properties in detail in a later section.

Methods of Point Estimation

Some of the important and frequently used methods of point estimation are:

a. Method of maximum likelihood

b. Method of moments
c. Method of least squares
d. Method of minimum chi-square
e. Method of minimum variance

Method of Maximum Likelihood

To understand the maximum likelihood function, we first have to understand a likelihood

function

Likelihood Function

5
If X 1 , X 2 , .. . , X n is a random sample of size n taken from a population with joint probability
density (mass) function f (X 1 , X 2 ,. . . , X n ,θ) of sample values, then its likelihood function is
denoted by L(θ) and is defined as follows:

L ( θ )=f (X 1 , X 2 , … , X n ,θ)

For the discrete case, we have

L ( θ )=P [ X 1=x 1 ] P [ X 2=x 2 ] … P[ X n=x n ]

For the continuous case, we have

L ( θ )=f ( x1 , θ ) . f ( x 2 , θ ) … f ( x n ,θ)

Let us take an example to understand the process of finding the likelihood of function
of a random variable.

If X 1 , X 2 , … , X n is a random sample of size n taken from exponential distribution θ whose pdf is

given by
−θx
f ( x , θ )=θ e ; x >0 , λ> 0.

Calculate the likelihood function.

The likelihood function of parameter θ can be obtained as follows

L ( θ )=f ¿

−θ x 1 −θ x 2 −θ xn
¿θe .θ e …θe
n
¿ ∏ θ e−θ(x + x +…+ x )1 2 n

i=1

n
−θ ∑ (xi )
n
L(θ)=θ e i=1

Therefore, the likelihood function gives the relative likelihoods for different values of
the parameters, given the sample data.

Now, let’s explain the concept of maximum of maximum likelihood with a

quantitative example

6
Suppose, we toss a coin 5 times and we observe 3 heads and 2 tails. Instead of
assuming that the probability of getting head is p = 0.5, we want to find / estimate the
value of p that makes the observed data most likely. Since number of heads follows
the binomial distribution, therefore, the probability (likelihood function) of getting 3
heads in 5 tosses is given by
5 x (n−x)
P ( X=3 )=C 3 p ( 1− p )

Let us assume that p=0.1, we have that

5 3 (2)
P ( X=3 )=C 3 (0.1) ( 1−0.1 ) =0.0081

Let consider different values of p, we can tabulate their different likelihood functions
as follows

S/N p L(θ)
1 0.1 0.0081
2 0.2 0.0512
3 0.3 0.1323
4 0.4 0.2304
5 0.5 0.3125
6 0.6 0.3456
7 0.7 0.3087
8 0.8 0.2048
9 0.9 0.0729

From the above, the maximum likelihood is likely to be at p=0.6 because the
likelihood function is maximum at that point.

We then define the maximum likelihood function as follows:

^ 1 , x 2 , .. . , x n) of the sample values such that θ^
If there exist an estimate, say θ(x
maximizes the likelihood function L(θ), then θ^ is known as the maximum likelihood
estimate.

Therefore, principle of maximum likelihood (ML) consists in finding an estimate for

the unknown parameter θ within the admissible range of θ, i.e. parameter space Θ,
which makes the likelihood function as large as possible, that is, maximize the
likelihood function.

Maximum Likelihood Estimation – Normal Distribution

7
Let X 1 , X 2 , … , X n be a sequence of iid normal random variables with mean μ and
variance δ 2. The generic probability density function of the sequence is given as

1
f x ( x , μ , δ )=
2 −¿¿ ¿
exp
√2 π δ 2

find the maximum likelihood estimators for μ and δ 2

Solution

We first take the likelihood function as follows

L ( μ , δ 2 )=f x ( x 1 , μ , δ 2 ) . f x ( x 2 , μ , δ 2 ) ... f x ( x n , μ , δ 2)

n
¿∏ ¿¿
i=1

n
−1

( ) exp
2∑
n ¿¿¿
1 2 δ i=0
¿ 2
2 π δ2

We take log on both sides

n
n n n 1
LogL= log 1− log 2 π − log δ − 2 ∑ 2( xi −μ)
2
2 2 2 2 δ i=1
Take partial derivative with respect to μ
n
∂ −1
LogL= 2 ∑ 2(x i−μ)(−1)
∂μ 2 δ i=1

∂ −n 2 1
LogL= log δ −
∂δ
2
2 2¿¿

To find the ML estimate, we equate the partial derivative to zero in both μ and.

∂
LogL=0
∂μ

We solve for ML estimate of μ

8
 n
2
⟹− 2 ∑ ( x i−μ ) (−1)=0
2 δ i=1
n
⟹−∑ ( x i−μ ) (−1)=0
i=1

n
⟹ ∑ ( x i−μ ) =0
i=1

n n
1
⟹ ∑ x i−n μ=0⟹ ^μ = ∑ x =X
i=1 n i=1 i

Thus, we conclude that the ML estimate for μ is the observed sample data mean X .

We now solve for the ML estimate of δ 2.

∂
2
LogL=0
∂δ

n 1 1
⟹− −(−1)
2δ 2
2¿¿

n
⟹−n δ + ∑ ¿ ¿ ¿2

i=1

n
⟹−n δ 2 + ∑ ¿ ¿
i=1

n
1
δ^ 2= ∑ ¿ ¿
n i=1