CHAPTER II

POINT ESTIMATION

Institute of Technology of Cambodia

Department of Applied Mathematics and Statistics (AMS)

October 26, 2025

Statistics ITC October 26, 2025 1 / 41

 Introduction to Point Estimation

Contents

1 Introduction to Point Estimation

2 Properties of Good Estimators

3 Methods for Finding Estimators

Statistics ITC October 26, 2025 2 / 41

 Introduction to Point Estimation

Estimators

Definition 1
POINT ESTIMATE The statistic (such as mean, standard deviation,
variance...), computed from sample information, that estimates a
population parameter.

Statistics ITC October 26, 2025 3 / 41

 Introduction to Point Estimation

POINT ESTIMATION
A POINT ESTIMATE is a single value, 𝜃, ˆ computed from a
sample X1 , X2 , . . . , Xn , that serves as our ”best guess” for an
unknown population parameter 𝜃.
For instance, if X̄ = 165cm from your sample, then 165 is the
point estimate of 𝜇.

POINT ESTIMATOR
A POINT ESTIMATOR is the specific formula or rule defining
how 𝜃ˆ is calculated. It is a random variable because its value
varies from sample to sample. Í
Xi
For example, the sample mean X̄ = n is a point estimator for
the population mean 𝜇.

Statistics ITC October 26, 2025 4 / 41

 Introduction to Point Estimation

Population Parameters and Estimators

Population Parameter (𝜃) ˆ

Corresponding Estimator (𝜃)

Mean (𝜇) Sample Mean (X̄ )

Variance (𝜎2 ) Sample Variance (S 2 )
Proportion (p) Sample Proportion (p̂)
Rate (𝜆) for Poisson ˆ for Poisson 𝜆
Sample Mean (𝜆)
Shape (𝛼) or Scale (𝛽) for Derived from sample moments
Gamma or likelihood

Statistics ITC October 26, 2025 5 / 41

 Introduction to Point Estimation

Revisiting Point Estimation

The Challenge
Since 𝜃ˆ is a random variable, different samples will yield different
estimates. How do we choose the ”best” estimator? We evaluate their
properties.

Statistics ITC October 26, 2025 6 / 41

 Properties of Good Estimators

Contents

1 Introduction to Point Estimation

2 Properties of Good Estimators

3 Methods for Finding Estimators

Statistics ITC October 26, 2025 7 / 41

 Properties of Good Estimators

Properties of Good Estimators

We primarily seek estimators that are:
1 Unbiased: – Gives the correct value on average.

For example, if you want to estimate the average height of all

students and your sample mean (average from a group of
students) usually equals the true population mean, your estimator
is unbiased.
If the true mean height = 165 cm, and when you repeat sampling
many times the average of your estimates is also 165 cm → it’s
unbiased.
2 Efficient: Precise, with minimal variability.
Suppose two different methods estimate average height. Both are
unbiased (correct on average).
But Method A’s estimates vary between 163–167 cm, while
Method B’s estimates vary between 150–180 cm.
Method A is more efficient because it’s more precise (less spread
out).

Statistics ITC October 26, 2025 8 / 41

 Properties of Good Estimators

Properties of Good Estimators

3 Consistent: Becomes more accurate (converge to the true

value), as sample size increases.
For example, With 10 students, your estimate of average height
might be 162 cm.
With 100 students, it becomes 164 cm.
With 1,000 students, it reaches 165 cm (the true value).
So the estimator is consistent because it improves as sample size
increases.

Statistics ITC October 26, 2025 9 / 41

 Properties of Good Estimators

Property 1: Unbiasedness

Definition 2
An estimator 𝜃ˆ is said to be an unbiased estimator of 𝜃 if its
expected value is equal to the true population parameter

ˆ =𝜃
E (𝜃)

Or if 𝜃ˆ is not unbiased, then the bias of 𝜃,

ˆ B(𝜃)
ˆ = E (𝜃)
ˆ − 𝜃 = 0.

Principal of Unbiased Estimation

When choosing among several different estimators of 𝜃, select an
unbiased one.

Statistics ITC October 26, 2025 10 / 41

 Properties of Good Estimators

Proposition 1
Let X ∼ Bin(n, p), where n is known and p ∈ (0, 1) is the parameter.
Then the sample proportion p̂ = X /n is an unbiased estimator of p.

Proposition 2
If X1 , X2 , . . . , Xn is a random sample from a dist. with mean 𝜇, then
the sample average 𝜇ˆ = X = ni=1 Xi /n is an unbiased estimator of 𝜇.
Í

Proposition 3
Let X1 , X2 , · · · , Xn be a random sample from a dist. with mean 𝜇 and
variance 𝜎2 . Then the sample variance
Ín  2
i=1 Xi − X
𝜎ˆ 2 = S 2 =
n−1
is unbiased for estimating 𝜎2 .
Statistics ITC October 26, 2025 11 / 41
 Properties of Good Estimators

Property 1: Unbiasedness

Example 3 (Sample Variance Estimators)

Let X1 , X2 , . . . , Xn be an i.i.d. sample from a distribution with mean 𝜇
and variance 𝜎2
1 Estimator 1: 𝜎 ˆ 12 = n1 ni=1 (Xi − X̄ )2
Í

E ( 𝜎ˆ 12 ) = n 𝜎 .
n−1 2
Thus, 𝜎ˆ 12 is biased.
Estimator 2: 𝜎ˆ 22 = S 2 = 1 Ín
2
n−1 i=1 (Xi − X̄ )2
E ( 𝜎ˆ 22 ) = E (S 2 ) = 𝜎2 . Thus, 𝜎ˆ 22 = S 2 is an unbiased estimator for
𝜎 .
2

Statistics ITC October 26, 2025 12 / 41

 Properties of Good Estimators

Proof of Unbiasedness of Sample Variance Estimator

Let X1 , X2 , . . . , Xn be i.i.d. random variables with mean 𝜇 and
variance 𝜎2 , and X̄ = n1 ni=1 Xi . We can express:
Í
n
Õ n
Õ
(Xi − X̄ ) = 2
(Xi − 𝜇)2 − n(X̄ − 𝜇)2
i=1 i=1
Taking expectations on both sides,
" n
#
𝜎2
Õ  
2 2 2
E (Xi − X̄ ) = n𝜎 − n Var(X̄ ) = n𝜎 − n = (n − 1)𝜎2
n
i=1

Estimator 1
n
1Õ
𝜎ˆ 12 = (Xi − X̄ )2
n
i=1
n−1 2
E ( 𝜎ˆ 12 ) = 𝜎 ⇒ 𝜎ˆ 12 is biased.
n
Statistics ITC October 26, 2025 13 / 41
 Properties of Good Estimators

Proof of Unbiasedness of Sample Variance Estimator

Estimator 2
Since " #
n
Õ
E (Xi − X̄ )2 = (n − 1)𝜎2
i=1

Then
" n
#
1 Õ 1
E (S ) = E 2
(Xi − X̄ )2 = (n − 1)𝜎2 = 𝜎2
n−1 n−1
i=1

⇒ S 2 is an unbiased estimator of 𝜎2 .

Remark: The term (n − 1) is called the degrees of freedom,

accounting for the estimation of X̄ instead of the true mean 𝜇.

Statistics ITC October 26, 2025 14 / 41

 Properties of Good Estimators

Property 2: Efficiency and CRLB

Definition 4
Among unbiased estimators, a more efficient estimator has a smaller
variance. If 𝜃ˆ1 and 𝜃ˆ2 are two unbiased estimators for 𝜃, and
V (𝜃ˆ1 ) < V (𝜃ˆ2 ), then 𝜃ˆ1 is more efficient than 𝜃ˆ2 .

Cramer-Rao Lower Bound (CRLB)

The CRLB provides a theoretical lower bound on the variance of any
ˆ
unbiased estimator 𝜃.
ˆ ≥ 1
V (𝜃)
n.I (𝜃)
where I (𝜃) is the Fisher Information, a measure of the amount of
information the data provides about the parameter 𝜃:

𝜕 𝜕2
   
I (𝜃) = V ln f (X |𝜃) = −E ln f (X |𝜃)
𝜕𝜃 𝜕𝜃 2
Statistics ITC October 26, 2025 15 / 41
 Properties of Good Estimators

Property 2: Efficiency and CRLB

Definition 5
An unbiased estimator 𝜃ˆ is said to be efficient if

ˆ = 1
V (𝜃) .
nℐ(𝜃)

Definition 6
An unbiased estimator that achieves this lower bound is called a
Minimum Variance Unbiased Estimator (MVUE).

Remark
An efficient estimator may not always exist; however, if it does and is
unbiased, then it is the MVUE.

Statistics ITC October 26, 2025 16 / 41

 Properties of Good Estimators

Property 2: Efficiency and CRLB

Example 7 (Estimating Mean of Normal Distribution)

For a random sample X1 , X2 , . . . , Xn from 𝒩(𝜇, 𝜎2 ) (with 𝜎2 is
known), the Fisher Information for 𝜇 is I (𝜇) = n/𝜎2 .
The CRLB for an unbiased estimator of 𝜇 is 𝜎2 /n
The sample mean X̄ has variance V (X̄ ) = 𝜎2 /n
Thus, X̄ is an MVUE for 𝜇 (when 𝜎2 is known).

Statistics ITC October 26, 2025 17 / 41

 Properties of Good Estimators

Property 3: Consistency

Definition 8
An estimator 𝜃ˆn (where n indicates dependence on sample size) is
consistent if it converges in probability to the true parameter 𝜃 as the
sample size n approaches infinity.
 
Formally: ∀𝜀 > 0, limn→+∞ P | 𝜃ˆn − 𝜃| < 𝜀 = 1.
p
This is denoted as 𝜃ˆn →
− 𝜃

Sufficient Condition for Consistency

An estimator 𝜃ˆn is consistent if:
1 It is asymptotically unbiased: limn→+∞ E (𝜃ˆn ) = 𝜃.
2 Its variance approaches zero: limn→+∞ V (𝜃ˆn ) = 0.

Statistics ITC October 26, 2025 18 / 41

 Properties of Good Estimators

Example 9 (Sample Mean Consistency)

The sample mean X̄ is a consistent estimator for 𝜇 because:
E (X̄ ) = 𝜇 (it’s unbiased for all n, so also asymptotically unbiased).
Var (X̄ ) = 𝜎2 /n. As n → +∞, V (X̄ ) → 0.
This means as you collect more data, your sample mean will almost
certainly be very close to the true population mean.

Statistics ITC October 26, 2025 19 / 41

 Properties of Good Estimators

Example 10 (Unbiasedness (Civil Engineering: Concrete

Strength))
A civil engineer tests 10 concrete cylinders to estimate the mean
compressive strength (MPa) used in bridge columns. The population
mean (design spec) is 𝜇 = 25.5 MPa. Concrete Strength (MPa) for
10 cylinders:

25.2, 26.0, 24.8, 25.5, 25.9, 26.3, 24.7, 25.1, 25.6, 25.4

1 Compute the sample mean X̄ and sample variance S 2 .

2 Is X̄ an unbiased estimator of 𝜇? Explain.
If the variance is computed with n1 (Xi − X̄ )2 instead of
Í
3
1 Í 2
n−1 (Xi − X̄ ) , explain why it is biased.
𝜎2
4 What is the bias in that case? B(𝜎ˆ 12 ) = E ( 𝜎ˆ 12 ) − 𝜎2 = −
.
n
5 Practically, why do we divide by n − 1 in engineering testing?

Statistics ITC October 26, 2025 20 / 41

 Properties of Good Estimators

Solution.
Data (MPa): 25.2, 26.0, 24.8, 25.5, 25.9, 26.3, 24.7, 25.1, 25.6, 25.4,
n = 10.
1) Point estimates
n
1Õ 255.5
X̄ = Xi = = 25.45 MPa
n 10
i=1

n
1 Õ
S2 = (Xi − X̄ )2 = 0.2694 MPa2 ( S ≈ 0.519 MPa )
n−1
i=1

2) Unbiasedness of the mean

E (X̄ ) = 𝜇 ⇒ X̄ is an unbiased estimator of 𝜇.

Statistics ITC October 26, 2025 21 / 41

 Properties of Good Estimators

Solution.
3) Why does dividing by n make the variance biased?
n
1Õ n−1 2 n−1 2
𝜎ˆ 12 = (Xi − X̄ )2 = S ⇒ E ( 𝜎ˆ 12 ) = 𝜎
n n n
i=1

so it underestimates 𝜎2 on average.
Numerically:
n−1 2
𝜎ˆ 12 = S = 0.2425 MPa2 ,
n
which is less than S 2 = 0.2694 MPa2 . This confirms that the “divide-by-n”
method gives a smaller (biased) variance.
4) Bias amount

𝜎2
B( 𝜎ˆ 12 ) = E ( 𝜎ˆ 12 ) − 𝜎 2 = − (< 0).
n
5) Why divide by n− 1? Using n− 1 removes this negative bias and makes
S 2 an unbiased estimator of the true variance 𝜎 2 , which is important for fair
quality control and safety margins in engineering tests. □
Statistics ITC October 26, 2025 22 / 41
 Methods for Finding Estimators

Contents

1 Introduction to Point Estimation

2 Properties of Good Estimators

3 Methods for Finding Estimators

Statistics ITC October 26, 2025 23 / 41

 Methods for Finding Estimators

Methods for Finding an Estimator of 𝜃

There are several methods for finding an estimator of 𝜃. Some of

these methods are:
1 Moment Method
2 Maximum Likelihood Method
3 Bayes Method
4 Least Squares Method
5 Minimum Chi-Squares Method
6 Minimum Distance Method

In this chapter, we only discuss the first two methods of estimating a

population parameter.

Statistics ITC October 26, 2025 24 / 41

 Methods for Finding Estimators

Method 1: The Method of Moments (MOM)

Definition 11 (Moments)
Let X be a random variable and c ∈ R a scalar. Then the k th moment
of X is:
E [X k ]
and the k th moment of X (about c) is:

E [(X − c)k ]

Usually, we are interested in the first moment of X :

𝜇 = E [X ],

and the second moment of X about 𝜇:

Var(X ) = E [(X − 𝜇)2 ].

Statistics ITC October 26, 2025 25 / 41
 Methods for Finding Estimators

Method 1: The Method of Moments (MOM)

Definition 12
Let X1 , X2 , . . . , Xn be a random sample from a probability mass
function (pmf) or probability density function (pdf) f (x). We then
define the method of moments (MoM) estimator as
n
1Õ k
E [X k ] = Xi .
n
i=1

The Methods of Moments (MOM) is a straightforward approach

where population moments are equated to their corresponding sample
moments.
The k th population moment about the origin is 𝜇′k = E (X k ).
Ín
The k th sample moment about the origin is Mk′ = 1
n
k
i=1 Xi .

Statistics ITC October 26, 2025 26 / 41

 Methods for Finding Estimators

Method 1: The Method of Moments (MOM)

Procedure
1 Express the population parameters (𝜃 , 𝜃 , . . . , 𝜃 ) as functions
1 2 m
of the first m population moments.
2 Equate the first m sample moments to the corresponding
population moments.
3 Solve the resulting system of m equations for the m unknown
parameters.

Statistics ITC October 26, 2025 27 / 41

 Methods for Finding Estimators

Example 13
Let X1 , X2 , . . . , Xn represent a random sample of service times of n
customers at a certain facility, where the underlying distribution is
assumed exponential with parameter 𝜆. Find the point estimator of 𝜆.

Solution.
i.i.d.
Given X1 , X2 , . . . , Xn ∼ Exponential(𝜆), 𝜆 > 0. we have

E (X ) = 𝜆, V (X ) = 𝜆2 ,

and its PDF is

1
 e −x/𝜆 , x > 0,


𝜆

f (x; 𝜆) =
 0,

otherwise.

□

Statistics ITC October 26, 2025 28 / 41

 Methods for Finding Estimators

Solution: MOM for Exponential Parameter 𝜆

Solution.
By the method of moments, we have:
n
1Õ
E (X ) = Xi .
n
i=1

Since E (X ) = 𝜆, then
=⇒ 𝜆 = X̄
i.e,

𝜆ˆ = X̄
Hence, the MOM estimator of 𝜆 is the sample mean X̄ . □

Statistics ITC October 26, 2025 29 / 41

 Methods for Finding Estimators

Method 1: The Method of Moments (MOM)

Example 14 (MOM Estimators for Gamma Distribution)

A Gamma distribution has two parameters: 𝛼 (shape) and 𝛽
(scale).
Its PDF: f (x |𝛼, 𝛽) = 1
Γ(𝛼)𝛽 𝛼
x 𝛼−1 e −x/𝛽 for x > 0.

Population Moments
E (X ) = 𝛼𝛽
E (X 2 ) = 𝛼𝛽 2 (1 + 𝛼)

Sample Moments
M1′ = X̄
Ín
M2′ = 1
n i=1 Xi
2

Statistics ITC October 26, 2025 30 / 41

 Methods for Finding Estimators

Example: MOM Estimators for Gamma Distribution

Given: A Gamma distribution with parameters 𝛼 (shape) and 𝛽
(scale):
1
f (x | 𝛼, 𝛽) = x 𝛼−1 e −x/𝛽 , x > 0.
Γ(𝛼)𝛽 𝛼
Population moments:
E (X ) = 𝛼𝛽, E (X 2 ) = 𝛼𝛽 2 (1 + 𝛼).
Sample moments:
n
1Õ 2
M1′ = X̄ , M2′ = Xi .
n
i=1

Step 1: Equate moments

n
1Õ 2
X̄ = 𝛼𝛽 (1) and Xi = 𝛼𝛽 2 (1 + 𝛼) (2).
n
i=1

Statistics ITC October 26, 2025 31 / 41

 Methods for Finding Estimators

Example: MOM Estimators for Gamma Distribution

X̄
Step 2: Solve for parameters 𝛼 and 𝛽: From equation (1): 𝛽 = .
𝛼
Substitute into equation (2):

2
X̄ 2

1Õ 2 X̄
Xi = 𝛼 (1 + 𝛼) = (1 + 𝛼)
n 𝛼 𝛼
1Õ 2
⇐⇒ 𝛼( Xi ) = X̄ 2 + 𝛼X̄
n
(X̄ )2
=⇒ 𝛼 = 1 Í 2
n Xi − (X̄ )2

Thus, the MOM estimators are:

1
Xi2 −(X̄ )2
Í
(X̄ )2
𝛼ˆ = 1 Í
Xi2 −(X̄ )2
𝛽ˆ = n
X̄
n

Statistics ITC October 26, 2025 32 / 41

 Methods for Finding Estimators

Method 2: Maximum Likelihood Estimation (MLE)

Likelihood Function
For a random sample X1 , X2 , . . . , Xn from a distribution with PDF (or
PMF) f (x |𝜃), the likelihood function is :
n
Ö
L(𝜃|x1 , . . . , xn ) = f (xi |𝜃)
i=1

This function expresses the joint probability of observing the given

sample data as a function of the unknown parameter(s) 𝜃.

Statistics ITC October 26, 2025 33 / 41

 Methods for Finding Estimators

Method 2: Maximum Likelihood Estimation (MLE)

Log-Likelihood Function
It is almost always easier to maximize the natural logarithm of the
likelihood function, called the log-likelihood:
n
Õ
ln L(𝜃) = ln f (xi |𝜃)
i=1

The value of 𝜃 that maximizes L(𝜃) also maximizes ln L(𝜃).

Statistics ITC October 26, 2025 34 / 41

 Methods for Finding Estimators

Method 2: Maximum Likelihood Estimation (MLE)

Procedure for Finding Estimator using MLE

1 Write down the likelihood function L(𝜃).
2 Take the natural logarithm to get the log-likelihood ln L(𝜃).
3 Take the first derivative of ln L(𝜃) with respect to each
parameter, set it to zero, and solve the resulting equations.
(These are called the likelihood equations).
4 Verify that these values correspond to a maximum (e.g., using the
second derivative test, or by checking the function’s behavior).

Statistics ITC October 26, 2025 35 / 41

 Methods for Finding Estimators

Method 2: Maximum Likelihood Estimation (MLE)

Remark
Sometimes calculus cannot be used to obtain mle’s.

Example 15
Example: MLE for Exponential Rate Parameter 𝜆 Assume
X1 , X2 , . . . , Xn are i.i.d. from an Exponential distribution with PDF
x
f (x |𝜆) = 𝜆1 e − 𝜆 , for x > 0. Find the MLE of 𝜆.

Statistics ITC October 26, 2025 36 / 41

 Methods for Finding Estimators

MLE for Exponential Scale (Mean) 𝜆

Solution.
i.i.d.
Assume X1 , . . . , Xn ∼ Exp(𝜆) with
1 −x/𝜆
f (x | 𝜆) = e , x > 0, 𝜆 > 0.
𝜆
1 Likelihood
n n
!
Ö 1 1Õ
L(𝜆) = e −Xi /𝜆 = 𝜆−n exp − Xi .
𝜆 𝜆
i=1 i=1

2 Log-likelihood
n
1Õ
ℓ (𝜆) = log L(𝜆) = −n log 𝜆 − Xi .
𝜆
i=1

Statistics ITC October 26, 2025 □37 / 41

 Methods for Finding Estimators

MLE for Exponential Scale (Mean) 𝜆

1 Score and MLE
n n
n 1 Õ ˆ = 0 =⇒ 𝜆ˆ = 1
Õ
ℓ ′(𝜆) = − + 2 Xi ⇒ ℓ ′(𝜆) Xi = X̄ .
𝜆 𝜆 n
i=1 i=1

2 Maximum check
n
′′ n 2 Õ ˆ = − n < 0 ⇒ maximum.
ℓ (𝜆) = 2 − 3 Xi , ℓ ′′(𝜆)
𝜆 𝜆 𝜆ˆ 2
i=1

Result
n
1Õ
The MLE of 𝜆 is 𝜆ˆ = X̄ = Xi .
n
i=1

Note: With parameterization f (x | 𝜃) = 𝜃e −𝜃x (rate 𝜃), the MLE is 𝜃ˆ = 1/X̄ .

Statistics ITC October 26, 2025 38 / 41
 Methods for Finding Estimators

Example 16 (Chemical & Food Technology: Sugar

Concentration)
A food engineer tests the sugar concentration (°Brix) of syrup
samples from a production line. The target mean concentration is
𝜇 = 15.0 °Brix.
Sample Data (°Brix): 14.8, 15.2, 14.7, 15.3, 15.1, 14.9, 15.4,
14.8
1 Compute the sample mean X̄ and sample variance S 2 .
2 Check if X̄ is an unbiased estimator of 𝜇.
1
(Xi − X̄ )2 and find the bias.
Í
3 Compute biased variance using n
4 Explain the importance of efficiency in quality control.
5 Using the Method of Moments (MOM), estimate 𝜆 assuming
an Exponential model.
6 If another method gives smaller variance, which estimator is
preferred and why?
Statistics ITC October 26, 2025 39 / 41
 Methods for Finding Estimators

Solution.
Given data (n = 8): 14.8, 15.2, 14.7, 15.3, 15.1, 14.9, 15.4, 14.8
1) Descriptive Statistics
1Õ
X̄ = Xi = 15.025, S 2 = 0.059, S = 0.243
8
2) Unbiasedness: E [X̄ ] = 𝜇 =⇒ X̄ is unbiased.
3) Biased variance:
n−1 2
𝜎ˆ 12 = S = 0.0518
n
𝜎2
B(𝜎ˆ 12 ) = − <0
n
4) Importance of Efficiency in Quality Control: In quality control, efficiency
ensures:
In quality control, an efficient estimator provides more precise and reliable
results.
It reduces variability in measurements, leading to tighter process control limits
and quicker detection of production issues.
With higher precision (small variability), fewer samples are needed, saving time
and resources while ensuring consistent decisions about batch acceptance.
Statistics ITC October 26, 2025 40 / 41
 Methods for Finding Estimators

Solution.
5) MOM Estimate (Exponential Model)

E [X ] = 𝜆 ⇒ 𝜆ˆ MOM = X̄ = 15.025

This means that the estimated average sugar concentration in the syrup samples is
approximately 15.025 °Brix, which is very close to the design target of 15.0 °Brix.
6) Preferred Estimator (Smaller Variance)
Between two unbiased estimators, choose the one with smaller variance → it
is more efficient.
If one estimator is slightly biased but has a much smaller variance, compare
them using:
MSE = Bias2 + Variance
The estimator with the smaller MSE is preferred, since it’s closer to the true
value on average.
In practice:
For regulatory QA (quality control): use the unbiased estimator.
For predictive control or production: use the estimator with
smaller MSE.
. Statistics ITC October 26, 2025 41 / 41