UNIT–1: Sampling Theory and Statistical Estimation

UNIT 1: Sampling Theory and Statistical Estimation

1. Population

Meaning

In statistics and data science, population refers to the entire group of individuals,
objects, or data points that are relevant to a particular study or analysis.

In simple terms, population means the complete set of all possible observations of
interest.

Examples

Population can be

• All students studying data science in a university

• All customers of an online shopping platform

• All tweets related to a specific product

• All patients suffering from a particular disease in a city

Population in Data Science

In real data science problems, the population is often very large or continuously
growing. Collecting data from the entire population is usually expensive, time
consuming, and sometimes impossible. Therefore, data scientists work with samples.

2. Sample

Meaning

A sample is a smaller group selected from the population for analysis.

In simple terms, a sample is a part of the population that is used to understand the
whole population.

Examples

Sample can be

• Five hundred students selected from all data science students

• Ten thousand customers selected from millions of users

• Fifty thousand tweets collected using an application programming interface

• Two hundred patients selected for a health study

Importance of Sample in Data Science

Using samples helps to

• Reduce data collection cost

• Save time

• Reduce computational effort

• Make analysis and model building feasible

3. Difference Between Population and Sample

Population includes all units under study, whereas a sample includes only some units
selected from the population. Population is usually large, while a sample is smaller.
Studying the entire population is costly and time consuming, whereas studying a
sample is practical and efficient.

4. Why Sampling Is Necessary in Data Science

In data science applications, datasets often contain millions of records. Analysing

the entire dataset may not be efficient during initial stages. Sampling allows data
scientists to perform exploratory data analysis, test models quickly, and validate
ideas before scaling to full data.
For example, instead of training a model using one billion records, a data scientist
may start with a smaller representative sample.

Sampling Techniques

1. Simple Random Sampling

Meaning

Simple random sampling is a sampling method in which every element of the

population has an equal chance of being selected.

Example

Consider a dataset containing ten thousand customer records of an online shopping

company. A data scientist selects one thousand customer records randomly using a
computer based random selection process. Each customer record has the same
probability of being chosen.

Data Science Example

During machine learning model development, a data scientist randomly selects

rows from a large dataset to create a training dataset or a validation dataset.

Methods of Simple Random Sampling

Lottery Method

In the lottery method, each element of the population is assigned a number. These
numbers are written on slips of paper and mixed thoroughly. The required number
of slips are then drawn randomly. The elements corresponding to the selected
numbers form the sample.

Example
If a class has fifty students and ten students need to be selected, each student is
assigned a number from one to fifty. Ten numbers are drawn randomly to select the
sample.

Random Number Method

In the random number method, random numbers are generated using random
number tables or computer software. Elements whose numbers match the
generated random numbers are included in the sample.

Example
From a customer database where each customer has a unique identification
number, random numbers generated by a computer are used to select the sample
records.

Computer Based Sampling

In this method, statistical software or programming languages such as Python or R

are used to randomly select samples from a dataset.

Example
A data scientist uses a data analysis tool to randomly select a subset of rows from a
large dataset for analysis or model training.

Key Point

Simple random sampling is most effective when the population is fairly uniform,
and each element is easily identifiable.

2. Stratified Sampling

Meaning

Stratified sampling divides the population into groups called strata based on specific
characteristics, and samples are selected from each group.

Example

A university has undergraduate and postgraduate students. If the study requires

representation from both groups, samples are taken separately from undergraduate
students and postgraduate students.

Data Science Example

In a loan approval dataset, customers are divided into income groups such as low,
medium, and high income. Samples are selected from each income group to ensure
balanced representation.

Key Point

This method ensures that all important subgroups are included in the sample.

2. Stratified Sampling

Meaning

Stratified sampling is a sampling method in which the population is divided into

smaller homogeneous groups called strata based on specific characteristics, and
samples are selected from each group.

Example

Consider a university with students enrolled in undergraduate and postgraduate

programs. To ensure representation from both groups, the population is divided
into two strata namely undergraduate students and postgraduate students. Samples
are then selected separately from each group.

Data Science Example

In a customer analytics dataset, customers are grouped based on income level such
as low income, middle income, and high income. Samples are selected from each
income group to ensure that all income categories are adequately represented in the
analysis.

Methods of Stratified Sampling

Proportionate Stratified Sampling

In proportionate stratified sampling, the sample size selected from each stratum is
proportional to the size of that stratum in the population.
Example
If a company has sixty percent male employees and forty percent female employees,
the sample is selected in the same proportion to maintain population balance.

Disproportionate Stratified Sampling

In disproportionate stratified sampling, the sample size selected from each stratum
is not proportional to the population size. This method is used when certain groups
are small but important for analysis.

Example
In a medical dataset, patients with a rare disease are oversampled to ensure
sufficient data for meaningful analysis.

Key Point

Stratified sampling improves representativeness and accuracy by ensuring that all

important subgroups of the population are included in the sample.

3. Systematic Sampling

Meaning

Systematic sampling is a sampling method in which elements are selected from the
population at regular intervals after choosing a random starting point.

Example

Consider a list of five thousand employee records arranged in an ordered manner. If

every tenth employee is selected after choosing a random starting employee, the
selected employees form a systematic sample.

Data Science Example

In a time series dataset containing daily sales data, a data scientist selects every fifth
day record to reduce data size while preserving overall trends.
Key Point

Systematic sampling is simple and efficient, but it should be avoided when the
population list contains hidden patterns that may bias the sample.

4. Cluster Sampling

Meaning

Cluster sampling is a sampling method in which the population is divided into

groups called clusters, and entire clusters are selected randomly for study.

Example

A city is divided into several residential areas. A few residential areas are randomly
selected, and all households within those areas are included in the sample.

Data Science Example

For a retail sales analysis across multiple regions, a data scientist selects a few
store locations and analyzes all transactions from those selected stores.

Methods of Cluster Sampling

Single Stage Cluster Sampling

In single stage cluster sampling, entire clusters are selected and all elements within
the selected clusters are studied.

Example
Selecting five schools from a district and collecting data from all students in those
schools.

Two Stage Cluster Sampling

In two stage cluster sampling, clusters are selected first, and then a sample of
elements is selected from each chosen cluster.
Example
Selecting villages randomly and then selecting households within those villages for
data collection.

Key Point

Cluster sampling reduces cost and effort, especially when the population is
geographically dispersed, but it may be less precise than stratified sampling.

5. Comparison of Sampling Methods

Sampling Basic Idea How Sample Is Best Used When

Method Selected

Simple Random Every element has Elements are Population is

Sampling equal chance selected randomly uniform

Stratified Population is Samples are taken Subgroups must be

Sampling divided into strata from each stratum represented

Systematic Selection at regular Every kth element Data is ordered

Sampling intervals is chosen

Cluster Population is Entire clusters are Population is large

Sampling divided into selected and scattered
clusters
Numerical Examples of Sampling Methods

1. Simple Random Sampling

Problem

A population consists of 50 students. The teacher wants to select 5 students using

simple random sampling.

Steps

1. Each student is assigned a unique number from 1 to 50.

2. Five numbers are selected randomly using a random number table or
computer program.
3. Suppose the selected numbers are 4, 12, 23, 35, and 47.

Result

The students corresponding to these numbers form the sample.

Conclusion

Each student had an equal chance of being selected.

2. Stratified Sampling

Problem

A company has 100 employees, consisting of 60 males and 40 females. A sample of

10 employees is required using stratified sampling.

Steps

1. The population is divided into two strata, males and females.

2. The sample size is allocated proportionally.
3. From males, 6 employees are selected.
4. From females, 4 employees are selected.

Result
The final sample consists of 6 male and 4 female employees.

Conclusion

Both groups are fairly represented in the sample.

3. Systematic Sampling

Problem

A dataset contains 200 records. A sample of 20 records is required using systematic

sampling.

Steps

1. The sampling interval is calculated as 200 divided by 20, which equals 10.
2. A random starting point between 1 and 10 is selected. Suppose the starting
point is 4.
3. Every 10th record is selected thereafter.

Selected Records

4, 14, 24, 34, 44, 54, 64, 74, 84, 94, 104, 114, 124, 134, 144, 154, 164, 174, 184,
and 194.

Conclusion

The sample is evenly spread across the population.

4. Cluster Sampling

Problem

A city has 20 schools, each with approximately 100 students. A study requires data
from 4 schools.

Steps
 1. Each school is treated as one cluster.
2. Four schools are selected randomly from the 20 schools.
3. All students from the selected schools are included in the study.

Result

The sample consists of students from 4 schools, totaling approximately 400 students.

Conclusion

Cluster sampling reduces effort by collecting data from selected groups.

Sampling Distribution

Introduction

In statistical inference, we rarely observe the entire population. Instead, we draw a

sample and compute statistics such as:

• Sample mean

• Sample variance

• Sample proportion

However, a natural question arises:

If we draw different samples from the same population, will we get the same statistic?

The answer is No.

Different samples produce different values of statistics.

The distribution of these values is called the Sampling Distribution.

Definition of Sampling Distribution

Let 𝑋1 , 𝑋2 , … , 𝑋𝑛 be a random sample from a population with distribution:

𝑋 ∼ 𝐹(𝑥; 𝜃)
A statistic 𝑇 = 𝑔(𝑋1 , 𝑋2 , … , 𝑋𝑛 )is a function of the sample.

The probability distribution of the statistic 𝑇is called the sampling distribution
of 𝑇.

Why Sampling Distribution is Important?

Sampling distributions form the foundation of:

• Confidence intervals

• Hypothesis testing

• Estimation theory

• Machine learning uncertainty quantification

• Bootstrap methods

In Data Science:

• It helps measure model uncertainty.

• It quantifies variability in parameter estimates.

• It explains why repeated experiments give slightly different results.

Sampling Distribution of Sample Mean

1. Introduction

In statistical inference, we use a sample to estimate a population parameter. One of

the most important statistics is the sample mean.

However, if we repeatedly draw samples of the same size from a population, the
sample mean will not always be the same. It varies from sample to sample.

The probability distribution of these possible values of the sample mean is called
the:

Sampling Distribution of the Sample Mean

Mathematical Definition

Let:

𝑋1 , 𝑋2 , … , 𝑋𝑛

be a random sample drawn from a population with:

𝐸(𝑋𝑖 ) = 𝜇
𝑉𝑎𝑟(𝑋𝑖 ) = 𝜎 2

The sample mean is defined as:

𝑛
1
𝑋ˉ = ∑ 𝑋𝑖
𝑛
𝑖=1

The distribution of 𝑋ˉis called the sampling distribution of the mean.

Sampling Distribution of the Mean – Properties

Let

𝑋1 , 𝑋2 , … , 𝑋𝑛

be a random sample drawn from a population with:

𝐸(𝑋𝑖 ) = 𝜇
𝑉𝑎𝑟(𝑋𝑖 ) = 𝜎 2

Assume the observations are independent.

Define the sample mean:

 𝑛
1
𝑋ˉ = ∑ 𝑋𝑖
𝑛
𝑖=1

Property 1: Mean of the Sampling Distribution

Statement

𝐸(𝑋ˉ) = 𝜇

Derivation

1
𝑋ˉ = (𝑋 + 𝑋2 + ⋯ + 𝑋𝑛 )
𝑛 1

Taking expectation:
𝑛
1
𝐸(𝑋ˉ) = 𝐸 ( ∑ 𝑋𝑖 )
𝑛
𝑖=1

Using linearity of expectation:

𝑛
1
𝐸(𝑋ˉ) = ∑ 𝐸( 𝑋𝑖 )
𝑛
𝑖=1

Since 𝐸(𝑋𝑖 ) = 𝜇,

1
𝐸(𝑋ˉ) = (𝑛𝜇)
𝑛
𝐸(𝑋ˉ) = 𝜇
Conclusion

The sample mean is an unbiased estimator of μ.

Property 2: Variance of the Sampling Distribution

Statement

𝜎2
𝑉𝑎𝑟(𝑋ˉ) =
𝑛

Derivation
𝑛
1
𝑋ˉ = ∑ 𝑋𝑖
𝑛
𝑖=1

Using variance property:

𝑉𝑎𝑟(𝑎𝑌) = 𝑎2 𝑉𝑎𝑟(𝑌)
1
𝑉𝑎𝑟(𝑋ˉ) = 𝑉𝑎𝑟 ( ∑𝑋𝑖 )
𝑛
1
= 𝑉𝑎𝑟( ∑𝑋𝑖 )
𝑛2

Since the 𝑋𝑖 are independent:

𝑉𝑎𝑟( ∑𝑋𝑖 ) = ∑𝑉𝑎𝑟(𝑋𝑖 )

= 𝑛𝜎 2

Substitute:
 1
𝑉𝑎𝑟(𝑋ˉ) = 2
(𝑛𝜎 2 )
𝑛
𝜎2
=
𝑛

Property 3: Standard Error

Taking square root:

𝑆𝐷(𝑋ˉ) = √𝑉𝑎𝑟(𝑋ˉ)
𝜎
=
√𝑛

This is called the Standard Error of the Mean.

Property 4: Distribution When Population is Normal

Statement

If

𝑋𝑖 ∼ 𝑁(𝜇, 𝜎 2 )

then
2
𝜎
𝑋ˉ ∼ 𝑁 (𝜇, )
𝑛

Derivation

A fundamental property of the normal distribution:

Any linear combination of independent normal random variables is normally
distributed.

Since

1
𝑋ˉ = ∑𝑋
𝑛 𝑖

is a linear combination of independent normal variables,

𝑋ˉ is normal.

Its mean and variance are already derived above:

𝐸(𝑋ˉ) = 𝜇
𝜎2
𝑉𝑎𝑟(𝑋ˉ) =
𝑛
Thus,
2
𝜎
𝑋ˉ ∼ 𝑁 (𝜇, )
𝑛
Example

Example: Sampling Distribution of the Mean

Problem

Consider a small population:

{2, 4, 6, 8}
Draw all possible samples of size 𝑛 = 2 without replacement.

1. Construct the sampling distribution of the sample mean.

2. Find the mean of the sampling distribution.

3. Verify that it equals the population mean.

Solution:
Step 1: Population Mean

2+4+6+8
𝜇=
4
20
𝜇= =5
4

Step 2: All Possible Samples of Size 2

Number of samples:

4
( )=6
2

Possible samples:

(2,4)
(2,6)
(2,8)
(4,6)
(4,8)
(6,8)

Step 3: Compute Sample Means

Sample Sample Mean

(2,4) 3

(2,6) 4

(2,8) 5

(4,6) 5

(4,8) 6
 (6,8) 7

Step 4: Sampling Distribution of 𝑿ˉ

Sample Mean Frequency Probability

3 1 1/6

4 1 1/6

5 2 2/6

6 1 1/6

7 1 1/6

This table represents the sampling distribution of the mean.

Step 5: Mean of Sampling Distribution

1 1 2 1 1
𝐸(𝑋ˉ) = 3 ( ) + 4 ( ) + 5 ( ) + 6 ( ) + 7 ( )
6 6 6 6 6
3 + 4 + 10 + 6 + 7
=
6
30
= =5
6
Final Result

𝐸(𝑋ˉ) = 5 = 𝜇
✔ Verified: The sample mean is unbiased.

Interpretation for Students

• Sample means vary: 3, 4, 5, 6, 7

• They fluctuate around the population mean (5)
• The average of all sample means equals the population mean

This demonstrates the fundamental property: 𝐸(𝑋ˉ) = 𝜇

Example: Sampling Distribution of the Mean (With Replacement)

Problem

Consider a small population:

{1, 3, 5}

Draw samples of size 𝑛 = 2with replacement.

1. Construct the sampling distribution of the sample mean.

2. Find the mean of the sampling distribution.

3. Verify that it equals the population mean.

Step 1: Population Mean

1+3+5
𝜇=
3
9
𝜇= =3
3

Step 2: All Possible Samples (With Replacement)

Since sampling is with replacement, the same element can appear more than once.

Total number of samples:

32 = 9

Possible ordered samples:

(1,1)
(1,3)
(1,5)
(3,1)
(3,3)
(3,5)
(5,1)
(5,3)
(5,5)

Step 3: Compute Sample Means

Sample Mean

(1,1) 1

(1,3) 2

(1,5) 3

(3,1) 2

(3,3) 3

(3,5) 4

(5,1) 3

(5,3) 4

(5,5) 5

Step 4: Sampling Distribution of 𝑿ˉ

Count frequencies:

Mean Frequency Probability

1 1 1/9

2 2 2/9

3 3 3/9

4 2 2/9

5 1 1/9
This table represents the sampling distribution of the mean.

Step 5: Mean of Sampling Distribution

1 2 3 2 1
𝐸(𝑋ˉ) = 1 ( ) + 2 ( ) + 3 ( ) + 4 ( ) + 5 ( )
9 9 9 9 9
1+4+9+8+5
=
9
27
=
9
=3

Final Result

𝐸(𝑋ˉ) = 3 = 𝜇
✔ Verified: The sample mean is unbiased even when sampling is with replacement.

Question

Consider the population:

{2, 4, 6, 8, 10}

Draw all possible samples of size 𝑛 = 3without replacement.

1. Construct the sampling distribution of the sample mean.

2. Find the mean of the sampling distribution.

3. Verify that it equals the population mean.

Answer

Step 1: Population Mean

2 + 4 + 6 + 8 + 10
𝜇=
5
30
𝜇= =6
5
Step 2: Number of Possible Samples

Since sampling is without replacement:

5
( ) = 10
3

Thus, there are 10 possible samples.

Step 3: List All Possible Samples and Their Means

Sample Sample Mean

(2,4,6) 4.00

(2,4,8) 4.67

(2,4,10) 5.33

(2,6,8) 5.33

(2,6,10) 6.00

(2,8,10) 6.67

(4,6,8) 6.00

(4,6,10) 6.67

(4,8,10) 7.33

(6,8,10) 8.00

Step 4: Construct the Sampling Distribution of the Mean

Now count the frequencies of each mean:

Sample Mean Frequency Probability

4.00 1 1/10
Sample Mean Frequency Probability

4.67 1 1/10

5.33 2 2/10

6.00 2 2/10

6.67 2 2/10

7.33 1 1/10

8.00 1 1/10

This table represents the sampling distribution of the mean.

Step 5: Mean of Sampling Distribution

1 1 2 2 2 1
𝐸(𝑋ˉ) = 4 ( ) + 4.67 ( ) + 5.33 ( ) + 6 ( ) + 6.67 ( ) + 7.33 ( )
10 10 10 10 10 10
1
+ 8( )
10

After calculation:

𝐸(𝑋ˉ) = 6

Final Conclusion

𝐸(𝑋ˉ) = 6 = 𝜇

Hence, the sample mean is an unbiased estimator of the population mean.

Important Observation
• Sample means vary from 4 to 8
• Most values cluster around 6
• The average of all sample means equals the population mean

This verifies the fundamental property:

𝐸(𝑋ˉ) = 𝜇