Resampling Methods
Dr. Abhilasha Chaudhuri
Assistant Professor
Department of Computer Science and Engineering
Sardar Vallabhbhai National Institute of Technology,
Surat, Gujrat, India
Bootstrapping
Bootstrappping
The bootstrap method is a resampling technique used to estimate statistics on a population
by sampling a dataset with replacement.
It can be used to estimate summary statistics such as the mean or standard deviation.
It lets you estimate:
Standard errors
Confidence intervals
Bias
Distribution shape
— without needing strong mathematical assumptions.
It is used in applied machine learning to estimate the skill of machine learning models when
making predictions on data not included in the training data.
The bootstrap method
The bootstrap method is a statistical technique for estimating quantities about a
population by averaging estimates from multiple small data samples.
Importantly, samples are constructed by drawing observations from a large data
sample one at a time and returning them to the data sample after they have
been chosen. This allows a given observation to be included in a given small
sample more than once. This approach to sampling is called sampling with
replacement.
The process for building one sample can be summarized as follows:
1. Choose the size of the sample.
2. While the size of the sample is less than the chosen size
1. Randomly select an observation from the dataset
2. Add it to the sample
The bootstrap method
The bootstrap method can be used to estimate a quantity of a population.
This is done by repeatedly taking small samples, calculating the statistic,
and taking the average of the calculated statistics.
We can summarize this procedure as follows:
1. Choose a number of bootstrap samples to perform
2. Choose a sample size
3. For each bootstrap sample
i. Draw a sample with replacement with the chosen size
ii. Calculate the statistic on the sample
4. Calculate the mean of the calculated sample statistics.
The bootstrap method
Reason for using “Sampling With Replacement”?
Sampling with replacement means:
◼ Some observations may appear multiple times.
◼ Some may not appear at all.
This mimics drawing new samples from the population.
The bootstrap method
Confidence Intervals (Most Common Use)
After generating 𝐵bootstrap statistics:
መ
𝜃
1
, 𝜃መ 2 , . . . , 𝜃መ 𝐵
You can form a 95% confidence interval using:
Percentile Method
Take the 2.5th and 97.5th percentiles.
This requires:
No normality assumption
No analytic formula for variance
Bootstrapping example
Original Data: Suppose we observe the sample:
4 10 7
Sample size: 𝑛 = 3
Sample mean:
ˉ 4 + 7 + 10 21
𝑥= = =7
3 3
We want to calculate the confidence interval for the population mean — but
without the theoretical formula for the standard error. So, we use
bootstrapping.
Bootstrapping example
We sample with replacement from {4, 7, 10}, drawing 3 observations
each time.
Since 𝑛 = 3, there are 33 = 27possible bootstrap samples
Bootstrapping example
Sample Mean
(4,4,4) 4
(4,4,7) 5
(4,4,10) 6
(4,7,4) 5
(4,7,7) 6
(4,7,10) 7
(4,10,4) 6
(4,10,7) 7
(4,10,10) 8
Bootstrapping example
Sample Mean
(7,4,4) 5
(7,4,7) 6
(7,4,10) 7
(7,7,4) 6
(7,7,7) 7
(7,7,10) 8
(7,10,4) 7
(7,10,7) 8
(7,10,10) 9
Bootstrapping example
Sample Mean
(10,4,4) 6
(10,4,7) 7
(10,4,10) 8
(10,7,4) 7
(10,7,7) 8
(10,7,10) 9
(10,10,4) 8
(10,10,7) 9
(10,10,10) 10
Bootstrapping example
Bootstrap Distribution of the Mean
Collect all 27 means:
4, 5,5,5, 6,6,6,6,6,6, 7,7,7,7,7,7,7, 8,8,8,8,8,8, 9,9,9, 10
Frequencies:
Mean Count Count
4 1
7
5 3 6 6
6 6
7 7 3 3
8 6
1 1
9 3 4 5 6 7 8 9 10
10 1
Bootstrapping example
This is the exact bootstrap sampling distribution of the mean.
Notice:
Centered at 7 (same as original mean)
Symmetric
Bootstrapping example
Bootstrap Standard Error
Compute variance of this distribution.
Using the frequency table:
Bootstrapping example
Mean Deviation Square Count Contribution
4 -3 9 1 9
5 -2 4 3 12
6 -1 1 6 6
7 0 0 7 0
8 1 1 6 6
9 2 4 3 12
10 3 9 1 9
Sum of contributions:
9 + 12 + 6 + 0 + 6 + 12 + 9 = 54
So:
54
Var = =2
27
Bootstrap standard error:
𝑆𝐸 = 2 ≈ 1.414
Bootstrapping example
95% Confidence Interval (Percentile Method)
We have 27 ordered values
95% CI → remove lowest 2.5% and highest 2.5%.
0.025 × 27 ≈ 0.675
So we drop roughly 1 value from each tail.
Lowest value = 4
Highest value = 10
So 95% bootstrap CI:
5 9
Bootstrapping example
Final Results:
Original mean: 7
Bootstrap standard error: 1.41
95% bootstrap Confidence Interval: 5 9
Interpretation: Based purely on resampling the observed data,
plausible values for the population mean lie between 5 and 9.