Detailed Notes on Sampling
Distribution, Standard Error &
Bootstrap
1. Sample Statistic
Definition: A sample statistic is a numerical measure (e.g., mean, median, variance)
calculated from a subset (sample) of the population.
Examples:
- Sample mean (x̄) = average income of 100 loan applicants.
- Sample variance (s²) = variability of 50 exam scores.
- Sample median = median house price in a neighborhood survey.
Problem Example:
If the incomes of 5 sampled people are: ₹40k, ₹50k, ₹60k, ₹70k, ₹80k.
- Sample mean = (40+50+60+70+80)/5 = 60k
- Sample median = 60k
2. Sampling Distribution
Definition: The sampling distribution of a statistic is the distribution of that statistic over
many samples from the same population.
Importance:
- Allows estimation of uncertainty in sample-based estimates.
- Key for confidence intervals and hypothesis testing.
Example:
Take 1000 samples of size 20 from income data → compute sample means → plot histogram.
Result: The histogram of means is bell-shaped and less variable than the raw data
distribution.
3. Necessity of Sampling Distribution
- A single sample gives only one estimate.
- Different samples yield different estimates → sampling variability.
- Sampling distribution:
- Helps quantify uncertainty.
- Justifies the use of probability models (normal, t-distribution).
- Essential for hypothesis testing and confidence intervals.
4. Standard Error (SE)
Definition: The standard deviation of a sample statistic’s sampling distribution.
Formula: SE = s / √n
where s = sample standard deviation, n = sample size.
Properties:
- Larger n → smaller SE.
- Square root rule: To halve SE, need 4× sample size.
Problem Example:
A sample of 100 students’ heights has s = 10 cm.
SE = 10/√100 = 1 cm → The sample mean varies about 1 cm across samples.
5. Central Limit Theorem (CLT)
Statement: For large enough n, the sampling distribution of the mean approximates a
normal distribution, regardless of the population shape.
Applications:
- Justifies using normal/t-distributions in inference.
- Basis for confidence intervals and hypothesis testing.
Example:
Even if income data is skewed, sample means of size 30 will be roughly normal.
6. The Bootstrap Method
Definition: Resampling method where we draw samples with replacement from observed
data to approximate the sampling distribution.
Algorithm:
1. Draw a resample (size = n) with replacement.
2. Compute statistic (e.g., mean).
3. Repeat many times (R = 1000).
4. Analyze distribution of results.
Advantages:
- Works without assumptions of normality.
- Applicable to complex statistics (median, correlation).
- Estimates bias, SE, confidence intervals.
Problem Example:
Median income = ₹62,000.
Bootstrap resampling → Bias ≈ –₹70, SE ≈ ₹209.
7. Resampling vs. Bootstrapping
Resampling: General term → taking repeated samples from observed data (with/without
replacement).
Bootstrapping: Specific resampling → always with replacement.
Permutation tests: Resampling without replacement, often used in hypothesis testing.
8. Multivariate Bootstrap Sampling
Concept: Resampling rows of multivariate data together to preserve relationships between
variables.
Applications:
- Stability of model parameters.
- Ensemble methods (bagging, random forest).
Example:
Bootstrapping rows of (income, loan default) dataset to estimate variability in logistic
regression coefficients.
9. Importance of Bootstrap in Sampling Distribution
- Provides non-parametric alternative when formulas are unavailable.
- Useful in estimating confidence intervals, determining sample size, and ensemble methods
in ML.
- Overcomes reliance on CLT for small or skewed samples.
10. Practice Problems
Problem 1: Standard Error
A sample of 64 exam scores has mean = 75, s = 16. Find SE of the mean.
SE = 16/√64 = 16/8 = 2
Problem 2: Central Limit Theorem
If the population of incomes is skewed with mean = ₹50k, SD = ₹15k, find the approximate
distribution of sample mean for n=100.
X̄ ~ N(50,000, 1500)
Problem 3: Bootstrap Median
Given sample incomes: {40k, 50k, 60k, 70k, 80k}, generate 5 bootstrap resamples of size 5
and compute medians.
Resample 1: {50,70,60,40,40} → Median = 50
Resample 2: {80,80,60,70,40} → Median = 70
... Distribution of medians ≈ Bootstrap sampling distribution.