Permutation & Random Sampling in Data Analysis
Detailed Notes with Worked Examples
PART 1: RANDOM SAMPLING
1.1 Definition
Random sampling is the process of selecting a subset of observations from a population in such a way that each observation has a known
probability of being chosen. It is the foundation of statistical inference — we study a sample to draw conclusions about the entire
population, since studying the whole population is often impossible, expensive, or time-consuming.
1.2 Why Random Sampling Matters
Avoids selection bias — a non-random sample may over/under-represent certain groups.
Enables generalization — results from a good sample can be extended to the population with a quantifiable margin of error.
Efficient — cheaper and faster than a full census.
Core to ML workflows — train-test splits, k-fold cross-validation, and bootstrapping all rely on random sampling.
1.3 Types of Random Sampling
1. Simple Random Sampling (SRS) Every member of the population has an equal chance of selection. Selection can be done with or
without replacement.
2. Stratified Sampling The population is divided into homogeneous subgroups (strata) based on a characteristic (e.g., gender, region),
and random samples are drawn from each stratum — often proportional to stratum size. This ensures every subgroup is represented.
3. Systematic Sampling Select every k-th element from an ordered list, starting from a random point. E.g., pick every 10th customer
from a list of 1,000.
4. Cluster Sampling Population divided into clusters (often geographically); a random subset of entire clusters is selected, and all
members within chosen clusters are studied.
5. Sampling With vs. Without Replacement
With replacement: an item can be picked more than once (used in bootstrapping).
Without replacement: once picked, an item is removed from the pool (standard SRS).
1.4 Worked Example — Simple Random Sampling
Problem: You have sales data for 10 stores. You want to randomly sample 4 stores to audit.
Store ID Sales ($)
S1 200
S2 450
S3 310
S4 275
S5 600
S6 190
S7 340
S8 500
S9 220
S10 410
Step 1: Assign each store equal probability = 1/10.
Step 2: Randomly draw 4 stores without replacement (e.g., using a random number generator).
import pandas as pd
data = {
"Store": ["S1","S2","S3","S4","S5","S6","S7","S8","S9","S10"],
"Sales": [200,450,310,275,600,190,340,500,220,410]
}
df = [Link](data)
sample = [Link](n=4, random_state=42, replace=False)
print(sample)
Possible Output:
Store Sales
8 S9 220
1 S2 450
5 S6 190
0 S1 200
Interpretation: These 4 stores were chosen purely by chance, with no bias toward high or low sales — a fair representation for auditing
purposes.
Estimating the population mean from the sample:
Sample mean = (220 + 450 + 190 + 200) / 4 = 265
True population mean (all 10 stores) = (200+450+310+275+600+190+340+500+220+410)/10 = 349.5
This shows sampling variability — a small sample may not perfectly reflect the population, which is why sample size and sampling method
matter, and why we use confidence intervals in inferential statistics.
PART 2: PERMUTATION
2.1 Definition
A permutation is an arrangement of elements in a specific order. In data analysis, “permutation” most often refers to randomly
shuffling/reordering data — either to test statistical significance (permutation testing) or to measure feature importance in machine
learning.
2.2 Permutation Testing (Randomization Test)
Concept: If two groups truly come from the same distribution (no real effect), then shuffling which observation belongs to which group
shouldn’t change the outcome much. If the observed difference between groups is much bigger than what we get from random shuffles,
that’s evidence of a real effect.
Null Hypothesis (H0): There is no difference between the two groups (labels are exchangeable).
Steps:
1. Compute the observed test statistic (e.g., difference in means) between Group A and Group B.
2. Pool all data together.
3. Randomly reshuffle the pooled data and split it into two groups of the original sizes.
4. Recompute the test statistic for this shuffled version.
5. Repeat steps 3-4 many times (e.g., 10,000 permutations) to build a null distribution.
6. p-value = proportion of permuted statistics that are as extreme as, or more extreme than, the observed statistic.
7. If p-value < significance level (e.g., 0.05), reject H0 — the difference is statistically significant.
2.3 Worked Example — Permutation Test
Problem: A company tests two website designs (A and B) and records time-on-page (in seconds) for 5 users each.
Group A Group B
12 20
15 22
14 19
10 25
13 21
Step 1 — Observed statistic:
Mean(A) = (12+15+14+10+13)/5 = 12.8
Mean(B) = (20+22+19+25+21)/5 = 21.4
Observed difference = 21.4 - 12.8 = 8.6
Step 2 — Pool all 10 values:
[12, 15, 14, 10, 13, 20, 22, 19, 25, 21]
Step 3 — Shuffle and split (example of ONE permutation):
Randomly reassign 5 values to “Group A’” and 5 to “Group B’”:
A’ = [20, 15, 10, 19, 13] → mean = 15.4
B’ = [12, 14, 22, 25, 21] → mean = 18.8
Permuted difference = 18.8 - 15.4 = 3.4
This is smaller than the observed 8.6. We repeat this shuffling process thousands of times and count how often the permuted difference is
≥ 8.6.
Step 4 — Full simulation in Python:
import numpy as np
group_a = [Link]([12, 15, 14, 10, 13])
group_b = [Link]([20, 22, 19, 25, 21])
observed_diff = group_b.mean() - group_a.mean()
pooled = [Link]([group_a, group_b])
n_a = len(group_a)
n_permutations = 10000
count = 0
diffs = []
for _ in range(n_permutations):
[Link](pooled)
perm_a = pooled[:n_a]
perm_b = pooled[n_a:]
diff = perm_b.mean() - perm_a.mean()
[Link](diff)
if diff >= observed_diff:
count += 1
p_value = count / n_permutations
print("Observed difference:", observed_diff)
print("p-value:", p_value)
Typical Output:
Observed difference: 8.6
p-value: 0.008
Interpretation: Since p-value (0.008) < 0.05, we reject H0. There is strong evidence that Design B genuinely leads to longer time-on-
page than Design A — the observed difference is very unlikely to have occurred by random chance alone.
2.4 Permutation Feature Importance
Concept: After training an ML model, to see how important a feature is, we randomly shuffle (permute) just that one feature’s column in
the test set (breaking its relationship with the target) while keeping other features intact, then measure how much the model’s
accuracy/score drops.
Large performance drop → feature was important.
No/small drop → feature was not useful to the model.
Worked Example:
Suppose a Random Forest model predicts house prices using size , bedrooms , and distance_to_city , with test R-squared = 0.85.
Feature Shuffled New R² Drop in R²
size 0.55 0.30
bedrooms 0.83 0.02
distance_to_city 0.70 0.15
Interpretation: size is the most important feature (biggest R² drop when permuted), bedrooms is nearly irrelevant, and
distance_to_city has moderate importance.
from [Link] import RandomForestRegressor
from [Link] import permutation_importance
from sklearn.model_selection import train_test_split
# X, y = features and target
X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=0)
model = RandomForestRegressor(random_state=0).fit(X_train, y_train)
result = permutation_importance(model, X_test, y_test, n_repeats=30, random_state=0)
for i in result.importances_mean.argsort()[::-1]:
print(f"{[Link][i]}: {result.importances_mean[i]:.3f} +/- {result.importances_std[i]:.3f}")
Summary Table
Concept Purpose Example Use
Random Sampling Select representative subset Auditing 4 out of 10 stores
Permutation Testing Test if group difference is real (not chance) Comparing website A vs B time-on-page
Permutation Feature Importance Measure feature’s contribution to model Ranking house price predictors
These notes are intended as a study reference for data analysis coursework. Code examples use Python (pandas, numpy, scikit-learn) and
can be run in any standard Python environment with these libraries installed.