Central Limit Theorem
The central limit theorem (CLT)
How would you explain the central limit theorem to one of your friends?
From the STAT0002 notes …
“… the sample mean of a very large random sample from a distribution with a finite mean and a finite
positive variance is approximately normally distributed, regardless of the shape of the population
distribution.”
“In short, sample means (or indeed sample sums) may have approximately a normal distribution
provided that the sample size is large enough.”
It is NOT correct to say something like
“If the sample size is large then the data are normal”
The CLT concerns sample means or sample sums
A more formal statement of the CLT
Suppose that 𝑋1 , . . . , 𝑋𝑛 are i.i.d. with finite mean 𝜇 and finite variance 𝜎 2 > 0 and the sample mean
1
𝑋 = ∑𝑛𝑖=1 𝑋𝑖
𝑛
Then
𝑋−𝜇 𝑑
𝑍𝑛 = →𝑍 as 𝑛 → ∞
𝜎/√𝑛
where 𝑍 ∼ 𝑁(0,1)
𝑑
𝑍𝑛 → 𝑍 if 𝑃(𝑍𝑛 ≤ 𝑧) → 𝑃(𝑍 ≤ 𝑧) = 𝛷(𝑧) as 𝑛 → ∞ for all 𝑧
A consequence of the CLT for large finite 𝑛
• Subject to the conditions on the previous slide …
⋅
• ∼ means “is approximately distributed as”
⋅
• If the sample size 𝑛 is large enough then
𝑋−𝜇 ⋅
∼ 𝑁(0,1)
𝜎/√𝑛 ⋅
or
⋅ 𝜎2
𝑋 ∼ 𝑁 (𝜇, )
⋅ 𝑛
What do you think determines how large 𝑛 needs to be?
A consequence of the CLT for large finite 𝑛
⋅
𝑋 ∼ 𝑁(𝜇, 𝜎 2 /𝑛)
⋅
What do you think determines how large 𝑛 needs to be?
1
• How close 𝑋1 , . . . , 𝑋𝑛 are to being normally distributed
– The more ‘normal’ they are the smaller is the sample size required
– 𝑛 = 1 is sufficient if 𝑋1 is normally distributed
• How close an approximation we need
Simulation in R
> n <- 10 # sample size n
> nsim <- 1000 # a large number of samples (its exact value is of no importance)
> # Simulate from exponential(1). Put numbers in an nsim by n matrix
> x <- matrix(rexp(n * nsim), ncol = n, nrow = nsim)
> # Add a column of the sample means (of n = 10 observations) for each sample
> xmeans <- rowMeans(x)
> x <- cbind(x, xmeans)
> # Add some names
> rownames(x) <- paste0("sample", 1:nsim)
> colnames(x) <- c(paste0("obs", 1:10), "xbar")
> # Show the first part of the matrix
> head(round(x, 2))
obs1 obs2 obs3 obs4 obs5 obs6 obs7 obs8 obs9 obs10 xbar
sample1 3.26 0.10 2.57 0.52 0.33 0.64 0.13 0.44 1.20 0.14 0.93
sample2 0.16 0.37 0.19 1.08 0.10 1.08 0.62 0.08 1.12 1.09 0.59
sample3 0.12 0.67 1.23 0.35 1.29 0.63 1.32 0.30 2.08 1.18 0.92
sample4 0.87 2.11 0.22 1.33 0.83 0.31 0.31 0.15 0.63 3.50 1.03
sample5 3.40 1.37 0.61 3.00 1.08 1.34 0.73 0.26 0.07 1.57 1.34
sample6 1.04 1.57 6.87 0.17 1.12 0.19 1.39 0.34 0.00 0.04 1.27
Can you see from the matrix what the R code is doing?
Sample means of 10 exponential(1) random variables
> hist(xmeans, prob = TRUE, main = "")
> curve(dnorm(x, mean = 1, sd = 1 / sqrt(n)), add = TRUE)
> legend("topright", legend = "N(1, 1 / n) p.d.f.", lty = 1)
Why the 1 and 1/𝑛 in 𝑁(1,1/𝑛)?
Sample means of 10 exponential(1) random variables
> hist(xmeans, prob = TRUE, main = "")
> curve(dnorm(x, mean = 1, sd = 1 / sqrt(n)), add = TRUE)
> legend("topright", legend = "N(1, 1 / n) p.d.f.", lty = 1)
2
Why the 1 and 1/𝑛 in 𝑁(1,1/𝑛)? The mean and variance of an exponential(1) random variable are
both equal to 1
A competitor for the sample mean
Suppose that we wish to estimate the location of a symmetric distribution
Which other sample statistic could we use?
A competitor for the sample mean
Suppose that we wish to estimate the location of a symmetric distribution
Which other sample statistic could we use?
The sample median
A limit theorem for the sample median
Suppose that 𝑋1 , . . . , 𝑋𝑛 are i.i.d. continuous random variables with median 𝑚 and a p.d.f. 𝑓(𝑥) that is
‘smooth’ near 𝑚. Suppose that 𝑛 is odd.
The sample median is 𝑋(𝑛+1) . Provided that 𝑓(𝑚) > 0
2
𝑋(𝑛+1) − 𝑚
2 𝑑
−1 → 𝑍 ∼ 𝑁(0,1) as 𝑛 → ∞
[2√𝑛𝑓(𝑚)]
Why must 𝑓(𝑚) > 0?
A limit theorem for the sample median
Suppose that 𝑋1 , . . . , 𝑋𝑛 are i.i.d. continuous random variables with median 𝑚 and a p.d.f. 𝑓(𝑥) that is
‘smooth’ near 𝑚. Suppose that 𝑛 is odd.
The sample median is 𝑋(𝑛+1) . Provided that 𝑓(𝑚) > 0
2
𝑋(𝑛+1) − 𝑚
2 𝑑
−1 → 𝑍 ∼ 𝑁(0,1) as 𝑛 → ∞
[2√𝑛𝑓(𝑚)]
3
Why must 𝑓(𝑚) > 0?
−1
• If 𝑓(𝑚) = 0 then we try to divide by 0 in [2√𝑛𝑓(𝑚)]
• The median must have positive probability density
A consequence for large finite 𝑛
• Subject to the conditions on the previous slide …
• If the sample size 𝑛 is large enough then
𝑋(𝑛+1) − 𝑚
2 ⋅
−1 ∼
⋅
𝑁(0,1)
[2√𝑛𝑓(𝑚)]
or
⋅ 1
𝑋(𝑛+1) ∼ 𝑁 (𝑚, )
2 ⋅ 4𝑛𝑓(𝑚)2
Sample mean vs sample median when 𝑛 is large
⋅ 𝑐1 ⋅ 𝑐2
𝑋 ∼ 𝑁 (𝜇, ) 𝑋(𝑛+1) ∼ 𝑁 (𝑚, )
⋅ 𝑛 2 ⋅ 𝑛
1
𝑐1 = 𝜎 2 𝑐2 =
4𝑓(𝑚)2
i.i.d.
Example 𝑋1 , . . . , 𝑋𝑛 ∼ 𝑁(𝜇, 1) ≡ 𝑁(𝑚, 1)
> 1 / (4 * dnorm(0) ^ 2) # the value of c2
[1] 1.570796
Which estimator of location do you prefer, 𝑋 or 𝑋(𝑛+1) ?
2
Sample mean vs sample median when 𝑛 is large
⋅ 𝑐1 ⋅ 𝑐2
𝑋 ∼ 𝑁 (𝜇, ) 𝑋(𝑛+1) ∼ 𝑁 (𝑚, )
⋅ 𝑛 2 ⋅ 𝑛
1
𝑐1 = 𝜎 2 𝑐2 =
4𝑓(𝑚)2
i.i.d.
Example 𝑋1 , . . . , 𝑋𝑛 ∼ 𝑁(𝜇, 1) ≡ 𝑁(𝑚, 1)
> 1 / (4 * dnorm(0) ^ 2)
[1] 1.570796
Which estimator of location do you prefer, 𝑋 or 𝑋(𝑛+1) ? 𝑋
2
𝑋 and 𝑋(𝑛+1) are unbiased but 𝑋 has the smaller variance
2
Student’s 𝑡-distribution with 𝜈 degrees of freedom (df)
𝜈
If 𝑋 ∼ 𝑡𝜈 then E(𝑋) = 0 for 𝜈 > 1 and var(𝑋) = 𝜈−2 for 𝜈 > 2
4
What is the median of 𝑋?
Student’s 𝑡-distribution with 𝜈 degrees of freedom (df)
𝜈
If 𝑋 ∼ 𝑡𝜈 then E(𝑋) = 0 for 𝜈 > 1 and var(𝑋) = 𝜈−2 for 𝜈 > 2
What is the median of 𝑋? 0
Perhaps you know to which distribution the 𝜈 = ∞ case corresponds?
Student’s 𝑡-distribution with 𝜈 degrees of freedom (df)
𝜈
If 𝑋 ∼ 𝑡𝜈 then E(𝑋) = 0 for 𝜈 > 1 and var(𝑋) = 𝜈−2 for 𝜈 > 2
5
What is the median of 𝑋? 0
Perhaps you know to which distribution the 𝜈 = ∞ case corresponds? 𝑁(0,1)
Sample mean vs sample median when 𝑛 is large
⋅ 𝑐1 ⋅ 𝑐2
𝑋 ∼ 𝑁 (𝜇, ) 𝑋(𝑛+1) ∼ 𝑁 (𝑚, )
⋅ 𝑛 2 ⋅ 𝑛
> df <- c(2.01, 3:4, 4.68, 5:6, 1000)
> c1 <- df / (df - 2)
> c2 <- 1 / (4 * dt(0, df = df) ^ 2)
> v <- cbind(df, c1, c2)
> colnames(v) <- c("v", "c1", "c2")
> round(v, 2)
v c1 c2
[1,] 2.01 201.00 2.00
[2,] 3.00 3.00 1.85
[3,] 4.00 2.00 1.78
[4,] 4.68 1.75 1.75
[5,] 5.00 1.67 1.73
[6,] 6.00 1.50 1.71
[7,] 1000.00 1.00 1.57
Why did we exclude 𝜈 ≤ 2?
Sample mean vs sample median when 𝑛 is large
⋅ 𝑐1 ⋅ 𝑐2
𝑋 ∼ 𝑁 (𝜇, ) 𝑋(𝑛+1) ∼ 𝑁 (𝑚, )
⋅ 𝑛 2 ⋅ 𝑛
> df <- c(2.01, 3:4, 4.68, 5:6, 1000)
> c1 <- df / (df - 2)
> c2 <- 1 / (4 * dt(0, df = df) ^ 2)
> v <- cbind(df, c1, c2)
> colnames(v) <- c("v", "c1", "c2")
> round(v, 2)
v c1 c2
[1,] 2.01 201.00 2.00
[2,] 3.00 3.00 1.85
[3,] 4.00 2.00 1.78
6
[4,] 4.68 1.75 1.75
[5,] 5.00 1.67 1.73
[6,] 6.00 1.50 1.71
[7,] 1000.00 1.00 1.57
Why did we exclude 𝜈 ≤ 2?
The variance of 𝑋 is not finite for 𝜈 ≤ 2, so the CLT does not apply
Sample mean vs sample median when 𝑛 is large
⋅ 𝑐1 ⋅ 𝑐2
𝑋 ∼ 𝑁 (𝜇, ) 𝑋(𝑛+1) ∼ 𝑁 (𝑚, )
⋅ 𝑛 2 ⋅ 𝑛
> df <- c(2.01, 3:4, 4.68, 5:6, 1000)
> c1 <- df / (df - 2)
> c2 <- 1 / (4 * dt(0, df = df) ^ 2)
> v <- cbind(df, c1, c2)
> colnames(v) <- c("v", "c1", "c2")
> round(v, 2)
v c1 c2
[1,] 2.01 201.00 2.00
[2,] 3.00 3.00 1.85
[3,] 4.00 2.00 1.78
[4,] 4.68 1.75 1.75
[5,] 5.00 1.67 1.73
[6,] 6.00 1.50 1.71
[7,] 1000.00 1.00 1.57
When do we prefer the sample mean and when the sample median?
Sample mean vs sample median when 𝑛 is large
⋅ 𝑐1 ⋅ 𝑐2
𝑋 ∼ 𝑁 (𝜇, ) 𝑋(𝑛+1) ∼ 𝑁 (𝑚, )
⋅ 𝑛 2 ⋅ 𝑛
> df <- c(2.01, 3:4, 4.68, 5:6, 1000)
> c1 <- df / (df - 2)
> c2 <- 1 / (4 * dt(0, df = df) ^ 2)
> v <- cbind(df, c1, c2)
> colnames(v) <- c("v", "c1", "c2")
> round(v, 2)
v c1 c2
[1,] 2.01 201.00 2.00
[2,] 3.00 3.00 1.85
[3,] 4.00 2.00 1.78
[4,] 4.68 1.75 1.75
[5,] 5.00 1.67 1.73
[6,] 6.00 1.50 1.71
[7,] 1000.00 1.00 1.57
When do we prefer the sample mean and when the sample median?
• the sample mean when 𝜈 ≥ 4.68
• the sample median otherwise
Further resources in R
To play with the CLT ‘movie’ from the CLT video use
> [Link]("smovie")
> smovie::movies()
7
and choose a distribution from the CLT option in the menu that pops up in a separate window
To compare the sample mean and sample median use the function mean_vs_median()
> ?mean_vs_median
See the Examples at the bottom of the page
The take-home messages
• The sample mean (or sum) of a large enough number of random variables has approximately a
normal distribution
• Some observations result from averaging or summing a large number of quantities, e.g. mean
temperatures, exam results
• A similar result holds for the sample median
• There are cases where the sample median is a better estimator of location than the sample
mean