Chapter 5 Statistics 2A STA02A2

CHAPTER 5: Limit Theorems

Lecture notes sections Content from corresponding textbook sections

1. Introduction 5.1. Introduction
2. Law of large numbers 5.2. Law of large numbers
3. Convergence in distribution and the central limit 5.3. Convergence in distribution and the central limit
theorem theorem

1. Introduction
This chapter is concerned with the limiting behaviour of the sum of independent random variables as the number
of the summands becomes large. The results are useful in statistics, since many commonly computed statistical
quantities, such as averages, can be represented as sums.

2. Law Of Large Numbers

It is commonly believed that if a fair coin is tossed many times the coin would land on heads approximately half
the time so that the probability of observing heads is close to 0.5. John Kerrick, a South African mathematician,
tested this belief empirically while detained as a prisoner in World War II. He tossed a coin 10000 times and
observed 5067 heads.

The law of large numbers (LLN) is the mathematical formulation of this belief and has a central role in probability
and statistics. It states that if you repeat an experiment independently a large number of times and average the
result, what you obtain should be close to the expected value. Therefore, the LLN is a proposition that provides a
set of sufficient conditions for the convergence of the sample mean to a constant. Typically, the constant is the
expected value of the distribution from which the sample has been drawn.

First, we will define the concepts of closeness and convergence:

➢ Closeness
Let  X n  be a sequence of random variables defined on the sample space Ω. Take a random variable X

and a strictly positive number   0 . Suppose we consider that X n is far away for X if X n − X   ,

then P ( X n − X   ) is the probability that X n is far away for X .

1
Chapter 5 Statistics 2A STA02A2

➢ Convergence
If  X n  converges to X , then the probability that X n is far away for X should become smaller and

smaller as n increases, therefore lim P ( X n − X   ) = 0 .

n →

There are two main versions of the LLN, namely the weak law of large numbers (WLLN) and the strong law of
large numbers (SLLN). The difference between them is mostly theoretical. There are many different LLN’s, but
we will focus on Chebyshev’s WLLN, and briefly discuss the SLLN formulation and comparison with a WLLN.

2.1 Weak law of large numbers

Let X 1 , X 2 , , X n be a sequence of independent and identically distributed (i.i.d.) random variables with

1 n
E ( X i ) =  , Var ( X i ) =  2 and sample mean X n =  X i . Then, for any   0 :
n i =1

(
lim P X n −    = 0
n →
)
(
 lim P X n −    = 1
n →
)

In other words, for a very small   0 , the probability that the difference between the sample mean and the
population mean is even smaller than  , forms a sequence of probability values, which increases with n, because
the sample size n approaches the population size N. This sequence converges to 1. Therefore, the sample mean
converges in probability to the expected value.

Proof/Derivation:
Since the X i ' s are independent and identically distributed, it follows that:

1 n  1 n 1 n n
E ( Xn ) = E   Xi  =  E ( Xi ) =   = =
 n i =1  n i =1 n i =1 n
1 n  1 n
Var ( X n ) = Var   X i  = 2 Var ( X i )
 n i =1  n i =1
1 n
n 2  2
=
n2
 2 =
i =1 n2
=
n

The result follows from Chebyshev’s inequality, namely:

Var ( X n ) 2
(
P Xn −     ) 2
= 2 → 0 as n → 
n
(
 lim P X n −    = 0
n →
)
2
Chapter 5 Statistics 2A STA02A2

2.2 Strong law of large numbers

Let X i be a sequence of independent and identically distributed (i.i.d.) random variables with E ( X i ) =  and

Var ( X i ) =  2 , for i = 1, 2, 
n
. Let X n = 1
n i =1
X i . Then,

P  lim ( X n =  )  = 1
 n→ 
(
 P lim X n −  = 0 = 1
n →
)
In other words, a sequence of differences between a sample mean of increasing size and the population mean,
vanishes to 0 with probability 1 as n →  . Therefore, the sample mean converges almost surely to the expected
value.

2.3 SLLN vs. WLLN

➢ SLLN refers to the assurance that something does happen, almost surely.
➢ WLLN refers to the assurance that what we want to see, will happen with increasing probability.
➢ SLLN implies WLLN, WLLN does not imply SLLN

3. Convergence in Distribution and the Central Limit Theorem

In applications, we often want to find P ( a  X  b ) when we do not know the cdf of X precisely. It is sometimes

possible to do this by approximating FX , often using some sort of limiting argument. The most famous limit

theorem in probability theory is the central limit theorem (CLT). The derivation of the CLT depends on the notion
of convergence in distribution and certain properties of the moment-generating function, namely that a
distribution is uniquely determined by its moment-generating function, and that convergence of a moment-
generating function implies convergence in distribution.

Convergence in distribution
Let X 1 , X 2 , be a sequence of random variables with cumulative distribution functions F1 , F2 , , and let X be

a random variable with cumulative distribution function F . Then X n converges in distribution to X if:

lim Fn ( x ) = F ( x )
n →

at every point at which F is continuous.

3
Chapter 5 Statistics 2A STA02A2

Continuity theorem
Let Fn be a sequence of cumulative distribution functions with corresponding moment-generating function M n .

Let F be a cumulative distribution function with moment-generating function M . If M n ( t ) → M ( t ) for al t in

an open interval containing zero, then Fn ( x ) → F ( x ) at all continuity points of F .

Central limit theorem

Let X 1 , X 2 , be a sequence of random variables with a mean of  , a variance of  2 , and the common

distribution function F and moment-generating function M defined in a neighbourhood of zero.

Let S n =  i =1 X i = nX , since X = 
n n
1
n i =1
X i . From the law of large numbers we know that:

• E ( X ) =   E ( Sn ) = E ( nX ) = n

2 2
• Var ( X ) =  Var ( S n ) = Var ( nX ) = n 2Var ( X ) = n 2  = n 2
n n

Note that the standardized form of X can be written as the standardized form of the sum (total) S n :

X − X − n S n − n
Z= =  =
  n  n
n n

Let G denote the cumulative distribution function of the standardized form of X (or S n ). Then, through

convergence in distribution and the continuity theorem, the limit of G is the cumulative distribution function of
the standard normal distribution, that is, the standardized form of X (or S n ) has a limiting standard normal

distribution.
 X − 
lim G = lim P   x  =  ( x)
 n
n → n →

 S − n 
lim G = lim P  n  x  =  ( x)
n → n →
  n 

The detailed proof of the conclusion is not for examination purposes.

4
Chapter 5 Statistics 2A STA02A2

The above theorem implies that X and S n are asymptotically normal, i.e., the distributions approach the normal

 2 
distribution as n increases. Therefore, if n is sufficiently large X ~ N   ,  and Sn ~ N ( n , n 2 ) . The CLT
 n 

is used in inferential statistics to derive the sampling distribution of a statistic and allows use to calculate
probabilities around the behaviour of such sample statistics. It can also be applied to approximate distributions
such as the Poisson (for large values of λ), the Binomial, and others.

Exercise 1
A harassed father, wishing to keep his son quiet, offers to pay him R100 if the average score of 100 throws of a
fair die exceeds 4. What is the probability that the father will have to pay out?

5
Chapter 5 Statistics 2A STA02A2

Exercise 2 (Question 13 in the textbook exercises)

A drunkard executes a “random walk” in the following way: Each minute he takes a step north or south, with
probability 0.5 each, and his successive step directions are independent. His step length is 50cm. Use the central
limit theory to approximate the probability distribution of his location after 1 hour. Where is he most likely to be?

6
Chapter 5 Statistics 2A STA02A2

Exercise 3 (Question 26 in the textbook exercises)

Suppose that a basketball player can score on a particular shot with probability 0.3. Use the central limit theorem
to find the approximate distribution of S, the number of successes out of 25 independent shots. Find the
approximate probabilities that S is less than or equal to 5, 7, 9, and 11, and compare these to the exact probabilities.