Continuous Distributions

Contents
1 Ch 3.1 - Random Variables of the Continuous Type 1
1.1 Cumulative Distribution Function (CDF), Probability Density
Function (PDF) . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1
1.2 MGF and Moments for Continuous Random Variables . . . . . . 5

2 Ch 3.2 - The Exponential, Gamma and Chi-Square Distribu-

tions 6
2.1 Exponential and Gamma Distributions . . . . . . . . . . . . . . 6
2.2 When to use Binomial, Geometric, Negative Binomial, Poisson,
Exponential, Gamma . . . . . . . . . . . . . . . . . . . . . . . . . 9
2.3 General Gamma and Chi-square . . . . . . . . . . . . . . . . . . 9
2.4 Quantiles and Percentiles - see also Ch 6.3 . . . . . . . . . . . . . 15

3 Ch 3.1 Uniform Distribution, CDF etc from PDF 17

3.1 Random Numbers . . . . . . . . . . . . . . . . . . . . . . . . . . 17

4 Normal Distribution - Ch 3.3 22

4.1 Definition . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22
4.2 MGF, Mean and SD, Percentiles . . . . . . . . . . . . . . . . . . 23
4.3 Standardised Random Variable, Z . . . . . . . . . . . . . . . . . 26
4.4 Normal and Chi-Square . . . . . . . . . . . . . . . . . . . . . . . 28
4.5 Central Limit Theorem and Law of Large Numbers - Ch 5.6 & 5.7 30

1 Ch 3.1 - Random Variables of the Continuous

Type
1.1 Cumulative Distribution Function (CDF), Probability
Density Function (PDF)
Part 3 of Accidents in a Workplace Ex.

Asked about the mean and standard deviation of time to 1st accident
Because the event that T1 > t (t > 0)is the same as the event that Xt = 0, for
a Poisson process with rate λ

P (T1 ≤ t) = 1 − P (T1 > t)

= 1 − P (Xt = 0)
= 1 − e−λt (1)

Problem: as a function of t this is continuous - see Figure 1

Contrasts with the discrete random variable Xt where the graph of P (Xt ≤ x)
has jumps - see Figures 2 and 3

1
 1.0
0.8
0.6
Prob.

0.4
0.2
0.0

0 2 4 6 8 10

Figure 1: Time to 1st Event of Poisson process - Cumulative Probability, P (T1 ≤

t)
1.0
0.8
0.6
PMF

0.4
0.2
0.0

0 1 2 3 4

Figure 2: PMF for X1 the number of accidents in a year

2
 1.0
0.8
0.6
CDF

0.4
0.2
0.0

−1 0 1 2 3 4 5

Figure 3: Cumulative Probs for X1 the number of accidents in a year, P (X1 ≤ x)

Cumulative Distribution Function and Probability Density Function

Define for any continuous random variable X the Cumulative Distribution Func-
tion, F , as the function defined for all real values x by

F (x) = P (X ≤ x) (2)

And call its derivative the Probability Density Function, f , so

f (x) = F 0 (x) (3)

Then Fundamental Theorem of Calculus gives

Z x
F (x) = f (y) dy (4)
−∞

Density Function for T1

CDF previously derived:

F (x) = P (T1 ≤ x)
(
0, x<0
=
1 − e−λx , x≥0

3
 0.1 0.2 0.3 0.4 0.5 0.6
PDF, f(x)

0 1 2 3 4

Figure 4: Probability Density Function, Called Exponential, for the RV T1

PDF now obtained by differentiating

f (x) = F 0 (x)
(
0, x<0
= (5)
λe−λx , x≥0

Important Properties of Continuous RVs

CDF It is assumed that the CDF is differentiable except at ”boundary” values

(such as 0 in the Exponential case)
CDF is always increasing and has limit 0 towards −∞ and limit 1 towards +∞
because of properties of probability
PDF is non-negative and continuous (in this course) except at ”boundary” val-
ues

PDF and CDF are defined for all real values x

Example - Accidents in a Workplace - Exponential

Suppose that accidents in the workplace occur at a rate of 0.62 per year.
What is the probability that the first accident occurs between the end of the
first year and the beginning of the third?

Solution - Accidents in a Workplace - Exponential

4
 P(1<T1 ≤ 2) =0.249

P(0<T1 ≤ 1) =0.462
PDF, f(x)

P(0<T1 ≤ 2) =0.711

=Blue Area+Red Area

0 1 2 3 4
x

Figure 5: Exponential Distribution

Let T1 be the time to the 1st accident. Then from equation

P (1 < T1 ≤ 2) = P (T1 ≤ 2) − P (T1 ≤ 1)

= (1 − e−0.62×2 ) − (1 − e−0.62×1 )
= 0.2485602

1.2 MGF and Moments for Continuous Random Variables

Mean, Variance and MGF for Continuous Random Variables

Discrete RV X with pmf f and a real-valued function h has

X
E(h(X)) = h(x)f (x)
x∈rangeX

Analogy for a continuous random variable X with probability density function

f Z ∞
E(h(X)) = h(x)f (x) dx (6)
−∞

Mean Z ∞
E(X) = xf (x) dx (7)
−∞

Mean, Variance and MGF for Continuous Random Variables

MGF Z ∞
tX
E(e )= etx f (x) dx (8)
−∞

5
Second Moment gives Variance Z ∞
2
E(X ) = x2 f (x) dx (9)
−∞

Rules the Same - Just swap integrals for sums

2 Ch 3.2 - The Exponential, Gamma and Chi-

Square Distributions
2.1 Exponential and Gamma Distributions
Example - Accidents in a Workplace - Gamma
Find the moment generating function, mean and standard deviation for the
time to the (a) first and (b) fourth accidents, assuming they occur according
to a Poisson process with rate 0.62 per year. Also find the probability density
function for the time to the fourth accident.

Solution Exponential - Moment Generating Function for T1

PDF is given in equation (3) so equation (8) gives the MGF, M of T1 as
M (t) = E(etX )
Z ∞
= etx 0.62e−0.62x dx
0

since the density is 0 for x < 0

PDF is given in equation (3) so equation (8) gives the MGF, M of T1 as
M (t) = E(etX )
Z ∞
= etx 0.62e−0.62x dx
0

since the density is 0 for x < 0 and this

Z ∞
0.62
= (0.62 − t)e−(0.62−t)x dx
0.62 − t 0

PDF is given in equation (3) so equation (8) gives the MGF, M of T1 by

M (t) = E(etX )
Z ∞
= etx 0.62e−0.62x dx
0

since the density is 0 for x < 0 and this

Z ∞
0.62
= (0.62 − t)e−(0.62−t)x dx
0.62 − t 0
The integrand is, provided t < 0.62, the exponential pdf with rate 0.62 − t,
so
0.62
M (t) = ×1
0.62 − t

6
since pdf’s integrate to 1 - this trick will be used frequently

Note on Exponential PDF

Must integrate to 1 because

Limit of P (T1 ≤ x) as x → ∞ is 1 (check this if you like in equation (1)) so

1 = lim P (T ≤ x)
x→∞
Z x
= lim f (x) dx
x→∞ −∞
Z ∞
= f (x) dx
−∞
Z ∞
= λe−λx dx
0

Solution Exponential - Mean and SD for T1

MGF derivatives, for t < 0.62 are:

0.62 2 × 0.62
M 0 (t) = , M 00 (t) =
(0.62 − t)2 (0.62 − t)3

Mean:
1
E(T1 ) = M 0 (0) = = 1.613
0.62
SD:
r
p p 2
V ar(T1 ) = M 00 (0) − (E(T1 ))2 = − (E(T1 ))2
0.622
= 1.613

Solution Gamma - MGF and Moments for T4

Important Just as in discrete time, the inter-accident times T1 , T2 − T1 , · · · are

independent and identically distributed - in continuous time, they all
have the exponential distribution with rate the same as the Poisson
process
Proof is similar to discrete time. For example, if t, s > 0, arguing informally
using the independence properties of the Poisson process:

P (T1 > t ∩ T2 − T1 > s) = P (T1 > t ∩ XT1 +s − XT1 = 0)

= P (T1 > t)P (XT1 +s − XT1 = 0)
= e−0.62t e−0.62s

7
Solution Gamma - MGF and Moments for T4 ctd

So T4 is the sum of 4 independent random variables each with the exponential

distribution with rate = 0.62
Hence using the fact that the MGF of a sum of independent random variables
is the product of their MGF’s, the MGF M4 of T4 is given by
 4
4 0.62
M4 (t) = (M (t)) =
0.62 − t

Expected value of any sum is the sum of the expectations, so

4
E(T4 ) = 4E(T1 ) = = 6.451
0.62

Variance of a sum of independent random variables is the sum of the variances,

so the standard deviation of T4 is
p p
V ar(T4 ) = 4V ar(T1 ) = 3.226

Solution Gamma - PDF for T4

CDF F4 can be obtained from the argument used in the example at the end of
Module 2, so for t > 0

F4 (t) = 1 − P (T4 > t)

= 1 − P (Xt ≤ 3)

= 1 − e−0.62t + 0.62te−0.62t

(0.62t)2 e−0.62t (0.62t)3 e−0.62t


+ +
2 6

Solution Gamma - PDF for T4 ctd

PDF f4 is obtained by differentiating - note the remarkable cancellation in the

telescoping sum:

f4 (t) = F40 (t)

= 0.62e−0.62t − 0.62e−0.62t + 0.622 te−0.62t − 0.622 te−0.62t
0.623 t2 e−0.62t 0.623 t2 e−0.62t 0.624 t3 e−0.62t
+ − +
2 2 6
0.624 t3 e−0.62t
=
6

In general

MGF of the time, Tn to the nth event of a Poisson process of rate λ is Mn given,
for t < λ by  n
λ
Mn (t) = (10)
λ−t

8
Mean & SD √
n n
E(Tn ) = , SD(Tn ) = (11)
λ λ
PDF fn is given by (
0, x<0
f (x) = λn xn−1 e−λx (12)
(n−1)! , x≥0

Gamma is the name of the distribution of Tn

Proofs

MGF & Moments: We found the MGF for T1 with rate 0.62 in the example -
general λ same. The MGF for Tn now follows, as in the example, by the
fact that the inter-event times Ti − Ti−1 are independent each with the
same exponential pdf as T1 . Differentiate and set to 0 for moments.
CDF WithXt a Poisson process of rate λ,
n−1
X e−λ λi
P (Tn ≤ x) = 1 − P (Xt ≤ n − 1) = 1 −
i=0
i!

PDF Differentiate the CDF with respect to x and get the same telescoping sum
as in the example - see text p. 106

2.2 When to use Binomial, Geometric, Negative Bino-

mial, Poisson, Exponential, Gamma
When to use?
Binomial, Geometric, Negative Binomial, Poisson, Exponential, Gamma
Time Assumptions Counts Inter-event Event
Times Times
Discrete Bernoulli trials - Binomial Independent Negative
independent, equal Geometric Binomial
probabilities
Continuous No multiple points, Poisson Independent Gamma
independence of Exponen-
counts in disjoint tial
time intervals,
mean count = in =
length intervals

2.3 General Gamma and Chi-square

General Gamma

Gamma function defined for any value α > 0 by

Z ∞
Γ(α) = y α−1 e−y dy (13)
0

9
Integer α PDFs integrate to one, so comparing with the PDF for Tα , the time
to the nth event in a Poisson process of rate 1,
Γ(α) = (α − 1)! (14)

PDF The probability density function for a general Gamma with shape param-
eter α > 0 and rate parameter λ > 0 is given by
(
0, x<0
f (x) = λα xα−1 e−λx (15)
Γ(α) , x≥0

Existence of Gamma Function

Gamma function for integer α is equal to (α − 1)! because, if f is the pdf of the
time to the αth event of a Poisson process of rate 1, then
Z ∞
1= f (y) dy
−∞
Z∞ α−1 −y
y e
= dy
0 (α − 1)!
So
Z ∞
(α − 1)! = y α−1 e−x dy (16)
0
= Γ(α), by the definition of Γ(α). (17)

Existence of Gamma Function Ctd

For non-integer α > 1, let [α] be the integer part of α, that is the largest integer
< α. Then for y > 0 (
α−1 1, y≤1
y < [α]
(18)
y , y>1
For α ≤ 1 (
y α−1 , y ≤ 1
y α−1 e−y < (19)
ye−y , y > 1

Existence of Gamma Function Ctd 2

Hence using (18) and (19)
Z ∞
Γ(α) = y α−1 e−y dy
0
Z 1 Z ∞
α−1 −y
= y dy +e y α−1 e−y dy
0 1
h i1 Z ∞
 α
 1 yα + ye−y dy, α ≤ 1
0 1
< Z ∞
1 + y [α] e−y dy, α>1

1
1 + 1,
(
α≤1
< α <∞
1 + ([α] + 1)!, α > 1

10
 with the 1 in the case α ≤ 1 coming from the fact that the integral from
1 to ∞ is bounded by the mean of an exponential pdf with rate 1 and
the factorial from (16)

Existence of Gamma PDF

Now that we know that the Gamma function evaluated at any α > 0 is a finite
number, can show that the pdf in (15) integrates to on
Z ∞ Z ∞ α α−1 −λx Z ∞ α−1 −y
λ x e y e
f (x)dx = dx = dy
−∞ 0 Γ(α) 0 Γ(α)
on letting y = λx so that dy 0=0 λdx (in the sense that the derivative of y
as a function of x is the constant λ).
So Z ∞ Z ∞
1
f (x)dx = y α−1 e−y dy = 1,
−∞ Γ(α) 0
as required, by the definition of the Gamma function.

General Gamma Ctd

Rate or Scale? To show that the PDF integrates to one for general λ and not just
the case λ = 1, a change of variables in the integral is necessary.
The number θ = 1/λ is more convenient in this calculation and
is thus called the scale of the Gamma distribution. The PDF
becomes (
0, x<0
f (x) = xα−1 e−x/θ (20)
θ α Γ(α) , x ≥ 0

PDF& CDF pictures for varying shape and scale parameters are given in Figures
6, 7, 8 and 9. For non-integer shape, the CDF does not have an
easy closed form just like the Gamma function for non-integers.

MGF & Moments

Follow as previously since it can be shown that the independence properties
continue to hold with non-integer shape.
Scale is the reciprocal of rate so for t < 1/θ the moment generating function,
M , for a Gamma distribution with shape parameter α > 0 and scale
parameter θ > 0 is
1
M (t) = (21)
(1 − θt)α

11
 α = 0.25 α=1 α=2 α=3

0.25

0.20

0.15 θ=4
f(x)

0.10

0.05

0.00
0 10 20 30
x

Figure 6: Gamma Probability Density Functions - Varying Shape

α = 0.25 α=1 α=2 α=3

1.00

0.75
F(x)

0.50

0.25

θ=4

0.00
0 10 20 30
x

Figure 7: Gamma Cumulative Distribution Functions - Varying Shape

12
 θ = 0.8 θ=1 θ=2 θ=4

0.3

0.2
f(x)

α=4

0.1

0.0
0 10 20 30
x

Figure 8: Gamma Probabiliity Density Functions - Varying Scale

θ = 0.8 θ=1 θ=2 θ=4

1.00

0.75
F(x)

0.50

0.25

α=4

0.00
0 10 20 30
x

Figure 9: Gamma Cumulative Distribution Functions - Varying Scale

13
 r=2 r=3 r=5 r=8

0.5

0.4

0.3
f(x)

0.2

0.1

0.0
0 10 20 30
x

Figure 10: Chi-Square PDFs - Varying Degrees of Freedom

Mean & SD The mean, µ, and standard deviation, σ, for the Gamma distri-
bution with shape parameter α > 0 and scale parameter θ > 0
are √
µ = αθ, σ = αθ (22)

Chi-Square
The Gamma density with shape being a multiple of 12 , say α = 2r , for some
integer r = 0, 1, 2, · · · and scale 2 is called the Chi- square distribution and
denoted χ2 (r). The pdf is:
(
0, x<0
f (x) = xr/2 −1 e−x/2 (23)
r
2 /2 Γ(r/ )
, x≥0
2

Degrees of Freedom - name given to the parameter r for the Chi-square dis-
tribution. There are many statistical applications of the
Chi-square distribution and in these applications, the name
degrees of freedom is natural. Figures 10 and 11 show the
Chi-square pdf and cdf for varying degrees of freedom

Chi-Square Distribution
MGF For t < 2 the MGF, M , for the χ2 (r) distribution is:
1
M (t) = r (24)
(1 − 2t) /2

Mean & SD The mean, µ, and standard deviation, σ, for the χ2 (r) distribution
are: √
µ = r, σ = 2r (25)

14
 r=2 r=3 r=5 r=8

1.00

0.75
F(x)

0.50

0.25

0.00
0 10 20 30
x

Figure 11: Chi-Square CDFs - Varying Degrees of Freedom

2.4 Quantiles and Percentiles - see also Ch 6.3

Quantiles
Values from the inverse of the CDF are called quantiles of the distribution.
Formally, the pth quantile for the CDF, πp , is the value solving
F (πp ) = p (26)

Discrete distributions may have no solutions to equation (26) and at values in

the range of the random variable there will be infinitely many distri-
butions, because the CDF is flat apart from jumps at the points of the
range. Usually the left-hand end point of a flat part is taken as the
percentile.
Continuous distributions have only one solution - provided their density has no
flat spots at 0 - so, for each 0 < p < 1 , πp is well defined

Percentiles

Percentiles
Percentiles The 100pth percentile is the pth quantile
Median is the 50th percentile

Example - Chi-Square
Suppose a random variable X has a chi-square distribution with two degrees
of freedom. What is its probability density function? Does it have another
name? What are the mean and median, as well as the 5th, 25th, 75th and 95th
percentiles for the random variable X?

15
 1.00

0.75
PDF

0.50 shaded area is 0.75

0.25

0.00
0 1 π0.75 2 3 4
x

Figure 12: 0.75 Quantile is π0.75 - PDF graph

1.00

0.75
CDF

0.50 inverse of CDF gives quantile

0.25

0.00
0 1 π0.75 2 3 4
x

Figure 13: 0.75 Quantile is π0.75 - CDF graph

16
Solution - Chi-Square

PDF is:

0, x<0
f (x) = 2 x
x /2 −1 e− /2
 2/ 2
, x≥0
2 2 Γ( /2 )
(
0, x<0
= e− /2
x

2 , x≥0

which is the pdf of exponential with rate 1/2 !

The mean is 2 and the quantiles, πp solve
1
F (πp ) = 1 − e− /2 πp = p

So
πp = −2 log(1 − p)

Solution - Chi-Square Ctd

Median is −2 log(0.5) = 1.39 - less than the mean because of the skewness
5th 25th,75th and 95th percentiles are −2 log(1 − p) for p = 0.05, 0.25, 0.75, 0.95
so they are
0.013, 0.575, 2.773, 5.991 (respectively)

3 Ch 3.1 Uniform Distribution, CDF etc from

PDF
3.1 Random Numbers
Example - Random Numbers
Random numbers generated by a computer between 2 and 4 are all equally
likely. What is the probability density for such random numbers? Sketch it.
What is the mean and standard deviation of the numbers? If 1,000,000 are gen-
erated, what should the mean and standard deviation of the generated numbers
approximate?

Solution - Random Numbers

Model the random numbers as coming from a continuous distribution as this

will cover all architectures and software generating the random numbers
= Likely random numbers between 2 and 4 means that the density must be
constant between 2 and 4 (0 elsewhere)

17
Let f be the density with value c (say) between 2 and 4. Then
Z ∞
1= f (x) dx
−∞
Z 4
= c dx
2
 4
= cx
2
= 2c

so c = 1/2 .

Solution - Random Numbers Ctd

Hence (
0, x < 2, x > 4
f (x) = 1
2, 2 ≤ x ≤ 4

Figure 14 shows the PDF of the random numbers

Mean If µ is the mean of the pdf f , then
Z ∞
µ= xf (x) dx
−∞
Z 4
x
= dx
2 2
 2 4
x
=
4 2
=3

Solution - Random Numbers Ctd 2

Variance If σ is the SD of the pdf f , then

Z ∞
2
σ = x2 f (x) dx − µ2
−∞
Z4 2
x
= dx − 9
2 2
 3 4
x
= −9
6 2
1
=
3

Hence
1
σ = √ = 0.5773
3

18
 0.5

0.4

0.3
PDF

0.2

0.1

0.0
1 2 3 4 5
x

Figure 14: PDF of the Random Numbers

Solution - Random Numbers Ctd 3

If 1,000,000 random numbers are generated, the mean and sd of the 1,000,000
numbers would be expected to be close to µ and σ.
How close? - see the subsection on the Central Limit Theorem

Example - CDF from the PDF

Find the CDF from the PDF of the uniform random numbers and plot it

Solution - CDF from the PDF

Let F be the CDF of the random numbers. For 2 ≤ x ≤ 4

Z x
F (x) = f (y) dy
−∞
Z x
1
= dy
2
2 x
y
=
2 2
x
= −1
2

x < 2: F (x) = 0
x > 4: F (x) = 1

19
 1.00

0.75
CDF

0.50

0.25

0.00
1 2 3 4 5
x

Figure 15: CDF of Random Numbers

Solution - CDF from the PDF Ctd

Hence 
0,
 x<2
x
F (x) = − 1, 2 ≤ x ≤ 4
2
1, x>4


Figure 15 shows this CDF

Example - Other Random Numbers

If U is a random variable with PDF which is 1 between 0 and 1, and 0
elsewhere, what is the CDF of 2U+2? If a computer only produces equally
likely random numbers between 0 and 1, how would you produce equally likely
numbers between 2 and 4?

Solution - Other Random Numbers

PDF of U is f given by (
0, x < 0, x > 1
f (x) =
1, 0 ≤ x ≤ 1
Figure 16 shows the relationship of events for 2U + 2 and U

20
 2u + 2 ≤ x u≤x 2−1

4
y

y = 2u + 2
x

2
0 x 2−1 1
u

x
Figure 16: Event [2U + 2 ≤ x] = [U ≤ 2 − 1]

CDF F of 2U + 2 is given, for 2 < x < 4 by

F (x) = P (2U + 2 ≤ x)
x−2
= P (U ≤ )
2
Z x−2
2
= 1 dy
0
  x−2
2
x
= y = −1
0 2

Solution - Other Random Numbers Ctd

Same as the CDF for the random numbers between 2 and 4

So take random numbers from a computer generating them only between 0 and
1, multiply 2 and add 2 to get random numbers between 2 and 4
Comment All random numbers are based on the equally likely ones between 0
and 1
Algorithms use generalisations of the argument we used here - see Ch 5.1 and
5.2, as well as 3.3 on the Normal Distribution standardized random
variable and square of it

Uniform Distribution

21
 µ=3 µ=4 µ=5 µ=6

0.5 σ=1

0.4

0.3
f(x)

0.2

0.1

0.0
0.0 2.5 5.0 7.5 10.0
x

Figure 17: Normal PDFs - Varying µ

Name of the distribution of U in the last example is the Uniform Distribution

with parameters 0 and 1
Name of the distribution for the equally likely random numbers between 2 and
4 is the Uniform Distribution with parameters 2 and 4
PDF CDF, Mean, SD for the general Uniform distribution are found by the
same arguments we used in the example on random numbers between 2
and 4

4 Normal Distribution - Ch 3.3

4.1 Definition
PDF

PDF of a Normal Distribution with parameters µ and σ is the function f given,

for any real x by

(x − µ)2
 
1
f (x) = √ exp − (27)
2πσ 2σ 2

Figures 17 and 18 show the PDF for varying values of µ and σ

Notation: a random variable X with density function f is written as X ∼

N (µ, σ 2 )

22
 σ = 0.4 σ = 0.8 σ=1 σ=6

1.00 µ=5

0.75
f(x)

0.50

0.25

0.00
0.0 2.5 5.0 7.5 10.0
x

Figure 18: Normal PDFs - Varying σ

CDF
CDF cannot be expressed as a finite sum of elementary functions like polyno-
mials or exponentials or logs or trignometric funtions
Showing that the pdf integrates to 1 requires a trick by squaring the integral,
rewriting the square as a double integral and then transforming to
polar co-ordinates
Figures 19 and 20 show normal CDFs with varying µ and σ
Normal probabilities are found from packages like R or Mathematica and his-
torically from tables

4.2 MGF, Mean and SD, Percentiles

MGF
Suppose X ∼ N (µ, σ 2 ) and that M is the moment generating function of X ,so
for any real t
M (t) = E(etX )
Z ∞
(x − µ)2
 
1
= exp [tx] √ exp − dx
−∞ 2πσ 2σ 2

Product of exp is exp of sum so the integrand exponent is:

(x − µ)2 1
−2txσ 2 − (x − µ)2
 
tx − =− 2
2σ 2 2σ
1
(x − (µ + tσ 2 ))2 − 2µσ 2 t − σ 4 t2
 
=− 2
2σ
on expanding (x − µ)2 and completing the square.

23
 µ=3 µ=4 µ=5 µ=6

1.00

0.75
F(x)

0.50 σ=1

0.25

0.00
0.0 2.5 5.0 7.5 10.0
x

Figure 19: Normal CDFs - Varying µ

σ = 0.4 σ = 0.8 σ=1 σ=6

1.00

0.75
F(x)

0.50 µ=5

0.25

0.00
0.0 2.5 5.0 7.5 10.0
x

Figure 20: Normal CDFs - Varying σ

24
MGF Ctd

Hence
∞
2µσ 2 t + σ 4 t2 (x − (µ + σ 2 t))2
 Z  
1
M (t) = exp √ exp − dx
2σ 2 −∞ 2πσ 2σ 2

But the integral is that of the N (µ + σ 2 t, σ 2 ) pdf so = 1 and hence

2µσ 2 t + σ 4 t2
 
M (t) = exp
2σ 2
σ 2 t2
 
= exp µt + (28)
2

Derivatives of MGF

First:
σ 2 t2
 
M 0 (t) = (µ + σ 2 t) exp µt + = (µ + σ 2 t)M (t)
2

Second: use the derivative product rule to get

σ 2 t2
 
0 2 2 2


M (t) = (µ + σ t) + σ exp µt +
2
= (µ + σ 2 t)2 + σ 2 M (t)

Moments

Mean
E(X) = (µ + σ 2 × 0)M (0) = µ (29)

Variance

V ar(X) = M 00 (0) − µ2
= (µ + σ 2 × 0)2 + σ 2 M (0) − µ2

= σ2 (30)

SD p
SD(X) = V ar(X) = σ (31)

Example - Normal calculations

If X ∼ N (3, 1), what is the moment generating function, mean and stan-
dard deviation of X? What are the median and the 5th, 25th, 75th and 95th
percentiles for X?

25
Solution - Normal Calculations

MGF: from equation (28) the MGF is

t2
 
M (t) = exp 3t +
2

Mean & SD: from equations (29) and (31)

E(X) = 3; SD(X) = 1

Percentiles other than the median - which is 3 by symmetry of the pdf - need
to come from R and here are the commands:
> p <- c(0.05, 0.25, 0.75,0.95)
> qnorm(mean = 3,sd = 1,p)
which give the 5th, 25th, 75th and 95th percentiles as 1.36, 2.33,
3.67 and 4.64 to two dp.

4.3 Standardised Random Variable, Z

Definition of Z

Suppose X ∼ N (µ, σ 2 ), then the standardised random variable is Z given by

X −µ
Z= (32)
σ

By the same argument as for the change from uniform random numbers on (0,1)
to those on (2,4), Z has a normal distribution
Rules for expectation and variance give

E(Z) = 0; V ar(Z) = 1

So
Z ∼ N (0, 1)
and Z is said to have the standard normal distribution

Z to X

Solving (32) for X gives

X = σZ + µ (33)

So normal probabilities for X can be found from those for Z:

P (X ≤ x) = P (σZ + µ ≤ x)
x−µ
= P (Z ≤ )
σ
ie the number x is turned into statistical units as x−µ
σ and the standard
normal CDF is applied to the number in statistical units
100pth percentile, πp , for Z gives the 100pth percentile for X as σπp + µ

26
 0.4 shaded area is 0.68

0.3
Stan. Norm. PDF

0.2

0.1

0.0
−3 −2 −1 0 1 2 −3
x

Figure 21: Standard Normal Distribution - 1 standard deviation from mean

0.4 shaded area is 0.95

0.3
Stan. Norm. PDF

0.2

0.1

0.0
−3 −2 −1 0 1 2 −3
x

Figure 22: Standard Normal Distribution - 2 standard deviations from mean

27
 0.4 shaded area is 0.997

0.3
Stan. Norm. PDF

0.2

0.1

0.0
−3 −2 −1 0 1 2 −3
x

Figure 23: Standard Normal Distribution - 3 standard deviations from mean

4.4 Normal and Chi-Square

Normal & Chi-Square

Let Z ∼ N (0, 1). Then the moment generating function of Z 2 is given by

Z ∞  2
1 z
E(exp Z 2 t ) = √ exp z 2 t exp −
   
dz
−∞ 2π 2
Z ∞  2 
1 z (1 − 2t)
= √ exp − dz
−∞ 2π 2
Z ∞ 1
(1 − 2t) /2
 2 
1 z (1 − 2t)
= (1 − 2t)− /2 √ exp − dz (34)
−∞ 2π 2
 
1
Integrand on the right hand side of (34) is the pdf of N 0, 1−2t so integrates
to 1
Thus the factor before the integral is the MGF of Z 2 and it is the MGF of χ21 ,
so Z 2 ∼ χ21

Example - Errors in Measurement

A sensing device for location has normally distributed errors in each of the
three directions that are independent and have mean 0 and standard deviation
0.1 m. What is the chance that the distance of the measured location is more
than 0.3 m from the true location in 3D?

Solution - Errors in Measurement

Let Di ∼ N (0, 0.12 ) be the three independent errors in measurement in height

and planar locations

28
 (D 0.1)2 > 9 D > 0.3

100

y = (D 0.1)2
y

9
0
0.0 0.3 1.0
D
h 2 i
D
Figure 24: Event [D > 0.3] is the same as 0.1 >9

Then p
the distance of the measured location from the true location is D =
D12 + D22 + D32 , so the required probability is
 2  2 !
D 0.3
P (D > 0.3) = P >
0.1 0.1

since the two events are the same because the random variable D is non-
negative - see Figure 24
Di , i = 1, 2, 3 be the three independent standardised ran-
Now, letting Zi = 0.1
dom variables,
 2
D
= Z12 + Z22 + Z32
0.1

Solution - Errors in Measurement Ctd

MGF of a sum is the product of the MGF’s and Zi2 ∼ χ21 so the MGF M (t) of
the sum of these three is given by
3
M (t) = (1 − 2t)− /2

which is that of χ23

Hence required probability can be found from R using the command 1-pgamma(9,rate=2,shape=3)
and the answer is < 7 × 10−8
Beware intuition about adding errors!

29
 1.0
0.8
0.6
P(Chisq(3) > x)

0.4
0.2
0.0

0 2 4 6 8

Figure 25: CDF of the sum of Chisq(3)

4.5 Central Limit Theorem and Law of Large Numbers -

Ch 5.6 & 5.7
Central Limit Theorem

Suppose X1 , X2 , · · · Xn are independent random variables all with the same dis-
tribution having mean µ and standard deviation σ
Pn
Xi
Then letting X̄ be the sample mean - X̄ = i=1
n . From previous results,
σ
E(X̄) = µ; SD(X) = √ (35)
n

CLT: Central Limit Theorem says that for any number z

p
(n)(X̄ − µ)
P( ) ≤ z) → P (Z ≤ z) (36)
σ
as n → ∞, where Z ∼ N (0, 1)

Central Limit Theorem Translation

If the the number of observations size is large, then the distribution of the
mean of independent random variables with the same distribution is approxi-
mately normal.
The approximate normal distribution is centred at the true population mean
- the Law of Large Numbers - and has standard deviation proportional to both
the population standard deviation and the inverse of the square root of the
number of observations.

30
Example - Conclusions in Opinion Polls
The Australian of 3rd April 2017 reported an opinion poll of 1708 voters
showed that (primary) support for the Government had slipped from last week’s
poll of 37% to 36% this week. The headline was ”Coalition in poll slip after tax
win”. Is this a fair headline assuming the support was actually exactly 37% in
the whole population?s

Solution - Conclusions in Opinion Polls

Let X1 , X2 , · · · X1708 record 0 or 1 (respectively) according to whether the 1st,

2nd, ... , 1708th voter in the sample would not/ would (respectively) vote
Liberal.
P1708
Then X = i=1 Xi has a Binomial distribution (assuming sampling with re-
placement this is exact and otherwise nearly exact from the work done on
the hypergeomtric distribution).
Assume the population proportion of 37% voting Liberal. Then the Binomial
distribution has n = 1, 708 and p = 0.37 so the probability that informs
out judgement is

P (X ≤ 0.37 ∗ 1708) = 0.191

using the R command pbinom(0.36*1708,size=1708,prob=0.37)

Solution - Conclusions in Opinion Polls

CLT approximation, since V ar(Xi ) = 0.37 × 0.63, is that

√
1708(X̄ − 0.37)
√ ≈ N (0, 1)
0.37 × 0.63

And the command to find the normal approximation is pnorm(-0.01*sqrt(1708/(0.37*0.63)))

which gives 0.196
There is 20% chance of getting a sample result of 36% or less with this sample
size - so the headline seems unfair as it could just be a sample fluctuation

Note the sample fluctuation explanation is even more plausible taking into ac-
count that the 37% result of the previous week came from another inde-
pendent poll with approximately the same sample size

31