Discrete Distributions

Contents
1 Ch 2.1 Random Variables of the Discrete Type 2
1.1 Definitions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2
1.2 Example - Soft Drinks . . . . . . . . . . . . . . . . . . . . . . . . 3
1.3 Probability Mass Function . . . . . . . . . . . . . . . . . . . . . . 4
1.4 Binomial Distribution (Start of Ch 2.4) . . . . . . . . . . . . . . 7
1.5 Derivation of Binomial Probabilities . . . . . . . . . . . . . . . . 8
1.6 Sampling with Replacement . . . . . . . . . . . . . . . . . . . . . 8
1.7 Pictures of Binomial Probabilities . . . . . . . . . . . . . . . . . . 9
1.8 Hypergeometric - Sampling without Replacement . . . . . . . . . 9
1.9 Hypergeometric - Derivation . . . . . . . . . . . . . . . . . . . . . 10
1.10 Hypergeometric - Pictures . . . . . . . . . . . . . . . . . . . . . . 10

2 Ch 2.2 Mathematical Expectation 11

2.1 Mean and variance of data - Section 6.1 . . . . . . . . . . . . . . 11
2.2 Definition of Expectation of RV . . . . . . . . . . . . . . . . . . . 12
2.3 Game Example . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13
2.4 Expectation of a Transform . . . . . . . . . . . . . . . . . . . . . 14

3 Ch 2.3 Special Mathematical Expectations 15

3.1 Variance of a RV . . . . . . . . . . . . . . . . . . . . . . . . . . . 15

4 Ch 2.2 - Back to Mathematical Expectation 16

4.1 Expectation Linear ie Distributive . . . . . . . . . . . . . . . . . 16
4.2 Application to Variance . . . . . . . . . . . . . . . . . . . . . . . 17
4.3 Example - Variance of Triangular PMF . . . . . . . . . . . . . . 17
4.4 Example - Minimum Squared Deviation . . . . . . . . . . . . . . 18

5 Ch 2.2 - Back to Mathematical Expectation 19

5.1 Example - Minimum Squared Deviation . . . . . . . . . . . . . . 19
5.2 Example - Sampling with and without replacement . . . . . . . . 21

6 Back to Ch 2.3 Special Mathematical Expectations 22

6.1 Moment Generating Function . . . . . . . . . . . . . . . . . . . . 22
6.2 Example - Moment Generating Function from PMF . . . . . . . 22
6.3 Example - Moments from MGF . . . . . . . . . . . . . . . . . . . 23
6.4 PMF from MGF . . . . . . . . . . . . . . . . . . . . . . . . . . . 24

7 Ch 4.1 - Independent Random Variables 25

7.1 Indep’t RVs - Var, MGF of Sum . . . . . . . . . . . . . . . . . . 25
7.2 MFG of a Sum . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26
7.3 Properties of Variance . . . . . . . . . . . . . . . . . . . . . . . . 26
7.4 Properties of Sample Mean . . . . . . . . . . . . . . . . . . . . . 27
7.5 Sample Proportion . . . . . . . . . . . . . . . . . . . . . . . . . . 27

1
8 Ch 2.4 - The Binomial Distribution 31
8.1 Recall PMF from Module 2 1 . . . . . . . . . . . . . . . . . . . . 31
8.2 Binomial Mean and Variance . . . . . . . . . . . . . . . . . . . . 32
8.3 Sampling with Replacement . . . . . . . . . . . . . . . . . . . . . 32
8.4 MGF for Binomial . . . . . . . . . . . . . . . . . . . . . . . . . . 32

9 Ch 2.5 - The Negative Binomial Distribution 33

9.1 PMF for Time till Success . . . . . . . . . . . . . . . . . . . . . . 33
9.2 Examples of PMF for Negative Binomial Distribution . . . . . . 33
9.3 MGF and Moments for Negative Binomial . . . . . . . . . . . . . 38
9.4 Example - Accidents in a Workplace . . . . . . . . . . . . . . . . 40

10 Ch 2.6 - the Poisson Distribution 41

10.1 Definition and Derivation . . . . . . . . . . . . . . . . . . . . . . 41
10.2 MGF and Moments . . . . . . . . . . . . . . . . . . . . . . . . . . 42
10.3 Poisson Process and Example . . . . . . . . . . . . . . . . . . . . 44

1 Ch 2.1 Random Variables of the Discrete Type

1.1 Definitions
Why do we need random variables?

Dealing with complicated sample spaces and calculating probabilities of events

by counting outcomes

Is often not practical especially when probabilities come from specific past data
or more general beliefs
Often the results of our experiment or observation is a number or numbers and
the events of interest centre on these numbers
For example, in the rocket failure example, we might only be interested in the
number of components that fail - since the components are as-
sumed identical
And in the soft drink example, we might only be concerned about the number
of chains that give more shelf space to P

And in the sampling lines of code example, we might only be concerned about
the total number of lines of code that could be improved

Definitions

Random Variable Given an outcome space S, a function or rule, X, which as-

sociates with each s ∈ S a number X(s) is called a random
variable, abbreviated rv
Range of rv The range or (Hogg and Tanis) space of a rv X is the set of real
numbers achieved by X ie {x : X(s) = x for some s ∈ S}

2
 X(s)
(P,P,P) 3
(P,C,P)
(P,P,C)
(C,P,P) 2

(C,P,C)
(P,C,C) 1
(C,C,P)
(C,C,C) 0
S

Figure 1: Rv X, the number of chains with more shelf space for P

Figure 2:

Abbreviation If B is a set of numbers, the notation [X ∈ B] or {X ∈ B} is an

abbreviation for {s : X(s) ∈ B} and P (X ∈ B) for P ({s : X(s) ∈
B})
The sample space takes a back seat - what matters are the probabilities associ-
ated with events defined by X
Discrete rv is one with a finite number of outcomes or an infinite number of
outcomes like 1, 2, · · ·

1.2 Example - Soft Drinks

Illustration - Soft Drinks

Illustration - Soft Drinks

The probabilities for X are:

3 chains
1
P (X = 3) = P ({(P, P, P )}) =
8
2 chains
3
P (X = 2) = P ({(P, P, C), (P, C, P ), (C, P, P )}) =
8
1 chain
3
P (X = 1) = P ({(P, C, C), (C, C, P ), (C, P, C)}) =
8
0 chain
1
P (X = 0) = P ({(C, C, C)}) =
8

3
Example - Soft Drinks rv
If X is the number of supermarket chains that give more shelf space to soft
drink P, what is the probability that X is even? odd?
4
P (X is even) = P (X = 0) + P (X = 2) =
8
4
P (X is odd) = P (X = 1) + P (X = 3) =
8

1.3 Probability Mass Function

Probability Mass Function

All the probabilities to do with X are given by P (X = 0), P (X = 1), P (X =

2), P (X = 3) because P (X = x) = 0 for any number x outside the range
{0, 1, 2, 3}

For this reason, the rule which gives for a number x the probability P (X = x)
is called the the probability mass function for the random variable
Small letters are used for arbitrary numbers and capital letters for the names
of random variables

General Definition

As long as the discrete rv X is the only quantity of interest in our experiment

or observation, the probability mass function is sufficient for calculations
The probability mass function, f (x), of real numbers, x, can be any function
satisfying:
(a) f (x) > 0 for x ∈ range(X) and f (x) = 0 for x ∈
/ range(X)
(b) X
f (x) = 1
x∈range(X)

for any set B of numbers,

(c) X
P (X ∈ B) = f (x)
x∈range(X)∩B

Rocket Failure - PMF Example

What is the probablity mass function (pmf) for the number of rocket com-
ponents that do not fail?

4
Rocket Failure PMF - Solution

Let X be the random variable giving the number of rocket components that do
not fail.
The range of X is {0, 1, 2, 3}.

By independence

P (X = 0) = P (A1 ∩ A2 ∩ A3 )
= P (A1 )P (A2 )P (A3 ) = 0.153 = 0.00338

Splitting [X = 1] into the three mutually exclusive events that partition it and
using independence gives

P (X = 1) = P (Ac1 ∩ A2 ∩ A3 ) + P (A1 ∩ Ac2 ∩ A3 ) + P (A1 ∩ A2 ∩ Ac3 )

= 3 × 0.152 × 0.85 = 0.057375

Rocket Failure PMF - Solution Ctd

Similarly

P (X = 2) = P (Ac1 ∩ Ac2 ∩ A3 ) + P (A1 ∩ Ac2 ∩ Ac3 ) + P (Ac1 ∩ A2 ∩ Ac3 )

= 3 × 0.15 × 0.852 = 0.325125
P (X = 3) = P (Ac1 ∩ Ac2 ∩ Ac3 )
= 0.853 = 0.614125

The probability mass function can be written as a table:

x 0 1 2 3
P(X=x) 0.003375 0.057375 0.325125 0.614125

Rocket Failure PMF - Solution Ctd 2

Alternatively,the probability mass function can be viewed as a dot plot:

Rocket Failure PMF - Solution Ctd 3

Or the probability mass function can be viewed as a probability histogram:

Rocket Failure PMF - Comment

The probability computed in the first Rocket Failure example in Module 1 was
P (X ≥ 1), which was computed as 1 − P (X = 0).
An alternative would have been to compute it as P (X = 1)+P (X = 2)+P (X =
3) but this is much more complicated.

5
 Figure 3: Probability Mass Function for X

DiscretePlot[PDF[BinomialDistribution[3, 0.85], k], {k, {0, 1, 2, 3}},

ExtentSize → Full, Ticks → {{0, 1, 2, 3}, Automatic}, AxesLabel → {x, pmf}]

In[1]:= DiscretePlot[PDF[BinomialDistribution[3, 0.85], k],

Figure {k,
4: Probability Histogram
{0, 1, 2, 3}}, for pmf
Ticks → {{0, - shaded
1, 2, blue areaAxesLabel
3}, Automatic}, =1 → {x, pmf}]

Out[1]=
6
1.4 Binomial Distribution (Start of Ch 2.4)
Similarities

Similar Examples In both the soft drink example and the rocket failure example,
the pmf calculations for the rv X were similar

Difference The only difference was the probability 0.5 in the soft drink exam-
ple (equally likely outcomes) compared to 0.85 in the rocket failure
example
Structure? Both examples had the idea of a successful outcome in each of 3
independent trials
Soft Drink Ex. the trials were the decisions of the three supermarket chain and
success was soft drink P having more shelf space
Rocket Ex. the trials were the performance of the components and success was
a component not failing

RV of interest? In both cases, the rv, X, was the number of successes

Bernoulli Trials Definition

Bernoulli Trials So we now consider a number, n, of independent trials

Two Outcomes Each trial has one of two possible outcomes success or failure
denoted S and F
Events of success The results of the trials define independent events A1 , A2 , · · · , An
representing success, S, on trials 1, 2, · · · , n

Events of failure So the events of failure, F , on the same trials are Ac1 , Ac2 , · · · , Acn

Binomial RV X

Binomial RV Consider the rv, X, which is the number of trials that result in
success

Range The range of X is {0, 1, 2, · · · , n}

Component RVs For i = 1, 2, · · · , n, let Xi be the rv which is 1 on the event Ai
and 0 on Aci , so that Xi is 1 for a success on trial i and 0 for
a failure on trial i

So The rv Xi counts the number of successes on trial i

Now The number of successes overall is X and this is the sum of the number of
successes on each trial so
n
X
X = X1 + X2 + · · · + Xn = Xi (1)
i=1

7
1.5 Derivation of Binomial Probabilities
PMF of Binomial RV - Informal

Probability of Success Let p be the constant probability of success on each trial

Component probs By independence, each way to get x successes in the n trials

has probability px (1 − p)n−x
Patterns of S and F The different ways to get x successes are just selections of
the trial numbers on which there are to be successes - each
of the remaining trials must have a failure

Number of patterns There are nx ways to get x successes in n trials

Hence For x = 0, 1, 2, · · · , n
 
n x
P (X = x) = p (1 − p)n−x (2)
x

PMF of Binomial RV - More Formal

S trials and F trials Suppose {i1 , i2 , · · · , ix } is a selection of x trials from {1, 2, · · · n}

to be S and that {j1 , · · · , jn−x } = {i1 , i2 , · · · , ix }c are the
trials to be F
Independence gives for each such selection

P (Ai1 ∩ · · · ∩ Aix ∩ Acj1 · · · ∩ Acjn−x ) = px (1 − p)n−x

Number of subsets There are nx subsets of size x from {1, 2, · · · , n} and the

corresponding events of the pattern of success and failure

are disjoint

Hence For x = 0, 1, 2, · · · , n
 
n x
P (X = x) = p (1 − p)n−x
x

1.6 Sampling with Replacement

Example - Sampling with Replacement
There is a probability of 0.9 that each member of the class has worked outside
class on this subject on the day before any particular lecture day. A spinner
chooses one of the numbers from 1 up to 21 randomly to find out whether that
member of the classes worked on this subject yesterday. This is repeated on
4 lecture days in total. What is the probability mass function for the random
variable X which gives the number of class members chosen who worked on this
subject on the day prior to the one when they were chosen?

8
 PDF[BinomialDistribution[4, 0.9], {0, 1, 2, 3, 4}]
{0.0001, 0.0036, 0.0486, 0.2916, 0.6561}

DiscretePlot[
Table[PDF[BinomialDistribution[40, p], k], {p, {0.1, 0.5, 0.7, 0.9}}] // Evaluate,
{k, 40}, PlotRange → All, PlotMarkers → Automatic, AxesLabel → {x, pmf}]

Figure 5: PMF’s for n = 40, p=0.1,0.5,0.7,0.9

Solution - Sampling with Replacement

There are 4 Bernoulli trials with success being that the class member chosen
worked on this subject on the day prior to them being chosen.
The trials are independent since the spinner choices are independent and we
are told the probability is constant.
Hence,

P (X = 0) = 0.14 = 0.0001
P (X = 1) = 4 × 0.9 × 0.13 = 0.0036
4×3
P (X = 2) = × 0.92 × 0.12 = 0.0486
2
P (X = 3) = 4 × 0.93 × 0.1 = 0.2916
P (X = 4) = 0.94 = 0.6561

1.7 Pictures of Binomial Probabilities

Binomial Probablities

1.8 Hypergeometric - Sampling without Replacement

Hypergeometric - Sampling without Replacement

Example In the example of improving lines of code in the computer, a sample

without replacement was taken at random
RV of interest was the total number, X, of lines of code in the sample that could
be improved

Notation Given a total number, t, of lines of code

Notation a sample size, n, for the sampled lines of code,

9
 Figure 6: PMF’s n = 40, p=0.1,0.5,0.7,0.9, t=50

Notation and a total number, b, of lines of code that can be improved

Hypergeometric PMF The same reasoning as the example gives, for x = 0, 1, 2, · · · , n
b

t−b

x n−x
P (X = x) = t

(3)
n

Note the top (and bottom) lines of the combination numbers in the numerator
add to the top (resp. bottom) line in the denominator

1.9 Hypergeometric - Derivation

Hypergeometric - Derivation

Samples Each
sample is a subset of size n from the t lines of code, so there are
t
n samples

Sample lines improved There are xb ways to choose the x lines in the sample

from the b lines that can be improved

t−b


Samples lines not There are n−x to choose the n − x lines in the sample from
the t − b lines that can’t be improved
Multiplication Principle says that the total number of ways to choose both the
lines that can, as well as the lines that can’t, be im-
proved is the multiplication of the two numbers

Equally likely random samples now completes the derivation

1.10 Hypergeometric - Pictures

Hypergeometric Pictures

Hypergeomtric Pictures - Difference?

10
 Figure 7: Same n and p but t = 10,000

Example - Capture-recapture experiment

Ten fish have been captured, tagged, and released to mix into their popula-
tion. Suppose the population consists of 80 fish. A new sample of 15 animals
selected at random. What is the probability that 3 in the new sample will come
from the tagged?

Capture-recapture

Let X be the number of tagged animals in the new sample.

Then X has the hypergeometric distribution with t = 80, n = 15 and b = 10
Therefore
10 70



3 12
P (X = 3) = 80

= 0.1924
15

Calculations like this are used in practice. Unlike the code example, it is the
total population size, T , about which inference is needed. Bayes Theorem
can be used. Governments use these, or alternative frequentist, techniques for
monitoring population sizes and making policy decisions.

2 Ch 2.2 Mathematical Expectation

2.1 Mean and variance of data - Section 6.1
Example - Travel Time

My travel time to work varies according to whether I drive early in the morning,
at peak hour, get dropped at the station, take tram and bus at peak hour or
take tram and bus outside peak hour. In the last case, a typical travel time is
45 minutes by tram and bus but it can be as little as 20 minutes by car early
in the morning. Over a two week period, my travel times were 45 min. ± a
departure time (min.) given by the following table:
Table of Departures from 45 Minutes

11
 Day 1 2 3 4 5 6 7 8 9 10
Dep. (min.) +5 -5 -5 +5 +5 -25 +10 -25 -5 -20
Find the mean and empirical variance of the departures from my standard
travel time of 45 minutes.

Definitions

For data x1 , x2 , · · · , xn the Mean is x̄ given by

x1 + x2 + · · · + xn
x̄ = (4)
n

and the Empirical Variance is v given by

(x1 − x̄)2 + (x2 − x̄)2 + · · · + (xn − x̄)2

v= (5)
n

Solution - Travel Time

For our departures from the standard travel time of 40 minutes we have the
mean or average departure as
5 − 5 + · · · − 20
x̄ = = −6 (6)
10

and the Empirical Variance is v given by

(5 + 6)2 + (−5 + 6)2 + · · · + (−20 + 6)2

v= = 15.4 (7)
10

2.2 Definition of Expectation of RV

Definition of Expectation

Random Selection Consider the experiment of randomly selecting one departure

time out of the 10 with equal chance for each of the 10 days.
Sample Space The sample space is 1, 2, ..., 10 and let X be the random variable
which records the departure from 45 minutes for the randomly
selected day.
Mean in Ex. The number x̄ = −6 is then called the Expectation or Mean of the
random variable X.
General In general, for any random variable X which is discrete
X
E(X) = X(s)P ({s}) (8)
s∈S

12
Definition of Expectation
Repetitions In computing the sum, we could collect together all the outcomes s
which have the same value of X(s)
Travel Times there were three repetitions of -5, three reps. of +5
and 2 reps. of -25
This gives
X
E(X) = xP (X = x) (9)
x∈range(X)

2.3 Game Example

Example - Game
In a simple game, a die is rolled and 1¢ is paid if the roll of the die gives 1,2
or 3, 5¢ for a roll of 4 or 5 and 35¢ for a 6. What is a fair charge for this game?

Solution - Game
Definition Let Y be the random variable which gives the prize one time I play
the game.
Long Run Frequencies Over many plays of the game, the prize will be y¢ (y =
1, 5 or 35) with approximate fraction P (Y = y) of the
time.
Long Run Gain Hence, adding over a long sequence of games, the average amount
of money (in ¢) I win will get closer to
1 × P (Y = 1) + 5 × P (Y = 5) + 35 × P (Y = 35) = E(Y )

Logical Fair Charge is this average amount of money

Solution - Game Ctd

PMF
3 2 1
P (Y = 1) = , P (Y = 5) = , P (Y = 35) =
6 6 6
Answer
3 2 1
E(Y ) = 1 × + 5 × + 35 × = 8
6 6 6
ie a fair charge is 8c.

Comments - Game
Actual electronic games machines operate like this with more complicated chance
mechanisms and more elaborate prize scales.
Legislation Often limits the charge to the expected return (that is the expecta-
tion just calculated) plus a defined profit margin.
Example In New South Wales clubs, the return is legislated to be at least 85%
of the money input.

13
 X(s) Y(s)
X u
roll 6 6 35
roll 5 5
roll 4 4
5
roll 3 3 1
roll 2 2
roll 1 1

S Y direct

Figure 8: Y is a composition of u with X

2.4 Expectation of a Transform

Game - Another Solution

Start with a different random variable, X, which is the number on the die so
that X has probability mass function P (X = x) = 16 for x = 1, 2, · · · , 6.
RV Y is obtained by applying the function u to X so that

X = u(Y )

where
u(1) = u(2) = u(3) = 1, u(4) = u(5) = 5, u(6) = 35

Diagram of X = u(Y)

Expectation of X
This can be equally found as:
X
E(Y ) = Y (s)P ({s}) (10)
s∈S
6
X
= u(x)P (X = x) (11)
x=1
X
= yP (Y = y) (12)
y=1,5or35

Different Levels of Agreggation

Aggregation at three levels in the three different ways in equations (10), (11),
(12) to compute E(Y ).

14
1 to 1 In this example, there was no aggregation from the sample space S to the
range of X - one to one correspondence between outcomes in the sample
space and results for Y .

But would not have had this if the sample space had two throws of the dice and
X recorded the outcome of the first throw - the 36 sample points aggregated
to 6 outcomes for X.
Aggregation from 6 outcomes in range of X via the function u to 3 prize values
for Y .

Different Levels of Agreggation

Right Hand Sides the same in equations (10), (11), (12) by the distributive law

Textbook uses the second one, (11), as the definition

PMF X known means the second one, (11), is a good choice
Rigorous maths, covering all kinds of random variables, starts with (10) for
random variable with finitely many outcomes.

Need X
E(|u(X)|) = |u(x)|P (X = x) < ∞
x

to be safe

3 Ch 2.3 Special Mathematical Expectations

3.1 Variance of a RV
Definition of Variance of a RV

Travel Time Example had Empirical Variance defined as the average of squared
deviations around the mean:
10
X 1
v= (xi − x̄)2 ×
x=1
n

Mean of RV is number E(X) = µ (say), then µ is just some number depending

on PMF of X - see (9)

Squared dev’s from µ are measured by the function u(x) = (x − µ)2

RV analogue of Empirical Variance is the Variance, often written σ 2 , of the
RV defined by

2 
V ar(X) = E (X − µ)2 = E X − E(X)
  
(13)
X
= (x − µ)2 P (X = x) (14)
x∈range(X)

15
Example - Variance of Game
Find the variance of the random variable Y in the simple game with the die

Solution - Variance of Game

From the previous example, we found E(Y ) = 8

So
3 2 1
V ar(Y ) = (1 − 8)2 × + (5 − 8)2 × + (35 − 8)2 × = 149
6 6 6

Expectation and Variance in one pass

Problem is that µ = E(X) was needed to define the function u and hence to
calculate the variance of the random variable
Needed are some properties of expectation that will enable the calculation of
variance at the same time as the calculation of the mean

4 Ch 2.2 - Back to Mathematical Expectation

4.1 Expectation Linear ie Distributive
Properties of Expectation

Constant If c is a constant, then E(c) = c

Multiplication If c is a constant and X is a random variable, then

E(cX) = cE(X) (15)

Addition If X and Y are random variables then

E(X + Y ) = E(X) + E(Y ) (16)

Linear If c1 , c2 , · · · , cn are constants and X1 , X2 , · · · , Xn are random variables,

then
Xn n

X
E ci Xi = ci E(Xi ) (17)
i=1 i=1

Demonstration

Constant Using the definition of expectation in (8), the fact that probabilities
of sample points add to one and the distributive law for numbers,
X X
E(c) = cP ({s}) = c P ({s}) = c
s∈S s∈S

Multiplication Using the distributive law again

X X
E(cX) = cX(s)P ({s}) = c X(s)P ({s}) = cE(X)
s∈S s∈S

16
Demonstration

Addition Using the definition of expectation in (8), and the distributive, com-
mutative and associative laws for numbers
X
E(X + Y ) = (X(s) + Y (s))P ({s})
s∈S
X X
= X(s)P ({s}) + Y (s)P ({s})
s∈S s∈S
= E(X) + E(Y )

Linear Follows from induction using the case n=2 and both the addition and
multiplication properties in order:

E(c1 X1 + c2 X2 ) = E(c1 X1 ) + E(c2 X2 )

= c1 E(X1 ) + c2 E(X2 )

4.2 Application to Variance

Demonstration
Using the definition of variance in (13), expanding the quadratic for each
sample outcome,recalling that µ = E(X) and using the Linearity property of
expectation in (17)

V ar(X) = E (X − µ)2
 

= E X 2 − 2µX + µ2
 

= E(X 2 ) − 2µE(X) + µ2
2
= E(X 2 ) − E(X)

(18)

4.3 Example - Variance of Triangular PMF

Example - Variance of Triangular PMF
Find the variance of a random variable which has probability mass function
f given by 
1/9 x = 1 or x = 5

f (x) = 2/9 x = 2 or x = 4

3/9 x = 3


Example - Triangular PMF

Solution - Variance of Triangular PMF

Mean
1 2 3
E(X) = (1 + 5) × + (2 + 4) × + 3 × = 3
9 9 9
Intuitive because the PMF is symmetric about 3

17
 pmf, f(x)
0.35
●
0.30

0.25
● ●
0.20

0.15

● ●
0.10

0.05

x
1 2 3 4 5

Figure 9: Find Mean and Variance for PMF

Mean Square
1 2 3 1
E(X 2 ) = (1 + 52 ) × + (22 + 42 ) × + 32 × = 10
9 9 9 3
Variance
2 1 1
V ar(X) = E(X 2 ) − E(X) = 10 − 9 = 1

3 3

4.4 Example - Minimum Squared Deviation

Example - Mean Minimises Squared Deviations 
Find the value of the number b which minimises E (X − b)2

Solution - Mean Minimises Squared Deviations

No calculus unlike your text Example 2.2-4

Method Guess and verify by the ”put in what you want and then make the
necessary correction” method

µ? Using the three properties of expectation in an extension of the argument

used to find

E (X − b)2 = E ((X − µ) − (b − µ))2

   

= E ((X − µ)2 − 2(b − µ))E (X − µ) + (b − µ)2

   

= E (X − µ)2 + (b − µ)2
 

Because the middle, cross-product term, is 0 : µ = E(X)!

RHS is ≥ V ar(X) since the first term is the variance and the second is ≥ 0 and
0 if, and only if, b = µ

18
5 Ch 2.2 - Back to Mathematical Expectation
5.1 Example - Minimum Squared Deviation
Example - Mean Minimises Squared Deviations 
Find the value of the number b which minimises E (X − b)2

Solution - Mean Minimises Squared Deviations

No calculus unlike your text Example 2.2-4

Method Guess and verify by the ”put in what you want and then make the
necessary correction” method

µ? Using the three properties of expectation in an extension of the argument

used to find

E (X − b)2 = E ((X − µ) − (b − µ))2

   

= E ((X − µ)2 − 2(b − µ))E ((X − µ) + (b − µ)2

   

= E ((X − µ)2 + (b − µ)2

 

Because the middle, cross-product term, is 0 : µ = E(X)!

RHS is ≥ V ar(X) since the first term is the variance and the second is ≥ 0 and
0 if, and only if, b = µ

Comment - Mean Minimises Squared Deviations

For data, descriptive statistics likes to make a numerical summary of all the
values: data value = summary + residual
For data x1 , x2 , · · · , xn and any number b, this is xi = b + (xi − b)
Criterion: b should be chosen to minimise the residuals in total, for example to
minimise the distance of the residuals from the origin

Euclidean Distance of the residuals from the origin is

v
u n
p uX
sum of squares of residuals = t (xi − b)2
i=1

Euclidean Distance Reminder

Comment - Mean Minimises Squared Deviations

Particular case of the Example, is the random variable,X, that chooses one of
the xi at random. The expected value of (X − b)2 , is related to the
Euclidean distance by
p
Euclidean distance = n ∗ E(X − b)2

19
 y

Euclidean Distance
( x2 , y2 )

(x − x ) +(y − y )
2 2

y2 − y1
2 1 2 1

( x1, y1 ) x2 − x1

Figure 10: Pythagoras Theorem gives Euclidean Distance

x Data, x, Summary, b, and residuals, arrowed

x1
b

xn

x2

1 2 !!!!! n i

Figure 11: Data, Summary and Residuals

Hence Euclidean distance of the residuals from the origin is minimised by choos-
ing b = x̄

Analogy for a general random variable is that b = µ = E(X) mnimises a

weighted Euclidean distance of the residuals from the origin with the
weights proportional to the probabilities of the values of X.

Data = Summary + Residuals

Standard Deviation (Ch 2.3)

Euclidean Distance takes the square root so that it is in the same units as the
measurements.
Standard Deviation of a random variable is the square root of V ar(X), often
denoted σ.

20
5.2 Example - Sampling with and without replacement
Example - Mean Number with a Disease
Find the mean number of people with a particular disease in a random
sample taken from a population with or without replacement. Assume that the
sample is size n and that the population has a proportion p of people with the
disease.

Solution - Mean Number with a Disease

Definition Let Xi , (i = 1, 2, · · · , n) be 1 if the ith person sampled has the disease

and 0 otherwise
Sample number X records the number in the sample who have the disease
n
X
X= Xi
i=1

Linearity of expectation gives

n
X
E(X) = E(Xi )
i=1

With Replacement makes the sample outcomes - diseased or not - the results of
Bernoulli trials with constant probability p.

Solution Mean Number with a Disease Ctd

So The answer for sampling with replacement is np = nE(X1 ) = n(1 × p + 0 ×

(1 − p)).
Without Replacement: perhaps not so clear, but true, that E(Xi ) = p, i =
1, 2, · · · , n, so that the answer for the mean number in
the sample with the disease is still np
Reason is that if the population size is t, we can imagine random orderings of
all members of the population with equally likely outcomes for each
random ordering
Without replacement sample is then taken as the first n members of the random
ordering of the whole population

Solution Mean Number with a Disease Ctd 2

Then using the fact that the orderings are equally likely, defining Xi (i = 1, 2, · · · , n)
to be 1 if the ith person in the sample has the disease and supposing there
are b out of the t people in the population with the disease

21
 number of outcomes in [Xi = 1]
P (Xi = 1) =
total number of outcomes
b × (t − 1)!
=
t!
b
= =p
t
since whichever of the b people with the disease are chosen in position i in the
random ordering, there are (t − 1)! orderings for the other t − 1 positions in the
ordering.

6 Back to Ch 2.3 Special Mathematical Expec-

tations
6.1 Moment Generating Function
Definition of Moment Generating Function
Suppose X is a random variable
Then the Moment Generating Function M (t) for a number t is defined as
M (t) = E(etX )
X
= etx P (X = x)
x∈range(X)
X
= etx f (x)
x∈range(X)

where f (x) is the probability mass function of X.

May not exist but does at t = 0 and usually does for −h < t < h for some h > 0

6.2 Example - Moment Generating Function from PMF

Example - MGF of Triangular Random Variable
Find the moment generating function, and its derivative, for the random
variable X whose probability mass function is the triangular one:

1/9 x = 1 or x = 5

f (x) = 2/9 x = 2 or x = 4

3/9 x = 3


What is the value of the derivative at 0? Interpret for X.

Solution - MGF of Triangular Random Variable

MGF
M (t) = E(etX )
= 1/9 et + 2/9 e2t + 3/9 e3t + 2/9 e4t + 1/9 e5t

22
Derivative M 0 (t) = 1/9 et + 2 × 2/9 e2t + 3 × 3/9 e3t + 4 × 2/9 e4t + 5 × 1/9 e5t
So M 0 (0) = 1/9 + 2 × 2/9 + 3 × 3/9 + 4 × 2/9 + 5 × 1/9 = 3

Interpretation: a complicated way to find E(X)!

6.3 Example - Moments from MGF

Why the name?

MGF M of rv X satisfies M (t) = E(etX ). Differentiating with respect to t,

under the sum which is the expectation, gives

M 0 (t) = E(XetX )

Putting t = 0 gives

E(X) = M 0 (0) (19)

Differentiating again

Gives
E(X 2 ) = M 00 (0) (20)

So

V ar(X) = E(X 2 ) − [E(X)]2

= M 00 (0) − [M 0 (0)]2 (21)

In general

Differentiating k times k = 0, 1, 2, · · ·

E(X k ) = M (k) (0) (22)

where the brackets (k) in the numerator indicates the kth deriva-
tive

Note

1 = E(X 0 ) = M (t)

Example - Finding Moments from MGF

Find the mean and variance of a random variable whose moment generating
function is M (t) = (q + pet )n where 0 < p < 1, q = 1 − p and n is a positive
integer

23
Solution - Finding Moments from MGF

Derivative
M 0 (t) = pet × n(q + pet )n−1

Mean E(X) = M 0 (0) = np

2nd Deriv

M 00 (t) = pet × n(q + pet )n−1 + p2 e2t × n(n − 1)(q + pet )n−2

So V ar(X) = M 00 (0) − [M 0 (0)]2 = np + n(n − 1)p2 − (np)2 = np(1 − p)

6.4 PMF from MGF

PMF from MGF

In general if the MGF exists −h < t < h, h > 0, then the PMF can be found
from the MGF
If it is known that there are (say) 5 values in the range of the rv, then evaluating
the MGF at 5 different points will give 5 simultaneous equations in the 5
unknown probabilities that can be solved

More complicated for other random variables, for example if there are infinitely
many possible values
However, the mgf is always good for finding expectations of the random variable
to all powers can be found by repeated differentiation

Moments Tell About Shape

Mean gives the central value for the pmf

Variance gives the extent of spread of the pmf around the mean
X−µ
Standardised random variable is σ where µ is the mean and σ is the standard
deviation
Skewness is the third moment for the standardised random variable - pmf skewed
to right or left
Kurtosis is the fourth moment for the standardised random variable - pmf flat
or peaked

Example - Finding Moments About PMF

24
 pmf(k)
Skewness = -0.2
0.12 Kurtosis = 3.0

0.10

0.08

0.06

0.04

0.02

k
10 20 30 40

Figure 12: One example - Mathematica Manipulate in Next Week’s Lab

7 Ch 4.1 - Independent Random Variables

7.1 Indep’t RVs - Var, MGF of Sum
Definition and Main Property, Variance and MGF
Definition Random variables S and Y are independent, if, for any sets of values
B1 , B2 , [X ∈ B1 ] and [Y ∈ B2 ] are independent events, that is
P (X ∈ B1 ∩ Y ∈ B2 ) = P (X ∈ B1 )P (Y ∈ B2 )

Extension to independent random variables X1 , X2 , · · · , Xn requires for any sets

of values B1 , B2 , · · · , Bn , the events [Xi ∈ Bi ], i = 1, 2, · · · , n are
independent
Main property (which will be shown in Module 4)
E(X1 X2 · · · Xn ) = E(X1 )E(X2 ) · · · E(Xn ) (23)

Useful If X1 , X2 , · · · , Xn are independent, so are f1 (X1 ), f2 (X2 ), · · · , fn (Xn ) for

any functions f1 , f2 , · · · fn

Application to Variance of a Sum

If X and Y are independent random variables, then
V ar(X + Y ) = V ar(X) + V ar(Y ) (24)

Demonstration
By linearity of expectation, the calculation formula for variance and the
main property for independent random variables
V ar(X + Y ) = E[(X + Y )2 ] − (E(X) + E(Y )))2
= E[X 2 + Y 2 + 2XY ] − (E(X) + E(Y ))2
= V ar(X) + V ar(Y ) + 2E(XY ) − 2E(X)E(Y ) (25)
= V ar(X) + V ar(Y )
just like Pythagoras’ theorem.

25
Variance of sum for non-independent random variables
Can be calculated using using equation (25) on the last slide since independence
only enters in using the Main Property (equation (23)) for independent
random variables
This will be discussed further in Module 4.
The variance for the number of successes in sampling without replacement is an
example - see later slides for pictures of the pmf in this case.

7.2 MFG of a Sum

Application to MGF of a Sum
If X and Y are independent random variables, then so are etX and etY , so
MX+Y (t) = E(et(X+Y ) )
= E(etX etY )
= E(etX )E(etY )
= MX (t)MY (t) (26)

7.3 Properties of Variance

Analogy to Expectation
Constant If a is a constant, then V ar(a) = 0
Multiplication If a is a constant and X is a random variable, then
V ar(aX) = a2 V ar(X)

Addition If a, b are constants and X, Y are independent random variables, then

V ar(aX + bY ) = a2 V ar(X) + b2 V ar(Y )

Linear Comb If a1 , a2 , · · · , an are constants and X1 , X2 , · · · , Xn are independent

random variables, then
Xn  X n
V ar ai Xi = a2i V ar(Xi )
i=1 i=1

Sample Mean of Independent RVs - All Same Distribution

So If X1 , X2 , · · · , Xn are independent random variables each with the same pmf,
then
 Pn 
i=1 Xi
V ar(X̄) = V ar
n
Xn
= n−2 V ar(Xi )
i=1
= n−2 × n × V ar(X1 )
V ar(X1 )
= (27)
n

26
 As n gets large, the variance of the sample mean gets small ie most values are
concentrated around the population mean, µ = E(X)

7.4 Properties of Sample Mean

Any RVs - All Same Distribution
Using linearity of expectation (as in the example for the mean number having
a disease)
 Pn 
i=1 Xi
E(X̄) = E
n
= n−1 × n × E(X1 )
= E(X1 ) (28)

Summary: Moments of Sample Mean

√
If the mean and standard deviation in the population are µ and σ = V ariance,
then the sample mean X̄ of n independent and identically distributed random
variables from this population has expectation µ and standard deviation √σn .

This is why the amount of information collected increases generally with the
square root of the sample size
In Module 5 we’ll study this relationship with sample size more
These connections are basic tools in practical statistical data science

7.5 Sample Proportion

Example - Sample Proportion
Find the mean, variance and standard deviation of the random variable
which gives the sample proportion in a random sample taken intending to vote
Liberal at the next election if the sample is taken with replacement and is
size 2000. Assume the population proportion intending to vote Liberal is (an
unknown number) p. Interpret. If the sample is taken without replacement,
when will these answers give an accurate guide?

Solution - Sample Proportion

The sample proportion is X̄ for the random variables X1 , X2 , · · · , X2000 with
Xi = 1 if the ith person in the sample intends to vote Liberal and Xi = 0
otherwise.
So using equation (28), the expectation or mean of the random variable which
gives the sample proportion is p, the population proportion intending to vote
Liberal, since E(X1 ) = 1 × p + 0 × (1 − p).
Also this is true with or without replacement in the sample
Interpretation: Repeated observations of the sample proportion - for example,
repeated opinion polls (provided the population proportion of
Liberal intending voters is not changing) - will centre around p,
the population proportion.

27
 pmf(k)
D = 0.00
0.015

Out[253]= 0.010 Without Repl.

With Repl.

0.005
k
900 950 1000 1050
Figure 13: Electorate=16m, p= 0.48 N ewsP oll 20 M arch 2017

Solution - Sample Proportion

Using (27) for random samples with replacement, the sampleq

proportion random
p(1−p) p(1−p)
variable has variance 2000 and standard deviation 2000 .

Interpretation is that the spread of the distribution for the sample mean is small
in a sample of 2000, since the standard deviation is maximised
when p = 12 and is then 0.011.
When the sample is taken without replacement, the expectation is exact.
Further the variance and standard deviation will be approximately right since
the population size of enrolled voters is about 16 million and this is
much larger than the sample size.

PMF - With & Without Replacement

Definition D in the pictures of the pmf’s with and without replacement is the
maximum difference in any probability computed with or without
replacement
Plot shows no difference for any probability in sampling with or without re-
placement for a population as large as 16 million, the size of the Australian
Electorate, when the sample size is 2000

On the other hand if the total size were just 2100 then there is a big difference
in probabilities with and without replacement, as the figure
shows:

28
 pmf(k)
0.08 D = 0.62
0.06
Without Repl.
Out[217]=
0.04 With Repl.

0.02
k
900 950 1000 1050
Figure 14: Pop Size=2,100, p= 0.48 N ewsP oll 20 M arch 2017

Discussion - With & Without Replacement

Shape very similar in both population sizes, 16 million and 2,100

Mean the same
Variance much smaller for 2100
What is the variance for population size 2000? See Module 4 for variance of
hypergeometric
Figures 15 & 16 show the PMFs with and without replacement for pop’n sizes
3,100 and 10,100

29
 pmf(k)
0.030 D = 0.25
0.025
0.020 Without Repl.
Out[229]=
0.015 With Repl.
0.010
0.005
k
900 950 1000 1050
Figure 15: Pop Size=3,100, p= 0.48 N ewsP oll 20 M arch 2017

pmf(k)
0.020 D = 0.05
0.015
Without Repl.
Out[241]=
0.010 With Repl.

0.005
k
900 950 1000 1050
Figure 16: Pop Size=10,100, p= 0.48 N ewsP oll 20 M arch 2017

PMF Shape Only

Standardized random variable, Z, is defined as

X −µ
σ
for any random variable X with E(X) = µ and V ar(X) = σ 2

Z takes value 0 if X = µ, value ±i, i = 1, 2, · · · if X = µ ± iσ, so standardiz-

ing changes units for X to centre at the mean and scale with the standard
deviation
Reason for standardizing will become clearer in Module 5, but showing the pmf
for the standardized random variable removes any differences between
random variables in both mean and variance

30
 pmf(z)
0.4

0.3
With Repl.
W'out Pop=2,100
Out[13]=
0.2 W'out Pop=3,100
W'out Pop=10,100
W'out Pop=16m
0.1

z
-3 -2 -1 1 2 3

Figure 17: Scaled pmf for p=0.48, with and w’out replacement, Pop Size varying

Because the units on the z axis increase by the reciprocal of the standard de-
viation, to preserve the probabilities adding to one, it is necessary to
scale the pmf by the standard deviation

Next figure shows the pmf for the standardized sample number with replacement,
and without replacement for population sizes 2,100, 3,100, 10,100 and
16,000,000

8 Ch 2.4 - The Binomial Distribution

8.1 Recall PMF from Module 2 1
Recall Bernoulli Trials from Module 2 1

Bernoulli Trials a number, n, of independent trials

Two Outcomes Each trial has one of two possible outcomes success or failure
denoted S and F
Binomial RV Consider the rv, X, which is the number of trials that result in
success

Component RVs For i = 1, 2, · · · , n, let Xi be the rv which is 1 if the ith trial

is a success and 0 on Aci , so that Xi is 1 for a success on trial
i and 0 for a failure on trial i
Now The number of successes overall is X and this is the sum of the number of
successes on each trial so
n
X
X = X1 + X2 + · · · + Xn = Xi (29)
i=1

31
PMF of Binomial RV

Probability of Success Let p be the constant probability of success on each trial

In Module 2 1 we showed that for x = 0, 1, 2, · · · , n
 
n x
P (X = x) = p (1 − p)n−x
x

8.2 Binomial Mean and Variance

Mean and Variance of Binomial RV

Follow , by the representation, (29) from the properties of the Sample Mean of
Independent Random Variables with the same Distribution

So for a Binomial random variable, X with parameters n (= 1, 2, · · · ) and p (0 <

p < 1)
E(X) = np

And
V ar(X) = np(1 − p)

8.3 Sampling with Replacement

Sampling with Replacement
is a special case and the number in the sample has Binomial distribution
with n equal to the sample size and p equal to the population proportion of
”successes”. The number with a disease and sample proportion voting Liberal
were examples of the Binomial distribution.

8.4 MGF for Binomial

MGF for Binomial

Bernoulli random variable X1 has MGF M1 given by

M1 (t) = E(etX1 )
= et×1 P (X1 = 1) + et×0 P (X1 = 0)
= (pet + q)

where q = 1 − p
MGF for sum of independent rv’s is product of MGF’s
So

M (t) = E(etX )
= M1 (t)M2 (t) · · · Mn (t)
= (pet + q)n

32
9 Ch 2.5 - The Negative Binomial Distribution
9.1 PMF for Time till Success
Time till First Success in Bernoulli Trials

Suppose X1 , X2 , · · · are random variables which record 1 on successes and 0 on

failures in an indefinitely long sequence of Bernoulli trials
So the X random variables have P (Xi ) = E(Xi ) = p for each i = 1, 2, · · · and
they are independent
Let T1 be the random variable which counts the number of trials until a success
occurs
Note on Time/Trial Number: trials number is often referred to as ”time” which
may, or may not, correspond to actual time de-
pending on the context
Then the probability mass function for T is given by

P (T = i) = P (X1 = 0, Xi−1 = 0, · · · , Xi = 1)
= P (F · · · F S)
= (1 − p)i−1 p, i = 1, 2, · · · (30)

Time till rth Success in Bernoulli Trials

Argument is similar for Tr , r = 1, 2, · · · which is the number of trials till the rth
success
Any pattern which has Tr = i has exactly r successes in the i trials with the rest
failures. Also, trial i must be a success for Tr = i. Otherwise, the trials for
the other r − 1 successes in the first i − 1 trials can be chosen freely.
i−1


There are r−1 ways to choose the trial numbers for the successes
So  
i−1 r
P (Tr = i) = p (1 − p)i−r i = r, r + 1, · · ·
r−1

Geometric and Negative Binomial

Geometric is the name for the distribution for the time to the first succcess in
Bernoulli trials
There is just one number p, the probability of a success on a trial, that deter-
mines the pmf
Negative Binomial is the name for the distribution of the time to the rth success
Negative Binomial has two numbers, r and p which determine the pmf

9.2 Examples of PMF for Negative Binomial Distribution

33
 ● p=0.9 ■ p=0.5 ◆ p=0.2 ▲ p=0.1
pmf(k)
●

0.8

0.6
■

0.4
■

0.2◆ ◆
◆
■
▲ ▲
● ◆
▲ ▲ ◆
▲
■ ◆
▲ ▲
◆ ▲
◆ ▲ ▲ ▲
■ ◆ ▲ ▲
k
■ ◆ ◆ ◆ ◆ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲
● ● ● ● ■
● ■
● ■
● ■
● ■
● ■
● ■
● ◆
■
● ◆
■
● ◆
■
● ◆
■
● ◆
■
● ◆
■
● ◆
■
● ◆
■
● ◆
■
● ▲
◆
■
● ▲
◆
■
● ▲
◆
■
●

5 10 15 20 25

Figure 18: Prob Mass Functions for Geometric

● p=0.9 ■ p=0.5 ◆ p=0.2 ▲ p=0.1

pmf(k)
0.8 ●

0.6

0.4

■ ■

0.2 ●
■

◆ ◆ ◆
■ ◆ ◆
◆ ◆ ◆ ◆
◆ ▲
■
▲ ▲ ▲ ▲ ▲ ◆
▲ ◆
▲ ▲
◆ ▲ ▲ ▲ ▲
▲ ▲ ■ ◆ ◆ ▲ ▲ ▲ ▲
k
▲ ● ■ ◆ ◆ ◆ ◆ ▲ ▲ ▲ ▲
▲ ● ● ● ● ● ■
● ■
● ■
● ■
● ■
● ■
● ■
● ■
● ■
● ■
● ■
● ◆
■
● ◆
■
● ◆
■
● ◆
■
● ◆
■
● ◆
■
●

5 10 15 20 25

Figure 19: PMFs for Trial Number of 2nd Success in Bernoulli Trials

34
 ● p=0.9 ■ p=0.5 ◆ p=0.2 ▲ p=0.1
pmf(k)
0.35 ●●

0.30
0.25
0.20 ●

0.15
0.10 ■■
■■
●■ ■
■
■
0.05
■
■ ■
■
■ ◆◆◆◆◆◆◆◆◆◆◆
■● ■ ◆ ◆◆◆◆ ◆◆◆◆◆
■■ ◆◆◆◆ ◆◆
◆◆◆ ◆◆◆▲◆
■ ◆■◆ ▲▲▲▲◆▲◆ ▲▲▲▲▲▲▲▲▲▲▲▲▲▲▲▲▲▲▲▲▲▲▲▲▲▲▲▲▲▲▲▲▲▲▲▲▲▲▲▲▲▲▲▲▲▲▲▲▲▲▲▲▲▲▲▲▲▲▲▲▲
▲◆
▲◆
k
● ◆◆
◆ ■■■■ ▲▲ ▲◆▲◆
▲◆ ▲◆
▲◆
▲
◆
■
■◆
▲◆
▲◆
▲◆
▲◆●
◆
▲◆
▲
▲◆
●
▲
●◆
◆
▲
●◆
▲
▲
●◆
▲
●
●◆
◆
▲
●◆
▲
▲
●◆
▲
●
●▲
●▲
●▲
▲
●●▲
●▲
●▲
●▲
▲
●■▲
●■▲
●●■▲
■▲
●●■▲
■▲
●●■▲
■▲
●●■▲
■▲
●●●■▲
■▲
■▲●●■▲
■▲
●●■▲
■▲
●●■▲
■▲
●●■▲
■▲
●●■▲
■▲
●■●
●■●
■●
■●
■●
■●
■●
■■
●■
●■
●■
●■
●■
●■
●■■◆
●●■◆
●■◆
●■◆
●■◆
●■◆
●■◆
●■◆
●■◆
●■◆
●■◆
●■◆
●■◆
●■◆
●■◆
●■
●◆
■
●◆
■
●◆
■
●◆
■
●◆
■
●◆
■◆
●■◆
●■◆
●■◆
●■◆
●■◆
●■◆
●■◆
●■◆
●■◆
●■◆
●■◆
●■◆
●■◆
●■◆
●■◆
●■◆
●■
●◆
■
●◆
■
●◆
■
●◆
■
●◆
■
●◆
■◆
●■◆
●■◆
●■◆
●■◆
●■◆
●■◆
●■◆
●■◆
●■◆
●■◆
●■◆
●■◆
●■◆
●■◆
●■◆
●●▲
▲
■◆
◆
■
●▲
◆
■
■
●
●▲
◆
■
●▲
▲
◆
◆
■
●▲
◆
■
■
●
●▲
◆
■
●▲
◆
●▲
■◆
●▲
■◆
●▲
■◆
●▲
■◆
●▲
■◆
●▲
■◆
●▲
■◆
●▲
■◆
●▲
■◆
●▲
■◆
●▲
■◆
●▲
■◆
●▲
■◆
●▲
■◆
●▲
▲
■◆
◆
■
●▲
◆
■
■
●
●▲
◆
■
●▲
▲
◆
◆
■
●▲
◆
■
■
●
●▲
◆
■
●▲
▲
◆
◆
■
●▲
◆
■
■
●
●▲
◆
■
●▲
◆
●▲
■◆
●▲
■◆
●▲
■◆
●▲
■◆
●▲
■◆
●▲
■◆
●▲
■◆
●▲
■◆
●▲
■◆
●▲
■◆
●▲
▲
■◆
◆
■
●▲
◆
■
■
●
●▲
◆
●▲
■◆
●▲
■◆
●▲
■◆▲
■◆
● ▲
■◆
● ▲
■◆
● ▲
■◆
● ▲
■◆
● ▲
■◆
● ▲
■◆
● ▲
■◆
● ▲
■◆
● ▲
■◆
● ▲
■◆
● ▲
■◆
● ▲
■◆
● ▲
■◆
● ▲
■◆
● ▲
■◆
● ▲
▲
■◆
●◆
■
●▲
◆
■
■
●
●▲
◆▲
■◆
● ▲
■◆
●■
●

50 100 150 200

Figure 20: PMFs for Trial Number of 10th Success in Bernoulli Trials

To see shapes
Need to standardize as with Binomial and Hypergeometric pictures. Scale
X −µ
to rv Z = and scale the pmf of this so the area under the curve is 1.
σ
Figures 23, 24 and 25 show the shapes by displaying the scaled pmf for Z.

Conclusion

For p=0.1 and p=0.5, as r gets larger see same shape as for Binomial and
Hypergeometric
For p=0.9, the successes come more quickly and the shape is not so clear

35
 p=0.9 p=0.7 p=0.5
pmf(k)

0.15

0.10

0.05

k
60 80 100 120

Figure 21: PMFs for Trial Number of 50th Success in Bernoulli Trials, High
Probs

p=0.2 p=0.1
pmf(k)
0.012
0.010
0.008
0.006
0.004
0.002
k
300 400 500 600 700

Figure 22: PMFs for Trial Number of 50th Success in Bernoulli Trials, Low
Probs

36
 n=10 n=50 n=100
scaled pmf(z)
0.4

0.3

0.2

0.1

z
-3 -2 -1 1 2 3

Figure 23: Standardized Time till rth success, p=0.1, r=10,50,100

n=10 n=50 n=100

scaled pmf(z)
0.4

0.3

0.2

0.1

z
-3 -2 -1 1 2 3

Figure 24: Standardized Time till rth success, p=0.5, n=10,50,100

37
 n=10 n=50 n=100
scaled pmf(z)
0.4

0.3

0.2

0.1

z
-3 -2 -1 1 2 3

Figure 25: Standardized Time till rth success, p=0.9, r=10,50,100

Note Skewness is evident for r = 10

Also There is some probability for |Z| > 3

Why the name Negative Binomial?

Binomial formula for positive powers simlarly demonstrates binomial probabil-

ities add to one:
n  
n
X n x n−x
1 = (p + q) = p q (31)
x=0
x

Binomial formula for negative powers: for −1 < w < 1

∞  
−r
X x−1
(1 − w) = wx−r (32)
x=r
r−1

So the sum of the pmf of the Negative Binomial is, putting w = 1 − p,

∞  
X x−1
pr (1 − p)x−r = pr (1 − (1 − p))−r = 1
x=r
r−1

9.3 MGF and Moments for Negative Binomial

Independence of Times Between Successes

Trial number of rth success is the sum of the times between the first r > 1
successes:
Tr = T1 + (T2 − T1 ) + · · · + (Tr − Tr−1 ) (33)

Notation ”iFS” means i failures followed by a success

38
And the random variables in the sum are independent and each distributed as
the Geometric(p) distribution
Because building on the case r = 2 by induction, for i, j ≥ 1, letting q = 1 − p
and using independence of the Bernoulli trials,
P (T1 = i ∩ T2 − T1 = j) = P (T1 = i)P (T2 − T1 = j|T1 = i)
= P ((i − 1)F S)P ((j − 1)F S|(i − 1)F S)
= q i−1 pq j−1 p
= P (T1 = i)P (T1 = j)

MGF for Geometric

Geometric Using (32) in the case r = 1 (the Geometric series)
M1 (t) = E(etT1 )
∞
X
= eti P (T1 = i)
i=1
∞
X
= eti q i−1 p
i=1
∞
X
= pet (et q)i−1
i=1
pet
= (34)
1 − qet
MGF for Negative Binomial
In General the MGF of a sum of independent random variables is product of
MGF’s and from the independence of the random variables in (33) :
Mr (t) = E(etTr )
= M1r (t)
r
pet

= (35)
1 − qet

Moments for Negative Binomial

Geometric
1
E(T1 ) = M10 (0) =
p
1−p
V ar(T1 ) = M100 (0) − (E(T1 ))2 = (36)
p2

General case follows from the fact that the random variables in (33) are inde-
pendent so:
r
E(Tr ) = rE(T1 ) =
p
r(1 − p)
V ar(Tr ) = rV ar(T1 ) = (37)
p2

39
Mathematica and R
Both use Tr −r, the number of failures before the rth success, rather than Tr
- affects values of times and means, but not variances and standard deviations,
since for any constant a and rv X with mean µX :

V ar(X + a) = E (X + a − µX+a )2

= E (X + a − (µX + a))2

= V ar(X) (38)

9.4 Example - Accidents in a Workplace

Example - Accidents in a Workplace
In a workplace, there is a probability of 0.05 that there is an accident in any
month. The number of accidents in successive months are independent.
1. What is the chance that there are no accidents in a year?
2. What is the chance that the 4th accident occurs more than ten years from
now?

3. What is the mean and standard deviation of the time in years to the
(a) first
(b) fourth

accident?

Solution - Accidents in a Workplace

Let Tr be the time to the rth accident and Xt be the number of acci-
dents in t years, so that Tr has Negative Binomial (r,0.05) pmf and Xn has
Binomial(n,0.05) pmf. Hence
1. the chance that there are no accidents in a year is P (T1 > 12) = (1 −
0.05)12 = 0.54036

2. the chance that the 4th accident occurs than 10 years from now is

P (T4 > 120) = P (X120 ≤ 3)

= (1 − 0.05)120 + 120 × (1 − 0.05)119 × 0.05
120 × 119
+ (1 − 0.05)118 × 0.052
2
120 × 119 × 118
+ (1 − 0.05)117 × 0.053
3×2
= 0.131707

40
Solution - Accidents in a Workplace Ctd

3. the mean and standard deviation of the time in years, by linearity prop-
erties of expectation, the mean and standard deviation divided by 12.
Hence
(a) the mean and standard deviation of the time
√ in years to√ the first
1 2 V ar(T1 ) 1−0.05
accident is E(T1 )/12 = 12×0.05 = 1 3 and 12 = 12×0.05 =
1.62447
(b) the mean and standard deviation of the time
√ in years√to the fourth
4 2 V ar(T4 ) 4×(1−0.05)
accident is E(T4 )/12 = 12×0.05 = 6 3 and 12 = 12×0.05 =
3.24893

10 Ch 2.6 - the Poisson Distribution

10.1 Definition and Derivation
Poisson introduced as Binomial Approximation

If accidents were reviewed on a daily basis, the mean number of accidents per
day might be expected to be about 1/30 of the mean number in a month ie
0.05/30
If X is the number of accidents in a time period subdivided into n parts with
p the probability of an accident in each one of the parts, then for any A ⊆
{0, 1, 2, · · · }
X (np)i X n X (np)i
P (X ∈ A) − e−np = pi (1 − p)n−i − e−np
i! i i!
i∈A i∈A i∈A
<p (39)

Proof requires advanced techniques called Stein’s method

Definition of Poisson probabilities

The probabilities that approximate Binomial probabilities closely when p is

small are called Poisson probabilities.
Definition For any λ > 0, the Poisson probability of i is

e−λ λi
.
i!

The number λ is called the parameter of the Poisson probabilities - n and p

are the parameters of the Binomial distribution and np was the
parameter of the approximate Poisson probabilities.
Figures 26,27 and 28 show how close are the Poisson and Binomial probabilities.
The number D is the maximum difference between any Poisson and the
corresponding Binomial probability.

41
 ● Poisson ■ Binomial
pmf(k) n =10
0.6●
■
p =5. × 10-2
D =1.18 × 10-2
0.5

0.4

0.3
■
●

0.2

0.1 ●
■

●
■
●
■
k
1 2 3 4

Figure 26: Binomial and Poisson Distributions, n=10, p=0.05

Exponential Series shows the Poisson probs are a pmf

Taylor series for exponential function: for any x

∞
X xi
ex = (40)
i=0
i!

So the Poisson probabilities add to one: for any λ > 0 the sum of the Poisson
probabilities is:
∞ ∞
X e−λ λi X λi
= e−λ = e−λ eλ = 1 (41)
i=0
i! i=0
i!

Poisson pmf, f (i), i =, 1, 2, · · · with parameter λ > 0 is defined as the ith entry
of the series

10.2 MGF and Moments

MGF and Mean of Poisson

42
 ● Poisson ■ Binomial
pmf(k) n =100
■
●
■
●
p =5. × 10-2
D =1.26 × 10-2
0.15 ●
■
●
■

■
0.10
●

●
■

●
■

0.05
● ●
■
■

●
■
●
■ ●
■
●
■ ●
■ ●
■ ●
■ ●
■
k
5 10 15

Figure 27: Binomial and Poisson Distributions, n=100, p=0.05

● Poisson ■ Binomial
pmf(k) n =250
■
●
p =5. × 10-2
■ ■
● ●
D =1.26 × 10-2
0.10 ■
●
■
●

0.08 ●
■
■
●

■
0.06
●

●
■

●
■

0.04 ●
■ ●
■

0.02 ●
■
●
■

●
■
●
■ ●
■
● ●
■
k
■ ●
■
●
■ ●
■ ●
■ ●
■ ●
■ ●
■ ●
■ ●
■ ●
■ ●
■ ●
■

5 10 15 20 25 30

Figure 28: Binomial and Poisson Distributions, n=250, p=0.05

43
If X has a Poisson pmf with parameter λ, then

M (t) = E(etX )
∞
X e−λ λi
= eti
i=0
i!
∞
X (et λ)i
= e−λ
i=0
i!
t t
= e−λ eλe = eλ(e −1)
(42)

So  
t
E(X) = M 0 (0) = λet eλ(e −1) (0) = λ

Variance of Poisson

MGF Second Deriv
2 t t

M 00 (t) = λet eλ(e −1) + λet eλ(e −1)

So
V ar(X) = M 00 (0) − λ2 = λ

10.3 Poisson Process and Example

Assumptions for Poisson process

Events occur in continuous time with no double occurrence of events at any

specific time
Counts of events in disjoint time intervals are independent random variables
Equal mean number of events in time intervals of the same length

Then an argument based on the Poisson approximation to Binomial shows that

the number of events in any time t has exactly the Poisson distribution
with parameter λt for some λ > 0

Poisson Process

The process of events is called a Poisson process with rate λ

Interpretation of λ is the mean number of events in a unit time interval since
this has an exact Poisson distribution with parameter λ × 1
If time is continuous, then the Binomial would only be an approximation to the
Poisson - it ignores the possibility in our accident example that there would
be more than one accident in a month
Figure 29 shows an example of a Poisson process - the events occur at random
in continuous time

44
 Cumulative Number of Events versus Time

5
4
Number of Events

3
2
1
0

1 2 3 4

Time of Event

Figure 29: Assumptions require that the jumps are all size one

Example - Accidents in a Workplace with Poisson

Suppose that accidents in the workplace occur at a rate of 0.62 per year.
Carry out the calculations for the first two parts of the Negative Binomial
example with these assumptions. What is the mean and variance for the number
of accidents per year? Comment on the answers.

Solution - Accidents in a Workplace with Poisson

Let Tr be the time to the rth accident and Xt be the number of accidents
in t years, so that Xt has Poisson(0.62t) pmf. Hence
1. the chance that there are no accidents in a year is P (T1 > 1) = P (X1 =
0) = e−0.62 = 0.537944
2. noting that X10 has Poisson(6.2) pmf, the chance that the 4th accident
occurs than 10 years from now is

P (T4 > 10) = P (X10 ≤ 3)

e−6.2 6.2
= e−6.2 +
1
e−6.2 6.22
+
2
e−6.2 6.23
+
6
= 0.13423

Solution - Accidents in a Workplace with Poisson Ctd

45
Negative Binomial probabilities of 0.54036 and 0.131707 are very close, since
both are based on the Binomial and Poisson probabilities
which are close

Mean and variance for the number of events in a year are both the rate 0.62
Mean for the Binomial(12,0.05) distribution is 0.6, close to 0.62
Can match Binomial to Poisson by the mean or the probability of zero - see Lab
this week

46