Figure 1: Our Subject Mostly 3 and 4

Module 1, Probability

Contents
1 MAST90105 Overview 1

2 Ch. 1.2 Methods of Enumeration 2

2.1 How to count? . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2
2.2 Example - Sampling without Replacement . . . . . . . . . . . . . 3

3 Ch 1.3 Conditional Probability 5

3.1 Bayesian and Frequentist Statistics . . . . . . . . . . . . . . . . . 5
3.2 Definition . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5
3.3 Example - Conditional Probability Definition . . . . . . . . . . . 6
3.4 Multiplication Rule . . . . . . . . . . . . . . . . . . . . . . . . . . 7
3.5 Example - System Failure . . . . . . . . . . . . . . . . . . . . . . 7

4 Ch 1.5 Bayes’ Theorem 8

4.1 Law of Total Probability . . . . . . . . . . . . . . . . . . . . . . . 8
4.2 Bayes’ Theorem . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8
4.3 Bayes Example - Sampling w’out Replacement . . . . . . . . . . 11

5 Ch 1.4 Independent Events 13

5.1 Definition . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13
5.2 Example - Rocket Failure . . . . . . . . . . . . . . . . . . . . . . 13
5.3 Example - Pairwise Independence - Soft Drinks . . . . . . . . . . 14

1 MAST90105 Overview
The Big Picture

1
4 Phases in Data Science

1. Producing Data: collecting it as it arises, or conducting an experiment or

random sample from a population
[Link] Data Analysis: organising the data (computer science), produc-
ing summaries (mean, standard deviation etc)
3. Probability: understanding the chance mechanisms that generated the data:-
random variables and their properties are important
4. Inference: using probability to draw inferences about the data: find esti-
mates of population quantities, understand the uncertainty in the
estimates, decide on hypotheses and predict

Probability - first half of our subject

Module 1: the basics for understanding random samples - important background:-

sets, permutations and combinations, functions (1:1 and onto)
Module 2: discrete distributions and random variables - important background:
series and the exponential function, differentiation
Module 3: continuous distributions and random variables - important back-
ground: integration

Module 4: distributions for two random variables, correlation - important back-

ground: two variable calculus
Module 5: transformations of random variables and limits - important back-
ground: limits, multivariable calculus

Statistical Inference - second half of our subject

Probability important because data = prob. model + residual is underlying idea

of statistical inference
Module 6: data description and point estimation - important background:- Mod-
ules 1 to 5
Module 7: quantifying uncertainty through interval estimation - important back-
ground: Modules 1 to 5
Module 8: tests of hypotheses - important background: Modules 1 to 5

Module 9: special cases including regression, analysis of variance and contin-

gency tables - important background: Modules 1 to 5

2
2 Ch. 1.2 Methods of Enumeration
2.1 How to count?
Two basic rules for counting

• Multiplication or Tree principle If one choice has n1 possibilities and an-

other has n2 possibilities, then a choice of both has n1 × n2 possibilities
• Addition principle If one event has n1 outcomes and a disjoint one has n2
outcomes, then the union has n1 + n2 outcomes

Permutations
The number of permutations or orderings of a subsetof size r from a set of
size n is

n Pr = n(n − 1) . . . (n − r + 1)
n(n − 1) . . . 1
=
(n − r)(n − r − 1) . . . 1
n!
=
(n − r)!

Combinations
The number of combinations or samples without replacement or subsets of
size r from a set of size n is
(n − 1) . . . (n − r + 1)
n Cr =
r(r − 1) . . . 1
n(n − 1) . . . 1
=
r(r − 1) . . . 1 × (n − r)(n − r − 1) . . . 1
n!
=
r!(n − r)!
 
n
=
r

The last equality is the definition of the bracket and this number n Cr is often
called a Binomial Coefficient.

2.2 Example - Sampling without Replacement

Example - Sampling without Replacement
Suppose that 10000 lines of code have 30 lines which could be improved
(for example by clarification of purpose, simplification or documentation) . If a
random sample of 10 lines of code is taken, what is the chance that:
(a) 1 or
(b) 0
lines of the code sampled could be improved?

3
Example Solution

• The sample

space has all the samples of size 10 taken without replacement
is 10,000
10 .
• The 1 line of code in the sample that could
be improved must be chosen
from the 30 improvable lines. There are 30
1 = 30 such choices.

• The other 9 lines of code in the sample must be chosen from the 10,
000 −
30 = 9, 970 lines of code that cannot be improved. There are 9,970
9 ways
to do that.
• The multiplication principle

says that the total number of samples with
one improvable line is 30 9,970


1 × 9 .
• Random sampling without replacement means that all the samples are
equally likely so the answer is
30 9,970



1 × 9
10,000


10

Example Solution Ctd 1

There are very big numbers in this calculation but modern symbolic packages
compute them and the ratio exactly. The next two slides show the calculations
for (a) and (b) using the package Mathematica. They show that the probability
of one line in the sample needing improvement is about 0.03.

Example Solution Ctd 2

This is the Mathematica command to find the number of sample points:
Binomial[10000, 10]
This is the output of it:
2, 743, 355, 077, 591, 282, 538, 231, 819, 720, 749, 000
This calculates the probability that one line of code in the sample could be
improved:
Binomial[30, 1] ∗ Binomial[10000 − 30, 9]/Binomial[10000, 10]
Here is the output:
2143751464028247883152007617
73351740042547661450048655635  2143751464028247883152007617 
This asks to evaluate the fraction as a decimal: N 73351740042547661450048655635
Here is the result:
0.0292256

Example Solution Ctd 3

Now for the probability that there are no lines of code that could be im-
proved, Input and Output:
Binomial[30, 0] ∗ Binomial[10000 − 30, 10]/Binomial[10000, 10]
1016852777770732245908435612997
1047882000607823735000695080500
 1016852777770732245908435612997 
N 1047882000607823735000695080500

0.970389

4
Ex. Sampling W’out Replacement - Comments, Questions
• There is a chance of 0.9995 that there is at most one line of code that
could be improved
• Do we really have to work out those big numbers? Approximations?
• What happens with sampling with replacement? Comparison?
• In practice, we’d observe the number of lines of code that could be im-
proved in a sample, and want to infer from that how good the code is in
total
• To do this we could use the probabilities calculated to make an inference
about the plausibility of 30 in total
• For example, if we observed 5 lines of code in our sample of 10 that could
be improved, then 30 in total is implausible

3 Ch 1.3 Conditional Probability

3.1 Bayesian and Frequentist Statistics
Conditional Probability
• Continuing the last example, given an observed number of 5 lines of code
in our sample, the probability that the total number of lines of code among
the 10,000 is 30 or 50 or 100 ... would help us decide the quality of the
code?
• This is the approach of Bayesian statistics

Conditional Probability
• The frequentist approach, on the other hand, would be to start with the
Assumption that there were 30 or 50 or 100 lines of code that could be
improved, and then to compute the probability of the observed 5 lines of
improvable code in the sample
• This probablity could then be plotted against the total number of lines,
and the maximum picked out - this is called the maximum likelihood esti-
mate
• Conditional probabilities are the heart of the Bayesian approach, but are
also very important for frequentists

3.2 Definition
Conditional Probability
Definition. The conditional probability of an event A, given that event B
has occurred, is defined by

P (A ∩ B)
P (A|B) = ,
P (B)
provided that P (B) > 0

5
3.3 Example - Conditional Probability Definition
Example - Conditional Prob Definition
Two cards are drawn without replacement from a deck of cards. (a) What
is the conditional probability of getting an ace on the second draw given that
the first draw is an ace? (b) What is the conditional probability of getting an
ace on the first draw given that the second draw is an ace?

Example Solution
Let A1 be the event that the first draw is an ace and A2 be the event that
the second draw is an ace. Then

n(A1 )
P (A1 ) = , so
n(S)
4 × 51
P (A1 ) =
52 × 51
n(A2 )
P (A2 ) = , so
n(S)
4 × 3 + 48 × 4
P (A2 ) =
52 × 51
n(A1 ∩ A2 ) 4×3
P (A1 ∩ A2 ) = , so P (A1 ∩ A2 ) =
n(S) 52 × 51

Example Solution Ctd

Answering (a)
P (A1 ∩ A2 )
P (A2 |A1 ) =
P (A1 )
which gives
4×3 3
P (A2 |A1 ) = =
4 × 51 51
and
P (A1 ∩ A2 )
P (A1 |A2 ) = ,
P (A2 )
so
4×3 3
P (A1 |A2 ) = =
4 × 51 51

Conditional Probability Example - Comment

• The answer to (a) is not perhaps surprising since there are 51 cards left
after the first choice of which 3 are aces given that the first drawer was an
ace
• But why should (b) have the same answer?
• A thought experiment leads to the idea of symmetry of events, in this case
called exchangeability

6
Rules of Conditional Probability
Conditional probability satisfies the same rules as ordinary probability. For
unions, the conditioning event needs to be the same. For example, if A1 and A2
are disjoint events and P (B) > 0,

P (A1 ∪ A2 |B) = P (A1 |B) + P (A2 |B)

3.4 Multiplication Rule

Multiplication Rule
The definition of conditional probability can be re-expressed as,

P (A ∩ B) = P (A|B) × P (B)
or equally

P (A ∩ B) = P (B|A) × P (A)

3.5 Example - System Failure

Example - System Failure
A device has two components. It will operate if at least one of the two
components is operating. The probability that one component will fail when
both are working in a one-year period is 0.01. However, when one fails, the
probability of the other failing is 0.03 in that one-year period due to added strain.
The two components cannot fail at the same time. What is the probability that
the device fails during a one-year period

Solution - System Failure

Let C1 (C2 ) be the event that the first (respectively second) component fails
first in the given year and B be the event that both components fail in the year.
Then, because B = (B ∩ C1 ) ∪ (B ∩ C2 ) and the events B ∩ C1 , B ∩ C2 are
disjoint,
P (B) = P (B ∩ C1 ) + P (B ∩ C2 )
Using the Multiplication Rule for Conditional Probabilities on each component

P (B) = P (B|C1 ) × P (C1 ) + P (B|C2 ) × P (C2 )

= 0.03 × 0.01 + 0.03 × 0.01
= 0.06

7
4 Ch 1.5 Bayes’ Theorem
4.1 Law of Total Probability
The Law of Total Probability

Assumption 1 Suppose A1 , A2 , · · · , An (n = 1, 2, . . . ) are disjoint events each

with non-zero probability.
Assumption 2 Suppose the event B is the union of B ∩ A1 , B ∩ A2 · · · B ∩ An
–ie the event B is partitioned into disjoint bits by the events
A1 , A2 . . . An .
Venn Diagram for B:

An ∩ B A1 ∩ B

. A2 ∩ B

. .

. .

The Law of Total Probability

Assumption 1 Suppose A1 , A2 , · · · , An (n = 1, 2, . . . ) are disjoint events each

with non-zero probability.
Assumption 2 Suppose the event B is the union of B ∩ A1 , B ∩ A2 · · · B ∩ An
–ie the event B is partitioned into disjoint bits by the events
A1 , A2 . . . An .
Rule (c) for Probability:

P (B) = P (B ∩ A1 ) + P (B ∩ A2 ) + · · · + P (B ∩ An )
Multiplication Rule gives the Law of Total Probability:

P (B) = P (B|A1 )P (A1 ) + P (B|A2 )P (A2 ) + · · · + P (B|An )P (An ) (1)

4.2 Bayes’ Theorem

The Law of Total Probability to Bayes Theorem

System Failure Ex. Assumptions were the probabilities P (B|Ci ), P (Ci ), i = 1

or 2, and could then find P (B)
Samp. W’out Repl. Ex. Assumptions were the total number of lines of code
and the total number that could be improved.
So could calculate probabilities for the number of lines of code in a random
sample that could be improved.

8
But to infer from an observed number in the sample to the total number of
lines of improvable code, the reverse conditional probabilities are needed

Enter the Reverend Thomas Bayes who had a remarkable posthumous essay
published in 1764 containing Bayes Theorem

The Reverend Thomas Bayes,1702 - 1761

Bayes Theorem - Assumptions as Before

Assumption 1 Suppose A1 , A2 , · · · , An (n = 1, 2, . . . ) are disjoint events each

with non-zero probability.
Assumption 2 Suppose the event B is the union of B ∩ A1 , B ∩ A2 · · · B ∩ An
–ie the event B is partitioned into disjoint bits by the events
A1 , A2 . . . An .

An ∩ B A1 ∩ B

. A2 ∩ B

. .

. .

Bayes Theorem

Assumption 1 Suppose A1 , A2 , · · · , An (n = 1, 2, . . . ) are disjoint events each

with non-zero probability.
Assumption 2 Suppose the event B is the union of B ∩ A1 , B ∩ A2 · · · B ∩ An
–ie the event B is partitioned into disjoint bits by the events
A1 , A2 . . . An .
Definition of Conditional Probability: gives for any i in 1 · · · n

P (B ∩ Ai )
P (Ai |B) = or P (B ∩ Ai ) = P (Ai |B)P (B)
P (B)
Law of Total Probability used on P (B) then gives Bayes Theorm:

P (B|Ai )P (Ai )
P (Ai |B) = (2)
P (B|A1 )P (A1 ) + P (B|A2 )P (A2 ) + · · · + P (B|An )P (An )

9
Figure 2: 1702 - 1761

10
4.3 Bayes Example - Sampling w’out Replacement
Bayes Example - Sampling w’out Replacement
Suppose that the standard for acceptable code is that at most 10% can be
improved. Past experience suggests that 90% of blocks of 1000 lines of code
meet the standard, with 10% failing to meet the standard. A random sample
of size 20 is taken from 1000 lines of code and, on inspection, it is observed
that 5 of the 20 lines of code can be improved. Does it seem plausible that the
standard has been met?

Bayes Ex. - Sampl. w’out Repl’t: Solution

Definitions Let Ai be the event that, in total, i(i = 0, 1, · · · , 1000) lines of code
can be improved and B be the event that in the random sample of
20 lines of code 5 could be improved .
Standard? For the standard to be met, one of the events A0 , · · · , A100 must be
true.
Sampl. w’out Repl’t Ex. means we can work out the probabilities P (B|Ai ) be-
cause, given the total number of lines of code that can
be improved,
the combinations calculations gave us:

i 1000−i



5 15
P (B|Ai ) = 1000

(3)
20

Bayes Ex. - Solution Ctd

What about P (Ai )(i = 0, 1, · · · , 1000)? We haven’t been told in detail

But We have been told that 90% meets the standard.

So, since the event that the code meets the standard is the union of the disjoint
events A0 , A1 , · · · , A100 ,
Rule (c) gives us:

0.9 = P (A0 ) + P (A1 ) + · · · + P (A100 ) (4)

Simplest assumption is that the events A0 , A1 , ...A100 are equally likely

So we assume
P (A0 ) = P (A1 ) = · · · = P (A100 ) = 0.9/101 = 0.00891 · · · (5)

11
Bayes Ex. - Solution Ctd 2

Simlarly we assume

P (A101 ) = P (A102 ) = · · · = P (A1000 ) = 0.1/900 = 0.00011 · · · (6)

Bayes Theorem (2) gives
P (A0 |B) = P (A1 |B) = · · · = P (A4 |B) = 0 (7)
since the number of lines overall that can be improved must be at least the
number in the sample that can be improved
and
P (B|A5 )P (A5 )
P (A5 |B) = (8)
P (B|A0 )P (A0 ) · · · + P (B|A1000 )P (A1000 )

Bayes Ex. - Solution Ctd 3

Combining (3) to (6) in (7) gives

(55)(995
15 ) 0.9
×
(1000
20 )
101
P (A5 |B) = (9)
(55)(995
15 ) 0.9 (1000 0
5 )(15) 0.1
5×0+ × + ··· + ×
(1000
20 )
101 1000
( 20 ) 900

The answer to this is small because a fluke gave us all five lines of code that
could be improved in our sample of 20
The numbers will come from Mathematica next week in the lab - and you will
discuss the interpretation!
But notice that (3) simplifies: for i ≥ 5
i 1000−i


 
5 15 20 i i − 4 1000 − i 1000 − i − 14
1000

= × ×···× × ×···× (10)
20
5 1000 996 995 981

Sampling with or without replacement

So if i is small relative to 1000

i 1000−i


  
5 15 20 i 5  i 15
1000

≈ × × 1− (11)
20
5 1000 1000

Which is the probability that we’ll see later applies when there is sampling with
replacement
The reason is that whatever happens during the first few selections of lines of
code in the sample does not change the probabilities much
This leads to the idea of independent events

12
5 Ch 1.4 Independent Events
5.1 Definition
Independent Events

Events are independent if the occurrence of any combination of them (or not)
does not change the probabilities of the others
So if A and B are independent, then P (A|B) = P (A), P (B|A) = P (B) and
P (A|B c ) = P (A)

The multiplication rule shows that, if A and B are independent, then

P (A ∩ B) = P (A)P (B) (12)

It turns out that (12) is enough to ensure that conditioning either event on the
other, or its complement, does not change the probabilities ie (12)
is enough to ensure independence of two events
For n events to be independent need that for any choices of the events Ai , Aj , · · · Az

P (Ai ∩ Aj ∩ · · · ∩ Az ) = P (Ai )P (Aj ) · · · P (Az ) (13)

5.2 Example - Rocket Failure

Example - Rocket Failure
A rocket has a built-in redundant system. In this system, if component K1
fails, it is bypassed and component K2 is used. If component K2 fails, it is
bypassed and component K3 is used. Suppose that the probability of failure
of any one of these components is 0.15 and assume that the failures of these
components are mutually independent events. What is the probability of that
the rocket does not fail?

Solution - Rocket Failure

Definition Let Ai denote the event that component component Ki , i = 1, 2, 3

fails.
Reasoning Because the system fails only if all components fail, the probability
that the system does not fail is given by

P ((A1 ∩ A2 ∩ A3 )c ) = 1 − P (A1 ∩ A2 ∩ A3 )

Independence gives the right hand side as

1 − P (A1 )P (A2 )P (A3 )

So the probability that the rocket does not fail is 1 − 0.153 = 0.9966

13
Comment - Rocket Failure
The desired probability is the probability that one of the components does
not fail. This is the probability of the union of three events. Try working the
probability out this way. Which way is easier?

5.3 Example - Pairwise Independence - Soft Drinks

Example - Soft drinks
Two companies producing soft drinks, P and C, compete for shelf space in 3
supermarket chains, B, D and X. At one time, competition is so fierce that the
possibilities are equally likely for which company has more shelf space in which
supermarket chain.
1. What is an appropriate sample space?
2. Consider the events PB ,PD , PX that P has more shelf space in supermarket
chains, B, D, X (respectively). Are these events independent?
3. What about the events BD, DX, BX that chains
(a) B and D,
(b) D and X,
(c) B and X (respectively)
have more shelf space for the same soft drink?

Solution - Soft drinks - 1

An appropriate sample space has sample points which are ordered triples
recording the soft drink that has more shelf space in supermarket chains B, D
and X in this order. Therefore,
S = {(P, P, P ), (P, P, C), (P, C, P ), (P, C, C),
(C, P, P ), (C, P, C), (C, C, P ), (C, C, C)}.
Notice that there are 2 × 2 × 2 = 8 outcomes in the sample space.

Example - Soft drinks

Two companies producing soft drinks, P and C, compete for shelf space in 3
supermarket chains, B, D and X. At one time, competition is so fierce that the
possibilities are equally likely for which company has more shelf space in which
supermarket chain.
1. What is an appropriate sample space?
2. Consider the events PB ,PD , PX that P has more shelf space in
supermarket chains, B, D, X (respectively). Are these events
independent?
3. What about the events BD, DX, BX that chains
(a) B and D,
(b) D and X,
(c) B and X (respectively)
have more shelf space for the same soft drink?

14
Solution - Soft drinks - 2

The event
PB = {(P, P, P ), (P, P, C), (P, C, P ), (P, C, C)}

so
4 1
P (PB ) = = .
8 2
Write out the outcomes in PD and PX to convince yourself that these events
also have probability one half.
The event
PB ∩ PD = {(P, P, P ), (P, P, C)}

so
2 1 1
P (PB ∩ PD ) = = × = P (PB ) × P (PD )
8 2 2

Solution - Soft drinks - 2 ctd

Hence, PB and PD are independent. Check for yourself that the pairs PD and
PX , as well as PB and PX , are also independent.
Finally,
PB ∩ PD ∩ PX = {(P, P, P )}

so
1
P (PB ∩ PD ∩ PX ) =
8
1 1 1
= × ×
2 2 2
= P (PB ) × P (PD ) × P (PX )

Hence, the events PB , PD , PX are independent.

Example - Soft drinks

Two companies producing soft drinks, P and C, compete for shelf space in 3
supermarket chains, B, D and X. At one time, competition is so fierce that the
possibilities are equally likely for which company has more shelf space in which
supermarket chain.
1. What is an appropriate sample space?
2. Consider the events PB ,PD , PX that P has more shelf space in supermarket
chains, B, D, X (respectively). Are these events independent?
3. What about the events BD, DX, BX that chains
(a) B and D,
(b) D and X,
(c) B and X (respectively)
have more shelf space for the same soft drink?

15
Solution - Soft drinks - 3

The event
BD = {(P, P, P ), (P, P, C), (C, C, P ), (C, C, C)}

so
4 1
P (BD) = = .
8 2
Write out the outcomes in DX and BX to convince yourself that these events
also have probability one half.
The event
BD ∩ BX = {(P, P, P ), (C, C, C)}

so
2 1 1
P (BD ∩ BX) = = × = P (BD) × P (BX)
8 2 2

Solution - Soft drinks - 3 ctd

Hence, BD and BX are independent. Check for yourself that the pairs BD
and DX, as well as DX and BX, are also independent.
Finally,

BD ∩ DX ∩ BX = {(P, P, P ), (C, C, C)} =! BD ∩ BX

so
1
P (BD ∩ DX ∩ BX) =
4
6= P (BD) × P (DX) × P (BX)

Hence, the events BD, DX, BX are not independent.

Knowing BD and BX occurred tells us that DX also occurred.

16