MATH40005: Probability and Statistics

Almut E. D. Veraart
Room 551, Huxley Building
Department of Mathematics
Imperial College London
180 Queen’s Gate, London, SW7 2AZ
E-Mail: [Link]@[Link]

Autumn 2023

(Last Update: Thursday 19th October, 2023)

1
Contents

1 Introduction 4
1.1 Why do we study probability? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4
1.2 Complementary reading . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4
1.3 Course overview (Autumn term) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4

2 Sample spaces and interpretations of probability 6

2.1 Notation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6
2.2 The sample space Ω . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6
2.2.1 Notation from set theory and Venn diagrams . . . . . . . . . . . . . . . . . . . . 7
2.2.2 Cardinality . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9
2.3 Interpretations of probability . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10
2.3.1 Naive definition of probability - classical interpretation . . . . . . . . . . . . . . . 10
2.3.2 Limiting frequency . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12
2.3.3 Subjective . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13

3 Counting 14
3.1 The multiplication principle . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14
3.2 Power sets . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15
3.3 Sampling with and without replacement . . . . . . . . . . . . . . . . . . . . . . . . . . . 15
3.3.1 Sampling with replacement – ordered . . . . . . . . . . . . . . . . . . . . . . . . 16
3.3.2 Sampling without replacement – ordered . . . . . . . . . . . . . . . . . . . . . . 16
3.3.3 The birthday problem . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17
3.3.4 Sampling without replacement – unordered . . . . . . . . . . . . . . . . . . . . . 18
3.3.5 Sampling with replacement – unordered . . . . . . . . . . . . . . . . . . . . . . . 20
3.3.6 Summary table . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22

4 Axiomatic definition of probability 23

4.1 The event space F . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23
4.2 Definition of probabilities and basic properties . . . . . . . . . . . . . . . . . . . . . . . . 25
4.2.1 Probability measure and probability space . . . . . . . . . . . . . . . . . . . . . . 25
4.2.2 Basic properties of the probability measure . . . . . . . . . . . . . . . . . . . . . 25
4.2.3 Examples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27

5 Conditional probabilities 28
5.1 Definition . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28
5.2 Examples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29
5.3 Multiplication rule . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29
5.4 Bayes’ rule and law of total probability . . . . . . . . . . . . . . . . . . . . . . . . . . . 30
5.4.1 Bayes’ rule . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30
5.4.2 Law of total probability . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30
5.4.3 General Bayes’ rule . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31
5.4.4 Bayes’ rule and law of total probability with additional conditioning . . . . . . . . 31

1
A. E. D. Veraart MATH40005: Probability and Statistics Autumn 2023

5.5 Examples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32
5.5.1 Examples: Cards and marbles . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32
5.5.2 Example: Testing for a rare disease . . . . . . . . . . . . . . . . . . . . . . . . . 32
5.5.3 Example: Monty Hall – Conditioning on the missing information . . . . . . . . . 35

6 Independence 37
6.1 Independence of events . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37
6.1.1 Conditional independence of events . . . . . . . . . . . . . . . . . . . . . . . . . 38
6.1.2 Continuity of the probability measure and product rule . . . . . . . . . . . . . . . 39

7 Discrete random variables 42

7.1 Pre-images and their properties . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42
7.2 Random variables . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43
7.3 Discrete random variables and probability distributions . . . . . . . . . . . . . . . . . . . 43
7.4 Common discrete distributions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45
7.4.1 Bernoulli distribution . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 46
7.4.2 Binomial distribution . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 46
7.4.3 Hypergeometric distribution . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 48
7.4.4 Discrete uniform distribution . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 49
7.4.5 Poisson distribution . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 50
7.4.6 Geometric distribution . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51
7.4.7 Negative binomial distribution . . . . . . . . . . . . . . . . . . . . . . . . . . . . 52
7.4.8 Exercise . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 54

8 Continuous random variables 55

8.1 Random variables and their distributions . . . . . . . . . . . . . . . . . . . . . . . . . . . 55
8.2 Continuous random variables and probability density function . . . . . . . . . . . . . . . 57
8.3 Common continuous distributions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 59
8.3.1 Uniform . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 59
8.3.2 Exponential . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 59
8.3.3 Gamma distribution . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 60
8.3.4 Chi-squared distribution . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 60
8.3.5 F-distribution . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 61
8.3.6 Beta distribution . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 61
8.3.7 Normal distribution . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 61
8.3.8 Cauchy distribution . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 63
8.3.9 Student t-distribution . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 63
8.4 Example of a random variable which is neither discrete nor continuous . . . . . . . . . . . 63

9 Transformations of random variables 65

9.1 The discrete case . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 65
9.2 The continuous case . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 66

10 Expectation of random variables 69

10.1 Definition of the expectation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 69
10.2 Law of the unconscious statistician (LOTUS) . . . . . . . . . . . . . . . . . . . . . . . . 70
10.3 Variance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 72

11 Bridging lecture: Multivariate calculus 73

11.1 Partial derivatives . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 73
11.2 Bivariate integrals . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 74
11.3 Change of variables formula . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 74
11.3.1 Example using polar coordinates . . . . . . . . . . . . . . . . . . . . . . . . . . . 75

2
A. E. D. Veraart MATH40005: Probability and Statistics Autumn 2023

12 Multivariate random variables 77

12.1 Multivariate distributions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 77
12.1.1 The bivariate case . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 77
12.1.2 The n-dimensional case . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 78
12.2 Independence . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 78
12.3 Multivariate discrete distributions and independence . . . . . . . . . . . . . . . . . . . . . 79
12.3.1 Independence . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 79
12.4 Multivariate continuous distributions and independence . . . . . . . . . . . . . . . . . . . 79
12.4.1 Independence . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 80
12.4.2 Examples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 80
12.5 Transformations of random vectors: The bivariate case . . . . . . . . . . . . . . . . . . . 82
12.6 Two dimensional law of the unconscious statistician (2D LOTUS) . . . . . . . . . . . . . 83
12.7 Covariance and correlation between random variables . . . . . . . . . . . . . . . . . . . . 84

13 Generating functions 87
13.1 Probability generating functions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 87
13.1.1 Common probability generating functions . . . . . . . . . . . . . . . . . . . . . . 88
13.1.2 Probability generating function of a sum of independent discrete random variables 89
13.1.3 Moments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 89
13.2 Moment generating functions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 91
13.2.1 Properties . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 92
13.3 Using m.g.f.s for finding all moments of the exponential and the standard normal distributions 93
13.4 Outlook: Characteristic function and Laplace transform . . . . . . . . . . . . . . . . . . . 93

14 Conditional distribution and conditional expectation 95

14.1 Discrete case: Conditional expectation and the law of total expectation . . . . . . . . . . . 95
14.1.1 Conditioning on a random variable . . . . . . . . . . . . . . . . . . . . . . . . . . 96
14.1.2 Example . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 96
14.2 Continuous case: Conditional density, conditional distribution and conditional expectation 97
14.2.1 Examples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 100

3
Chapter 1

Introduction

1.1 Why do we study probability?

Probability

• is a beautiful branch of mathematics with a long history going back to the early works by Cardano
(16th century), Fermat and Pascal (17th century), Laplace (19th century). Modern (axiomatic) prob-
ability theory, however, is a much younger discipline which goes back to the influential work by
Kolmogorov published in 1933,

• is a very dynamic discipline with a strong interplay between theory and applications,
• is ubiquitous in every day life and in most sciences,
• is the foundation for statistics,
• enables us to interpret and quantify uncertainty.

1.2 Complementary reading

• These lecture notes are self contained. They are mainly based on the textbooks Grimmett & Welsh
(1986), Blitzstein & Hwang (2019) and Anderson et al. (2018).
You can get the first and third book from our library and the second book is available on-line at
[Link] where you can also find addi-
tional exercises and solutions.

• Complementary reading material can be found in the following textbooks: Ross (2014).

1.3 Course overview (Autumn term)

1. Interpretations of probability; limiting frequency; classical (symmetry between equally likely out-
comes) ; subjective (degree of personal belief)
2. Counting: multiplication principle; binomial coefficients; the inclusion-exclusion principle; stars and
bars arguments
3. Formal probability: probability axioms; conditional probability; Bayes’ theorem; independence
4. Random variables: mass and density functions; common discrete and continuous distributions, trans-
formations of random variables, expectation and variance; probability and moment generating func-
tions

4
A. E. D. Veraart MATH40005: Probability and Statistics Autumn 2023

5. Multivariate random variables: Joint mass and density functions; independence; covariance
6. Conditional distribution: Conditional probability mass function, conditional density, conditional ex-
pectation, law of total expectation

5
Chapter 2

Sample spaces and interpretations of

probability

The material of this chapter is based on Blitzstein & Hwang (2019), p.1-8, Anderson et al. (2018), p.1-5,
Proschan & Shaw (2016), p.9-10.

2.1 Notation
Throughout the lecture notes we denote the natural numbers by N = {1, 2, . . . } and we define N0 =
N ∪ {0}. Moreover, we denote the integers by Z = {. . . , −1, 0, 1, . . . }, the real numbers by R. For real
numbers a < b we write [a, b] for closed intervals and (a, b) for open intervals.

2.2 The sample space Ω

Probability theory is based on set theory which was introduced in the course Introduction to University
Mathematics. We will now explain how set theory enters in probability theory and review some of the key
concepts briefly.
Definition 2.2.1 (Sample space). The sample space Ω is defined as the set of all possible outcomes of an
experiment. The elements of Ω are typically denoted by ω and called sample points.

Example 2.2.2. We start with the classical example of flipping a (fair) coin. We write H for heads and T
for tails. The sample space is given by

Ω = {H, T }.

Figure 2.1: Flipping a fair coin.

6
A. E. D. Veraart MATH40005: Probability and Statistics Autumn 2023

Example 2.2.3. Consider the experiment where we roll a standard six-sided (fair) die. The sample space
associated with this experiment is given by

Ω = {1, 2, 3, 4, 5, 6}.

2.2.1 Notation from set theory and Venn diagrams

Let us recap some concepts from set theory which you studied in MATH40001.

• Subsets of Ω are collections of elements of Ω and called events. Notation: A is a subset of Ω can be
written as A ⊆ Ω meaning that every element of A is also an element of Ω.
• We write ω ∈ A if the element ω is a member of A and ω ̸∈ A if the element ω is not a member of
A.
• We denote the empty set by ∅. Note that the empty set contains no points, i.e. ω ̸∈ ∅ for all ω ∈ Ω.

• Every subset A of the sample space Ω satisfies ∅ ⊆ A ⊆ Ω.

Example 2.2.4. Let Ω = {1, 2, 3, 4, 5, 6}. For instance, we can say that 1 ∈ Ω and {1} ⊆ Ω.
Suppose that A, B ⊆ Ω are events, then
• the union A ∪ B = {ω ∈ Ω : ω ∈ A or ω ∈ B} is the event that at least one of A and B occurs (this
is the inclusive ”or”),

• the intersection A ∩ B = {ω ∈ Ω : ω ∈ A and ω ∈ B} is the event that both A and B occur,

• the complement Ac = Ω \ A = {ω ∈ Ω : ω ̸∈ A} is the event that occurs if and only if A does not
occur.
We typically use so-called Venn diagrams to illustrate concepts from set theory such as the union, in-
tersection and complement of sets introduced above. Consider a sample space Ω with subsets A, B ⊆ Ω.

7
A. E. D. Veraart MATH40005: Probability and Statistics Autumn 2023

Ω Ω

B B

A A A∪B

(a) A, B ⊆ Ω (b) A ∪ B

Ω Ω

B B
A∩B
A A
Ac

(c) A ∩ B (d) Ac

Figure 2.2: We consider a sample space Ω with subsets A, B ⊆ Ω which are depicted in Figure 2.2a. The
grey area in Figure 2.2b depicts the union A ∪ B. The grey area in Figure 2.2c depicts the intersection
A ∪ B, and the grey area in Figure 2.2d depicts the complement Ac .

In MATH40001 you studied De Morgan’s laws and distributivity for propositions. These results imply
the following very useful identities for sets.
Let I denote a general index set, e.g. I = {1, 2} or I = N or I = [0, ∞). Suppose that Ai ⊆ Ω for all
i ∈ I and B ⊆ Ω. Then the following identities hold.

• De Morgan’s laws:
!c !c
\ [ [ \
Ai = Aci , and Ai = Aci ,
i∈I i∈I i∈I i∈I

• Distributivity:
! !
[ [ \ \
B∩ Ai = (B ∩ Ai ), and B∪ Ai = (B ∪ Ai ).
i∈I i∈I i∈I i∈I

Let us prove the first De Morgan’s law as an exercise. First we consider the easier case when I = {1, 2}.
c
Exercise 2.2.5. Using the notation above, show that (A1 ∩ A2 ) = Ac1 ∪ Ac2 .
Proof. Here we want to prove an identity between two sets. As a general strategy, recall that we can prove
the equality by first showing that the set on the left hand side is a subset of the set on the right hand side
and then that the set on the right hand side is a subset of the set on the left hand side.
Proof of (A1 ∩ A2 )c ⊆ Ac1 ∪ Ac2 :
Let a ∈ (A1 ∩ A2 )c .
⇒ Then a is not an element of (A1 ∩ A2 ).

8
A. E. D. Veraart MATH40005: Probability and Statistics Autumn 2023

⇒ There is at least one i∗ ∈ I = {1, 2} such that a ̸∈ Ai∗ .

⇒ There is at least one i∗ ∈ I = {1, 2} such that a ∈ Aci∗ .
Since Aci∗ ⊆ Ac1 ∪ Ac2 , we deduce that a ∈ Ac1 ∪ Ac2 .
c
Proof of (A1 ∩ A2 ) ⊇ Ac1 ∪ Ac2 :
c c
Let a ∈ A1 ∪ A2 .
⇒ There is at least one i∗ ∈ I such that a ∈ Aci∗ . I.e. there is at least one i∗ ∈ I such that a ̸∈ Ai∗ .
Since A1 ∩ A2 ⊆ Ai∗ , we deduce that a ̸∈ A1 ∩ A2 .
(Suppose that a ∈ A1 ∩ A2 . Then a ∈ A1 and a ∈ A2 , which is a contradiction to the statement that
there is at least one i∗ ∈ I such that a ̸∈ Ai∗ .)
c
Since the proposition a ̸∈ A1 ∩ A2 is equivalent to a ∈ (A1 ∩ A2 ) , we are done with the proof.

We can use the same arguments to prove the general case:

T
c S
Exercise 2.2.6. Using the notation above, show that i∈I Ai = i∈I Aci .
T
c S
Proof. Proof of i∈I Ai ⊆ i∈I Aci :
T
c
Let a ∈ i∈I Ai .

T
⇒ Then a is not an element of i∈I Ai .
⇒ There is at least one i∗ ∈ I such that a ̸∈ Ai∗ .
⇒ There is atSleast one i∗ ∈ I such that a ∈SAci∗ .
Since Aci∗ ⊆ i∈I Aci , we deduce that a ∈ i∈I Aci .
T
c S
Proof ofS i∈I Ai ⊇ i∈I Aci :
Let a ∈ i∈I Aci .
∗ c ∗
⇒ ThenT there is at least one i ∈ I such thatTa ∈ Ai∗ . I.e. there is at least one i ∈TI such
that a ̸∈ Ai∗ .
c
Since i∈I Ai ⊆ Ai∗ , we deduce that a ̸∈ i∈I Ai which is equivalent to a ∈ A
i∈I i .

End of lecture 1. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

2.2.2 Cardinality
Definition 2.2.7 (Cardinality). For any set A, we define the cardinality of A as the number of elements in
A. We typically write card(A) or simply |A| (the latter should not be confused with the absolute value!).

Example 2.2.8. Let us compute the cardinality of the sample spaces Ω considered in the two examples
above.
• Flipping of a (fair) coin. Here we have Ω = {H, T }, then

card(A) = 2.

• Rolling a (fair) die. Here we have Ω = {1, 2, 3, 4, 5, 6}, then

card(A) = 6.

Definition 2.2.9. Two sets have the same cardinality if there is a bijection between the two sets.

Bijective functions were introduced in MATH40001. Let us recall their definition again.
Definition 2.2.10 (Injective, surjective and bijective functions).
• A function f : A 7→ B is called injective if ∀a1 , a2 ∈ A, f (a1 ) = f (a2 ) ⇒ a1 = a2 .

• A function f : A 7→ B is called surjective if ∀b ∈ B, ∃a ∈ A such that f (a) = b.

9
A. E. D. Veraart MATH40005: Probability and Statistics Autumn 2023

• A function f : A 7→ B is called bijective if it is both injective and surjective.

Definition 2.2.11 (Finite, countably infinite and uncountably infinite sets). • A set A is said to be fi-
nite, if it has a finite number of elements.
• A set A is called countably infinite if there is a bijection between the elements of A and the natural
numbers N = {1, 2, . . . }.
• A set A is called countable if it is either finite or countably infinite.
• If the set A is neither finite nor countably infinite, we call it uncountable or uncountably infinite.
Example 2.2.12. The sample space of an experiment can be
• finite (e.g. Ω = {1, . . . , 6}),
• countably infinite (e.g. Ω = N = {1, 2, . . . }), or
• uncountably infinite (e.g. Ω = [0, 1]).
Exercise 2.2.13. Show that Z = {. . . , −2, −1, 0, 1, 2, . . .} and N ∪ {0} = {0, 1, . . . } have the same
cardinality.
Proof. We use the convention that 0 is even. We define a function f : N ∪ {0} → Z such that
 x
2, if x is even,
f (x) = (x+1)
− 2 , if x is odd.

We need to show that this function is bijective.

First, we show that it is injective: For all x, y ∈ N ∪ {0} with f (x) = f (y) we have that f (x) and f (y)
have the same sign. So either
x y
f (x) = = = f (y) ⇒ x = y,
2 2
or
x+1 y+1
f (x) = − =− = f (y) ⇒ x = y.
2 2

Next, we show that f is surjective. For all y ∈ Z, y < 0, choose x = −2y − 1 (which is odd), then
f (x) = − −2y−1+1
2 = y. For all y ∈ Z, y ≥ 0, choose x = 2y (which is even), then f (x) = 2y
2 = y.

2.3 Interpretations of probability

Let us briefly discuss the three main interpretations of probability, for an extended survey please see Hájek
(2012).

2.3.1 Naive definition of probability - classical interpretation

Consider the case when the sample space Ω is finite, i.e. card(Ω) < ∞, and suppose you want to assign a
probability to the event A ⊆ Ω. A naive definition of a probability is obtained when we count the elements
in A and divide by the total number of elements in Ω:
Definition 2.3.1 (Naive definition of probability). Suppose that the sample space Ω is finite, i.e. card(Ω) <
∞ and consider an event A ⊆ Ω. Then the naive probability of A is defined as

card(A)
PNaive (A) = .
card(Ω)

10
A. E. D. Veraart MATH40005: Probability and Statistics Autumn 2023

In addition to the assumption that the sample space is finite, the naive definition of a probability also
assumes that each possible outcome has the same weight. When is such a definition applicable? In sym-
metric settings when all outcomes are equally likely (for instance when we toss a fair coin or roll a fair
die), or in settings where the outcomes are equally likely due to the design of a study (for instance when
I randomly select 10 students out of the entire year group assuming that the selection mechanism is such
that all subsets of 10 students are equally likely).
Example 2.3.2. Consider the example of rolling a six-sided fair die. What is the (naive) probability that I
roll either a 1 or a 2? We have Ω = {1, . . . , 6} and A = {1, 2}. Hence

card(A) 2 1
PNaive (A) = = = .
card(Ω) 6 3

Let us consider the complement: Ac = {3, 4, 5, 6}. The probability of the complement can be computed as

card(Ac ) card(Ω) − card(A) card(A) 2

PNaive (Ac ) = = =1− = 1 − PNaive (A) = .
card(Ω) card(Ω) card(Ω) 3

Note that for A ⊆ Ω we always have that P(Ac ) = 1 − P(A), not just in the case of the naive
probability.
The classical interpretation applies when we have outcomes that are equally likely. In our naive defini-
tion above, we have only covered the case when card(Ω) < ∞. If Ω is uncountably infinite, but of finite
area, e.g. choose a disk of radius 1: Ω = {(x, y) ∈ R2 : x2 + y 2 ≤ 1} and the event A is some subset of
Ω, then we could assume that the probability of the event A should be uniform on Ω, i.e.
area of A
P(A) = .
area of Ω
For instance, for A = {(x, y) ∈ R2 : x2 + y 2 ≤ 0.52 }, we have

area of A 0.52 π
P(A) = = = 0.25.
area of Ω π

A x
−1 1

−1

Figure 2.3: Illustration of the classical interpretation of probability in the case when Ω is uncountably
infinite.

Remark 2.3.3. In order for the classical/naive definition to work, we need that the number of elements
in Ω is either finite, or, if Ω is uncountably infinite, then we require that the area of Ω is finite. In either
scenario we can then define a uniform distribution, which we will study in more detail later in the course.
Note that there is no uniform distribution on N or on R.

11
A. E. D. Veraart MATH40005: Probability and Statistics Autumn 2023

2.3.2 Limiting frequency

Consider ntotal replications of an experiment and let nA denote the number of times event A occurs (out
of ntotal ). Then we could interpret the probability of event A occuring as
nA
P(A) = lim .
ntotal →∞ ntotal
The problem with this interpretation is that ntotal → ∞ may be difficult to conceive, and any finite version
may not be representative.

Example 2.3.4. Consider a fair coin toss and let A = {H} denote the event that Heads appears. Figure1
2.4 illustrates a possible outcome of the experiment.

Figure 2.4: One possible outcome when tossing a fair coin repeatedly.

Let us compute the relative frequency of A and report and plot them in Table 2.1 and Figure 2.5,
respectively.

ntotal 1 2 3 4 5 6 7 8 9 10 ···
nA
ntotal 0/1 0/2 1/3 2/4 2/5 3/6 4/7 5/8 6/9 6/10 ···

Table 2.1: Relative frequencies of heads when repeatedly tossing a fair coin.
1 The pictures of the one pound coin are attributed to Sir Magnus Fluffbrains [CC BY-SA 4.0
([Link] [Link]
[Link] and [Link]

12
A. E. D. Veraart MATH40005: Probability and Statistics Autumn 2023

0.6

0.6
Relative frequency nA ntotal

Relative frequency nA ntotal

0.5

0.5
0.4

0.4
0.3

0.3
0.2

0.2
0.1

0.1
0.0

0.0
0 20 40 60 80 100 0 1000 2000 3000 4000 5000

Number of experiments ntotal Number of experiments ntotal

Figure 2.5: Relative frequencies of heads when repeatedly tossing a fair coin.

2.3.3 Subjective
For an event A, we can assign the probability P(A) according to our personal “degree of belief”. This
could be done according to historical information or local knowledge. This probability will not need to be
the same for each individual. The subjective approach may be difficult to implement in practice, but is a
valid and universal interpretation of probability.

Remark 2.3.5. It is important to remember that all three interpretations of probability depend on assump-
tions about experimental conditions.

13
Chapter 3

Counting

The material of this chapter is based on Blitzstein & Hwang (2019), p.8-28, Anderson et al. (2018),
p.4-11.

In order to compute (naive) probabilities we need to be able to count events in possibly large (but fi-
nite) sample spaces. The area of mathematics which deals with counting is called Combinatorics. We will
study some key ideas from combinatorics and show their interplay with probability theory.

3.1 The multiplication principle

Theorem 3.1.1 (The multiplication principle). Consider two experiments: Experiment A has a possible
outcomes, and Experiment B has b possible outcomes. Then the compound experiment of performing
Experiment A and B (in any order) has ab possible outcomes.
Proof. Without loss of generality we assume that we conduct Experiment A first. We draw a tree diagram
consisting of a branches with one branch for each possible outcome of Experiment A. For each of theses
branches we then generate b branches for each possible outcome of Experiment B. We can then directly
read off that there are = ab possibilities.
b + ··· + b
| {z }
a

We illustrate the proof of the multiplication principle in Figure 3.1 in the case when the outcomes
of Experiment A are labelled as A1 , A2 (for a = 2) and the outcomes of Experiment B are labelled as
B1 , B2 , B3 (for b = 3).

A1 A2

B1 B2 B3 B1 B2 B3

Figure 3.1: Illustration of the proof of the multiplication principle in the case when the outcomes of Exper-
iment A are labelled as A1 , A2 (for a = 2) and the outcomes of Experiment B are labelled as B1 , B2 , B3
(for b = 3).

14
A. E. D. Veraart MATH40005: Probability and Statistics Autumn 2023

Exercise 3.1.2. We flip a fair coin three times. We write ”0” for heads and ”1” for tails. Find the sample
space Ω and PNaive ({ω}) for all ω ∈ Ω. Also, compute the probability that the first and the third flip are
tails.
Proof. Here the sample space is given by

Ω = {(0, 0, 0), (0, 0, 1), . . . , (1, 1, 0), (1, 1, 1)},

which are all the ordered triplets of zeros and ones. By the multiplication principle we have that card(Ω) =
23 = 8, hence PNaive ({ω}) = 1/8 for all ω ∈ Ω.
Let us now consider the event B := the first and the third flip are tails = {(1, 0, 1), (1, 1, 1)}. Then
PNaive (B) = 2/8 = 1/4.

Remark 3.1.3. If we deal with repetitions of experiments (coin toss, rolling a die), the corresponding
sample spaces are given by Cartesian product spaces.
For sets A1 , . . . , An , we define the Cartesian product as

A1 × · · · × An = {(x1 , . . . , xn ) : xi ∈ Ai for i = 1, . . . , n}.

So in our example above, we can write Ω = {0, 1} × {0, 1} × {0, 1} = {0, 1}3 .
End of lecture 2. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

3.2 Power sets

Exercise 3.2.1. Consider a set Ω with card(Ω) = n ∈ N elements. Use the multiplication principle to
show that there are 2n possible subsets of Ω if you include the empty set ∅ and Ω.
Proof. For each element in Ω, you can choose whether or not to include it in the subset, so you have two
options each. Hence you have = 2n possible outcomes.
2 · ·
| {z }· 2
n

Definition 3.2.2 (Power set). A power set of a set A, denoted as P(A) is defined as the set of all possible
subsets of A including ∅ and A.

We have already proven the following result:

Theorem 3.2.3 (Cardinality of the power set). Consider a sample space Ω with card(Ω) < ∞. Then
card(P(Ω)) = 2card(Ω) .
Exercise 3.2.4. Let Ω = {A, B, C}. Find card(P(Ω)) and P(Ω).

Proof. From the theorem above, we know that the corresponding power set P(Ω) consists of card(P(Ω)) =
23 = 8 elements. Also, we have

P(Ω) = {∅, {A}, {B}, {C}, {A, B}, {A, C}, {B, C}, Ω}.

3.3 Sampling with and without replacement

We can use the multiplication principle to derive the number of outcomes when sampling with or without
replacement. In the following, we will state the results in the context when we have an urn with n ∈ N
balls which are labelled {1, 2, . . . , n} and we draw k ∈ N balls from the urn.

15
A. E. D. Veraart MATH40005: Probability and Statistics Autumn 2023

Figure 3.2: Consider an urn with n ∈ N balls which are labelled {1, . . . , n}. How many possible ways are
there to draw k ∈ N balls with or without replacement?

3.3.1 Sampling with replacement – ordered

Consider an urn with n ∈ N balls which are labelled {1, . . . , n}. Let k ∈ N. Suppose that you take a ball
out of the urn and write down its number. Then you put it back into the urn (i.e. you replace it). You do this
k times in total and write down the labels in the order in which they appear. The sample space Ω of this
experiment can be expressed in the following way: We denote by S = {1, . . . , n} the labels of the balls in
the urn. A possible outcome of the experiment can be written as ω = (s1 , . . . , sk ), where si denotes the
number of the ith ball for i ∈ {1, . . . , n}. Hence

Ω= = S k = {(s1 , . . . , sk ) : si ∈ S for i = 1, . . . , k}.

S × ··· × S
| {z }
k times

Theorem 3.3.1 (Sampling with replacement). In the case of sampling k balls with replacement from an
urn containing n balls as described above, there are card(Ω) = nk possible outcomes when the order of
the objects matters.
Proof. The result is a direct consequence of the multiplication principle: Each time we draw a ball, there
are n possible outcomes. We carry out this experiment k times, so there are nk ways of obtaining a sample
consisting of k balls.

3.3.2 Sampling without replacement – ordered

Again we consider an urn with n ∈ N balls which are labelled {1, . . . , n}. Let k ∈ N. Suppose that you
take a ball out of the urn and write down its number. Then you remove the ball and do not put it back into
the urn (i.e. you do not replace it). You do this k times in total. Since you are removing balls from the urn
permanently, k cannot be larger than n.
The sample space Ω of this experiment can be expressed in the following way: We denote by S =
{1, . . . , n} the labels of the balls in the urn. Then

Ω = {(s1 , . . . , sk ) : si ∈ S for i = 1, . . . , k, and si ̸= sj if i ̸= j}.

Theorem 3.3.2 (Sampling without replacement). In the case of sampling k balls without replacement from
an urn containing n balls as described above, there are card(Ω) = n(n − 1) · · · (n − (k − 1)) = (n)k
possible outcomes when the order of the objects matters.

Note that we use the convention that (n)1 = n.

Proof. Also this result is a direct consequence of the multiplication principle: The first time, we draw a
ball, there are n possible outcomes, the second time, there are n−1 possible outcomes and so on, and when
we draw the kth ball, there are n − (k − 1) possible labels left for the kth ball. Multiplying the number of
possible outcomes for each sub-experiment together leads to the stated result.

16
A. E. D. Veraart MATH40005: Probability and Statistics Autumn 2023

Definition 3.3.3 (Factorial). Let n ∈ N. The factorial of n, denoted by n!, is defined asQ the product of all
n
natural numbers less than or equal to n, i.e. n! = n · (n − 1) · (n − 2) · · · 3 · 2 · 1 = i=1 i. We define
0! = 1.
Definition 3.3.4 (Descending factorial). For k, n ∈ N with k ≤ n, we define the descending factorial,
Qk−1 Qn
denoted by (n)k as (n)k = n(n − 1) · · · (n − k + 1) = i=0 (n − i) = j=n−k+1 j with the convention
n!
that (n)1 = n. We note that the descending factorial can be expressed as (n)k = (n−k)! .

The factorial arises naturally in the context of so-called permutations. Consider the set of numbers
{1, 2, . . . , n}. A permutation brings these numbers into a certain order. As a consequence of Theorem
3.3.2 with k = n, we deduce that the numbers in the set {1, 2, . . . , n} can be arranged in exactly n!
possible ways.

Example 3.3.5. Consider the set {1, 2, 3}. How many permutations (i.e. possible orderings) are there? We
can write (1, 2, 3), (1, 3, 2), (2, 1, 3), (2, 3, 1), (3, 1, 2), (3, 2, 1), so we have 3! = 6 possible permutations.

3.3.3 The birthday problem

Let us now study the famous birthday problem:
Example 3.3.6. Assume there are k ∈ N, k ≤ 365 people in a room and assume that each person’s
birthday is equally likely to be any of the 365 days of the year (with the 29th February excluded). What is
the probability that at least two people in the room have the same birthday?

• Due to our assumption, the naive probability definition is applicable here. First we count how many
possible ways there are to assign birthdays to the k people in the room.
• This problem can be viewed as sampling with replacement, so we have 365k possible birthday com-
binations.

• Next, we need to count how many scenarios there are such that at least two people have the same
birthday. It appears that this is rather challenging...
• What is easier to compute is the complement, i.e. the number of scenarios such that no two people
share the same birthday. This number can be computed using sampling without replacement, which
leads to (365)k possible outcomes.

Combining the results, we get

PNaive (At least two people in the room have the same birthday)
= 1 − PNaive (All people in the room have distinct birthdays)
(365)k 365 364 365 − (k − 1)
=1− =1− ··· =: f (k).
365k 365 365 365

17
A. E. D. Veraart MATH40005: Probability and Statistics Autumn 2023

1.0
Probability of birthday match
0.8
0.6
0.4
0.2
0.0

0 20 40 60 80 100
k

Figure 3.3: We compute the probability f (k) that, out of k people in a room, at least two share the same
birthday.

We plot the probabilities f (k) for k = 1, . . . , 100 in Figure 3.3. Note that f (22) ≈ 0.476 and f (23) ≈
0.507, so you need to have at least 23 people in the room to have a probability of at least 50% such that at
least two people share the same birthday.
Exercise 3.3.7. A college has 10 non-overlapping time slots for its courses and assigns courses to time
slots randomly and independently. A student randomly chooses three of the courses to enroll in. What is
the probability that there is a conflict in the student’s schedule?
Proof. Using the multiplication principle and the naive probability, we can compute the probability of no
schedule conflict as 10·9·8
103 (= 0.72). So the probability that there is at least one schedule conflict is given
by 1 − 10·9·8
10 3 (= 0.28).

Exercise 3.3.8. A fair die is rolled 6 times. What is the probability that some value is repeated?
Proof. There are 66 possible outcomes when rolling a die 6 times. There are 6! configurations where each
number appears exactly once. Hence P(no value repeated) = 66!6 and P(at least one value is repeated) =
1 − 66!6 (≈ 0.9845).

End of lecture 3. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

3.3.4 Sampling without replacement – unordered

Consider the case of an urn with n ∈ N balls, where we take out k ∈ N (k ≤ n) balls and write down their
labels, which are distinct numbers in {1, . . . , n}. We do not care about the order in which the balls are
collected. (For instance, think of drawing the winning numbers in the lottery!) Note that you could view
this experiment as drawing k balls at once rather than one at a time. Hence the outcome of the experiment
is a subset of size k from S = {1, . . . , n}. Hence we can write Ω = {ω ⊆ S : card(ω) = k}.
Definition 3.3.9 (Binomial coefficient). For any k, n ∈ N ∪ {0},
the binomial coefficient is defined as the
number of subsets of size k for a set of size n. It is denoted by nk and we say “n choose k”.

18
A. E. D. Veraart MATH40005: Probability and Statistics Autumn 2023

Theorem 3.3.10 (Binomial coefficient). For any k, n ∈ N ∪ {0}, we have

 
n n(n − 1) · · · (n − (k − 1)) (n)k n!
= = = .
k k! k! (n − k)!k!


for k ≤ n, and nk = 0 for k > n.
Proof. Consider the setting
of the urn with n balls, where we draw k balls at once as described above.
Clearly, if k > n, then nk = 0. Suppose now that k ≤ n. By Theorem 3.3.2, we know that there are
(n)k possible choices if we draw k balls without replacement and care about the ordering. Now we need to
make an adjustment for the overcounting since we do not care about the order any more. For each subset
of size k, we have k! permutations. So we conclude that when we divide (n)k by k! we have adjusted for
the overcounting and obtain the result.
Example 3.3.11. Recall the binomial theorem, which states that for any x, y ∈ R and n ∈ N we have
n  
X
n n
(x + y) = xk y n−k .
k
k=0

The theorem can be proven as follows: We expand (x + y)n = (x + y) · · · (x + y) in n factors and then
pick either the x or the

y from the first factor, multiply this to either the x or the y of the second factor and
so on. There are nk ways of picking exactly k x’s and each of these choices leads to a term of the form
xk y n−k . Summing over all possible k leads the result.

Exercise 3.3.12. A family has 6 children consisting of 3 boys and 3 girls. Assuming that all birth orders
are equally likely, what is the probability that the 3 eldest children are the 3 girls?
Proof. Label the girls as 1, 2, 3, and the boys as 4, 5, 6. Then the birth order is a permutation of 1, 2, 3, 4,
5, 6. So, 236514 means that child 2 was born first, then child 3 etc. The number of possible permutations is
6!. For the three girls to be the eldest children, we need a permutation of 1,2,3, followed by a permutation
of 4,5,6. Hence
3!3! 1
P(the 3 girls are the 3 eldest children) = = = 0.05.
6! 20


Alternative proof: There are 63 ways to chose where the three girls appear in the birth order (without
taking ordering of the girls into account). Of these cases, there is only one where the three girls are the
three eldest children. Hence
1 3!3! 1
P(the 3 girls are the 3 eldest children) =

6 = 6! = 20 = 0.05.
3

Exercise 3.3.13. Consider a group of four people.

1. How many ways are there to choose a two-person committee?

2. How many ways are there to break the people into two teams of two?
Proof. Part 1: We use two approaches to show that there are 6 possibilities:
• We could list all possibilities: Label the people as 1, 2, 3, 4. Then the possibilities are

{1, 2}, {1, 3}, {1, 4}, {2, 3}, {2, 4}, {3, 4},

i.e. we have 6 possibilities.

19
A. E. D. Veraart MATH40005: Probability and Statistics Autumn 2023

• Alternatively, we can use the multiplication rule and account for overcounting:
There are 4 possibilities for choosing the first person on the committee and 3 to choose the second
person. Note, however, that this counts every possibility twice since picking 1 and 2 is the same as
picking 2 and 1. We have overcounted by a factor of 2. So the number of possibilities is given by
  
4·3 4
=6 = .
2 2

Part 2: We use three approaches to show that there are 3 possibilities:

• Again, we could list all possibilities and count them:

{1, 2}, {3, 4}; {1, 3}, {2, 4}; {1, 4}, {2, 3},

which is rather tedious...

• Alternatively, we could just specify the first person’s teammate, then the other team is determined.
There are 3 ways of doing this.
• Or, we use Part 1) to deduce that there are 6 possibilities of choosing one team. Here we overcount
by a factor of 2 since picking {1, 2} as a team is equivalent to picking {3, 4} as a team. Hence we
have 6/2 = 3 possibilities.

Exercise 3.3.14. How many ways are there to permute the letters in the word STATISTICS? Note that there
are 10 letter in total, ”S” and ”T” appear three times, ”I” twice and ”A” and ”C” once.

Proof. • Approach 1: There are 10 positions in total, first we choose the 3 positions for the ”S”s out
of 10, then the 3 positions for the ”T”s out of the remaining 7 positions etc. Hence we get

          = 50400
10 7 4 3 1
3 3 1 2 1
| {z } |{z} |{z} |{z} |{z}
”S” ”T ” ”A” ”I” ”C”

• Approach 2: Start with 10! permutations of the 10 letters and adjust for overcounting:

10!
= 50400.
3!3!2!

3.3.5 Sampling with replacement – unordered

As before, let k, n ∈ N. Let us consider the case that we have an urn with n balls with labels in {1, . . . , n},
and we want to choose k balls one after the other with replacement. Assuming the order of the balls does
not matter, how many possible outcomes are there? The sample space for our experiment is given by
Ω = {ω : ω is a k − element multiset with elements from {1, . . . , n}}.
Theorem 3.3.15 (Sampling with replacement when the order does not matter). In the sampling with re-
placement problem
described above and assuming that the order of the balls does not matter, we have
card(Ω) = n+k−1
k possibilities.
The proof of the theorem relies on the so-called stars and bars argument presented in Feller (1957),
which is a graphical tool for counting. Note that since we sample with replacement, we can have that
k > n.

20
A. E. D. Veraart MATH40005: Probability and Statistics Autumn 2023

Proof. Consider n distinguishable boxes representing the n distinct labels of the balls in the urn. We
can draw these n boxes using n + 1 bars which represent the walls of the boxes, see Figure 3.4 for an
illustration. Next, we have k indistinguishable balls, which we now draw as stars and place them into the
boxes, i.e. between the bars.

Figure 3.4:

Illustration

of the stars and bars method in the case when n = 6 and k = 10. Here we have
n+k−1 15
k = 10 = 3003 possible outcomes.

I.e. we can view the stars as “check marks” which count how often a particular label gets selected.
Between the two outer walls (i.e. the first and the last bar), which are fixed, there are n − 1 bars and k stars,
i.e. n + k − 1 symbols which can be arranged in any possible order. Out of the n + k − 1 possible positions,

we choose k positions for the stars and fill the remaining positions with bars. Hence card(Ω) = n+k−1 k .
Note that we could have picked the n − 1 bars instead and then filled in the remaining stars, which leads to
the identity
   
n+k−1 n+k−1
card(Ω) = = .
k n−1

It is important to remember that you should not use the above result in connection with the naive
probability since the unordered samples are typically not equally likely.
Exercise 3.3.16. [Exam question 2020] How many possibilities are there to write the number 7 as an
ordered sum of 3 positive integers? [E.g. 7=1+3+3 would be one possible case and 7=3+1+3 would be
another case.]
Proof. We present an elementary solution here: We can write down all possible 3-tuples of numbers which
sum up to 7:
• (1, 3, 3) with 3!/2! = 3 possible arrangements (where we adjusted for over-counting since the num-
ber 3 appears twice),

• (1, 1, 5) with 3!/2! = 3 possible arrangements,

• (2, 2, 3) with 3!/2! = 3 possible arrangements,
• (1, 2, 4) with 3! = 6 possible arrangements.

Hence there are 3 + 3 + 3 + 6 = 15 possibilities of writing the number 7 as a sum of 3 positive integers.
Exercise 3.3.17. [Exam question 2020] Let k, n ∈ N = {1, 2, . . .}. How many possibilities are there to
write the number k as an ordered sum of n positive integers?

21
A. E. D. Veraart MATH40005: Probability and Statistics Autumn 2023

Proof. We can use a stars and bars argument: There are 0 possibilities if k < n.
Now suppose that k ≥ n. Then we represent the number k as k stars which we would like to place
in n bins such that each bin contains at least one object (since we have the restriction that all addends are
positive integers). We can first write the k stars in one line. Then there are k − 1 possible gaps between
the stars, where a bar could be inserted to separate
the bins. We need to select n − 1 gaps out of the k − 1
k−1
gaps, to create the n bins, so in total we have n−1 possibilities.
The above proof can be further illustrated by associating the stars with ”1”s and then writing

k = 1 + 1 + 1 + · · · + 1,

i.e. we express k as a sum of k ”1”s. Each ”+” sign is a possible location for a bar. There are k − 1 ”+”signs
and we need to choose n − 1 of them which leads to the result.
For instance, in the case of Exercise 3.3.16, one possible configuration is ∗| ∗ ∗ ∗ | ∗ ∗∗, where we have
1 star in the first bin, followed by 3 stars in the next bin, followed by 3 stars in the last bin. This can be
viewed is writing

7 = 1 + (1 + 1 + 1) + (1 + 1 + 1).

3.3.6 Summary table

We can summarise the preceding discussion as follows: Consider an urn with n ∈ N balls, where you draw
k ∈ N balls. The number of possible outcomes is then given as follows:

Ordered Unordered
n+k−1


With replacement nk k
Without replacement

n
(for k ≤ n only) (n)k k

End of lecture 4. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

22
Chapter 4

Axiomatic definition of probability

The material of this chapter is based on Blitzstein & Hwang (2019), p.21-26, Anderson et al. (2018),
p.1-21, Grimmett & Welsh (1986), p.3-9.
In this chapter we will focus on an axiomatic definition of probability and derive some of the key
properties of the probability measure.

4.1 The event space F

Recall Definition 2.2.1, which states that the sample space Ω is defined as the set of all possible outcomes
of an experiment. In our previous discussion we assigned (naive) probabilities to events which were subsets
of Ω. We typically denote by F the event space, which contains the events we are allowed to consider. This
is a rather vague statement, which we will need to make precise! In fact, in probability theory, we always
require that the event space F is a so-called σ-algebra (which is the same as a σ-field).
Definition 4.1.1 (Algebra and σ-algebra). A collection of subsets of Ω denoted by F is called
1. an algebra (or a field) on Ω if

(a) ∅ ∈ F,
(b) F is closed under complements, i.e. A ∈ F ⇒ Ac ∈ F, and
(c) F is closed under unions of pairs of members, i.e. A1 , A2 ∈ F ⇒ A1 ∪ A2 ∈ F.
2. an σ-algebra (or a σ-field) on Ω if

(a) ∅ ∈ F,
(b) F is closed under complements, i.e. A ∈ F ⇒ Ac ∈ F, and
(c) F is closed under countable union, i.e. for any countable 1 index set I, we have Ai ∈ F, for
all i ∈ I ⇒ ∪i∈I Ai ∈ F.

Let us consider two examples of algebras, which are not a σ-algebras.

Example 4.1.2. Let Ω = R and define F as the collection of finite disjoint unions of sets of the form
(a, b], (−∞, a], (b, ∞) for all a, b ∈ R.
Then F is an algebra:

1. ∅ = (a, b] ∈ F for b < a,

1 Recall Definition 2.2.11: A countable index set could have countably infinitely many values, e.g. when I = N or finitely many

values, e.g. when I = {1, 2}.

23
A. E. D. Veraart MATH40005: Probability and Statistics Autumn 2023

2. ∅c = R = (−∞, a] ∪ (a, ∞) ∈ F. Also, for all a < b we have

(a, b]c = (−∞, a] ∪ (b, ∞) ∈ F,

(−∞, a]c = (a, ∞) ∈ F,
(b, ∞)c = (−∞, b] ∈ F.

These results can be further extended to the corresponding finite disjoint unions of sets specified
above and imply that for any A ∈ F, we also have that Ac ∈ F.

3. The closedness under unions of pairs follows directly from the definition of F.
Hence F is an algebra.
However, it is NOT a σ-algebra. To see this, note that
 
1
Ai = 0, 1 − ∈ F,
i

but
∞
[      
1 1 1
Ai = 0, 1 − ∪ 0, 1 − ∪ 0, 1 − ∪ · · · = (0, 1) ̸∈ F.
i=1
1 2 3

Example 4.1.3. Let Ω = R and F = {A ⊂ Ω : A is finite or Ac is finite}. Then F is an algebra since:

1. ∅ ∈ F since it is finite,
2. For any A ∈ F, we have either

A is finite ⇒ (Ac )c = A is finite, hence Ac ∈ F,

or

Ac is finite ⇒ Ac ∈ F.

3. For any A1 , A2 ∈ F, we have either

A1 , A2 are both finite ⇒ A1 ∪ A2 finite ⇒ A1 ∪ A2 ∈ F,

or at least one Aci is finite for i = 1, 2. Without loss of generality assume that Ac2 is finite. Then, by
De Morgan’s law,

(A1 ∪ A2 )c = (Ac1 ∩ Ac2 ) ⊆ Ac2 is finite ⇒ A1 ∪ A2 ∈ F.

Hence F is an algebra.
However, F is NOT a σ-algebra. To see this, note that
Ai = {i} ∈ F since it is finite, but ∪∞ c
i=1 Ai = N ̸∈ F since it is not finite and N = R \ N is not finite
either!
Note that if we work with Ω = N instead in the above example, we still get that F is an algebra, but not
a σ-algebra. Here we could take Ai = {2i}, the ∪∞ i=1 Ai are the even natural numbers, which are infinite,
and (∪∞ c ∞
i=1 Ai ) are the odd natural numbers, which are infinite, too. So ∪i=1 Ai ̸∈ F.

Remark 4.1.4. 1. Any algebra is closed under finite unions and finite intersections.

2. Any σ-algebra is closed under countable intersections since for any Ai ∈ F, ∩∞ ∞ c c

i=1 Ai = (∪i=1 Ai )
c
and each Ai ∈ F.
3. Any (σ-)algebra contains Ω = ∅c .

24
A. E. D. Veraart MATH40005: Probability and Statistics Autumn 2023

Example 4.1.5. Consider any sample space Ω. Then the so-called trivial σ-algebra is defined as Ftrivial =
{∅, Ω} and the total or power σ-field is defined as F = P(Ω) = {all subsets of Ω}.
Example 4.1.6. Consider a sample space Ω with A ⊆ Ω. Then {∅, Ω, A, Ac } is a σ-algebra (in fact the
smallest σ-algebra including A).

Throughout the course, we shall assume that F is a σ-algebra. This will allow us to consider countable
infinite rather than finite unions. Note that this is clearly more restrictive than assuming that F is an algebra.
So why do we care about algebras at all? In probability we typically define a probability measure first
on an algebra and extend it to a σ-algebra. The details behind this construction are beyond the scope of this
introductory course, but you can learn more about this in our measure theory course.

4.2 Definition of probabilities and basic properties

4.2.1 Probability measure and probability space
Definition 4.2.1 (Probability measure). A mapping P : F → R is called a probability measure on (Ω, F)
if it satisfies three conditions:
(i) P(A) ≥ 0 for all events A ∈ F,

(ii) P(Ω) = 1,
(iii) For any countable2 sequence of disjoint events (Ai )i∈I with Ai ∈ F, for all i ∈ I, we have
!
[ X
P Ai = P(Ai ).
i∈I i∈I

[Note that by ”disjoint events” we mean that Ai ∩ Aj = ∅ for all i ̸= j.]

Definition 4.2.2 (Probability space). We define a probability space as the triplet (Ω, F, P), where Ω is a
set (the sample space), F is a σ-algebra (the event space) consisting of subsets of Ω and P is a probability
measure on (Ω, F).

4.2.2 Basic properties of the probability measure

Theorem 4.2.3. Consider a probability space (Ω, F, P). Then, for any events A, B ∈ F, we have
1. P(Ac ) = 1 − P(A).

2. If A ⊆ B, then P(A) ≤ P(B).

3. P(A ∪ B) = P(A) + P(B) − P(A ∩ B).
Proof. 1. Show that P(Ac ) = 1−P(A): Since A and Ac are disjoint and Ω = A∪Ac , the second axiom
(ii) leads to 1 = P(Ω) = P(A ∪ Ac ) and the third axiom (iii) leads to P(A ∪ Ac ) = P(A) + P(Ac ).
Altogether, we have P(A) + P(Ac ) = 1.
2. Show that, if A ⊆ B, then P(A) ≤ P(B): We can express B as a union of two disjoint sets:
B = (B ∩ A) ∪ (B ∩ Ac ). Since A ⊆ B, we have that B ∩ A = A. So, using the axiom (iii), we
have that

P(B) = P(B ∩ A) + P(B ∩ Ac ) = P(A) + P(B ∩ Ac ).

2 Recall Definition 2.2.11: A countable index set could have countably infinitely many values, e.g. when I = N or finitely many

values, e.g. when I = {1, 2}.

25
A. E. D. Veraart MATH40005: Probability and Statistics Autumn 2023

Using the fact that the probability measure is nonnegative (axiom (i)), we conclude that P(B ∩Ac ) ≥
0 and hence

P(B) = P(A) + P(B ∩ Ac ) ≥ P(A).

3. Show P(A ∪ B) = P(A) + P(B) − P(A ∩ B): We express A and B in terms of disjoint unions:

A = (A ∩ B) ∪ (A ∩ B c ), B = (B ∩ A) ∪ (B ∩ Ac ).

By axiom (iii), we have that

P(A) = P(A ∩ B) + P(A ∩ B c ), P(B) = P(B ∩ A) + P(B ∩ Ac ).

Hence

P(A) + P(B) − P(A ∩ B) = P(A ∩ B) + P(A ∩ B c ) + P(B ∩ Ac ).

Also the union A ∪ B can be expressed as a union of disjoint sets

A ∪ B = (A ∩ B) ∪ (A ∩ B c ) ∪ (B ∩ Ac ).

By axiom (iii), we have that

P(A ∪ B) = P(A ∩ B) + P(A ∩ B c ) + P(B ∩ Ac ).

So, indeed,

P(A ∪ B) = P(A) + P(B) − P(A ∩ B).

Some graphical illustrations for the arguments presented in the above proof are given in Figure 4.1.

Ω Ω

B B
A∩B B ∩ Ac A ∩ Bc A∩B B ∩ Ac
A A

(a) (b)

Figure 4.1: We consider a sample space Ω with subsets A, B ⊆ Ω. Figures 4.1a and 4.1b depict the settings
we are considering in the proofs of Theorem 4.2.3 part 2 and 3, respectively.

End of lecture 5. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Remark 4.2.4. The above theorem implies that P(∅) = 0. To see this, note that Ω and ∅ are disjoint since
Ω ∩ ∅ = ∅. Also, Ω = Ω ∪ ∅. So, altogether we have 1 = P(Ω) = P(Ω ∪ ∅) = P(Ω) + P(∅) = 1 + P(∅).
Hence, P(∅) = 0.

26
A. E. D. Veraart MATH40005: Probability and Statistics Autumn 2023

4.2.3 Examples
Example 4.2.5. We continue with the classical example of flipping a fair coin.

Figure 4.2: Flipping a fair coin.

We write H for heads and T for tail. The sample space is given by Ω = {H, T }. The event space can be
taken as F = {∅, {H}, {T }, Ω} which is the collection of all subsets of Ω. Since we are considering a
”fair” coin, we have that P({H}) = P({T }) = 21 , where we typically shorten the notation to:

1
P(H) = P(T ) = .
2
Moreover, we have P(∅) = 0 and P(Ω) = 1.
P e−λ λx
Exercise 4.2.6. Let Ω = N ∪ {0}, F = P(N ∪ {0}), P : F → R with P(A) = x∈A x! for λ > 0.
Show that (Ω, F, P) is a probability space.

Proof. We know from lectures that the power set is a σ-algebra. So we only need to show that P is a
probability measure. We note that P : F → R and hence it remains to check the three axioms of the
definition of a probability measure:
P∞ x
Axiom (ii) We use the fact that x=0 λx! = eλ , to deduce that
∞
X λx
P(Ω) = e−λ = eλ e−λ = 1.
x=0
x!

Axiom (i) Since λ > 0, we have that for any A ∈ F,

X λx X∞
λx −λ
0 ≤ P(A) = e−λ ≤ e = 1.
x! x=0
x!
x∈A

Axiom (iii) Let Ai ∈ F for i ∈ I be disjoint, then

!
[ X λx X X λx X
P Ai = e−λ = e−λ = P(Ai ),
x! x!
i∈I x∈∪i∈I Ai i∈I x∈Ai i∈I

where we interchanged the order of summation (to be covered in Analysis!).

27
Chapter 5

Conditional probabilities

The material of this chapter is based on Blitzstein & Hwang (2019), p.45-63, Anderson et al. (2018),
p.43-56, Grimmett & Welsh (1986), p.11-12.
After having introduced the axiomatic definition of a probability measure, we will now turn our atten-
tion to so-called conditional probabilities. We will learn how probabilities can be computed based on some
given evidence. Conditional probabilities play a key role in almost all subsequent probability and statistics
course und you will learn that they constitute a powerful concept for computing unconditional probabilities
as well.

5.1 Definition
Definition 5.1.1 (Conditional probability). Consider a probability space (Ω, F, P). Consider events A, B ∈
F with P(B) > 0. Then the conditional probability of A given B, denoted by P(A|B), is defined as
P(A ∩ B)
P(A|B) = .
P(B)
Remark 5.1.2. Interpretation: You could call P(A) the prior probability of event A and P(A|B) the
posterior probability of A. Here we view B as additional evidence which becomes available, and the
prior probability is formulated without knowledge of the additional evidence and the posterior probability
describes the updated probability based on the additional evidence.
Let us now show that the conditional probability measure does indeed satisfy the axioms of a probability
measure:
Theorem 5.1.3 (Conditional probability). Let B ∈ F with P(B) > 0 and define Q : F → R by Q(A) =
P(A|B). Then (Ω, F, Q) is a probability space.
Proof. The only thing we need to show is that Q satisfies the axioms of a probability measure on (Ω, F).
Axiom (i): Since P is a probability measure satisfying axiom (i), we deduce that, for any A ∈ F,
P(A ∩ B)
Q(A) = ≥ 0.
P(B)
Axiom (ii):
P(Ω ∩ B) P(B)
Q(Ω) = = = 1.
P(B) P(B)
Axiom (iii): Consider disjoint events Ai ∈ F for i ∈ I. Then
! S

S
P
[ P i∈I Ai ∩ B P ∈I (Ai ∩ B) P(Ai ∩ B)
Q Ai = = = i∈I
P(B) P(B) P(B)
i∈I

28
A. E. D. Veraart MATH40005: Probability and Statistics Autumn 2023

X P(Ai ∩ B) X
= = Q(Ai ),
P(B)
i∈I i∈I

where for the third equality we used the fact that the events Ai ∩ B ∈ F are disjoint and that P satisfies
axiom (iii).

5.2 Examples
Example 5.2.1. We roll a single fair die. Hence Ω = {1, 2, 3, 4, 5, 6}. What is the probability that the
score is greater than 3 given that the score is even? We define the events

B = {ω ∈ Ω : ω is even} = {2, 4, 6},

A = {ω ∈ Ω : ω > 3} = {4, 5, 6}.

Then, A ∩ B = {4, 6}. We have P(A) = 1/2, P(B) = 1/2 and P(A ∩ B) = 2/6 = 1/3.

P(A ∩ B) 1/3 2
P(A|B) = = = .
P(B) 1/2 3

Example 5.2.2. A family has two children. Assume that Female (F)/Male (M) are equally likely and
successive births are independent. We write Ω = {F F, F M, M F, M M }, where e.g. F M stands for the
event that the first child is Female and the second child is Male. Note that all four outcomes are equally
likely, so P(ω) = 1/4 for all ω ∈ Ω.
1. If one child is a boy, what is the probability that both children are boys?
2. If the eldest is a boy, what is the probability that both children are boys?
Let A = {M M } both male, B = {M M, M F, F M } at least one male, C = {M M, M F } eldest is male.
Then
P(A ∩ B) 1/4 1
P(A|B) = = = ,
P(B) 3/4 3

and

P(A ∩ C) 1/4 1
P(A|C) = = = .
P(C) 2/4 2

Remark 5.2.3. Consider the case of a finite state space Ω where all events are equally likely and hence
the classical interpretation of probability can be used. Then for two events A, B ⊆ Ω, we have that
card(A∩B)
card(A ∩ B) card(Ω) P(A ∩ B)
P(A|B) = = card(B)
= .
card(B) P(B)
card(Ω)

5.3 Multiplication rule

Suppose that A, B ∈ F with P(A) > 0, P(B) > 0. Then

P(A ∩ B)
P(A|B) = ⇔ P(A ∩ B) = P(A|B)P(B).
P(B)

29
A. E. D. Veraart MATH40005: Probability and Statistics Autumn 2023

Similarly,

P(A ∩ B)
P(B|A) = ⇔ P(A ∩ B) = P(B|A)P(A).
P(A)

Let us extend the above result to three events: Let C ∈ F with P(C) > 0. Then

P(A ∩ B ∩ C) = P(A|B ∩ C)P(B ∩ C) = P(A|B ∩ C)P(B|C)P(C).

Repeating the arguments above, we obtain the following result:

Theorem 5.3.1 (Multiplication rule). Let n ∈ N, then for any events A1 , . . . , An with P(A2 ∩· · ·∩An ) > 0,
we have

P(A1 ∩ · · · ∩ An ) = P(A1 |A2 ∩ · · · ∩ An )P(A2 |A3 ∩ · · · ∩ An ) · · ·

P(An−2 |An−1 ∩ An )P(An−1 |An )P(An ),

where the right hand side is a product of n terms.

Clearly, the ordering of the events in the theorem above can be changed, so there are n! possible
formulations of the above theorem!
End of lecture 6. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

5.4 Bayes’ rule and law of total probability

5.4.1 Bayes’ rule
Bayes’ rule is a famous and extremely useful result for computing conditional probabilities:
Theorem 5.4.1. Let A, B ∈ F with P(A) > 0, P(B) > 0. Then

P(B|A)P(A)
P(A|B) =
P(B)

Proof. This is an immediate consequence of the definition of conditional probability and the multiplication
rule.

5.4.2 Law of total probability

Next we study the so-called law of total probability, which is an extremely useful tool for computing
complicated probabilities in terms of simpler pieces based on conditional probabilities.

Definition 5.4.2 (Partition). A partition of the sample space Ω is a collection {Bi : i ∈ I} (for a S
countable
index set I) of disjoint events (meaning that Bi ∈ F and Bi ∩ Bj = ∅ for i ̸= j) such that Ω = i∈I Bi .
Remark 5.4.3. We note that a partition of the sample space is often not unique and the choice of the
particular partition typically very much depends on the problem we want to solve!

Theorem 5.4.4 (Law of total probability). Let {Bi : i ∈ I} denote a partition of Ω, with P(Bi ) > 0 for
all i ∈ I. Then, for all A ∈ F,
X X
P(A) = P(A ∩ Bi ) = P(A|Bi )P(Bi ).
i∈I i∈I

30
A. E. D. Veraart MATH40005: Probability and Statistics Autumn 2023

Proof. Using the properties of a partition and the distributivity property, we deduce that
!
[ [
A=A∩Ω=A∩ Bi = (A ∩ Bi ),
i∈I i∈I

where the events A ∩ Bi are disjoint. Using axiom (iii) of the definition of the probability measure leads to
X
P(A) = P(A ∩ Bi ),
i∈I

and using the multiplication formula leads to

X X
P(A) = P(A ∩ Bi ) = P(A|Bi )P(Bi ).
i∈I i∈I

A ∩ B1 A ∩ B2 A ∩ B3
A A ∩ B4

B1 B2 B3 B4

P4
Figure 5.1: Illustration of the law of total probability. Here we have that P(A) = i=1 P(A ∩ Bi ).

5.4.3 General Bayes’ rule

When we combined Bayes’ rule with the law of total probability, we get the following useful result:

Theorem 5.4.5. Consider a partition {Bi : i ∈ I} of Ω with P(Bi ) > 0 for all i ∈ I, then for any event
A ∈ F with P(A) > 0, we have

P(A|Bi )P(Bi ) P(A|Bi )P(Bi )

P(Bi |A) = =P .
P(A) k∈I P(A|Bk )P(Bk )

5.4.4 Bayes’ rule and law of total probability with additional conditioning
The Bayes’ rule and the law of total probability also hold with extra conditioning.
Theorem 5.4.6 ( Bayes’ rule with extra conditioning). For events A, B, E with P(A∩E) > 0, P(B ∩E) >
0, we have
P(B|A ∩ E)P(A|E)
P(A|B ∩ E) = .
P(B|E)

31
A. E. D. Veraart MATH40005: Probability and Statistics Autumn 2023

Theorem 5.4.7 (Law of total probability with additional conditioning). Consider events A, E with P(E) >
0 and let {Bi : i ∈ I} denote a partition of Ω, with P(Bi ∩ E) > 0 for all i ∈ I. Then,
X P(A ∩ Bi ∩ E) X
P(A|E) = = P(A|Bi ∩ E)P(Bi |E).
P(E)
i∈I i∈I

The proof of these two results is left as an exercise, see Exercise 3- 3.

5.5 Examples
5.5.1 Examples: Cards and marbles
Example 5.5.1. We pick 2 cards at random from a well-shuffled 52-card deck. We consider the event
E:=”2nd card is red”. What is P(E)?
We use the law of total probability to deduce that

P(E) = P(E| 1st card red)P(1st card red) + P(E|1st card black)P(1st card black)
25 1 26 1 1
= · + · = .
51 2 51 2 2
Example 5.5.2. You have three bags that each contain 100 marbles. Bag 1 has 70 red and 30 green
marbles. Bag 2 has 60 red and 40 green marbles. Bag 3 has 50 red and 50 green marbles. 1) If you choose
one bag at random and then pick a marble at random from the chosen bag. What is the probability that the
chosen marble is red? 2) Suppose that the chosen marble was red, what is the probability that bag 1 was
chosen?

1. We define the events Bi :=bag i was chosen, P(Bi ) = 31 , i = 1, 2, 3. R := marble is red. Then
7 6 5
P(R|B1 ) = , P(R|B2 ) = , P(R|B3 ) = ,
10 10 10
We use the law of total probability:
 
7 6 5 1
P(R) = P(R|B1 )P(B1 ) + P(R|B2 )P(B2 ) + P(R|B3 )P(B3 ) = + + = 0.6.
10 10 10 3

2. We use Bayes’ rule:

7 1
P(R|B1 )P(B1 ) 10 · 3 7 1
P(B1 |R) = = 6 = ≈ 0.38 > .
P(R) 10
18 3

5.5.2 Example: Testing for a rare disease

Example 5.5.3. Let us look in more detail into visualisation of conditional probabilities, which often arise
in medical screening tests. We quote the example from the article (Spiegelhalter et al. 2011, p. 1396).
Consider a
‘mammography test on a population with a 1% prevalence of breast cancer. The test is positive
for around 90% of women with cancer, but it is also positive for around 10% of women without
cancer’. ((Spiegelhalter et al. 2011, p. 1396))
What is the probability that a woman whose mammography test is positive has breast cancer?
Define the events: B := breast cancer present; T P = test is positive. We know that P(B) = 0.01 and
P(T P |B) = 0.9 and P(T P |B c ) = 0.1. We can derive that P(B c ) = 1 − P(B) = 0.99. Further, by the
law of total probability,

P(T P ) = P(T P |B)P(B) + P(T P |B c )P(B c )

32
A. E. D. Veraart MATH40005: Probability and Statistics Autumn 2023

= 0.9 · 0.01 + 0.1 · 0.99 = 0.108.

Using Bayes’ Theorem, we get

P(B ∩ T P ) P(T P |B)P(B) 0.9 · 0.01

P(B|T P ) = = = ≈ 8%
P(T P ) P(T P ) 0.108

33
A. E. D. Veraart MATH40005: Probability and Statistics Autumn 2023

Figure 5.2: Visualising conditional probabilities in the case of testing for a rare disease. This picture is a
copy of Figure 4 in the article Spiegelhalter et al. (2011).

34
A. E. D. Veraart MATH40005: Probability and Statistics Autumn 2023

5.5.3 Example: Monty Hall – Conditioning on the missing information

As a general strategy, we typically use conditioning on the information we wish we had to make our
probability computations easier. To illustrate this strategy, we consider a famous example. In the TV Game
show Let’s make a deal, hosted by Monty Hall, a contestant selects one of three doors; behind one of the
doors there is a prize (a car), and behind the other two there are no prizes (in fact, there are goats!). After
the contestant selects a door, the game-show host opens one of the remaining doors, and reveals that there
is no prize behind it. The host then asks the contestant whether they want to SWITCH their choice to
the other unopened door, or STICK to their original choice. Is it probabilistically advantageous for the
contestant to SWITCH doors, or is the probability of winning the prize the same whether they STICK or
SWITCH ? (Assume that the host selects a door to open, from those available, with equal probability).

Figure 5.3: In the Monty Hall example it is advantageous to SWITCH!

Label the doors 1, 2 and 3 and assume without loss of generality that the candidate selects door 1. Monty
Hall then opens a door and reveals a goat. When deciding whether or not to switch, which information
would the contestant like to have? She would like to know the location of the car.
So let us consider the partition: {Ci , i = 1, 2, 3} where Ci is the event that the car is behind the ith
door (for i = 1, 2, 3). Also, we denote by H2 the event that Monty Hall opens door 2. Then P(C1 ) =
P(C2 ) = P(C3 ) = 1/3 and P(H2 |C1 ) = 1/2, P(H2 |C2 ) = 0 and P(H2 |C3 ) = 1. We want to compare
the probabilities of P(C1 |H2 ) (STICK) with P(C3 |H2 ) (SWITCH). Using the law of total probability, we
have

P(H2 ) = P(H2 |C1 )P(C1 ) + P(H2 |C2 )P(C2 ) + P(H2 |C3 )P(C3 )
11 1 1 1
= +0 +1 = .
23 3 3 2

35
A. E. D. Veraart MATH40005: Probability and Statistics Autumn 2023

Then, using the (general) Bayes’ rule implies that

11
P(H2 |C1 )P(C1 ) 23 1
P(C1 |H2 ) = = 1 = .
P(H2 ) 2
3

Similarly

P(H2 |C3 )P(C3 ) 11 2

P(C3 |H2 ) = = 13 = .
P(H2 ) 2
3

So it is better to SWITCH, see Figure 5.3.

End of lecture 7. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

36
Chapter 6

Independence

The material of this chapter is based on Blitzstein & Hwang (2019), p.63-65, Anderson et al. (2018),
p.51-56, Grimmett & Welsh (1986), p.12-16.

6.1 Independence of events

We will call two events A, B ∈ F independent if the occurrence of one of them does not affect the
probability that the other one occurs, meaning that, if P(A) > 0, P(B) > 0,
P(A|B) = P(A), and P(B|A) = P(B). (6.1.1)
Now, recall the definition of the conditional probability as
P(A ∩ B)
P(A|B) = .
P(B)
Hence, the following definition appears suitable:
Definition 6.1.1 (Independent events). The events A, B are called independent if
P(A ∩ B) = P(A)P(B), (6.1.2)
and dependent otherwise.
Remark 6.1.2. The definition given in equation 6.1.2 is more general than the one in equation 6.1.1 since
it does not require that A and B have nonzero probabilities.
Theorem 6.1.3. If the events A and B are independent, then the same is true for each of the pairs Ac and
B, A and B c , and Ac and B c .
Proof. We only prove that P(Ac ∩ B) = P(Ac )P(B) (the proof for the remaining pairs follows the same
arguments). Let us start from the left hand side of the equation:
P(Ac )P(B) = (1 − P(A))P(B) = P(B) − P(A)P(B).
From the law of total probability, we deduce that
P(B) = P(B ∩ A) + P(B ∩ Ac ),
also
P(A)P(B) = P(A ∩ B),
since A and B are independent. Overall we get
P(Ac )P(B) = P(B ∩ A) + P(B ∩ Ac ) − P(B ∩ A) = P(B ∩ Ac ).

37
A. E. D. Veraart MATH40005: Probability and Statistics Autumn 2023

Let us now generalise the definition of independence to more than two events.
Definition 6.1.4 (Independence of events (general case)). 1. A finite collection of events A1 , . . . , An is
defined to be independent if

P(Ai1 ∩ Ai2 ∩ · · · ∩ Aik ) = P(Ai1 )P(Ai2 ) · · · P(Aik )

for every subcollection {i1 , . . . , ik } of {1, . . . , n}, k = 1, . . . , n.

2. A countable or uncountably infinite collection of events is defined to be independent if each finite
subcollection is independent.

Remark 6.1.5. Note that pairwise independence of (Ai , Aj ) is in general not sufficient to conclude the
independence of (A1 , . . . , An ).
Example 6.1.6. The three events A1 , A2 , A3 are independent if and only if

P(A1 ∩ A2 ∩ A3 ) = P(A1 )P(A2 )P(A3 ),

P(A1 ∩ A2 ) = P(A1 )P(A2 ),
P(A1 ∩ A3 ) = P(A1 )P(A3 ),
P(A2 ∩ A3 ) = P(A2 )P(A3 ),

Example 6.1.7. We roll two fair dice and write the sample space as Ω = {(i, j) : i, j = 1, . . . , 6}. We note
that card(Ω) = 62 = 36 and all outcomes are equally likely. We define three events: A1 = first roll is odd,
A2 = second roll is odd, A3 = sum is odd. Then A1 , A2 , A3 are pairwise independent, but they are not
independent since P(A1 ∩ A2 ∩ A3 ) = 0 ̸= P(A1 )P(A2 )P(A3 ).

6.1.1 Conditional independence of events

Definition 6.1.8 (Conditional independence of events). Consider events A, B, C ∈ F with P(C) > 0.
Then we say that A and B are conditionally independent given C if

P(A ∩ B|C) = P(A|C)P(B|C). (6.1.3)

If we, in addition, assume that P(B ∩ C) > 0, then equation (6.1.3) is equivalent to the condition

P(A|B ∩ C) = P(A|C).

The equivalence can be shown as follows:

P(A ∩ B ∩ C) P(A ∩ B|C)P(C)

P(A|B ∩ C) = = = P(A|C)
P(B ∩ C) P(B ∩ C)
P(B ∩ C)
⇔ P(A ∩ B|C) = P(A|C)
P(C)
⇔ P(A ∩ B|C) = P(A|C)P(B|C).

Example 6.1.9. You have a fair and an ”unfair” coin. The unfair coin lands heads with probability 34 . You
pick one coin at random and toss it three times. It lands heads three times. Given this information, what is
the probability that you picked the fair coin?
Let A:=the chosen coin lands heads three times, F := you picked the fair coin. We want to find P(F |A).
We use the generalised Bayes rule and the fact that
 3  3
1 1 3
P(F ) = = P(F c ), P(A|F ) = , P(A|F c ) = .
2 2 4

38
A. E. D. Veraart MATH40005: Probability and Statistics Autumn 2023

Then:


1 3 1
P(A|F )P(F ) P(A|F )P(F ) 2 · 2
P(F |A) = = =

≈ 0.23.
P(A) P(A|F )P(F ) + P(A|F c )P(F c ) 1 3 1 3 3 1
2 · 2 + 4 · 2
After having seen that it landed heads three times, what is the probability that, if we toss the same coin
a fourth time, it lands heads a fourth time?
Let H := chosen coin lands heads at 4th toss. We want to find P(H|A). Here we use the law of total
probability with extra conditioning:
P(H|A) = P(H|A ∩ F )P(F |A) + P(H|A ∩ F c )P(F c |A).
Note that H and A are conditional independent given F since
 3
1 1 1
P(H ∩ A|F ) = 4 = · = P(H|F )P(A|F ).
2 2 2
Hence P(H|A ∩ F ) = P(H|F ) = 21 . Similarly, P(H|A ∩ F c ) = P(H|F c ) = 43 . Hence
1 3
P(H|A) ≈ · 0.23 + (1 − 0.23) = 0.69.
2 4

6.1.2 Continuity of the probability measure and product rule

Definition 6.1.10. The set difference between two sets A, B ⊆ Ω, denoted by A \ B is defined as
A \ B = {ω ∈ Ω : ω ∈ A and ω ̸∈ B} = A ∩ B c .
Lemma 6.1.11. Any countable union can be written as a countable union of disjoint sets. I.e. let A1 , A2 , · · · ∈
F and define D1 = A1 , D2 = A2 \ A1 , D3 = A3 \ (A1 ∪ A2 ), . . . . Then {Di } is a collection of disjoint
sets and ∪ni=1 Ai = ∪ni=1 Di for n being any positive integer or ∞.
The proof of the Lemma is left as an exercise, see Exercise 3- 5.
Definition 6.1.12 (Increasing and decreasing sets). A sequence of sets (Ai )∞ i=1 is said to increase to A,
i.e. Ai ↑ A, if A1 ⊆ A2 ⊆ · · · and ∪∞ ∞
i=1 Ai = A. Similarly, a sequence of sets (Ai )i=1 is said to decrease
∞
to A, i.e. Ai ↓ A, if A1 ⊇ A2 ⊇ · · · and ∩i=1 Ai = A.
Note that Ai ↑ A if and only if Aci ↓ Ac .

A3
A1 A2

∪ni=1 Ai = A

Figure 6.1: Example of increasing sets in R2 .

39
A. E. D. Veraart MATH40005: Probability and Statistics Autumn 2023

Exercise 6.1.13. Give an example of an increasing and a decreasing sequence of sets.

Many examples are possible. We will be discussing two examples here.
1

Example 6.1.14. Example of an increasing sequence of sets: Let Ω = (0, ∞) and define An := 0, 1 − n .
Then
   
1 2
A1 = (0, 0] = ∅ ⊆ A2 = 0, ⊆ A3 = 0, ⊆ ···
2 3
with
∞
[
An ↑ A := An = (0, 1).
n=1


We note that the sequence of complements is decreasing: Acn ↓ Ac : We have Acn = 1 − n1 , ∞ and
hence
   
1 2
Ac1 = (0, ∞) = Ω ⊇ Ac2 = , ∞ ⊇ Ac3 = ,∞ ⊇ ···
2 3
with
∞
\
Acn ↓ Ac = Acn = [1, ∞).
n=1

Example 6.1.15. Example of an increasing sequence of sets: Let Ω = R and define An := (−∞, n]. Then
A1 = (−∞, 1] ⊆ A2 = (−∞, 2] ⊆ A3 = (−∞, 3] ⊆ · · ·
with
∞
[
An ↑ A := An = Ω = R.
n=1

We note that the sequence of complements is decreasing: Acn ↓ Ac : We have Acn = (n, ∞) and hence
Ac1 = (1, ∞) = Ω ⊇ Ac2 = (2, ∞) ⊇ Ac3 = (3, ∞) ⊇ · · ·
with
∞
\
Acn ↓ Ac = Acn = ∅.
n=1

Next we will state and prove the continuity property of the probability measure1 .

Theorem 6.1.16. If A1 , A2 , · · · ∈ F and Ai ↑ A or Ai ↓ A, then

lim P(Ai ) = P(A).
i→∞

The above theorem states that, for increasing or decreasing sets, we can interchange the limit operation
and the probability measure, i.e. we have
lim P(Ai ) = P( lim Ai ),
i→∞ i→∞

where the set limit on the right hand side needs to be understood as taking an infinite union or intersection
for increasing and decreasing sequences, respectively.
1 Recall that a sequence of real numbers (x ) is said to converge to a real number x if for all ϵ > 0 there exists an n ∈ N such
n 0
that for all n ≥ n0 we have |xn − x| < ϵ.

40
A. E. D. Veraart MATH40005: Probability and Statistics Autumn 2023

Proof. Suppose that Ai ↑ A. Then using Lemma 6.1.11, we write A = ∪∞ ∞

i=1 Ai = ∪i=1 Di , where the
i−1
Di = Ai \ (∪k=1 Ak ) are disjoint. By axiom (iii) of the definition of the probability measure (applied
twice), we deduce that
∞
X n
X
P(A) = P(Di ) = lim P(Di )
n→∞
i=1 i=1
= lim P(∪ni=1 Di ) = lim P(∪ni=1 Ai ) = lim P(An ).
n→∞ n→∞ n→∞

Now let Ai ↓ A. Then Fi = Aci ↑ F = Ac . Then, we deduce from the first part of the proof that
limi→∞ P(Fi ) = P(F ). Using the properties of a probability measure, since P(Fi ) = 1 − P(Ai ) and
P(F ) = 1 − P(A), we deduce that limi→∞ P(Ai ) = P(A).

End of lecture 8. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
We can now formulate the so-called product rule for countable number of independent sets:
Theorem 6.1.17. If A1 , A2 , . . . is a countably infinite set of independent events, then
∞
! ∞
\ Y
P Ai = P(Ai ).
i=1 i=1

The proof of the above theorem relies on the continuity property of the probability measure.

Proof of Theorem 6.1.17. Let Bn = ∩ni=1 Ai . Then Bn ↓ B = ∩∞

i=1 Ai , so by the continuity property of
the probability measure, see Theorem 6.1.16, we deduce that

P(∩∞
i=1 Ai ) = P(B) = lim P(Bn )
n→∞
n
Y ∞
Y
= lim P(∩ni=1 Ai ) = lim P(Ai ) = P(Ai ).
n→∞ n→∞
i=1 i=1

41
Chapter 7

Discrete random variables

The material of this chapter is based on Blitzstein & Hwang (2019), p.103-120, Grimmett & Welsh
(1986), p.24-28.
In this and the following chapter, we will be introducing discrete and continuous random variables and
their distributions.

7.1 Pre-images and their properties

We will be defining random variables using the notion of so-called pre-images which you will study in
Analysis. Here we will briefly review their definition and some key properties which we will need in
probability.
Definition 7.1.1. Consider a function with domain X and co-domain Y, i.e. f : X → Y.
• For any subset A ⊆ X , we define the image of A under f as

f (A) = {y ∈ Y : ∃x ∈ A : f (x) = y} = {f (x) : x ∈ A}.

If A = X , then we call f (X ) = Imf the image of f .

• For any subset B ⊆ Y, we define the pre-image of B under f as

f −1 (B) = {x ∈ X : f (x) ∈ B}.

Please not that the pre-image should not be confused with the inverse function (despite the fact that we
are using the same notation). The pre-image is well-defined for any function, whereas the inverse function
obviously only exists when the function f is invertible.
The definition of the pre-image implies that

x ∈ f −1 (B) ⇔ f (x) ∈ B.

Note that in the case when B is a singleton, i.e. B = {b} for an element b ∈ Y, then we often simplify
the notation to f −1 ({b}) = f −1 (b).
Lemma 7.1.2. For any collection of subsets Bi ⊆ Y, i ∈ I where I denotes an (arbitrary) index set, we
have that
!
[ [
−1
f Bi = f −1 (Bi ).
i∈I i∈I

42
A. E. D. Veraart MATH40005: Probability and Statistics Autumn 2023

Proof. We have that

!
[ [
−1
x∈f Bi ⇔ f (x) ∈ Bi ⇔ ∃ i ∈ I such that f (x) ∈ Bi
i∈I i∈I
[
⇔ ∃ i ∈ I such that x ∈ f −1 (Bi ) ⇔ x ∈ f −1 (Bi ).
i∈I

7.2 Random variables

We consider a probability space (Ω, F, P) and we will think of a random variable (r.v.) as a function from
the sample space to the real numbers R, i.e.

X : Ω → R.

The function needs to satisfy some properties, which we introduce in the formal definition below. Note that
• Despite the name, a random variable is a function and not a variable.

• We typically use capital letters such as X, Y, Z to denote random variables.

• The value of the random variable X at the sample point ω is given by X(ω) and is called a realisation
of X.
• The randomness stems from ω ∈ Ω (we don’t know which outcome ω appears in the random exper-
iment), the mapping itself given by X is deterministic.

7.3 Discrete random variables and probability distributions

Definition 7.3.1 (Discrete random variable). A discrete random variable on the probability space (Ω, F, P)
is defined as a mapping X : Ω → R such that
(i) the image/range of Ω under X denoted by ImX = {X(ω) : ω ∈ Ω} is a countable subset of R,

(ii) X −1 (x) = {ω ∈ Ω : X(ω) = x} ∈ F for all x ∈ R.

Remark 7.3.2. The name discrete stems from the first condition in the above definition, which says that
the random variable can only take countably many values in R. In most applications, we deal with discrete
random variables taking values in (a subset of) N or Z.
Remark 7.3.3. Let us clarify the second condition in the above definition: We note that the set appearing
there is the so-called pre-image of x defined as

X −1 (x) = {ω ∈ Ω : X(ω) = x},

i.e. the set of all ω which X maps to x. We require that this set is an event in F (for all possible x) so that
we can later assign probabilities to these events.

Definition 7.3.4 (Probability mass function). The probability mass function (pmf) of the discrete random
variable X is defined as the function pX : R → [0, 1] given by

pX (x) = P({ω ∈ Ω : X(ω) = x}) = P(X −1 (x)). (7.3.1)

43
A. E. D. Veraart MATH40005: Probability and Statistics Autumn 2023

We typically shorten the notation significantly and write pX (x) = P(X = x). Keep in mind, that this
is short hand notation for equation (7.3.1).
Note that the definition of the pmf implies the following properties:

pX (x) = 0 if x ̸∈ ImX.

Note that for x1 , x2 ∈ ImX with x1 ̸= x2 , then

X −1 (x1 ) ∩ X −1 (x2 ) = {ω ∈ Ω : X(ω) = x1 and X(ω) = x2 } = ∅.

Using axiom (iii) in the definition of the probability measure, we have

!
X X [
−1 −1
pX (x) = P(X (x)) = P X (x)
x∈ImX x∈ImX x∈ImX
!
[
=P {ω ∈ Ω : X(ω) = x} = P(Ω) = 1.
x∈ImX

The above equation is often written as

X
pX (x) = 1,
x∈R

since only countably many values of x result in non-zero values for the pmf and hence non-zero contribu-
tions to the sum.
Example 7.3.5. Consider the experiment where we toss a fair coin twice. Write H for heads and T for
tails. Then Ω = {HH, HT, T H, T T }. Define random variables on (Ω, P(Ω)).
• X = number of heads:

X(HH) = 2, X(HT ) = H(T H) = 1, X(T T ) = 0.

• Y = number of tails: Y = 2 − X.
• I = 1 if first toss lands heads and 0 otherwise.

I(HH) = I(HT ) = 1, I(T H) = I(T T ) = 0.

This is a so-called indicator random variable indicating whether or not the first toss lands heads
(1=”yes”, 0=”no”).
We briefly check that X is a discrete random variable on (Ω, P(Ω)): Clearly, X : Ω → R and
ImX = {0, 1, 2} is finite. Also, X −1 (0) = {T T } ∈ P(Ω), X −1 (1) = {HT, T H} ∈ P(Ω), and
X −1 (2) = {HH} ∈ P(Ω).
For Y and I the arguments are analogous.
We can write 1 for H and 0 for T, then Ω = {(1, 1), (1, 0), (0, 1), (0, 0)} and P(Ω). I.e. ω = (ω1 , ω2 ) ∈
Ω if ωi ∈ {0, 1}, i = 1, 2. Then we can express the three random variables defined above as follows:

X(ω1 , ω2 ) = ω1 + ω2 ,
Y (ω1 , ω2 )) = 2 − ω1 − ω2 ,
I(ω1 , ω2 ) = ω1 .

Note that we can define the event A := {(1, 1), (1, 0)}, then I = IA . Also, note that

P(IA = 1) = P({ω ∈ Ω : IA (ω) = 1}) = P({ω ∈ Ω : ω ∈ A}) = P(A).

44
A. E. D. Veraart MATH40005: Probability and Statistics Autumn 2023

End of lecture 9. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Theorem 7.3.6. Let I denote a countable (index) set. Suppose that S = {si : i ∈ I} is a countable set of
distinct real numbers and {πi : i ∈ I} is a collection of numbers satisfying
X
πi ≥ 0 for all i ∈ I, and πi = 1,
i∈I

then there exists a probability space (Ω, F, P) and a discrete random variable X on that probability space
such that its probability mass function is given by

pX (si ) = πi , for all i ∈ I

pX (s) = 0, if s ̸∈ S.

Proof. This is a constructive proof: Take Ω = S, let F = P(Ω) be the power set (i.e. the set of all subsets
of Ω) and set
X
P(A) = πi for all A ∈ F.
i:si ∈A

The discrete random variable X : Ω → R is then defined as X(ω) = ω for all ω ∈ Ω.

The above theorem is incredibly useful for our further study of discrete random variables. It implies that
we do not need to worry about sample spaces, event spaces and probability measures too much. Instead,
we can just say that we study a random variable X taking the value si with probability πi for i ∈ I and we
know that such a random variable actually exists!

Exercise 7.3.7. Can you think of an example of a probability space (Ω, F, P) and one function X : Ω → R
which is a random variable and one function Y : Ω → R which is not a random variable on that probability
space?
Please try solving the exercise before consulting the model solutions below. There are many possible
examples, one is stated below and you might find other ones which are equally correct!
Proof. Consider the sample space Ω = {1, 2, 3, 4, 5, 6} and define the event space F = {∅, Ω, {1, 3, 5}, {2, 4, 6}}.
Define the probability measure P to be the naive probability measure, i.e. P(A) = card(A)/card(Ω) for
A ∈ F.
(i) Define X : Ω → R such that X(ω) = 1 if ω is even and X(ω) = −1 if ω is odd. Then ImX =
{−1, 1} is finite and X −1 ({−1}) = {1, 3, 5} ∈ F and X −1 ({1}) = {2, 4, 6} ∈ F. For all
x ̸∈ ImX we have that X −1 ({x}) = ∅ ∈ F. Hence X is a discrete random variable.
(ii) Define Y : Ω → R such that Y (ω) = ω. Then e.g. Y −1 ({1}) = {1} ̸∈ F, hence Y is not a
(discrete) random variable with respect to the given sigma-algebra F. (It would be one if we had
chosen F = P(Ω) to be the power sigma-algebra of Ω as in the proof of Theorem 7.3.6!)

7.4 Common discrete distributions

In this section, we will introduce some widely used discrete distributions.

45
A. E. D. Veraart MATH40005: Probability and Statistics Autumn 2023

7.4.1 Bernoulli distribution

Definition 7.4.1 (Bernoulli distribution). A discrete random variable X is said to have Bernoulli distri-
bution with parameter p ∈ (0, 1), if X can only take two possible values, 0 and 1, i.e. ImX = {0, 1}
and

pX (1) = P(X = 1) = p, pX (0) = P(X = 0) = 1 − p, pX (x) = 0 if x ̸∈ 0, 1.

We write X ∼ Bern(p).
Note that for any event there is a natural way of associating a Bernoulli random variable with it: We
can define the so-called indicator variable of the event:
Definition 7.4.2 (Indicator variable). Consider an event A ∈ F, we denote by

1, if ω ∈ A,
IA (ω) =
0, if ω ̸∈ A,

the indicator variable of the event A.

Note that the random variable IA ∼ Bern(p) with p = P(A), since

P(IA = 1) = P(A), P(IA = 0) = P(Ac ) = 1 − P(A), P(IA = x) = 0 for x ̸∈ {0, 1}.

Background: Think of an experiment with two possible outcomes ”success” or ”failure” (but not both).
We call such an experiment a Bernoulli trial. We can think of a Bernoulli random variable as an indicator
of success, where an outcome of 1 represents success and an outcome of 0 represents failure. Hence we
often call the parameter p in the Bernoulli distribution the success probability.

7.4.2 Binomial distribution

Consider a sequence of n ∈ N independent and identical Bernoulli trials with success probability p ∈ (0, 1)
and count the number of successes and denote it by the random variable X [e.g. count the number of heads
when tossing a coin repeatedly]. Then X can take the values ImX = {0, 1, . . . , n}. Let x ∈ ImX, and
suppose we have x successes and n − x failures. Since the trials are independent, the
probability of any
sequence with x successes and n − x failures is px (1 − p)n−x . In total, there are nx possible sequences
with x successes and n − x failures, hence
 
n x
P(X = x) = p (1 − p)n−x .
x

Definition 7.4.3 (Binomial distribution). A discrete random variable X is said to follow the binomial
distribution with parameters n ∈ N and p ∈ (0, 1) if ImX = {0, 1, . . . , n} and
 
n x
P(X = x) = p (1 − p)n−x , for x ∈ {0, 1, . . . , n},
x

and P(X = x) = 0 otherwise. We write X ∼ Bin(n, p).

We depict the probability mass function for three random variables with binomial distribution and
parameters n = 10 and p ∈ {0.25, 0.5, 0.75} in Figure 7.1. We observe that for p = 0.5 the pmf is
symmetric about n/2 = 5 and skewed when p ̸= 0.5. We will show prove this finding on the problem
sheet.

46
A. E. D. Veraart MATH40005: Probability and Statistics Autumn 2023

0.25
0.20
Pmf of Bin(10,0.25)

0.15
0.10
0.05
0.00

0 2 4 6 8 10

(a) P.m.f. of X ∼ Bin(10, 0.25)

0.25
0.20
Pmf of Bin(10,0.5)
0.15
0.10
0.05
0.00

0 2 4 6 8 10
x

(b) P.m.f. of X ∼ Bin(10, 0.5)

0.25
0.20
Pmf of Bin(10,0.75)

0.15
0.10
0.05
0.00

0 2 4 6 8 10

x
47
(c) P.m.f. of X ∼ Bin(10, 0.75)

Figure 7.1: We depict the probability mass function for three random variables with binomial distribution
and parameters n = 10 and p ∈ {0.25, 0.5, 0.75}. Note that for p = 0.5 the pmf is symmetric about 5 and
skewed when p ̸= 0.5.
A. E. D. Veraart MATH40005: Probability and Statistics Autumn 2023

7.4.3 Hypergeometric distribution

Consider an urn filled with N balls, with K ∈ N being white balls and N − K being black. When we
draw n ∈ N balls with replacement, we obtain a Bin(n, K/N ) distribution for the number of white balls
drawn. Suppose now we draw without replacement, then the number of white balls follows the so-called
hypergeometric distribution.
Definition 7.4.4 (Hypergeometric distribution). A discrete random variable X is said to follow the hy-
pergeometric distribution with the three parameters N ∈ N ∪ {0}, K, n ∈ {0, 1, . . . , N } if ImX =
{0, 1, . . . , min(n, K)} and


K N −K


x n−x
P(X = x) = N

, for x ∈ {0, 1, . . . , K} and n − x ∈ {0, 1, . . . , N − K},
n

and P(X = x) = 0 otherwise. We write X ∼ HGeom(N, K, n).

Remark 7.4.5. We think of N as the size of the population, K the number of success states in the population
(e.g. number of white balls), n the number of draws and x is the number of observed successes.
In Figure 7.2 we show how the pmf of the hypergeometric distributions with N = 500 and K = 200
shifts when we increase the number of draws from n = 10 to 30 and then 50.
0.25

HGeom(500,200,10)
HGeom(500,200,30)
HGeom(500,200,50)
0.20
0.15
Pmf

0.10
0.05
0.00

0 10 20 30 40 50

Figure 7.2: This graph shows the probability mass function of the hypergeometric distribution with param-
eters N = 500, K = 200 and n ∈ {10, 30, 50}.

If we would like to show that the probability mass function of the hypergeometric distribution is indeed
a valid probability mass function, we typically use the Vandermonde’s identity which we will study next.

48
A. E. D. Veraart MATH40005: Probability and Statistics Autumn 2023

Lemma 7.4.6 (Vandermonde’s identity). For k, n, m ∈ N ∪ {0}, k ≤ n + m, we have

  Xk   
m+n m n
= .
k i=0
i k−i

m


Remark 7.4.7. Note that we use the convention that, for n, m ∈ N0 , we set n = 0 if n > m.
Proof. Story proof/Combinatorial proof:
Consider selecting a committee of k people from a group of people consisting of m men and n women.
The left hand side describes the number of possibilities of selecting k from m + n people (without replace-
ment, order irrelevant). On the right hand side we consider all possible combinations when we choose i
men out of m men, then we need to choose k −i women out of n women to obtain a committee of k people.
We then need to sum of all possible values of i which gives us the right hand side.
Algebraic proof:
Using the binomial theorem, we get

X
m+n 
m+n k
m+n
(1 + x) = x ,
k
k=0

and also (1 + x)m+n = (1 + x)m (1 + x)n with

Xm   n   m n   
m i X n j X X m n i+j
(1 + x)m (1 + x)n = x x = x .
i=0
i j=0
j i=0 j=0
i j

Next, we change the summation indices: We have that 0 ≤ i ≤ m, 0 ≤ j ≤ n. We set k = i + j, then

0 ≤ k ≤ m + n, 0 ≤ i ≤ k. Hence
n   
m X
X m+n
XX k   
m n m n
(1 + x)m+n = xi+j = xk
i=0 j=0
i j i k−i
k=0 i=0

So, overall, we found that

X
m+n 
m+n k
m+n
XX k  
m n

m+n
(1 + x) = x = xk .
k i=0
i k − i
k=0 k=0

We note that two polynomials are identical if they have the same degree and the corresponding coefficients
are identical, which implies that for all 0 ≤ k ≤ n + m, we have
  Xk   
m+n m n
= .
k i=0
i k−i

7.4.4 Discrete uniform distribution

Definition 7.4.8 (Discrete uniform distribution). Let C denote a finite nonempty set of numbers. We say
that a discrete random variable X follows the discrete uniform distribution on C, i.e. X ∼ DUnif(C), if
ImX = C and
1
P(X = x) = ,
card(C)

for x ∈ C and P(X = x) = 0 otherwise.

49
A. E. D. Veraart MATH40005: Probability and Statistics Autumn 2023

Example 7.4.9. Let C = {1, . . . , n}. If X ∼ DUnif(C), then P(X = x) = 1/n for all x ∈ {1, . . . , n}
and 0 otherwise.

0.20
0.15
Pmf of DUnif({1,2,3,4,5})

0.10
0.05
0.00

0 1 2 3 4 5 6 7

Figure 7.3: This graph shows the probability mass function of the discrete uniform distribution on the set
C = {1, 2, 3, 4, 5}.

7.4.5 Poisson distribution

We will now introduce the Poisson distribution which is widely used for counting the number of events/-
successes in a certain time period, e.g. the number of earthquakes in some region in the world.

Definition 7.4.10 (Poisson distribution). A discrete random variable X is said to follow the Poisson dis-
tribution with parameter λ > 0, i.e. X ∼ Poi(λ), if ImX = {0, 1, 2, . . .} = N ∪ {0} and

λx −λ
P(X = x) = e , for x = 0, 1, 2, . . . .
x!
We typically call the parameter λ in the Poisson distribution the rate or intensity [of the occurrence of
(rare) events].

50
A. E. D. Veraart MATH40005: Probability and Statistics Autumn 2023

Pois(1)
Pois(5)
Pois(10)
0.3
0.2
Pmf

0.1
0.0

0 5 10 15 20

Figure 7.4: This graph shows the probability mass function of the Poisson distribution with three different
rate parameters: λ ∈ {1, 5, 10}.

7.4.6 Geometric distribution

Definition 7.4.11 (Geometric distribution). A discrete random variable X is said to follow the geometric
distribution with parameter p ∈ (0, 1), i.e. X ∼ Geom(p) if ImX = N and

P(X = x) = (1 − p)x−1 p, for x = 1, 2, . . . .

We can think of an experiment where we carry out repeated (independent) Bernoulli trials with success
probability p. We stop the experiment after the first success. We denote by X the number of trials to obtain
the first success. Then we obtain that X ∼ Geom(p). Warning: If we set Y to be the number of failures
until first success we obtain a slightly different definition of the geometric distribution. Here we have that
ImY = N ∪ {0} and

P(Y = x) = (1 − p)x p, for x = 0, 1, 2, . . . .

51
A. E. D. Veraart MATH40005: Probability and Statistics Autumn 2023

0.7
Geom(0.7)
Geom(0.5)
Geom(0.2)
0.6
0.5
0.4
Pmf

0.3
0.2
0.1
0.0

2 4 6 8 10

Figure 7.5: This graph shows the probability mass function of the Geometric distribution with three differ-
ent success probabilities: p ∈ {0.7, 0.5, 0.2}.

7.4.7 Negative binomial distribution

Definition 7.4.12 (Negative binomial distribution). A discrete random variable X is said to follow the
negative binomial distribution with parameters r ∈ N and p ∈ (0, 1), written X ∼ NBin(r, p), if ImX =
N ∪ {0} and
 
x+r−1 r
P(X = x) = p (1 − p)x , for x = 0, 1, . . . . (7.4.1)
r−1

The negative binomial distribution arises as the distribution of the number of failures in a sequence of
independent Bernoulli trials with success parameter p before r successes have occurred. To see this, let us
consider strings of ”0” (for failure) and ”1” (for success). Each string of r ”1”s and x ”0”s has probability
pr (1 − p)x . Now we need to find the number of such strings: We stop when we reach the rth success, so
the last element in the string will always be a ”1”. This leaves us with r + x − 1 positions, to which we
need to assign the remaining r − 1 ”1”s. Hence we obtain equation (7.4.1).
Remark 7.4.13. Recall that in the case of a Bin(n, p) distribution, we also consider a sequence of inde-
pendent Bernoulli trials, but we fix the number of trials n and count the number of successes.

52
A. E. D. Veraart MATH40005: Probability and Statistics Autumn 2023

0.07
NBin(5,0.3)
NBin(10,0.3)
0.06

NBin(20,0.3)
0.05
0.04
Pmf

0.03
0.02
0.01
0.00

0 20 40 60 80

Figure 7.6: This graph shows the probability mass function of the negative binomial distribution with
parameter p = 0.3 and r ∈ {5, 10, 20}.

In order to show that the pmf of the negative binomial distribution is a valid pmf, we study the general-
isation of the binomial coefficient:
Definition 7.4.14. For α ∈ C, k ∈ N, we define
 
α α(α − 1) · · · (α − k + 1)
:= .
k k!
The generalised binomial formula is then given by
X∞  
α k
(1 + x)α = x , for |x| < 1.
k
k=0

Lemma 7.4.15. For x ∈ N ∪ {0}, r ∈ N we have the following identity

   
x+r−1 x −r
= (−1) .
r−1 x
Proof. To see this, note that
 
x+r−1 (x + r − 1)! (x + r − 1)(x + r − 2) · · · r
= =
r−1 x!(r − 1)! x!
 
x (−r)(−r − 1) · · · (−r − x + 1) x −r
= (−1) = (−1) .
x! x

Lemma 7.4.15 together with the generalised Binomial formula stated above can be used to show that if
X ∼ NBin(r, p) for p ∈ (0, 1), then
X∞ X∞   X∞  
x+r−1 r x r x −r
P(X = x) = p (1 − p) = p (−1) (1 − p)x
x=0 x=0
r − 1 x=0
x

53
A. E. D. Veraart MATH40005: Probability and Statistics Autumn 2023

∞ 
X 
−r
=p r
(p − 1)x = pr (1 + (p − 1))−r = pr p−r = 1,
x=0
x

where the generalised Binomial theorem was applicable since |1 − p| = (1 − p) < 1.

7.4.8 Exercise
Exercise 7.4.16. Verify P
that all the probability mass functions listed above are valid in the sense that
pX (x) ≥ 0 for all x and x pX (x) = 1.

End of lecture 10. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

54
Chapter 8

Continuous random variables

The material of this chapter is based on Blitzstein & Hwang (2019), p.121-123, 213-244, Grimmett &
Welsh (1986), p.56-65.

8.1 Random variables and their distributions

So far, we have only considered discrete random variables which can take at most countably many values.
Now we will give a more general definition which is also suitable for broader applications.
Definition 8.1.1 (Random variable). A random variable on the probability space (Ω, F, P) is defined as
the mapping X : Ω → R which satisfies

X −1 ((−∞, x]) = {ω ∈ Ω : X(ω) ≤ x} ∈ F for all x ∈ R. (8.1.1)

Note that a discrete random variable (see Definition 7.3.1) satisfies the above definition of a random
variable. To see this, we can write
[
{ω ∈ Ω : X(ω) ≤ x} = {ω : X(ω) = y}.
y∈ImX:y≤x

The right hand side is a countable union of elements of F and (according the definition of the sigma-
algebra) hence also an element of F.

Remark 8.1.2. Note that, similarly to our previous definition, we call the set X −1 ((−∞, x]) = {ω ∈
Ω : X(ω) ≤ x} the pre-image of (−∞, x]. We can only make probability statements about the set
X −1 ((−∞, x]) if it is an element of the event space F which motivates our definition of a random variable.
Definition 8.1.3 (Cumulative distribution function (c.d.f.)). Suppose that X is a random variable on
(Ω, F, P), then the cumulative distribution function (c.d.f.) of X is defined as the mapping FX : R → [0, 1]
given by

FX (x) = P({ω ∈ Ω : X(ω) ≤ x}) = P(X −1 ((−∞, x])),

which is typically abbreviated to FX (x) = P(X ≤ x).

Example 8.1.4. Consider a Bernoulli random variable X ∼ Bern(p). The c.d.f. of a Bernoulli random
variable is given by

X  0, for x < 0,
FX (x) = P(X ≤ x) = P(X = k) = 1 − p, for x ∈ [0, 1),

k≤x 1, for x ≥ 1.

55
A. E. D. Veraart MATH40005: Probability and Statistics Autumn 2023

P (X = k) 1.2

0.6
1
0.6

0.8

0.4

F (x)
0.4 0.6

0.4
0.2
0.2

0
0 −1 −0.5 0 0.5 1 1.5 2
−1 0 1 2 k x

(a) P.m.f. of X ∼ Bern(0.4) (b) C.d.f. of X ∼ Bern(0.4)

Figure 8.1: Consider a Bernoulli random variable X with parameter p = 0.4. Its probability mass function
is depicted in Figure 8.1a and its cumulative distribution function in Figure 8.1b.

Let us now derive important properties of the c.d.f..

Theorem 8.1.5 (Properties of the c.d.f.). 1. FX is monotonically non-decreasing, i.e. for all x ≤ y we
have FX (x) ≤ FX (y).
2. FX is right-continuous, i.e. if xn ↓ x (which is short-hand notation for a sequence (xn )n∈N , which
is monotonically non-increasing, i.e. x1 ≥ · · · ≥ xn ≥ xn+1 ≥ · · · ≥ x and converging to x,
i.e. limn→∞ xn = x), then FX (xn ) → FX (x) as n → ∞.
3. limx→−∞ FX (x) = 0 and limx→∞ FX (x) = 1.
Proof. 1. Monotonicity: Let x ≤ y. Then

{ω ∈ Ω : X(ω) ≤ x} ⊆ {ω ∈ Ω : X(ω) ≤ y}.

Then the result follows from the monotonicity of the probability measure, see the second statement
in Theorem 4.2.3.
2. Right continuity: We prove that if xn ↓ x (which is short-hand notation for a sequence (xn )n∈N ,
which is monotonically non-increasing, i.e. x1 ≥ · · · ≥ xn ≥ xn+1 ≥ · · · ≥ x and converging to x,
i.e. limn→∞ xn = x), then FX (xn ) → FX (x) as n → ∞. Define events
∞
\
En := {ω : X(ω) ≤ xn } ↓ En = {ω : X(ω) ≤ x} =: E.
n=1

Using the continuity of the probability measure, see Theorem 6.1.16, P(En ) → P(E). Since
P(En ) = FX (xn ), P(E) = FX (x), we have that FX (xn ) → FX (x) as n → ∞.
3. Limit behaviour at ±∞: Define for an xn

En = {ω : X(ω) ≤ xn }.

Then En ↓ ∅ as xn ↓ −∞ and En ↑ Ω as xn ↑ ∞. Using the continuity property of P again, we

deduce that FX (xn ) = P(En ) → P(∅) = 0 as xn → −∞, and FX (xn ) = P(En ) → P(Ω) = 1 as
xn → ∞.

56
A. E. D. Veraart MATH40005: Probability and Statistics Autumn 2023

Remark 8.1.6. One can show that for any function F which satisfies the three conditions stated in Theorem
8.1.5, there exists a probability space and a random variable on that space which has F as its c.d.f..
Note that in applications, we often use the following result:
Theorem 8.1.7. For a < b, we have P(a < X ≤ b) = FX (b) − FX (a).
Proof. Note that for a < b, we have

{ω ∈ Ω : X(ω) ≤ b} = {ω ∈ Ω : X(ω) ≤ a} ∪ {ω ∈ Ω : a < X(ω) ≤ b},

where the two events on the right hand side are disjoint. Hence

P({ω ∈ Ω : X(ω) ≤ b}) = P({ω ∈ Ω : X(ω) ≤ a}) + P({ω ∈ Ω : a < X(ω) ≤ b}),

which implies that

P({ω ∈ Ω : a < X(ω) ≤ b}) = FX (b) − FX (a).

Remark 8.1.8. A cumulative distribution function (c.d.f) of a random variable X, say, is right continuous,
but not in general left continuous. To see the latter, consider a point x ∈ R and an arbitrary sequence
(xn )n∈N approaching x from the left, i.e. x1 ≤ x2 ≤ · · · ≤ xn ≤ · · · ≤ x and limn→∞ xn = x. Then
∞
[
En := {ω ∈ Ω : X(ω) ≤ xn } ↑ En = {ω ∈ Ω : X(ω) < x} =: E.
n=1

Hence, by the continuity of the probability measure

lim FX (xn ) = lim P(En ) = P(E)

n→∞ n→∞
= P({ω ∈ Ω : X(ω) < x})
= P({ω ∈ Ω : X(ω) ≤ x}) − P({ω ∈ Ω : X(ω) = x})
≤ P({ω ∈ Ω : X(ω) ≤ x} = FX (x).

So, we observe that FX is (left) continuous in x if and only if P({ω ∈ Ω : X(ω) = x}) = 0.

End of lecture 11. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

8.2 Continuous random variables and probability density function

When looking at the c.d.f. of a Bernoulli random variable, see Figure 8.1b, we noted that the c.d.f. looks
like a step function. Indeed, all discrete random variables have c.d.f.s which are right-continuous step
functions (with possibly (many) more steps than in the Bernoulli case). In the remainder of this chapter we
will now focus on random variables with a smooth c.d.f.:
Definition 8.2.1 (Continuous random variable and probability density function). A random variable X is
called continuous if its c.d.f. can be written as
Z x
FX (x) = P(X ≤ x) = fX (u)du, for all x ∈ R, (8.2.1)
−∞

where the function fX : R → R satisfies

(i) fX (u) ≥ 0 for all u ∈ R,

57
A. E. D. Veraart MATH40005: Probability and Statistics Autumn 2023

R∞
(ii) −∞
fX (u)du = 1.
We call fX the probability density function (p.d.f.) of X (or just the density).1
The so-called Fundamental Theorem of Calculus guarantees that a function FX given as in Definition
′
8.2.1 is differentiable at every point x where f is continuous with FX (x) = fX (x).
Remark 8.2.2. Note that fX (x) is not a probability and while fX is non-negative it is not restricted to be
smaller than 1.
We compare properties of the p.m.f. and the p.d.f. in the following table:

Discrete random variable Continuous random variable

pX (x) ≥ 0, for all x ∈ R R ∞ ≥ 0, for all x ∈ R
fX (x)
P
p (x) = 1 f (x)dx = 1
P X
x∈ImX −∞ XR
x
FX (x) = u∈ImX:u≤x pX (u) FX (x) = −∞ fX (u)du

Table 8.1: Comparing discrete and random variables with p.m.f. pX and p.d.f. fX , respectively.

It turns out that, although fX (x) is not a probability, it can be linked to a probability when we scale it
appropriately. Consider a small quantity which we shall denote by dx > 0. Then the probability that X is
close to x can be written as
Z x+dx
P(x < X ≤ x + dx) = FX (x + dx) − FX (x) = fX (u)du ≈ fX (x)dx.
x

So, we can view the quantity fX (x)dx as the continuous analogue to a probability mass function pX (x).
The reason why we typically do not consider point probabilities for continuous random variables be-
comes clear in the next theorem.
Theorem 8.2.3. For a continuous random variable X with density fX , we have
P(X = x) = 0, for all x ∈ R, (8.2.2)
and
Z b
P(a ≤ X ≤ b) = fX (u)du, for all a, b ∈ R with a ≤ b. (8.2.3)
a

Proof. Consider any x ∈ R with a sequence xn ↑ x (i.e.(xn )n∈N , with limn→∞ xn = x and x1 ≤ x2 ≤
· · · ≤ xn ≤ · · · ≤ x), and define events
En = {ω : xn < X(ω) ≤ x} ↓ E = {ω : X(ω) = x}.
Using the continuity of the probability measure, see Theorem 6.1.16, P(En ) → P(E). Hence we can write
P(X = x) = lim P(En ) = lim P({ω : xn < X(ω) ≤ x})
n→∞ n→∞
= lim (FX (x) − FX (xn ))
n→∞
Z x
= lim fX (u)du = 0.
n→∞ xn

Now, let a ≤ b, then we know from the above that P(X = a) = 0, hence
P(a ≤ X ≤ b) = P(a < X ≤ b) = FX (b) − FX (a),
where we used Theorem 8.1.7.

1 In a later analysis/measure course we will say that equation 8.2.1 means that the ”c.d.f. of a continuous random variable is

absolutely continuous with respect to the Lebesgue measure”.

58
A. E. D. Veraart MATH40005: Probability and Statistics Autumn 2023

Remark 8.2.4. Combining the results from Remark 8.1.8 and Theorem 8.2.3 we conclude that the c.d.f. of
a continuous random variable is continuous. It is important to remember that the definition of the con-
tinuous random variable guarantees the existence of the density and then the continuity of the associated
c.d.f. follows from the properties of the (improper) Riemann integral. Note that if you only assumed that a
random variable X has a continuous c.d.f. with, in particular, P(X = x) = 0 for all x, then the existence
of a density function is not guaranteed. A notorious example of such a case is the so-called Cantor function
which you might study in a later analysis/measure course and another example we will discuss later, see
Example 8.4.1.

8.3 Common continuous distributions [Reading material]

8.3.1 Uniform
Definition 8.3.1 (Uniform distribution). A continuous random variable X is said to have the uniform
distribution on the interval (a, b) for a < b, i.e. X ∼ U(a, b), if its density function is given by
 1
fX (x) = b−a , if a < x < b,
0, otherwise.

Its cumulative distribution function is given by


 0, if x ≤ a,
x−a
FX (x) =
 b−a , if a < x < b,
1, if x ≥ b.

8.3.2 Exponential

λ=2 1
λ=1
λ = 0.5
FX (x)
fX (x)

0
0 1 2 3 4 5 0 1 2 3 4 5
x x

Figure 8.2: Plot of the p.d.f. (left) and the c.d.f. (right) of an Exp(λ) random variable for λ ∈ {0.5, 1, 2}.

59
A. E. D. Veraart MATH40005: Probability and Statistics Autumn 2023

Definition 8.3.2 (Exponential distribution). A continuous random variable X is said to have the exponen-
tial distribution with parameter λ > 0, i.e. X ∼ Exp(λ), if its density function is given by

λe−λx , if x > 0,
fX (x) =
0, otherwise.

Its cumulative distribution function is given by


0, if x ≤ 0,
FX (x) =
1 − e−λx , if x > 0.

8.3.3 Gamma distribution

The Gamma distribution is – as the exponential distribution – also supported on the positive real line only
and extends the exponential distribution discussed above.
For t > 0 we define the Gamma function by
Z ∞
Γ(t) = xt−1 e−x dx,
0

which has the following properties:

Γ(t) = (t − 1)Γ(t − 1), for t > 1,

and in the case when t ∈ N we have Γ(t) = (t − 1)!.

Definition 8.3.3 (Gamma distribution). A continuous random variable X is said to have the Gamma dis-
tribution with shape parameter α > 0 and rate parameter β > 0, i.e. X ∼ Gamma(α, β), if its density
function is given by
( α
β α−1 −βx
fX (x) = Γ(α) x e , if x > 0,
0, otherwise.

Its cumulative distribution function is not available in closed form.

In the case when α = n ∈ N, we often call the Gamma distribution the Erlang distribution which has
density
(
βn n−1 −βx
fX (x) = (n−1)! x e , if x > 0,
0, otherwise.

8.3.4 Chi-squared distribution

Definition 8.3.4 (Chi-squared distribution). A continuous random variable X is said to have the chi-
squared distribution with n ∈ N degrees of freedom, i.e. X ∼ χ2 (n) (or also X ∼ χ2n ) , if its density
function is given by
(

1 x n/2−1 −x/2
e , if x > 0,
fX (x) = 2Γ(n/2) 2
0, otherwise.

Its cumulative distribution function is not available in closed form.

We note that the χ2 (n) distribution is the same as the Gamma(n/2, 1/2) distribution.

60
A. E. D. Veraart MATH40005: Probability and Statistics Autumn 2023

8.3.5 F-distribution
Definition 8.3.5 (F-distribution). A continuous random variable X is said to have the F-distribution with
d1 , d2 > 0 degrees of freedom, i.e. X ∼ F (d1 , d2 ) (or also X ∼ Fd1 ,d2 ) , if its density function is given by
  d /2
 d +d
 Γ( 1 2 2 ) d12
d 1
xd1 /2−1
(d +d )/2 , if x > 0,
fX (x) =

d d d 1 2
Γ( 21 )Γ( 22 ) 1+ d1 x

 2
0, otherwise.

Its cumulative distribution function is not available in closed form.

We note that the positive parameters d1 , d2 are not restricted to be integer-valued.
Note that if we have independent random variables X1 ∼ χ2n and X2 ∼ χ2m , then the random variable

X1 /n
X= ∼ Fn,m .
X2 /m

8.3.6 Beta distribution

For α, β > 0 denote by
Z 1
Γ(α)Γ(β)
B(α, β) = xα−1 (1 − x)β−1 dx =
0 Γ(α + β)

the so-called Beta function.

Definition 8.3.6 (Beta distribution). A continuous random variable X is said to have the Beta distribution
with parameters α, β > 0, i.e. X ∼ Beta(α, β), if its density function is given by
 1 α−1
B(α,β) x (1 − x)β−1 , if 0 ≤ x ≤ 1,
fX (x) =
0, otherwise.

Its cumulative distribution function is not available in closed form.

8.3.7 Normal distribution

Definition 8.3.7 (Standard normal distribution). A random variable X has the standard normal/standard
Gaussian distribution if it has density function f (x) = ϕ(x) with
1 2
ϕ(x) = √ e−x /2 , for x ∈ R.
2π
Note that we typically write X ∼ N(0, 1) since a standard normal random variable has mean zero and
variance one. The c.d.f. is then denoted by F (x) = Φ(x) with
Z x
1 2
Φ(x) = √ e−t /2 dt, for x ∈ R.
2π −∞
Unfortunately there is no explicit formula for the integral appearing in the c.d.f.!

61
A. E. D. Veraart MATH40005: Probability and Statistics Autumn 2023

1.00 y

0.75
2 Rx 2
φ(x) = √1 e−x /2
Φ(x) = √1 e−t /2
dt
2π 2π −∞
0.50

0.25 √
1/ 2π ≈ 0.3989

−2 −1 0 1 x 2

Figure 8.3: The red solid line depicts the standard Gaussian probability density function and the blue
dashed line the corresponding cumulative distribution function.

y = φ(x)

Φ(−x0 ) 1 − Φ(x0 )
x
−x0 x0

Figure 8.4: Note that the standard normal density is symmetric around 0, i.e. ϕ(x) = ϕ(−x) for all x. This
also implies that Φ(−x) = 1 − Φ(x).

Definition 8.3.8 (Normal distribution). Let µ denote a real number and let σ > 0. A random variable X
has the normal/ Gaussian distribution with mean µ and variance σ 2 if it has density function
1 (x−µ)2
f (x) = √ e− 2σ 2 , for x ∈ R.
2πσ 2
Note that we typically write X ∼ N(µ, σ 2 ).

62
A. E. D. Veraart MATH40005: Probability and Statistics Autumn 2023

1.00 y

µ = 0, σ = 0.5
0.75
(x−µ)2
f (x) = √1
σ 2π
e− 2σ2
µ = 0, σ = 1
0.50

µ = 0.5, σ = 0.5
0.25

−2 −1 0 1 2

Figure 8.5: The red line depicts the standard Gaussian probability density function and the two blue lines
show non-standard Gaussian probability density functions.

8.3.8 Cauchy distribution

Definition 8.3.9 (Cauchy distribution). A continuous random variable X is said to have the Cauchy dis-
tribution, if its density function is given by
1
fX (x) = , for x ∈ R.
π(1 + x2 )

Its cumulative distribution function is given by

1 1
FX (x) = arctan(x) + , for x ∈ R.
π 2
We note that if we have two independent standard normal random variables X, Y ∼ N(0, 1), then their
ratio Z = X/Y follows the Cauchy distribution.

8.3.9 Student t-distribution

Definition 8.3.10 ((Student’s) t-distribution). A continuous random variable X is said to have the (Stu-
dent’s) t-distribution with ν > 0 degrees of freedom, if its density function is given by

 − ν+1
Γ ν+1
2 x2 2

fX (x) = √ ν

1 + , for x ∈ R.
νπΓ 2 ν

Its cumulative distribution function is not available in closed form.

8.4 Example of a random variable which is neither discrete nor con-

tinuous
Throughout this course we will typically focus on random variable which are either discrete or continuous.
However, it is important to note that random variables exist which are neither discrete nor continuous! Let
us study such an example in the following.

63
A. E. D. Veraart MATH40005: Probability and Statistics Autumn 2023

Example 8.4.1. We flip an unfair coin infinitely many times and assume that we obtain heads with proba-
bility p ∈ (0, 1). We denote the outcomes by X1 , X2 , . . . with Xi = 0 if we obtain tails in the ith flip and
Xi = 1 if we obtain heads in the ith flip. Define a random number

Y = 0.X1 X2 X3 . . . (in base 2), i.e.

 2  3
1 1 1
Y = X1 · + X2 + X3 · + ··· .
2 2 2

Then
• X1 determines whether Y is in the first half [0, 21 ) (if X1 = 0) or in the second half [ 21 , 1] (if X1 = 1).
• X2 determines whether Y is in the first half (if X2 = 0) (either in [0, 14 ) or in [ 12 , 34 )) or in the second
half (if X2 = 1) (either in [ 41 , 12 ) or in [ 43 , 1]) of the previous half.

• etc.
We note that for any y = 0.x1 x2 x3 . . . in base 2 representation, we have

P(Y = y) = P(X1 = x1 )P(X2 = x2 ) · · · = 0

since each term in the product is either equal to p or 1 − p which are both smaller than 1, so their infinite
product will converge to 0. Hence Y cannot be a discrete random variable.
One (not we!) can show that for p ̸= 0.5, Y does not have a density, whereas if p = 0.5, then Y is
uniformly distributed on [0, 1] and hence has a density.

End of lecture 12. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

64
Chapter 9

Transformations of random variables

The material of this chapter is based on Blitzstein & Hwang (2019), p.123-129, Grimmett & Welsh
(1986), p.28-29, 65-67.
Let us consider a random variable X on (Ω, F, P) and a (deterministic) function g : R → R. Clearly
Y = g(X) is a mapping from Ω to R with Y (ω) = g(X(ω)). In this chapter, we would like to study under
which conditions Y is itself a random variable and we would like to study its distribution

9.1 The discrete case

Let us first consider the case when X is a discrete random variable.
Theorem 9.1.1. Let X be a discrete random variable on (Ω, F, P) and let g : R → R denote a deter-
ministic function. Then Y = g(X) is a discrete random variable with probability mass function given
by
X
pY (y) = P(X = x), (9.1.1)
x∈ImX:g(x)=y

for all y ∈ ImY and 0 otherwise.

Proof. We observe that Y : Ω → R and that Im(Y ) = {g(X(ω)) : ω ∈ Ω} is countable since Im(X) =
{X(ω) : ω ∈ Ω} is countable. Moreover, for all y ∈ R, we have

Y −1 ({y}) = {ω ∈ Ω : Y (ω) = y} = {ω ∈ Ω : g(X(ω)) = y}

= {ω ∈ Ω : X(ω) ∈ {x ∈ ImX : g(x) = y}}
[
= {ω ∈ Ω : X(ω) ∈ {x}}
x∈ImX:g(x)=y
 
[
= X −1  {x}
x∈ImX:g(x)=y
Lemma7.1.2
[
= X −1 ({x})
x∈ImX:g(x)=y
[
= {ω ∈ Ω : X(ω) = x} ∈ F, (9.1.2)
x∈ImX:g(x)=y

since each event {ω ∈ Ω : X(ω) = x} ∈ F for all x ∈ R and, by the definition of a σ-algebra, a countable
union of elements of F is in F, too. Since X is discrete, we indeed have that {x ∈ ImX : g(x) = y} ⊆
ImX is (at most) countably infinite. Hence we can conclude that Y = g(X) is indeed a discrete random
variable.

65
A. E. D. Veraart MATH40005: Probability and Statistics Autumn 2023

We can compute the p.m.f. of Y as follows:

X
pY (y) = P(Y = y) = P(g(X) = y) = P(X = x),
x∈ImX:g(x)=y

so we are just summing up the probabilities for all values x for which g(x) = y. Here we used the fact
that the union in (9.1.2) is a countable union of disjoint events, hence Axiom (iii) of the definition of the
probability measure applies, see also the discussion before Example 7.3.5 for more details.

In the special case when g is invertible (i.e. bijective), then the pmf of Y can be expressed as

pY (y) = pX (g −1 (y)) for all y ∈ ImY.

9.2 The continuous case

For the continuous (or more general case) recall that Y = g(X) is only a random variable if Y satisfies
condition (8.1.1), i.e.

{ω ∈ Ω : Y (ω) ≤ y} ∈ F for all y ∈ R.

This condition is only satisfied if g satisfies some additional properties (e.g. if it is continuous or mono-
tone1 ).
Example 9.2.1. Consider a linear transformation of the random variable X. I.e. let a > 0, b ∈ R and
define g(x) = ax + b. Then Y = g(X) = aX + b is indeed a random variable and, for any y ∈ R its
c.d.f. is given by
   
y−b y−b
FY (y) = P(Y ≤ y) = P(aX + b ≤ y) = P X ≤ = FX .
a a

Assuming that X is a continuous random variable, we can now differentiate (with respect to y– using the
chain rule) and obtain
   
y−b 1 1 y−b
fY (y) = FY′ (y) = FX
′
= fX .
a a a a

In the previous example, we have seen that in the case that the function g can be inverted, we can find
an explicit formula for the corresponding density of the transformed random variable. We can now state
and prove this result in a more general form.
Theorem 9.2.2. Suppose that X is a continuous random variable with density fX and g : R → R is
strictly increasing/decreasing and differentiable with inverse function denoted by g −1 , then Y = g(X) has
density

d −1
fY (y) = fX (g −1 (y)) [g (y)] , for all y ∈ R. (9.2.1)
dy

Proof. First, suppose that g is strictly increasing. Then, for any y ∈ R, we have

FY (y) = P(Y ≤ y) = P(g(X) ≤ y) = P(X ≤ g −1 (y)) = FX (g −1 (y)).

Differentiating w.r.t. y and an application of the chain rule leads to

d d
fY (y) = FY′ (y) = FX (g −1 (y)) = fX (g −1 (y)) [g −1 (y)].
dy dy
1 More generally, we will need that g is Borel-measurable, but this concept is beyond the scope of this course.

66
A. E. D. Veraart MATH40005: Probability and Statistics Autumn 2023

Second, suppose that g is strictly decreasing. Then, for any y ∈ R, we have

FY (y) = P(Y ≤ y) = P(g(X) ≤ y) = P(X ≥ g −1 (y)) = 1 − FX (g −1 (y)),

and hence
d −1 d −1
fY (y) = −fX (g −1 (y)) [g (y)] = fX (g −1 (y)) [g (y)] ,
dy dy
d −1
since in this case dy [g (y)] < 0.

Remark 9.2.3. In the proof above, we used the following result from Analysis: If g : R → R is a strictly
increasing (decreasing) function, then its inverse function denoted by g −1 is also strictly increasing (de-
creasing).
d −1
Remark 9.2.4. We typically call the term dy [g (y)] the Jacobian of the transformation.

Remark 9.2.5. You might remember equation (9.2.1) more easily when you write x = g −1 (y) and note
that
dx
fY (y) = fX (x) .
dy

In the case when g is strictly increasing, you can remove the absolute value signs and you get the pretty
symmetric formula:

fY (y)dy = fX (x)dx.

If X is continuous and we want to find the c.d.f. and/or the p.d.f. of Y = g(X) in the case when g is
not necessarily strictly increasing/decreasing, then we compute

FY (y) = P(Y ≤ y) = P(g(X) ≤ y) = P(X ∈ {x ∈ ImX : g(x) ≤ y})

Z
= fX (x)dx.
{x∈ImX:g(x)≤y}

If Y is continuous, we would then differentiate its c.d.f. to obtain its p.d.f..

Example 9.2.6. Let X ∼ N(0, 1), g(x) = x2 and set Y = g(X) = X 2 . We would like to find the c.d.f. and
the p.d.f. of Y . First we compute the c.d.f. of Y . Clearly, for y < 0, we have FY (y) = P(Y ≤ y) = 0 and
hence fY (y) = 0. Now let y ≥ 0, then
√ √
FY (y) = P(Y ≤ y) = P(X 2 ≤ y) = P(− y ≤ X ≤ y)
√ √ √ √
= FX ( y) − FX (− y) = Φ( y) − Φ(− y).

Differentiating leads to
 
1 −1/2 √ 1 −1/2 √
fY (y) = y fX ( y) − − y fX (− y)
2 2
1 √ √
= √ [ϕ( y) + ϕ(− y)]
2 y
    
1 1 1 1 1
= √ 2 √ exp − y =√ exp − y ,
2 y 2π 2 2πy 2

which is the density of a χ21 -random variable.

Example 9.2.7. Consider a random variable X : Ω → R with c.d.f. FX .

67
A. E. D. Veraart MATH40005: Probability and Statistics Autumn 2023

1. Find the c.d.f. of Y = max{X, 3}: Let y ∈ R, then

FY (y) = P(max{X, 3} ≤ y) = P(X ≤ y, 3 ≤ y) = P(X ≤ y)I[3,∞) (y)


FX (y), for y ≥ 3,
=
0, for y < 3.

2. Find the c.d.f. of Y = |X|. Let y ∈ R, then


0, for y < 0,
FY (y) = P(|X| ≤ y) =
P(−y ≤ X ≤ y), for y ≥ 0.

0, for y < 0,
=
FX (y) − FX ((−y)−), for y ≥ 0.

Recall that FX ((−y)−) is the left limit of FX at the point −y.

End of lecture 13. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

68
Chapter 10

Expectation of random variables

The material of this chapter is based on Blitzstein & Hwang (2019), p.149-174, Grimmett & Welsh
(1986), p.29-32, 67-70, 90-92.
This chapter introduces the expectation of a random variable. We will distinguish the two cases of
a discrete and continuous random variable and study the so-called law of the unconscious statistician
(LOTUS). We will also learn that the expectation is a linear operator and we will introduce the concept of
a variance and other (higher) moments.

10.1 Definition of the expectation

Next we define the expectation of a discrete random variable.
Definition 10.1.1 (Expectation of discrete random variable). Let X denote a discrete random variable,
then the expectation of X is defined as
X
E(X) = xP(X = x)
x∈ImX
P
whenever the sum on the right hand side converges absolutely, i.e. when we have x∈ImX |x|P(X = x) <
∞.1
The expectation of X is also called expected value or mean. Note that we typically simplify the notation
and write
X X
E(X) = xP(X = x) = xpX (x).
x x

Definition 10.1.2 (Expectation of a continuous random variable). For a continuous random variable X
with density fX , we define the expectation of X as
Z ∞
E(X) = xfX (x)dx,
−∞
R∞
provided that −∞
|x|fX (x)dx < ∞.
As in the discrete case, we often refer to the expectation as mean or expected value.
Remark 10.1.3. Recall that we said that pX (x) for a discrete random variable is comparable to fX (x)dx
for a continuous random variable. Also, in the discrete case, we deal with sums, whereas in the continuous
case we have integrals. Using these analogies it makes sense to use the definition
 P
E(X) = R ∞ x xpX (x) if X is discrete,
−∞
xf X (x)dx if X is continuous.
1 This assumption matters in the case when ImX is infinite. If the sum converges absolutely, then the sum takes the same value

irrespectively of the order of summation.

69
A. E. D. Veraart MATH40005: Probability and Statistics Autumn 2023

Remark 10.1.4. Note that the above definition of the expectation ensures that the expectation is finite. We
can relax that definition slightly and allow for infinite expectations as well. In doing that, we need to be a
bit more careful with the precise definition. We can proceed as follows.
Suppose X is a non-negative discrete/continuous random variable with pmf pX /pdf fX . Then, define
 P
E(X) = R ∞x≥0 xpX (x) if X is discrete,
(10.1.1)
0
xf X (x)dx if X is continuous.
Here the sum/integral is allowed to take the value +∞.
For a general discrete/continuous random variable, which is not necessarily restricted to be non-
negative, we can define its positive and negative parts as follows:
X + = max{0, X}, X − = max{0, −X}.
Since both X + and X − are non-negative, we can now define E(X + ) and E(X − ) as in (10.1.1). Here the
expectations can take the value +∞.
Since
X = X + − X −,
we can then define E(X) as
E(X) = E(X + ) − E(X − ),
provided the right hand side is not of the form ∞ − ∞, in which case we would say that the corresponding
expectation is undefined.

10.2 Law of the unconscious statistician (LOTUS)

Consider the situation that we have a transformation of a random variable Y = g(X) and we would like
to find its expectation. The law of the unconscious statistician will tell us that we do not need to find the
p.m.f./p.d.f. of the transformed variable but rather use the following formula:
Theorem 10.2.1 (LOTUS: Discrete case). Let X be a discrete random variable and g : R → R, then
X
E(g(X)) = g(x)P(X = x),
x∈ImX

whenever the sum on the right hand side converges absolutely.

Proof. We note that if Y = g(X), then according to equation (9.1.1) the p.m.f. of Y is given by
X
P(Y = y) = P(g(X) = y) = P(X = x).
x∈ImX:g(x)=y

Hence
X X
E(Y ) = yP(Y = y) = yP(g(X) = y)
y∈ImY y∈ImY
X X
= y P(X = x)
y∈ImY x∈ImX:g(x)=y
X X
= yP(X = x)
x∈ImX y∈ImY :y=g(x)
X X
= g(x)P(X = x)
x∈ImX y∈ImY :y=g(x)
X
= g(x)P(X = x),
x∈ImX

where we were allowed to interchange the order of summation since the sum converges absolutely.

70
A. E. D. Veraart MATH40005: Probability and Statistics Autumn 2023

Example 10.2.2. Consider a Bernoulli random variable X ∼ Bern(p). We compute its mean as follows:
X
E(X) = xP(X = x) = 0 · P(X = 0) + 1 · P(X = 1) = P(X = 1) = p.
x

Next, we want to find E(X 2 ). Using Theorem 10.2.1, we find

X
E(X 2 ) = x2 P(X = x) = 02 · P(X = 0) + 12 · P(X = 1) = P(X = 1) = p.
x

Example 10.2.3. Let X ∼ Poi(λ) for λ > 0. Find E(X!).

∞
X ∞
X X∞
LOTUS λn −λ geom. series,|λ|<1 1
E(X!) = n!P(X = n) = n! e = e−λ λn = e−λ ,
n=0 n=0
n! n=0
1−λ

if |λ| = λ < 1 and E(X!) = ∞ for λ ≥ 1.

We will now state (without proof) the LOTUS for the continuous case:
Theorem 10.2.4 (LOTUS: Continuous case). Let X be a continuous random variable with density fX ,
consider a function2 g : R → R, then
Z ∞
E(g(X)) = g(x)fX (x)dx,
−∞
R∞
provided that −∞
|g(x)|fX (x)dx < ∞.

Example 10.2.5. Take g(x) = xk for k ∈ N. Then E(X k ) is called the kth moment of X (provided it
exists).
The LOTUS theorems imply the linearity of the expectation in the following sense:
Theorem 10.2.6. Consider a discrete/continuous random variable X with finite expectation.
1. If X is non-negative, then E(X) ≥ 0.
2. If a, b ∈ R, then E(aX + b) = aE(X) + b.
Proof. The proof is left as an exercise, see Exercise 6- 2.
Example 10.2.7. Let X be a continuous random variable with density fX (x) = cx2 for x ∈ [0, 2] and
fX (x) = 0 otherwise. Find c and E(X) and E(X 2 ). The probability density function needs to be nonneg-
ative and integrate to 1, hence we set
Z ∞ Z 2 2
1 8
1= fX (x)dx = c x2 dx = c x3 =c ,
−∞ 0 3 0 3

which implies that c = 38 (≥ 0).

Then
Z ∞ Z 2 2
3 3 31 4 3
E(X) = xfX (x)dx = x dx = x = .
−∞ 0 8 84 0 2

Using the LOTUS, we get

Z ∞ Z 2 2
3 4 31 5 12
E(X 2 ) = x2 fX (x)dx = x dx = x = .
−∞ 0 8 85 0 5

2 We assume that the function g is such that g(X) is also a continuous random variable.

71
A. E. D. Veraart MATH40005: Probability and Statistics Autumn 2023

10.3 Variance
While the expectation tells you something about the centre of the distribution, in many applications we also
want to know about the dispersion of X about its mean value. Hence we introduce the so-called variance
Definition 10.3.1 (Variance). Let X be a discrete/continuous random variable. Then its variance is defined
as

Var(X) = E[(X − E(X))2 ],

provided that it exists. Often we write σ 2 = Var(X).

From Theorem 10.2.6 we deduce that the variance of a random variable is always non-negative.
If we are considering a random variable which Pis just given by a deterministic constant, e.g. for some
c ∈ R we have P(X = c) = 1, then E(X) = x xP(X = x) = c · P(X = c) = c and Var(X) =
E[(X − E(X))2 ] = E[(c − c)2 ] = 0. That means that only true randomness generates a non-zero variance.
In practice, it is often easier to work with a slightly different expression for the variance which we shall
derive next.
Theorem 10.3.2. For a discrete/continuous random variable with finite variance we have that

Var(X) = E(X 2 ) − [E(X)]2 .

Proof. In order to simplify the notation we write µ = E(X). In the discrete case we have, using Theorem
10.2.1,
X
Var(X) = E[(X − µ)2 ] = (x2 − 2µx + µ2 )pX (x)
x
X X X
2
= x pX (x) + (−2µx)pX (x) + µ2 pX (x)
x x x
X X X
= x2 pX (x) − 2µ xpX (x) + µ2 pX (x)
x x x
= E(X 2 ) − 2µE(X) + µ2 = E(X 2 ) − [E(X)]2 .

In the continuous case, we have after applying Theorem 10.2.4,

Z ∞
Var(X) = E[(X − µ)2 ] = (x2 − 2µx + µ2 )fX (x)dx
−∞
Z ∞ Z ∞ Z ∞
= x2 fX (x)dx + (−2µx)fX (x)dx + µ2 fX (x)dx
−∞ −∞ −∞
Z ∞ Z ∞ Z ∞
= x2 fX (x)dx − 2µ xfX (x)dx + µ2 fX (x)dx
−∞ −∞ −∞
2 2 2 2
= E(X ) − 2µE(X) + µ = E(X ) − [E(X)] .

A very useful property of the variance is that it is not affected by deterministic additions and a multi-
plicative constant can be taken out of the variance provided we square it:
Theorem 10.3.3. Let X be a discrete/continuous random variable with finite variance and consider deter-
ministic constants a, b ∈ R. Then

Var(aX + b) = a2 Var(X).

Proof. This is left as an exercise, see Exercise 6- 3.

End of lecture 14. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

72
Chapter 11

Bridging lecture: Multivariate calculus

The material of this chapter is based on Blitzstein & Hwang (2019), p.594–596.
In this bridging lecture we will state the main concepts from multivariate calculus which we will need
in the Y1 Probability and Statistics course in order to be able to study multivariate random variables. More
details of this material will be provided in the Analysis and Calculus courses at a later point in time.
In this lecture, we will focus on bi-variable calculus only, but the concepts will extend to the general
multivariate case.

11.1 Partial derivatives

Suppose you have a bivariate function f : R2 → R defined as

f (x, y) = xy 2 + x3 y.

We would like to find the partial derivative with respect to x. Then we treat the variable y as a constant
and differentiate the function g(x) := f (x, y) in the usual way with respect to x. This leads to
∂f (x, y) d
= g(x) = y 2 + 3x2 y.
∂x dx
Similarly, if we would like to find the partial derivative with respect to y, then we treat the variable x
as a constant and differentiate the function h(y) := f (x, y) in the usual way with respect to y. This leads
to
∂f (x, y) d
= h(y) = 2xy + x3 .
∂y dy
The partial derivatives above are so-called first order partial derivatives. Repeating the steps above, by
taking partial derivatives of the partial derivatives, we get second order partial derivatives. In our example,
this leads to

 
∂ 2 f (x, y) ∂ ∂f (x, y)
= = 2y + 3x2 .
∂y∂x ∂y ∂x
We note that it does not matter in which order we differentiate and we get the same result when we compute
 
∂ 2 f (x, y) ∂ ∂f (x, y)
= = 2y + 3x2 .
∂x∂y ∂x ∂y
Under mild technical assumptions, we have for a general function f that
∂ 2 f (x, y) ∂ 2 f (x, y)
=
∂x∂y ∂y∂x

73
A. E. D. Veraart MATH40005: Probability and Statistics Autumn 2023

Consider a transformation which maps x = (x1 , x2 ) to y = (y1 , y2 ). Then the Jacobian of the
transformation is defined as the 2x2 matrix of all possible first order partial derivatives:
!
∂y1 ∂y1
∂y ∂x1 ∂x2
= ∂y2 ∂y2
∂x ∂x1 ∂x2

11.2 Bivariate integrals

Let A ⊆ R2 and f : R2 → R. Then (under mild conditions), we have that the order of integration does not
matter and that
Z Z Z Z
f (x, y)dxdy = f (x, y)dydx,
A A

and the joint integral can be computed by iteratively computing the univariate integrals. I.e., similar to the
concept of partial derivatives, we can compute multiple integrals by treating the variable which is not the
integration variable constant and integrate out one variable at a time. We illustrate this idea in an example.
Let A = {(x, y) : 0 ≤ x ≤ y ≤ 1} ⊆ R2 and f (x, y) = xy 2 + x3 y. First we will be integrating with
respect to x and then with respect to y:

Z Z Z 1 Z y Z 1  y
1 2 2 1 4
f (x, y)dxdy = (xy 2 + x3 y)dxdy = x y + x y dy
A 0 0 0 2 4 0
Z 1  
1 4 1 5
= y + y dy
0 2 4
1
11 5 11 6 1 1 17
= y + y = + = .
25 46 0 10 24 120
Alternatively, we can also integrate with respect to y first and then with respect to x. When switching the
order of integration, we need to carefully check the area of integration defined by A. Then we have

Z Z Z Z Z  !
1 1 1 1
1 3 1 3 2
f (x, y)dydx = (xy 2 + x3 y)dydx = xy + x y dx
A 0 x 0 3 2 x
Z 1  
1 1 3 1 4 1 5
= x + x − x − x dx
0 3 2 3 2
1
11 2 11 4 11 5 11 6
= x + x − x − x
23 24 35 26 0
1 1 1 1 17
= + − − = .
6 8 15 12 120

11.3 Change of variables formula

As in the univariate case, a change of variables formula also exist for the multivariate case. In that formula,
a so-called Jacobian appears. We will explain the key ideas again in the bivariate setting.
Let f : R2 → R. We define the mapping T : R2 → R2 by
T (x, y) = (u(x, y), v(x, y)),
and assume that T is a bijection from the domain D ⊆ R2 to some range S ⊆ R2 . Then we can write
T −1 : S → D for the inverse mapping of T , i.e. (x, y) = T −1 (u, v). For the first component we write
x = x(u, v) and for the second y = y(u, v). The Jacobian determinant of T −1 is defined as the determinant
   ∂x ∂x 
∂(x, y) ∂u ∂v
∂x ∂y ∂x ∂y
J(u, v) = det = det ∂y ∂y = − .
∂(u, v) ∂u ∂v ∂u ∂v ∂v ∂u

74
A. E. D. Veraart MATH40005: Probability and Statistics Autumn 2023

The change of variable formula states that (under mild conditions1 )

Z Z Z Z
f (x, y)dxdy = f (x(u, v), y(u, v))|J(u, v)|dudv. (11.3.1)
D S

11.3.1 Example using polar coordinates

Suppose we want to compute the integral
Z ∞Z ∞  
1 2 2
I := exp − (x + y ) dxdy.
−∞ −∞ 2
Now we consider the (invertible) transformation to polar coordinates:

x = r cos(θ), y = r sin(θ),

for r > 0 and θ ∈ [0, 2π). Then, we compute the Jacobian determinant J(r, θ) of the transformation as
follows:
   
∂(x, y) cos(θ) −r sin(θ)
J(r, θ) = det = det = r(cos2 (θ) + sin2 (θ)) = r.
∂(r, θ) sin(θ) r cos(θ)
Then,

Z 2π 
Z ∞ 
1
I= exp − (r2 cos2 (θ) + r2 sin2 (θ)) |J(r, θ)|drdθ
0 0 2
Z 2π Z ∞  
1
= exp − r2 rdrdθ.
0 0 2

You can now do another variable transformation (or integrate directly): We set u = r2 /2, the du = rdr
and
Z 2π Z ∞  Z 2π Z 2π
∞

I= e−u du dθ = −e−u u=0 dθ = 1dθ = 2π.
0 0 0 0
√ √
Hence, we have that I = 2π. Now you might wonder why we do these kind of computations in the
probability course...
Suppose you would like to check that the standard normal density is indeed a valid density and, apart
from being nonnegative, satisfies

Z ∞  2
1 x
√ exp − dx = 1. (11.3.2)
−∞ 2π 2
It turns out that this integral can be computed using the transformation to polar coordinates used above. To
this end, note that showing equation (11.3.2) is equivalent to showing that

Z ∞  
x2 √
exp − dx = 2π.
−∞ 2
Squaring both sides leads to

Z ∞  2  2
x
exp − dx = 2π.
−∞ 2
1 For instance, we need that the partial derivatives in the Jacobian exist and are continuous and that the Jacobian determinant is

never 0.

75
A. E. D. Veraart MATH40005: Probability and Statistics Autumn 2023

Now we expand the left hand side as follows:

Z  2  2 Z ∞
∞  2   Z ∞  2 
x x y
exp − dx = exp − dx · exp − dy
−∞ 2 −∞ 2 −∞ 2
Z ∞Z ∞  
1
= exp − (x2 + y 2 ) dxdy = I.
−∞ −∞ 2
R∞
Since we have already shown that I = 2π, we can conclude that −∞ ϕ(x)dx = 1.
In the next chapter, we will also show how the change of variables formula can be applied in the context
of transformations of multivariate random variables.
End of bridging lecture . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

76
Chapter 12

Multivariate random variables

The material of this chapter is based on Blitzstein & Hwang (2019), p.129-133, 303-306, 312-313,
Grimmett & Welsh (1986), p.36-43, 75-88.

12.1 Multivariate distributions

12.1.1 The bivariate case
Let us now consider two (arbitrary, i.e. not restricted to discrete or continuous) random variables X and Y
on the same probability space (Ω, F, P). We would like to understand how they relate to each other and
whether or not they are independent. We will write them as a random vector (X, Y ) taking values in R2 .
Definition 12.1.1 (Joint distribution function). The joint distribution function of the random vector (X, Y )
is defined as the mapping FX,Y : R2 → [0, 1] given by
FX,Y (x, y) = P({ω ∈ Ω : X(ω) ≤ x, Y (ω) ≤ y}), for any x, y ∈ R.
Using our shortened notation, we typically write
FX,Y (x, y) = P(X ≤ x, Y ≤ y), for any x, y ∈ R.
We can now list some of the key properties of joint distribution functions:
• FX,Y is non-decreasing in each variable, meaning that
FX,Y (x1 , y1 ) ≤ FX,Y (x2 , y2 ) if x1 ≤ x2 and y1 ≤ y2 .

• FX,Y is continuous from above (the multivariate version of right-continuity), i.e., for two sequences
(xn ), (yn ) which approach x and y from the right as n → ∞ we get that FX,Y (xn , yn ) → FX,Y (x, y).
• We have the following two limits:
lim FX,Y (x, y) = 0, lim FX,Y (x, y) = 1.
x→−∞,y→−∞ x→∞,y→∞

• They determine the marginal distributions uniquely, i.e.

FX (x) = lim FX,Y (x, y), FY (y) = lim FX,Y (x, y).
y→∞ x→∞

Example 12.1.2. Let FX,Y denote the joint c.d.f. of (X, Y ). For x, y ∈ R, find an expression for P(X ≤
x, Y ≥ y) in terms of FX,Y . We note that
Law of total prob.
P(X ≤ x, Y ≥ y) + P(X ≤ x, Y < y) = P(X ≤ x) = FX,Y (x, ∞).
Also, P(X ≤ x, Y < y) = FX,Y (x, y−). Hence,
P(X ≤ x, Y ≥ y) = FX,Y (x, ∞) − FX,Y (x, y−).

77
A. E. D. Veraart MATH40005: Probability and Statistics Autumn 2023

12.1.2 The n-dimensional case

The extension to the n-dimensional case (for n ∈ N) is now straightforward: We consider random variables
X1 , . . . , Xn on the same probability space (Ω, F, P). We write X = (X1 , . . . , Xn ) and x = (x1 , . . . , xn ).
Then the joint distribution function of X is given by FX : Rn → [0, 1]:

FX (x) = P(X1 ≤ x1 , . . . , Xn ≤ xn ), for all x ∈ Rn .

12.2 Independence
We have now all the tools to define what we mean by independence of (general) random variables: We call
random variables X and Y independent if the events {ω ∈ Ω : X(ω) ≤ x} and {ω ∈ Ω : Y (ω) ≤ y} are
independent for all x, y ∈ R. I.e. we define:

Definition 12.2.1 (Independence of random variables). The random variables X and Y are independent if
and only if

P(X ≤ x, Y ≤ y) = P(X ≤ x)P(Y ≤ y), for all x, y ∈ R,

which is equivalent to saying that the joint distribution function factorises as the product of the two
marginal distribution functions:

FX,Y (x, y) = FX (x)FY (y), for all x, y ∈ R.

We call the random variables X1 , . . . , Xn independent if

P(X1 ≤ x1 , . . . , Xn ≤ xn ) = P(X1 ≤ x1 ) · · · P(Xn ≤ xn ), for all x ∈ Rn

or equivalently if

FX (x) = FX1 (x1 ) · · · FXn (xn ), for all x ∈ Rn

Definition 12.2.2 (Pairwise independence for n ∈ N, n > 2 random variables). We call the random
variables X1 , . . . , Xn pairwise independent if

FXi ,Xj (xi , xj ) = FXi (xi )FXj (xj ), for all xi , xj ∈ R whenever i ̸= j.

Remark 12.2.3. Independence of random variables implies pairwise independence, the reverse statement,
however, is not true in general.
Finally, we define what we mean by independence of a family of (infinitely many) random variables.
Definition 12.2.4 (Independence of a family of random variables). Let I ⊂ R denote an index set. A
family of random variables {Xi : i ∈ I} is said to be independent if for all finite subsets J ⊆ I and all
xj ∈ R, j ∈ J , the following product rule holds:
Y
P(∩j∈J {Xj ≤ xj }) = P(Xj ≤ xj ).
j∈J

Remark 12.2.5. Note that for independent random variables X1 , . . . Xn , and continuous functions fi :
R → R for i = 1, . . . , n the transformed random variables Y1 = f1 (X1 ), . . . , Yn = fn (Xn ) are also
independent. [It would be sufficient to assume that functions fi are Borel-measurable, but this concept is
beyond the scope of this course.]

78
A. E. D. Veraart MATH40005: Probability and Statistics Autumn 2023

12.3 Multivariate discrete distributions and independence

Definition 12.3.1 (Joint probability mass function). Let X, Y denote discrete random variables on (Ω, F, P).
Their joint probability mass function denoted by pX,Y is defined as the function pX,Y : R2 → [0, 1] given
by

pX,Y (x, y) = P({ω ∈ Ω : X(ω) = x, Y (ω) = y}),

which is typically shortened to

pX,Y (x, y) = P(X = x, Y = y).

P P
We have that pX,Y (x, y) ≥ 0 for all x, y ∈ R and x y pX,Y (x, y) = 1.
The marginal probability mass functions of X and Y are then given by
X X
pX (x) = pX,Y (x, y), and pY (y) = pX,Y (x, y).
y x

It turns out that for any ”nice” set A ⊆ R2 , we obtain that

X X
P((X, Y ) ∈ A) = P(X = x, Y = y).
(x,y)∈A

12.3.1 Independence
Definition 12.2.1 covers the case of general random variables. In the discrete (or continuous) case, we can
formulate equivalent independence conditions:
Definition 12.3.2 (Independence of discrete random variables). Suppose that X and Y are discrete random
variables on a probability space (Ω, F, P). X and Y are said to be independent if the pair of events
{ω ∈ Ω : X(ω) = x} and {ω ∈ Ω : Y (ω) = y} are independent for all x, y ∈ R, i.e. if

P(X = x, Y = y) = P(X = x)P(Y = y), for all x, y ∈ R. (12.3.1)

Condition (12.3.1) is equivalent to saying that

pX,Y (x, y) = pX (x)pY (y), for all x, y ∈ R.

Random variables which are not independent are called dependent.

12.4 Multivariate continuous distributions and independence

We can also extend the concept of a continuous random variable to random vectors. Again, we will be
focussing on the bivariate case, but the n-dimensional case works in exactly the same way.
Definition 12.4.1 (Continuous random vector). We call the random vector (X, Y ) on (Ω, F, P) (jointly)
continuous if
Z x Z y
FX,Y (x, y) = fX,Y (u, v)dvdu,
u=−∞ v=−∞

for a function fX,Y : R2 → R satisfying

(i) fX,Y (u, v) ≥ 0 for all u, v ∈ R,
R∞ R∞
(ii) −∞ −∞ fX,Y (u, v)dvdu = 1.
We call fX,Y the (joint) density function of (X, Y ).

79
A. E. D. Veraart MATH40005: Probability and Statistics Autumn 2023

Similar to the univariate case, we typically obtain the joint density by differentiating the joint distribu-
tion function. I.e. we take
(
∂2
fX,Y (x, y) = ∂x∂y FX,Y (x, y), if this derivative exists at (x, y),
0, otherwise.

It turns out that for any ”nice” set A ⊆ R2 , we obtain that

Z Z
P((X, Y ) ∈ A) = fX,Y (x, y)dxdy.
(x,y)∈A

We do not prove this result here formally.

We note that the marginal densities can be obtained from the joint density as follows:
Z x Z ∞
d d
fX (x) = FX (x) = fX,Y (u, v)dvdu
dx dx u=−∞ v=−∞
Z ∞
= fX,Y (x, v)dv,
v=−∞

and also
Z ∞
fY (y) = fX,Y (u, y)du.
u=−∞

12.4.1 Independence
From our definition of independence of random variables, see Definition 12.2.1, we can immediately de-
duce by differentiating/integrating that jointly continuous random variables X and Y are independent if
and only if their joint density factorises:
fX,Y (x, y) = fX (x)fY (y), for all x, y ∈ R.

12.4.2 Examples
Example 12.4.2. Suppose the joint density of (X, Y ) is given by
( 2
√7 e−x /2−7y , if − ∞ < x < ∞, y > 0,
fX,Y (x, y) = 2π
0, otherwise.

We want to check whether or not X and Y are independent and compute P(X > 2, Y < 1). We note that
for x ∈ R, y > 0, we can write
1 2
fX,Y (x, y) = √ e−x /2 · 7e−7y ,
2π
which is in fact the product of a standard normal random variable and an Exp(7) random variable. Hence,
we have fX,Y (x, y) = fX (x)fY (y) for all x, y ∈ R which implies independence. Hence
P(X > 2, Y < 1) = P(X > 2)P(Y < 1) = (1 − Φ(2))FY (1)
= (1 − Φ(2))(1 − e−7 ).
The following is a worked example where you can practice doing computations involving bivariate
p.d.f.s and c.d.f.s.
Example 12.4.3 (Reading material: Worked example). Consider jointly continuous random variables
X, Y with joint density given by

c(x2 + y 2 ), for 0 < x < 2, 0 < y < 2,
fX,Y (x, y) =
0, otherwise.

80
A. E. D. Veraart MATH40005: Probability and Statistics Autumn 2023

1. Find c such that fX,Y is a p.d.f.: We need that c ≥ 0 and

Z Z Z  Z  !
∞ 2 2 2 2
2 2 1
1= fX,Y (x, y)dydx = c(x + y )dy dx = c x2 y + y 3 dx
−∞ x=0 y=0 x=0 3 y=0
Z    !
2 2
2 8 2 3 8 32 3
=c 2x + dx = c x + x =c ⇔c= .
x=0 3 3 3 x=0 3 32

2. Find the joint c.d.f. of (X, Y ).

Case 1: Let x ≤ 0 or y ≤ 0: FX,Y (x, y) = 0.

Case 2: Let x, y ∈ (0, 2):
Z x Z y  Z x  y 
1
FX,Y (x, y) = c(u2 + v 2 )dv du = c u2 v + v 3 du
u=0 v=0 u=0 3 v=0
Z x    x   
2 1 3 1 3 1 3 1 3 1 3
=c u y + y du = c u y+ y u =c x y+ y x
u=0 3 3 3 u=0 3 3
1 3
= (x y + xy 3 ).
32
1

1 3
Case 3: Let x ∈ (0, 2), y ≥ 2: FX,Y (x, y) = 32 2x3 + 8x = 16 x + 14 x = FX (x).
1

1 3
Case 4: Let y ∈ (0, 2), x ≥ 2: FX,Y (x, y) = 32 2y 3 + 8y = 16 y + 14 y = FY (y).
Case 5: Let x, y ≥ 2: FX,Y (x, y) = 1.
3. Differentiate the c.d.f. to obtain the p.d.f.:
Case 1: Let x ≤ 0 or y ≤ 0: FX,Y (x, y) = 0. Hence

∂ 2 FX,Y (x, y)
fX,Y (x, y) = = 0.
∂x∂y
1 3
Case 2: Let x, y ∈ (0, 2): FX,Y (x, y) = 32 (x y + xy 3 ). Hence
   
∂ 2 FX,Y (x, y) ∂ ∂ 1 3 3 ∂ 1 2 3
fX,Y (x, y) = = (x y + xy ) = (3x y + y )
∂x∂y ∂y ∂x 32 ∂y 32
1 3 2
= (3x2 + 3y 2 ) = (x + y 2 ).
32 32

1 3 ∂ 2 FX,Y (x,y)
Case 3: Let x ∈ (0, 2), y ≥ 2: FX,Y (x, y) = 16 x + 14 x. Hence fX,Y (x, y) = ∂x∂y = 0.
2
1 3 ∂ FX,Y (x,y)
Case 4: Let y ∈ (0, 2), x ≥ 2: FX,Y (x, y) = 16 y + 14 y. Hence fX,Y (x, y) = ∂x∂y = 0.
2
∂ FX,Y (x,y)
Case 5: Let x, y ≥ 2: FX,Y (x, y) = 1. Hence fX,Y (x, y) = ∂x∂y = 0.
4. Find the marginal densities of X and Y .
Z Z  !
∞ 2 2
1
fX (x) = fX,Y (x, y)dy = c(x + y )dy = c x2 y + y 3
2 2
−∞ 0 3 y=0
 
8 3 2 1
= c 2x2 + = x + ,
3 16 4
3 2
for x ∈ (0, 2) and fX (x) = 0 otherwise. By symmetry, fY (y) = 16 y + 14 , for y ∈ (0, 2) and
fY (y) = 0 otherwise.

81
A. E. D. Veraart MATH40005: Probability and Statistics Autumn 2023

5. Show that X and Y are not independent.

We have
  
3 2 1 3 2 1
fX (x)fY (y) = x + y + ̸ fX,Y (x, y),
=
16 4 16 4
for x, y ∈ (0, 2), hence X and Y are not independent.
6. Find the marginal c.d.f.s of X and Y .
1 3
FX (x) = FX,Y (x, ∞) = 16 x + 41 x for x ∈ (0, 2), FX (x) = 0 for x ≤ 0 and FX (x) = 1 for
1 3
x ≥ 2. Also, FY (x) = FX,Y (∞, y) = 16 y + 14 y for y ∈ (0, 2), FY (y) = 0 for y ≤ 0 and
FY (y) = 1 for y ≥ 2.

12.5 Transformations of random vectors: The bivariate case

In Chapter 9 we discussed how we can compute the p.d.f. of a transformed random variable. Here we
will illustrate how the methodology works in bivariate setting, where the change of variable formula, see
equation (11.3.1), discussed in the bridging lecture plays a key role.
Consider the case of jointly continuous random variables (X, Y ) with density fX,Y . Let u, v : R2 → R
denote deterministic functions and define a new pair of random variables by

U = u(X, Y ), V = v(X, Y ).

We would like to find the joint density of (U, V ).

We define the mapping T : R2 → R2 by

T (x, y) = (u(x, y), v(x, y)),

and assume that T is a bijection from the domain D = {(x, y) : fX,Y (x, y) > 0} ⊆ R2 to some range
S ⊆ R2 . Then we can write T −1 : S → D for the inverse mapping of T , i.e. (x, y) = T −1 (u, v). For the
first component we write x = x(u, v) and for the second y = y(u, v). The Jacobian determinant of T −1
is defined as the determinant
 ∂x ∂x 
∂u ∂v
∂x ∂y ∂x ∂y
J(u, v) = det ∂y ∂y = − .
∂u ∂v ∂u ∂v ∂v ∂u

Then the joint density of (U, V ) is given by


fX,Y (x(u, v), y(u, v))|J(u, v)|, if (u, v) ∈ S,
fU,V (u, v) =
0, otherwise.

Example 12.5.1. Let us demonstrate how the methodology works in practice, see Grimmett & Welsh (1986,
p. 87).
Suppose that X, Y ∼ Exp(1) are independent. Define
X
U := X + Y, V := .
X +Y
We want to find the joint density of (U, V ) and the marginal densities of U and V .
First, we note that the joint density of (X, Y ) is – due to independence – given by

fX,Y (x, y) = fX (x)fY (y) = e−(x+y) , if x, y > 0,

and zero otherwise.

In our case, the mapping T is given by
 
x
T (x, y) = (u, v) = x + y, ,
x+y

82
A. E. D. Veraart MATH40005: Probability and Statistics Autumn 2023

where T maps the set D = {(x, y) : x, y > 0}, onto S = {(u, v) : 0 < u < ∞, 0 < v < 1}.
Next, we find the inverse function of T :

T −1 (u, v) = (x, y) = (uv, (1 − v)u).

The Jacobian of T −1 is given by

 ∂x ∂x
  
∂u ∂v v u
J(u, v) = det ∂y ∂y = det = −uv − (1 − v)u = −u.
∂u ∂v
(1 − v) −u

Then, for (u, v) ∈ S, we have

fU,V (u, v) = fX,Y (x(u, v), y(u, v))|J(u, v)|

= exp(−(uv + (1 − v)u)) | − u| = u exp(−u),

and zero otherwise. I.e.


ue−u , for u > 0, 0 < v < 1,
fU,V (u, v) =
0 otherwise.

The marginal density of U is given by

Z 1 Z 1
fU (u) = fU,V (u, v)dv = u exp(−u)dv = u exp(−u), for u > 0,
0 0

and zero otherwise. Hence, U ∼ Gamma(2, 1). The marginal density of V is given by
Z ∞ Z ∞
fV (v) = fU,V (u, v)du = u exp(−u)du = Γ(2) = 1, for 0 < v < 1,
0 0

and zero otherwise. Hence V ∼ U(0, 1).

We notice that fU,V (u, v) = fU (u)fV (v) for all u, v, which implies that U and V are independent.

End of lecture 15. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

12.6 Two dimensional law of the unconscious statistician (2D LO-

TUS)
Using exactly the same arguments as in the univariate case, we can also formulate a law of the unconscious
statistician applied to a function of a (bivariate) random vector. We will only state the result here.
Theorem 12.6.1 (2D LOTUS: discrete case). Let X, Y denote discrete random variables on (Ω, F, P) and
let g : R2 → R. Then Z = g(X, Y ) is also a discrete random variable on (Ω, F, P) and its expectation is
given by
X X
E(g(X, Y )) = g(x, y)P(X = x, Y = y).
x∈ImX y∈ImY

Theorem 12.6.2 (2D LOTUS: continuous case). Let X, Y be jointly continuous random variables with
density fX,Y and let h : R2 → R. Then
Z ∞Z ∞
E[h(X, Y )] = h(x, y)fX,Y (x, y)dxdy.
−∞ −∞

Theorems 12.6.1,12.6.2 can be used to prove the linearity of the expectation:

83
A. E. D. Veraart MATH40005: Probability and Statistics Autumn 2023

Theorem 12.6.3 (Linearity of expectation). Let X, Y denote jointly discrete/continuous random variables
on (Ω, F, P), and a, b ∈ R, then

E(aX + bY ) = aE(X) + bE(Y ),

provided that E(X) and E(Y ) exist.

Proof. In the discrete case, we apply Theorem 12.6.1 with g(x, y) = ax + by. Then
XX
E(aX + bY ) = (ax + by)P(X = x, Y = y)
x y
X X X X
=a x P(X = x, Y = y) + b y P(X = x, Y = y)
x y y x
X X
=a xP(X = x) + b yP(Y = y)
x y

= aE(X) + bE(Y ).

In the continuous case, we apply Theorem 12.6.2 with g(x, y) = ax + by. Then
Z ∞Z ∞
E(aX + bY ) = (ax + by)fX,Y (x, y)dxdy
−∞ −∞
Z ∞Z ∞ Z ∞Z ∞
=a xfX,Y (x, y)dxdy + b yfX,Y (x, y)dxdy
−∞ −∞ −∞ −∞
Z ∞ Z ∞
=a xfX (x)dx + b yfY (y)dy
−∞ −∞
= aE(X) + bE(Y ).

Using induction, one can easily deduce that for n ∈ N and random variables X1 , . . . , Xn with finite
expectations and constants a1 , . . . , an ∈ R we have

E(a1 X1 + · · · + an Xn ) = a1 E(X1 ) + · · · + an E(Xn ). (12.6.1)

Remark 12.6.4. It is important to remember that the linearity of the expectation (12.6.1) holds in general
without assuming any independence between the random variables.

12.7 Covariance and correlation between random variables

Definition 12.7.1. Consider two (one-dimensional) random variables X and Y on the same sample space
with expectations µX = E(X) and µY = E(Y ). The covariance of X and Y is defined as

Cov(X, Y ) = E[(X − µX )(Y − µY )]

if the expectation on the right hand side takes a finite value. Also, we define the correlation of X and Y as
Cov(X, Y )
Cor(X, Y ) = p .
Var(X)Var(Y )

When we set X = Y , then the covariance simplifies to the variance:

Cov(X, X) = E[(X − µX )2 ] = Var(X).

For concrete computations it is often useful to work with the following alternative expression for the
covariance.

84
A. E. D. Veraart MATH40005: Probability and Statistics Autumn 2023

Theorem 12.7.2 (Covariance). For jointly discrete/continuous random variables X, Y with finite expecta-
tions, we have

Cov(X, Y ) = E(XY ) − E(X)E(Y ).

Proof. See Exercise 6- 9.

The following example follows on from Example 12.4.3. Here you can practice deriving the covariance
and correlation of two random variable.
Example 12.7.3 (Reading material: Worked example). Consider the same jointly continuous random
variables X, Y as in Example 12.4.3. Their joint density given by

c(x2 + y 2 ), for 0 < x < 2, 0 < y < 2,
fX,Y (x, y) =
0, otherwise.
3
for c = 32 .

1. Find Cov(X, Y ).
We note that Cov(X, Y ) = E(XY ) − E(X)E(Y ).
Recall that
Z Z  !
∞ 2 2
1
fX (x) = fX,Y (x, y)dy = c(x + y )dy = c x2 y + y 3
2 2
−∞ 0 3 y=0
 
8 3 2 1
= c 2x2 + = x + ,
3 16 4
for x ∈ (0, 2) and fX (x) = 0 otherwise. Then
Z ∞ Z 2  2
3 3 1 3 1 4 1 1 2 3 1 5
E(X) = xfX (x)dx = x + x dx = · x + · x = + = .
−∞ 0 16 4 16 4 4 2 x=0 4 2 4

By symmetry, we also have that E(Y ) = 45 . Also,

Z ∞ Z ∞ Z 2 Z 2 
LOTUS 3 3
E(XY ) = xyfX,Y (x, y)dydx = c (x y + xy )dy dx
x=−∞ y=−∞ x=0 y=0
Z 2  ! Z 2  
2 2
1 3 2 1 4 3

2 4 4 2
=c x y + xy dx = c 2x + 4x dx = c x + x
x=0 2 4 y=0 x=0 4 2 x=0
3 3
= (8 + 8) = .
32 2
Hence
3 52 1
Cov(X, Y ) = E(XY ) − E(X)E(Y ) = − =− .
2 42 16

2. Find Cor(X, Y ).
Recall that Var(X) = E(X 2 ) − (E(X))2 . Here we have
Z ∞ Z 2  2
2 LOTUS 2 3 4 1 2 3 1 5 1 1 3
E(X ) = x fX (x)dx = x + x dx = · x + · x
−∞ 0 16 4 16 5 4 3 x=0
6 2 28
= + = .
5 3 15
Hence
28 52 73
Var(X) = E(X 2 ) − (E(X))2 = − 2 = ,
15 4 240

85
A. E. D. Veraart MATH40005: Probability and Statistics Autumn 2023

73
and by symmetry Var(Y ) = 240 . Hence

Cov(X, Y ) −1 15
Cor(X, Y ) = p = 7316 = − ≈ −0.2.
Var(X)Var(Y ) 240
73

Remark 12.7.4. It is important to note that independent random variables always have zero covariance,
but the converse does not hold in general!
A very important property of independent random variables is the fact that the expectation of their
product can be written as the product of their expectation (a property which does not hold in general!):

Theorem 12.7.5. Let X, Y denote independent and jointly discrete/continuous random variables with
finite expectation, then

E(XY ) = E(X)E(Y ). (12.7.1)

Proof. In the case when X, Y are jointly discrete, we use Theorem 12.6.1 with g(x, y) = xy. Then
XX
E(XY ) = xyP(X = x, Y = y)
x y
XX
= xyP(X = x)P(Y = y) (by independence)
x y
X X
= xP(X = x) yP(Y = y) (using the existence of E(X), E(Y ))
x y

= E(X)E(Y ).

Using Theorem 12.6.2 and similar computations as above gives us the result for the jointly continuous
case.

Remark 12.7.6. It is important to note that if E(XY ) = E(X)E(Y ), then this does not in general imply
that X and Y are independent, see Exercise 6- 6.
The results stated in Theorem 12.7.5 can be extended to the n ∈ N dimensional case by induction: If
X1 , . . . , Xn are independent, then

E(X1 · · · Xn ) = E(X1 ) · · · E(Xn ).

In statistics, we often deal with sums of random variables. How can we compute their variance? The
following theorem gives an answer.
Theorem 12.7.7 (Variance of a sum of random variables). Let X, Y denote two jointly discrete/continuous
random variables with finite variances. Then

Var(X + Y ) = Var(X) + Var(Y ) + 2Cov(X, Y ).

Proof. See Exercise 6- 10.

End of lecture 16. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

86
Chapter 13

Generating functions

The material of this chapter is based on Blitzstein & Hwang (2019), p.279-293, Grimmett & Welsh
(1986), p.45-52.
In probability theory we often use so-called generating functions to derive/prove statements regarding
the distribution of random variables/vectors or to compute moments. In this course, we study so-called
probability generating functions and moment generating functions and we will give an outlook on what
characteristic functions are.

13.1 Probability generating functions

First of all, we introduce so-called probability generating functions and explain why they are extremely
useful!
Throughout this section, we will only consider discrete random variables taking values in the non-
negative integers, i.e. ImX ⊆ N ∪ {0} = {0, 1, 2, . . . }.
Definition 13.1.1 (Probability generating function (p.g.f.)). Let X denote a discrete random variable with
ImX ⊆ N ∪ {0}. We denote by
( ∞
)
X
x
SX = s ∈ R : |s| P(X = x) < ∞ .
x=0

Then the probability generating function (pgf) of X is defined as the function GX : SX → R given by
∞
X
GX (s) = E(sX ) = sx P(X = x).
x=0

We observe that the pgf is well-defined for |s| ≤ 1 since

∞
X ∞
X
|s|x P(X = x) ≤ P(X = x) = 1 < ∞.
x=0 x=0

Also, GX (0) = P(X = 0) and GX (1) = 1.

The reason why probability generating functions are extremely useful is that they uniquely determine
the probability mass function (i.e. the distribution) of a discrete random variable:
Theorem 13.1.2. Let X, Y denote discrete random variables with ImX, ImY ⊆ N ∪ {0}. Their p.g.f.s are
denoted by GX and GY , respectively. Then
GX (s) = GY (s), for all s ∈ SX ∩ SY , (13.1.1)
if and only if
P(X = x) = P(Y = x), for all x = 0, 1, 2, . . . . (13.1.2)

87
A. E. D. Veraart MATH40005: Probability and Statistics Autumn 2023

Proof. Assume that (13.1.1) holds. First we note that GX (0) = GY (0) implies P(X = 0) = P(Y = 0).
When we differentiate1 the pgfs we get
∞
X
G′X (s) = xsx−1 P(X = x).
x=1

When we plug in s = 0 in the first derivative, we get G′X (0) = P(X = 1). Hence, G′X (0) = P(X = 1) =
P(Y = 1) = G′Y (0). This procedure can be repeated and we obtain

dn
GX (s) = n!P(X = n),
dsn s=0

the same can be done for GY , and the identity of the two pgfs implies the result.
The other direction of the proof is trivial.
Example 13.1.3. Let X be a discrete random variable with ImX ⊆ N ∪ {0}. Suppose that
1 1 5 1 10 4
GX (s) = + s + s + s12 .
3 5 5 15
Find the p.m.f. of X. Recall that
∞
X
GX (s) = E(sX ) = sx P(X = x) = P(X = 0) + sP(X = 1) + s2 P(X = 2) + · · · .
x=0

Hence, in the example above, we can just read off the probabilities:
1 1 1 4
P(X = 0) = , P(X = 5) = , P(X = 10) = , P(X = 12) = ,
3 5 5 15
and P(X = x) = 0 for x ̸∈ {0, 5, 10, 12}.

13.1.1 Common probability generating functions

We will now list the p.g.f.s of some common discrete distributions.
Example 13.1.4 (Bernoulli distribution). Let X ∼ Bern(p). Then

GX (s) = E(sX ) = s0 P(X = 0) + s1 P(X = 1) = 1 − p + sp

for all s ∈ R.
Example 13.1.5 (Binomial distribution). Let X ∼ Bin(n, p). Then
n  
X n
GX (s) = E(sX ) = sx px (1 − p)n−x = (1 − p + sp)n ,
x=0
x

for all s ∈ R, by an application of the binomial theorem.

Example 13.1.6 (Poisson distribution). Let X ∼ Poi(λ). Then

∞
X X∞
λx −λ (sλ)x
GX (s) = E(sX ) = sx e = e−λ = e−λ esλ = exp(λ(s − 1)),
x=0
x! x=0
x!

for all s ∈ R. Here we used the series expansion of the exponential function.
1 You will learn in the real analysis course under which conditions we are allowed to interchange the infinite sum and the derivative.

For the purpose of this course, we will just assume that the above computation is valid.

88
A. E. D. Veraart MATH40005: Probability and Statistics Autumn 2023

13.1.2 Probability generating function of a sum of independent discrete random

variables
Theorem 13.1.7. Let X, Y be independent discrete random variables with ImX, ImY ⊆ N ∪ {0}. Then

GX+Y (s) = GX (s)GY (s), for all s ∈ SX ∩ SY .

Proof. Let s ∈ SX ∩ SY . Since X and Y are independent, Exercise 6- 7 implies that sX and sY satisfy the
product formula for expectations (in fact, they are also independent and Theorem 12.7.5 applies). Hence
using we conclude that

GX+Y (s) = E(sX+Y ) = E(sX sY ) = E(sX )E(sY ) = GX (s)GY (s).

An immediate consequence of the above results is, that for independent non-negative integer-valued
random variables X1 , . . . , Xn (n ∈ N), we have
n
Y
GPni=1 Xi (s) = GXi (s),
i=1

for all s ∈ ∩ni=1 SXi .

13.1.3 Moments
We have already introduced, the mean and variance of a (discrete) random variable X. More generally, for
k ∈ N, we call E(X k ) the kth moment of X provided it exists. It turns out that we can use the probability
generating function for deriving moments of random variables. More precisely, we differentiate the pgf k
times and plug in s = 1:
Theorem 13.1.8. Let X be a discrete random variable with ImX ∈ N ∪ {0}. Let k ∈ N. Then the kth
derivative of the pgf is given by

dk
GX (s) = G(k) (1) = E[X(X − 1) · · · (X − k + 1)].
dsk s=1

Proof. As before, we assume that we are allowed to interchange derivatives and summation under suitable
conditions. Then
d d
GX (s) = E(sX ) = E(XsX−1 ).
ds ds
Hence
d
GX (s) = E(X).
ds s=1

Similarly,

dk d
k
GX (s) = E(sX ) = E[X(X − 1) · · · (X − k + 1)sX−k ].
ds ds
Hence
dk
GX (s) = E[X(X − 1) · · · (X − k + 1)].
dsk s=1

89
A. E. D. Veraart MATH40005: Probability and Statistics Autumn 2023

Example 13.1.9 (Computing the variance using pgfs). The above theorem can be used for computing the
variance of a discrete non-negative integer-valued random variable X. We note that
G′′X (1) = E[X(X − 1)] = E(X 2 ) − E(X).
Hence,
Var(X) = E(X 2 ) − [E(X)]2 = G′′X (1) + G′X (1) − (G′X (1))2 .
Example 13.1.10. Compute the mean and variance of the Bernoulli, Binomial and Poisson distributions
using the probability generating functions.
• For X ∼ Bern(p), we have, for s ∈ R,
LOTUS
X
GX (s) = E(sX ) = sx P(X = x) = s0 P(X = 0) + s1 P(X = 1) = 1 − p + sp.
x

Then
d d2
GX (s) = p, GX (s) = 0.
ds s=1 ds2 s=1
Hence
E(X) = G′X (1) = p, Var(X) = G′′X (1) + G′X (1) − (G′X (1))2 = p − p2 = p(1 − p).

• For X ∼ Bin(n, p), we have, for s ∈ R,

X n
X   Xn  
LOTUS n x n
GX (s) = E(sX ) = sx P(X = x) = sx p (1 − p)n−x = (sp)x (1 − p)n−x
x x=0
x x=0
x
Binomial theorem n
= (sp + 1 − p) .
Then
d
GX (s) = n(sp + 1 − p)n−1 p s=1
= np,
ds s=1
d2
GX (s) = n(n − 1)(sp + 1 − p)n−2 p2 = n(n − 1)p2 .
ds2 s=1
s=1

Hence
E(X) = G′X (1) = np,
Var(X) = G′′X (1) + G′X (1) − (G′X (1))2 = n2 p2 − np2 + np − (np)2 = np(1 − p).

• For X ∼ Poi(λ), we have, for s ∈ R,

X ∞
X ∞
LOTUS λx −λ X (sλ)x −λ
GX (s) = E(sX ) = sx P(X = x) = sx e = e = eλs e−λ = exp(λ(s − 1)).
x x=0
x! x=0
x!
Then
d
GX (s) = exp(λ(s − 1))λ|s=1 = λ,
ds s=1
d2
GX (s) = exp(λ(s − 1))λ2 = λ2 .
ds2 s=1
s=1

Hence
E(X) = G′X (1) = λ,
Var(X) = G′′X (1) + G′X (1) − (G′X (1))2 = λ2 + λ − λ2 = λ.

End of lecture 17. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

90
A. E. D. Veraart MATH40005: Probability and Statistics Autumn 2023

13.2 Moment generating functions

Definition 13.2.1 (Moment generating functions). Let X be a random variable. Then its moment generat-
ing function (m.g.f.) is defined as
MX (t) = E(etX ),
provided the expectation exists in some neighbourhood of zero, i.e. the expectation exists for all |t| < ϵ for
some ϵ > 0.
We compute the m.g.f. as follows:
 P tx
MX (t) = E(etX ) = R ∞x e txpX (x), if X is discrete,
−∞
e fX (x)dx, if X is continuous,
whenever the sum/integral is (absolutely) convergent.
Remark 13.2.2. Relation between the p.g.f. and the m.g.f. for discrete random variables: Let X be a
discrete random variable with ImX ⊆ N ∪ {0}. Then
MX (t) = E(etX ) = E((et )X ) = GX (et ).
Remark 13.2.3. Why is MX called the moment generating function?
Let X be a random variable. Then its moment generating function (m.g.f.) is defined as

MX (t) = E(etX ),
provided the expectation exists in some neighbourhood of zero, i.e. the expectation exists for all |t| < ϵ for
some ϵ > 0. Then
∞
! ∞
X (tX)n (∗) X tn
tX
MX (t) = E(e ) = E = E(X n ) .
n=0
n! n=0
n!
Note that, in general, we are not allowed to interchange an infinite sum with the expectation. However,
here the equality in (*) holds, since we assume the existence of the moment generating function in a neigh-
bourhood of zero.
Also, we can do a Taylor series expansion of MX (t) around 0, which leads to
∞
X tn
MX (t) = M n (0) .
n=0
n!

Clearly, for the two infinite series to be the same, we need that M (n) (0) = E(X n ).
Example 13.2.4. Let X ∼ N(0, 1). Then we use the trick of ”completing the square” in the second line:
Z ∞ Z ∞
tx 1 − 21 x2 1 1 2
MX (t) = e √ e dx = √ e− 2 x +tx dx
−∞ 2π −∞ 2π
Z ∞
1 − 1 (x2 −2tx+t2 ) t2
= √ e 2 e 2 dx
−∞ 2π
Z ∞
t2 1 1 2 t2
=e 2 √ e− 2 (x−t) dx = e 2 ,
−∞ 2π
since the latter integral is equal to 1 since it is the integral of a N(t, 1) density function. We note that the
m.g.f. exists for all t ∈ R in this case.
Let us now consider an example where the m.g.f. does not exist for all t ∈ R.
Example 13.2.5. Let X ∼ Exp(λ), then
Z ∞ Z ∞
MX (t) = E(etX ) = etx λe−λx dx = e(t−λ)x λdx
0 0
 λ
λ−t , if t < λ,
=
∞, if t ≥ λ.

91
A. E. D. Veraart MATH40005: Probability and Statistics Autumn 2023

13.2.1 Properties
Theorem 13.2.6. If X has a m.g.f., then for k ∈ N, the kth moment of X is given by

(k) dk
E(X k ) = MX (0) = MX (t) .
dtk t=0

Proof. We give a sketch proof of the theorem. Assuming we can interchange expectation and differentia-
tion, we write
 k 
dk dk tX d tX

MX (t) = k E(e ) = E e = E X k etX ,
dtk dt dtk

and then plug in t = 0.

Theorem 13.2.7. For a, b ∈ R, we have MaX+b (t) = ebt MX (at).
Proof.

MaX+b (t) = E{exp[t(aX + b)]} = E[etaX+tb ] = etb E[exp(taX)] = etb MX (at).

Example 13.2.8. Let Z ∼ N(0, 1). Let µ ∈ R, σ > 0, X := µ + σZ, then

2 2 2 2
MX (t) = etµ MZ (σt) = eµt eσ t /2
= eµt+σ t /2
,

which is the m.g.f. of an N(µ, σ 2 ) distributed random variable. We can find the mean of X, by
2 2
′
E(X) = MX (0) = eµt+σ t /2
(µ + σ 2 t) = µ.
t=0

Theorem 13.2.9. Let X1 , . . . , Xn denote a sequence of independent random variables with m.g.f.s MX1 , . . . , MXn .
Then
n
Y
MPni=1 Xi (t) = MXi (t).
i=1

Proof. Since the expectation of a product of independent random variables is the product of their corre-
sponding expectation, see Theorem 12.7.5, we have
" n
!# " n # n n
X Y Y Y
MPni=1 Xi (t) = E exp t Xi =E exp(tXi ) = E(etXi ) = MXi (t).
i=1 i=1 i=1 i=1

Here we used that continuous transformations (with f (x) = etx ) of independent random variables are
independent, too, see Remark 12.2.5.
We will now state without proof the famous characterisation theorem:
Theorem 13.2.10 (Characterisation). If the m.g.f.s of the random variables X and Y exist and MX (t) =
MY (t) in a neighbourhood of zero, then

FX (u) = FY (u) for all u.

The above theorem states that m.g.f. characterise the distribution of a random variable uniquely.

92
A. E. D. Veraart MATH40005: Probability and Statistics Autumn 2023

13.3 Using m.g.f.s for finding all moments of the exponential and the
standard normal distributions
Example 13.3.1. We would like to find all the moments of the exponential distribution:
• Let X ∼ Exp(1). Then MX (t) = (1 − t)−1 for all t < 1. Using the geometric series for |t| < 1, we
have
∞
X X∞ X∞
1 tn tn
MX (t) = = tn = n! = E(X n ) ,
1 − t n=0 n=0
n! n=0 n!

hence E(X n ) = n! for all n ∈ N.

• Now consider the general case, when Y ∼ Exp(λ). Then X := λY ∼ Exp(1). To see this, note
that, for x > 0,

FX (x) = P(X ≤ x) = P(λY ≤ x) = P(Y ≤ x/λ) = FY (x/λ) = 1 − exp(−x),

and FX (x) = 0 for x < 0, which is the c.d.f. of an Exp(1)-distributed random variable. Then, we
can deduce that, for all n ∈ N, we have
n!
E(X n ) = n! = E(λn Y n ) = λn E(Y n ) ⇔ E(Y n ) = .
λn
In particular, E(X) = λ−1 , Var(X) = λ−2 .
Example 13.3.2. We would like to find all the moments of the standard normal distribution: Let X ∼
N(0, 1). Then
X∞ X∞ ∞
X ∞
X
2 (t2 /2)n t2n (2n)! t2n t2n
MX (t) = et /2
= = n
= n
· = E(X 2n ) .
n=0
n! n=0
2 n! n=0 2 n! (2n)! n=0 (2n)!

(2n)!
I.e. E(X 2n ) = 2n n! and E(X 2n−1 ) = 0 for all n ∈ N.
The even moments can be computed using the following identity:
(2n)!
Lemma 13.3.3. 2n n! = (2n − 1)(2n − 3) · · · 3 · 1, for n ∈ N.
Proof. We can give a story proof/proof by interpretation. Both sides count how many ways there are to
break a group of 2n people into n pairs: Left hand side: Take 2n people and label them 1 to 2n. We can
line up the 2n people (there are (2n)! possible permutations) and say that the first two are a pair, the next
two are a pair etc. Here we overcount by a factor of n! since the order of the pairs does not matter and by a
factor of 2n since the order within each pair does not matter. Right hand side: There are 2n − 1 ways to
choose a partner for the first person, then there are 2n − 3 choices for person 2 (or 3 if 2 was already paired
to person 1) etc.

13.4 Outlook: Characteristic function and Laplace transform

Note that moment generating functions do not exist for all distributions.
Hence we often work with the characteristic function of a random variable X instead which is defined
as

ϕX (t) = E(eitX ) = E[cos(tX) + i sin(tX)], for all t ∈ R,

√
where i = −1. It turns out that characteristic functions exist indeed for all distributions and hence they
are a useful tool for general proofs in probability theory. However, we will defer the detailed discussion
of complex-valued objects to a later probability/statistics (and analysis!) course.

93
A. E. D. Veraart MATH40005: Probability and Statistics Autumn 2023

For a non-negative random variable X we sometimes work with the Laplace transform instead which
is defined as

LX (t) = E(e−tX ), for all t ≥ 0.

We note that LX (t) = MX (−t) for t ≥ 0.

Example 13.4.1. Let X ∼ Exp(λ), then, for t ≥ 0,

Z ∞ Z ∞
LX (t) = E(e−tX ) = e−tx λe−λx dx = e−(t+λ)x λdx
0 0
λ
= .
λ+t

End of lecture 18. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

94
Chapter 14

Conditional distribution and

conditional expectation

The material of this chapter is based on Blitzstein & Hwang (2019), p.306-311, 313-321, Grimmett &
Welsh (1986), p.32-33, 88-89, 92-95.
Let us now study conditional distributions both for discrete and continuous random variables. They
allow us to define the conditional expectation, which is a really useful concept as we shall see when stating
the law of total expectation.

14.1 Discrete case: Conditional expectation and the law of total ex-
pectation
We have already introduced the notation of conditional probabilities. Now we are going to define the
conditional distribution of a discrete random variable.

Definition 14.1.1 (Conditional distribution and conditional expectation). Let Y denote a discrete random
variable on the probability space (Ω, F, P). Consider an event B ∈ F such that P(B) > 0. The condi-
tional distribution of Y given B is defined as

P({Y = y} ∩ B)
P(Y = y|B) = , for y ∈ R.
P(B)

Further, the conditional expectation of Y given B is defined as

X
E(Y |B) = yP(Y = y|B),
y∈ImY

provided the sum is absolutely convergent.

Similarly to the ideas presented in the law of total probability, it can often be useful to consider a
partition of the probability space to compute an (unconditional) expectation via conditional expectations as
we describe in the following theorem.
Theorem 14.1.2. [Law of total expectation] Consider a partition {Bi : i ∈ I} of Ω with P(Bi ) > 0 for
all i ∈ I. Let Y denote a discrete random variable with finite expectation. Then
X
E(Y ) = E(Y |Bi )P(Bi ),
i∈I

whenever the sum converges absolutely.

95
A. E. D. Veraart MATH40005: Probability and Statistics Autumn 2023

Proof. First we use the definition of the expectation, followed by the law of total probability (Theorem
5.4.4):
X X X
E(Y ) = yP(Y = y) = y P(Y = y|Bi )P(Bi )
y y i∈I
X X X
= P(Bi ) yP(Y = y|Bi ) = P(Bi )E(Y |Bi ).
i∈I y i∈I

We use the fact that the series is absolutely convergent to justify that we are allowed to change the order of
summation.

14.1.1 Conditioning on a random variable

Suppose (X, Y ) are jointly discrete random variables. In the above definition, consider the event B =
{X = x} for some x ∈ R such that pX (x) = P(X = x) > 0. Then the conditional distribution/probability
mass function of Y given X = x is given by

pX,Y (x, y)
pY |X (y|x) = P(Y = y|X = x) = , for y ∈ R.
pX (x)

Also, the conditional expectation of Y given X = x is given by

X
E(Y |X = x) = ypY |X (y|x),
y

provided the sum is absolutely convergent.

Also, the LOTUS for conditional expectations says that
X
E(g(Y )|X = x) = g(y)pY |X (y|x).
y

Note that we can also formulate an independence condition in terms of conditional p.m.f.s: Discrete X
and Y are independent if and only if

P(Y = y|X = x) = P(Y = y)

for all x, y such that P(X = x) > 0. Also, we get a Bayes’ type result of the form

pY |X (y|x)pX (x) = pX,Y (x, y) = pX|Y (x|y)pY (y),

for all x, y for which pX (x), pY (y) > 0.

14.1.2 Example
Let us study an example:
Example 14.1.3. Suppose you sit in Heathrow waiting for your flight to go on your well deserved holiday.
You denote by N the total (random) number of planes arriving while you wait and you assume that, for
some λ > 0, N ∼ Poi(λ). Each plane, independently, turns out to be a British Airways plane with
probability p ∈ (0, 1), hence with probability 1 − p it will be a plane from another airline. We write
N = X + Y where X represents the number of British Airways planes and Y the number of planes from
other airlines. You are wondering what might be the joint probability mass function of X and Y .
You recall that the Bernoulli distribution describes binary outcomes with success probability p. So,
every time a plane you observe turns out to be a British Airways plane, you view this as a success and a
failure otherwise.

96
A. E. D. Veraart MATH40005: Probability and Statistics Autumn 2023

You recall that the number of success given the total number of trials follows a Binomial distribution,
so more precisely, in your case you have for n ∈ N
X|N = n ∼ Bin(n, p), and Y |N = n ∼ Bin(n, 1 − p).
Given this information, you try to compute P(X = x, Y = y). For this, it would be really useful to know
N , so let us apply the law of total probability given information on N . For x, y ∈ N ∪ {0}:
∞
X
P(X = x, Y = y) = P(X = x, Y = y|N = n)P(N = n).
n=0

Clearly P(X = x, Y = y|N = n) > 0 ⇔ x + y = n. So, in the sum, we can get rid off all the terms which
result in conditional probabilities being equal to 0.
∞
X
P(X = x, Y = y) = P(X = x, Y = y|N = n)P(N = n)
n=0
X
= P(X = x, Y = y|N = n)P(N = n)
n:x+y=n

= P(X = x, Y = y|N = x + y)P(N = x + y).

Conditional on the event that {N = x + y}, the events {X = x} and {Y = y} contain exactly the same
information, hence we get
P(X = x, Y = y|N = x + y)P(N = x + y) = P(X = x|N = x + y)P(N = x + y).
It remains to plug in the Binomial and Poisson p.m.f.s:
P(X = x, Y = y) = P(X = x|N = x + y)P(N = x + y)
 
x+y x λx+y −λ
= p (1 − p)y e
x (x + y)!
(x + y)! x λx+y −λ
= p (1 − p)y e
x!y! (x + y)!
(λp)x −λp (1 − p)y λy −λ(1−p)
= e · e ,
x! y!
which is in fact the product of the p.m.f. of a Poi(pλ) and a Poi((1 − p)λ) random variable. Hence, we
conclude that X and Y are independent and X ∼ Poi(pλ) and Y ∼ Poi((1 − p)λ).

14.2 Continuous case: Conditional density, conditional distribution

and conditional expectation
Let us now consider two jointly continuous random variables (X, Y ). We cannot proceed as above to define
the conditional distribution P(Y ≤ y|X = x) since we now have that P(X = x) for all x ∈ R. Hence we
need to condition on an event with non-zero probability. Let ϵ > 0, then we have
P(Y ≤ y, x ≤ X ≤ x + ϵ)
P(Y ≤ y|x ≤ X ≤ x + ϵ) =
P(x ≤ X ≤ x + ϵ)
R x+ϵ R y R
1 x+ϵ y
R
u=x v=−∞ X,Y
f (u, v)dvdu ϵ u=x v=−∞ fX,Y (u, v)dvdu
= R x+ϵ = R
1 x+ϵ
.
x
fX (u)du ϵ x fX (u)du
Ry
We define H(y, u) := v=−∞ fX,Y (u, v)dv. Then we can write
R
1 x+ϵ
ϵ u=x H(y, u)
P(Y ≤ y|x ≤ X ≤ x + ϵ) = 1 R x+ϵ .
ϵ x fX (u)du

97
A. E. D. Veraart MATH40005: Probability and Statistics Autumn 2023

Now we let ϵ → 0 and get

Ry Z
fX,Y (x, v)dv y
H(y, x) −∞ fX,Y (x, v)dv
lim P(Y ≤ y|x ≤ X ≤ x + ϵ) = = = = G(y).
ϵ→0 fX (x) fX (x) −∞ fX (x)

So G is a distribution function with density function

fX,Y (x, y)
g(y) = , for y ∈ R.
fX (x)

The derivations above only work in the case when fX (x) > 0. Let us now state our formal definition:
Definition 14.2.1 (Conditional distribution and conditional density). For two jointly continuous random
variables X, Y , we define the conditional density of Y given X = x as

fX,Y (x, y)
fY |X (y|x) = , (14.2.1)
fX (x)

for all y ∈ R and for all x ∈ R for which fX (x) > 0. The corresponding conditional distribution function
of Y given X = x is then given by
Ry
fX,Y (x, v)dv
FY |X=x (y|x) = −∞ ,
fX (x)

for all y ∈ R and for all x ∈ R for which fX (x) > 0.

Note that we can now also formulate an independence condition in terms of conditional p.d.f.s: Jointly
continuous random variables X and Y are independent if and only if

fY |X (y|x) = fY (y),

for all x, y such that fX (x) > 0.

Remark 14.2.2. Note that (14.2.1) also implies a Bayes’ type formula:

fY |X (y|x)fX (x) = fX,Y (x, y) = fX|Y (x|y)fY (y),

provided that fX (x), fY (y) > 0.

We note that we can now formulate a continuous version of the law of total probability1 :
Proposition 14.2.3. Let (X, Y ) be two jointly continuous random variables with joint probability density
function fX,Y and marginal probability densities denoted by fX and fY , respectively. Let FY |X=x denote
the conditional distribution function of Y given X = x. Then
Z
P(Y ≤ y) = FY |X=x (y|x)fX (x)dx, y ∈ R.
{x:fX (x)>0}

Proof. For y ∈ R, we have

Z y
P(Y ≤ y) = fY (v)dv,
−∞
Z ∞
fY (v) = fX,Y (x, v)dx,
−∞
fX,Y (x, v)
fY |X=x (y|x) = , for x such thatfX (x) > 0.
fX (x)
1 The proof was part of an exam question in 2021.

98
A. E. D. Veraart MATH40005: Probability and Statistics Autumn 2023

Hence, we can write

Z y Z y Z ∞
P(Y ≤ y) = fY (v)dv = fX,Y (x, v)dxdv
−∞ −∞ −∞
Z y Z
fX,Y (x, v)
= fX (x)dxdv
−∞ {x:fX (x)>0} fX (x)
Z y Z
= fY |X=x (v|x)fX (x)dxdv
−∞ {x:fX (x)>0}
Z Z y
= fY |X=x (v|x)dvfX (x)dx
{x:fX (x)>0} −∞
Z
= FY |X=x (y|x)fX (x)dx.
{x:fX (x)>0}

We note that the interchange of the integrals can be justified by Tonelli’s theorem, which states that for
non-negative functions the order of the integration can be interchanged. The proof of Tonelli’s theorem is
beyond the scope of this course.
Similarly to the discrete case, we can now define the conditional expectation and formulate the law of
total expectation.
Definition 14.2.4 (Conditional expectation). For two jointly continuous random variables X, Y , we define
the conditional expectation of Y given X = x as
Z ∞ Z ∞
fX,Y (x, y)
E(Y |X = x) = yfY |X (y|x)dy = y dy,
−∞ −∞ fX (x)
provided that fX (x) > 0.
Recall that, in the discrete case, Theorem 14.1.2 implies that for jointly discrete random variables X, Y
with E|Y | < ∞, we have
X
E(Y ) = E(Y |X = x)P(X = x),
x:P(X=x)>0

whenever the sum converges absolutely. The continuous analogue reads as follows:
Theorem 14.2.5 (Law of total expectation). For jointly continuous random variable X, Y with E|Y | < ∞,
we have
Z
E(Y ) = E(Y |X = x)fX (x)dx.
{x:fX (x)>0}

Proof. We use the definition of the expectation, the fact that the marginal density of Y can be obtained by
integrating out the joint density and equation (14.2.1):
Z Z Z
E(Y ) = yfY (y)dy = yfX,Y (x, y)dxdy
Z Z
= yfY |X (y|x)fX (x)dxdy
Z Z 
= yfY |X (y|x)dy fX (x)dx
Z
= E(Y |X = x)fX (x)dx,

where we assume that the integrals range over the appropriate values for x and y.

End of lecture 19. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

99
A. E. D. Veraart MATH40005: Probability and Statistics Autumn 2023

14.2.1 Examples
Example 14.2.6. Consider two jointly continuous random variables with joint density (for λ > 0):
 2 −λy
λ e , for 0 ≤ x ≤ y < ∞,
fX,Y (x, y) =
0, otherwise.

• Find fY |X :
fX,Y (x,y)
Recall that fY |X (y|x) = fX (x) . We compute the marginal density of X first:
Z ∞ Z ∞ ∞
(−1) −λy
fX (x) = fX,Y (x, y)dy = λ2 e−λy dy = λ2 e = λe−λx ,
−∞ x λ y=x

for x ≥ 0 and fX (x) = 0 for x < 0. Hence,

 2 −λy
λ e
= λe−λ(y−x) , for 0 ≤ x ≤ y < ∞,
fY |X (y|x) = λe−λx
0, otherwise.

• Find E(Y |X = x).

For x ≥ 0, we have
Z ∞ Z ∞ Z ∞
E(Y |X = x) = yfY |X (y|x)dy = yλe−λ(y−x) dy = eλx λye−λy dy
−∞ x x
Z ∞  Z ∞ 
u:=λy eλx ∞
= eλx ue−u duλ−1 = u(−e−u ) u=λx
− (−e−u )du
λx λ λx
eλx  1
= λxe−λx + e−λx = (λx + 1).
λ λ

Example 14.2.7. Let ρ ∈ (−1, 1). The standard bivariate normal distribution has joint density given by
 
1 1 2 2


fX,Y (x, y) = p exp − x − 2ρxy + y
2π 1 − ρ2 2 (1 − ρ2 )

for x, y ∈ R. We want to demonstrate some of the concepts introduced earlier.

1. What is the marginal density of X? We compute

Z ∞
fX (x) = fX,Y (x, y)dy
−∞
Z ∞  
1 1 2 2


= p exp − x − 2ρxy + y dy
−∞ 2π 1 − ρ
2 2 (1 − ρ2 )
Z ∞  
1 1  2 2 2


= p exp − (y − ρx) + x 1 − ρ dy
−∞ 2π 1 − ρ
2 2 (1 − ρ2 )
Z  
1 −x2 /2 ∞ 1 (y − ρx)2
=√ e p exp − dy.
2π −∞ 2π (1 − ρ2 ) 2 (1 − ρ2 )

We observe that the integrand in the above integral is the density of an N(ρx, 1−ρ2 ) random variable
and hence the integral equals 1. Hence
1 2
fX (x) = √ e−x /2 .
2π
I.e. X ∼ N(0, 1) (and also Y ∼ N(0, 1)).

100
A. E. D. Veraart MATH40005: Probability and Statistics Autumn 2023

2. What is the conditional density of Y given X = x? We have

fX,Y (x, y)
fY |X (y|x) =
fX (x)
 
1 1 2 2

√ 2
= p exp − x − 2ρxy + y 2πex /2
2π 1 − ρ2 2 (1 − ρ2 )
 
1 1 2 2

2
= √ p exp − x − 2ρxy + y + x /2
2π 1 − ρ2 2 (1 − ρ2 )
 
1 1 2 2 2 2


= √ p exp − x − 2ρxy + y − x (1 − ρ )
2π 1 − ρ2 2 (1 − ρ2 )
 
1 (y − ρx)2
= p exp − .
2π(1 − ρ2 ) 2 (1 − ρ2 )

This is in fact the density of an N(ρx, 1 − ρ2 ) random variable.

3. What is the conditional expectation of Y given X = x?
Using the definition of the conditional expectation, we have
Z ∞
E(Y |X = x) = yfY |X (y|x)dy = ρx,
−∞

given our above finding that Y |X = x ∼ N(ρx, 1 − ρ2 ).

4. Formulate a condition which ensures that X and Y are independent.
We know that X and Y are independent, if and only if fX,Y (x, y) = fX (x)fY (y) for all x, y ∈ R.
For any x, y ∈ R we have

fX,Y (x, y) = fX (x)fY (y)

⇐⇒
 
1 1
1 2 1 2
p exp − 2
x − 2ρxy + y 2
= √ e−x /2 √ e−y /2
2π 1 − ρ2 2 (1 − ρ2 ) 2π 2π
⇐⇒
 
1 1 2 2

1 − 1 (x2 +y2 )
p exp − 2
x − 2ρxy + y = e 2
2π 1 − ρ2 2 (1 − ρ ) 2π
⇐⇒ ρ = 0.

So X and Y are independent if and only if ρ = 0.

5. Find the covariance between X and Y .
We note that since E(X) = 0 = E(Y ), we have that
Z ∞
Cov(X, Y ) = E(XY ) − E(X)E(Y ) = E(XY ) = E(XY |X = x)fX (x)dx,
−∞

where we used the law of the total expectation, see Theorem 14.2.5. Note that E(XY |X = x) =
E(xY |X = x) = xE(Y |X = x) = ρx2 , hence
Z ∞ Z ∞
Cov(X, Y ) = ρx2 fX (x)dx = ρ x2 fX (x)dx = ρE(X 2 )
−∞ −∞
= ρVar(X) = ρ.

101
A. E. D. Veraart MATH40005: Probability and Statistics Autumn 2023

So, we have

ρ = E(XY ) − E(X)E(Y ),

which, with our findings above, implies the following important result: Assume that X, Y follow a
bivariate (standard) normal distribution. Then X and Y are independent if and only if E(XY ) =
E(X)E(Y ).
Warning: As soon as you drop the assumption that you are dealing with jointly normal random
variables, then we only know that if we assume that they are independent, then the product formula
for the expectations holds. However, if we have only verified that the product formula for the expec-
tations holds, then that does not imply in general independence of the random variables. We will
illustrate this in the following remark and example.
Remark 14.2.8. • If (X, Y ) is bivariate normal and Cov(X, Y ) = 0 ⇒ X, Y are independent.
• However, if X and Y follow a univariate normal distribution and Cov(X, Y ) = 0 ̸⇒ X, Y are
independent.
Example 14.2.9. Let X ∼ N(0, 1). Let Z be a discrete random variable, independent of X with P(Z =
−1) = P(Z = 1) = 21 . Let Y := Z · X. We want to show that
1. Y ∼ N(0, 1),
2. Cov(X, Y ) = 0,
3. X and Y are not independent.

1. Y ∼ N(0, 1):
Let y ∈ R. Then, using the law of total probability and the independence of Z and X, we have

FY (y) = P(Y ≤ y) = P(ZX ≤ y)

= P(ZX ≤ y|Z = −1)P(Z = −1) + P(ZX ≤ y|Z = 1)P(Z = 1)
1 1
= (P(−X ≤ y) + P(X ≤ y)) = (P(X ≥ −y) + Φ(y))
2 2
1 1
= (1 − P(X ≤ −y) + Φ(y)) = (1 − Φ(−y) + Φ(y))
2 2
1
= (1 − (1 − Φ(y)) + Φ(y)) = Φ(y).
2
Hence Y ∼ N(0, 1).
2. Cov(X, Y ) = 0:

independence of X,Z
Cov(X, Y ) = E(XY ) − E(X)E(Y ) = E(X 2 Z) = E(X 2 )E(Z) = 1 · 0 = 0.

3. X and Y are not independent:

We note that |X| = |Y | is always true, hence X and Y are not independent.

As a side remark, we note that the sum of X and Y is not normally distributed (and not even continuous!):

P(X + Y = 0) = P(X + ZX = 0) = P(X(1 + Z) = 0)

= P(X(1 + Z) = 0|Z = 1)P(Z = 1) + P(X(1 + Z) = 0|Z = −1)P(Z = −1)
1 1 1
= P(2X = 0) · + P(0 = 0) · = ̸= 0.
2 2 2
Conclusions: If we only know that X and Y follow univariate normal distributions, we cannot con-
clude that

102
A. E. D. Veraart MATH40005: Probability and Statistics Autumn 2023

• (X, Y ) has a bivariate normal distribution,

• (X, Y ) are jointly continuous,
• Cov(X, Y ) = 0 ⇒ X, Y are independent.

End of lecture 20. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

103
Bibliography

Anderson, D. F., Seppäläinen, T. & Valkó, B. (2018), Introduction to probability, Cambridge Mathematical
Textbooks, Cambridge University Press, Cambridge.
Blitzstein, J. K. & Hwang, J. (2019), Introduction to probability, Texts in Statistical Science Series, CRC
Press, Boca Raton, FL. Second edition.
Feller, W. (1957), An introduction to probability theory and its applications. Vol. I, John Wiley and Sons,
Inc., New York; Chapman and Hall, Ltd., London. 2nd ed.
Grimmett, G. & Welsh, D. (1986), Probability: an introduction, Oxford Science Publications, The Claren-
don Press, Oxford University Press, New York.
Hájek, A. (2012), Interpretations of probability, in E. N. Zalta, ed., ‘The Stanford Encyclopedia of Philos-
ophy’, Winter 2012 edn, Metaphysics Research Lab, Stanford University.
Proschan, M. A. & Shaw, P. A. (2016), Essentials of probability theory for statisticians, Chapman &
Hall/CRC Texts in Statistical Science Series, CRC Press, Boca Raton, FL.
Ross, S. (2014), A first course in probability, ninth edn, Macmillan Co., New York; Collier Macmillan Ltd.,
London.

Spiegelhalter, D., Pearson, M. & Short, I. (2011), ‘Visualizing uncertainty about the future’, Science
333(6048), 1393–1400.

104