MATH5741M: Statistical Theory and

Methods
Stuart Barber ([Link]@[Link])
Sunday October 01, 2023

Module information

Frequently asked questions

How is the module structured?

The content is split into the following chapters, each to be covered in a period of around
one to two weeks.1

1. Introduction

2. Exploratory Data Analysis

3. Statistical Models

4. Estimation

5. Frequentist Inference

6. Bayesian Inference

How is the module assessed?

You will be assessed via an exam (80% of your module mark) and a coursework
assignment (20%). The weekly exercises do not count towards your final mark.
The exam timetable will eventually be published at
[Link] I do not know precisely when, and I
cannot alter the exam timetable. Given that the exam timetable sometimes changes
unexpectedly, you are expected to be in Leeds for the entire exam period.

What kind of feedback is provided?

After the deadline of each weekly homework, you will be given detailed solutions. You will
also receive feedback on your coursework before the exam.

Is there any compulsory extra reading?

I will suggest some relevant reading at the beginning of each chapter, but that reading is
not compulsory. Some relevant books are:

1. Probability and Statistics, by DeGroot and Schervish;

2. Probability and Statistical Inference, by Hogg, Tanis and Zimmerman;

3. Biostatistics with R: an introductory guide for field biologists, by Leps And

Smilauer.

These are available online through the library. The first two are similar, but the one by
DeGroot and Schervish is generally stronger and has more content, and that’s the one I
will refer you to most often (although having access to both will give you a larger
collection of exercises and examples). The third book might be of particular interest to
those of you with a biology background, and it also has the benefit of using the R
programming language.

Remember that all of these books contain some topics that are not covered in this
module. So please don’t imagine that you need to read them thoroughly, and try not to
get too distracted by topics that are not required here (e.g. ANOVA and linear models are
interesting, but you don’t need them right now).

Of course, there are many other helpful books not mentioned here. You are very strongly
encouraged to find books that meet your own specific needs, for example by simply
visiting the library. Finding materials to support your own learning is an important skill, so
don’t be lazy and don’t expect your lecturers to provide you with everything.
Even books on A-Level statistics could be helpful sources of examples and exercises,
particularly for probability and hypothesis testing (A-Levels are the exams taken by most
students in England before going to university). You might find cheap second-hand
copies in charity shops. Even very old A-Level textbooks are likely to be relevant. But
please remember that we will cover many topics in this module that go beyond A-Level.

Is programming required?

Yes, it definitely is. You cannot avoid programming in this module. R will be used in the
practical classes and for the coursework. Questions related to R could appear on the
exam (e.g. interpretation of code or output). You should install R ([Link]
[Link]/) on your computer before attempting any part of the module. I also
recommend installing the integrated development environment RStudio
([Link] which makes writing R scripts, saving them and sharing them a lot
easier.

What is expected of students in this module?

You should try to attend all lectures and practical classes.

You should work on any relevant gaps in your maths knowledge.

You should fully engage with the teaching materials (attempt all exercises, run all
code provided, and ask me about anything that is not clear).

You should work on the coursework continuously throughout the module (it will
be treated as part of your learning, not just as an assessment).

I will set the exam under the assumption that you have done all of the above.

Is there a PDF version of the notes? Will I get a printed copy?

The notes are in HTML files to make them more accessible for students who are visually
impaired. You can convert the HTML files to PDF yourself in a few easy steps:

1. Open the HTML file in your browser.

2. Wait a few seconds for the equations to reach full size.

3. Click “Print” and set the destination to “save as PDF”.

 4. Choose your desired settings before saving (e.g. to fit more on each page you
can reduce the scale, or you can choose “two pages per sheet”).

I will not hand out printed copies because the notes are likely to get small updates or
corrections throughout the semester. If you print the HTML notes on paper, you should
first wait a few seconds for the equations to reach their full size, otherwise your printout
might look weird.

Mathematical prerequisites
Some knowledge of maths is assumed in the module. If you feel that you are struggling
with this then please let me know. If you don’t tell me, there’s nothing I can do to help. If
you do tell me, then I’m happy to spend some extra time with you to explain anything you
don’t understand.

For revision of basic maths (around the standard of first-year A-level modules, including
calculus), you shouldn’t have much difficulty finding books at an appropriate level in the
library or in any bookshop. For example, the library provides free access to:

Mathematics for Engineers and Technologists, by Fox and Bolton (Chapters 1

and 3 are most relevant);

Probability and Statistics: A Course for Physicists and Engineers, by Mathai and
Haubold (Chapter 1 is most relevant).

Notation
In maths and statistics modules, the following symbols are quite common:

⇒ means “implies”. For example, “x < 3 ⇒ x < 4 ” means “x < 3 implies

that x < 4 ” or, equivalently, “if x < 3 then x < 4 ”. This symbol is used to relate
two statements (one before the arrow, one after the arrow) and tells us that if the
first statement is true then the second statement must also be true.

⇔ means “if and only if” or “is equivalent to”. For example, “
x = y/3 ⇔ y = 3x” means “the expression x = y/3 is equivalent to the
 expression y = 3x ”.

∼ (tilde) usually means “has the distribution”. This will become clearer when
we discuss statistical models later in the module.

It is important to remember that we can only say two statements are equivalent (
A ⇔ B) if each statement implies the other (both A ⇒ B and B ⇒ A). For example,

we cannot say x < 3 ⇔ x < 4 because it is not true that x < 4 ⇒ x < 3 .

None of these three symbols above mean “equal to”. For example, ⇔ denotes logical
equivalence between two statements (statements that can be either true or false), it is
not used to denote equality between two numbers. So writing A ⇔ B (which makes
sense if A and B are logical statements) is not the same as writing A = B (which
makes sense if A and B take numerical values). Also, ∼ does not mean “approximately
equal to” (which would be denoted ≈).

Sequences
You won’t need a deep understanding of concepts such as convergence of sequences,
but you should at least make sure that you are comfortable with the notation for finite or
infinite sequences of numbers:

x1 , x2 , x3 , …

where xi is, for each i = 1, 2, 3, … , some number indexed by i . The notation

n n

∑ xi = x1 + x2 + … + xn and ∏ xi = x1 × x2 × … × xn

i=1 i=1

denote the sum and product (respectively) of the first n terms of our sequence. We will
sometimes write

xi → L as i → ∞,

which means that our sequence “tends to” (or “converges to” or “approaches”) the
number L when we make i extremely large. We say that L is the limit of the sequence.
Sets
We sometimes work with sets of numbers. This includes the set of all real numbers
(denoted R), the set of all integers (denoted Z , which includes all positive and negative
“whole numbers”, and also zero), and the set of all natural numbers (denoted N, which
includes only positive integers). For example, the notation

x ∈ R

should be read as “x is in the set of real numbers” or, equivalently, “x is an element of

the real numbers” (so ∈ just means “in”). The notation

Z ⊂ R

should be read as “the integers are a subset of the real numbers” (i.e. every x in the
integers must also be in the reals). Taking this notation further, we can define specific
subsets of R, Z and N. For example, we can write the set of even integers as

{x ∈ Z : x is even},

which we read as “all x in the integers such that x is even” (we read the colon as “such
that”). We might write the set of all real numbers between 2 and 4 as

{x ∈ R : 2 < x < 4},

which we read as “all real x such that x is more than 2 and less than 4”. This set is
actually an interval of the real line, i.e. a set that contains all real numbers between two
endpoints (with the endpoints here being 2 and 4). It is an open interval because it does
not contain the endpoints 2 and 4. We usually write open intervals as

(a, b) = {x ∈ R : a < x < b}.

We write closed intervals (which contain the endpoints) as

[a, b] = {x ∈ R : a ≤ x ≤ b}.

We can even define half-open intervals that contain only one of the endpoints,

[a, b) = {x ∈ R : a ≤ x < b} or (a, b] = {x ∈ R : a < x ≤ b}.

We can define new sets as combinations of two other sets (denoted here by A and B):

The intersection is A ∩ B = {x : x ∈ A and x ∈ B} .

 The union is A ∪ B = {x : x ∈ A or x ∈ B} .

The difference is A ∖ B = {x : x ∈ A and x ∉ B} , where ∉ means “not

in”.

For example, please check that you understand why

(1, 4) ∪ (3, 100) = (1, 100) and (1, 4] ∩ (3, 100] = (3, 4].

The notation for intersections, unions and differences doesn’t just apply to sets of
numbers. We can consider sets of objects, sets of people, sets of outcomes of an
experiment, and so on.

Functions
To denote a function or map from a set A to a set B, we will write

f : A → B,

which we read as “f maps each element of A to an element of B”. For each x ∈ A, we

denote by f (x) the corresponding element of B. The important feature of a function is
that it maps each element of the domain A to exactly one element of the codomain B.
We will sometimes use the “maps to” notation such that, for example, we would write the
function f (x) = x
2
as

2
f : x ↦ x .

We will denote the inverse of f by f −1 . This is the function f −1 : B → A such that if

f : a ↦ b then f : b ↦ a. In other words, such that f
−1 −1
(f (x)) = x for all x ∈ A .
For example, the exponential function
x
exp(x) = e

(where e = 2.718282 … ) has inverse log(x), called the natural logarithm, such that

log(exp(x)) = x = exp(log(x)).

Unless stated otherwise, any logarithm in this module is assumed to be a natural (base
e) logarithm (often written as ln(x) in books and in other modules). You are expected to

be familiar with the basic rules for logarithms and exponents. For example,
log(ab) = log a + log b log a , b
,
= b log a exp(a + b) = exp(a) exp(b) and
exp(ab) = exp(a)
b
.
Calculus
You need to be able to differentiate and integrate simple functions (e.g. polynomials,
logarithms and the exponential function). You need to be able to differentiate using both
the product rule,

d df dg
(f (x) × g(x)) = (x) × g(x) + f (x) × (x),
dx dx dx

and the chain rule,

d dg df
f (g(x)) = (x) × (g(x)).
dx dx dx

We will sometimes integrate products of functions using integration by parts,

df
∫ f (x) × g(x) dx = f (x) × ∫ g(x) dx − ∫ ( (x) × ∫ g(x) dx) dx.
dx

I will use integration by parts in lectures, but I will not expect you to use it yourself. You
will not be expected to evaluate difficult integrals, for example integrals that require the
use of partial fractions. You definitely don’t need to know how to integrate or differentiate
trigonometric functions.

Some textbooks will use the notation

df
′
(x) = f (x),
dx

which has the benefit of simplifying formulae. For example,

′ ′ ′ ′ ′ ′
(f × g) = f × g + f × g and (f ∘ g) = g × f ∘ g

are the product rule and chain rule, respectively (here f ∘ g just denotes the composition
of f and g).

Recent students have found [Link] and

[Link] useful. To get the most benefit
out of either of these links, focus on differentiation of functions that involve polynomials,
exponential functions and natural logarithms. Try some questions that involve the product
rule or chain rule. Don’t bother with questions that require the quotient rule or
differentiation of trigonometric functions.
It might even be helpful to look for books on A-Level or AS-Level Pure Mathematics.
Again, it does not matter much how old they are. Just look for the ones that introduce
students to differentiation and that should be more than enough.

1 Introduction
Relevant reading: DeGroot and Schervish, Chapters 1 and 2.

1.1 Statistics and data science

“I would claim that the unique strength of the statistics discipline is the extent of
its relevance to the whole of the natural and social sciences.” - Peter Diggle,
Statistics: a data science for the 21st century

As the quote from Professor Diggle suggests, statistics is very widely applicable to
science, including physics, biology and chemistry. The uses of statistics are particularly
well established in medicine, where randomised clinical trials allow us to assess the
effectiveness of medical treatments and where probabilistic sensitivity analyses allow us
to compare the cost-effectiveness of different treatments. As an exercise, please think
about your favourite field of scientific research and investigate the applications of
statistics or probability in that field.

But how is statistics related to data science? Do they overlap? Are they just the same
subject under different names? Are they in competition with each other? Is one just a
subset of the other? These questions have been discussed by statisticians who are
much more experienced than I am:

“There’s so much that goes on with “If data science is a close relation of
data that is about computing, not statistics, its increasing popularity must
statistics. I do think it would be fair to surely present us [statisticians] with an
consider statistics as a subset of data opportunity. We should embrace data
science. The tech industry has always science, proudly assert what we can
had to deal with databases and coding; offer it and humbly acknowledge what
that stuff is a necessity. The statistical we can learn from it. What can we
part of data science is more of an offer? Crucially, we can assert that
option.” - Andrew Gelman, Columbia uncertainty is ubiquitous and that
University probability is the correct way to deal
with uncertainty. We understand the
“Statistics can do all sorts of things. I uncertainty in our data by building
love statistics! But it’s not the most stochastic models, and in our
important part of data science, or even conclusions by probabilistic inference.” -
close.” - Andrew Gelman, Columbia Peter Diggle, Statistics: a data science
University for the 21st century

If your main interest is data science, Professor Gelman’s comments might lead you to
believe that statistics can be safely avoided. This might even be true. But it does not
mean that statistics cannot be very useful in data science, as it is in other sciences. As
Professor Diggle suggests, there is potential for statisticians (and other scientists with a
strong knowledge of statistics) to make meaningful contributions to data science.

In particular, Diggle’s comment highlights the fact that statistical methods offer the
correct way to handle uncertainty. This is important because, when we make conclusions
based on observational or experimental data, there will always be uncertainty in the data
and therefore uncertainty in our conclusions. Statistical models allow us to describe
mathematically the uncertainty in our data, and then statistical inference allows us to
quantify the uncertainty in our conclusions. Any good scientist (or data scientist) should
always try to measure the uncertainty in their own conclusions, and statistics gives them
the tools required to do this.

1.2 Probability
Before we can do anything with statistics, we need to understand some probability. Most
of the ideas here will be intuitively obvious, but we should make the notation and
definitions as clear as possible. We will achieve this through an example. Throughout
this module, motivating examples will be used to introduce any theory.

Example 1.1 (prevalence of a disease) In a population of people, the prevalence of a

disease is the proportion of people in the population who have that disease. Suppose
that the prevalence of a specific disease is summarised in Figure 1.2.
## Warning: The `returnclass` ar
gument of `ne_download()` sp as
of rnaturalearth 1.0.0.
## ℹ Please use `sf` objects with
{rnaturalearth}, support for Spa
tial objects
## (sp) will be removed in a f
uture release of the package.
## This warning is displayed onc
e every 8 hours.
## Call `lifecycle::last_lifecyc
le_warnings()` to see where this
warning was
## generated.

Figure 1.2: Prevalence of the

disease in England, Scotland,
Wales and Northern Ireland (1% ,
5% , 2% and 2% , respectively).

Figure 1.1: Proportion of the UK

population in England, Scotland,
Wales and Northern Ireland (84%,
 8% , 5% and 3% , respectively).

We ask two questions:

1. What is the overall prevalence in the UK?

2. What proportion of individuals with the disease are in Scotland?

To rephrase these questions in the language of probability, we consider an experiment

where we randomly select a person from the UK, in such a way that each person is
equally likely to be selected. Our questions become:

1. What is the probability this person will have the disease?

2. If we see that they have the disease, then what is the probability they are from
Scotland?

With some common sense, it would be possible to answer both questions without turning
them into questions about probabilities. However, we introduce probability here for two
reasons: to demonstrate that many of the ideas used in probability align well with what
we think of as common sense, and because probability gives us the tools to eventually
answer more complex questions for which common sense might not be enough.

1.2.1 Basic definitions

We have rephrased our questions in terms of a random experiment. But what do we
mean by “equally likely”? If the size of the UK population is N , then we have N possible
outcomes for our experiment (N people who could be selected). “Equally likely” means
that if we could (hypothetically) repeat our experiment infinitely many times, no outcome
would occur more or less frequently than any other. In other words, we might say that

1
P(ω) =
N

is the probability of each possible outcome ω (for us, ω would be a person in the UK
population). But we need to go further than this. We need define probabilities more
generally, so that we can apply them in situations where outcomes might not be equally
likely or where there are infinitely many possible outcomes.
 Definition 1.1 A sample space (usually denoted Ω) is the set of all possible
outcomes of an experiment. An event is a set of possible outcomes (a subset of Ω).

In our example, Ω is the UK population. We can define events E, S , W and I to be the

populations of England, Scotland, Wales and Northern Ireland, respectively.

Definition 1.2 Let A and B be events. We can define further events via set
operations:

The complement, Ac , is set of outcomes not in A.

The intersection, A ∩ B, is the set of outcomes in both A and B.

The union, A ∪ B, is the set of outcomes in A or B (possibly both).

The difference, A ∖ B , is the set of outcomes in A and not in B.

Let’s describe the following events in plain English.

Ω means “from the UK”.

S means “from Scotland”.

E ∩ S means “from England and from Scotland” (which is just an empty set).

E ∪ S means “from England or from Scotland”.

S
c
means “from the UK and not from Scotland”.

Ω ∖ S means “from the UK and not from Scotland”.

Ω ∩ S
c
means “from the UK and not from Scotland”.

E ∪ S ∪ W ∪ I means “from England or Scotland or Wales or Northern

Ireland”.
No two of the events E, S , W and I can occur simultaneously, so we say that these
four events are mutually exclusive. Their union is the full sample space, forming a
partition:

Ω = E ∪ S ∪ W ∪ I.

Definition 1.3 Let B1 , B2 , … , Bk be events. If Bi ∩ Bj is empty (contains no

outcomes) for every pair of events (i ≠ j), we say these k events are mutually
exclusive (some textbooks use the terms disjoint or mutually incompatible
instead).

Notation. For the union of many events or sets, B1 ∪ B2 ∪ … ∪ Bk , we sometimes

k
write ⋃i=1 Bi .

Definition 1.4 Let B1 , B2 , … , Bk be mutually exclusive. If

⋃ Bi = Ω

i=1

then these k events are a partition of Ω.

In other words, events are mutually exclusive if they don’t overlap (cannot happen
together). They are a partition if they don’t overlap and, when combined, they include
every possible outcome.

Definition 1.5 Given a sample space Ω, consider a function

B ↦ P(B)

that maps every event B to a number P(B) ∈ [0, 1] . Suppose that P has the
following two properties: P(Ω) = 1 and
 P (B 1 ∪ B 2 ∪ B 3 ∪ …) = P(B 1 ) + P(B 2 ) + P(B 3 ) + ⋯

whenever B1 , B2 , B3 , … are mutually exclusive. Then, for events B ⊂ Ω , we can

say the numbers P(B) are probabilities.

It is important to remember that there is no single correct way to assign probabilities. In

other words, this definition does not imply that there is a unique function P for any
sample space Ω. It simply gives us the properties that such a function P must have.

The truth is that we will rarely use this definition directly, so we don’t need to remember it
(we state it here for completeness and so that we understand it if we see it in textbooks).
Instead, we usually just think of P as a way to assign to any event a number between 0
and 1 that describes how likely that event is. The bigger the number, the more likely the
event (the more often it will occur if our experiment is repeated over and over again). The
two properties required in the definition just mean that these numbers are only really
considered to be probabilities if:

Ω has probability 100% (which make perfect sense because Ω it contains all
possible outcomes).

The probability of any union of non-overlapping events is just the sum of their
individual probabilities.

When phrased like this, the definition seems less intimidating and just fits nicely with our
intuitive understanding of how probabilities should work. When carrying out probability
calculations practice, rather than using the definition, we use some consequences of the
definition, such as the following theorem.

Theorem 1.1 For any events A and B, we have:

c
P(A ) = 1 − P(A),

P(A ∪ B) = P(A) + P(B) − P(A ∩ B),

P(A ∖ B) = P(A) − P(A ∩ B).

Based on Figure 1.1, if each person in the UK is equally likely to be selected then we
should have the following probabilities,

P(E) = 0.84, P(W ) = 0.05, P(S ) = 0.08, P(I ) = 0.03.

Consistent with Definition 1.5,

P(Ω) = P(E ∪ S ∪ W ∪ I ) = P(E) + P(S ) + P(W ) + P(I ) = 1.

As quick examples, it follows from the theorem that

P(E ∪ S ) = P(E) + P(S ) − P(E ∩ S ) = 0.84 + 0.08 − 0 = 0.92

and
c
P(E ) = 1 − P(E) = 1 − 0.84 = 0.16.

Thus, there is a 92% probability of selecting someone from England or Scotland, and a
16% probability of selecting someone not from England. We can even combine the rules

from the theorem to see that

c
P ((E ∪ S ) ) = 1 − P(E ∪ S ) = 1 − 0.92 = 0.08,

so there is only an 8% chance of selecting someone not from England or Scotland.

1.2.2 Conditional probability

Continuing our disease prevalence example, let D be the set of individuals in the UK
who have the disease. Then P(D) can be thought of as the prevalence. Can we use
Figure 1.2 to immediately write down P(D)? No, because Figure 1.2 actually gives us
conditional probabilities. For example, 5% is the probability of a random person having
the disease given that they are from Scotland.

Definition 1.6 The conditional probability of A given B is

P(A ∩ B)
P(A ∣ B) = .
P(B)
Of course, we assume in this definition that P (B) > 0 , otherwise the conditional
probability would not be defined. Throughout this module, whenever we work with
conditional probabilities, we will assume that the events we condition on have nonzero
probability.

All rules for probabilities also hold for conditional probabilities. Therefore we can handle
unions, intersections and complements for conditional probabilities just as we would for
ordinary probabilities. For example, the probability of someone not having the disease
given that they are from Scotland would be
c
P(D ∣ S ) = 1 − P(D ∣ S ) = 1 − 0.05 = 0.95.

Since P(S ) = 0.08, the definition allows us to calculate the proportion of the UK
population who are both in Scotland and have the disease

P(D ∩ S ) = P(D ∣ S )P(S ) = 0.05 × 0.08 = 0.004.

To answer question 1, we need P(D). But we only know the conditional probabilities:

P(D ∣ E) = 0.01, P(D ∣ S ) = 0.05, P(D ∣ W ) = 0.02, P(D ∣ I ) = 0.02.

Since E, S , W and I form a partition, it might make intuitive sense to write

P(D) = P(D ∩ E) + P(D ∩ S ) + P(D ∩ W ) + P(D ∩ I )

= P(D ∣ E) P(E) + P(D ∣ S ) P(S ) + P(D ∣ W ) P(W ) + P(D ∣ I ) P(I )

since every outcome in D must be in either D ∩ E or D ∩ S or D ∩ W or D ∩ I . It

turns out that this actually is the correct calculation to perform, and it is an example of
the partition theorem.

Theorem 1.2 (Partition Theorem) Let B1 , B2 , … , Bk be a partition. Then, for

any event A,

P(A) = ∑ P(A ∣ B i )P(B i ).

i=1

This also applies when the partition consists of infinitely many events, B1 , B2 , …
(just replace k with ∞ in the sum). The partition theorem is sometimes called the
law of total probability.
In general, we usually apply the partition theorem when we want the probability of an
event but only have conditional probabilities for that event. Our previous equation is just
an example of this theorem, where we set B1 = E, B2 = S , B3 = W , B4 = I and
A = D to get

P(D) = ∑ P(D ∣ B i )P(B i )

i=1

= P(D ∣ E)P(E) + P(D ∣ S )P(S ) + P(D ∣ W )P(W ) + P(D ∣ I )P(I )

= 0.01 × 0.84 + 0.05 × 0.08 + 0.02 × 0.05 + 0.02 × 0.03

= 0.014.

This answers question 1: overall prevalence in the UK is 1.4% .

1.2.3 Bayes’ theorem

To answer question 2, we apply the definition of conditional probability to write P(S ∣ D)

in terms of P(D ∣ S ):

P(S ∩ D)
P(S ∣ D) =
P(D)

P(D ∣ S )P(S )
=
P(D)

0.05 × 0.08
= ≈ 0.3,
0.014

which tells us that around 30% of people with the disease are in Scotland. This
calculation is an example of Bayes’ theorem.

Theorem 1.3 (Bayes' Theorem) Let B1 , B2 , … , Bk be a partition of Ω, then

P(A ∣ B i )P(B i ) P(A ∣ B i )P(B i )

P(B i ∣ A) = =
k
P(A) ∑ P(A ∣ B j )P(B j )
j=1

for any event A with P(A) > 0 .

But we don’t really need to memorise these complicated formulae. As long as we know
the definition of conditional probability and the partition theorem, Bayes’ theorem follows
quite easily.

1.2.4 Independence
We should notice from our conditional probabilities in this example that someone’s
probability of having the disease does depend on where they come from. They are not
independent events. But what would it mean for two events to be independent?

Definition 1.7 Events A and B are independent if and only if

P(A ∩ B) = P(A)P(B) .

Independence is equivalent to P(A ∣ B) = P(A) and P(B ∣ A) = P(B). In other

words, learning that event B occurred does not change our beliefs about how likely
event A is (and vice versa). We can extend this idea to any number of events:

Definition 1.8 Consider a finite collection of events. These events are mutually
independent if and only if

P(B 1 ∩ B 2 ∩ … ∩ B j ) = P(B 1 )P(B 2 ) … P(B j )

whenever B1 , B2 , … , Bj are distinct events from this collection.

Please remember that being mutually independent is not equivalent to being mutually
exclusive.

1.3 Statistical models

Now that we understand probability, we can discuss how random quantities can be
modelled using probability distributions. This section will be a preview of the ideas that
we will discuss in more depth over the next ten weeks. As usual, we will try to understand
the concepts involved via an example.
Example 1.2 (hospital stay) Consider a type of major surgery that requires patients to
spend long periods of time recovering in the hospital before they can return home.
Duration of hospital stay (which we will denote by X , measured in days) can be thought
of as a number that varies randomly from patient to patient (across the population of
patients who undergo that particular surgery). We could therefore consider X to be a
random variable.2

If a variable can be thought of as random, we can use it to define events and can
consider their probabilities. For example, we can consider P(X ∈ C ) , the probability of
the event X ∈ C (that X will belong to some specific set C ).

We usually denote random variables in upper case (e.g. X ) and fixed numbers in lower
case (e.g. x ). In particular, X = x denotes the event that we observe X to take some
specific value x . We would say that the observed value x is a realisation of the random
variable X . We will use this convention throughout the module.3

In our hospital stay example, let’s think about how we might calculate the probability

P(X ∈ [20, 40])

of the event that a random patient’s hospital stay would be between 20 and 40 days
long.4

1.3.1 Probability distributions

Definition 1.9 The distribution of X is the collection of all probabilities of the

form P(X ∈ C ) for sets C ⊂ R .

To propose a distribution of hospital stay, it seems that we need to write down

P(X ∈ [a, b]) for all possible intervals [a, b] . But there are infinitely many intervals, so

how can we ever hope to fully specify a distribution for X ? This seems like an impossible
task.

Instead of writing down every probability, we try to write down a statistical model from
which we can calculate any P(X ∈ [a, b]) whenever we need to.
 Figure 1.3: The normal distribution curve. The area of the shaded
region over the interval [a, b] represents the probability of the event
X ∈ [a, b] . At each x ∈ R , the height of the curve is f (x) .

When we think of a statistical model, we often imagine a curve such as the normal
distribution’s familiar bell shape. For a normally distributed X , the shaded area under the
curve in the figure above would be

P(X ∈ [a, b])

for any interval [a, b] . Along with the rules for probabilities seen in the previous section,
the ability to calculate this area allows us to calculate any probability P(X ∈ C ) that we
want. Thus, if X really is normally distributed, this fairly simple curve gives us the entire
probability distribution of X .

To calculate these areas, we would need to know the density function f (x) that defines
the curve. For a normal distribution, this is

2 2
exp[−(x − μ) /2σ ] b

f (x) = so that P(X ∈ [a, b]) = ∫ f (x)dx,

−−−−
√2πσ 2 a

where μ and σ > 0 are two real numbers called parameters.

1.3.2 Parameter estimation
Before we can calculate probabilities, we must determine what the parameters should be
in our model. The parameters can have a big effect on the properties of our distribution
and therefore on the probabilities that we calculate, as seen in Figure 1.4.

Figure 1.4: The normal distribution curve for different values of μ and
σ
2
. It seems quite clear that μ determines the centre of the
distribution, while σ 2 determines the spread of the distribution.

How can we choose μ and σ ? First we should think about what they represent. It can be
shown that μ and σ are the average and the standard deviation (respectively) of X
across the population of interest.

So, for example, we might try to estimate μ via an experiment. Rather than just
considering the hospital stay X of just one patient, we would consider the hospital stays
X1 , X2 , … , Xn of n randomly chosen patients.
 Definition 1.10 Condiser n independent random variables X1 , X2 , … , Xn that
all have the same probability distribution. We call these variables an independent
and identically distributed (IID) random sample from that distribution.5

We would then observe the realised6 values of these n hospital stays:

X 1 = x1 , X 2 = x2 , …, X n = xn .

Finally, these observations could be used calculate the average hospital stay within this
sample,

x1 + x2 + … + xn
¯¯
¯
x = .
n

We would then think of this number x̄

¯¯
(which we now know) as our estimate of μ (which
remains unknown). Similarly, we could estimate σ by calculating the standard deviation
of our observations.

1.3.3 Dealing with uncertainty

We must remember that each xi is just one observed realisation of a normally
distributed random variable. Our estimate is based on a random sample and is therefore
also just one realisation of a random “statistic”.

Definition 1.11 Let X1 , X2 , … , Xn be a random sample and let s be a real-

valued function of n variables. The random variable S = s(X1 , X2 , … , Xn ) is a
statistic. In other words, any number that we calculate from our random sample is a
statistic.

Our estimate, ¯¯
x , is therefore just a realisation (observed value) of the (random) statistic
¯

¯¯¯¯ X1 + X2 + … + Xn
X =
n
which varies randomly from sample to sample (if we performed the experiment over and
over again, taking n new patients each time, we would observe a different realisation of
¯¯¯¯
X each time).

The randomness of the statistic is why different studies, with the same design and same
population, can still produce very different conclusions. Because of this, there will always
be uncertainty in our estimates and we should ask the following questions:

¯¯¯¯
a. Is X likely to be close to the true value of μ ?

b. Are there better ways to estimate μ , and what is the best estimator of μ ?

c. After observing our estimate ¯¯

x , what are the plausible values of μ ?
¯

These are very important questions because, as we saw in Figure 1.4, if we incorrectly
specify μ (or σ 2 ) in our model then that could affect our conclusions about the
distribution of X . But they are not easy questions. We will spend most of this module
gradually developing statistical tools to answer them.

To answer questions a and b, we will need to understand different methods of estimation

and the concept of a sampling distribution for an estimator (see Chapter 4). To answer
question c, we will need to understand hypothesis testing, confidence intervals and
Bayesian inference (see Chapters 5 and 6). And all of this will rely on having a solid
grasp of statistical modelling (see Chapter 3).

By the end of this module, it is hoped that you will feel comfortable answering all three of
these questions.

1. This notes are based on previous versions developed by Dr Benjamin Thorpe

and Professor Charles Taylor. ↩︎

2. Thinking of random variables as “numbers that vary randomly” will be enough for
us to understand the ideas presented in this module. But, if we wanted to be
more mathematically precise, would define a random variable to be a function
X : Ω → R defined on a sample space Ω . In our example, X assigns a real

number (days in hospital) to each patient in some population of patients (i.e. our
chosen sample space). ↩︎
3. The difference between a realisation x and a random variable X often confuses
students who are new to these concepts. It can help to think of X and x as
representing the same quantity before and after it is observed (respectively).

For example, let X be the height of the next person you will see today. You don’t
necessarily know who this will be (they will be a random person from some
population of people), so you don’t know a specific value for X . Because there is
some randomness in who this person will be, you can think about probabilities for
X by thinking of X as a random variable that varies from person to person.

However, once you meet this person, you observe their height x , which is some
fixed number, let’s say x = 5.5 feet. This observed value 5.5 is then something
that you know, so it is not random. Instead, it is what we call a realisation. It
would not make sense to think about probabilities for x , because it is just a fixed
number.

In summary, upper and lower case are used to distinguish between random and
non-random quantities. We can use X when we talk about what might happen,
and we use x when we talk about what actually did happen. For example, X is
used in planning experiments and x would then represent the actual result of
that experiment. ↩︎

4. Rather than using the notation X ∈ [20, 40], some textbooks might write this
event as as the set of all patients whose stay is 20 to 40 days:

{ω ∈ Ω : X(ω) ∈ [20, 40]},

where Ω is our sample space (patient population). This is would be more

consistent with Definition 1.1, which requires an event to be a subset of the
sample space (in our example, a subset of the patient population). However, it is
common and intuitively clear to just write this event more simply as
X ∈ [20, 40] , as we have. We will use this simpler notation throughout the

module. More generally, for any set C of real numbers, we will use X ∈ C as
our notation for the event of X being in C , rather than the slightly more accurate
but much more cumbersome notation {ω ∈ Ω : X(ω) ∈ C }). ↩︎

5. Although we tend to use the word “sample” to refer to a set of random variables,
it is also common to see “sample” used much more loosely. For example,
sometimes people say that our sample is the subset of the population for which
 we have information (in our example, that would be the set of n patients whose
hospital stays we can observe). Sometimes “sample” is used to refer to the data,
which are our collection of observations (in our example, this would be the
numerical values of the hospital stays of our n patients). This usually will not
cause much confusion. ↩︎

6. We think of a random sample X1 , X2 , … , Xn as a set of random variables,

denoted in upper case. Upon observing them we have x1 , x2 , … , xn , a set of
realisations, which we think of as fixed numbers, denoted in lower case. ↩︎