LECTURE 2: Fundamental Concepts in Probability
Theory
Somabha Mukherjee
National University of Singapore
1 / 11
Outline
1 The Probability Space
2 Conditional Probability
3 Independence
2 / 11
Sample Space and Events
Suppose that a random experiment is performed, and its outcomes are
observed. The set of all possible outcomes is called the sample space,
denoted by Ω.
Roll a die once. The sample space
Ω = {1, 2, 3, 4, 5, 6}.
Toss a coin thrice. The sample space
Ω = {HHH, HHT , HTH, HTT , THH, THT , TTH, TTT }.
An event is a subset of the sample space Ω, i.e. a collection of some possible
outcomes.
For example, obtaining an odd number of heads in the first three tosses of a
coin is the event {HTT , THT , TTH, HHH}.
3 / 11
Probability Measure
A probability measure is a rule of assigning a valid chance (likelihood) to
each event.
Mathematically speaking, a probability measure is a function
P : Set of all events → [0, 1], satisfying the following two conditions:
1 P(Ω) = 1; T
2 If A1 , A2 , . . . is a countable collection of disjoint events (i.e. Ai Aj = ∅ for
i ̸= j), then
[ X
P An = P(An ).
n≥1 n≥1
If each outcome in a sample space is equally likely, one can imagine of
assigning the same probability to all the outcomes.
In this case, the probability of an event is given by:
|A| Total number of favorable outcomes
P(A) = = .
|Ω| Total number of possible outcomes
4 / 11
Some Properties of Probability:
P(Ac ) = 1 − P(A):
[
1 = P(Ω) = P A Ac = P(A) + P(Ac ).
If A ⊆ B then P(A) ≤ P(B):
[
P(B) = P A (B \ A) = P(A) + P(B \ A) ≥ P(A).
Boole’s Inequality: If A1 , A2 , . . . are events, then
[ X
P An ≤ P(An ) EXERCISE!;
n≥1 n≥1
S T
P (A B) = P(A) + P(B) − P (A B) EXERCISE!
5 / 11
Outline
1 The Probability Space
2 Conditional Probability
3 Independence
6 / 11
What is Conditional Probability?
For two events A and B with P(B) > 0, the conditional probability of A given
B is defined as: T
P (A B)
P(A|B) = .
P(B)
It follows directly from the above definition, that:
\
P A B = P(A|B)P(B) = P(B|A)P(A).
P(B|A)P(A)
Bayes Rule I: P(A|B) = P(B) EXERCISE !
Bayes Rule II: Let A1 , A2 , . .S. be a partition of the sample space Ω (i.e.
A1 , A2 , . . . are disjoint and j≥1 Aj = Ω.) Then for any event B with
P(B) > 0, we have:
P(B|Ai )P(Ai )
P(Ai |B) = P EXERCISE !
j≥1 P(B|Aj )P(Aj )
7 / 11
An Application of Bayes Formulae in Doping Tests
A certain doping test for football players is 99% accurate, meaning that it
shows a true positive result for players using drug with probability 0.99, and a
true negative result for players not using any drug with probability 0.99.
Let us assume that 0.1% of all football players use drug before a match.
Question: If a player selected at random tests positive, what is the probability
that he is actually a drug user?
Let A denote the event that the player is a drug user, and let B denote the
event that he tests positive. We are interested in P(A|B).
By Bayes formula, we have:
P(B|A)P(A)
P(A|B) =
P(B|A)P(A) + P(B|Ac )P(Ac )
0.99 × 0.001
= = 0.09.
(0.99 × 0.001) + (0.01 × 0.999)
Surprisingly, even if a player tests positive, there is only 9% chance that he is
actually drug user!
8 / 11
Outline
1 The Probability Space
2 Conditional Probability
3 Independence
9 / 11
What do we Mean by Independent Events?
Events A1 , A2 , . . . , An are said to be independent, if for every k ≤ n and
every 1 ≤ i1 < i2 < . . . < ik ≤ n,
\ \
P Ai1 . . . Aik = P(Ai1 ) . . . P(Aik ).
Independence is different from pairwise independence, which means
\
P Ai Aj = P(Ai )P(Aj ) for all i ̸= j .
Clearly, two events A and B are independent if and only if
\
P A B = P(A)P(B).
Note that for two events A and B with P(B) > 0, A and B are independent
if and only if
P(A|B) = P(A).
if and only if “Conditional Probability of A given B does not depend on B.”
10 / 11
What will we do Next Day?
Introduce “random variables”, which are the central characters in probability
theory.
Introduce a notion of the average of a random variable, which is known as
“expectation”.
Talk about “conditional distributions” of random variables and introduce the
notion of “conditional expectations”.
All these concepts will be immensely useful in our future study of Markov
chains.
11 / 11