MATH2411: Applied Statistics | Dr.
YU, Chi Wai
Chapter 2: PROBABILITY
1 WHAT IS PROBABILITY?
Probability is the study of randomness. A probability of an uncertain outcome/event is the
chance (or likelihood) that it will occur.
The probability must be between 0 and 1, inclusively.
Our goal:
We would like to develop terms and results that form the basis of probability theory, which
will allow us to quantify the likelihood or chance of outcomes of interest.
2 BASIC PROBABILITY TERMS
Random experiment
o Is a process which generates observations.
o The outcome of the experiment cannot be known BEFORE hand and is only
known AFTER the experiment is performed.
Typical examples: Toss a coin, roll a die, etc.
Sample Space
o Is the set of all possible outcomes for a given random experiment, and is often
denoted by S.
For instance, S = {H, T}, S = {1, 2, 3, 4, 5, 6} = {integers from 1 to 6}, etc.
Event
o Is a subset/portion of outcomes from the sample space that we are interested
in, often denoted by capital letters.
For instance, A = {H} = {Head}, B = {Roll a 1, 2 or 3}, etc.
~1~
MATH2411: Applied Statistics | Dr. YU, Chi Wai
3 OPERATION ON EVENTS (OR BASIC SET THEORY)
3.1 UNION
The union of two events A and B, denoted by 𝐴 ∪ 𝐵, is the event that either A or B AND
both occur.
o If the union of some events which are in the same sample space S is exactly the
same as S, then those events are said to be exhaustive.
o 𝐴 ∪ 𝐵 = 𝐵 if 𝐴 is in 𝐵.
3.2 INTERSECTION
The intersection of two events A and B, denoted by 𝐴 ∩ 𝐵, is the event that both A and
B occur.
o If two events have no common element, then their intersection is
an empty set 𝜙 (a set without element) and the events are said to be disjoint.
o If two or more events are considered and they are pairwise disjoint, then all of
these events are said to be mutually exclusive.
o 𝐴 ∩ 𝐵 = 𝐴 if 𝐴 is in 𝐵.
3.3 COMPLEMENT
For any event A in the sample space S, the complement of A, denoted by 𝐴𝐶 , is the
event that contains all points in S but not in A. That is, it is the event that A does not
occur.
o 𝑆 𝐶 = 𝜙 and 𝜙 𝐶 = 𝑆.
3.4 SYMMETRIC DIFFERENCE
The symmetric difference of two events A and B, denoted by, 𝐴∆𝐵, is the event that
either A or B (BUT not both) occurs. More simply, it is the union without the
intersection.
o 𝐴∆𝜙 = 𝐴 and 𝐴∆𝐴 = 𝜙.
~2~
MATH2411: Applied Statistics | Dr. YU, Chi Wai
4 VENN DIAGRAM
Venn diagrams are the graphs which can illustrate the relationships among events.
Note that in Venn diagram, a rectangle is often used to represent the sample space S, and a
circle (or other patterns) inside the rectangle represents a particular event in the sample space.
Usually, the relationship between two sets A and B can be drawn as
The two circles overlap, indicating that some, but not all, of the elements of A is also elements
of B.
In particular, if all elements of B are elements of A, but some of elements of A are not elements
of B (That is, B is strictly in A), then we have the following Venn diagram.
~3~
MATH2411: Applied Statistics | Dr. YU, Chi Wai
UNION INTERSECTION
EXHAUSTIVE MUTUALLY EXCLUSIVE
EXHAUSTIVE AND MUTUALLY EXCLUSIVE --- PARTITION
COMPLEMENT SYMMETRIC DIFFERENCE
~4~
MATH2411: Applied Statistics | Dr. YU, Chi Wai
5 PROPERTIES OF EVENTS
𝐴 ∩ 𝐵 = 𝐵 ∩ 𝐴, and 𝐴 ∪ 𝐵 = 𝐵 ∪ 𝐴
(Associate laws)
𝐴 ∩ (𝐵 ∩ 𝐷) = (𝐴 ∩ 𝐵) ∩ 𝐷, and 𝐴 ∪ (𝐵 ∪ 𝐷) = (𝐴 ∪ 𝐵) ∪ 𝐷
(Distributive laws)
𝐴 ∩ (𝐵 ∪ 𝐷) = (𝐴 ∩ 𝐵) ∪ (𝐴 ∩ 𝐷), and 𝐴 ∪ (𝐵 ∩ 𝐷) = (𝐴 ∪ 𝐵) ∩ (𝐴 ∪ 𝐷)
(DeMorgan’s laws)
(𝐴 ∩ 𝐵)𝐶 = 𝐴𝐶 ∪ 𝐵 𝐶 , and (𝐴 ∪ 𝐵)𝐶 = 𝐴𝐶 ∩ 𝐵 𝐶
6 KOLMOGOROV’S AXIOM SYSTEM
In 1933, a Russian mathematician named
Andrei Nikolayevich Kolmogorov
published the axiomatic structure of
probability theory. The following is the
definition of a probability under
Kolmogorov’s system.
Definition: A probability on the sample space S is an assignment of a value, say 𝑃(𝐴),
to an event A in S via a function 𝑃(⋅) such that:
1. 𝑃(𝑆) = 1.
2. 𝑃(𝐸) ≥ 0, for any event 𝐸 in 𝑆.
3. For any sequence of mutually exclusive events 𝐸1 , 𝐸2 , …,
∞ ∞
𝑃 (⋃ 𝐸𝑖 ) = ∑ 𝑃(𝐸𝑖 ) .
𝑖=1 𝑖=1
~5~
MATH2411: Applied Statistics | Dr. YU, Chi Wai
QUESTION
Consider a sample space S of all outcomes of a fair die and events A = {1, 3} and B = {2, 4}. Use
Kolmogorov’s system to find the probabilities of A, B and their union.
7 PROPERTIES OF PROBABILITY
𝑃(𝐴) = 1 − 𝑃(𝐴𝐶 )
𝑃(𝜙) = 0
𝑃(𝐴) ≤ 𝑃(𝐵), if 𝐴 is in 𝐵.
𝑃(𝐴) = 𝑃(𝐵), if 𝐴 = 𝐵 (i.e. 𝐴 is in 𝐵 and 𝐵 is in 𝐴)
𝑃(𝐴 ∪ 𝐵) = 𝑃(𝐴) + 𝑃(𝐵) − 𝑃(𝐴 ∩ 𝐵)
~6~
MATH2411: Applied Statistics | Dr. YU, Chi Wai
8 PROBABILITY OF AN EVENT IN SAMPLE SPACE WITH FINITE EQUALLY LIKELY
OUTCOMES
One simple scenario in probability is to consider the sample space with finite equally likely
outcomes.
Outcomes are said to be equally likely if they have equal chances to occur.
(Theorem)
Consider a sample space 𝑆 = {𝑤1 , 𝑤2 , … , 𝑤𝑁 } with N equally likely outcomes, where N is a finite
positive integer. Let 𝐸 be any event in 𝑆. Then
𝑛𝑢𝑚𝑏𝑒𝑟 𝑜𝑓 𝑜𝑢𝑡𝑐𝑜𝑚𝑒𝑠 𝑖𝑛 𝐸
𝑃(𝐸) = .
𝑁
EXAMPLE
Flip a fair coin three times. We have the sample space
S = { HHH, HHT, HTH, THH, TTH, THT, HTT, TTT }.
Denote by E the event that “heads” occurs at the first time. Thus,
𝐸 = {HHH, HHT, HTH, HTT},
and its probability can be found by
4 1
𝑃(𝐸) = = .
8 2
9 CONDITIONAL PROBABILITY
Conditional probability is the probability of one event given that another event (conditional
event) has already occurred. The conditional probability of A given that an event B has already
occurred or known is denoted by 𝑃(𝐴|𝐵).
A typical example: Toss a fair die once. P(getting odd number) = 1/2, and
P(getting odd number | number is known to be greater than 1) = 2/5.
Note that the sample space is reduced to {2, 3, 4, 5, 6} if the conditional event occurs.
~7~
MATH2411: Applied Statistics | Dr. YU, Chi Wai
This example tells us that sometimes the knowledge/information that an event B has occurred
influences the probability that A will occur. We can “update” the probability of an event A
happening after obtaining some “additional” information from B.
Definition: Let A and B be two events such that 𝑃(𝐵) > 0. Then the conditional
probability that A will occur given that B has occurred is defined to be
𝑃(𝐴 ∩ 𝐵)
𝑃(𝐴|𝐵) = .
𝑃(𝐵)
Referring to the typical example, we re-do the question of finding the conditional probability by
the above definition.
Let A be the event “getting odd number” and B be the event “getting number greater than 1”.
Thus,
𝑃(𝐴 ∩ 𝐵) 2/6 2
𝑃(𝐴|𝐵) = = = .
𝑃(𝐵) 5/6 5
Note that the formula of the conditional probability implies that 𝑃(𝐴 ∩ 𝐵) = 𝑃(𝐴|𝐵)𝑃(𝐵),
known as the multiplication rule, which can be easily generalized to a case with 3 or more
events. For instance,
Multiplication rule for three events:
𝑃(𝐴 ∩ 𝐵 ∩ 𝐶) = 𝑃(𝐶|(𝐴 ∩ 𝐵))𝑃(𝐵|𝐴)𝑃(𝐴) .
Note that the above result requires one condition. What is it? Discuss it in class.
Example: A box of fuses contains 20 fuses, of which 5 are defective. If three of the fuses are
selected randomly and removed from the box in succession without replacement, the
probability that all three fuses are defective can be found in the following way:
Let A be the event that the first selected fuse is defective. Similarly, let B and C be the
respective events that the second and third selected fuses are defective.
Obviously, 𝑃(𝐴) = 5/20, 𝑃(𝐵|𝐴) = 4/19 and 𝑃(𝐶|(𝐴 ∩ 𝐵)) = 3/18 . Thus, the required
probability is
3 4 5 1
𝑃(𝐴 ∩ 𝐵 ∩ 𝐶) = 𝑃(𝐶|(𝐴 ∩ 𝐵))𝑃(𝐵|𝐴)𝑃(𝐴) = × × = .
18 19 20 114
~8~
MATH2411: Applied Statistics | Dr. YU, Chi Wai
10 PROPERTIES OF CONDITIONAL PROBABILITY
𝑃(𝐴|𝐵) = 1 − 𝑃(𝐴𝐶 |𝐵)
𝑃((𝐴 ∪ 𝐵)|𝐷) = 𝑃(𝐴|𝐷) + 𝑃(𝐵|𝐷) − 𝑃((𝐴 ∩ 𝐵)|𝐷)
𝑃((𝐴 ∩ 𝐵)|𝐷) = 𝑃(𝐴|(𝐵 ∩ 𝐷))𝑃(𝐵|𝐷)
11 INDEPENDENCE
Note that it is possible that the knowledge/information that an event B has occurred DOES NOT
INFLUENCE the probability that A will occur. That is, 𝑃(𝐴|𝐵) = 𝑃(𝐴). In this case, we would say
that the events A and B are independent. By the above multiplication rule, we can notice that
events A and B are independent if and only if
𝑷(𝑨 ∩ 𝑩) = 𝑷(𝑨)𝑷(𝑩) .
If the events are NOT independent, then they are said to be dependent.
QUESTION
A fair coin is tossed three times. Consider the following events:
A: the event that a head occurs on each of the first two tosses,
B: the event that a tail occurs on the third toss, and
C: the event that exactly two tails occur in the three tosses.
Are A and B independent? Are B and C independent?
~9~
MATH2411: Applied Statistics | Dr. YU, Chi Wai
The following definition is for the independence of three events:
Definition: Events 𝐴1 , 𝐴2 and 𝐴3 are independent if and only if the following
conditions are ALL true:
12 LAW OF TOTAL PROBABILITY
By the Venn diagram, we can notice that (𝐴 ∩ 𝐵) and (𝐴 ∩ 𝐵 𝐶 ) are disjoint. Therefore, we can
write the event A as
𝐴 = (𝐴 ∩ 𝐵) ∪ (𝐴 ∩ 𝐵 𝐶 )
and then have the result often known as “LAW OF TOTAL PROBABILITY”
𝑃(𝐴) = 𝑃(𝐴 ∩ 𝐵) + 𝑃(𝐴 ∩ 𝐵 𝐶 )
For 0 < 𝑃(𝐵) < 1,
𝑃(𝐴) = 𝑃(𝐴|𝐵)𝑃(𝐵) + 𝑃(𝐴|𝐵 𝐶 )𝑃(𝐵 𝐶 )
More generally,
(LAW OF TOTAL PROBABILITY --- GENERAL VERSION)
If 𝐵1 , 𝐵2 , , … , 𝐵𝑘 are mutually exclusive and exhaustive events with probabilities between 0 and
1 exclusively, then for any event A, we have
𝑘
𝑃(𝐴) = ∑ 𝑃(𝐴|𝐵𝑖 )𝑃(𝐵𝑖 ) .
𝑖=1
~ 10 ~
MATH2411: Applied Statistics | Dr. YU, Chi Wai
QUESTION
In a certain assembly plant, three machines, I, II and III, make 30%, 45%, and 25%, respectively,
of the products. It is known from past experience that 2%, 3%, and 2% of the products made by
each machine, respectively, are defective. Now, suppose that a finished product is randomly
selected. What is the probability that it is defective?
~ 11 ~
MATH2411: Applied Statistics | Dr. YU, Chi Wai
13 BAYES’ THEOREM
Bayes’ theorem is a famous theorem in probability related the
conditional probability 𝑃(𝐴|𝐵) to its “inverse” counterpart
𝑃(𝐵|𝐴). One typical application of using this theorem in practice
is for a diagnostic test --- determine the probability of a person
infected by a particular virus, given that the test on this person
shows a positive result.
Thomas Bayes
(1701 - 1761)
(Bayes’ theorem --- 1st form)
If 𝑃(𝐴) > 0 and 𝑃(𝐵) > 0, then we have
𝑃(𝐵)
𝑃(𝐵|𝐴) = 𝑃(𝐴|𝐵) .
𝑃(𝐴)
𝑃(𝐴|𝐵) and 𝑃(𝐵|𝐴) if and only if 𝑃(𝐴) = 𝑃(𝐵). In general, 𝑃(𝐴|𝐵) and 𝑃(𝐵|𝐴) are
different. Thus, please don’t jump to conclude that smokers will have large chance to
have lung cancer if we observed that a large proportion of lung cancer patients are
smokers.
The unconditional probability 𝑃(𝐵) is also called the prior probability of 𝐵 in the sense
that it does not take any information about 𝐴 into account.
The conditional probability of 𝐵 given 𝐴, 𝑃(𝐵|𝐴), is also called the posterior probability
in contrast to the prior probability 𝑃(𝐵). It is “posterior” in the sense that some
additional information (occurrence of 𝐴) has been taken into account.
Together with the law of total probability, we have the following 2nd form:
(Bayes’ theorem --- 2st form)
If 𝐵1 , 𝐵2 , , … , 𝐵𝑘 are mutually exclusive and exhaustive events with probabilities between 0 and
1 exclusively, then for any event 𝐵𝑗 , we have
𝑃(𝐵𝑗 )
𝑃(𝐵𝑗 |𝐴) = 𝑃(𝐴|𝐵𝑗 ) .
∑𝑘𝑖=1 𝑃(𝐴|𝐵𝑖 )𝑃(𝐵𝑖 )
~ 12 ~
MATH2411: Applied Statistics | Dr. YU, Chi Wai
QUESTION
Given 𝑃(𝐵) = 0.03, 𝑃(𝐴𝐶 |𝐵) = 0.01, and 𝑃(𝐴|𝐵 𝐶 ) = 0.02,
where 𝐴 is the event that a test shows a positive result as
carrier of a particular virus and 𝐵 is the event that a person is
infected by this virus. Then, what is the probability of a person
infected by the virus, given that the test on this person shows a
positive result?
Huh? Positive?!?!?!?!
~ 13 ~