Chapter 3: Bayesian inference for

discrete random variables

Instructor: Truong Buu Chau

Email: truongbuuchau@[Link]

Faculty of Mathematics and Statistics

August 18, 2025

Truong Buu Chau C03031 - Chapter 3 August 18, 2025 1 / 31

Contents

1 Two equivalent ways of using Bayes’ theorem

2 Bayes’ theorem for Binomial with discrete prior
3 Important consequences of Bayes’ theorem
4 Bayes’ theorem for Poisson with discrete prior

Truong Buu Chau C03031 - Chapter 3 August 18, 2025 2 / 31

1

▶ Bayesian universe consists of the all possible ordered

pairs (xi , yj ) for i = 1, . . . , I and j = 1, . . . , J.

(x1 , y1 ) (x1 , y2 ) . . . (x1 , yj ) . . . (x1 , yJ )

.. .. .. .. .. ..
. . . . . .
(xi , y1 ) (xi , y2 ) . . . (xi , yj ) . . . (xi , yJ )
.. .. .. .. .. ..
. . . . . .
(xI , y1 ) (xI , y2 ) . . . (xI , yj ) . . . (xI , yJ )
▶ Each of the joint probabilities in the Bayesian universe:
f (xi , yj ) = g (xi ) × f (yj |xi )

Truong Buu Chau C03031 - Chapter 3 August 18, 2025 3 / 31

1

The marginal distribution of Y is found by summing the

columns.
prior y1 . . . yi . . . yJ
x1 g (x1 ) f (x1 , y1 ). . . f (x1 , yi ). . . f (x1 , yJ )
.. .. .. .. .
. . . ... . . . . ..
xi g (xi ) f (xi , y1 ). . . f (xi , yi ). . . f (xi , yJ )
.. .. .. .. .
. . . ... . . . . ..
xI g (xI ) f (xI , y1 ). . . f (xI , yi ). . . f (xI , yJ )

f (y1 ) . . . f (yi ) . . . f (yJ )

The joint and marginal distributions of X and Y.

Truong Buu Chau C03031 - Chapter 3 August 18, 2025 4 / 31

1

▶ When we observe Y = yj , the reduced Bayesian

universe is the set of ordered pairs in the j th column
.. .. ..
. . ... (x1 , yj ) . . . .
.. .. .. .. .. ..
. . . . . .
.. .. ..
. . ... (xi , yj ) . . . .
.. .. .. .. .. ..
. . . . . .
.. .. ..
. . ... (xi , yj ) . . . .
The reduced Bayesian universe given Y = yj

Truong Buu Chau C03031 - Chapter 3 August 18, 2025 5 / 31

1

▶ The posterior probability function of X given Y = yj is

given by
g (xi ) × f (yj |xi )
g (xi |yj ) = Pn
i =1 g (xi ) × f (yj |xi )

g (xi ) is prior
f (yj |xi ) is likelihood function
g (xi |yj ) is posterior
Pn
i =1 g (xi ) × f (yj |xi ) is constant

Truong Buu Chau C03031 - Chapter 3 August 18, 2025 6 / 31

1

The prior distribution of the discrete random variable is

given by the prior probability function g (xi ) for
i = 1, . . . , n. This is what we believe the probability of
each xi to be before we look at the data. It must come
from prior experience, not from the current data.

Truong Buu Chau C03031 - Chapter 3 August 18, 2025 7 / 31

1

Since we observed Y = yj , the likelihood of the discrete

parameter random variable is given by the f (yj |xi ) for
i = 1, . . . , n. This is the conditional probability function
of given X = xi evaluated at yj , the value that actually
occurred and where X is allowed to vary over its whole
range for xi , . . . , xn .

Truong Buu Chau C03031 - Chapter 3 August 18, 2025 8 / 31

1

The posterior probability distribution of the discrete

random variable is given by the posterior probability
function g (xi |yj ) evaluated at xi for i = 1, . . . , n, given
Y = yj .

Truong Buu Chau C03031 - Chapter 3 August 18, 2025 9 / 31

1

▶ Example 6.1 in page 112

There is an urn containing a total of 5 balls, some of which
may be red and the rest of which are green. We do not
know how many of the balls are red. Let the random
variable X be the number of red balls in the urn. Possible
values of X are xi = i for i = 0, . . . , 5. Since we do not
have any idea about the number of red balls, we will assume
all possible values are equally likely. Our prior distribution of
X is g (0) = g (1) = g (2) = g (3) = g (4) = g (5) = 1/6.

Truong Buu Chau C03031 - Chapter 3 August 18, 2025 10 / 31

1

▶ Example 6.1 in page 112

We will draw a ball at random from the urn. The random
variable Y is equal to 1 if draw is red and 0 otherwise.
Conditional observation distribution of Y |X is
P (Y = 1|X = xi ) = i /5
and
P (Y = 0|X = xi ) = (5 − i )/5
The joint probabilities are found by multiplying the prior
probabilities times the conditional observation probabilities.
The marginal probabilities of Y are found by summing the
joint probabilities down the columns.

Truong Buu Chau C03031 - Chapter 3 August 18, 2025 11 / 31

1

▶ Example 6.1 in page 112

The joint and marginal probability distributions

Truong Buu Chau C03031 - Chapter 3 August 18, 2025 12 / 31
1
▶ Example 6.1 in page 112
Suppose the selected ball is red, so the reduced universe is
in the column labelled yj = 1. The conditional observation
probabilities in that column are highlighted. They form the
likelihood function.

Finding the posterior probabilities of X |Y = 1

Truong Buu Chau C03031 - Chapter 3 August 18, 2025 13 / 31
1
▶ Example 6.1 in page 112
A simpler way of finding the posterior probability is to use
only the column in the reduced universe

Simplified table for finding the posterior probabilities of

X |Y = 1
Truong Buu Chau C03031 - Chapter 3 August 18, 2025 14 / 31
1. Two equivalent ways of using Bayes’ theorem

Analyzing the observations sequentially one at a

time.
Suppose that we randomly draw a second ball out of the
urn without replacing the first. Suppose the second draw
resulted in a green ball, so Y = 0.
We want to find the posterior probabilities of X given the
results of the two observations, red first, green second.

Truong Buu Chau C03031 - Chapter 3 August 18, 2025 15 / 31

1. Two equivalent ways of using Bayes’ theorem

We will use the posterior probabilities from the first draw as

the prior probabilities for the second draw.

The posterior probability distribution after second

observation

Truong Buu Chau C03031 - Chapter 3 August 18, 2025 16 / 31

1. Two equivalent ways of using Bayes’ theorem

Analyzing the observations all together in a single

step.
Alternatively, we could consider both draws together. Let
Y1 and Y2 be the outcome of the first and second draw,
respectively. The probabilities of the second draw depend
on the balls left after the first draw. By the multiplication
rule, the observation probability conditional on X is
f (y1 , y2 |x ) = f (y1 |x ) × f (y2 |y1 , x )

Truong Buu Chau C03031 - Chapter 3 August 18, 2025 17 / 31

1. Two equivalent ways of using Bayes’ theorem
The first ball was red, second was green, so the reduced
universe probabilities are in column yj1 , yj2 = 1, 0. The
likelihood function given by the conditional observation
probabilities in that column are highlighted.

The joint distribution of X , Y1 , Y2 and marginal distribution

of Y1 , Y2
Truong Buu Chau C03031 - Chapter 3 August 18, 2025 18 / 31
1. Two equivalent ways of using Bayes’ theorem

The posterior probability distribution given Y1 = 1 and

Y2 = 0

Truong Buu Chau C03031 - Chapter 3 August 18, 2025 19 / 31

1. Two equivalent ways of using Bayes’ theorem

The posterior probability distribution after both

observations

Truong Buu Chau C03031 - Chapter 3 August 18, 2025 20 / 31

1. Two equivalent ways of using Bayes’ theorem

▶ This shows that it makes no difference whether you

analyze the observations one at a time in
sequence using the posterior after the previous step as
the prior for the next step, or whether you analyze all
observations together in a single step starting with
your initial prior.

Truong Buu Chau C03031 - Chapter 3 August 18, 2025 21 / 31

2. Bayes’ theorem for Binomial with discrete prior

▶ The likelihood function:

 
n
f (y |π) = y π y (1 − π)n−y

for 0 ≤ π ≤ 1
▶ We decide on our prior probability distribution of the
parameter. They give our prior belief about each
possible value of the parameter π . If we have no idea
beforehand, we can choose the prior distribution that
has all values equally likely.

Truong Buu Chau C03031 - Chapter 3 August 18, 2025 22 / 31

2. Bayes’ theorem for Binomial with discrete prior

▶ The joint probability distribution of the parameter π

and the observation Y is found by multiplying the
conditional probability of Y |π by the prior probability of
π.
▶ The marginal distribution of Y is found by summing the
joint distribution down the columns.

Truong Buu Chau C03031 - Chapter 3 August 18, 2025 23 / 31

2. Bayes’ theorem for Binomial with discrete prior
▶ Example 6.2 in page 117
Let Y |π be binomial(n = 4; π ). Suppose we consider that
there are only three possible values for π = 0.4, 0.5, and
0.6. We will assume they are equally likely. Suppose Y = 3
was observed.

Y = 3 was observed, so the binomial probabilities of Y = 3

(the likelihood) are highlighted
Truong Buu Chau C03031 - Chapter 3 August 18, 2025 24 / 31
2. Bayes’ theorem for Binomial with discrete prior

▶ Example 6.2 in page 117

The prior distribution of π , joint probability distribution of
(π ; Y ), and marginal probability distribution of Y are
shown in Table

The joint and marginal probability distributions. Y = 3 was

observed, so those probabilities are highlighted.

Truong Buu Chau C03031 - Chapter 3 August 18, 2025 25 / 31

2. Bayes’ theorem for Binomial with discrete prior

▶ Example 6.2 in page 117

The prior distribution of π , joint probability distribution of
(π , Y), marginal probability distribution of Y, and posterior
probability distribution of π|Y = 3 are shown in Table

The joint, marginal, and posterior probability distribution of

π given Y = 3.

Truong Buu Chau C03031 - Chapter 3 August 18, 2025 26 / 31

2. Bayes’ theorem for Binomial with discrete prior

▶ Example 6.2 in page 117

The simplified table for finding posterior distribution given

Y = 3.

Truong Buu Chau C03031 - Chapter 3 August 18, 2025 27 / 31

2. Bayes’ theorem for Binomial with discrete prior

▶ Example 6.2 in page 117

The simplified table for finding posterior distribution given

Y = 3. Note we are using the proportional likelihood where
we have absorbed that part of the Binomial distribution
that does not depend on π into the constant.

Truong Buu Chau C03031 - Chapter 3 August 18, 2025 28 / 31

3. Important consequences of Bayes’ theorem

▶ Multiplying all the prior probabilities by a constant does

not change the result of Bayes’ theorem.
▶ Multiplying the likelihood by a constant does not
change the result of Bayes’ theorem.

Truong Buu Chau C03031 - Chapter 3 August 18, 2025 29 / 31

4. Bayes’ theorem for Poisson with discrete prior

▶ The likelihood function:

µy e −µ
f (y |µ) =
y!

∝ µy e −µ
for µ > 0
▶ Example 6.3 in page 121
Let Y |µ be Poisson(µ). Suppose that we believe there are
only four possible values for µ = 1, 1.5, 2, and 2.5. Suppose
we consider that the two middle values, 1.5 and 2, are twice
as likely as the two end values 1 and 2.5. Suppose y = 2
was observed.
Truong Buu Chau C03031 - Chapter 3 August 18, 2025 30 / 31
4. Bayes’ theorem for Poisson with discrete prior

The simplified table for finding posterior distribution given

Y =2

Truong Buu Chau C03031 - Chapter 3 August 18, 2025 31 / 31