Bayesian Machine Learning1

Machine Learning

1
Thanks to MIT Open Courseware, 18.05
Machine Learning Bayesian Machine Learning
Bayes Rule Revisited

P(X |Y )P(Y )
P(Y |X ) =
P(X )
Or
P(Data|Hypothesis)P(Hypothesis)
P(Hypothesis|Data) =
P(Data)

Bayes Rule can be used as a rule of inference for solving ML

problems. Given data and prior information, derive:
1. Parameter value (hypotheses are paratemeter values)
2. Predictions (hypotheses are values of a random variable)
3. Preference over models (hypotheses are models)
In all cases, we have to understand Bayes rule as an update of
priors to posteriors (which become priors for the next update etc.)

Machine Learning Bayesian Machine Learning

Discrete Hypotheses, Discrete Data I

I Assume we have a model with a parameter Θ that takes

values from a discrete set {θ1 , θ2 , . . . , θn }. We want to work
out which value of Θ = θ is best for the data d we have have
observed
I d is one of many data sets d1 , d2 , . . . , dk that could have been
drawn. So, d is the value of some random variable D
I θ is one of many values θ1 , θ2 , . . . , θn that could be assigned to
the random variable Θ
I For each combination di , θi , the probability p(di , θi ) is a value
from the joint distribution p(D, Θ)
I Here, we are considering the case where D and Θ are discrete
r.v.’s
I Hypotheses are statements like Θ = θ. So P(Hypothesis) =
P(Θ = θi ) where P is the p.m.f. of Θ

Machine Learning Bayesian Machine Learning

Discrete Hypotheses, Discrete Data II

I Bayes:

P(D = d|Θ = θ)P(Θ = θ)

P(Θ = θ|D = d) =
P(D = d)
I Example
I Suppose we have a coin, whose probability of heads could be
one of 0.5, 0.6 or 0.9, with prior probabilities 0.4, 0.4 and 0.2
respectively.
I The coin is flipped once, and it lands heads. That is the data
is D = h
I Then:

H Prior Likelihood Numerator Denom Posterior

Θ = 0.5 0.4 0.5 0.2 0.62 0.3226
Θ = 0.6 0.4 0.6 0.24 0.62 0.3871
Θ = 0.9 0.2 0.9 0.18 0.62 0.2903

Machine Learning Bayesian Machine Learning

Discrete Hypotheses, Discrete Data III

I NOTE: The maximum likelihood hypothesis is different to the

hypothesis with the maximum posterior probability

Machine Learning Bayesian Machine Learning

Continuous Hypotheses, Discrete Data I

I Assume we have a model structure with some parameter Θ,

and we want to work out which value of Θ = θ is best for the
data d we have observed
I We now consider the case where D is a discrete r.v. and Θ is
a continuous r.v
I Hypotheses are statements like Θ = θ
I Correctly: “Θ lies in an interval dθ around θ”. So
P(Hypothesis) = f (Θ = θ)dθ where f is the p.d.f
I Bayes:

P(D = d|Θ = θ)f (Θ = θ)dθ

f (Θ = θ|D = d)dθ =
P(D = d)

Machine Learning Bayesian Machine Learning

Continuous Hypotheses, Discrete Data II
I This is usually simplified to:

P(d|θ)f (θ)
f (θ|d) =
P(d)

(BUT: remember how we got here, with the dθ’s)

I Example
I Suppose we have a biased coin, whose probability of heads is
some unknown value θ. The prior p.d.f is f (θ) = 2θ
I The coin is flipped once, and it lands heads
I Then:
Data. This is D = heads (or D = h for short)
Hypothesis. This is Θ = θ (θ is not known)
Likelihood. This is P(D = h|Θ = θ) = θ
Prior. f (θ) dθ = 2θ dθ
Numerator. 2θ2 dθ R1
Denominator. P(D = h) = 0 2θ2 dθ = 2/3

Machine Learning Bayesian Machine Learning

Continuous Hypotheses, Discrete Data III

Posterior. (2θ2 dθ)/(2/3) = 3θ2 dθ

Posterior p.d.f 3θ2
I A very flexible function for specifying the prior p.d.f. of a
continuous r.v. uses the Beta distribution
I The Beta distribution is a 2-parameter distribution, specifying
the p.d.f.

Γ(α + β) α−1
Beta(α, β) = f (θ; α, β) = θ (1 − θ)β−1
Γ(α)Γ(β)

We omit the proof here that f is a density function (that is,

R1
0 f (θ; α, β) dθ = 1. Varying α and β gives us many
different kinds of p.d.f’s:

Machine Learning Bayesian Machine Learning

Continuous Hypotheses, Discrete Data IV

I In general, any p.d.f. of the form:

f (r ) = c r α−1 (1 − r )β−1
Γ(α+β)
for 0 ≤ r ≤ 1 is necessarily Beta(α, β) with c = Γ(α)Γ(β)

Machine Learning Bayesian Machine Learning

Continuous Hypotheses, Discrete Data V

I Let us take the case of data drawn from a Binomial

distribution. That is, the number of successes yN (heads) is
the value of a r.v. YN ∼ Binom(N, θ)
I Example
I Suppose we observe 8 heads in 12 tosses of a coin with
P(heads) = θ. Assume the prior p.d.f is f (θ) = 1. What is the
posterior p.d.f for θ?
Data. Y12 = y12 = 8 (8 h’s in 12 trials)
Hypothesis. Θ =
θ8
Likelihood. 12 8 θ (1 − θ)
4

Prior. 1dθ

Numerator. 12 8 4
8
θR (1 − θ) dθ
12 1 8
Denominator. 8 0 θ (1 − θ)4 dθ
Posterior. cθ8 (1 − θ)4 dθ = Beta(9, 5)dθ
I So, the posterior p.d.f. is Beta(9, 5)

Machine Learning Bayesian Machine Learning

Continuous Hypotheses, Discrete Data VI
I BUT: the prior p.d.f. (f (θ) = 1) is just Beta(1, 1). So, after
observing 8 heads and 4 tails, a Beta(1, 1) prior p.d.f has been
updated to a Beta(1 + 8, 1 + 4) posterior p.d.f
I GENERALISE: if we started with a Beta(a, b) prior p.d.f and
we observe s heads and t tails, the posterior p.d.f will be
Beta(a + s, b + t)
I This property (prior and posterior both having a Beta
distribution) happens because the similarity of the functional
form of the likelihood (Binomial) and the prior p.d.f (Beta):
Likelihood. c1 θs (1 − θ)t
Prior. c2 θa−1 (1 − θ)b−1 dθ
Posterior c3 θs+a−1 (1 − θ)b+t−1 dθ
I The Beta distribution is said to be a conjugate prior p.d.f. for
a Binomial likelihood function.
I For every such conjugage prior-likelihood pair, the posterior
will have the same functional form as the prior

Machine Learning Bayesian Machine Learning

Continuous Hypotheses, Discrete Data VII

I There are other pairs with this property (Dirichlet-Multinomial;

Gaussian-Gaussian etc.)
I Conjugate prior-likelihood pairs make Bayesian posterior
probability calculation easy (easier)

Machine Learning Bayesian Machine Learning

Discrete Hypotheses, Continuous Data I

I Assume we have a model with a parameter Θ that takes

values from a discrete set {θ1 , θ2 , . . . , θn }. We want to work
out which value of Θ = θ is best for the data x (previously we
used d) we have have observed
I x is the value of some r.v. X
I θ is one of many values θ1 , θ2 , . . . , θn that could be
I Here we want to consider the case where X is a continuous
r.v. and Θ is a discrete r.v.
I The only thing that changes is that likelihoods are calculated
using p.d.f’s. That is:
Hypothesis. Θ = θ
Data. X in an interval dx around x
Prior. P(Θ = θ)
Likelihood. f (x|θ)dx (correctly, fX |Θ (x|θ)dx)
Numerator. f (x|θ)dxP(Θ = θ)

Machine Learning Bayesian Machine Learning

Discrete Hypotheses, Continuous Data II
R1
Denominator. f (x)dx = ( 0 f (θ)dθ) dx
Posterior. f (θ|x)dθ
I Bayes:

f (x|Θ = θ)dxP(Θ = θ)
P(Θ = θ|x) =
f (x)dx

(sometimes, the dx’s are not shown)

I Example
I Suppose data are drawn from one of 2 clusters whose centres
are at µ1 = 2 and µ2 = 5, both with prior probability 0.5.
Data from a cluster µi are drawn according N(µi , 1)
I An instance x = 3 is observed. What is the posterior
probability of the clusters given this data?
I Then:

Machine Learning Bayesian Machine Learning

Discrete Hypotheses, Continuous Data III

H Prior Likelihood Numerator Denom Posterior

2
Θ=2 0.5 0.4e −(3−2) /2 dx 0.12 dx 0.19 dx 0.63
2
Θ=5 0.5 0.4e −(3−5) /2 dx 0.07 dx 0.19 dx 0.37

Machine Learning Bayesian Machine Learning

Continuous Hypotheses, Continuous Data I

I Assume we have a model structure with some parameter Θ,

and we want to work out which value of Θ = θ is best for the
data x (previously we used d) we have observed.
I As before, θ and x will be values of random variables Θ and X
(used to be D)
I Both Θ and X are now continuous r.v.’s
I The only thing that changes is that both likelihood and priors
are calculated using p.d.f’s. That is:
Hypothesis. Θ = θ
Data. X in an interval dx around x
Prior. f (θ)dθ
Likelihood. f (x|θ)dx (correctly, fX |Θ (x|θ)dx)
Numerator. f (x|θ)dxf (θ)dθ
R1
Denominator. f (x)dx = ( 0 f (θ)dθ) dx
Posterior. f (θ|x)dθ

Machine Learning Bayesian Machine Learning

Continuous Hypotheses, Continuous Data II

I Example
I Suppose X ∼ N(θ, 1) and Θ ∼ N(2, 1)
I Then:
2
Prior. c1 e −(θ−2) /2 dθ
2
Likelihood. c2 e −(5−θ) /2 dx
2
Numerator. c3 e −(2θ −14θ+29)/2 dxdθ =
−((θ 2 −7/2)2 +9/4) 2 2
cR3 e dxdθ = c4 e −(θ −7/2) dxdθ
−(θ 2 −7/2)2
Denominator. ( c4 e dθ)dx = c5 dx
−(θ 2 −7/2)2
Posterior. c6 e dθ
I The posterior
√ p.d.f. is N(7/2, σ) where 2σ 2 = 1 and
c6 = 1/(σ 2π)
I So a Gaussian is conjugate prior for Gaussian likelihood.

Machine Learning Bayesian Machine Learning

Summary

I Bayes rule forms the basis for several inference tasks in

Bayesian Machine Learning
I 3 such inference tasks are: parameter estimation, structure
estimation, and prediction
I Each of these can be formulated as the problem of inferring
the posterior probability of a hypothesis, given data
I Hypotheses and data are both seen as values taken by random
variables. The r.v. could be discrete or continuous, but the
basic approach is the same (differences arise only in the use of
p.m.f’s or p.d.f’s)

Machine Learning Bayesian Machine Learning