Introduction to Bayesian Inference

Comprehensive Lecture Notes with Complete Proofs

These notes develop Bayesian inference from rst principles, with every central result stated
precisely and proved. Beyond the core (Bayes' theorem, prior/likelihood/posterior, conju-
gacy), we treat the logical foundations (coherence and de Finetti's theorem), suciency,
worked objective priors, hierarchical and empirical Bayes, decision theory, model compari-
son via Bayes factors, the computational toolkit (Monte Carlo, MCMC, Gibbs, the Laplace
approximation), large-sample theory, and a careful comparison with frequentist methods 
closing with worked examples and solved exercises.
Contents
1 Motivation and Philosophy 2
1.1 What inference is . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2
1.2 A little history . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
1.3 Two readings of probability . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
1.4 Why beliefs must obey the probability axioms . . . . . . . . . . . . . . . . . . . . 3

2 Probability Foundations 3
3 Bayes' Theorem 4
3.1 The odds form: evidence as weight . . . . . . . . . . . . . . . . . . . . . . . . . . 5

4 The Four Ingredients 5

5 Exchangeability and de Finetti's Theorem 5
6 The Likelihood Function and Suciency 6
6.1 Suciency: when a summary loses nothing . . . . . . . . . . . . . . . . . . . . . 6

7 Prior Distributions 7
7.1 A taxonomy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7
7.2 Improper priors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7
7.3 The Jereys prior, with worked derivations . . . . . . . . . . . . . . . . . . . . . . 7
7.4 The prior predictive (and how to elicit a prior) . . . . . . . . . . . . . . . . . . . 8
7.5 Mixture priors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8

8 The Posterior and Sequential Updating 8

9 Conjugate Models, with Full Proofs 8
9.1 BetaBinomial: a probability . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8
9.2 GammaPoisson: a rate . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9
9.3 NormalNormal: a mean (known variance) . . . . . . . . . . . . . . . . . . . . . . 9
9.4 Inverse-GammaNormal: a variance (known mean) . . . . . . . . . . . . . . . . . 9
9.5 DirichletMultinomial: category probabilities . . . . . . . . . . . . . . . . . . . . 9
9.6 Normal with unknown mean and variance . . . . . . . . . . . . . . . . . . . . . . 10
9.7 Reference table . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10

1
10 The Exponential Family and General Conjugacy 10
11 A Capstone: Bayesian Linear Regression 10
12 Hierarchical Models and Empirical Bayes 12
12.1 Empirical Bayes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12

13 Point Estimates and Bayesian Decision Theory 12

13.1 Risk and admissibility . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13

14 Credible Intervals 13
15 The Posterior Predictive Distribution 13
16 Bayesian Model Comparison 14
17 Computational Methods 14
17.1 Monte Carlo integration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14
17.2 Importance sampling . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15
17.3 Markov chain Monte Carlo (MCMC) . . . . . . . . . . . . . . . . . . . . . . . . . 15
17.4 Gibbs sampling . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15
17.5 The Laplace approximation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16

18 Bayesian versus Frequentist Inference 16

19 Large-Sample Theory 17
20 Fully Worked Examples 17
21 Exercises (with solutions) 18
22 Common Misconceptions 19
23 Summary 19
A Appendix: Distribution Quick Reference 20

1 Motivation and Philosophy

1.1 What inference is

Statistical inference is the task of learning about an unknown quantity θ  a parameter, a hy-
pothesis, or a future observation  from observed data x. Two traditions answer this dierently.

ˆ Frequentist. θ is a xed but unknown constant. Probability refers only to the long-run
frequency of outcomes under hypothetical repetitions of the experiment. One never writes
P (θ ∈ A), because θ is not random.

ˆ Bayesian. θ is treated as a random variable. Our uncertainty about θ is itself a probability

distribution that we update as data arrive.

2
 The central Bayesian idea is learning as belief updating:
observe x
prior belief −−−−−−→ posterior belief .
| {z } | {z }
before data after data

The engine that performs the update is Bayes' theorem; almost everything that follows elab-
orates that one equation.

1.2 A little history

The rule is named for the Reverend Thomas Bayes (17011761), whose posthumous 1763 essay
solved a special case (inferring a binomial probability from observed successes). Pierre-Simon
Laplace (17491827) independently rediscovered and vastly generalised the idea, applying it
to astronomy, demography and jurisprudence; for much of the nineteenth century the method
was simply called inverse probability. The twentieth-century frequentist movement (Fisher,
Neyman, Pearson) pushed it aside over concerns about subjective priors, but the Bayesian
programme was rebuilt on rigorous decision-theoretic and axiomatic foundations by Ramsey,
de Finetti, Jereys, Savage and others, and  powered by modern computation  is now a
dominant framework across statistics and machine learning.

1.3 Two readings of probability

The split above mirrors a deeper disagreement about what probability is.

ˆ The frequency interpretation: a probability is the limiting relative frequency of an event in

an innite sequence of identical trials. It struggles to assign meaning to one-o statements
(the probability this drug works).

ˆ The epistemic (Bayesian) interpretation: a probability is a degree of rational belief, cali-

brated against bets one would accept. It applies to unique events and to parameters with no
repetition.

1.4 Why beliefs must obey the probability axioms

A reasonable worry is that degree of belief  is too woolly to do mathematics with. Two classical
results answer it.

The Dutch-book argument (coherence). Suppose you quote betting odds on events.
If your quoted fair prices for events violate the probability axioms (non-negativity, nor-
malisation, additivity), then a clever opponent can construct a nite set of bets  a Dutch
book  each of which you regard as fair, but whose net eect is that you lose money with
certainty, regardless of outcome. Conversely, if your prices obey the axioms, no such sure-
loss book exists. Hence: to be coherent (immune to sure loss), your degrees of belief must
be a probability measure.

Cox's theorem reaches the same conclusion from dierent premises: if degrees of plausibility
are real numbers obeying a few qualitative desiderata (consistency, and agreement with Boolean
logic in the certain limits), then they are isomorphic to probabilities and must combine by the
sum and product rules. Either way, rational belief is probability, and updating that belief on
evidence is forced to be Bayes' theorem. This is the philosophical bedrock of everything below.

3
2 Probability Foundations
We isolate the facts on which Bayes' theorem rests. Throughout, (Ω, F, P ) is a probability space
and conditioning events have positive probability.

Denition 2.1 (Conditional probability) . For events A, B with P (B) > 0, P (A | B) =

P (A ∩ B)
.
P (B)
Lemma 2.2 (Multiplication rule) . P (A ∩ B) = P (A | B)P (B) = P (B | A)P (A) whenever
P (A), P (B) > 0.

Proof. Multiply P (A | B) = P (A ∩ B)/P (B) by P (B); symmetrically for the other equality.
Both equal P (A ∩ B).

Denition 2.3 (Independence; conditional independence). Events A, B are independent if

P (A ∩ B) = P (A)P (B). Random variables X, Y are conditionally independent given Z , written
X ⊥⊥ Y | Z , if p(x, y | z) = p(x | z) p(y | z) for all z . Conditional independence of observations
given θ is the workhorse assumption that turns a likelihood into a product.

Theorem 2.4 (Law of Total Probability) . If {B1 , . . . , Bn } partition Ω (disjoint, exhaustive,

each P (Bi ) > 0) then for any event A,
n
X
P (A) = P (A | Bi ) P (Bi ).
i=1
S
Proof. Since the Bi cover Ω, A = i (A ∩ Bi )P
, a disjoint union because (A ∩ Bi ) ∩ (A ∩ Bj ) ⊆
Bi ∩ Bj = ∅. Finite additivity gives P (A) = i P (A ∩ Bi ), and Lemma 2.2 rewrites each term
as P (A | Bi )P (Bi ).
Z
The continuous analogue, used constantly below, is p(x) = p(x | θ)π(θ) dθ.
Θ

Denition 2.5 (Expectation and variance) . For a random variable θ with density π , E[θ] =
Var(θ) = E[θ2 ] − (E[θ])2 .
R
θ π(θ) dθ and The law of total expectation and law of total variance
state
   

E[Y ] = E E[Y | Z] , Var(Y ) = E Var(Y | Z) + Var E[Y | Z] ,
both of which recur when we marginalise nuisance parameters and analyse hierarchical models.

3 Bayes' Theorem
P (A | B) P (B)
Theorem 3.1 (Bayes' theorem, event form). For P (A), P (B) > 0, P (B | A) = .
P (A)

Proof. By denition P (B | A) = P (A ∩ B)/P (A) and P (A | B) = P (A ∩ B)/P (B). The second

gives P (A ∩ B) = P (A | B)P (B); substitute into the rst.

Corollary 3.2 (Extended / hypothesis form) . If {H1 , . . . , Hn } partition the hypothesis space
and D is data,
P (D | Hi )P (Hi )
P (Hi | D) = P .
j P (D | Hj )P (Hj )

Proof. Theorem 3.1 with B = Hi , A = D over P (D), then expand P (D) by Theorem 2.4.

4
Theorem 3.3 (Continuous parametric form) . With prior density π(θ) on Θ and data density
p(x | θ),
p(x | θ)π(θ)
π(θ | x) = Z
p(x | θ′ )π(θ′ ) dθ′
Θ

| θ′ )π(θ′ ) dθ′ ∈ (0, ∞).

R
wherever m(x) = Θ p(x

Proof. The joint density factors two ways,

R p(θ, x) = p(x | θ)π(θ) = π(θ | x) m(x), where
m(x) = p(θ, x) dθ is the marginal of x. Equate and divide by m(x) > 0.

3.1 The odds form: evidence as weight

For two hypotheses H1 , H2 , dividing the two instances of Bayes' theorem cancels the awkward
denominator P (D) entirely:

P (H1 | D) P (D | H1 ) P (H1 )
= × .
P (H2 | D) P (D | H2 ) P (H2 )
| {z } | {z } | {z }
posterior odds Bayes factor prior odds

Posterior odds = Bayes factor × prior odds. Taking logarithms,

P (H1 | D) P (H1 ) P (D | H1 )
log = log + log ,
P (H2 | D) P (H2 ) P (D | H2 )

so evidence adds on the log-odds scale. Independent data sets contribute additively, which
is why log-odds is the natural currency of sequential learning.

4 The Four Ingredients

Bayes' theorem reads as a sentence:

likelihood prior
z }| { z}|{
p(x | θ) π(θ)
π(θ | x) = R .
| {z } p(x | θ′ )π(θ′ ) dθ′
posterior | {z }
evidence

Symbol Name Meaning

π(θ) Prior Beliefs about θ before data.

p(x | θ) Likelihood How probable the observed data is, as a

function of θ.
m(x) Evidence Data probability averaged over all θ; the
normaliser.

π(θ | x) Posterior Updated beliefs after data.

5
 The proportionality shortcut. Because m(x) is free of θ, it is just the constant making
the posterior integrate to 1. To nd the shape,

π(θ | x) ∝ p(x | θ) π(θ) (posterior ∝ likelihood × prior),

R
then recover the constant by π(θ | x) dθ = 1 and recognise the kernel of a known
distribution.

5 Exchangeability and de Finetti's Theorem

A natural objection to the whole setup: where do the parameter θ and its prior even come
from? A pure subjectivist who only believes in observable data need never mention them. The
remarkable answer is that a mild symmetry assumption forces the parametric-model-plus-prior
structure into existence.

Denition 5.1 (Exchangeability). An innite sequence X1 , X2 , . . . is exchangeable if for every

n and every permutation σ of {1, . . . , n}, the joint distribution is invariant:

p(x1 , . . . , xn ) = p(xσ(1) , . . . , xσ(n) ).

Exchangeability says the order of observations carries no information  a far weaker and
more defensible assumption than independence.

Theorem 5.2 (de Finetti, binary case) .

X1 , X2 , · · · ∈ {0, 1} is an innite
If exchangeable se-
quence, then there exists a unique distribution π on [0, 1] such that for all n,

Z 1Yn 
p(x1 , . . . , xn ) = θxi (1 − θ)1−xi π(θ) dθ.
0 i=1
Pn
Idea of proof. Let Sn = i=1 Xi . Exchangeability makes
p(x1 , . . . , xn ) depend only on Sn . One
shows the empirical frequency Sn /n converges almost surely to a random limit θ , and that
conditionally on θ the Xi behave as i.i.d. Bernoulli(θ); the distribution π of the limit θ is the
prior. (The general version, due to Hewitt and Savage, replaces Bernoulli(θ) by an arbitrary
sampling distribution indexed by θ .)

Why this matters. de Finetti's theorem says that believing your data are exchangeable is
mathematically equivalent to believing they are i.i.d. from some distribution with an unknown
parameter θ carrying a prior π. The parameter, the likelihood, and the prior are not
arbitrary metaphysical add-ons  they are an unavoidable consequence of a symmetry you
already accept. This is the deepest justication for the Bayesian model.

6 The Likelihood Function and Suciency

Let data x = (x1 , . . . , xn ) follow p(x | θ).
Denition 6.1 (Likelihood). L(θ | x) = p(x | θ), viewed as a function of θ with x xed.
R
Two cautions: (i) it is not a distribution over θ  generally L dθ ̸= 1; it ranks how well
each θ explains the data. (ii) The same formula is a density in x but a likelihood in θ. For
conditionally independent data,

n
Y n
X
L(θ | x) = p(xi | θ), ℓ(θ) = log p(xi | θ).
i=1 i=1

6
6.1 Suciency: when a summary loses nothing

Denition 6.2 (Sucient statistic). A statistic T (x) is sucient for θ if the conditional distri-
bution of x given T (x) does not depend on θ.

Theorem 6.3 (FisherNeyman factorisation) . T (x) is sucient for θ if and only if there exist
functions g, h ≥ 0 with


p(x | θ) = g T (x), θ h(x).

Proof (discrete case). (⇐) Suppose the factorisation holds. t At = {x :

P PFix a value and let
T (x) = t}. Then P (T = t | θ) = x∈At g(t, θ)h(x) = g(t, θ) x∈At h(x). For x ∈ At ,

g(t, θ)h(x) h(x)

P (X = x | T = t, θ) = P ′
=P ′
,
g(t, θ) x′ ∈At h(x ) x′ ∈At h(x )

which is free of θ; hence T is sucient. (⇒) If T is sucient, write p(x | θ) = P (X = x | T =

T (x), θ) P (T = T (x) | θ); the rst factor is free of θ (call it h(x)) and the second is a function
of T (x) and θ (call it g ).

Corollary 6.4 (Suciency in Bayesian terms). If T (x) is sucient then π(θ | x) = π θ | T (x) :

the posterior depends on the data only through T .

Proof. Using Theorem 6.3, π(θ | x) ∝ p(x | θ)π(θ) = g(T (x), θ)h(x)π(θ) ∝ g(T (x), θ)π(θ) after
dropping the θ-free factor h(x). The right side depends on data only through T (x).

Remark 6.5 (The Likelihood Principle) . Proportional likelihoods carry identical information
about θ. Bayesian inference obeys this automatically, since π(θ | x) depends on data only
through L(θ | x). A practical consequence: the experimenter's stopping rule (stop after n trials
vs. stop after k successes) does not aect Bayesian conclusions, though it famously aects
frequentist p-values.

7 Prior Distributions
The prior π(θ) encodes information available before the current data. Choosing it well is where
modelling judgement enters.

7.1 A taxonomy

ˆ Informative priors encode genuine knowledge (a physical constant, a prior study's eect
size). Stronger priors need more data to overturn.

ˆ Weakly informative priors are deliberately broad: enough to exclude the absurd, while
letting data dominate (e.g. a N (0, 102 ) prior on a standardised regression coecient).

ˆ Non-informative / objective priors aim to let the data speak (at, Jereys, reference
priors).

ˆ Conjugate priors keep the posterior in the prior's family (Ÿ9).

ˆ Hierarchical priors place priors on prior hyperparameters (Ÿ12).

7.2 Improper priors

R
A prior with
R dθ = ∞ (e.g. π ∝ 1
π(θ) on R) is improper. It can still yield a proper posterior
provided m(x) = p(x | θ) π(θ) dθ < ∞  but propriety must be checked, not assumed, or the
posterior is meaningless.

7
7.3 The Jereys prior, with worked derivations
p
The at prior is not invariant: at in θ is not at in g(θ). Jereys' x is πJ (θ) ∝ I(θ) with
I(θ) = −E ∂θ2 log p(X | θ) .
 
Fisher information

Proposition 7.1 (Invariance) . Under a smooth invertible reparameterisation ϕ = g(θ), the

Jereys rule gives the same prior whether applied in θ or ϕ.

Proof. Fisher information transforms as I(ϕ) = I(θ)(dθ/dϕ)2 , and densities transform as π (ϕ) =
p p p J
πJ (θ)|dθ/dϕ|. Hence πJ (ϕ) ∝ I(θ) |dθ/dϕ| = I(θ)(dθ/dϕ)2 = I(ϕ).

Example 7.2 (Jereys priors for three standard models).

ˆ Bernoulli(θ): log p = x log θ+(1−x) log(1−θ), so I(θ) = 1
θ(1−θ) and πJ (θ) ∝ θ−1/2 (1−θ)−1/2 ,
1 1
i.e. Beta( 2 , 2 )  the arcsine prior.

ˆ Poisson(λ): I(λ) = 1/λ, so πJ (λ) ∝ λ−1/2 (improper).

ˆ Normal mean µ, known σ 2 : I(µ) = 1/σ 2 is constant, so πJ (µ) ∝ 1 (the at prior is already
invariant here).

7.4 The prior predictive (and how to elicit a prior)

Before any data, the model predicts observables through the prior predictive (a.k.a. marginal)
distribution Z
m(x) = p(x | θ) π(θ) dθ.
Θ
This is invaluable for prior elicitation and checking : simulate θ ∼ π, then x ∼ p(· | θ), and ask
whether the resulting fake data sets look plausible. If a vague prior generates absurd data, it
is not actually vague in a good way. Elicitation in practice means translating expert statements
(the rate is probably between 2 and 6) into hyperparameters (e.g. a Gamma with that central
range).

7.5 Mixture priors

P P
A nite mixture π(θ) = k wk πk (θ) (wk ≥ 0, k wk = 1) encodes several competing hypotheses
(e.g. the coin is fair vs. it could be anything). Mixtures are closed under updating:

Z
X wk mk (x)
π(θ | x) = w̃k πk (θ | x), w̃k = P , mk (x) = p(x | θ)πk (θ) dθ,
k j wj mj (x)

so the data re-weight the components by their marginal likelihoods  a clean preview of model
comparison (Ÿ16).

8 The Posterior and Sequential Updating

The posterior is the complete answer: a full distribution from which any summary follows.

Theorem 8.1 (Coherent sequential updating) . If x1 , x2 are conditionally independent given θ,

then π(θ | x1 , x2 ) ∝ p(x2 | θ) π(θ | x1 ). Batch and online updating coincide.

Proof. π(θ | x1 , x2 ) ∝ p(x1 | θ)p(x2 | θ)π(θ). Group as p(x2 | θ) · [p(x1 | θ)π(θ)]; the bracket
is ∝ π(θ | x1 ). Both processing orders normalise the same product, so they give the same
posterior.

8
 Yesterday's posterior is today's prior. This coherence is unique to the Bayesian
framework and is exactly what makes streaming/online learning principled: no special
corrections, no double-counting, no dependence on the order data happened to arrive.

9 Conjugate Models, with Full Proofs

Denition 9.1 (Conjugacy) . A family F of priors is conjugate to a likelihood if a prior in F
always yields a posterior in F.

Each proof uses only the proportionality shortcut and recognise the kernel.

9.1 BetaBinomial: a probability

Model. x | θ ∼ Binomial(n, θ). Prior. θ ∼ Beta(α, β).

Theorem 9.2. θ | x ∼ Beta(α + x, β + n − x).

Proof. Dropping θ-free factors, π(θ | x) ∝ θx (1 − θ)n−x · θα−1 (1 − θ)β−1 = θ(α+x)−1 (1 −

θ)(β+n−x)−1 , the kernel of Beta(α + x, β + n − x).

Corollary 9.3 (Shrinkage) . With µ0 = α

α+β and θ̂ = x
n, E[θ | x] = α+x
α+β+n = wθ̂ + (1 − w)µ0 ,
n
w= n+α+β .

α+x n x a α
Proof. a+n = a+n n + a+n a with a = α + β. As n → ∞, w → 1 (prior washes out); as a → ∞,
w → 0.

9.2 GammaPoisson: a rate

Model. iid
Prior.
P
xi ∼ Poisson(λ), S = xi . λ ∼ Gamma(α, β).

Theorem 9.4. λ | x ∼ Gamma(α + S, β + n), so E[λ | x] = α+S

β+n = n
β+n x̄ + β
β+n µ0 .

Proof. π(λ | x) ∝ λS e−nλ · λα−1 e−βλ = λ(α+S)−1 e−(β+n)λ , a Gamma kernel. The mean splits as
in Corollary 9.3.

9.3 NormalNormal: a mean (known variance)

Model. iid
xi ∼ N (µ, σ 2 ), σ 2 known. Prior. µ ∼ N (µ0 , τ02 ).

Theorem 9.5. µ | x ∼ N (µn , τn2 ) 1 1 n

µn = τn2 µ0 nx̄


with
τn2
= τ02
+ σ2
and
τ02
+ σ2
.

Proof. Up to constants in µ, log π(µ | x) = − 21 [Aµ2 − 2Bµ] with A= n

σ2
+ 1
τ02
, B = nx̄
σ2
+ µτ 20
0
B 2 B2
(xi − µ)2 =
P 2
xi − 2µnx̄ + nµ2 ). Aµ2 − 2Bµ = A(µ −
P
(using Completing the square,
A ) − A ,
so the posterior is Normal with variance 1/A and mean B/A.

Precisions add. With precision = 1/variance: 1

τn2
=1
τ02
+ σn2 , and µn is the precision-

weighted average of µ0 and x̄. As n → ∞, µn → x̄ 2

and τn → 0.

9
9.4 Inverse-GammaNormal: a variance (known mean)

Model. iid
xi ∼ N (µ, σ 2 ), µ
2
known, σ2 unknown. Prior. σ 2 ∼ Inv-Gamma(α, β), density
∝ (σ 2 )−α−1 e−β/σ .
Theorem 9.6. σ 2 | x ∼ Inv-Gamma α + n2 , β + 12 i (xi − µ)2 .
P

2 2 −n/2 exp(− W ).
P
Proof. Let W = i (xi − µ) . The likelihood is ∝ (σ ) 2σ 2
Multiplying by the prior,

 
π(σ 2 | x) ∝ (σ 2 )−(α+n/2)−1 exp − β+W/2
σ 2 ,

the kernel of Inv-Gamma(α + n2 , β + W

2 ).

9.5 DirichletMultinomial: category probabilities

Model. counts x = (x1 , . . . , xK ) with

P
k xk = n follow Multinomial(n, p), p on the simplex.
Prior. p ∼ Dirichlet(α), density ∝ k pαk k −1 .
Q

Theorem 9.7. p | x ∼ Dirichlet(α1 + x1 , . . . , αK + xK ).

Q xk Q αk −1 Q (αk +xk )−1
Proof. π(p | x) ∝ k pk · k pk = k pk , a Dirichlet kernel. (This is the K -category
generalisation of BetaBinomial; K = 2 recovers Theorem 9.2.)

9.6 Normal with unknown mean and variance

When both µ and σ 2 are unknown, the conjugate prior is NormalInverse-Gamma : σ 2 ∼

Inv-Gamma(a, b) and µ | σ 2 ∼ N (µ0 , σ 2 /κ0 ). The posterior is again NormalInverse-Gamma,
and marginally
µ|x ∼ Student-t,

a heavier-tailed distribution reecting the extra uncertainty from estimating σ2. This is the
Bayesian counterpart of why the frequentist t-distribution (not the normal) governs inference
for a mean with unknown variance.

9.7 Reference table

Likelihood Conjugate prior Posterior update

P P
Binomial / Bernoulli (θ) Beta(α, β) α+ xi , β + (1 − xi )
P
Poisson (λ) Gamma(α, β) α+ xi , β + n
P
Exponential (λ) Gamma(α, β) α + n, β + xi
Normal, known σ 2 (µ) N (µ0 , τ02 ) precision update (Thm. 9.5)

(σ 2 ) α + n2 , β + 1
(xi − µ)2
P
Normal, known µ Inv-Gamma(α, β) 2
Multinomial (p) Dirichlet(α) α + counts

10 The Exponential Family and General Conjugacy

The conjugacy coincidences are one theorem in disguise. A family is an exponential family if

p(x | θ) = h(x) g(θ) exp η(θ)⊤ T (x) ,

with natural parameter η(θ) and sucient statistic T (x) (note Theorem 6.3: T is automatically
sucient).

10
Theorem 10.1 (Conjugate prior for an exponential family) .
For an i.i.d. sample, the prior
π(θ | χ, ν) ∝ g(θ)ν ⊤
exp(η(θ) χ) is conjugate, with posterior hyperparameters
n
X
χ 7→ χ + T (xi ), ν 7→ ν + n.
i=1

| θ) = [ i h(xi )] g(θ)n exp(η(θ)⊤ i T

Q Q P
Proof. The likelihood is i p(xi P(xi )). Multiplying by the
θ-free factors, π(θ | x) ∝ g(θ)
ν+n exp(η(θ)⊤ (χ +
prior and dropping i T (xi ))), the same form
with the stated updates.

ν behaves as a prior sample size and χ as prior sucient statistics, to which the data
P
simply add their own count n and statistic i T (xi ). BetaBinomial, GammaPoisson,
NormalNormal, Inverse-GammaNormal, and DirichletMultinomial are all special cases.

11 A Capstone: Bayesian Linear Regression

We now combine almost everything  a multivariate conjugate prior, the exponential family,
completing the square, and the predictive distribution  in the single most-used model in
statistics.
Model. Stack the responses into y ∈ Rn and predictors into the design matrix X ∈ Rn×d .
Assume
y | β ∼ N (Xβ, σ 2 In ), σ2 known for now,

p(y | β) ∝ exp − 2σ1 2 ∥y − Xβ∥2 .

so the likelihood is
Prior. A conjugate Gaussian prior on the coecient vector, β ∼ N (m0 , V0 ).

Theorem 11.1 (Conjugate posterior for the coecients) . The posterior is Gaussian, β |y∼
N (mn , Vn ), with

 −1  
Vn = V0−1 + 1
σ2
X⊤ X , mn = Vn V0−1 m0 + 1
σ2
X⊤ y .

Proof. Work with the log-posterior up to constants in β. The two quadratic forms are

− 2σ1 2 ∥y − Xβ∥2 − 12 (β − m0 )⊤ V0−1 (β − m0 ).

β . The quadratic part is − 12 β⊤ σ12 X⊤ X + V0−1 β = − 21 β⊤ Vn−1 β ,

Expand and keep terms in
⊤ 1 X⊤ y + V −1 m . Completing the square in the multivariate form,


and the linear part is β 0 0
σ2
− 21 β⊤ Vn−1 β + β⊤ b = − 21 (β − Vn b)⊤ Vn−1 (β − Vn b) + const with b = σ12 X⊤ y + V0−1 m0 , we read o
a Gaussian with covariance Vn and mean mn = Vn b.

Corollary 11.2 (Ridge regression is a MAP estimate). Take the isotropic prior β ∼ N (0, τ 2 Id ).
Then the posterior mode (equivalently the mean, by Gaussian symmetry) is

 −1
σ2
mn = X⊤ X + I
τ2 d
X⊤ y,

which is exactly the ridge / Tikhonov estimator with penalty λ = σ 2 /τ 2 .

Proof. Substitute V0 = τ 2 Id , m0 = 0 into Theorem 11.1: Vn−1 = σ12 X⊤ X + 1

I and
τ2 d
mn =
2
Vn · σ12 X⊤ y = (X⊤ X + στ 2 I)−1 X⊤ y after multiplying through by σ 2 .

11
 Regularisation is a prior. The familiar penalised estimators are MAP estimates under
explicit priors: a Gaussian prior gives ridge (ℓ2 ) regularisation, and a Laplace (double-
exponential) prior gives the lasso (ℓ1 ), whose MAP estimate is sparse. Choosing a penalty
strength is choosing a prior variance. The Bayesian view additionally supplies the full
posterior  and hence honest uncertainty on every coecient  not just a point estimate.

Proposition 11.3 (Predictive distribution for a new case) . For a new input x∗ ∈ Rd with
y∗ | β ∼ N (x⊤∗ β, σ 2 ), the posterior predictive is Gaussian,
y∗ | y ∼ N x⊤∗ mn , σ 2 + x⊤∗ Vn x∗ .

Proof. Since y∗ = x⊤∗ β + ε with ε ∼ N (0, σ 2 ) independent of the Gaussian posterior β | y ∼

N (mn , Vn ), y∗ | y is a linear image of a Gaussian plus independent Gaussian noise, hence
⊤ ⊤ 2 ⊤
Gaussian; its mean is x∗ mn and, by independence, its variance is Var(x∗ β | y) + σ = x∗ Vn x∗ +
2
σ .
The predictive variance has two parts: irreducible noise σ2 plus the term x⊤∗ Vn x∗ from
parameter uncertainty, which grows for inputs x∗ far from the observed data  the model
knows where it is extrapolating.

12 Hierarchical Models and Empirical Bayes

Often parameters are themselves related. Suppose we observe groups j = 1, . . . , J (schools,
hospitals, batches) with group means θj . Rather than treat the θj as unrelated (no pooling) or
identical (complete pooling), a hierarchical (multilevel) model places a shared prior on them:

xij | θj ∼ p(· | θj ), θj | ϕ ∼ π(· | ϕ), ϕ ∼ π(ϕ),

where ϕ are hyperparameters. The joint posterior factorises along the hierarchy:

J
Y Y
π(θ, ϕ | x) ∝ π(ϕ) π(θj | ϕ) p(xij | θj ).
j=1 i

Partial pooling and borrowing strength. The shared prior pulls each group estimate
toward the overall mean by an amount that depends on how noisy that group is: data-
poor groups are shrunk more, data-rich groups less. This borrowing of strength typically
beats both no-pooling and complete-pooling estimates  the celebrated Stein phenomenon,
recovered here automatically.

12.1 Empirical Bayes

A fully Bayesian treatment integrates over ϕ. A pragmatic shortcut, empirical Bayes, instead
estimatesϕ from the data  typically by maximising the marginal likelihood ϕ̂ = arg maxϕ m(x |
R
ϕ) = arg maxϕ p(x | θ)π(θ | ϕ) dθ  and then proceeds as if ϕ = ϕ̂.
Example 12.1 (Gaussian empirical Bayes / shrinkage) . xj | θj ∼ N (θj , σ 2 ) (σ 2 known)
Let
τ2
and θj ∼ N (0, τ 2 ). By Theorem 9.5, E[θj | xj ] =
τ 2 +σ 2
xj =: (1 − B)xj with shrinkage factor
σ2
B= σ 2 +τ 2
. Empirical Bayes estimates B from the spread of the observed xj (since marginally

xj ∼ N (0, σ 2 + τ 2 )), giving the data-driven shrinkage estimator θ̂j = (1 − B̂)xj . The estimates
are pulled toward 0, and the worse the signal-to-noise ratio, the harder the pull.

Remark 12.2. Empirical Bayes ignores uncertainty in ϕ̂ and so tends to produce slightly over-
condent intervals; the fully Bayesian (integrate-out) approach is preferable when feasible, and
modern computation (Ÿ17) usually makes it feasible.

12
13 Point Estimates and Bayesian Decision Theory
Given a loss L(θ, a), the Bayes action minimises the posterior expected loss ρ(a | x) = E[L(θ, a) |
x]. Three classical losses give the three classical summaries.

Theorem 13.1 (Squared-error ⇒ mean). arg mina E[(θ − a)2 | x] = E[θ | x].

Proof. With µ = E[θ | x], E[(θ − a)2 | x] = Var(θ | x) + (µ − a)2 (the cross term vanishes since
E[θ − µ | x] = 0), minimised uniquely at a = µ.

Theorem 13.2 (Absolute-error ⇒ median). arg mina E[|θ − a| | x] is any posterior median.

Proof.
′ (a) = 2F (a) − 1, where F is the
g(a) = E[|θ − a| | x] is convex with right-derivative g+
posterior CDF. This is ≤ 0 below a median and ≥ 0 above it, so the minimum is attained exactly
−
where F (a ) ≤
1
2 ≤ F (a).

Theorem 13.3 (01 loss ⇒ mode (MAP)). Under Lϵ (θ, a) = 1{|θ − a| > ϵ}, the Bayes action
tends toarg maxθ π(θ | x) as ϵ ↓ 0.
R a+ϵ
Proof. E[Lϵ | x] = 1 − a−ϵ π(θ | x) dθ is minimised by maximising local posterior mass; as ϵ ↓ 0,
1
R a+ϵ
2ϵ a−ϵ π(θ | x) dθ → π(a | x), so the maximiser tends to the mode.

13.1 Risk and admissibility

The (frequentist) risk of an estimator δ is R(θ, δ) = Ex|θ [L(θ, δ(x))]

R , averaging loss over data for
xed θ. The Bayes risk averages risk over the prior, r(π, δ) = R(θ, δ)π(θ) dθ.

Theorem 13.4 (Bayes rules minimise Bayes risk). The estimator δπ that, for each x, minimises
ρ(a | x) also minimises the Bayes risk r(π, δ).
the posterior expected loss
RR R
Proof. By Fubini, r(π, δ) = L(θ, δ(x))p(x | θ)π(θ) dθ dx = ρ(δ(x) | x) m(x) dx. Since
m(x) ≥ 0, minimising the inner integrand ρ(δ(x) | x) pointwise in x minimises the whole
integral.

A decision rule is admissible if no other rule has risk ≤ its risk for all θ with strict inequality
somewhere. A standard result: Bayes rules with proper priors and nite Bayes risk are admissi-
ble, giving Bayesian estimators an automatic optimality guarantee that frequentist constructions
must earn case by case.

Summary. squared error → mean ; absolute error → median ; 01 loss → mode ; and Bayes
rules minimise Bayes risk and are typically admissible.

14 Credible Intervals
Denition 14.1 (Credible set).
R
100(1 − α)% credible set is any C with C π(θ | x) dθ = 1 − α.
A
The equal-tailed interval removes α/2 from each tail; the HPD (highest posterior density) set is
{θ : π(θ | x) ≥ c} with c chosen to give probability 1 − α.

Proposition 14.2 (HPD is shortest) . For a unimodal posterior, the HPD set has the smallest
length among credible sets of its level.

Proof. Any competitor C of equal probability swaps a high-density piece of the HPD set H
(density ≥ c) for a low-density piece outside (< c). Equal probability forces the swapped masses
equal; since density is larger on the discarded piece, its length is smaller, so |C| ≥ |H|.

13
 A 95% credible interval genuinely means: given the data, there is a 95% probability θ lies
inside. This is the statement people think a condence interval makes  valid only in the
Bayesian framework (Ÿ18).

15 The Posterior Predictive Distribution

Z
Theorem 15.1 (Posterior predictive). If x̃ ⊥⊥ x | θ, then p(x̃ | x) = p(x̃ | θ) π(θ | x) dθ.
Θ
R R R
Proof. p(x̃ | x) = p(x̃, θ | x) dθ = p(x̃ | θ, x)π(θ | x) dθ = p(x̃ | θ)π(θ | x) dθ, the last step
by conditional independence.

This averages the model over the posterior, propagating parameter uncertainty into the fore-
cast  strictly more honest than plugging in θ̂, and it yields wider, better-calibrated predictive
intervals.

Example 15.2 (Laplace's Rule of Succession) . With prior Beta(α, β) and x successes in n
α+x x+1
trials, P (x̃ = 1 | x) = E[θ | x] = α+β+n . Under the uniform prior this is n+2 , never exactly 0
or 1: a built-in guard against overcondence from limited data. (Laplace used it to bound the
probability the sun rises tomorrow.)

Example 15.3 (A predictive that is not just a plug-in) . For the GammaPoisson model the
posterior predictive for a new count is Negative Binomial, not Poisson: integrating Poisson(λ)
against Gamma(α + S, β + n) inates the variance beyond the Poisson's mean-equals-variance,
correctly reecting that λ is still uncertain. Over-dispersion is the visible ngerprint of honest
uncertainty propagation.

16 Bayesian Model Comparison

Bayesian reasoning extends from parameters to models. Let M1 , M2 be competing models
with parameters θk and within-model priors πk (θk ). The key quantity is each model's marginal
likelihood (evidence): Z
p(x | Mk ) = p(x | θk , Mk ) πk (θk ) dθk .

p(x | M1 )
Denition 16.1 (Bayes factor). BF12 = , the factor by which the data shift the odds
p(x | M2 )
between the two models:
P (M1 | x) P (M1 )
= BF12 · .
P (M2 | x) P (M2 )
| {z } | {z }
posterior odds prior odds

Bayesian Occam's razor. The marginal likelihood automatically penalises complexity.

A exible model spreads its prior predictive mass over many possible data sets, so it assigns
less density to the one actually observed; a model just complex enough to t the data
concentrates its predictions and wins. No explicit penalty term is added  parsimony
falls out of integrating over parameters. The Laplace approximation (Ÿ17) makes this
explicit: log p(x | M) ≈ ℓ(θ̂)− d2
log n+ const, whose − d2 log n term is precisely the Bayesian
Information Criterion (BIC) complexity penalty for d parameters and n data points.

A caution: marginal likelihoods can be sensitive to the within-model priors πk (unlike posteriors,
the dependence does not vanish as n → ∞), so model comparison demands carefully justied
priors.

14
Remark 16.2 (Lindley's paradox) . For a point null M2 : θ = θ0 against a diuse alternative
M1 , it is possible for the same data to reject the null at the frequentist 5% level while the
Bayes factor favours the null. The resolution is that a very diuse alternative spreads its prior
predictive mass thinly, so even a signicant deviation is better explained by the sharp null than
by an alternative that predicted almost nothing in particular. The paradox is a vivid reminder
that p-values and posterior model probabilities answer dierent questions.

17 Computational Methods
R
Outside conjugate models the evidence m(x) = R p(x | θ) π(θ) dθ is an intractable integral, and
so is any posterior expectation E[f (θ) | x] = f (θ)π(θ | x) dθ. Modern Bayesian practice
replaces exact integration with computation.

17.1 Monte Carlo integration

If we can draw θ(1) , . . . , θ(M ) ∼ π(θ | x), then by the law of large numbers

M
d] = 1
X a.s.
E[f f (θ(m) ) −−−→ E[f (θ) | x],
M
m=1

with Monte Carlo error O(M −1/2 ) independent of the dimension of θ  the property that makes
sampling so powerful in high dimensions.

17.2 Importance sampling

When we cannot sample π(θ | x) but can evaluate it up to a constant, draw θ(m) ∼ q from a
tractable proposal and reweight:

(m) )
p(x | θ(m) )π(θ(m) )
P
wm f (θ
mP
E[f (θ) | x] ≈ , wm = .
m wm q(θ(m) )
The self-normalised form cancels the unknown m(x). Eciency hinges on q resembling the
posterior; heavy-tailed q guards against rare huge weights.

17.3 Markov chain Monte Carlo (MCMC)

The dominant general tool builds a Markov chain whose stationary distribution is the posterior,
then runs it and treats the (correlated) states as samples.

Theorem 17.1 (MetropolisHastings has the right stationary law) . Let the target be π ∗ (θ) ∝
p(x | θ)π(θ). From state θ, propose θ′ ∼ q(θ′
| θ) and accept it with probability
n π ∗ (θ′ ) q(θ | θ′ ) o
α(θ, θ′ ) = min 1, ∗ ,
π (θ) q(θ′ | θ)
otherwise stay at θ. The resulting chain has π∗ as a stationary distribution.

Proof. It suces to verify detailed balance : π ∗ (θ)P (θ → θ′ ) = π ∗ (θ′ )P (θ′ → θ) for θ ̸= θ′ , where
′ ′ ′
the transition density is P (θ → θ ) = q(θ | θ)α(θ, θ ). Compute

∗ ′ )q(θ|θ ′ )
n o
π ∗ (θ)q(θ′ | θ)α(θ, θ′ ) = π ∗ (θ)q(θ′ | θ) min 1, ππ∗(θ = min π ∗ (θ)q(θ′ | θ), π ∗ (θ′ )q(θ | θ′ ) .

′
(θ)q(θ |θ)

The nal expression is symmetric in(θ, θ′ ), hence ∗ ′ ′

R ∗ equals π (θ′ )q(θ | θ )α(θ
′ , θ) = π ∗ (θ ′ )P (θ ′ → θ).

Integrating detailed balance over θ gives π (θ)P (θ → θ ) dθ = π (θ ), i.e. π ∗ is stationary.

∗ ′

Note the unknown normaliser m(x) cancels in the ratio, so only the unnormalised posterior is
needed.

15
Remark 17.2. Stationarity plus irreducibility and aperiodicity of the chain yields convergence
of the time-average to the posterior expectation (an ergodic theorem). In practice one discards
an initial burn-in, monitors convergence (e.g. the R̂ statistic across several chains), and accounts
for autocorrelation via the eective sample size.

17.4 Gibbs sampling

When the full conditionals π(θj | θ−j , x) are available in closed form (as they are in conjugate
hierarchical models), Gibbs sampling cycles through them:

(t+1) (t) (t) (t+1) (t+1) (t+1)

θ1 ∼ π(θ1 | θ2 , . . . , θd , x), . . . , θd ∼ π(θd | θ1 , . . . , θd−1 , x).
Proposition 17.3. Gibbs sampling is a special case of MetropolisHastings in which every
proposal is accepted (α ≡ 1); hence its stationary distribution is the joint posterior.

Sketch. Take the proposal for the j -th update to be the full conditional q = π(θj′ | θ−j , x).
∗
Substituting into the acceptance ratio, the joint factors as π (θ) = π(θj | θ−j , x) π(θ−j | x), and
the θ−j factors and the proposal terms cancel, leaving the ratio equal to 1.

17.5 The Laplace approximation

For a smooth, sharply peaked posterior, a second-order Taylor expansion of log π ∗ (θ) about its
mode θ̂ gives a Gaussian approximation

−1
Σ = − ∇2 log π ∗ (θ̂)


π(θ | x) ≈ N θ̂, Σ , ,

and a closed-form estimate of the evidence m(x) ≈ π ∗ (θ̂) (2π)d/2 |Σ|1/2 . It is fast and underlies
the BIC, but degrades for skewed or multimodal posteriors, where MCMC or variational infer-
ence (which ts the closest tractable distribution by minimising KullbackLeibler divergence) is
preferred.

18 Bayesian versus Frequentist Inference

Aspect Frequentist Bayesian
Nature of θ Fixed unknown constant Random variable with a
distribution

Probability Long-run frequency over Degree of belief / plausibility

repeats

Prior information Not formally used Encoded in the prior

Main output Estimate, condence interval, Full posterior distribution

p-value
Interval meaning 95% of such intervals cover θ 95% probability θ is inside,
given data

Nuisance Often awkward Integrated out

parameters

Sequential data Needs multiple-testing care Posterior becomes next prior

Stopping rules Aect p-values Irrelevant (Likelihood

Principle)

Optimality Earned case by case Bayes rules admissible

(Thm. 13.4)

16
 The condence-interval subtlety. A 95% CI [L(X), U (X)] satises P (L(X) ≤ θ ≤
U (X)) = 0.95 over the random data X with θ xed. After observing data and computing
[2.1, 3.8], the truth either is or is not inside; the  95% is a property of the procedure's
long-run coverage, not of this interval. Only a credible interval licenses there is a 95%
probability θ ∈ [2.1, 3.8].

Advantages of the Bayesian approach.

1. Direct probability statements about θ and about hypotheses.

2. Principled incorporation of prior knowledge.

3. Coherent sequential updating (Theorem 8.1).

4. Automatic uncertainty propagation via the posterior predictive.

5. Clean marginalisation of nuisance parameters.

6. Respect for the Likelihood Principle (stopping rules irrelevant).

7. Built-in decision theory and admissibility.

8. Exactness for any sample size.

Honest caveats. prior sensitivity with little data; the appearance of subjectivity; compu-
tational cost outside conjugate models; and the communication risk that audiences misread
credible intervals.

19 Large-Sample Theory
Theorem 19.1 (Posterior consistency, informal) . Under regularity conditions, if the data are
generated by θ0 and the prior assigns positive probability to every neighbourhood of θ0 , then the
posterior concentrates: for any neighbourhood U of θ0 , P (θ ∈ U | x1:n ) → 1 almost surely as
n → ∞. The truth is eventually learned regardless of the (non-dogmatic) prior.

Theorem 19.2 (Bernsteinvon Mises, informal) . Under regularity conditions, the posterior is
asymptotically Gaussian and prior-free:

π(θ | x) ≈ N θ̂MLE , In (θ̂)−1 ,

where In is the observed Fisher information. Consequently Bayesian credible intervals and fre-
quentist condence intervals coincide to leading order.

With abundant data and a reasonable prior, the two paradigms agree numerically: the
likelihood overwhelms the prior, and the posterior centres on the MLE with the classical
variance. The frameworks therefore diverge most  and the Bayesian machinery earns its
keep  precisely when data are scarce or genuine prior information exists.

20 Fully Worked Examples

Example 20.1 (Medical test  base-rate neglect) . Prevalence P (D) = 0.01, sensitivity P (+ |
D) = 0.99, false-positive rate P (+ | Dc ) = 0.05. By Corollary 3.2,

(0.99)(0.01) 0.0099
P (D | +) = = ≈ 0.167.
(0.99)(0.01) + (0.05)(0.99) 0.0594

17
A positive test implies only a ≈ 17% disease probability: with a rare disease the prior dominates,
and ignoring base rates yields badly wrong intuition.

Example 20.2 (Coin bias  BetaBinomial). Prior Beta(2, 2), data x = 15 heads in n = 20.
Posterior Beta(17, 7) (Theorem 9.2); posterior mean 17 24 ≈ 0.708, between prior mean 0.5 and
20
MLE 0.75 with data weight 24 ≈ 0.833. A 95% equal-tailed credible interval is ≈ [0.51, 0.87]:
given the data, a 95% probability the bias lies there.

Example 20.3 (Arrival rate  GammaPoisson) . Prior Gamma(3, 1), data x = (5, 2, 7, 6),
S = 20, n = 4. Posterior Gamma(23, 5) (Theorem 9.4); mean 4.6, variance 0.92. The sample
4
mean 5 shrinks prior mean 3 with weight
toward
5 , and 0.8(5) + 0.2(3) = 4.6. A new-count
forecast uses the Negative-Binomial posterior predictive, wider than a plug-in Poisson.

Example 20.4 (Two-school comparison  hierarchical shrinkage) . Two schools report mean
test gains x1 = 8 (σ = 4) x2 = 2 (σ = 4) on small classes. A no-pooling estimate trusts
and
each number; a hierarchical model with θj ∼ N (µ, τ 2 ) shrinks both toward the common mean
x̄ = 5, by more when τ is small (schools are similar) and less when τ is large. The hierarchical
estimates  say 6.5 and 3.5 for moderate τ  are more reliable predictors of next year's gains
than the raw, noisy class averages.

Example 20.5 (Bayesian A/B testing  the probability one variant is better) . An online
experiment shows variant A to nA = 200 users with xA = 30 conversions, and variant B to
nB = 180 users with xB = 36 conversions. Put independent uniform priors θA , θB ∼ Beta(1, 1).
By Theorem 9.2,

θA | data ∼ Beta(31, 171), θB | data ∼ Beta(37, 145).

31 37
Posterior means are
202 ≈ 0.153 and 182 ≈ 0.203. The decision-relevant quantity is not a p-value
but the direct probability

Z 1Z 1
P (θB > θA | data) = 1{θB > θA } π(θA )π(θB ) dθA dθB ,
0 0

easily estimated by Monte Carlo: draw many pairs from the two posteriors and report the
fraction with θB > θA (here ≈ 0.91). One can equally report P (θB /θA > 1.1 | data)  the
probability B beats A by at least 10%  or the posterior of the dierence θB − θA for a full
picture. This is exactly the question a product team asks, answered directly; the frequentist test
answers a dierent, more circuitous one.

21 Exercises (with solutions)

Exercise 1. A box holds a fair coin and a two-headed coin. One is drawn at random and ipped,
landing heads. Find the probability it was the two-headed coin.

1 1
Solution. With H2 = two-headed, P (H2 ) = P (F ) = 2, P (heads | H2 ) = 1, P (heads | F ) = 2.
1· 12 1/2
By Bayes, P (H2 | heads) = 1· 12 + 21 · 12
= 3/4 = 32 .

Exercise 2. For a Beta(α, β) prior and x successes in n trials, give the posterior variance and
its rate in n.
′ ′
Solution. Posterior Beta(a′ , b′ ) with a′ = α + x, b′ = β + n − x. Its variance is (a′ +b′ )2a(ab ′ +b′ +1) .
′ ′ 3
Since a + b = α + β + n, the denominator grows like n while the numerator grows like n , so
2

the variance is O(1/n)  posterior uncertainty shrinks at the usual root-n rate.

Exercise 3. Prove the Inverse-Gamma posterior of Theorem 9.6 from scratch.

18
 2)
Solution. See the proof of Theorem 9.6: multiply the likelihood ∝ (σ 2 )−n/2 e−W/(2σ by the
∝ (σ 2 )−α−1 e −β/σ 2 Inv-Gamma(α + n2 , β + W
prior and collect exponents to recognise
2 ) with
(xi − µ)2 .
P
W =
Exercise 4. Show the BetaBinomial posterior predictive for k successes in m new trials is the
BetaBinomial distribution.
m

R1
Solution. Integrate the binomial pmf against the Beta(a′ , b′ ) posterior: P (k | x) = k
0 θ (1 −
k
′ ′
θa −1 (1−θ)b −1 m B(a′ +k, b′ +m−k) ′ ′
θ)m−k


B(a′ ,b′ ) dθ = k B(a′ ,b′ ) , which is exactly the BetaBinomial(m, a , b ) pmf.

Exercise 5. Using Theorem 17.1, explain why the unknown normalising constant m(x) never
needs to be computed in MetropolisHastings.

Solution. The acceptance ratio depends on the target only through π ∗ (θ′ )/π ∗ (θ). Writing
∗
π = π̃/m(x) with π̃ = p(x | θ)π(θ) the unnormalised posterior, the factor m(x) cancels, so only
π̃  the easy part  is required.
Exercise 6. In Bayesian linear regression with prior β ∼ N (0, τ 2 I), show that as τ 2 → ∞ the
posterior mean tends to the ordinary least-squares estimator β̂OLS = (X X)
⊤ −1 X⊤ y .

⊤ σ 2 −1 ⊤ 2 σ2
Solution. By Corollary 11.2, mn = (X X + 2 I) X y . As τ → ∞ the term 2 I → 0, so
τ τ
mn → (X⊤ X)−1 X⊤ y = β̂OLS . A vague prior recovers the frequentist estimate, consistent with
Bernsteinvon Mises.

Exercise 7. A coin is ipped until the rst head, which occurs on trial k (a Geometric likelihood,
p(k | θ) = (1 − θ)k−1 θ). Show Beta is still conjugate and nd the posterior.

Solution. π(θ | x) ∝ θ(1 − θ)k−1 · θα−1 (1 − θ)β−1 = θ(α+1)−1 (1 − θ)(β+k−1)−1 , so θ | k ∼

Beta(α + 1, β + k − 1). Note the posterior depends on the data only through the sucient
count k , and  by the Likelihood Principle  it matches what BetaBinomial would give for
 1 success in k trials, even though the stopping rule diers.

22 Common Misconceptions
1. The likelihood is the probability of θ. No  it is the probability of the data read as a
function of θ, and does not integrate to 1 over θ (Ÿ6).

2. The prior is just an arbitrary fudge. A prior is an explicit, auditable assumption; frequentist
procedures also encode assumptions, only less visibly. With ample data the prior washes out
(Bernsteinvon Mises, Thm. 19.2); with little data, being forced to state assumptions is a
feature, not a bug.

3. A 95% condence interval has a 95% chance of containing θ. False for condence intervals;
true only for credible intervals (Ÿ14).

4. Flat priors are always non-informative. A prior at in θ is informative in g(θ); non-
informative is not reparameterisation-invariant, which is why Jereys priors exist (Ÿ7).

5. A bigger Bayes factor always means a better model. Bayes factors depend on within-model
priors and can behave counter-intuitively (Lindley's paradox); they answer which model
predicted these data better, not which model is true.

6. MCMC gives independent samples from the posterior. The draws are correlated ; eective
sample size, burn-in, and convergence diagnostics matter (Ÿ17).

7. The MAP estimate is the best summary. The mode ignores the shape of the posterior and
is not invariant under reparameterisation; the mean or median, plus an interval, is usually
more informative (Ÿ13).

19
23 Summary
Bayes' theorem is the unique coherent rule for updating beliefs:
p(x | θ)π(θ)
π(θ | x) = R ∝ p(x | θ) π(θ) .
p(x | θ′ )π(θ′ ) dθ′ | {z } |{z}
likelihood prior

ˆ Foundations. Coherence (Dutch-book) and Cox's theorem force rational belief to be prob-
ability; de Finetti's theorem (Ÿ5) shows exchangeability is equivalent to the model-plus-prior
structure.

ˆ Mechanics. prior, likelihood, evidence, posterior; suciency (Ÿ6) compresses data without
loss; conjugacy (Ÿ9) and the exponential-family rule (Thm. 10.1) give closed-form updates
whose means are weighted averages of prior and data.

ˆ Structure and decisions. hierarchical models and empirical Bayes (Ÿ12) borrow strength
across groups; decision theory (Ÿ13) identies mean/median/mode as Bayes estimators and
guarantees admissibility; credible intervals make direct probability statements; the posterior
predictive forecasts honestly.

ˆ Comparison and computation. Bayes factors (Ÿ16) compare models with a built-in Occam
penalty; MCMC, Gibbs, importance sampling, and the Laplace approximation (Ÿ17) make
all of it computable.

ˆ Relation to frequentism. The two agree asymptotically (Bernsteinvon Mises), and the
Bayesian approach is most valuable when data are scarce or prior knowledge is real, at the
cost of prior sensitivity and computation.

Bayesian inference treats learning as the disciplined revision of a probability distribution 

posterior ∝ likelihood × prior  returning not a single number but a full, interpretable
description of belief.

A Appendix: Distribution Quick Reference

Distribution Density / pmf (kernel) Mean
α
Beta(α, β) θα−1 (1 − θ)β−1 on (0, 1)
α+β
α
Gamma(α, β) (rate) λα−1 e−βλ on (0, ∞)
β
2 β
Inv-Gamma(α, β) (σ 2 )−α−1 e−β/σ (α > 1)
α−1
2

N (µ, σ 2 ) exp − (θ−µ)
2σ 2 µ
Q αk −1 α
Dirichlet(α) k pk on simplex Pk
j αj
Poisson(λ) λx e−λ /x! λ

20