Example i.

Sequential
learning

Example ii.
Normal-
normal
Lecture 1:
Turning the
Overview of Bayesian Econometrics
Bayesian
crank
Prior predictive
Posterior
Posterior
Hedibert Freitas Lopes
predictive
Sequential Bayes
Model
probability
The University of Chicago Booth School of Business
Posterior odds 5807 South Woodlawn Avenue, Chicago, IL 60637
Bayes factor [Link]
Marginal
likelihood
hlopes@[Link]
Example iii.
Multiple linear
regression

Real data
exercise

Example iv.
SV model
 Outline
Example i.
Sequential
learning
1 Example i. Sequential learning
Example ii. 2 Example ii. Normal-normal
Normal-
normal
3 Turning the Bayesian crank
Turning the
Bayesian Prior predictive
crank
Prior predictive Posterior
Posterior
Posterior Posterior predictive
predictive
Sequential Bayes Sequential Bayes
Model
probability
Posterior odds
Model probability
Bayes factor
Marginal
Posterior odds
likelihood
Bayes factor
Example iii.
Multiple linear Marginal likelihood
regression

Real data 4 Example iii. Multiple linear regression

exercise

Example iv. 5 Real data exercise

SV model
6 Example iv. SV model
 Example i. Sequential learning
Example i.
Sequential
learning
• John claims some discomfort and goes to the doctor.
Example ii.
Normal-
normal

Turning the • The doctor believes John may have the disease A.
Bayesian
crank
Prior predictive
Posterior • θ = 1: John has disease A; θ = 0: he does not.
Posterior
predictive
Sequential Bayes
Model
probability • The doctor claims, based on his expertise (H), that
Posterior odds
Bayes factor
Marginal
likelihood P(θ = 1|H) = 0.70
Example iii.
Multiple linear
regression • Examination X is related to θ as follows
Real data 
exercise P(X = 1|θ = 0) = 0.40, positive test given no disease
Example iv.
SV model
P(X = 1|θ = 1) = 0.95, positive test given disease
 Observe X = 1
Example i.
Sequential
learning

Example ii. Exam’s result: X = 1

Normal-
normal
P(θ = 1|X = 1) ∝ P(X = 1|θ = 1)P(θ = 1)
Turning the
Bayesian
crank
∝ (0.95)(0.70) = 0.665
Prior predictive
Posterior P(θ = 0|X = 1) ∝ P(X = 1|θ = 0)P(θ = 0)
Posterior
predictive
Sequential Bayes
∝ (0.40)(0.30) = 0.120
Model
probability
Posterior odds Consequently
Bayes factor
Marginal
likelihood
P(θ = 0|X = 1) = 0.120/0.785 = 0.1528662 and
Example iii.
Multiple linear
regression
P(θ = 1|X = 1) = 0.665/0.785 = 0.8471338
Real data
exercise The information X = 1 increases, for the doctor, the
Example iv. probability that John has the disease A from 70% to 84.71%.
SV model
 Posterior predictive
Example i.
Sequential John undertakes the test Y , which relates to θ as follows1
learning

Example ii.
Normal- P(Y = 1|θ = 1) = 0.99 and P(Y = 1|θ = 0) = 0.04
normal

Turning the
Bayesian
crank
Then, the predictive of Y = 0 given X = 1 is given by
Prior predictive
Posterior P(Y = 0|X = 1) = P(Y = 0|X = 1, θ = 0)P(θ = 0|X = 1)
Posterior
predictive
Sequential Bayes + P(Y = 0|X = 1, θ = 1)P(θ = 1|X = 1)
Model
probability = P(Y = 0|θ = 0)P(θ = 0|X = 1)
Posterior odds
Bayes factor
Marginal
+ P(Y = 0|θ = 1)P(θ = 1|X = 1)
likelihood
= (0.96)(0.1528662) + (0.01)(0.8471338)
Example iii.
Multiple linear
regression
= 15.52%
Real data
exercise

Example iv.
SV model Key condition: X and Y are conditionally independent given
θ.
1
Recall that P(X = 1|θ = 1) = 0.95 and P(X = 1|θ = 0) = 0.40.
 Model criticism
Example i.
Sequential
learning Suppose the observed result was Y = 0. This is a reasonably
Example ii. unexpected result as the doctor only gave it roughly 15%
Normal-
normal chance.
Turning the
Bayesian
crank He should at least consider rethinking the model based on this
Prior predictive
Posterior result. In particular, he might want to ask himself
Posterior
predictive
Sequential Bayes
Model
probability
1 Did 0.7 adequately reflect his P(θ = 1|H)?
Posterior odds
Bayes factor
Marginal
likelihood
2 Is test X really so unreliable?
Example iii.
Multiple linear
regression
3 Is the sample distribution of X correct?
Real data
exercise 4 Is the test Y so powerful?
Example iv.
SV model
5 Have the tests been carried out properly?
 Observe Y = 0
Example i.
Sequential
learning Let H2 = {X = 1, Y = 0}. Then, Bayes theorem leads to
Example ii.
Normal- P(θ = 1|H2 ) ∝ P(Y = 0|θ = 1)P(θ = 1|X = 1)
normal

Turning the ∝ (0.01)(0.8471338) = 0.008471338

Bayesian
crank P(θ = 0|H2 ) ∝ P(Y = 0|θ = 0)P(θ = 0|X = 1)
Prior predictive
Posterior
Posterior
∝ (0.96)(0.1528662) = 0.1467516
predictive
Sequential Bayes
Model Therefore,
probability
Posterior odds
Bayes factor
P(Y = 0, θ = 1|X = 1)
Marginal
likelihood P(θ = 1|X = 1, Y = 0) = = 0.0545753
P(Y = 0|X = 1)
Example iii.
Multiple linear
regression

Real data
exercise
 0.7000 , H0 : before X and Y
Example iv.
P(θ = 1|Hi ) = 0.8446 , H1 : after X=1 and before Y
0.0546 , H2 : after X=1 and Y=0
SV model

 Example ii. Normal-normal
Example i.
Sequential
learning
Consider a simple measurement error model
Example ii.
Normal-
normal
X =θ+ε ε ∼ N(0, σ 2 )
Turning the
Bayesian
crank where
Prior predictive
Posterior θ ∼ N(θ0 , τ02 ).
Posterior
predictive
Sequential Bayes
Model
probability
Posterior odds The quantities (σ 2 , θ0 , τ02 ) are known.
Bayes factor
Marginal
likelihood

Example iii.
Multiple linear
The posterior distribution of θ (after X = x is observed) is
regression

Real data p(x|θ)p(θ)

exercise p(θ|x) =
Example iv.
p(x)
SV model
 More precisely,
p(θ|x) ∝ exp{−0.5(θ2 − 2θx)/σ 2 }exp{−0.5(θ2 − 2θθ0 )/τ02 }
× exp{−0.5(θ2 (1/σ 2 + 1/τ02 ) + 2θ(x/σ 2 + θ0 /τ02 )}
Example i.
Sequential = exp{−0.5(θ2 /τ12 + 2θτ12 (x/σ 2 + θ0 /τ02 )/τ12 }
learning
= exp{−0.5(θ2 + 2θθ1 )/τ12 }.
Example ii.
Normal-
normal

Turning the
Bayesian
Therefore, θ|x is normally distributed with
crank
Prior predictive E (θ|x) = τ12 (x/σ 2 + θ0 /τ02 )
Posterior
Posterior
predictive and
−1
Sequential Bayes
Model
probability
V (θ|x) = 1/σ 2 + 1/τ02 .
Posterior odds
Bayes factor
Marginal
likelihood Notice that
Example iii.
Multiple linear E (θ|x) = ωθ0 + (1 − ω)x
regression
where
Real data
exercise σ2
ω=
Example iv. σ 2 + τ02
SV model
measures the relative information contained in the prior
distribution with respect to the total information (prior plus
 Illustration
Example i.
Sequential
learning
Prior A: Physicist A (large experience): θ ∼ N(900, (20)2 )
Example ii.
Normal-
normal Prior B: Physicist B (not so experienced): θ ∼ N(800, (80)2 ).
Turning the
Bayesian Model: (X |θ) ∼ N(θ, (40)2 ).
crank
Prior predictive
Posterior
Posterior
predictive
Observation: X = 850
Sequential Bayes
Model
probability (θ|X = 850, HA ) ∼ N(890, (17.9)2 )
Posterior odds
Bayes factor
Marginal
(θ|X = 850, HB ) ∼ N(840, (35.7)2 )
likelihood

Example iii.
Multiple linear
regression
Information (precision)
Real data
exercise Physicist A: from 0.002500 to 0.003120 (an increase of 25%)
Example iv.
SV model Physicist B: from 0.000156 to 0.000781 (an increase of 400%)
Example i.
Sequential

0.020
learning

Example ii.
Normal-
normal Physicist A: prior
Physicist B: prior
Turning the Likelihood
0.015
Physicist A: posterior
Bayesian Physicist B: posterior
crank
Prior predictive
Posterior
Posterior
predictive
0.010

Sequential Bayes
Model
probability
Posterior odds
Bayes factor
Marginal
likelihood
0.005

Example iii.
Multiple linear
regression

Real data
exercise
0.000

Example iv.
SV model 600 700 800 900 1000

θ
 Turning the Bayesian crank
Example i.
Sequential
learning

Example ii. We usually decompose

Normal-
normal

Turning the p(θ, x|H)

Bayesian
crank
Prior predictive into
Posterior
Posterior
predictive
p(θ|H) and p(x|θ, H)
Sequential Bayes
Model
probability
Posterior odds
Bayes factor
Marginal
likelihood
The prior predictive distribution
Example iii. Z
Multiple linear
regression p(x|H) = p(x|θ, H)p(θ|H) dθ = Eθ [p(x|θ, H)]
Θ
Real data
exercise

Example iv.
if of key importance in Bayesian model assessment.
SV model
 Posterior distribution
Example i.
Sequential
learning

Example ii.
Normal-
normal
The posterior distribution of θ is obtained, after x is observed,
Turning the
Bayesian by Bayes’ Theorem:
crank
Prior predictive
p(θ, x|H)
Posterior
Posterior
p(θ|x, H) =
predictive p(x|H)
Sequential Bayes
Model
probability
p(x|θ, H)p(θ|H)
Posterior odds
=
Bayes factor
Marginal
p(x|H)
likelihood

Example iii.
Multiple linear ∝ p(x|θ, H)p(θ|H).
regression

Real data
exercise

Example iv.
SV model
 Posterior predictive distribution
Example i.
Sequential
learning Let y be a new set of observations conditionally independent of
Example ii. x given θ, ie.
Normal-
normal

Turning the
p(x, y |θ) = p(x|θ, H)p(y |θ, H).
Bayesian
crank
Prior predictive
Posterior
Posterior
predictive Then,
Sequential Bayes
Model Z
probability
Posterior odds p(y |x, H) = p(y , θ|x, H)dθ
Bayes factor
Marginal
likelihood
ZΘ
Example iii. = p(y |θ, x, H)p(θ|x, H)dθ
Multiple linear Θ
regression
Z
Real data = p(y |θ, H)p(θ|x, H)dθ
exercise Θ
Example iv. = Eθ|x [p(y |θ, H)]
SV model
Example i.
Sequential
learning

Example ii.
Normal- In general, but not always (time series, for example) x and y
normal
are independent given θ.
Turning the
Bayesian
crank
Prior predictive
Posterior It might be more useful to concentrate on prediction rather
Posterior
predictive
Sequential Bayes
than on estimation since the former is verifiable.
Model
probability
Posterior odds
Bayes factor
Marginal x and y can be (and usually are) observed; θ can not!
likelihood

Example iii.
Multiple linear
regression

Real data
exercise

Example iv.
SV model
 Sequential Bayes theorem
Example i.
Sequential
learning
Experimental result: x1 ∼ p1 (x1 |θ)
Example ii.
Normal-
normal
p(θ|x1 ) ∝ l1 (θ; x1 )p(θ)
Turning the
Bayesian
crank Experimental result: x2 ∼ p2 (x2 |θ)
Prior predictive
Posterior
Posterior
predictive p(θ|x2 , x1 ) ∝ l2 (θ; x2 )p(θ|x1 )
Sequential Bayes
Model
probability
∝ l2 (θ; x2 )l1 (θ; x1 )p(θ)
Posterior odds
Bayes factor
Marginal Experimental results: xi ∼ pi (xi |θ), for i = 3, . . . , n
likelihood

Example iii.
Multiple linear p(θ|xn , . . . , x1 ) ∝ ln (θ; xn )p(θ|xn−1 , . . . , x1 )
regression " n #
Y
Real data
exercise ∝ li (θ; xi ) p(θ)
Example iv. i=1
SV model
 Model probability
Example i.
Sequential
learning

Example ii.
Normal-
normal
Suppose that the competing models can be enumerated and
Turning the
are represented by the set
Bayesian
crank
Prior predictive M = {M1 , M2 , . . .}
Posterior
Posterior
predictive
Sequential Bayes and that the true model is in M (Bernardo and Smith, 1994).
Model
probability
Posterior odds
Bayes factor
Marginal
The posterior model probability of model Mj is given by
likelihood

Example iii. f (y |Mj )Pr (Mj )

Multiple linear Pr (Mj |y ) =
regression
f (y )
Real data
exercise

Example iv.
SV model
 Ingredients
Example i.
Sequential
learning Prior predictive density of model Mj
Example ii. Z
Normal-
normal f (y |Mj ) = f (y |θj , Mj )p(θj |Mj )dθj
Turning the
Bayesian
crank
Prior predictive
Posterior
Posterior
predictive Prior model probability of model Mj
Sequential Bayes
Model
probability
Posterior odds Pr (Mj )
Bayes factor
Marginal
likelihood

Example iii.
Multiple linear
regression Overall prior predictive
Real data
exercise
X
f (y ) = f (y |Mj )Pr (Mj )
Example iv.
SV model Mj ∈M
 Posterior odds
Example i.
Sequential The posterior odds of model Mj relative to Mk is given by
learning
Pr (Mj |y ) Pr (Mj ) f (y |Mj )
Example ii. = × .
Normal-
normal
Pr (Mk |y ) Pr (Mk ) f (y |Mk )
| {z } | {z } | {z }
Turning the
Bayesian
posterior odds prior odds Bayes factor
crank
Prior predictive
Posterior
Posterior The Bayes factor can be viewed as the weighted likelihood ratio
predictive
Sequential Bayes of Mj to Mk .
Model
probability
Posterior odds
Bayes factor
Marginal
The main difficulty is the computation of the marginal
likelihood
likelihood or normalizing constant f (y |Mj ).
Example iii.
Multiple linear
regression

Real data
Therefore, the posterior model probability for model j can be
exercise obtained from
Example iv.
SV model 1 X Pr (Mk )
= Bkj .
Pr (Mj |y ) Pr (Mj )
Mk ∈M
 Bayes factor
Example i.
Sequential
Jeffreys (1961) recommends the use of the following rule of
learning thumb to decide between models j and k:
Example ii.
Normal- log10 Bjk Bjk Evidence against k
normal
0.0 to 0.5 1.0 to 3.2 Not worth more than a bare mention
Turning the
Bayesian 0.5 to 1.0 3.2 to 10 Substantial
crank 1.0 to 2.0 10 to 100 Strong
Prior predictive
Posterior >2 > 100 Decisive
Posterior
predictive
Sequential Bayes
Model
probability Kass and Raftery (1995) argue that “it can be useful to
Posterior odds
Bayes factor
consider twice the natural logarithm of the Bayes factor, which
Marginal
likelihood
is on the same scale as the familiar deviance and likelihood
Example iii.
ratio test statistics”. Their slight modification is:
Multiple linear
regression
2 loge Bjk Bjk Evidence against k
Real data
exercise
0.0 to 2.0 1.0 to 3.0 Not worth more than a bare mention
2.0 to 6.0 3.0 to 20 Substantial
Example iv.
SV model 6.0 to 10.0 20 to 150 Strong
> 10 > 150 Decisive
 Marginal likelihood
Example i.
Sequential A basic ingredient for model assessment is given by the
learning
predictive density
Example ii. Z
Normal-
normal
f (y |M) = f (y |θ, M)p(θ|M)dθ ,
Turning the
Bayesian
crank which is the normalizing constant of the posterior distribution.
Prior predictive
Posterior
Posterior
predictive
Sequential Bayes
The predictive density can now be viewed as the likelihood of
Model
probability model M.
Posterior odds
Bayes factor
Marginal
likelihood It is sometimes referred to as predictive likelihood, because it is
Example iii.
Multiple linear obtained after marginalization of model parameters.
regression

Real data
exercise The predictive density can be written as the expectation of the
Example iv. likelihood with respect to the prior:
SV model

f (y ) = Ep [f (y |θ)].
 Example iii. Multiple linear
Example i.
Sequential
learning
regression
Example ii.
Normal-
The standard Bayesian approach to multiple linear regression is
normal

Turning the
Bayesian
yi = xi0 β + i
crank
Prior predictive
Posterior
for i = 1, . . . , n, xi a q-dimensional vector of regressors and
Posterior
predictive residuals i iid N(0, σ 2 ).
Sequential Bayes
Model
probability
Posterior odds
Bayes factor
Marginal
In matrix notation,
likelihood

Example iii.
Multiple linear (y |X , β, σ 2 ) ∼ N(X β, σ 2 In )
regression

Real data
exercise
where y = (y1 , . . . , yn ), X = (x1 , . . . , xn )0 is the (n × q), design
Example iv.
matrix and q = p + 1.
SV model
 Example iii. Maximum likelihood
Example i.
Sequential
learning
estimation
Example ii.
It is well known that
Normal-
normal β̂ = (X 0 X )−1 X 0 y
Turning the
Bayesian Se (y − X β̂)0 (y − X β̂)
crank σ̂ 2 = =
Prior predictive n−q n−q
Posterior
Posterior
predictive are the OLS estimates of β and σ 2 , respectively.
Sequential Bayes
Model
probability
Posterior odds
Bayes factor
The conditional and unconditional sampling distributions of β̂
Marginal
likelihood are
Example iii.
Multiple linear (β̂|σ 2 , y , X ) ∼ N(β, σ 2 (X 0 X )−1 )
regression

Real data (β̂|y , X ) ∼ tn−q (β, Se (X 0 X )−1 )

exercise

Example iv. respectively, with

SV model

(σ̂ 2 |σ 2 ) ∼ IG (n − q)/2, ((n − q)σ 2 /2 .

 Example iii. Conjugate prior
Example i.
Sequential
learning
The prior distribution of (β, σ 2 ) is NIG (b0 , B0 , n0 , S0 ), i.e.
Example ii.
Normal- β|σ 2 ∼ N(b0 , σ 2 B0 )
normal

Turning the
σ 2 ∼ IG (n0 /2, n0 S0 /2)
Bayesian
crank
Prior predictive
for known hyperparameters b0 , B0 , n0 and S0 .
Posterior
Posterior
predictive
Sequential Bayes
Model
probability
For clarification, when σ 2 ∼ IG (a, b), if follows that
Posterior odds
Bayes factor
 
b
p(σ 2 ) ∝ (σ 2 )−(a+1) exp − 2
Marginal
likelihood

Example iii. σ
Multiple linear
regression
with
Real data
exercise
b b2
Example iv. E (σ 2 ) = and V (σ 2 ) =
SV model a−1 (a − 1)2 (a − 2)
 Example iii. Conditionals
Example i.
Sequential It is easy to show that
learning

Example ii. (β|σ 2 , y , X ) ∼ N(b1 , σ 2 B1 )

Normal-
normal

Turning the where

Bayesian
crank
Prior predictive
B1−1 = B0−1 + X 0 X
B1−1 b1 = B0−1 b0 + X 0 y .
Posterior
Posterior
predictive
Sequential Bayes
Model
probability
Posterior odds
Bayes factor
Marginal
likelihood
It is also easy to show that
Example iii.
Multiple linear (σ 2 |β, y , X ) ∼ IG (n1 /2, n1 S11 (β)/2)
regression

Real data
exercise
where
Example iv.
SV model n1 = n0 + n
n1 S11 (β) = n0 S0 + (y − X β)0 (y − X β).
 Example iii. Marginals
Example i.
Sequential
learning

Example ii.
Normal-
normal It can be shown that
Turning the
Bayesian (σ 2 |y , X ) ∼ IG (n1 /2, n1 S1 /2)
crank
Prior predictive
Posterior where
Posterior
predictive
Sequential Bayes
Model
probability
n1 S1 = n0 S0 + (y − Xb1 )0 y + (b0 − b1 )0 B0−1 b0 .
Posterior odds
Bayes factor
Marginal
likelihood

Example iii.
Multiple linear Consequently,
regression
(β|y , X ) ∼ tn1 (b1 , S1 B1 ).
Real data
exercise

Example iv.
SV model
 Example iii. MLE versus Bayes
Example i.
Sequential Distributions of the estimators β̂ and σ̂ 2
learning
(σ̂ 2 |σ 2 , y , X ) ∼ IG (n − q)/2, ((n − q)σ 2 /2


Example ii.
Normal-
normal (β̂|β, y , X ) ∼ tn−q (β, Se (X 0 X )−1 ).
Turning the
Bayesian
crank
Prior predictive
Marginal posterior distributions of β and σ 2
Posterior
Posterior
predictive
(σ 2 |y , X ) ∼ IG (n1 /2, n1 S1 /2)
Sequential Bayes
Model (β|y , X ) ∼ tn1 (b1 , S1 B1 ).
probability
Posterior odds
Bayes factor

Vague prior: When B0−1 = 0, n0 = −q and S0 = 0

Marginal
likelihood

Example iii.
Multiple linear b1 = β̂
regression

Real data B1 = (X 0 X )−1

exercise
n1 = n − q
Example iv.
SV model
n1 S1 = (y − X β̂)0 y = (y − X β̂)0 (y − X β̂) = (n − q)σ̂ 2
S1 B1 = σ̂ 2 (X 0 X )−1 .
 Example iii. Predictive
Example i.
Sequential
learning The predictive density can be obtained by
Example ii.
Normal- Z
normal
p(y |X ) = p(y |X , β, σ 2 )p(β|σ 2 )p(σ 2 )dβdσ 2
Turning the
Bayesian
crank
Prior predictive or (via Bayes’ theorem) by
Posterior
Posterior

p(y |X , β, σ 2 )p(β|σ 2 )p(σ 2 )

predictive
Sequential Bayes
Model p(y |X ) =
probability
Posterior odds
p(β|σ 2 , y , X )p(σ 2 |y , X )
Bayes factor
Marginal
likelihood which is valid for all (β, σ 2 ).
Example iii.
Multiple linear
regression
Closed form solution for the multiple normal linear regression:
Real data
exercise (y |X ) ∼ tn0 (Xb0 , S0 (In + XB0 X 0 )).
Example iv.
SV model
 Real data exercise
Example i.
Sequential
learning

Example ii.
Normal-
normal
To better understand the differential role of the prior in
Turning the estimation and model comparison, consider the following simple
Bayesian
crank linear regression application, illustrated using a sample of
Prior predictive
Posterior
n = 1, 217 observations from the National Longitudinal Survey
Posterior
predictive of Youth (NLSY):
Sequential Bayes
iid
Model
probability • M0 : yi = β0 + β1 xi + i , i ∼ N(0, σ 2 ).
Posterior odds
iid
Bayes factor
Marginal • M1 : yi = β0 + i , i ∼ N(0, σ 2 ).
likelihood

Example iii.
Multiple linear yi : log hourly wage received by individual i.
regression

Real data xi : education in years of schooling completed by individual i.

exercise

Example iv.
SV model
 Years of schooling completed
Example i.
Sequential
learning

Example ii.
Normal-

40
normal

Turning the
Bayesian
crank
Prior predictive
30

Posterior
Posterior
predictive
Sequential Bayes
Percentage

Model
probability
20

Posterior odds
Bayes factor
Marginal
likelihood

Example iii.
Multiple linear
10

regression

Real data
exercise

Example iv.
0

SV model
6 7 8 9 10 11 12 13 14 15 16 17 18 19 20

Years of schooling completed

 Log hourly wage
Example i.
Sequential
learning

Example ii.
Normal-
normal

0.8
Turning the
Bayesian
crank
Prior predictive
0.6

Posterior
Posterior
predictive
Sequential Bayes
Density

Model
probability
Posterior odds
0.4

Bayes factor
Marginal
likelihood

Example iii.
Multiple linear
0.2

regression

Real data
exercise

Example iv.
0.0

SV model
0 1 2 3 4

Log hourly wage

 MLE regression
Example i.
Sequential
learning

Example ii. ●
●
●

Normal- ●
●

4
●
normal ●

● ● ●
●
● ●
Turning the ●
●
● ●
●
●
●
●
●
● ●
●
●

● ● ●
Bayesian ●

●
●
●
●
● ●
●
● ●
● ●
crank ●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ● ●
● ●
● ●
● ● ● ● ●
● ● ●
● ●
Prior predictive 3 ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
● ●
●
● ●
● ●
● ●
● ●
● ●
●
● ● ●
● ●
●
● ●
● ●
●
●
● ● ● ●
● ● ●
Posterior ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
● ● ●
● ●
● ●
● ●
●
● ●
●
● ● ●
● ● ●
● ●
● ●
● ●
Posterior ● ●
● ● ●
● ●
● ●
●
● ●
●
● ● ● ●
Log hourly wage

● ● ●
● ●
● ● ● ●
● ●
● ● ●
● ● ●
● ● ● ●
● ● ● ●
● ●
● ● ●
● ● ●
predictive ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
● ●
● ● ●
● ● ● ● ●
● ● ●
● ●
● ● ● ● ● ●
● ●
Sequential Bayes ●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
● ● ● ●
● ●
● ● ● ● ●
●
● ● ● ●
● ● ● ● ●
● ●
Model ●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
● ●
2

● ●
● ●
● ● ● ●
probability ● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ● ● ●
● ● ● ●
● ● ●
● ● ● ●
●
Posterior odds ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
● ● ● ● ● ●
●
● ● ●
● ● ●
Bayes factor ●
●
● ● ●
●
●
●
●
●
●
●
●
●
● ● ●
● ● ●
● ●
●
● ●
Marginal ●
●
●
●
●
●
●
●

likelihood ●
●
● ●
●
● ●
●
●
● ● ● ●
1

● ●
Example iii. ●
●
●

●
●
● ●
Multiple linear ●
● ● ●
● ●

●
regression ●
●
●
●

●
●

Real data ●

exercise
0

6 8 10 12 14 16 18 20

Example iv. Years of schooling completed

SV model

β̂ = (1.17766, 0.09101)0 and σ̂ 2 = 0.2668455.

Example i.
Sequential
learning Recall the conjugate prior for (β, σ 2 ) is
Example ii.
Normal-
normal β|σ 2 ∼ N(b0 , σ 2 B0 ) and σ 2 ∼ IG (n0 /2, n0 S0 /2).
Turning the
Bayesian
crank
Prior predictive
Posterior
Posterior Let us assume that b0 = 0, n0 = 6 and S0 = 0.1333.
predictive
Sequential Bayes
Model
probability
Posterior odds
Bayes factor
Let us consider two different prior variance for β:
Marginal
likelihood • Prior I: B0 = 1.0 × 101 I2 ,
Example iii.
Multiple linear • Prior II: B0 = 1.0 × 10100 I2 .
regression

Real data
exercise

Example iv.
SV model
 Posterior summary for the complete model M0
Example i.
Sequential
Parameter Prior I Prior II
learning Post. Mean Post Std. Post Mean Post Std.
Example ii.
Normal-
β0 1.17439 (0.08626) 1.17439 (0.08637)
normal
β1 0.09125 (0.00654) 0.09125 (0.00655)
Turning the
Bayesian σ2 0.26587 (0.01073) 0.26587 (0.01073)
crank
Prior predictive Posterior summary for the restricted model M1
Posterior
Posterior β0 2.35963 (0.01592) 2.35963 (0.01591)
predictive
Sequential Bayes σ2 0.30815 (0.01244) 0.30815 (0.01244)
Model
probability
Posterior odds Log Bayes factor of M0 versus M1
Bayes factor
Marginal log B01 84.7294 -29.8886
likelihood

Example iii.
Multiple linear
regression log B01 (Prior I) = log p(y |X , M0 , Prior I) − log p(y |X , M1 , Prior I)
Real data = −2458.713 − (−2543.442) = 84.7294
exercise

Example iv. log B01 (Prior II) = log p(y |X , M0 , Prior II) − log p(y |X , M1 , Prior II)
SV model
= −2686.406 − (−2656.517) = −29.8886
 Example iv. Stochastic volatility
Example i.
Sequential One of the most used models in financial econometrics is the
learning

Example ii.
diffusive stochastic volatility model, where log-returns are
Normal-
normal
normally distributed
Turning the
Bayesian yt |θt , H ∼ N(0; e θt )
crank
Prior predictive
Posterior
Posterior
with heteroscedasticity modeled as
predictive
Sequential Bayes
Model
probability
θt |θt−1 , γ, H ∼ N(α + βθt−1 , σ 2 )
Posterior odds
Bayes factor
Marginal
likelihood
for t = 1, . . . , T and γ = (α, β, σ 2 ), known for now.
Example iii.
Multiple linear
regression
The model is completed with
Real data
exercise

Example iv. θ0 |γ, H ∼ N(m0 , C0 )

SV model

for known hyperparameters (m0 , C0 ).

 Example iv. Posterior distribution
Example i.
Sequential For θ = (θ1 , . . . , θT )0 , it follows that
learning

Example ii. T  
Normal- Y
−θt /2 1
normal p(θ|y , H) ∝ e exp − yt2 e −θt
2
Turning the t=1
Bayesian
crank T  
Y 1
Prior predictive
Posterior × exp − 2 (θt − α − βθt−1 )2
Posterior 2σ
predictive t=1
Sequential Bayes  
Model 1
probability × exp − (θ0 − m0 )2
Posterior odds
Bayes factor
2C0
Marginal
likelihood

Example iii. Unfortunately, closed form solutions are rare!

Multiple linear
regression • How to compute E (θ43 |y , H) or V (θ11 |y , H)?
Real data
exercise • How to obtain a 95% credible region for (θ35 , θ36 |y , H)?
Example iv.
SV model
• How to sample from p(θ|y , H)?
• How to compute p(y |H) or p(yT +1 , . . . , yT +k |y , H)?