Introduction to Risk Management and Business

Intelligence
Topic 9. Bayesian Statistics

Xuhu Wan

Department of ISOM, HKUST

October 25, 2020

Xuhu Wan Topic 9. Bayesian Statistics Department of ISOM, HKUST

Outline

1 Motivation

2 Bayesian Theorem

3 Markov Chain Monte Carlo Approximation, MCMC

4 Explanation of MCMC with Plain Codes

5 Impletmentation of MCMC with PyMC3

Xuhu Wan Topic 9. Bayesian Statistics Department of ISOM, HKUST

Why Bayesian?

Bayesian approach v.s. Frequentist approach

. For frequentists, probabilities are fundamentally related to
frequencies of events and the basic parameters of probability
distribution is constant.
For Bayesians, the concept of probability is extended to cover degrees
of certainty about statements and probabilities are fundamentally
related to our own knowledge about an event. The paramenters are
sampled from the distribution.

Bayesian is much more meaningful in practice. Starting with a belief (a

distribution, we call it prior) about parameters, we will update our belief
(posterior) about parameters with the arrival of data. We will continue to
update with new batch of data.
For example, you work in a small lending company, and you can update
your model with new data of lenders, without updating the model with all
historical data, because the new batch of cutomers may be different.
Xuhu Wan Topic 9. Bayesian Statistics Department of ISOM, HKUST
Outline

1 Motivation

2 Bayesian Theorem

3 Markov Chain Monte Carlo Approximation, MCMC

4 Explanation of MCMC with Plain Codes

5 Impletmentation of MCMC with PyMC3

Xuhu Wan Topic 9. Bayesian Statistics Department of ISOM, HKUST

Bayesian Theorem

P(X |θ)P(θ)
P(θ|X ) =
P(X )
P(θ) is prior probability,P(θ|X ) is posterior probability
P(X ) is the model evidence or marginal likelihood .
P(X |θ) is the likelihood of evidence.

Example (An example with discrete prior)

Suppose the parameter θ is the chance of a coin landing head, the prior
probability is P(θ = 0.5) = 0.2 and P(θ = 0.1) = 0.8. If X = head, what
is the chance the coin is fair?

Solution
0.5 ∗ 0.2
P(θ = 0.5|head) = = 0.56
0.1 ∗ 0.8 + 0.5 ∗ 0.2
Xuhu Wan Topic 9. Bayesian Statistics Department of ISOM, HKUST
Continuous Prior

In the example, the prior is binary, either 0.5 or 0.1

the prior could be wither discrete or continuous.
Continuous prior and high dimensional θ make the computation of
posterior probability difficult because of the marginal likelihood.
Z
P(X ) = P(X |θ)P(θ)dθ

Only with a few kind of priors, the posterior is easy to compute, i.e.
conjugate priors.

Xuhu Wan Topic 9. Bayesian Statistics Department of ISOM, HKUST

Bayesian Updating With Conjugate Priors

Definition
In Bayesian probability theory, if the posterior distributions P(θ|x) are in
the same probability distribution family as the prior probability distribution
P(θ), the prior and posterior are then called conjugate distributions, and
the prior is called a conjugate prior for the likelihood function P(x|θ).

Beta distribution is a conjugate prior for Bernoulli trials. Suppose we

have total n trials and k success. If Beta(α, β) is the prior, then
posterior distribution is Beta(α + k, β + (n − k)). So conveneient!
Besides, for Beta(α, β), the mean and variance can be computed
α
easily µ = α+β , σ 2 = (α+β)2αβ
(α+β+1)
.
Given prior Beta(1, 10), what is the mean if there are 8 successes out
of 10 trials?

Xuhu Wan Topic 9. Bayesian Statistics Department of ISOM, HKUST

Beta Prior
Definition
The beta distribution is a family of continuous probability distributions
defined on the interval [0, 1] parameterized by two positive shape
parameters, denoted by α and β, that control the shape of the distribution.
The beta distribution has been applied to model the behavior of random
variables limited to intervals of finite length in a wide variety of disciplines,
for example, the probability

Xuhu Wan Topic 9. Bayesian Statistics Department of ISOM, HKUST

Coin Toss with Beta Prior (1)

Problem
The probability of head is θ, the prior is Beta(1, 1). The likelihood
function is Bernouli.
If toss the coin 10 times. We simulate the data
X = [Link](p=0.5, size=10)
it generates a data with 10 trials: [0 0 1 1 1 1 0 0 1 0] There are totall 5
heads and 5 tails, then the posterior distribution is Beta(1 + 5, 1 + 5).

Xuhu Wan Topic 9. Bayesian Statistics Department of ISOM, HKUST

Coin Toss with Beta Prior (2)

Problem
Find posterior distribution with uniform prior and different numbers of
trials.

trials = [0, 1, 3, 5, 10, 25, 50, 100, 500]

X = [Link](p=0.5, size=trials[-1])
theta = [Link](0, 1, 100)
alpha = beta = 1

for i, n in enumerate(trials):
data =X[:n]
heads = sum(data)
tails = n - heads
update = [Link](theta, alpha + heads ,beta + tails)

Xuhu Wan Topic 9. Bayesian Statistics Department of ISOM, HKUST

Coin Toss with Uninformative Prior: Beta(1, 1)
trial=0, success=0 2 trial=1, success=0 trial=3, success=1
1.04
1.5
1.5
1.02
1
1 1

0.98 0.5
0.5

0.96
0 0
0 0.2 0.4 0.6 0.8 1 0 0.2 0.4 0.6 0.8 1 0 0.2 0.4 0.6 0.8 1
2.5 3
trial=5, success=1 trial=10, success=3 4 trial=25, success=10

2
2 3
1.5
2
1
1
0.5 1

0 0 0
0 0.2 0.4 0.6 0.8 1 0 0.2 0.4 0.6 0.8 1 0 0.2 0.4 0.6 0.8 1

trial=50, success=25 8 trial=100, success=52 trial=500, success=275

15
6
4
10
4

2
2 5

0 0 0
0 0.2 0.4 0.6 0.8 1 0 0.2 0.4 0.6 0.8 1 0 0.2 0.4 0.6 0.8 1

Figure: Posterior distribtions with uniformative prior and different numbers of

trials.

Remark
The distribution of success rate converges as the number of trials increases.
Xuhu Wan Topic 9. Bayesian Statistics Department of ISOM, HKUST
Coin Toss with Informative Prior: Beta(5, 1)
5 trial=0, success=0 6 trial=1, success=1 trial=3, success=2
3
4
4 2
3

2
2 1
1

0 0 0
0 0.2 0.4 0.6 0.8 1 0 0.2 0.4 0.6 0.8 1 0 0.2 0.4 0.6 0.8 1

trial=5, success=2 3 trial=10, success=4 trial=25, success=10

4

2
3
2

2
1
1
1

0 0 0
0 0.2 0.4 0.6 0.8 1 0 0.2 0.4 0.6 0.8 1 0 0.2 0.4 0.6 0.8 1

6 trial=50, success=20 8 trial=100, success=43 trial=500, success=248

15
6
4
10
4

2
5
2

0 0 0
0 0.2 0.4 0.6 0.8 1 0 0.2 0.4 0.6 0.8 1 0 0.2 0.4 0.6 0.8 1

Figure: Posterior distribtions with informative prior and different numbers of trials.

Remark
Even if the priors are different, the posterior distributions converge.
Xuhu Wan Topic 9. Bayesian Statistics Department of ISOM, HKUST
Updating the Posterior Distribution of Stock Return (1)

Problem
We will update our belief about the probability that the daily change of
S&P500 is positive.

sp500["return"]=sp500["Close"].pct_change()
sp500["trial"]=[1 if x>0 else 0 for x in sp500["return"]]
Date Open High Low Close Volume return trial
2015-10-23 187.854196 188.488683 186.993106 188.089859 144442300 NaN 0
2015-10-26 187.899536 187.962978 187.228785 187.627609 69033000 -0.002458 0
2015-10-27 186.902456 187.627590 186.530823 187.265030 77905800 -0.001932 0
2015-10-28 187.627592 189.422287 186.911532 189.395096 135906700 0.011375 1
2015-10-29 188.851241 189.685140 188.724344 189.286316 90525500 -0.000574 0

Xuhu Wan Topic 9. Bayesian Statistics Department of ISOM, HKUST

Updating the Posterior Distribution of Stock Return (2)

6
Days=0, up=0 Days=50, up=20
1.04

1.02 4

2
0.98

0.96
0
0 0.2 0.4 0.6 0.8 1 0 0.2 0.4 0.6 0.8 1

8 Days=100, up=48 Days=500, up=270

15
6

10
4

2 5

0 0
0 0.2 0.4 0.6 0.8 1 0 0.2 0.4 0.6 0.8 1

Xuhu Wan Topic 9. Bayesian Statistics Department of ISOM, HKUST

 Live Update of the Posterior Distribution of Stock Return
(1)

Problem
We will update the posterior probability in a rolling way and take the
mode for each day with Beta(3, 2).

sp500["Mode"]=0
alpha=3
beta=2
for index in [Link]:
if [Link][index,"trial"]==1:
alpha=alpha+1
update = [Link](theta, alpha , beta )
else:
beta=beta+1
update = [Link](theta, alpha , beta )
[Link][index,"Mode"]=theta[[Link]()]

Xuhu Wan Topic 9. Bayesian Statistics Department of ISOM, HKUST

 Live Update of the Posterior Distribution of Stock Return
(2)

Mode Probability Updated 350

0.5
300

250
0.4
200
CLose Price of sp500

0 200 400 600 800 1,000 1,200

Xuhu Wan Topic 9. Bayesian Statistics Department of ISOM, HKUST

Outline

1 Motivation

2 Bayesian Theorem

3 Markov Chain Monte Carlo Approximation, MCMC

4 Explanation of MCMC with Plain Codes

5 Impletmentation of MCMC with PyMC3

Xuhu Wan Topic 9. Bayesian Statistics Department of ISOM, HKUST

Approximation of Posterior Probability

Challenge of Bayesian Inference

It is hard to compute
Z
P(X ) = P(X |θ)P(θ)dθ

If we can’t solve it directly, we can approximate it. We can take samples

indirectly from posterior distribution and generate the posterior
distribution by Monte Carlo simulation. More specifically, it is called
Markov Chain Monte Carlo Approximation (MCMC).

Xuhu Wan Topic 9. Bayesian Statistics Department of ISOM, HKUST

Nontechnical Explanation of MCMC With Beta Prior (1)

Problem
Consider Bernouli trials
xi ∼ Bernoulli(θ),
then the data X = {x1 , x2 , . . . , xn } and the likelihood function is

P(X |θ) = P(x1 |θ) × P(x2 |θ) . . . × P(xn |θ)

We assume the prior is Beta:

θ ∼ Beta(α, β)

We can prove the posterior

P(θ|X ) ∼ Beta(α + k, β + n − k), where k is number of successes.

Xuhu Wan Topic 9. Bayesian Statistics Department of ISOM, HKUST

Nontechnical Explanation of MCMC with Beta Prior (2)
Choose a initial value for θ,say
θt = 0.2, where t=0
Then choose next θt+1 by jummping to a new position (“Markove
chain”), centering around θt with certain standard deviation (σS ).
Metropolis sampler takes a sample from normal(θt , σS2 )
θt+1 ∼ Normal(θt , σS2 ),
θt+1 has to be in [0, 1]
Decide whether to keep θt+1 : keep it if
P(θt+1 |X )
> [Link](), rand() is Uniform in [0,1]
P(θt |X )
that is
P(X |θt+1 )P(θt )
> [Link]()
P(X |θt+1 )P(θt )

Xuhu Wan Topic 9. Bayesian Statistics Department of ISOM, HKUST

Nontechnical Explanation of MCMC with Beta Prior(3)

If we accept θt+1 , we will continue, and try to find θt+2 , θt+3 , . . .. If

not, we will find another θt+1 .
We have a list of θ: Θ = {θ0 , θ1 , θ2 , θ3 , . . .}, which is empirical
posterior distribution of θ
Θ approximates the posterior distribution.

How to validate MCMC?

We will apply MCMC to find many possible sample values of θ. We will
compare the histogram of all these θ with the density fucntion of
Beta(α + k, β + n − k. If very close, it demonstrates the MCMC algorithm
can get samples indirectly from posterior distribution.

Xuhu Wan Topic 9. Bayesian Statistics Department of ISOM, HKUST

Outline

1 Motivation

2 Bayesian Theorem

3 Markov Chain Monte Carlo Approximation, MCMC

4 Explanation of MCMC with Plain Codes

5 Impletmentation of MCMC with PyMC3

Xuhu Wan Topic 9. Bayesian Statistics Department of ISOM, HKUST

Evidence and Prior

Generate a evidence with real probability 0.2

X=[Link](p=0.2, size=200)
The prior is Beta(5, 1)
The real posterior with data X is
Beta(5 + no. of success, 1 + no. of fail)
Prior Distribution Posterior Distribution

4 6

4
2
2

0 0
0 0.5 1 0 0.5 1

Xuhu Wan Topic 9. Bayesian Statistics Department of ISOM, HKUST

MCMC Approximation of Posterior: Step by Step (1)

Generate θ0
posterior_sample=[]
sigma_S=3
theta0=0.4
Generate θ1
theta1=[Link](theta0,sigma,size=1)[0]
while theta1>1 or theta1<0:
theta1=[Link](theta0,sigma,size=1)[0]

Xuhu Wan Topic 9. Bayesian Statistics Department of ISOM, HKUST

MCMC Approximation of Posterior: Step by Step (2)

whether to reject or accept:

likelihood0=[Link]([Link](), 200, theta0)
likelihood1=[Link]([Link](),200, theta1)
prior0=[Link](theta0,5,1)
prior1=[Link](theta1,5,1)
ratio=(likelihood1*prior1)/(likelihood0*prior0)
acceptance=min(ratio,1)
u=[Link](0,1,size=1)[0]
if u<acceptance:
posterior_sample.append(theta0)
theta0=theta1

Xuhu Wan Topic 9. Bayesian Statistics Department of ISOM, HKUST

MCMC Approximation of Posterior: Step by Step (3)

Visualization
We run these steps recursively, we get θ0 , θ1 , . . . , we plot the histograms
as follows with density curve of posterior Beta.

Empirical Distribution of Posterior Distribution of Posterior

15 15

10 10

5 5

0 0
0.1 0.2 0.3 0.4 0.5 0.1 0.2 0.3 0.4 0.5
Xuhu Wan Topic 9. Bayesian Statistics Department of ISOM, HKUST
MCMC Approximation of Posterior with Small Data

Visualization
If our data is small ,len(X ) = 50, the histograms of posterior are as follows
with density curve of Beta, posterior.

Empirical Distribution of Posterior Distribution of Posterior

15 15

10 10

5 5

0 0
0.1 0.2 0.3 0.4 0.5 0.1 0.2 0.3 0.4 0.5
Xuhu Wan Topic 9. Bayesian Statistics Department of ISOM, HKUST
Outline

1 Motivation

2 Bayesian Theorem

3 Markov Chain Monte Carlo Approximation, MCMC

4 Explanation of MCMC with Plain Codes

5 Impletmentation of MCMC with PyMC3

Xuhu Wan Topic 9. Bayesian Statistics Department of ISOM, HKUST

Brief Introduction of PyMC3

The BEST python bayesian library!!!

State-of-the-art
The speed is very fast (GPU computation).

How to install
In the cell of notebook, type “!pip install pymc3”. Do not install it in
virtual machine.

Xuhu Wan Topic 9. Bayesian Statistics Department of ISOM, HKUST

How to Simulate the Posterior Distribution with PyMC3(1)

with [Link]():
theta0=[Link]("theta0",alpha=5,beta=1) # prior
X_Obs=[Link]("X",p=theta0,observed=X)# posterior
step=[Link]()
post_sample=[Link](10000,step)

Xuhu Wan Topic 9. Bayesian Statistics Department of ISOM, HKUST

How to Simulate the Posterior Distribution with PyMC3(2)
Visualization
len(X ) = 50, the histograms of posterior are as follows with density curve
of Beta posterior , implemented by PyMC3.

Empirical Distribution of Posterior Distribution of Posterior

15 15

10 10

5 5

0 0
0 0.2 0.4 0 0.2 0.4
Xuhu Wan Topic 9. Bayesian Statistics Department of ISOM, HKUST