Bayesian Inference and Deep Learning

Dhruv Patel
Department of Mechanical Engineering
Stanford University

Email: dvpatel@[Link]

Wokshop on Scientific Computing, Modeling, and Deep Learning

January 6, 2024
Artificial Intelligence, Machine Learning and Deep Learning

What is machine learning?

▶ Collect a set of data

S = {xi : 1 ≤ i ≤ n} or S = {(xi , yi ) : 1 ≤ i ≤ n}.

▶ Train an algorithm to discover patterns or relation between samples.

▶ Use algorithm to make future prediction on new data.

Figure: ML Methods

D. Patel Bayesian Inference and Deep Learning 1

Outline

1. Bayesian Deep Learning

2. Generative adversarial networks (GANs)

3. Deep learning in inverse problems

3.1 Bayesian formulation for inverse problems
3.2 GANs as prior
3.3 GANs as posterior

D. Patel Bayesian Inference and Deep Learning 2

Outline

1. Bayesian Deep Learning

2. Generative adversarial networks (GANs)

3. Deep learning in inverse problems

3.1 Bayesian formulation for inverse problems
3.2 GANs as prior
3.3 GANs as posterior

D. Patel Bayesian Inference and Deep Learning 3

Regression problem

▶ Given {(x1 , y1 ), (x2 , y2 ), · · · , (xn , yn )}

▶ Want ŷ = f (x̂; W )
▶ W can represent any set of parameters (e.g. linear regression, neural
networks)
▶ Best W is the one that minimizes the loss function ||ŷ − f (x̂; W )||22 (i.e. makes
ŷ close to “true” y)

Figure: Linear Regression

D. Patel Bayesian Inference and Deep Learning 4

Learning/Training by Maximum Likelihood Estimation

Ŵ = arg maxp(data|W )
W

Figure: Max Likelihood training

D. Patel Bayesian Inference and Deep Learning 5

Bayesian approach

▶ We make use of prior information

▶ W is unlikely to be 1 million or negative 1 billion.

p(data|W )p(W )
p(W |data) =
p(data)

▶ MAP (maximum a posteriori) estimate of Bayesian approach is same as

optimal solution of deterministic approach with regularization.

Ŵ = arg maxp(y |x, W )p(W )

W

D. Patel Bayesian Inference and Deep Learning 6

Distribution of predictions

Figure: Posterior predictive distribution

D. Patel Bayesian Inference and Deep Learning 7

Comparative Summary

Figure: Comparison

D. Patel Bayesian Inference and Deep Learning 8

Outline

1. Bayesian Deep Learning

2. Generative adversarial networks (GANs)

3. Deep learning in inverse problems

3.1 Bayesian formulation for inverse problems
3.2 GANs as prior
3.3 GANs as posterior

D. Patel Bayesian Inference and Deep Learning 9

Generating samples from a probability distribution

Given: A set S = {xi : xi ∈ ΩX ⊂ RNX , 1 ≤ i ≤ n} of samples from some PX .

Goal: Discover PX from S and generate new samples .

D. Patel Bayesian Inference and Deep Learning 10

Generating samples from a probability distribution

Given: A set S = {xi : xi ∈ ΩX ⊂ RNX , 1 ≤ i ≤ n} of samples from some PX .

Goal: Discover PX from S and generate new samples .

x i ∈ R2 :

x ∼ U ([0, 1]2 ) x ∼ N(µ, Σ)

D. Patel Bayesian Inference and Deep Learning 10

Generating samples from a probability distribution

Given: A set S = {xi : xi ∈ ΩX ⊂ RNX , 1 ≤ i ≤ n} of samples from some PX .

Goal: Discover PX from S and generate new samples .

x i ∈ R2 :

x ∼ U ([0, 1]2 ) x ∼ N(µ, Σ)

xi ∈ RN×N :
(images)

Binary phase microstructure Handwritten MNIST digits Shepp-Logan phantom

Representing this data in the form of a prior is hard!

D. Patel Bayesian Inference and Deep Learning 10
Generating samples from a probability distribution

Given: A set S = {xi : xi ∈ ΩX ⊂ RNX , 1 ≤ i ≤ n} of samples from some PX .

Goal: Discover PX from S and generate new samples .

x i ∈ R2 :
Data-driven
generative algorithms

x ∼ U ([0, 1]2 ) x ∼ N(µ, Σ)

xi ∈ RN×N :
(images)

Binary phase microstructure Handwritten MNIST digits Shepp-Logan phantom

Representing this data in the form of a prior is hard!

D. Patel Bayesian Inference and Deep Learning 10
Generative adversarial network (GAN)

Designed by Goodfellow et al. (2014) to learn and sample from a target PX .

Two networks with some suitable architectures.

D. Patel Bayesian Inference and Deep Learning 11

Generative adversarial network (GAN)

Designed by Goodfellow et al. (2014) to learn and sample from a target PX .

Two networks with some suitable architectures.

Generator network g(.; θ):
▶ Generates fake samples x̃
▶ g : ΩZ → ΩX .
▶ Latent variable z ∈ ΩZ ⊂ RNz .
▶ z ∼ PZ simple distribution, e.g. Gaussian.
▶ Nz ≪ Nx .

D. Patel Bayesian Inference and Deep Learning 11

Generative adversarial network (GAN)

Designed by Goodfellow et al. (2014) to learn and sample from a target PX .

Two networks with some suitable architectures.

Generator network g(.; θ): Critic network d(.; ϕ):
▶ Generates fake samples x̃ ▶ Distinguishes fake samples from real
▶ g : ΩZ → ΩX . ▶ d : ΩX → R.
▶ Latent variable z ∈ ΩZ ⊂ RNz . ▶ x ∼ PX .
▶ z ∼ PZ simple distribution, e.g. Gaussian. ▶ d(x) large for x ∼ PX , small otherwise.
▶ Nz ≪ Nx .

D. Patel Bayesian Inference and Deep Learning 11

Generative adversarial network (GAN)

Designed by Goodfellow et al. (2014) to learn and sample from a target PX .

Two networks with some suitable architectures.

Generator network g(.; θ): Critic network d(.; ϕ):
▶ Generates fake samples x̃ ▶ Distinguishes fake samples from real
▶ g : ΩZ → ΩX . ▶ d : ΩX → R.
▶ Latent variable z ∈ ΩZ ⊂ RNz . ▶ x ∼ PX .
▶ z ∼ PZ simple distribution, e.g. Gaussian. ▶ d(x) large for x ∼ PX , small otherwise.
▶ Nz ≪ Nx .

For a metric M on P(ΩX ), define the loss

Π(g, d) := Π(θ, ϕ) = M(PX , g# PZ ).
Solve the MinMax problem
∗ ∗
(g , d ) = arg min arg max Π(g, d) −→ Adversarial Training
g d

D. Patel Bayesian Inference and Deep Learning 11

Wasserstein GAN

Proposed by Arjovsky et al. (2017), using the Wasserstein-1 metric

W1 (P1 , P2 ) = inf E [∥x1 − x2 ∥]
γ∈J(P1 ,P2 ) (x1 ,x2 )∼γ

Using the Kantorovich-Rubinstein dual characterization, we have

 
W1 (P1 , P2 ) = sup E [f (x)] − E [f (x)]
∥f ∥Lip ≤1 x∼P1 x∼P2

D. Patel Bayesian Inference and Deep Learning 12

Wasserstein GAN

Proposed by Arjovsky et al. (2017), using the Wasserstein-1 metric

W1 (P1 , P2 ) = inf E [∥x1 − x2 ∥]
γ∈J(P1 ,P2 ) (x1 ,x2 )∼γ

Using the Kantorovich-Rubinstein dual characterization, we have

 
W1 (P1 , P2 ) = sup E [f (x)] − E [f (x)]
∥f ∥Lip ≤1 x∼P1 x∼P2

Set the loss function as

Π(g, d) = E [d(x)] − E [d(g(z))]
x∼PX z∼PZ

Under the constraint ∥d∥Lip ≤ 1, find

d ∗ (g) = arg max Π(g, d) = W1 (PX , g# PZ )
d

D. Patel Bayesian Inference and Deep Learning 12

Wasserstein GAN

Proposed by Arjovsky et al. (2017), using the Wasserstein-1 metric

W1 (P1 , P2 ) = inf E [∥x1 − x2 ∥]
γ∈J(P1 ,P2 ) (x1 ,x2 )∼γ

Using the Kantorovich-Rubinstein dual characterization, we have

 
W1 (P1 , P2 ) = sup E [f (x)] − E [f (x)]
∥f ∥Lip ≤1 x∼P1 x∼P2

Set the loss function as

Π(g, d) = E [d(x)] − E [d(g(z))]
x∼PX z∼PZ

Under the constraint ∥d∥Lip ≤ 1, find

d ∗ (g) = arg max Π(g, d) = W1 (PX , g# PZ )
d

Thus, for the optimal generator g ∗

g ∗ = arg minW1 (PX , g# PZ )
g

D. Patel Bayesian Inference and Deep Learning 12

Wasserstein GAN

Proposed by Arjovsky et al. (2017), using the Wasserstein-1 metric

W1 (P1 , P2 ) = inf E [∥x1 − x2 ∥]
γ∈J(P1 ,P2 ) (x1 ,x2 )∼γ

Using the Kantorovich-Rubinstein dual characterization, we have

 
W1 (P1 , P2 ) = sup E [f (x)] − E [f (x)]
∥f ∥Lip ≤1 x∼P1 x∼P2

Set the loss function as

Π(g, d) = E [d(x)] − E [d(g(z))]
x∼PX z∼PZ

Under the constraint ∥d∥Lip ≤ 1, find

d ∗ (g) = arg max Π(g, d) = W1 (PX , g# PZ )
d

Thus, for the optimal generator g ∗

g ∗ = arg minW1 (PX , g# PZ )
g

Finally, convergence in W1 implies weak convergence of measures

E [ℓ(x)] = E [ℓ(g ∗ (z))] , ∀ ℓ ∈ Cb (ΩX )
x∼PX z∼PZ

−→ moments converge.
D. Patel Bayesian Inference and Deep Learning 12
Wasserstein GAN

In practice, at the discrete level

▶ Generate/obtain the finite dataset S = {xi : xi ∈ ΩX , 1 ≤ i ≤ n}.

▶ Compute expectations using Monte Carlo

n n
1X 1 X
E [d(x)] ≈ d(xi ), E [d(g(z))] ≈ d(g(zi ))
x∼PX n z∼PZ n
i=1 i=1,zi ∼PZ

▶ Iterative solve the MinMax problem:

▶ Take N (typically N ≥ 4) optimization steps for d
▶ Take 1 optimization step for g

▶ Add a gradient penalty term (Gulrajani, 2017) to constraint d to be 1-Lipschitz

n
1X
λ (∥∇x d(xj )∥ − 1)2
n
j=1

D. Patel Bayesian Inference and Deep Learning 13

What a GAN can do

Results by Karras et al. (2018) from NVIDIA.

CELEBA-HQ dataset, Nz = 512, Nx = 1024 × 1024 × 3 = 3.14 × 106
−→ dimension reduction!

D. Patel Bayesian Inference and Deep Learning 14

Outline

1. Bayesian Deep Learning

2. Generative adversarial networks (GANs)

3. Deep learning in inverse problems

3.1 Bayesian formulation for inverse problems
3.2 GANs as prior
3.3 GANs as posterior

D. Patel Bayesian Inference and Deep Learning 15

Inverse problems

“We call two problems inverses of one another if the formulation of each involves
all or part of the solution of the other. Often, for historical reasons, one of the two
problems has been studied extensively for sometime, while the other is newer and
not so well understood. In such cases, the former is called the direct problem,
while the latter is called the inverse problem.”
– Joseph Keller, 1976

D. Patel Bayesian Inference and Deep Learning 16

Inverse problems

Consider the elliptic PDE for the (steady-state) temperature field u with
conductivity κ
−∇ · (κ∇u) = b(ξ), ∀ξ∈Ω
u(ξ) = 0, ∀ ξ ∈ ∂Ω
f f −1
Direct: Given {PDE, b, κ} −
→u Inverse: Given {PDE, b, u} −−→ κ

D. Patel Bayesian Inference and Deep Learning 17

Inverse problems

Consider the elliptic PDE for the (steady-state) temperature field u with
conductivity κ
−∇ · (κ∇u) = b(ξ), ∀ξ∈Ω
u(ξ) = 0, ∀ ξ ∈ ∂Ω
f f −1
Direct: Given {PDE, b, κ} −
→u Inverse: Given {PDE, b, u} −−→ κ

Challenges with inverse problems:

▶ Inverse map is not well posed.
▶ Noisy measurements from direct problem.
▶ Need to encode prior knowledge about inferred field.

Two approaches: regularization and Bayesian inference.

D. Patel Bayesian Inference and Deep Learning 17
Bayesian formulation

Uncertainty in inferred field critical for applications with high-stake decisions.

Example: Medical imaging to detect liver lesions

Example: Inferring basal sliding friction from surface ice velocity of Antarctic ice-shelf

D. Patel Bayesian Inference and Deep Learning 18

Bayesian formulation

Notations: We assume all quantities are discretized on some grid

▶ Parameter we wish to infer x ∈ ΩX ⊂ RNx (e.g. κ on Nx grid points).

▶ Measured response from direct problem y ∈ ΩY ⊂ RNy (e.g. u on Ny grid

points).

▶ Direct map f : ΩX → ΩY (e.g. discrete PDE solver). Sometimes,

y = f (x) + η → (additive noise)

where η is noise with distribution Pη .

▶ Assume that x and y are modelled using random variables X and Y .

D. Patel Bayesian Inference and Deep Learning 19

Bayesian formulation

Bayes theorem gives us:

PY |X (y|x)PX (x)
PX |Y (x|y) =
PY (y)
We apply this to the inverse problem: given a measurement y and prior information, infer x

D. Patel Bayesian Inference and Deep Learning 20

Bayesian formulation

Bayes theorem gives us:

PY |X (y|x)PX (x)
PX |Y (x|y) =
PY (y)
We apply this to the inverse problem: given a measurement y and prior information, infer x
prior
▶ PX (x) = PX (x): prior distribution, obtained from samples or other constraints.

D. Patel Bayesian Inference and Deep Learning 20

Bayesian formulation

Bayes theorem gives us:

PY |X (y|x)PX (x)
PX |Y (x|y) =
PY (y)
We apply this to the inverse problem: given a measurement y and prior information, infer x
prior
▶ PX (x) = PX (x): prior distribution, obtained from samples or other constraints.
▶ PY |X (y|x) = PYlike (y|x): the likelihood of observing the measurement y given x. For
additive noise
PYlike (y|x) = Pη (y − f (x)) → (embedding physics).

D. Patel Bayesian Inference and Deep Learning 20

Bayesian formulation

Bayes theorem gives us:

PY |X (y|x)PX (x)
PX |Y (x|y) =
PY (y)
We apply this to the inverse problem: given a measurement y and prior information, infer x
prior
▶ PX (x) = PX (x): prior distribution, obtained from samples or other constraints.
▶ PY |X (y|x) = PYlike (y|x): the likelihood of observing the measurement y given x. For
additive noise
PYlike (y|x) = Pη (y − f (x)) → (embedding physics).

▶ PY (y) = Q: the evidence/normalizing term

Z
prior
Q= Pη (y − f (x))PX (x)dx → (hard to compute when Nx ≫ 1).

D. Patel Bayesian Inference and Deep Learning 20

Bayesian formulation

Bayes theorem gives us:

PY |X (y|x)PX (x)
PX |Y (x|y) =
PY (y)
We apply this to the inverse problem: given a measurement y and prior information, infer x
prior
▶ PX (x) = PX (x): prior distribution, obtained from samples or other constraints.
▶ PY |X (y|x) = PYlike (y|x): the likelihood of observing the measurement y given x. For
additive noise
PYlike (y|x) = Pη (y − f (x)) → (embedding physics).

▶ PY (y) = Q: the evidence/normalizing term

Z
prior
Q= Pη (y − f (x))PX (x)dx → (hard to compute when Nx ≫ 1).

post
▶ PX |Y (x|y) = PX (x|y): the posterior distribution of x given y.

D. Patel Bayesian Inference and Deep Learning 20

Bayesian formulation

Bayes theorem gives us:

PY |X (y|x)PX (x)
PX |Y (x|y) =
PY (y)
We apply this to the inverse problem: given a measurement y and prior information, infer x
prior
▶ PX (x) = PX (x): prior distribution, obtained from samples or other constraints.
▶ PY |X (y|x) = PYlike (y|x): the likelihood of observing the measurement y given x. For
additive noise
PYlike (y|x) = Pη (y − f (x)) → (embedding physics).

▶ PY (y) = Q: the evidence/normalizing term

Z
prior
Q= Pη (y − f (x))PX (x)dx → (hard to compute when Nx ≫ 1).

post
▶ PX |Y (x|y) = PX (x|y): the posterior distribution of x given y.

Bayesian formulation:
prior
post Pη (y − f (x))PX (x) prior
PX (x|y) = ∝ Pη (y − f (x))PX (x)
Q

D. Patel Bayesian Inference and Deep Learning 20

Bayesian formulation

Posterior distribution

PXpost (x|y) ∝ Pη (y − f (x))PXprior (x)

Steps:
▶ Construct/obtain an explicit expression for PXprior .
▶ For a given y, use Markov Chain Monte Carlo (MCMC) to sample from PXpost .
▶ Generate a Markov chain whose stationary distribution is P post .
X
▶ Need to burn the first part of the chain.

One could also use variational inference, which would find the best approximation
of PXpost among a parametrised family.

D. Patel Bayesian Inference and Deep Learning 21

Bayesian formulation: challenges

▶ MCMC is prohibitively expensive when Nx is large.

▶ Characterization of priors for complex data.
2
 
Typical Gaussian prior PXprior (x) = √2πσ
1
exp − |x|
2σ 2

However, prior knowledge may be samples like:

Representing this data in the form of a prior is hard!

Resolve both issues using GANs

D. Patel Bayesian Inference and Deep Learning 22

Outline

1. Bayesian Deep Learning

2. Generative adversarial networks (GANs)

3. Deep learning in inverse problems

3.1 Bayesian formulation for inverse problems
3.2 GANs as prior
3.3 GANs as posterior

D. Patel Bayesian Inference and Deep Learning 23

Recall the WGAN

Learn and sample from a target PX .

Generator network g(.; θ): Critic network d(.; ϕ):

▶ Generates fake samples x̃ ▶ Distinguishes fake samples from real
▶ g : ΩZ → ΩX . ▶ d : ΩX → R.
▶ Latent variable z ∈ ΩZ ⊂ RNz . ▶ x ∼ PX .
▶ z ∼ PZ simple distribution, e.g. Gaussian. ▶ d(x) large for x ∼ PX , small otherwise.
▶ Nz ≪ Nx (dimension reduction)

D. Patel Bayesian Inference and Deep Learning 24

Recall the WGAN

Learn and sample from a target PX .

Generator network g(.; θ): Critic network d(.; ϕ):

▶ Generates fake samples x̃ ▶ Distinguishes fake samples from real
▶ g : ΩZ → ΩX . ▶ d : ΩX → R.
▶ Latent variable z ∈ ΩZ ⊂ RNz . ▶ x ∼ PX .
▶ z ∼ PZ simple distribution, e.g. Gaussian. ▶ d(x) large for x ∼ PX , small otherwise.
▶ Nz ≪ Nx (dimension reduction)

Solve the MinMax problem:

 
∗ ∗
(g , d ) = arg min arg max Π(g, d) = arg min arg max E [d(x)] − E [d(g(z))]
g d g d x∼PX z∼PZ

D. Patel Bayesian Inference and Deep Learning 24

Recall the WGAN

Learn and sample from a target PX .

Generator network g(.; θ): Critic network d(.; ϕ):

▶ Generates fake samples x̃ ▶ Distinguishes fake samples from real
▶ g : ΩZ → ΩX . ▶ d : ΩX → R.
▶ Latent variable z ∈ ΩZ ⊂ RNz . ▶ x ∼ PX .
▶ z ∼ PZ simple distribution, e.g. Gaussian. ▶ d(x) large for x ∼ PX , small otherwise.
▶ Nz ≪ Nx (dimension reduction)

Solve the MinMax problem:

 
∗ ∗
(g , d ) = arg min arg max Π(g, d) = arg min arg max E [d(x)] − E [d(g(z))]
g d g d x∼PX z∼PZ

Convergence in W1 =⇒ weak convergence

 ∗ 
E [ℓ(x)] = E ℓ(g (z)) , ∀ ℓ ∈ Cb (ΩX )
x∼PX z∼PZ

−→ moments converge.
D. Patel Bayesian Inference and Deep Learning 24
GAN as prior

Given:
▶ A set S = {x1 , ..., xn }, where xi ∼ PXprior .
▶ The direct map f (x) (exactly or approximately).
▶ The noise distribution Pη
▶ A noisy measurement y

Goal: Determine PXpost and evaluate statistics w.r.t. it.

• GAN-based Priors for Uncertainty Quantification, by Patel & Oberai, SIAM/ASA Journal on Uncertainty
Quantification 9(3):1314-1343, 2021.
• Solution of Physics-based Bayesian Inverse Problems with Deep Generative Priors, by Patel, Ray &
Oberai, arXiv:2107.02926, 2021.
D. Patel Bayesian Inference and Deep Learning 25
GAN as prior

Step 1: Using S, train a WGAN with generator g ∗ .

Assume:
▶ g ∗ is the optimal generator satisfying the weak relation
E [ℓ(x)] = E [ℓ(g ∗ (z))] , ∀ ℓ ∈ Cb (ΩX ).
x∼PX
prior z∼PZ

▶ f and Pη are continuous.

Choose
1
ℓ(x) = ℓ̂(x)Pη (y − f (x)), ℓ̂ ∈ Cb (ΩX ).
Q
post
Then, we can get an expression for PX
   
1 1 
E ℓ̂(x)Pη (y − f (x)) = E ℓ̂(g ∗ (z))Pη y − f (g ∗ (z))
x∼PX
prior Q z∼PZ Q
h i h i
=⇒ E ℓ̂(x) = E ℓ̂(g ∗ (z))
post post
x∼PX z∼PZ

where
post 1    
PZ (z|y) = Pη y − f (g ∗ (z)) PZ (z) ∝ Pη y − f (g ∗ (z)) PZ (z)
Q

post post
Sampling x from PX ≡ sampling z from PZ and evaluating x = g ∗ (z).

D. Patel Bayesian Inference and Deep Learning 26

GAN as prior

Step 2: Generate an MCMC approximation PZmcmc (z|y) ≈ PZpost (z|y).

Step 3: Evaluate statistics using Monte Carlo

Nsamples
1 X
E [ℓ(x)] ≈ ℓ(g ∗ (z))), z ∼ PZmcmc (z|y).
post
x∼PX Nsamples
i=1

What do we gain?
▶ Ability to represent complex prior, if S is available.
▶ Nz ≪ Nx makes MCMC computational tractable.

D. Patel Bayesian Inference and Deep Learning 27

Inferring thermal conductivity

Given u, find κ satisfying

−∇ · (κ∇u) = b(ξ), ∀ ξ ∈ Ω ⊂ R2
u(ξ) = 0, ∀ ξ ∈ ∂Ω
Problem setup:
▶ Measurement y, noisy temperature field u on a 2D grid.
▶ Infer x, nodal values of conductivity κ.
▶ Non-linear forward map f solves the PDE. Implemented in Fenics.
▶ Noise is assumed to be Gaussian iid.

D. Patel Bayesian Inference and Deep Learning 28

Inferring thermal conductivity (MNIST)

Assume that κ is given by MNIST digits (Nx = 784, NZ = 100)

True Generated

D. Patel Bayesian Inference and Deep Learning 29

Inferring thermal conductivity (MNIST)

Solving the inference problem on test data

D. Patel Bayesian Inference and Deep Learning 30

Inferring thermal conductivity (microstructure)

Microstructure profile given by Cahn-Hilliard (Nx = 4096, NZ = 100)

True Generated

D. Patel Bayesian Inference and Deep Learning 31

Inferring thermal conductivity (microstructure)

Solving the inference problem on test data

D. Patel Bayesian Inference and Deep Learning 32

Inverse Radon transform (CT)

Find the tissue density ρ : Ω ⊂ R2 → R given the line Radon transforms

Z
Rt,ψ = ρdγ
γt,ψ

where γt,ψ is the line at an angle ψ and at a signed-distance of t from the center of
Ω.

Problem setup:
▶ Infer x, nodal values of ρ.
▶ Linear forward map f , Radon transform.
▶ Measurement y, noisy Radon transforms on a set of lines.
▶ Noise is assumed to be Gaussian iid.

D. Patel Bayesian Inference and Deep Learning 33

Inverse Radon transform (CT)

ρ given by perturbed Shepp-Logan phantoms (Nx = 16384, NZ = 100)

True Generated

D. Patel Bayesian Inference and Deep Learning 34

Inverse Radon transform (CT)

Solving the inference problem on test data

D. Patel Bayesian Inference and Deep Learning 35

Outline

1. Bayesian Deep Learning

2. Generative adversarial networks (GANs)

3. Deep learning in inverse problems

3.1 Bayesian formulation for inverse problems
3.2 GANs as prior
3.3 GANs as posterior

D. Patel Bayesian Inference and Deep Learning 36

Conditional WGANs

Learning distributions conditioned on another field. Based on work by Adler et al.

(2018) & Almahairi et al. (2018).

Generator network: Critic network:

▶ g : ΩZ × ΩY → ΩX . ▶ d : ΩX × ΩY → R.
▶ z ∼ P Z , Nz ≪ Nx . ▶ d(x, y) large for real x, small
▶ (x, y) ∼ PXY otherwise.

D. Patel Bayesian Inference and Deep Learning 37

Conditional WGANs

▶ Objective function


L(g, d) = E d(x, y) − d g(z, y), y
(x,y)∼PXY
z∼PZ

▶ g and d determined (with constraint ∥d∥Lip ≤ 1) through

(g ∗ , d ∗ ) = arg max arg minL(g, d)

d g

▶ For the optimal generator g ∗ and given y

g ∗ (., y) = arg minW1 (PX |Y , g# (., y)PZ )

g

▶ Convergence in W1 implies weak convergence

E [ℓ(x)] = E [ℓ(g(z, y))] , ∀ ℓ ∈ Cb (ΩX ).

x∼PX |Y z∼PZ

D. Patel Bayesian Inference and Deep Learning 38

GANs as posterior

Given:
▶ A set S = {(x1 , y1 ), ..., (xn , yn )}, where xi ∼ PXprior and yi ∼ PY |X .
▶ A noisy measurement y

Goal: Determine PXpost and evaluate statistics wrt it.

Step 1: Using S, train a WGAN with generator g ∗ (z, y).

Using Bayes and weak convergence of conditional WGAN for a given y

E [ℓ(x)] = E [ℓ(g ∗ (z, y))] , ∀ ℓ ∈ Cb (ΩX )

post z∼PZ
x∼PX

Sampling x from PXpost ≡ sampling z from PZ and evaluating x = g ∗ (z, y).

D. Patel Bayesian Inference and Deep Learning 39

GANs as posterior

Step 2: Evaluate statistics using Monte Carlo

Nsamples
1 X
E [ℓ(x)] ≈ ℓ(g ∗ (z, y))), z ∼ PZ .
post
x∼PX Nsamples
i=1

What do we gain?
▶ Ability to represent complex prior, if S is available.
▶ Nz ≪ Nx .
▶ Sampling from a GAN is very simple.

D. Patel Bayesian Inference and Deep Learning 40

Inferring thermal conductivity

Given u, find κ satisfying

−∇ · (κ∇u) = 10, ∀ ξ ∈ Ω ⊂ R2
u(ξ) = 0, ∀ ξ ∈ ∂Ω

Problem setup:
▶ Infer x, nodal values of conductivity κ.
▶ Measurement y, noisy temperature field u on a 2D grid.
▶ Generate S by sampling x ∼ PXprior and evaluating y = f (x) + η.
▶ Train WGAN on S

D. Patel Bayesian Inference and Deep Learning 41

Inferring thermal conductivity

Assume κ is given by circular inclusions (Nx = Ny = 4096, NZ = 50)

D. Patel Bayesian Inference and Deep Learning 42

Inferring thermal conductivity

Solving the inference problem

D. Patel Bayesian Inference and Deep Learning 43

Inferring thermal conductivity

D. Patel Bayesian Inference and Deep Learning 44

Inferring the initial condition

Given u(ξ, T ), find u0

∂u
− ∇ · (2∇u) = 0, ∀ (ξ, t) ∈ Ω × (0, 1)
∂t
u(ξ, 0) = u0 (ξ), ∀ξ∈Ω
u(ξ, t) = 0, ∀ (ξ, t) ∈ ∂Ω × (0, 1)
Severely ill-posed problem!

Problem setup:
▶ Infer x, initial temperature field u on a 2D grid.
▶ Measurement y, noisy temperature field u on a 2D grid.
▶ Generate S by sampling x ∼ PXprior and evaluating y = f (x) + η.
▶ Train WGAN on S

D. Patel Bayesian Inference and Deep Learning 45

Inferring the initial condition

Assume u0 is given by MNIST (Nx = Ny = 784, NZ = 100)

D. Patel Bayesian Inference and Deep Learning 46

Inferring the initial condition

Solving the inference problem

D. Patel Bayesian Inference and Deep Learning 47

Inferring the initial condition

Solving the inference problem

D. Patel Bayesian Inference and Deep Learning 48

Comparing the two approaches

GAN as prior GAN as posterior

Learning type Unsupervised Supervised

Data generation x ∼ PXprior x ∼ PXprior , y ∼ PY |X

∂f
Forward model Need f and ∂x
Possibly need f to generate data

Sampling GAN and MCMC Only GAN

Generalizability Hard to control Better control

D. Patel Bayesian Inference and Deep Learning 49

Final remarks

▶ Neural networks are good universal approximators.

▶ GANs can be used to learn distributions from data and generate new
samples.
▶ Using GANs to overcome challenges with Bayesian inference:
▶ GANs as priors.
▶ GANS as posterior.

▶ Ability to capture complex prior information.

▶ Dimensional reduction using latent space.

▶ Generate point estimates to quantify uncertainty in inferred field.

▶ There are many, many other variants of GANs.

▶ GANs are not the only generative algorithms – Variational Autoencoders

(VAEs), normalizing flows, Diffusion models,etc.

D. Patel Bayesian Inference and Deep Learning 50

References

I. Goodfellow, Y. Bengio, A. Courville.

Deep Learning, MIT Press.
[Link] (2016).
N. Thuerey, P. Holl, M. Mueller, P. Schnell, F. Trost, K. Um.
Physics-based Deep Learning.
[Link] (2021).
A. Pinkus.
Approximation theory of the MLP model in neural networks.
Acta Numerica, Vol. 8, 143–195, 1999.
P. Kidger,T. Lyons
Universal Approximation with Deep Narrow Networks.
PMLR,125, 2306–2327, 2020.
D. Yarotsky, A. Zhevnerchuk
The phase diagram of approximation rates for deep neural networks.
arXiv:1906.09477, 2021.

D. Patel Bayesian Inference and Deep Learning 51

References

J. B. Keller.
Inverse Problems.
The American Mathematical Monthly, 83:107–118, 1976.
I. J. Goodfellow, J. P. -Abadie, M. Mirza, B. Xu, D. W. -Farley, S. Ozair,
A. Courville, Y. Bengio
Generative Adversarial Networks.
Advances in Neural Information Processing Systems, 2672–2680, 2014.
M. Arjovsky, S. Chintala, L. Bottou
Wasserstein Generative Adversarial Networks.
PMLR, 70:214-223, 2017
T. Karras, T. Aila, S. Laine, J. Lehtinen
Progressive Growing of GANs for Improved Quality, Stability, and Variation.
arXiv:1710.10196, 2018
J. Adler, O. Öktem
Deep Bayesian Inversion.
arXiv:1811.05910, 2018

D. Patel Bayesian Inference and Deep Learning 52

References

A. Almahairi, S. Rajeswar, A. Sordoni, P. Bachman, A. Courville

Augmented CycleGAN: Learning Many-to-Many Mappings from Unpaired
Data.
PMLR, 80:195-204, 2018
D. Patel, A. A. Oberai
GAN-based Priors for Uncertainty Quantification
arXiv:2003.12597, 2020
D. Patel, D. Ray, A. A. Oberai
Solution of Physics-based Bayesian Inverse Problems with Deep Generative
Priors
arXiv:2107.02926, 2021

D. Patel Bayesian Inference and Deep Learning 53