BLG 566E
Week 2
Normalizing Flows
Dr. Yusuf H. Sahin
Deriven from:
Deep Learning: Foundations and Concepts (2024), Christopher M. Bishop, Hugh Bishop.
Machine Learning for Graphs and Sequential Data, Stephan Günnemann
Density Estimation using Real NVP (2017), Laurent Dinh, Jascha Sohl-Dickstein, Samy
Bengio
Deep Learning Course, François Fleuret
1
Transformation of Probability Densities
• Used in normalizing flows (generative models).
• Consider a single variable 𝑥 and suppose we make a change of variables 𝑥 = 𝑔 𝑦 , then a function 𝑓(𝑥)
becomes a new function 𝑓(𝑦)( defined by
𝑓( 𝑦 = 𝑓(𝑔(𝑦))
• Given a probability density 𝑝! 𝑥 and a transformation 𝑥=𝑔(𝑦), the new density 𝑝" 𝑦 is:
𝑑𝑥 𝑑𝑔
𝑝" 𝑦 = 𝑝! 𝑥 = 𝑝! 𝑔(𝑦)
𝑑𝑦 𝑑𝑦
• The modulus ∣.∣ ensures density scaling remains positive.
2
Jacobian matrix
• Consider a density 𝑝(𝒙) over a D-
dimensional variable 𝒙 = 𝑥# , … , 𝑥$ % .
• We want to transform to a new variable
𝒚 = 𝑦# , … , 𝑦$ % where 𝒙 = 𝑔(𝒚)
𝑝" 𝑦 = 𝑝! 𝑥 det 𝑱
3
Statistical Generative Models
learning
• Model family
+ •
•
Loss function
Optimization algorithm
= 𝑝(𝒙)
• …
Data Prior Knowledge
4
Normalizing Flows
𝑋#
• We assume that the data 𝒙 follows a probability distribution 𝑝(𝒙), where 𝒙= … .
𝑋$
• We can do two interesting things with a distribution:
• Data sampling
• Generate data sample 𝒙& following the distribution 𝑝(𝒙).
• Density evaluation
• Given any 𝒙& , compute the probability density at this point 𝑝(𝒙& ).
5
Normalizing Flows
• Normalizing Flows (NF) can model flexible distributions for data sampling and density evaluation.
• Normalizing Flows are based on the change of variable formula.
• It substitutes a variable 𝒛 to another variable 𝒙 by using a transformation function.
6
Change of Variables: Examples
• 𝐷=1, 𝑝# (𝑧) = 𝑈𝑛𝑖𝑓([0,1]), 𝑓(𝑧) = 2𝑧 + 1 = 𝑥
• The probability in the input space should be the same as in the output space.
()
• Abusing the notations leads to 𝑝' 𝑥 = 𝑝# 𝑧 (!
.
7
Change of Variables: Examples
• 𝐷=1, 𝑝# (𝑧) = 𝑈𝑛𝑖𝑓([0,1]), 𝑓(𝑧) = 2𝑧 + 1 = 𝑥
𝑓 *# 𝑥 = 𝑧
𝜕𝑧 𝜕 𝑥−1 1 𝑥−1 1
𝑥−1 = = 𝑝' 𝑥 = 𝑝#
𝑓 *# 𝑥 = 𝜕𝑥 𝜕𝑥 2 2 2 2
2
8
Change of Variables: Examples
• 𝐷=2, 𝑝# (𝑧) = 𝑈𝑛𝑖𝑓( 0,1 ' ), 𝑓(𝑧) = 𝑧 + 𝑠ℎ𝑖𝑓𝑡 = 𝑥
Applying a constant shift does not change the area after the
transformation.
𝑝' 𝑥 = 𝑝# 𝑧
𝑎 𝑏
• 𝐷=2, 𝑝# (𝑧) = 𝑈𝑛𝑖𝑓( 0,1 ' ),𝑓(𝑧) = 𝑀𝑧 = 𝑥, 𝑀 =
𝑐 𝑑
The linear transformation 𝑀 changes the area from 1 to
𝑎𝑑−𝑏𝑐=det(𝑀).
1
𝑝' 𝑥 = 𝑝# 𝑧
det(𝑀)
9
Change of Variables: Examples
2 1
𝑓 𝑧 = 𝑀𝑧 = 𝑥, 𝑀=
1 2
(3,3)
(1,2)
1
𝑝' 𝑥 = 𝑝# 𝑧
det(𝑀)
(0,1) (1,1)
(2,1) 1
𝑝' 𝑥 = 𝑝# 𝑀*# 𝑥
(0,0) (1,0) det(𝑀)
1
𝑝' 𝑥 = 𝑝# 𝑀*# 𝑥
3
10
Python Example
𝑧# 𝑧# 𝑧#
𝑝# = 𝑁 0, 𝐼' , 𝑓 𝑧 =
𝑧' ' 𝑧' + 0.25 (𝑧#' − 1)
def f(z):
z1, z2 = z[0], z[1]
x1 = z1
x2 = z2 + 0.25 * (z1**2 - 1.0)
return [Link]([x1, x2])
def finv(x):
x1, x2 = x[0], x[1]
z1 = x1
z2 = x2 - 0.25 * (x1**2 - 1.0)
return [Link]([z1, z2])
rng = [Link].default_rng(7)
N = 6000
z = rng.standard_normal((2, N))
x = f(z)
11
Change of Variable Formula
• İf 𝐷∈ℕ, 𝑝# (𝒛) a 𝐷-dimensional distribution, 𝑓 𝒛 = 𝒙 an invertible and differentiable transformation, then
distribution 𝑝' (𝒙) is
𝜕𝑓 *# (𝒙)
𝑝' 𝒙 = 𝑝# 𝑓 *# (𝒙) . det
𝜕𝒙
(+(𝒙)
• The term (𝒙
is called Jacobian of G, which is a 𝐷×𝐷 matrix.
12
Normalizing Flow Conditions
• Invertibility
• The input and output space of the mapping should have the same dimension D.
• If D=1, the mapping 𝑓should be strictly monotonic (increasing or decreasing)
• If the transformation 𝑓 is linear , its determinant should be nonzero: det f ≠ 0.
• Differentiability
(/ !"(!)
• The mapping 𝑓should be smooth. The Jacobian should exist.
(!
• Note: Differentiability is a sufficient condition; in theory, the mapping 𝑓does not have to be
differentiable everywhere. In practice we usually use only differentiable transformation.
13
Stacking
• We can apply the change of variable formula multiple time.
𝜕𝑓#*# (𝑧# )
𝑝# 𝑧# = 𝑝0 𝑧0 det
𝜕𝑧# 𝜕𝑓#*# (𝑧# ) 𝜕𝑓'*# (𝑧' )
𝑝' 𝑧' = 𝑝0 𝑧0 det det
𝜕𝑧# 𝜕𝑧'
𝜕𝑓'*# (𝑧' )
𝑝' 𝑧' = 𝑝# 𝑧# det
𝜕𝑧'
14
Stacking
• We can apply the change of variable formula multiple time.
• The change of variable formula becomes:
1
𝜕𝑓&*# (𝑧# )
log 𝑝1 𝒙 = log 𝑝0 𝑧0 + X 𝑙𝑜𝑔 det
𝜕𝑧&
&2#
15
Normalizing flow
• A NF transforms a simple distribution (e.g. uniform, Gaussian) in a complex distribution. For some
Normalizing Flows the universality theorem has been proven.
• NFs stack valid transformations to model a complex mapping between the input and output space. The input
variable flows through the transformations.
• The determinant terms normalize the distribution in the output space.
16
Exercises
• Is 𝑓(𝑧) = 1 − 𝑧 a valid transformation?
• Is f(𝑧) = 2 − 3𝑧 a valid transformation?
−𝑧, 𝑧 ∈ [0,1]
• Is f(𝑧) = [ a valid transformation?
1 − 𝑧, 𝑧 ∈ [1,2]
17
Forward and Reverse Parametrization
• The change of variable formula does a mapping between two
distributions. 𝜕𝑓 *# (𝑥)
*#
𝑝' 𝑥 = 𝑝# 𝑓 (𝑥) . det
𝜕𝑥
• Reverse parametrization: How to use the change of variable formula to
evaluate 𝑝' 𝑥 at any point x.
• Forward parametrization: How to use the change of variable formula to
sample points from 𝑝' 𝑥 .
• There exist many different flows which flows generally differ in the
parametrization of the transformation 𝑓.
18
Reverse Parametrization
• We parametrize the inverse transformation 𝑔 = 𝑓 *# that we know analy[cally.
• 𝑔3 𝒙 = 𝒛 is computable and parameter 𝜑 could be learned.
• We know that the inverse 𝑓 = 𝑔*# exists, but we might not know it analytically.
• 𝑔3*# 𝒛 = 𝒙 might not be easily computable.
• The change of variable formula with reverse parametrization is
𝜕𝑔3 (𝒙)
𝑝' 𝒙 = 𝑝# 𝑔3 (𝒙) . det
𝜕𝒙
• Given any point 𝒙𝒊 , we may compute 𝑝' 𝒙𝒊 .
19
Reverse Parametrization
• We can also stack transformations with reverse parametrization.
• While 𝑝0 (𝒛) has a simple shape, 𝑝0 (𝒙) can capture very complex structure.
20
Reverse Parametrization
• To compute the density
• For any 𝒙𝒊 , we can set 𝒙𝒊 = 𝒛𝑲 .
(+$ (𝒛𝒋,𝒊 )
%
• For every transformation i, compute 𝐳𝒋,𝒊*𝟏 = 𝒈𝝋𝒊 (𝒛𝒋,𝒊 ) and det (𝒛𝒋,𝒊
• Given 𝐳𝒋,𝟎 , we can compute 𝐩𝟎 (𝐳𝒋,𝟎 )
21
Normalizing Flows in Machine Learning
=
• Given a dataset 𝐷 = 𝑥 < <2# usually consisting
of i.i.d. samples, find the parameters that best
explain the data.
22
Forward Parametrization
• We parametrize the transformation 𝑓 that we know analytically.
• 𝑓> 𝒛 = 𝒙 is computable and parameter 𝜃 can be learned
• We know that the inverse 𝑓 *# exists, but we might not know it
analytically.
• The change of variable formula with reverse parametrization is
*#
𝜕𝑓> 𝒛
𝑝' 𝒙 = 𝑝# 𝒛 . det
𝜕𝑧
• The formula only uses the known parametrized function 𝑓> .
• We can also stack transformations with forward parametrization.
23
Forward Parametrization
• The forward parametrization enables sampling from the distribution 𝑝1 (𝒙).
• Sample 𝐳𝒋,𝟎 ~𝒑𝟎 𝒛𝟎 (e.g. Gaussian or uniform).
*#
(/"# 𝒛𝒋,𝒊&𝟏
• For every transformation i, compute 𝐳𝒋,𝒊 = 𝒇𝜽𝒊 (𝒛𝒋,𝒊*𝟏 ) and det
(𝒛𝒋,𝒊&𝟏
• For the particular sample 𝐱 𝐣 = 𝐳𝒋,𝑲 , we can compute 𝐩𝐊 (𝒙𝒋 ).
24
RealNVP
• Non-volume preserving transformations.
• Invertible neural network architecture for
exact density estimation.
• Uses affine coupling layers to ensure
tractable Jacobian determinant.
25
RealNVP: Affine Coupling Layer
• Given a D dimensional input x and d < D, the output y of an affine coupling layer follows the equations.
• 𝑠 and 𝑡 are neural networks.
• Jacobian is triangular, hence easy to calculate the
determinant.
𝜕𝑦
log det = X 𝑠(𝑥#:C )
𝜕𝑥 %
26
RealNVP: Affine Coupling Layer
• Jacobian becomes triangular by using a binary mask Boolean vector 𝑏 ∈ 0,1 C .
27
RealNVP: Affine Coupling Layer
• Jacobian becomes triangular by using a binary mask Boolean vector 𝑏 ∈ 0,1 C .
28
RealNVP: 2D Data
29
RealNVP: Handling images
• Images could be masked before convolution.
30
GLOW: Generative Flow with Invertible 1×1
Convolutions
• A scalable normalizing flow for high-dimensional
image generation.
• Extends RealNVP with invertible 1×1
convolutions.
• ActNorm: Per-channel affine normalization.
31
Planar Flows
• 10 planar transformations can transform simple distributions into a more complex one.
Base distribution: Gaussian
Base distribution: Uniform
32
Planar Flows
• Matrix Determinant Lemma
det 𝐴 + 𝑢𝑣 % = (1 + 𝑣 % 𝐴*# 𝑢) det(𝐴)
• Using the planar flow:
𝒙 = 𝑓> 𝒛 = 𝒛 + 𝒖 ⊙ ℎ(𝒘% 𝒛 + 𝑏)
𝜕𝑓> 𝑧
det = det(𝐼 + ℎD 𝒘% 𝒛 + 𝑏 𝒖𝒘% ) = 1 + ℎD 𝒘% 𝒛 + 𝑏 𝒖𝒘%
𝜕𝑧
• Mapping should be invertible: 𝒖𝒘% > −1.
• ℎ is generally choosen as tanh.
33
Jacobian Determinant Computation
• The change of variable formula involves the Jacobian determinant.
• Jacobian computation can be hard/slow:
34
Some Determinant Properties
• Because of the given properties, it is easy to evaluate diagonal and triangular Jacobians.
35