Generative Deep Learning

Chapter: 06

Normalizing Flow Models

By
Fahad Hussain and Muhammad Atif Tahir
 What is Normalizing Flows
• Normalizing Flows are a method for constructing complex distributions by
transforming a probability density through a series of invertible mappings

• By repeatedly applying the rule for change of variables, the initial density
'flows' through the sequence of invertible mappings
 Change of Variable
• Suppose we have a probability distribution pX(x) defined over a
rectangle X in two dimensions (x = x1 , x1 ), as shown in Figure 6.2
• This function integrates to 1 over the domain of the distribution (i.e., x1
in the range [1, 4] and x2 in the range [0, 2]), so it represents a well-
defined probability distribution. We can write this as follows:
Change of Variable
Change of Variable

Note that this function is invertible.

Change of Variable
 The Jacobian Determinant
The Jacobian of a function z = f (x) is the matrix of its first-order partial derivatives,
as shown here:

If we take the partial derivative of z1 with respect to x1, we obtain 1/3. If we take the
partial derivative of z1 with respect to x2, we obtain 0. Similarly, if we take the partial
derivative of z2 with respect to x1, we obtain 0. Lastly, if we take the partial derivative
of z2 with respect to x2, we obtain 1/2.
 The Change of Variables Equation
We can now write down a single equation that describes the process for changing
variables between X and Z. This is known as the change of variables equation.
The change of variables equation

• How does this help us build a generative model?

• The key is understanding that if pZ(z)is a simple distribution from which we

can easily sample (e.g., a Gaussian), then in theory, all we need to do is find
an appropriate invertible function f(x) that can map from the data X into Z
• and the corresponding inverse function g(z) that can be used to map a
sampled z back to a point x in the original domain
• We can use the preceding equation involving the Jacobian determinant to
find an exact, tractable formula for the data distribution p(x)
Applying in Practice
• Two major issues

• Firstly, calculating the determinant of a high-

dimensional matrix is computationally extremely
expensive—specifically, it is O(n3)

• This is completely impractical to implement in

practice, as even small 32 × 32–pixel grayscale
images have 1,024 dimensions
Applying in Practice
• Secondly, it is not immediately obvious how we
should go about calculating the invertible function
f(x)

• We could use a neural network to find some

function f(x) but we cannot necessarily invert this
network—neural networks only work in one
direction
Applying in Practice
• To solve these two problems, we need to use a
special neural network architecture that ensures
that the change of variables function f is invertible
and has a determinant that is easy to calculate

• We shall see how to do this in the following section

using a technique called Realvalued Non-Volume
Preserving (RealNVP) transformation
 Real NVP Algorithm
The Two Moons Dataset
The dataset we will use for this example is created by the make_moons function from
the Python library sklearn
This creates a noisy dataset of points in 2D that resemble two crescents, as shown in
Figure 6-4

Make a noisy, unnormalized moons

dataset of 3,000 points

Normalize the dataset to have mean 0

and standard deviation 1

Figure 6-4. The two moons dataset in two dimensions

data = datasets.make_moons(3000, noise=0.05)[0].astype("float32")
norm = [Link]()
[Link](data)
normalized_data = norm(data)
 Coupling Layer
We will build a RealNVP model that can generate points in 2D that follow a similar
distribution to the two moons dataset

Whilst this is a very simple example, it will help us understand how a normalizing flow
model works in practice, in fine detail

First, however, we need to introduce a new type of layer, called a coupling layer

A coupling layer outputs two tensors that are the same shape as the input:
a scaling factor (s) and a translation factor (t)
 Coupling Layer
• The input to the Coupling layer block in our example has
two dimensions

• The scaling stream is a stack of Dense layers of size 256.

Four stack in the book. For images, Conv2D is being used
instead of Dense Layers

• The final scaling layer is of size 2 and has tanh activation

• The translation stream is a stack of Dense layers of size 256

• The final translation layer is of size 2 and has linear

activation

• The Coupling layer is constructed as a Keras Model with two

outputs (the scaling and translation factors)
 Passing data through a coupling layer
The architecture of a coupling layer is not particularly interesting—what makes it
unique is the way the input data is masked and transformed as it is fed through the
layer, as shown in Figure 6-6

Notice how only the first d

dimensions of the data are fed
through to the first coupling
layer—the remaining D−d
dimensions are completely
masked (i.e., set to zero)
In our simple example with D=
2, choosing d= 1 means that
instead of the coupling layer
seeing two values, x1, x2, the
layer sees x1, 0
Figure 6-6. The process of
transforming the input x through a
coupling layer
Passing data through a coupling layer
Example
• Lets assume x = [2,3]
• Also assume that [2,0] passed through Coupling
layer with output s = [-0.5,0.1] and t = [0.4,0.5]
• Due to reverse masking s = [0,0.1], t = [0,0.5] will be
passed
• So, z1 = x1 = 2
• z2 = x2 * exp(0.1) + 0.5 = 3.815512754
Passing data through a coupling layer

Figure 6-8. The inverse function x = g(z)

 Stacking coupling layers

if we stack coupling layers, flipping the masking each time, we can build a neural
network that is able to transform the whole input tensor, while retaining the essential
properties of having a simple Jacobian determinant and being invertible.
Figure 6-9 shows the overall structure.
 Training the RealNVP Model
Now that we have built the RealNVP model, we can train it to learn the complex
distribution of the two moons dataset
Remember, we want to minimize the negative log-likelihood of the data under the model
− log pX (x) . Using Equation below first,, we can write this as second:

We choose the target output distribution pZ (z) of the forward process f to be a

standard Gaussian, because we can easily sample from this distribution
We can then transform a point sampled from the Gaussian back into the original
image domain by applying the inverse process g
The show in the next slide;
Training the RealNVP Model
 Analysis of the RealNVP Model
• Once the model is trained, we can use it to transform the training set into the
latent space (using the forward direction, f) and
• More importantly, to transform a sampled point in the latent space into a
point that looks like it could have been sampled from the original data
distribution (using the backward direction, g)
• Figure shows the output from the network before any learning has taken
place the forward and backward directions just pass information straight
through with hardly any transformation

The RealNVP model inputs (left)

and outputs (right) before
training, for the forward process
(top) and the reverse process
(bottom)
 Analysis of the RealNVP Model
After training the forward process is able to convert the points from the
training set into a distribution that resembles a Gaussian
Likewise, the backward process can take points sampled from a
Gaussian distribution and map them back to a distribution that
resembles the original data

The RealNVP model inputs (left)

and outputs (right) after
training, for the forward process
(top) and the reverse process
(bottom)
 Analysis of the RealNVP Model
The loss curve for the training process is shown below

The loss curve for the RealNVP training process

 Other Normalizing Flow Models - GLOW
The research paper GLOW, presented at NeurIPS 2018, introduced a new way to build
normalizing flow models
This method allowed the models to generate high-quality samples and manipulate
them in a meaningful way
The key difference from previous models was replacing a complex setup with a simple
1x1 convolutional layer. This change ensured all data was processed and maintained
the efficiency of the model

Random samples from the GLOW model

 Other Normalizing Flow Models - FFJORD
RealNVP and GLOW are discrete time normalizing flows, employing a finite number of
coupling layers to transform input
FFJORD (Free-Form Continuous Dynamics for Scalable Reversible Generative Models),
introduced at ICLR 2019, introduces a novel approach by modeling the transformation as
a continuous time process
This involves representing the dynamics with an ordinary differential equation (ODE)
whose parameters are generated by a neural network (fθ)
A blackbox solver is utilized to solve the ODE at a given time point, such as finding z1 from
an initial point z0 sampled from a Gaussian distribution.

FFJORD models the transformation between the data distribution and a standard
Gaussian via an ordinary differential equation, parameterized by a neural network
 References

1. Laurent Dinh et al., “Density Estimation Using Real NVP,” May 27, 2016,
https:// [Link]/abs/1605.08803v3.

2. Diedrick P. Kingma and Prafulla Dhariwal, “Glow: Generative Flow with

Invertible 1x1 Convolutions,” July 10, 2018, [Link]

3. Will Grathwohl et al., “FFJORD: Free-Form Continuous Dynamics for Scalable

Reversible Generative Models,” October 22, 2018,
[Link]
Thanks