Scribd
Upload
17 views26 pages

Paper 8

This document provides an introduction to deep generative models (DGMs), which are neural networks designed to approximate complex probability distributions. It discusses the mathematical challenges in training DGMs, including issues related to hyperparameters, sample similarity quantification, and latent space dimension determination. The paper also presents three main approaches to DGMs—normalizing flows, variational autoencoders, and generative adversarial networks—while emphasizing the need for further research in this rapidly evolving field.

Uploaded by

Shahnawaz Alam
Copyright
© All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF or read online on Scribd
17 views26 pages

Paper 8

This document provides an introduction to deep generative models (DGMs), which are neural networks designed to approximate complex probability distributions. It discusses the mathematical challenges in training DGMs, including issues related to hyperparameters, sample similarity quantification, and latent space dimension determination. The paper also presents three main approaches to DGMs—normalizing flows, variational autoencoders, and generative adversarial networks—while emphasizing the need for further research in this rapidly evolving field.

Uploaded by

Shahnawaz Alam
Copyright
© All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF or read online on Scribd
arXiv:2103.05180v2 [[Link]] 12 Apr 2021 An Introduction to Deep Generative Modeling Lars Ruthotto! and Eldad Haber? Upepartmont of Mathematics, Fmory University, Atlanta, GA, USA *Department of Earth and Ocean Sciences, University of British Columbia, Vancouver, BC, Canada, April 13, 2021 Abstract. Deep generative models (DGM) are neural networks with many hidden layers trained to approximate complicated, high-dimensional probability distributions using a lange number of samples. When trained ‘successfully, we can use the DGMs to estimate the likelihood of each observation and to create new sata- ples from the underlying distribution. Developing DGMs has become one of the most hotly researched fields in artificial intelligence in recent years. ‘The literature on DMs has become vast and is growing rapidly. Some advanees have even reached the public sphere, for example, the rovent successes in gen erating realsticlooking images, voices, or movies; so-called deep fakes. Despite these successes, several mathematical and practical issues limit the broader use of DGMs: given a specific dataset, it remains challenging to design and train a DGM and even more challenging to find out why a particular model is or is not effective, To help advance the theoretical understanding of DGMs, we introduce DGMs and provide concise mathematical framework for modeling the three most populat approaches: normalizing flows (NF), variational autooncoders (VAR), and generative adversarial networks (GAN). We illustrate the advantages and disadvantages of these basic approaches using numerical experiments. Our goal is to enable and motivate the reader to contribute to this proliferating research area. Our presentation also ‘emphasizes relations betwaen generative modeling, and optimal transport. Keywords: Deep Generative Models, Machine Learning, Deep Learning, Optimal Transport, Nor. rmalizing Flow, Variational Autoencoder, Generative Adversatial Network 1 Motivation Applications of deep generative models (DGM), stich as creating fake portraits from celebrity images, have recently made headlines. ‘The advent of these so-called deep fakes poses considerable societal and legal challenges, but also promise new beneficial technologies (10]- Those include new scientific applications of GMs, for example, in physics and computational chemistry (9, 35, 7) Fueled by these headlines and the potential applications across scientific diseiplines, there has bee an explosion in research activity in gonerative modeling in recent years. Due to the high volume and frequency of publications in this area, this article does not attempt to provide a comprehensive re- view, Instead, we seek to provide a mathematical introduction to the field, use at in-depth diseussion of three main approaches to show the potential of DGMs, and expose open challenges. We also airs at illustrating similarities between generative modeling and other felde of applied mathematics, most Importantly, optimal transport (OT) (14, 49, 38]. For a more comprehensive view of the field, we refer fo the monographs on deep learning [18, 24], variational autoencoders (VB) [29, 42, 30], and gen- erative adversarial nets (GAN) [17]. ‘To enable progress in this area, we provide the codes used to generate examples in thie paper as wll as interactive iPython notebooks in our Github repository at ‘avtps://github, con/EzoryMLIP/DeepGenerativeMedelingintre Deep generative models ate neural networks with many hidden layers trained to approximate com= plicated, high-dimensional probability distributions. In short, the ambitious goal in DGM training is to learn an unknown or intractable probability distribution from a typically small number of independent and identically distributed samples. When trained successfully, we can use the DGM to estimate the likelihood of a given sample and to create new samples that are similar to samples from the unknown distribution. These probloms have been at the core of probability and statistics for decades but remain ‘computationally challenging to tolve, particularly in high dimensions Despite many recent advances and success stories, several open challenges remain in the field of generative modeling. This paper focuses on explaining three key mathematical challenges. 1. DGM training is an ill-posed problem since uniquely identifying a probability distribution from a finite number of samples is impossible. Hence, the performance of the DGM will depend heavily on so-called hyperparameters, which include the design of the network, the choice of training objective, regularization, and training algorithms 2. ‘Training the generator requires a way to quantify its samples’ similarity to those from the intractable distribution, In the approaches considered here, this either requires the inversion of the generator ‘or comparing the distribution of generated samples to the given dataset, Both of these avemues Ihave their distinct challenges. Inverting the generator is complicated in most cases, particularly ‘when itis modeled by neural network that is noulinear by design. Quantifying the similatity of ‘wo probability distributions from samples loads to two-sample test problems, which are expecially dificult without prior assumptions on the distributions 3. Most common approaches for training DGMs assume that we can approximate the intractable distribution by transforming a known and much simpler probability distribution (for instance, a Gaussian) in a latent space of known dimension. In most practical cases, determining the latent space dimension is impossible and is left as a hyperparameter that the user needs to choase. This choice is both difficult and important, With an overly conservative estimate, the generator may not approximate the data well enough, and an overestimate can render the generator non-injective, which complicates the training. To imcroase the accessibility of the paper, we will keep the discussion as informal as possible and sacrifice generality for clarity where needed, ‘The remainder of the paper is organized as follows. In Section 2-we describe the generative modeling problem mathematically. In Section 3, we prosent finite [41, 12, and continuous normalizing flows [20, 51, 50, 15, $6). In Section 4, we introduce variational autoencoders [29, 42, 30). In Section 5, we introduce Generative Adversarial Networks [19, 3). In Section 6, we provide a detailed discussion and compariso ‘of the three approaches. In Section 7, we conclude the paper and highlight a few important disections of future research, 2 Mathematical Formulation and Examples ‘This section establishes our notation, defines and illustrates the deep gener presents two numerical examples used {o demonstrate the different approaché overview of the DGM training problem, ive modeling problem, and provides a high-level 2.1 General Set Up ‘The key goal in generative modeling is to learn a representation of an intractable probability distribution X defined over i", where n typically is large, and the distribution is complicated; consider, for example, ‘multimodal distribution with disjoint support. To this end, we can use a potentially large, but typically finite, number of independent and identically distributed (ii.d) samples from % that we refer to as the ‘raining data, Unlike standard statistical inference where @ mathematical expression for the probability is sought, the goal is to obtain a generator a:R! OR” a that maps samples from a tractable distribution Z supported in RY to points im R* that resomble the given data, In other words, we assume that for each sample x ~ A there is at least one point a ~ Z, such that g(a) ~ x. We denote the transformation of the latent distribution Z as g(2). Having a generator ‘hat can map points fom the simple distribution, Z, to the intractable distribution, ¥, allows us to generate samples from the complicated space 2, which is desired in many applications Figue 1: A deep generative model, go, is trained to map samples from a simple distribution, 2, (bottom right) to the more complicated distribution go(Z) (top right), which is similar to the true distribution 2° (top left). Finding an objective function that quantifies the discrepancy between the generated samples and the original examples is the key obstacle to training generative models, This is particularly difficult in the absence of point-to-point correspondences between data samples and latent variables. Since the vector 2 that results in a given vector x is generally unlmown, it is common to refer to it as the latent variable and call Z the latent space. As is common, we will assume that Z is a univariate Gaussian in R?, This is without lose of generality and in principle, Z can be any tractable distribution; ‘hat is, we require the ability to sample from Z and, im some cases, compute the probability pz(z). We illustrate our notation in Figure 1 tis important to note that the latent space dimension, g, will generally be different from the dimension of the data space, n. For example, high-resolution images with mallions of pixels do not really "live" in such a high-dimensional space since their content mostly gets preserved when reducing the resolution. Instead, there is « hidden manifold of typically unknown dimension in which the images reside, This further complicates the problem, and we discuss this point later in the paper. Assuming the generator g is known, we can generate new data points by sampling 2 ~ 2 and computing 9{2). In many applications, ranging from deep fakes to Bayesian statistics, generating now ‘samples is the only goal. In addition, the generator can also be used to compute the likelihood or evidence of a particular sample x using marginalization x(x) I Pa(xlz)p2(2)de, 2) where the likelihood p,(x'z) measures how close g(z) is to x. Note that the exact computation of (2) is, in general, intractable due to the highedimensionality of the integral. The choice of the likelihood function depends on the properties of the data. For real-valued data, it is common to use a Gaussian which leads to on0)-F exp (—Llgte) xi?) ®) ‘whore the choice of « > 0 controls how marrow the likelihood is around the samples. For binary data, ‘one typically assumes 4 Bernoulli distribution, which leads to ole) Thome ose? “ Polxin) Deriving 9 from first principles is impossible or infeasible for most data sets of interest. For example, is may be challenging to model the process that transforms a sample from # univariate Gaussian to an image of a celebrity, ‘Therefore, it has become common in recent years to use generic function approximators such as neural networks with many hidden layats. This is the fundarsental design concept in deep generative models (DGM), where g is modeled using a feed-forward deep neural network (DNN). Advantages of DNNs include their ability to approximate functions in high dimensions effectively. We enote the DNN generator by go and its weights by @.¢ 2™* Defining the DNN architecture that defines go, that is, choosing the number of layers and the opera tons involved in each layer, isa topic in its own right that we will not discuss in detail here. Instead, we will review a few examples ftom the literature in our numerical experiments below and refer the reader +o the excellent introduction [24] and the comprehensive textbook [18] for indepth discussions and more ‘options. Our choice is made for conciseness and should not divert from the fact that choosing an effective architocture is critical and complicated by the lack of theoretical guidelines, For example, the quality of ‘the architecture impacts our ability to model the generative process and our ability to solve the learning problem, that is, to train the parameters of the generator 2.2. Testbed Examples Wo use two exaraples to illustrate and compare the different approaches to deep generative modeling, (Our goal is not to improve the state-of-the-art on those common benchmark problems but to keep the models and their implementation as simple as poseible and closely match the presented derivations. We encourage the reader to look under the hood and perform more in-depth experiments and provide our implementation at https: //github..con/EnoryMLIP/DeepGenerativeNodel ingIntro. Example 1 (Moons) We use a two-dimensional eeample to help visualize the deep generative modeling problem, the data and latent distributions, ond the intermediate steps of the generation process, Here, tue consider the moons ezample fram the seikit-Learn package [38]. The implementation provides an infinite muber of (pscudo) random samples from a complicated distribution whose support ts spit sno two disjoint resions of equal mass shaped lke halfmoons; see Figure 2 for a visualization. The user can control the width of the halfmmoon shapes. Por the setting used in our plot, the width i sufficiently luide such that the support of the underlying distribution has non-zero volume sn R!. Hence, we use a = 2 dimensional latent space and seek to find a generator thal maps the standard normal distribution fo the data distribution. It is important to note thet if we reduced the width of the moon shapes con siderably, their intrinsic dimension would reduce to one, and we would expect this approach to fal or 4 least provide suboptimal results. Even though the optimal generator ts discontinuous, one common approach to generative modeling where q = n sto restrict the search to smooth and invertible models; see Section $. This modeling choice allows us to efficiently compute and optimize cach sample's likelihood since we can determine the latent varvable associated with each sample. Sill, we expect large derivatives of the generator in parts of the latent space as we try to transform a uni-modal Gaussian to bi-modal ‘istribution; for more insight on this ise and ways to stabilize the generator using a mizture model as latent distribution, see {25}. As we all see, upon suitable construction of the generator, is results in 0 relatively straightforward training problem compared tothe mare advanced methods needed when 9 21 Example 2 (MNIST) As « high-dimensional example in which the data’s intrinsic dimensionality és clearly less than n, we consider the well-studied MNIST dataset [81]; see Figure 9. The dataset consists of gray-valued digital images, each having 28 x 28 pixels and showing one hand-uritten digit. The dataset provides a finite number of images that are divided into 60,000 training and 10,000 test vmages. To train the generator, we do not require labels; however, we demonstrate in Figure 8 that the embedding of the data points into the latent space in this example roughly clusters the samples based on the digit shown. ‘The first obstacle to setting up the DCM training 1s that the intrinsic dimension of the MNIST dataset is unknown, which renders choosing the dimension of the latent space non-trivial, Whale each image contains n = 784 pazels, the support of X will bkely We in a subset of a much lower dimension Also, since the images are grouped into ten different classes, one can expect the support to be disjoint lunth 2 sukstantil distance between the different clusters Despite these conceptual challenges, we will demonstrate that DGMs can be trained effectively and at modest computational costs to create realistic-looking images. In our example, we seck to train a DGM such that it maps samples from the two-dimensional standard normal distribution to realistic smages. Figure 2: Visualizing the training data of the two moons dataset in Example 1. Here, the data distribution (samples represented by blue squares in right subplot) form two disjoint half moon shaped clusters. As the dataset does not live on a lower dimensional manifold, we assume an intrinsic dimension of two and try to find a generator from the q = 2 dimensional normal distribution (left subplot) to the data, In practice, itis common to model the generator as an invertible transformation, which simplifies the training; see Section 3, ‘This choice is made so that we can easily visualize the latent space; however, we note that using 4 larger latent space dimension may improve the quality of the generated images. Similar to {40), we define the generator as a three-layer convolutional neural network (CNN) that transforms an input sample x © RE to.a vectorized smage go(a) © R™ using the following three steps wl ana (1 (e+), 2 ona (W (IC™ 42)) ® so(2) = anem (KW =) Here, KK), KO) are linear operators, bY, b®), 8 are bias terms, N’ denotes a batch normalization layer, onevuln) = max{x, 0} i the rectified linear wi, ond aeiga(2) ~ (1+ exp (—z))"* ts the sigmoid function. The activation functions, ansits and dvem, are applied element-wise. For ease of notation, we collectively denote the parameters of the model as 8; that ts, 6 is a vector containing the parameters of KOO KO BO BOB We now descrite our generative model in more detail. The matrix K" in the first layer is of size (64-7-7) x2, that is, i transforms the input 2 € R? into 64 images of size 77, also called channels. We ‘add the bias vector bY © RY channel-wise and apply batch normalization (26] the output of the affine transformation before, finally, using the activation function, ‘The matrar K') in the second layer of size (32-14-14) x (64-77). It maps the input images w"? to $2 images with 14 x 14 pizels each by ‘computing the transpose of strided convolutions whose stencils are 4 x 4. As in the first layer, the bias be ER™ shifts each channel separately. In the final layer, the matriz Kis of size 784 (82-14-14); that is it provides a sinale image with 28 x 28 pizels by computing a linear combination of transposed, strided convolutions applied to all the channels. ‘The bias, 3 18 a scalar. Due to the sigmoid activation function, the entries of g(a) are all Between zero and one Overall, the generator has No = 42,913 trainable weights that we initialize randomly and then train by sninimizing an objective function. The specific construction ofthe objective function is the main diference hetween the approaches. Note that we cannot assume the generator to admit an inverse, unlike in the ‘two-dimensional example 2.3 Training the Generator: A High-level Overview In the remainder of the paper, we will discuss three main approaches for training the DGM gy using zmples from X. ‘Their common goal is to learn a parameter 0 such that now samples, ga(z) where z~ Z, are statistically indistinguishable from samples from the training data. In other words, we train, generator, gg IR? + B™ Figure 3: Illustration of the MNIST image generation process in Example 2. Here, the intrinsie dimension of the dataset (right) is unknown but assumed to be much less than the number of pixels por image, n= 784. In our example, we define the latent variable to be distributed according to the q = 2-dimensional standard, normal distribution (left). Compared to the moons example (see Figure 2), the generator cannot be assumed to be invertible. This complicates the density estimation and the training process. 0 so that go transforms the latent probability distribution, Z, ta the probability distribution of the data, X. Determining the distance between two distributions is a two-sample hypothesis test problem, which is very difficul, especially for complicated distributions in high dimensions, Therefore, we will also judge the quality of the generator visually ‘We will see correspondences between the latent variables and the data samples can help avoid the two-sample test problem. One key obstacle to this goal is the unsupervised nature of the problem; that is, wwe do not have pairs (x, ) of corresponding samples from the data and latent distribution, respectively Instead, we have many samples from the data space, X’, and the function, go that we can use to create nw sazaples. ‘Therefore, if we want to avoid the challenging two-sample problem, we need to establish ‘correspondences, for example, by inverting ge or using statistical inference techniques ‘We eck to exemplify these challenges using numerical experiments with the examples from Section 22. ‘The three state-of-the-art approaches, which are built on different assumptions on the data, modeling choices, and numerical techniques, are + In Sec, 3, we deseribe two ways to construct an invertible D@M go and apply them to the moons dataset from Example I. In the frst one, we concatenate a finite number of invertible functions In the second one, we model ge as a trainable dynamical system. ‘These approaches have become Jnown as normalizing flows and continuous normalizing ows, respectively. When ge and its inverse are continuously differentiable, we can avoid the integral in (2) and compute the likelihood of a sample using the change of variables forzaula, Hence, thie assumption simplifies the training snd also enables links to optimal transport (OT). The invertibility of the model assumes that g = n, which ig limiting in many real-world datasets. Still, we can use (continous) normalizing flows a¢ building blodks in more powerful approaches; for example, we tzy to reduce the space dimension land then use the Gow in the latent space. ‘+ In Sec. 4, we discuss variational autoencoders (VAKs) [30] that use a probabilistic model to establish rolations between the latent variables and data samples for non-invertible generators go. VAEs are broadly applicable, for instance, in the realistic case where q (3 fa (x9)? —togast vas" (x9) + Botan), (8) TA fonction 9: RY RY ie diffeomorphic WW ie invertible and both g and go! orientation-preserving, det Vole) > 0 are continuously diferentiable, If 9 i ako Figure 4: The objective function for a simple affine function 9g (2) = Oz for a single example. where x2 x2,....2e are bid, samples from 2 and s is also called the batch size, We also note that KL(pxllpo) = Jseu(8) + Exxx [—logp(x)] and since the second term is constant with respect to 8, rainimizing Jy. (0) is equivalent to minimizing the Kullback-Leibler divergence i (7). ‘The terms in Equation (8) can be also interpreted by a more classical regularized approach, The first term is minimized when 95'(x) = 0 irrespectably’ of x, that is, it prefers transformations that shrink the space. The second term can be viewed as mapping the volume around x and it has the exact opposite ‘effct. It prefers transformations that are volume preserving, A plot of the function fora transformation im a single dimension is plotted in Figure 4, The approximation in (8) comes through the sampling of ‘the distribution in 4 it is important to understand that the quality of the minimizer is only as good as ‘the quality of this approximation, which for complicated, high-dimensional distributions can be poor: ‘The asymmetry of the KL divergence has important implications for the generators resulting from ‘maximum Likelihood training; see ao [2]. On the one hand, the objective function will assign lange values for points x at which the actual likelihood exceeds that implied by the generator; that is, pa(x) 2 pel) Hence, when trained well, the support of the density of the generator should cover the samples. On the ‘other hand, the generator may produce samaples with small likelihood since the KL divergence becomes stall when p(X) < po(X) "To approximately minimize Jy, we use stochastic approximation methods such as stochastic gradiont escent (SGD) and its variants; seo the excellent survey (6). In short, their iterations minimize the objective using gradients that are estimated from the current minibatch, which is re-sarapled at every step. By avoiding using the entire dataset, SGD type methods seek to save computational costs and, ‘empirically, often provide meaningful neural network models Lot us now turn to the question of how to design the function ge, We assumed that itis a reversible diffeomorphic function that preserves the orientation of the data, In the following, we discuss how to build such a generator. 3.2 Finite Normali A finite normalizing flow [41, 12] is constructed by concatenating diffeomorphic transformations with tractable Jacobian determinants, which leads to the genorator fx © fn 8-0 file) O) In deep learning it is common to call f, the layers of the network and K the depth of the network Assuming that efficient expressions for the inverses of the layer functions f, are available, we ean compute ‘the maximum likelihood loss (8) using ng Flows go(2) siN)= Joho fe ove Fgh) amd lgar Wap") = Fo togder 75" (9"). 00) (79, yl, yl are the hidden features, y= = (), Note that we can perform maximum likelihood training as long as we can compute the inverse of the onerator and the log-determsinant ofits Jacobian. However, efficent sampling, wich is our goal, requires ‘ficient forward calculations as well. The key tradeoff in normalizing lows is designing the layers f, to bye expressive while also leading to tractable Jacobian determinants, and ideally same cost for evaluating 4 and sts inverse. ‘Those considerations allow us to group existing approaches by their abity to compute ‘bo 99°, oF bots + Examples of normalizing fows that can evaluate both the generator and its inverse efficiently are nonlinear independent components estimation (NICR) [11] and real non-volume preserving, (ral VP) flow [12], which we prasent im more detail below. A ley idea in these approaches is that the layers partition the variables into two blocks and use components that are easy to snvert. Both of these approaches belong, to the more general class of invertible neural networks; see, eg, the excellent literature review and application to inverse probletas in (1 Here, y' jo'(x), and we have with"? ‘+ Heamples of normalizing flows that can compute gp efficiently but not its inverse include the planar and radial flows [A1) and inverse autoregressive flows [28]. These approaches lack a closed-form expression for the inverse, which is needed to train the generator using the maximua likelihood objective function. Instead, they are commonly used in variational autoencoders, which we will discuss next ‘+ An example of a normalizing flow that can compute 95? efficiently but not the generator is the tasked autoregressive flow [37]. While these models can be trained straightforwardly using maxi- ‘mum likelihood training and provide efficient density estimates, their use to produce new samples is limited Numerical Example: Real NVP For Moons Dataset We apply the real preserving (seal NVP) flow (12) to our moons problom presented in Example 1. Our architecture and implementation is adapted fom the excellent tutorial [s). The jth layer splits its input y © R? into its components y)") and yy), When j is an even number, keeps the Srst component unchanged and transforms the second component using an affine transformation parameterized by y2?, that is, Jed nom-volume 16°) =[3) clo 60) +060) |: o where s),t) : RR are neural networks that model scaling and translation, respectively. We collect the trainable weights from all the layers in 8. The Jacobian of the jth layer reads BA) =| woo} 60) +60) mole 60) | aervt (y) = exp (s (9) (2) also note that, independent of the specific choice of the networks 5; and fj, the inverse of the layer is 16") [Bea gryowtnr)) 0 When j is an odd number, the roles of the componente of y are interchanged. We note that this leads to.a well-conditioned flow as long as the magnitude of the scaling remains bounded to a small number In our example, we use a normalizing flow with K’= 6 real NVP layers. Por every j = 1,2,-..,6 ‘the neural networks that parameterize the scaling and translation of the layers (3, and ¢,, respectively) ‘each consist of three affine layers with a hidden dimension of 128, The firet two layers use a leaky 7 x Bi Figure 5: Normalizing flow results for the moon problem described im Example 1. Here, we use a real NVP approach with six hidden layers to transform samples from the standard normal distribution (ed dots) to match the given data (blue squares). In the frst subplot ftom the left, we show one batch of the training data. In the second subplot, we show the likeliiood estimate superizaposed by generated samples (blue squaze). Here, it is worth noting that, due to the flow model's smoothness, the two half-moons appear to be connected. In the thind subplot, we show the generator's hidden layers for six randomly chosen latent variables as inputs (red dots). Due to the altemating fashion of the layers, the transformation for every layer is limited to one ofthe coordinates. In the fourth subplot, we show a two-dimensional histogram of the inverse transformation applied to 50,000 samples from the moon data set, superimposed by a few randomly chosen examples, As expected, the latent variables do approximately, but not perfectly, match a Gaussian distribution; see, for example, a narrow blue line passing approximately vertically through the origin that separates the parts associated with each cluster of the dataset. ReLU nonlinearity with a slope parameter of 0.01. Overall, the fow network has 205,848 trainable weights, which we train using the stochastic approximation scheme ADAM [27]. We perform 20,000 steps of maximum likelihood training, each approximating the gradient of the objective function using a rminibatch containing 256 points sampled from the true distribution X. We show the result of the training in Figure 5, ‘The real NVP approach has also been applied to higher dimensional examples [12). Here, the partitions ‘of the vatiables are more involved, and a certain depth of the network is required to ensure full coupling between the variables 3.3. Continuous Normalizing Flows ‘While we can compute finite normalizing ows efficent only for a specific choice of layers, we cam obtain more flexibility in the framework of continuous normalizing fows (CNB). In a CNE [20), we define the generator se go(z) = y(Z) where T’ > 0 is some terminal time and y : [0,7] —> RP aataies the initial value problem Bey(t) = valy(,0, where y(0) =x 4) Here, vo : RY xR > R" is an atbitrary neural network parameterized by the weights @ © 2"*, For a sufficiently yegular v», the mapping «>> y(T) is invertible and in principle, one may define the inverse (0) where p : [0,7] -> RY satisfies the final value problem - p(t) Here, we integrate backward in time as indicated by ~2). While this provess is straightforward in theory, it is important to note that the generator’s stability and its inverse depends erueially on the design of ve, choice of weights, and the numerical integration used to solve (15). Simple integrators such as the (010.9, whee p(T) =x (03) commonly used forward Euler or even bigherordor Runge-Kutta schemes without step size control can be prone to large errors when integrating backward; especially when the velocity in (I) changes rapidly along the curve ¥() ‘There are several ways to avoid such issues, First, one may use ymplactic integrators that can be reversed analytically up lo machine procision, Second, as we will demonstrate in our experiment, wo 10 ean favor curves y()) that are simple (ideally, straight lines) by regulating the velocity. ‘This simplifies ‘the intogration and improves the inverse consistency. When using a non-consorvative integrator we rocommend also monitoring the inverse errors | 95 (ge(2)) ~ z\| and llga(gq"(«)) ~ x! ‘To compute the logarithm of the doterminant of gg we employ the Jacobi identity also used in [51, 20) and obtain dogdet Voo(ay = fence (Vyve( p00), 0) (8) In practice, this computation can be combined with the numerical approximation of the characteris- ties (15). Relation to Optimal Transport To shed more light into the CNF problem, we point out ita similarities and diferences to opttnal transport (14,10, 39 ‘fo this end, we take a macroscopic view on the acting ofthe generator defined in (14) on the latent distribution Z Let us denote the density function associated with Z by po, that is, go is the density function of a univariate Gaussian. Then, the push forward of po under the transformation that maps # to ¥(r) by integrating (14) until some 7 ¢ [0,7] is given by (7), the solution to the continuity equation p(x, 8) + 9 (pl, t}vo (x,t) f%,0) = pol) Here, we see that the neural network vp takes the role of an non-stationary velocity, We also note ‘that (14) computes the characteristic curve originating, in forward in time and, similarly, (15) computes ‘the same curve backward in time from point x. We can now formulate the CNF problem as a PDE-constrained optimization problem wink, [log p(x,7)] subject to (27). (8) Since the objective function solely depends on the density at the final time, the above problem does not attain a unique solution. ‘To be precise, all velocity felds with the same initial and endpoints of the ‘characteristics are assigned the samo function value. Noting that (14) and (15) are trivial to solve whon the velocity does not change along the characteristics, motivates us to add the La transport cost and consider the regularized problem sain [ Heeb. \*P0u axa + a8 [log 7] subject to (7 9) where a > O is regularization parameter that balances between minimizing the transport costs and tnasimiring the lg likelihood, We view this problem as a relaxed version of the dynasnic optimal transport formulation [5] or more precisely as a mean field game [43]. The key difference to standard optimal transport settings is that the target density i unknown, Fora the optimal transport theory it follows that (19) attains a unique solution fr which the characteristics are traght lines, We not (19) ean be reformulated into » convex optimization problem and can be solved efciently using PDE constrained optimization techniques in dimensions n © 3 [22 Several approaches that ad traeport costs to the CNF problem have been proposed recently [5 50, 15, 32, 3. While thove approaches dilfer in some factors, inching the definition of the objective function, network design, and numerical implementation, they provide ample numetial evidence to suggest that optimal traneport techniques improve the traning ofthe CNP. Sine the methods ae applied to'machine leering benchmark datasets of tensor hundreds of dimension, all mimerical schemes rly on neural network parameterizatons ofthe velo and compite an approximate solution to (19) king stochastic epproximation techniques, There i some numerical evidence thet penalizing vieations of the Mamilton-Jacobi-Bellman (IIJB) equations, which are the necessary and suftcient first-order optimality ‘conditions of (19), improves the practical performance [50, 36] Numerical Example: OT-Flow for Moons Dataset We apply the O'T-Flow approach intro: ‘éacod in [35] to the moons problem; soe Example 1. ‘This approach involves a Lagrangian PDE solver to solve the continuity equation (17) in a mesh-tee manne, which renders the scheme scalable to high dimensions. Following the optimal transport theory, the approach defines the dynamics of the flow as the sadient ofa patestial, thas, ~Véo(x,¢). Farther, it hss been shown empirically that adding u oO Figure 6: Continuous normalizing flow results for the moon problem described in Example 1. Here, we use the OT-Flow approach [36] to transform samples from the standard normal distribution (red dots) to match the given data (blue squares). In the fisst subplot from the left, we show one batch of the training data, In the second subplot, we show the likelihood estimate superimposed by generated samples (blue square). Despite the smoothness of the model, the two half-moons appear almost disconnected. In the third subplot, wwe show the hidden layers of the generator for six randomly chosen latent variables as inputs (red dots). Since the training is regularized by the transport costs, the characteristics are almost straight lines, which allows to invert the flow by integrating backwards in time. In the fourth subplot, we show a two-dimensional hhistogram of the inverse transformation applied to 50,000 samples from the moon data set, superimposed by a few randomly chosen examples. As expected, the latent variables do approximately, but not entirely, match a Gaussian distribution, In particular, note the narrow gap in the center of the domain that separates the parts associated with each cluster of the datasct z x Bi ‘penalty function that enforces the Hamilton-Jacobi-Bellman (HJB) equations along the characteristic ‘carves improves performance. We parameterize by as the sum of a quadratic form and a two-layer residual network whoee second layer has 32 neurons; see [86] for details. This model has 1,229 trainable parameters, around two orders of magnitude fewer than the real NVP model used above. During the training, we compute the characteristics using a fourth-order Runge-Kutta scheme with equidistant time steps. As in tho real NVP example, we train the network tsing 20,000 training steps of the ADAM scheme, eack based on a rminibatch containing 256 randomly aarapled points. We show the training results in Figure 6. Ilere, the trained Gow provides meaningful sarmplee froma the datasot as wll as realistic density estimates. Due to the penalization of transport costs, the characteristics are almost straight, which also helps reduce the inverse error nearly to machine precision 4 Variational Autoencoders 1 most practical situations, we eanot assume that the latent space dimension and that of the data space are equal. ‘This prohibits a direct use of the flow models from the previous section since the generator is not invertible and the KI, divergence may be unbounded or not well-defined (3). Variational Autoencoders (VAE) [29, 42, 30] are a popular framework to overcome this limitation and, typically, uso a latent space of much smaller dimension than the data space, that is, ¢ {* are learning rates that are typically chosen a priori by the user. Empirically, SGD. variants such as ADAM [27], and RMSProp [48] ate typically more efficent and lead to better scluions than the plain stochastic gradient scheme shown above ‘Tho critical hurdle during training is balancing betweon the two subprobloms in (30). Consider, for example, the beginning of training, when it is relatively easy to distinguish the actual and generated samples. On the one hand, training the discriminator to optimality would make it impossible for the generator to improve, since the gradient [Link](0, @") would be close other hand, not training the discriminator well enough would make it challenging to update the weights of the generator Another common problem that also presents theoretical challenges to the GAN formulation ie known 1s rode collapse, ‘To gain sone intuition, comsider the extreme case when the generator maps the entire zero, On istribution Z to a single data point, say x°) ~ 2, that is, go(z) =x") for almost all x~ 2 In this case, the optimal « training would terminate, jinator would yield dg-(x") = 4 and dye (x) = 1 for all j > 1 and the tis important to note that we can easly detect the above example of mode collapse by inspecting 4 few samples from the generator, which will all be identical A more dealt ease would be when the tenerator mapped almost no point from Z close to x". Ifthe data set contains a few thousand data points or more, such failure is almost impossible to detect by analyzing a fnite number of generated ‘proposed to reduce the rik of mode collapse; far examaple, ome eas auld distance terms that compate the statistic of the minibatches or apply one-sided label smoothing [24] samples. Several houristics have be Numerical Experiment: DCGAN for MNIST. We continue our MNIST Example 2 and seele to train the generator along with a discriminator whose architecture is similar to the one used in Deep Convolutional GAN (DCGAN) [td]. ‘To be specific, we define the discriminator using two convolution layers and one fully connected layer: that ie, given the input feature x © R", we predict the probability )7 in 20 equidistant steps. While the images from the samples ‘that is sampled from the true dataset using v = ama (W (Eke ebis)) v= renae (W (Ba +B) @) g(x) = ouiem (dean) v + dcax) Cll and Cha are convolution operators, can i a vector, ell, el2ye Scan are hiss terms, N is's batch normalization ayer, and oem i the leaky ReLU activation ‘2 220 02 che ‘To abbreviate the notation, we collect the trainable parameters in Cf! in the vector @ , ‘The fast two layers contain the convolution operators Clk and C{Zha, whose structure i identical to the operators Cy and Clty used in the VAR example. In addition to the different convolution stencils, the main diference here is the behavior of the activation function for negative entries in the feature vector. As inthe VAE example, the output ofthe second layer a vector of length 7-7-64, which ‘we multiply with deaw € R%® and shift by the scalar daqy € R before using the sigmoid function to ‘obtain the fine value In training, we perform the steps in (31) with gradients approximated using minibatches of size 64 and using the ADAM scheme. We use fixed learning rates of 0.0002 and, as proposed in [44], « momentum of 05. We observed that the training performance is highly dependent on these parameter choices and that, for instance, changes in the batch size can quickly load to complete failure of the training. We perform a xed mimber of 50,000 training steps ‘We compare two ways of initializing the weights, Fist, we use the default random initialization imple. rented in pytorch for all the generator and discriminator weights, Second, we initialize the discriminator randomly as above but use the optimal weights from the VAE example in the generator. ‘We show random samples and samples obtained by interpolating across the latent space in the second column of Figure 9 and the middle row of Figure 10, respectively. ‘The similarity of the samples to true MNIST images varies considerably. The best images are almost indistinguishable from the actual distribution, but many images do not appear to contain any ofthe digits ‘Aa the GAN training see to find a saddle point of Joa in (29), monitoring its value during training doce not provide useful insight into the quality of the generator. Often the best option is to inspect a few tenerated samples at some intermediate steps visually. As described above, this can be tisleadin, for example, due to mode collapse. To abtain some insight into the convergence of the method, we estimate the distance between and go(Z) using the multivatiatec test for equal distributions suggested in (47). As can be soon in Figure 11, this distance is reduced during the GAN training both from the random initialization (blue dashed line) and when initializing the generator with the weights from the VAE. training (coli blue line). The latter considerably reduces the number of training steps needed and obtains « better score overall renas(2) pr Scas 5.2 Discriminators based on Transport Costs ‘The idea of Wasserstein GANs [3] is to uso an approximation of the earth movar distance (EMD) to measure the distance between go(Z) and a. The EMD distance, also known as Wasserstein-1 distance, can also be sven as the cost of the optimal transport plan that moves g¢(Z) to X, The Wasserstein-] distance can be written as Wilge(Z),¥} nf Bee) ~> [I~ xl) (a) where Il denotes the set of all distributions 4(3,x) whose marginals are X and go(2), respectively ‘Tho value of >(¥,x) indicates how much mass is moved between the two locations, and the distance is measured using the Euclidean norm. a ance rang steps 108 Figure 1: Estimating the distance between ¥ and g9(Z) for the MNIST example using the multivariate ¢ test for equal distributions [47,, A lower value suggests an improved performance of the generator. The DCGAN (blue) and WGAN (red) approaches reduce this distance measure effectively when initialized randomly (dashed) and when starting from the VAE solution (solid). According to this measure, starting ftom the ‘VAE solution improves the results overall and the WGAN slightly outperforms the DCGAN, Instead of the formulation ($4), most practical implementations of GANs use the equivalent formu lation Wa(gol(2),4) = Exc [f (90(2))) ~ Zev FC)» (85) lcs ‘which is also known as the Kantorovich and Rubinstein norm; see [39] for more details. Here, the maximum is taken over all functions f : B” > B that are Lipschitz-1 continuous. Computing such f is far from trivial, especially in high dimensions, In the context of GANs it has become common to approximate the function f with a neural network fy. We note, however, that ensuring the Lipschitz, ‘continuity of the neural network approximation is dificult ‘Wasserstein GANs have several appealing theorotical advantages over the discriminator-based GANS, including the ones found in [3]. For example, the lose function is continuous a long a go is continuous and differentiable almost everywhere when gy is locally Lipschitz continuous. However, it is unclear ‘which of these advantages can be realized or is even desirable in practical applications. In fact there are examples in which training J and thus approximating Wa(ge(Z),X) more accurately can reduce the performance of the generator (46) ‘Numerical Experiment: WGAN for MNIST We continue the MNIST example and train the generator described in Example 2 using a WGAN approach. ‘The architecture for the potential fg is ‘equal to the one used as the discriminator in (88) except that the sigaoid im the last layer is removed. Following [3] we impose bound constraints on ¢ to regularize fg. This will, in general, not be sufficient to ensure the Lipschitz continuity. More promising (but also more involved) ways to incorporate this constraint is using gradient penalty methods (21] or spectral normalization [84] We train the network using 50 iterations of RMSprop. As before, we perform two experiments. One uses the default random initialization in pytorch and one starts from the solution obtained by the VAE. ‘We show random samples and interpolated images, respectively, in the third column of Figure 9 and ‘the bottom row of Figure 10 are qualitatively similar to the DCGAN images. ‘The shaxpness of the images resembles that of the true images, and many samples are very realistic. However, many images do not contain a hand-written digit and are thus easy for the human eye to be recognized as fakes Both for the random and the VAB initialization, the WGAN approach outperforms the DCGAN, in terms of the multivariate © test [47], see Figure 11. Again, initializing the gonerator with the VAR, 2

You might also like