Deep learning

Unit – 4
Deep generative models
A Generative Model is a powerful way of learning any kind of data distribution
using unsupervised learning and it has achieved tremendous success in just few
years. All types of generative models aim at learning the true data distribution of
the training set so as to generate new data points with some variations. But it is
not always possible to learn the exact distribution of our data either implicitly or
explicitly and so we try to model a distribution which is as similar as possible to
the true data distribution.
Two of the most commonly used and efficient approaches are Variational
Autoencoders (VAE) and Generative Adversarial Networks (GAN). VAE aims
at maximizing the lower bound of the data log-likelihood and GAN aims at
achieving an equilibrium between Generator and Discriminator.

Boltzmann Machine and Auto-encoders

Boltzmann Machine was invented by Geoffrey Hinton and Terry Sejnowski in
1985.
A surprising feature of this network is that it uses only locally available
information. The change of weight depends only on the behavior of the two
units it connects, even though the change optimizes a global measure” - Ackley,
Hinton 1985.
Some important points about Boltzmann Machine −
 They use recurrent structure.
 They consist of stochastic neurons, which have one of the two possible
states, either 1 or 0.
 Some of the neurons in this are adaptive freestatefreestate and some are
clamped frozenstatefrozenstate.
 If we apply simulated annealing on discrete Hopfield network, then it
would become Boltzmann Machine.

Objective of Boltzmann Machine

The main purpose of Boltzmann Machine is to optimize the solution of a
problem. It is the work of Boltzmann Machine to optimize the weights and
quantity related to that particular problem.

Architecture
The following diagram shows the architecture of Boltzmann machine. It is
clear from the diagram, that it is a two-dimensional array of units. Here,
weights on interconnections between units are –p where p > 0. The weights of
self-connections are given by b where b > 0.

As we know that Boltzmann machines have fixed weights, hence there will be
no training algorithm as we do not need to update the weights in the network.
However, to test the network we have to set the weights as well as to find the
consensus function CFCF.
Boltzmann machine has a set of units Ui and Uj and has bi-directional
connections on them.
 We are considering the fixed weight say wij.
 wij ≠ 0 if Ui and Uj are connected.
 There also exists a symmetry in weighted interconnection, i.e. wij = wji.
 wii also exists, i.e. there would be the self-connection between units.
 For any unit Ui, its state ui would be either 1 or 0.
The main objective of Boltzmann Machine is to maximize the Consensus
Function CFCF which can be given by the following relation
CF=∑i∑j⩽iwijuiuj
Now, when the state changes from either 1 to 0 or from 0 to 1, then the change
in consensus can be given by the following relation −
ΔCF=(1−2ui)(wij+∑j≠iuiwij)ΔCF=(1−2ui)(wij+∑j≠iuiwij)
Here ui is the current state of Ui.

Probability of the network to accept the change in the state of the unit is given
by the following relation −
AF(i,T)=11+exp[−ΔCF(i)T]AF(i,T)=11+exp[−ΔCF(i)T]
Here, T is the controlling parameter. It will decrease as CF reaches the
maximum value.

Auto-encoders:

Autoencoder is a type of neural network where the output layer has the same
dimensionality as the input layer. In simpler words, the number of output
units in the output layer is equal to the number of input units in the input
layer. An autoencoder replicates the data from the input to the output in an
unsupervised manner and is therefore sometimes referred to as a replicator
neural network.

The autoencoders reconstruct each dimension of the input by passing it

through the network. It may seem trivial to use a neural network for the
purpose of replicating the input, but during the replication process, the size of
the input is reduced into its smaller representation. The middle layers of the
neural network have a fewer number of units as compared to that of input or
output layers. Therefore, the middle layers hold the reduced representation of
the input. The output is reconstructed from this reduced representation of the
input.

Architecture of autoencoders
An autoencoder consists of three components:

 Encoder: An encoder is a feedforward, fully connected neural

network that compresses the input into a latent space representation
and encodes the input image as a compressed representation in a
reduced dimension. The compressed image is the distorted version
of the original image.
 Code: This part of the network contains the reduced representation
of the input that is fed into the decoder.
 Decoder: Decoder is also a feedforward network like the encoder
and has a similar structure to the encoder. This network is
responsible for reconstructing the input back to the original
dimensions from the code.

First, the input goes through the encoder where it is compressed and stored in
the layer called Code, then the decoder decompresses the original input from
the code. The main objective of the autoencoder is to get an output identical
to the input.

Note that the decoder architecture is the mirror image of the encoder. This is
not a requirement but it’s typically the case. The only requirement is the
dimensionality of the input and output must be the same.
Variational Auto-encoders
We know that we can use an autoencoder to encode an input image to a much
smaller dimensional representation which can store latent information about the
input data distribution. But in a vanilla autoencoder, the encoded vector can only
be mapped to the corresponding input using a decoder. It certainly can’t be used
to generate similar images with some variability.
To achieve this, the model needs to learn the probability distribution of the
training data. VAE is one of the most popular approach to learn the complicated
data distribution such as images using neural networks in an unsupervised
fashion. It is a probabilistic graphical model rooted in Bayesian inference i.e.,
the model aims to learn the underlying probability distribution of the training
data so that it could easily sample new data from that learned distribution. The
idea is to learn a low-dimensional latent representation of the training data
called latent variables (variables which are not directly observed but are rather
inferred through a mathematical model) which we assume to have generated our
actual training data. These latent variables can store useful information about the
type of output the model needs to generate. The probability distribution of latent
variables z is denoted by P(z). A Gaussian distribution is selected as a prior to
learn the distribution P(z) so as to easily sample new data points during inference
time.

Now the primary objective is to model the data with some parameters which
maximizes the likelihood of training data X. In short, we are assuming that a
low-dimensional latent vector has generated our data x (x ∈ X) and we can map
this latent vector to data x using a deterministic function f(z;θ) parameterized by
theta which we need to evaluate (see fig. 1[1]). Under this generative process,
our aim is to maximize the probability of each data in X which is given as,

Pө(X) = ∫Pө(X, z)dz = ∫Pө(X|z)Pө(z)dz (1)

Here, f(z;θ)has been replaced by a distribution Pө(X|z).

The intuition behind this maximum likelihood estimation is that if the model can
generate training samples from these latent variables then it can also generate
similar samples with some variations.
Obviously it is a tedious task to manually specify the relevant information we
would like to encode in latent vector to generate the output image. Rather we
rely on neural networks to compute z just with an assumption that this latent
vector can be well approximated as a normal distribution so as to sample easily
at inference time. If we have a normal distribution of z in n dimensional space
then it is always possible to generate any kind of distribution using a sufficiently
complicated function and the inverse of this function can be used to learn the
latent variables itself.
The idea of VAE is to infer P(z) using P(z|X) which we don’t know. We infer
P(z|X) using a method called variational inference which is basically an
optimization problem in Bayesian statistics. We first model P(z|X) using simpler
distribution Q(z|X) which is easy to find and we try to minimize the difference
between P(z|X) and Q(z|X) using KL-divergence metric approach so that our
hypothesis is close to the true distribution. This is followed by a lot of
mathematical equations which I will not be explaining here but you can find it in
the original paper. But I must say that those equations are not very difficult to
understand once you get the intuition behind VAE.

The final objective function of VAE is :-

The best thing of VAE is that it learns both the generative model and an
inference model.

Generative adversarial networks

Yann LeCun says that adversarial training is the coolest thing since sliced bread.
Seeing the popularity of Generative Adversarial Networks and the quality of the
results they produce, I think most of us would agree with him. Adversarial
training has completely changed the way we teach the neural networks to do a
specific task. Generative Adversarial Networks don’t work with any explicit
density estimation like Variational Autoencoders. Instead, it is based on game
theory approach with an objective to find Nash equilibrium between the two
networks, Generator and Discriminator. The idea is to sample from a simple
distribution like Gaussian and then learn to transform this noise to data
distribution using universal function approximators such as neural networks.

This is achieved by adversarial training of these two networks. A generator

model G learns to capture the data distribution and a discriminator model D
estimates the probability that a sample came from the data distribution rather
than model distribution. Basically the task of the Generator is to generate natural
looking images and the task of the Discriminator is to decide whether the image
is fake or real. This can be thought of as a mini-max two player game where the
performance of both the networks improves over time. In this game, the
generator tries to fool the discriminator by generating real images as far as
possible and the discriminator tries not to get fooled by the generator by
improving its discriminative capability. Below image shows the basic
architecture of GAN.
Fig.3. Building block of Generative Adversarial Network

We define a prior on input noise variables P(z) and then the generator maps this
to data distribution using a complex differentiable function with parameters өg.
In addition to this, we have another network called Discriminator which takes in
input x and using another differentiable function with parameters өd outputs a
single scalar value denoting the probability that x comes from the true data
distribution Pdata(x). The objective function of the GAN is defined as

We maximize the above function w.r.t parameters of Discriminator using

Gradient Ascent and minimize the same w.r.t parameters of Generator using
Gradient Descent. But there is a problem in optimizing generator objective. At
the start of the game when the generator hasn’t learned anything, the gradient is
usually very small and when it is doing very well, the gradients are very high.
But we want the opposite behaviour. We therefore maximize E[log(D(G(z))]
rather than minimizing E[log(1-D(G(z))]
One of the earliest model on GAN employing Convolutional Neural Network
was DCGAN which stands for Deep Convolutional Generative Adversarial
Networks. This network takes as input 100 random numbers drawn from a
uniform distribution and outputs an image of desired shape. The network
consists of many convolutional, deconvolutional and fully connected layers. The
network uses many deconvolutional layers to map the input noise to the desired
output image. Batch Normalization is used to stabilize the training of the
network. ReLU activation is used in generator for all layers except the output
layer which uses tanh layer and Leaky ReLU is used for all layers in the
Discriminator. This network was trained using mini-batch stochastic gradient
descent and Adam optimizer was used to accelerate training with tuned
hyperparameters.

Autoregressive models
Autoregression is a time series model that uses observations from previous time
steps as input to a regression equation to predict the value at the next time step.

It is a very simple idea that can result in accurate forecasts on a range of time
series problems.

As an equation, that looks like this: X_{t+1} = \sum^t_i \delta _i X_{t-i} +

cXt+1=∑itδiXt−i+c . This should look a lot like normal (online) linear
regression and that’s because mathematically it effectively is. It uses the
previous m terms to predict the next term where m is a constant called the lag or
receptive field. The next item, X_{t+1}Xt+1, is predicted based on the product
of a learnable parameter, , times a previous item in the sequence, X_iXi, plus a
learnable constant or bias, c.
The cool part of these networks is their unique capabilities. They are sequential
models but still feedforward. They are generative but still supervised. These
facts allow us to apply optimization, training and acceleration techniques that
have been around for years to sequential generative models which makes them
faster, more stable and gives us a better understanding of them.
DARNs (Deep AutoRegressive Networks) are generative sequential models,
and are therefore often compared to other generative networks like GANs or
VAEs; however, they are also sequence models and show promise in traditional
sequence challenges like language processing and audio generation. But before
we jump into the network comparisons, let’s define exactly what people are
referring to when they say DARN and how these networks function.
Technically, any network that used previous data from a sequence to predict a
future value in that sequence could be considered autoregressive, but when in
the context of deep learning, autoregression almost always refers to the relation
of prior outputs as inputs as opposed to recurrent models which take a set
amount of predefined input. To clarify, outputs are fed back into the model as
input and this is what makes the model autoregressive. Usually, the
implementation ends up being a convolutional layer or series of convolutional
layers with autoregressive connections. The graphic below is one of the best
ways to understand this relationship. Notice how the first prediction is generated
only on the prior data but for every prediction after that, the model takes the
output from the previous step as input. Also, notice how the width of the input
window is constant and so after many interactions, the original data isn’t even
part of the input and its regression solely on data the model has
generated.
It’s important to note that we need to use causal convolution to ensure that no
data is allowed to leak backward in time i.e. all predictions are made using only
data from previous time steps, this is necessary to preserve the validity of the
chain rule mentioned above. Both the traditional model and the deep learning
probabilistic model use previous data to predict future data, but the deep
learning model is more powerful for several reasons primarily its ability to deal
with large amounts of high dimensionality [Link] network architecture helps
the chain rule represent complex data in such a way that structure prediction is
possible. For example, when trained on an image dataset, the deep learning
model allows the chain rule to implicitly represent not only the probability of
each pixel but also the relation between them. From here, it’s a normal
supervised learning technique where the loss is correlated with the difference
between the predicted output and the observed next item in the sequence.
During training, the input is the observed (original training) data not the
predicted data which is notably not autoregressive, but it allows for a very high
level of parallelization which speeds up training by a factor of 5 or 10 or more.
It’s only during inference when it’s important that the output is fed back in as
input. Conceptually, that’s all you really need to know about autoregressive
networks. The interesting part is in the special cases, applications, and oddly
enough the implementation itself.

Generative Image Models

Generative models are one of the most promising approaches towards this
goal. To train a generative model we first collect a large amount of data in some
domain (e.g., think millions of images, sentences, or sounds, etc.) and then train
a model to generate data like it.

The trick is that the neural networks we use as generative models have a number
of parameters significantly smaller than the amount of data we train them on, so
the models are forced to discover and efficiently internalize the essence of the
data in order to generate it.

Generative models have many short-term applications. But in the long run, they
hold the potential to automatically learn the natural features of a dataset,
whether categories or dimensions or something else entirely.