Module 5: Auto-encoders and Generative models.

Autoencoders, Variational Auto-Encoder-under complete Auto-encoder,

stochastic encoder, denoising encoder, Applications of Autoencoders.
Generative models - Boltzmann machines, Deep Belief Networks,
Generative Adversarial Networks.
Autoencoders
Autoencoders are a special type of neural networks that learn to compress data
into a compact form and then reconstruct it to closely match the original input.
They consist of an:
 Encoder that captures important features by reducing dimensionality.
 Decoder that rebuilds the data from this compressed representation.
The model trains by minimizing reconstruction error using loss functions like Mean
Squared Error or Binary Cross-Entropy. These are applied in tasks such as noise
removal, error detection and feature extraction where capturing efficient data
representations is important.

Architecture of Autoencoder

An autoencoder’s architecture consists of three main components that work

together to compress and then reconstruct data which are as follows:
1. Encoder

It compress the input data into a smaller, more manageable form by reducing its
dimensionality while preserving important information. It has three layers which
are:
 Input Layer: This is where the original data enters the network. It can be
images, text features or any other structured data.
 Hidden Layers: These layers perform a series of transformations on the input
data. Each hidden layer applies weights and activation functions to capture
important patterns, progressively reducing the data's size and complexity.
 Output(Latent Space): The encoder outputs a compressed vector known as
the latent representation or encoding. This vector captures the important
features of the input data in a condensed form helps in filtering out noise and
redundancies.

2. Bottleneck (Latent Space)

It is the smallest layer of the network which represents the most compressed
version of the input data. It serves as the information bottleneck which force the
network to prioritize the most significant features. This compact representation
helps the model learn the underlying structure and key patterns of the input helps
in enabling better generalization and efficient data encoding.

3. Decoder
It is responsible for taking the compressed representation from the latent space
and reconstructing it back into the original data form.
 Hidden Layers: These layers progressively expand the latent vector back into a
higher-dimensional space. Through successive transformations decoder
attempts to restore the original data shape and details
 Output Layer: The final layer produces the reconstructed output which aims to
closely resemble the original input. The quality of reconstruction depends on
how well the encoder-decoder pair can minimize the difference between the
input and output during training.

Loss Function in Autoencoder Training

During training an autoencoder’s goal is to minimize the reconstruction loss which

measures how different the reconstructed output is from the original input. The
choice of loss function depends on the type of data being processed:

 Mean Squared Error (MSE): This is commonly used for continuous data. It
measures the average squared differences between the input and the
reconstructed data.

 Binary Cross-Entropy: Used for binary data (0 or 1 values). It calculates the

difference in probability between the original and reconstructed output.

During training the network updates its weights using backpropagation to minimize
this reconstruction loss. By doing this it learns to extract and retain the most
important features of the input data which are encoded in the latent space.

Efficient Representations in Autoencoders

Constraining an autoencoder helps it learn meaningful and compact features from

the input data which leads to more efficient representations. After training only the
encoder part is used to encode similar data for future tasks. Various techniques
are used to achieve this are as follows:
 Keep Small Hidden Layers: Limiting the size of each hidden layer forces the
network to focus on the most important features. Smaller layers reduce
redundancy and allows efficient encoding.
 Regularization: Techniques like L1 or L2 regularization add penalty terms to
the loss function. This prevents overfitting by removing excessively large
weights which helps in ensuring the model to learns general and useful
representations.
 Denoising: In denoising autoencoders random noise is added to the input
during training. It learns to remove this noise during reconstruction which helps
it focus on core, noise-free features and helps in improving robustness.
 Tuning the Activation Functions: Adjusting activation functions can promote
sparsity by activating only a few neurons at a time. This sparsity reduces model
complexity and forces the network to capture only the most relevant features.

Applications of Autoencoders
Compression of data

Even though autoencoders are meant to compress data, they are rarely employed for this
reason in practice. The following are the reasons –

 Lossy compression − The Autoencoder's output is not identical to the input, but it is a near but degraded
representation. They are not the best option for lossless compression.
 Data-specific − Autoencoders can only compress data identical to the data on which they were trained.
They differ from traditional data compression algorithms like jpeg or gzip in that they learn features
relevant to the provided training data. As a result, we can't anticipate a landscape photo to be compressed
by an autoencoder trained on handwritten digits.

Autoencoders are rarely used for compression because we now have more efficient and
straightforward algorithms as jpeg, LZMA, and LZSS (used in WinRAR in conjunction with
Huffman coding). Autoencoders have been used for picture denoising and dimensionality
reduction in recent years. Image denoising is used to gain accurate information about the
image's content.
Reduction of Dimensionality

The autoencoders reduce the input to a reduced representation stored in the middle layer
called code. By separating this layer from the model, the information from the input has
been compressed, and each node can now be handled as a variable. As a result, we may
determine that by deleting the decoder, an autoencoder with the coding layer as the output
can be used for dimensionality reduction.

Extraction of Features

Autoencoders' encoding segment aids in the learning of critical hidden features present in
the input data, reducing the reconstruction error. A new set of unique feature combinations
is formed during the encoding process.

Image Production

The VAE (Variational Autoencoder) is a generative model used to produce images that the
model has not yet seen. The concept is that the system will generate similar images based
on input photographs such as faces or scenery. The purpose is to:

 Create new animated characters

 Create fictitious human images
 Colourization of an image

One of the purposes of autoencoders is to convert a black-and-white image to a colored

image. A colorful image can also be converted to grayscale.

Types of Autoencoders

Following are the different types of Autoencoders which are designed for specific
tasks with unique features:
1. Denoising Autoencoder
Denoising Autoencoder is trained to handle corrupted or noisy inputs, it learns to
remove noise and helps in reconstructing clean data. It prevent the network from
simply memorizing the input and encourages learning the core features.
2. Sparse Autoencoder
Sparse Autoencoder contains more hidden units than input features but only allows
a few neurons to be active simultaneously. This sparsity is controlled by zeroing
some hidden units, adjusting activation functions or adding a sparsity penalty to the
loss function.
3. Variational Autoencoder
Variational autoencoder (VAE) makes assumptions about the probability
distribution of the data and tries to learn a better approximation of it. It
uses stochastic gradient descent to optimize and learn the distribution of latent
variables. They used for generating new data such as creating realistic images or
text.
It assumes that the data is generated by a Directed Graphical Model and tries to
learn an approximation to qϕ(z∣x) to the conditional property

qθ(z∣x) where ϕ and θ are the parameters of the encoder and the decoder
respectively.
4. Convolutional Autoencoder
Convolutional autoencoder uses convolutional neural networks (CNNs) which
are designed for processing images. The encoder extracts features using
convolutional layers and the decoder reconstructs the image
through deconvolution also called as upsampling.

Stochastic Encoder
The word stochastic means random. So, a stochastic encoder introduces controlled
randomness in creating the compressed representation of input data. A Stochastic Encoder is a
type of encoder used in deep learning that does not produce a single fixed output for an input.
Instead, it produces a probability distribution and then samples the latent representation from
that distribution.
Function of Stochastic Encoder
When an input (image / text / audio) is given:

1. The encoder processes the input using a neural network.

2. Instead of giving one output vector, it outputs:
o Mean (μ) → the central value of the latent representation
o Standard Deviation (σ) → how much variation is allowed
3. A random noise (ε) is added to create the final latent vector z.

Formula

This step ensures the output is not fixed, but still meaningful.
 3. Purpose of Stochastic Encoder

 To prevent the model from memorizing the input.

 To make the latent space smooth and continuous.
 To allow the model to generate new samples by sampling different values of z.
 To represent uncertainty (how confident the model is about features).

Variational AutoEncoders

Variational Autoencoders (VAEs) are type of generative model in machine learning
that create new data similar to the input they are trained on. They not only compress
and reconstruct data like traditional autoencoders but also learn a continuous
probabilistic representation of the underlying features. This unique approach helps
VAEs to generate new, realistic data samples that closely resemble the original
input. In this article, we will see more about VAEs and its core concepts.

Architecture of Variational Autoencoder

VAE is a special kind of autoencoder that can generate new data instead of just
compressing and reconstructing it. It has three main parts:

1. Encoder (Understanding the Input)

The encoder takes input data like images or text and learns its key features. Instead
of outputting one fixed value, it produces two vectors for each feature:
 Mean (μ): A central value representing the data.
 Standard Deviation (σ): It is a measure of how much the values can vary.
These two values define a range of possibilities instead of a single number.

2. Latent Space (Adding Some Randomness)

Instead of encoding the input as one fixed point it pick a random point within the
range given by the mean and standard deviation. This randomness lets the model
 create slightly different versions of data which is useful for generating new, realistic
samples.

3. Decoder (Reconstructing or Creating New Data)

The decoder takes the random sample from the latent space and tries to reconstruct
the original input. Since the encoder gives a range, the decoder can produce new
data that is similar but not identical to what it has seen.

Variational Autoencoder

Where,

is a random standard normal samples, such as 0.5, -0.2 etc.

Applications of VAEs
 Generative modeling. The core advantage of VAEs is their ability to generate new
data samples that are similar to the training data but not identical to any specific
instance. For example, in image synthesis, VAEs can create new images that
 resemble the training set but with variations, making them useful for tasks like
creating new artwork, generating realistic faces, or producing new designs in fashion
and architecture.

 Anomaly detection. By learning the distribution of normal data, VAEs can identify
deviations from this distribution as anomalies. This is particularly useful in
applications like fraud detection, network security, and predictive maintenance.

 Data imputation and denoising. One of VAEs' strong points is reconstructing data
with missing or noisy parts. By sampling from the learned latent distribution, they are
able to predict and fill in missing values or remove noise from corrupted data. This
makes them valuable in applications such as medical imaging, where accurate data
reconstruction is essential, or in restoring corrupted audio and visual data.

 Semi-supervised learning. In semi-supervised learning scenarios, VAEs can

improve classifier performance by using the latent space to capture underlying data
structures, thereby enhancing the learning process with limited labeled data.

 Latent space manipulation. VAEs provide a structured and continuous latent

space that can be manipulated for various applications. For instance, in image
editing, specific features (like lighting or facial expressions) can be adjusted by
navigating the latent space. This feature is particularly useful in creative industries
for modifying and enhancing images and videos.

Denoising AutoEncoders(DAE)

Autoencoders are neural networks for unsupervised learning that compress input
data into a low-dimensional space (using an encoder) and then reconstruct it (using
a decoder), training the network to minimize the reconstruction error between the
original input and its reconstructed output. If the hidden layer is too large,
autoencoders may simply learn to replicate the input perfectly, functioning as an
identity mapping and failing to extract meaningful features.
 Denoising autoencoders address this by providing a deliberately noisy or
corrupted version of the input to the encoder, but still using the original, clean
input for calculating loss.
 This trains the model to learn useful, robust features and reduces the chance of
simply replicating the input.

Architecture of DAE

The denoising autoencoder (DAE) architecture resembles a standard autoencoder and

consists of two main components:
Encoder
 A neural network (one or more layers) that transforms noisy input data into a lower-
dimensional encoding.
 Noise can be introduced by adding Gaussian noise or randomly masking/missing
some inputs.
Decoder
 A neural network (one or more layers) that reconstructs the original data from the
encoding.
 The loss is calculated between the decoder’s output and the original clean input, not
the noisy one.
 DAE architecture

Generative Models
A generative model is a type of machine learning model that aims to learn
underlying patterns or distributions of data to generate new, similar data. This is
used in unsupervised machine learning to describe phenomena in data, enabling
computers to understand the real world. In this article, we will discuss some
applications and examples of generative models.
Generative modeling is the use of probability in artificial intelligence (AI), statistics,
and applications to create a representation or abstraction of observed phenomena
or target variables that can be computed from observations. These models are
capable of generating new data instances that are similar to your training data
Generative models aim to understand the underlying data distribution of the
training set and generate new samples from this distribution. They can learn the
joint probability distribution and can generate both input data X and target labels Y.
 Generative models learn the data distribution of an input training set, to
generate new data points similar to the initial training set. It means that these
models can understand and replicate the nuances of your data. From image
generation to natural language understanding and synthesis, this Generative
Model has a wide range of applications and is the basis of the latest generation of
AI systems powered by large language models (LLMs).

Importance of Generative Model in Artificial Intelligence

Artificial intelligence (AI) is rapidly transforming our world, and its importance is
undeniable. Generative models are a cornerstone of modern artificial intelligence
(AI), providing essential capabilities that drive innovation and expand the
boundaries of what AI systems can achieve. Their importance in AI stems from
their ability to model complex data distributions, generate new data, and enable a
wide range of applications that would otherwise be challenging or impossible.

Examples of Generative Models

Generative models are a class of models in machine learning that aim to model the
underlying distribution of data in order to generate new samples from that
distribution. Here are some common types of generative models:

1. Probabilistic Models:

Probabilistic models use probability distributions to represent the data. They aim to
estimate the joint probability of the observed data and the latent variables. Two
notable examples of probabilistic models are Bayesian Networks and Hidden
Markov Models.
 Bayesian Network: Bayesian Network is a graphical representation of the
various possible relationships between a given set of random variables. is a
classifier that has no dependence on features i.e. it is condition-independent
They are used to model uncertainty in complex domains by encoding
probabilistic relationships among variables. These networks are particularly
effective in scenarios where we need to understand causal relationships and
make decisions under uncertainty.

 Hidden Markov Models (HMM): Hidden Markov Models are statistical models
that represent systems that transition between states in a Markov process,
where the system's state is partially observable through a set of observations. It
is predicated on the idea that there is an underlying process with hidden states,
each of which has a known outcome. The probabilities of switching between
hidden states and emitting observable symbols are defined by the model.

2. Neural Network-Based Models

Neural network-based generative models leverage the power of deep learning to

capture intricate patterns in data. These are The two popular types in this category
are Generative Adversarial Network and Variational Autoencoders.
 Generative Adversarial Networks (GAN): This model is based on machine
learning and deep neural networks. Two stochastic neural networks – a
generator and a discriminator of the models compete against each other to
provide more accurate predictions and realistic information. Basically, A GAN is
an unsupervised learning technique that makes it possible to automatically find
and learn different patterns in input data. GANs are also used to create
incredibly lifelike renderings of various objects, people, and scenes that are
challenging for even a human brain to recognize as replicas.
 Variational Autoencoders (VAEs): VAEs are probabilistic models that learn to
encode and decode data. They learn a latent representation of the data and can
 generate new samples by sampling from the learned latent space. Like GAN,
VAE is a model is also based on a neural network autoencoder, which has a
combination of 2 neural networks – encoder and decoder.

3. Some Other type Model

Apart from the mainstream generative models, there are also some other types of
models available that provide unique approaches to data generation.
 Flow-Based Models: Flow-based models utilize invertible neural networks to
learn the exact likelihood of data. These models, such as RealNVP and Glow,
allow for both efficient sampling and exact log-likelihood computation.
 Energy-Based Models: Energy-based models define a scalar energy function
that assigns low energy to data points that resemble the training data and high
energy to unlikely data points. These models, including Boltzmann Machines
and their variants, focus on learning the energy landscape of the data
distribution.


Applications of Generative models

Generative models have a wide range of applications across various fields. Here's
a breakdown of some of the key areas :

1. Image Generation
 Generative Adversarial Networks (GANs), have revolutionized image
generation. GANs can produce highly realistic images that are often
indistinguishable from real photographs. Basically, it takes a Prompt from the
User to analyze the meaning of the query and pass it forward to creating an
image. These models are used in various applications, including creating
synthetic faces, landscapes, and objects for media, entertainment, and virtual
reality environments.
 Image-to-image translation involves converting images from one domain to
another, such as transforming sketches into photorealistic images or converting
daytime photos into nighttime scenes.
Examples models: Stable Diffusion, Midjourney,OpenAI DALL-E

2. Text Generation
 Generative models play a crucial role in natural language processing (NLP),
enabling machines to understand and generate human-like text. There are
some hugging face models such as Mistral 7b and LLama 2 7b that are used in
Natural Language Processing.
 Text Completion and Summarization: Text generation models are adept at
completing sentences or paragraphs based on a given prompt. They are also
used for text summarization, where they condense long documents into concise
summaries while preserving the original meaning.
Examples models: Google PaLM, Meta LLaMA,OpenAI GPT-4, Mistral, Zephyr

3. Audio and Music Generation

 Generative models have significantly improved speech synthesis, allowing the
creation of highly natural and expressive synthetic speech. Models like WaveNet
and Tacotron produce high-quality speech that is used in virtual assistants,
audiobooks, and voice-overs.
 In the music industry, generative models can compose original pieces by learning
from existing music datasets. Models like Musenet and Jukedeck create music
across genres and styles, help musicians compose, and provide background
scores for multimedia content.
Examples models: BachBot, WaveNet

4. Data Augmentation
 Enhancing Training Dataset: Generative models are used to augment training
datasets, especially in scenarios where obtaining large amounts of labeled data
is challenging. By generating synthetic data that resembles real-world samples,
these models improve the performance of machine learning algorithm.
 Creating synthetic data: Model can create entirely synthetic datasets that
simulate real-world distributions. This capability is critical for tasks that require
privacy-preserving data, such as in healthcare and finance, where synthetic
data can be used to train models without revealing sensitive information.
Examples models: StyleGAN

5. Healthcare Applications
 Generative models are transforming medical image analysis by enhancing and
generating medical images. It can be used to analyze medical images like X-
rays or MRIs, potentially aiding in early disease detection or treatment planning.
These applications assist in disease diagnosis, treatment planning, and medical
research.
 In drug discovery, models are used to design new molecules with desired
properties. Models like Variational Autoencoders (VAEs) and GANs help in
exploring the vast chemical space, predicting molecular properties, and
generating novel compounds.

Challenges and Limitations of the Generative Model

While generative models offer a wide range of capabilities, they also face several
challenges:
Computational complexity of training: Generative models, especially sophisticated
models such as GANs, require significant computational resources and time. Training
them requires powerful hardware and can be resource-intensive.
 Quality of output generated: Output generated from generative models may not
always be accurate or error-free. This can be due to a number of factors, including
insufficient data, insufficient training, or overly complex models
 Security: Generative AI systems can be used to fake information or propaganda by
creating realistic and believable fake videos, images and text
 Trustworthy Concern The ability of generative models to generate realistic
content raises ethical issues, especially in the creation of deep imitation or fake
content. Ensuring responsible use is paramount to prevent abuse or fraud.
 Data dependencies: The quality of the output generated depends heavily on the
quality of the training data. If the training data is biased or unrepresentative, the
model output will reflect those biases.

Boltzmann machine

A Boltzmann machine is an unsupervised deep learning model in which every node is

connected to every other node. It is a type of recurrent neural network, and the nodes
make binary decisions with some level of bias.

These machines are not deterministic deep learning models, they are stochastic
or generative deep learning models. They are representations of a system.

A Boltzmann machine has two kinds of nodes

 Visible nodes: These are nodes that can be measured and are measured.
 Hidden nodes: These are nodes that cannot be measured or are not measured.

a Boltzmann machine can be called a stochastic Hopfield network which has hidden
units. It has a network of units with an ‘energy’ defined for the overall network.

Boltzmann machines seek to reach thermal equilibrium. It essentially looks to optimize

global distribution of energy. But the temperature and energy of the system are relative
to laws of thermodynamics and are not literal.

A Boltzmann machine is made up of a learning algorithm that enables it to discover

interesting features in datasets composed of binary vectors. The learning algorithm
tends to be slow in networks that have many layers of feature detectors but it is possible
to make it faster by implementing a learning layer of feature detectors.
They use stochastic binary units to reach probability distribution equilibrium (to minimize
energy). It is possible to get multiple Boltzmann machines to collaborate together to
form far more sophisticated systems like deep belief networks.

The Boltzmann machine is named after Ludwig Boltzmann, an Austrian scientist who
came up with the Boltzmann distribution. However, this type of network was first
developed by Geoff Hinton, a Stanford Scientist.

Boltzmann machine

Boltzmann distribution

The Boltzmann distribution is a probability distribution that gives the probability of a

system being in a certain state as a function of that state's energy and the temperature
of the system.

It was formulated by Ludwig Boltzmann in 1868 and is also known as the Gibbs
distribution.
Uses of Boltzmann Machines
The main aim of a Boltzmann machine is to optimize the solution of a problem. To do
this, it optimizes the weights and quantities related to the specific problem that is
assigned to it. This technique is employed when the main aim is to create mapping and
to learn from the attributes and target variables in the data. If you seek to identify an
underlying structure or the pattern within the data, unsupervised learning methods for
this model are regarded to be more useful. Some of the most widely used unsupervised
learning methods are clustering, dimensionality reduction, anomaly detection and
creating generative models.

All of these techniques have a different objective of detecting patterns like identifying
latent grouping, finding irregularities in the data, or even generating new samples from
the data that is available. You can even stack these networks in layers to build deep
neural networks that capture highly complicated statistics. Restricted Boltzmann
machines are widely used in the domain of imaging and image processing as well
because they have the ability to model continuous data that are common to natural
images. They are even used to solve complicated quantum mechanical many-particle
problems or classical statistical physics problems like the Ising and Potts classes of
models.

Working Principle of Boltzmann Machine

Boltzmann machines are non-deterministic (stochastic) generative Deep
Learning models that only have two kinds of nodes - hidden and visible nodes. They
don’t have any output nodes, and that’s what gives them the non-deterministic feature.
They learn patterns without the typical 1 or 0 type output through which patterns are
learned and optimized using Stochastic Gradient Descent.

A major difference is that unlike other traditional networks (A/C/R) which don’t have any
connections between the input nodes, Boltzmann Machines have connections among
the input nodes. Every node is connected to all other nodes irrespective of whether they
are input or hidden nodes. This enables them to share information among themselves
and self-generate subsequent data. You’d only measure what’s on the visible nodes
and not what’s on the hidden nodes. After the input is provided, the Boltzmann
machines are able to capture all the parameters, patterns and correlations among the
data. It is because of this that they are known as deep generative models and they fall
into the class of Unsupervised Deep Learning.

Types of Boltzmann machines

There are three types of Boltzmann machines. These are:

 Restricted Boltzmann Machines (RBMs)

 Deep Belief Networks (DBNs)
 Deep Boltzmann Machines (DBMs)

1. Restricted Boltzmann Machines (RBMs)

While in a full Boltzmann machine all the nodes are connected to each other and the
connections grow exponentially, an RBM has certain restrictions with respect to node
connections.
 In a Restricted Boltzmann Machine, hidden nodes cannot be connected to each other
while visible nodes are connected to each other.

2. Deep Belief Networks (DBNs)

In a Deep Belief Network, you could say that multiple Restricted Boltzmann Machines
are stacked, such that the outputs of the first RBM are the inputs of the subsequent
RBM. The connections within individual layers are undirected, while the connections
between layers are directed. However, there is an exception here. The connection
between the top two layers is undirected.

A deep belief network can either be trained using a Greedy Layer-wise Training
Algorithm or a Wake-Sleep Algorithm.

3. Deep Boltzmann Machines (DBMs)

Deep Boltzmann Machines are very similar to Deep Belief Networks. The difference
between these two types of Boltzmann machines is that while connections between
layers in DBNs are directed, in DBMs, the connections within layers, as well as the
connections between the layers, are all undirected.

Deep Belief Network (DBN)


Deep Belief Networks (DBNs) are sophisticated artificial neural networks used in the
field of deep learning, a subset of machine learning. They are designed to discover
and learn patterns within large sets of data automatically. Imagine them as multi-
layered networks, where each layer is capable of making sense of the information
received from the previous one, gradually building up a complex understanding of
the overall data.
DBNs are composed of multiple layers of stochastic, or randomly determined, units.
These units are known as Restricted Boltzmann Machines (RBMs) or other similar
structures. Each layer in a DBN aims to extract different features from the input data,
with lower layers identifying basic patterns and higher layers recognizing more
abstract concepts. This structure allows DBNs to effectively learn complex
representations of data, which makes them particularly useful for tasks like image
and speech recognition, where the input data is high-dimensional and requires a
deep level of understanding.
The architecture of DBNs also makes them good at unsupervised learning, where
the goal is to understand and label input data without explicit guidance. This
characteristic is particularly useful in scenarios where labelled data is scarce or
when the goal is to explore the structure of the data without any preconceived
labels.

Working Principle of Deep Belief Networks

DBNs work in two main phases: pre-training and fine-tuning. In the pre-training phase,
the network learns to represent the input data layer by layer. Each layer is trained
independently as an RBM, which allows the network to learn complex data
representations efficiently. During this phase, the network learns the probability
distribution of the inputs, which helps it understand the underlying structure of the data.
In the fine-tuning phase, the DBN adjusts its parameters for a specific task, like
classification or regression. This is typically done using a technique known as
backpropagation, where the network’s performance on a task is evaluated, and the
errors are used to update the network’s parameters. This phase often involves
supervised learning, where the network is trained with labelled data.

Concepts Related to Deep Belief Networks (DBNs)

 Restricted Boltzmann Machines (RBMs): These are the building blocks of DBNs.
An RBM is a two-layered neural network that learns the probability distribution of the
input data. Each layer in a DBN is typically an RBM.
  Stochastic Units: DBNs use units that make decisions probabilistically. This
stochastic nature allows the network to explore and learn more complex patterns in
the data.
 Layer-wise Training: DBNs are trained one layer at a time, which is efficient and
helps in learning deep representations of data.
 Unsupervised and Supervised Learning: DBNs are versatile, capable of both
unsupervised learning (learning from unlabeled data) and supervised learning
(learning from labeled data).
 Greedy Algorithm: This is used during the pre-training phase of DBNs. Each layer
is trained greedily, meaning it’s trained independently of the others, which simplifies
the training process.
 Backpropagation: In the fine-tuning phase, backpropagation is used for supervised
learning tasks. It adjusts the network’s parameters to improve its performance on
specific tasks.


DBNs, with their deep architecture and efficient learning capabilities, have been pivotal
in advancing the field of deep learning, particularly in handling complex tasks like image
and speech recognition.

Generative Adversarial Network (GAN)


Generative Adversarial Networks (GAN) help machines to create new, realistic data
by learning from existing examples. It is introduced by Ian Goodfellow and his team
in 2014 and they have transformed how computers generate images, videos, music
and more. Unlike traditional models that only recognize or classify data, they take a
creative way by generating entirely new content that closely resembles real-world
data. This ability helped various fields such as art, gaming, healthcare and data
science. In this article, we will see more about GANs and its core concepts.
Architecture of GAN

GAN consist of two main models that work together to create realistic synthetic data
which are as follows:

1. Generator Model

The generator is a deep neural network that takes random noise as input to
generate realistic data samples like images or text. It learns the underlying data
patterns by adjusting its internal parameters during training
through backpropagation. Its objective is to produce samples that the discriminator
classifies as real.

Generator Loss Function: The generator tries to minimize this loss:

where
 JG measure how well the generator is fooling the discriminator.
 G(zi) is the generated sample from random noise zi
 D(G(zi)) is the discriminator’s estimated probability that the generated sample is
real.
The generator aims to maximize D(G(zi)) meaning it wants the discriminator to
classify its fake data as real (probability close to 1).

2. Discriminator Model

The discriminator acts as a binary classifier helps in distinguishing between real and
generated data. It learns to improve its classification ability through training, refining
its parameters to detect fake samples more accurately. When dealing with image
data, the discriminator uses convolutional layers or other relevant architectures
which help to extract features and enhance the model’s ability.
Discriminator Loss Function: The discriminator tries to minimize this loss:

 JD measures how well the discriminator classifies real and fake samples.
 xi is a real data sample.
 G(zi) is a fake sample from the generator.
 D(xi) is the discriminator’s probability that xi is real.
 D(G(zi)) is the discriminator’s probability that the fake sample is real.
The discriminator wants to correctly classify real data as real (maximize
logD(xi) and fake data as fake (maximize log(1−D(G(zi)))

MinMax Loss

GANs are trained using a MinMax Loss between the generator and discriminator:

where,

 G is generator network and is D is the discriminator network

 pdata(x) = true data distribution
 pz(z)= distribution of random noise (usually normal or uniform)
 D(x) = discriminator’s estimate of real data
 D(G(z))= discriminator’s estimate of generated data

The generator tries to minimize this loss (to fool the discriminator) and the
discriminator tries to maximize it (to detect fakes accurately).
 GAN

Working of GAN
GAN train by having two networks the Generator (G) and the Discriminator (D)
compete and improve together. Here's the step-by-step process

1. Generator's First Move

The generator starts with a random noise vector like random numbers. It uses this
noise as a starting point to create a fake data sample such as a generated image.
The generator’s internal layers transform this noise into something that looks like
real data.

2. Discriminator's Turn

The discriminator receives two types of data:

 Real samples from the actual training dataset.
 Fake samples created by the generator.
D's job is to analyze each input and find whether it's real data or something G
cooked up. It outputs a probability score between 0 and 1. A score of 1 shows the
data is likely real and 0 suggests it's fake.

3. Adversarial Learning

 If the discriminator correctly classifies real and fake data it gets better at its job.
 If the generator fools the discriminator by creating realistic fake data, it receives a
positive update and the discriminator is penalized for making a wrong decision.

4. Generator's Improvement

 Each time the discriminator mistakes fake data for real, the generator learns from
this success.
 Through many iterations, the generator improves and creates more convincing
fake samples.

5. Discriminator's Adaptation

 The discriminator also learns continuously by updating itself to better spot fake
data.
 This constant back-and-forth makes both networks stronger over time.


6. Training Progression
 As training continues, the generator becomes highly proficient at producing
realistic data.
 Eventually the discriminator struggles to distinguish real from fake shows that the
GAN has reached a well-trained state.
 At this point, the generator can produce high-quality synthetic data that can be
used for different applications.

Types of GAN

There are several types of GANs each designed for different purposes. Here are
some important types:

1. Vanilla GAN

Vanilla GAN is the simplest type of GAN. It consists of:

 A generator and a discriminator both are built using multi-layer perceptrons
(MLPs).
 The model optimizes its mathematical formulation using stochastic gradient
descent (SGD).
While foundational, Vanilla GAN can face problems like:
 Mode collapse: The generator produces limited types of outputs repeatedly.
 Unstable training: The generator and discriminator may not improve smoothly.

2. Conditional GAN (CGAN)

Conditional GAN (CGAN) adds an additional conditional parameter to guide the

generation process. Instead of generating data randomly they allow the model to
produce specific types of outputs.
Working of CGANs:
 A conditional variable (y) is fed into both the generator and the discriminator.
 This ensures that the generator creates data corresponding to the given condition
(e.g generating images of specific objects).
 The discriminator also receives the labels to help distinguish between real and
fake data.
Example: Instead of generating any random image, CGAN can generate a specific
object like a dog or a cat based on the label.
3. Deep Convolutional GAN (DCGAN)

Deep Convolutional GAN (DCGAN) are among the most popular types of GANs
used for image generation.
They are important because they:
 Uses Convolutional Neural Networks (CNNs) instead of simple multi-layer
perceptrons (MLPs).
 Max pooling layers are replaced with convolutional stride helps in making the
model more efficient.
 Fully connected layers are removed, which allows for better spatial understanding
of images.
DCGANs are successful because they generate high-quality, realistic images.

4. Laplacian Pyramid GAN (LAPGAN)

Laplacian Pyramid GAN (LAPGAN) is designed to generate ultra-high-quality

images by using a multi-resolution approach.
Working of LAPGAN:
 Uses multiple generator-discriminator pairs at different levels of the Laplacian
pyramid.
 Images are first down sampled at each layer of the pyramid and upscaled again
using Conditional GAN (CGAN).
 This process allows the image to gradually refine details and helps in reducing
noise and improving clarity.
Due to its ability to generate highly detailed images, LAPGAN is considered a
superior approach for photorealistic image generation.
5. Super Resolution GAN (SRGAN)

Super-Resolution GAN (SRGAN) is designed to increase the resolution of low-

quality images while preserving details.
Working of SRGAN:
 Uses a deep neural network combined with an adversarial loss function.
 Enhances low-resolution images by adding finer details helps in making them
appear sharper and more realistic.
 Helps to reduce common image upscaling errors such as blurriness and
pixelation.