UNSUPERVISED DEEP LEARNING

Auto encoders – Standard – Sparse – De noising – Contractive – Variational Auto encoders –

Adversarial Generative Networks – Deep Boltzmann Machine (DBM)

Unsupervised learning is an intriguing area of machine learning that reveals hidden structures and patterns in
data without requiring labelled samples. Because it investigates the underlying relationships in data, it's an
effective tool for tasks like anomaly identification, dimensionality reduction, and clustering. There are several
uses for unsupervised learning in domains like computer vision, natural language processing, and data
analysis. Through self-sufficient data interpretation, it provides insightful information that enhances decision-
making and facilitates comprehension of intricate data patterns.

Unsupervised Neural Network

An unsupervised neural network is a type of artificial neural network (ANN) used in unsupervised learning
tasks. Unlike supervised neural networks, trained on labelled data with explicit input-output pairs,
unsupervised neural networks are trained on unlabelled data. In unsupervised learning, the network is not
under the guidance of features. Instead, it is provided with unlabelled data sets (containing only the input data)
and left to discover the patterns in the data and build a new model from it. Here, it has to figure out how to
arrange the data by exploiting the separation between clusters within it. These neural networks aim to discover
patterns, structures, or representations within the data without specific guidance.

There are several components of unsupervised learning. They are:

1. Encoder-Decoder: As the name itself suggests that it is used to encode and decode the data. Encoder
basically responsible for transforming the input data into lower dimensional representation on which
the neural network works. Whereas decoder takes the encoded representation and reconstruct the input
data from it. There architecture and parameters are learned during the training of the network.

2. Latent Space: It is the immediate representation created by the encoder. It contains the abstract
representation or features that captures important information about the data's structures. It is also
known as the latent space.

3. Training algorithm: Unsupervised neural network model use specific training algorithms to get the
parameters. Some of the common optimization algorithms are Stochastic gradient descent, Adam etc.
They are used depending on the type of model and loss function.

4. Loss Function: It is a common component among all the machine learning models. It basically
calculates the model's output and the actual/measured output. It quantifies how well the model
understands the data.
At the heart of deep learning lies the neural network, an intricate interconnected system of nodes that mimics
the human brain’s neural architecture. Neural networks excel at discerning intricate patterns and
representations within vast datasets, allowing them to make predictions, classify information, and generate
novel insights. Autoencoders emerge as a fascinating subset of neural networks, offering a unique approach
to unsupervised learning. Autoencoders are an adaptable and strong class of architectures for the dynamic field
of deep learning, where neural networks develop constantly to identify complicated patterns and
representations. With their ability to learn effective representations of data, these unsupervised learning
models have received considerable attention and are useful in a wide variety of areas, from image processing
to anomaly detection.

What are Autoencoders?

Autoencoders are a specialized class of algorithms that can learn efficient representations of input data with
no need for labels. It is a class of artificial neural networks designed for unsupervised learning. Learning to
compress and effectively represent input data without specific labels is the essential principle of an automatic
decoder. This is accomplished using a two-fold structure that consists of an encoder and a decoder. The encoder
transforms the input data into a reduced-dimensional representation, which is often referred to as “latent space”
or “encoding”. From that representation, a decoder rebuilds
the initial input. For the network to gain meaningful patterns
in data, a process of encoding and decoding facilitates the
definition of essential features.

Architecture of Autoencoder in Deep Learning

The general architecture of an autoencoder includes an

encoder, decoder, and bottleneck layer.

Architecture of Autoencoder in Deep Learning

The general architecture of an autoencoder includes an encoder, decoder, and bottleneck layer.

Encoder

➢ Input layer take raw input data

➢ The hidden layers progressively reduce the dimensionality of the input, capturing important features
and patterns. These layers compose the encoder.
➢ The bottleneck layer (latent space) is the final hidden layer, where the dimensionality is significantly
reduced. This layer represents the compressed encoding of the input data.

Decoder
 ➢ The bottleneck layer takes the encoded representation and expands it back to the dimensionality of the
original input.
➢ The hidden layers progressively increase the dimensionality and aim to reconstruct the original input.
➢ The output layer produces the reconstructed output, which ideally should be as close as possible to the
input data.

The loss function used during training is typically a reconstruction loss, measuring the difference between
the input and the reconstructed output. Common choices include mean squared error (MSE) for continuous
data or binary cross-entropy for binary data.

During training, the autoencoder learns to minimize the reconstruction loss, forcing the network to capture
the most important features of the input data in the bottleneck layer.

After the training process, only the encoder part of the autoencoder is retained to encode a similar type of
data used in the training process. The different ways to constrain the network are.

SPARSE:

Sparse autoencoders are a specialized type of autoencoder used in deep learning, primarily for feature learning
and dimensionality reduction. They introduce a sparsity constraint that encourages the model to learn a
compact representation of the input data.

In a standard autoencoder, the network learns to encode and decode data without any constraints on the hidden
layer's activations. Sparse autoencoders modify this behavior by adding a sparsity constraint, which forces the
hidden units to activate only a small number of neurons at a time. This encourages the network to discover
more meaningful and interpretable features.

Mathematical Formulation

The objective function for a sparse autoencoder can be expressed as:

Sparsity Constraint

• The sparsity constraint can be enforced through various techniques:

• L1 Regularization: Adds a penalty proportional to the absolute values of the weights.
• KL Divergence: Measures the difference between the average activation of the hidden neurons and a
target sparsity level.

Denoising Autoencoders

Autoencoders are types of neural network architecture used for unsupervised learning. The architecture
consists of an encoder and a decoder. The encoder encodes the input data into a lower dimensional space
while the decoder decodes the encoded data back to the original input. The network is trained to minimize
the difference between decoded data and input. Autoencoders have the risk of becoming an Identify
function meaning the output equals the input which makes the whole neural network of autoencoders
useless. This generally happens when there are more nodes in the hidden layer than there are inputs.

Denoising Autoencoder (DAE)

Now, a denoising autoencoder is a modification of the original autoencoder in which instead of giving the
original input we give a corrupted or noisy version of input to the encoder while decoder loss is calculated
concerning original input only. This results in efficient learning of autoencoders and the risk of
autoencoder becoming an identity function is significantly reduced.

Architecture of DAE

The denoising autoencoder (DAE) architecture resembles a standard autoencoder and consists of two main
components:

Encoder:

• The encoder is a neural network with one or more hidden layers.

• It receives noisy input data instead of the original input and generates an encoding in a low-dimensional
space.

• There are several ways to generate a corrupted input. The most common being adding a Gaussian noise
or randomly masking some of the inputs.

Decoder:

• Similar to encoders, decoders are implemented as neural networks with one or more hidden layers.

• It takes the encoding generated by the encoder as input and reconstructs the original data.

• When calculating the Loss function it compares the output values with the original input, not with the
corrupted input.

What DAE Learns?

The above architecture of using a corrupted input helps decrease the risk of overfitting and prevents the
DAE from becoming an identity function.

• If DAEs are trained with partially corrupted inputs (e.g., with masking values), they learn to impute or
fill in missing information during the reconstruction process. This makes them useful for tasks
involving incomplete datasets.

• If DAEs are trained with partially noisy inputs (gaussian noise) DAEs tend to generalize well to
unseen, real-world data with different levels of noise or corruption as they learn to extract robust
features. This is beneficial in various applications where data quality is compromised, such as image
denoising or signal processing.

Objective Function of DAE

The objective of DAE is to minimize the difference between the original input (clean input without the
notice) and the reconstructed output. This is quantified using a reconstruction loss function. Two types of
loss function are generally used depending on the type of input data.

Mean Squared Error (MSE):

If we have input image data in the form of floating pixel values i.e. values between (0 to 1) or (0 to
255) we use mse

Here,

• each of xi is the pixel value of input data

• yi is the pixel value of reconstructed data

o yi = D(E(xi*noise) )

o Where E represents encoder and D represents decoder

• this is summed over all the training set

Binary Cross-Entropy (log-loss):

If we have input image data in the form of bits pixel values i.e. values will be either 0 or 1 only then we
can use binary cross entropy loss for each pixel value
Here

• each of xi is the pixel value of input data with value being only 0 or 1

• yi is the pixel value of reconstructed data.

o yi = D(E(xi*noise))

• Where E represents encoder and D represents decoder

• this is summed over all the training set

Training Process of DAE

The training of DAE consists of below steps:

• Initialze ender and decoer with random weights

• Noise is intentionally added to the input data.

• Feedforward the input data through encoder and decoder to get the reconstructed image

• Calculate the reconstruction loss as defined in our objective function w

• Do backprogoagation and update weights. The goal during training is to minimize the reconstruction
loss.

The training is typically done through optimization algorithms like stochastic gradient descent (SGD)
or its variants.

Applications of DAE

• Image Denoising: DAEs are widely employed for cleaning and enhancing images by removing noise.

• Audio Denoising: DAEs can be applied to denoise audio signals, making them valuable in speech-
enhancement tasks.

• Sensor Data Processing: DAEs are valuable in processing sensor data, removing noise, and extracting
relevant information from sensor readings.

• Data Compression: Autoencoders, including DAEs, can be utilized for data compression by learning
compact representations of input data.

• Feature Learning: DAEs are effective in unsupervised feature learning, capturing relevant features in
the data without explicit labels.
 Implementation of DAE

Let us implement DAE in PyTorch for MNIST dataset.

1. Import Libraries

• torch. [Link] provides tools for working with datasets and data loaders in PyTorch.

• torch-vision is a PyTorch library specifically designed for computer vision tasks. datasets contain
popular datasets (like MNIST, CIFAR-10, etc.), and transforms provide image transformations and
preprocessing functions.

• nn provides building blocks for constructing neural network architectures, and optim includes
optimization algorithms (like SGD, Adam, etc.) for training neural networks.

• If a GPU is available, it sets the device variable to 'cuda'; otherwise, it sets it to 'CPU'
• The first row is the corrupted image

• The second row is the reconstructed image

• The last row is the actual image before addition of noise

CONTRACTIVE

What is Contarctive AutoEncoder?

Contractive Autoencoder was proposed by researchers at the University of Toronto in 2011 in the paper
Contractive auto-encoders: Explicit invariance during feature extraction. The idea behind that is to make
the autoencoders robust to small changes in the training dataset.

To deal with the above challenge that is posed by basic autoencoders, the authors proposed adding another
penalty term to the loss function of autoencoders. We will discuss this loss function in detail.

Loss Function of Contactive AutoEncoder

Contractive autoencoder adds an extra term in the loss function of autoencoder, it is given as:

i.e. the above penalty term is the Frobenius Norm of the encoder, the Frobenius norm is just a
generalization of the Euclidean norm.
In the above penalty term, we first need to calculate the Jacobian matrix of the hidden layer, calculating a
Jacobian of the hidden layer with respect to input is similar to gradient calculation. Let’s first calculate the
Jacobian of the hidden layer:

where \phi is non-linearity. Now, to get the jth hidden unit, we need to get the dot product of the ith feature
vector and the corresponding weight. For this, we need to apply the chain rule.

The above method is similar to how we calculate the gradient descent, but there is one major difference,
that is we take h(X) as a vector-valued function, each as a separate output. Intuitively, For example, we
have 64 hidden units, then we have 64 function outputs, and so we will have a gradient vector for each of
that 64 hidden units.

Let diag(x) be the diagonal matrix, the matrix from the above derivative is as follows:

Now, we place the diag(x) equation to the above equation and simplify:

Relationship with Sparse Autoencoder

In sparse autoencoder, our goal is to have the majority of components of representation close to 0, for this
to happen, they must be lying in the left saturated part of the sigmoid function, where their corresponding
sigmoid value is close to 0 with a very small first derivative, which in turn leads to the very small entries
in the Jacobian matrix. This leads to highly contractive mapping in the sparse autoencoder, even though
this is not the goal in sparse Autoencoder.

Relationship with Denoising Autoencoder

The idea behind denoising autoencoder is just to increase the robustness of the encoder to the small
changes in the training data which is quite similar to the motivation of Contractive Autoencoder. However,
there is some difference:

• CAEs encourage robustness of representation f(x), whereas DAEs encourage robustness of

reconstruction, which only partially increases the robustness of representation.

• DAE increases its robustness by stochastically training the model for the reconstruction, whereas CAE
increases the robustness of the first derivative of the Jacobian matrix.

VARIATIONAL AUTO ENCODERS

Variational Autoencoders (VAEs) are generative models in machine learning (ML) that create new data
similar to the input they are trained on. Along with data generation they also perform common autoencoder
tasks like denoising. Like all autoencoders VAEs consist of:

• Encoder: Learns important patterns (latent variables) from input data.

• Decoder: It uses those latent variables to reconstruct the input.

Unlike traditional autoencoders that encode a fixed representation VAEs learn a continuous probabilistic
representation of latent space. This allows them to reconstruct input data accurately and generate new data
samples that resemble

the original input.

Architecture of Variational Autoencoder

VAE is a special kind of autoencoder that can generate new data instead of just compressing and
reconstructing it. It has three main parts:

1. Encoder (Understanding the Input)

• The encoder takes the input data like an image or text and tries to understand its most important
features.

• Instead of creating a fixed compressed version like a normal autoencoder it creates two things:

o Mean (μ): A central value representing the data.

o Standard Deviation (σ): It is a measure of how much the values can vary.
• These two values define a range of possibilities instead of a single number.

2. Latent Space (Adding Some Randomness)

• Instead of encoding input into a fixed number VAEs introduce randomness to create variations.

• The model picks a point from the range to create different variations of the data.

• This is what makes VAEs great for generating new slightly different but realistic data.

3. Decoder (Reconstructing or Creating New Data)

• The decoder takes this sampled value and tries to reconstruct the original input.

• Since the encoder creates a range of possibilities instead of a fixed number the decoder can generate
new similar data instead of just memorizing the input.

Mathematics behind Variational Autoencoder

Variational autoencoder uses KL-divergence as its loss function the goal of this is to minimize the
difference between a supposed distribution and original distribution of dataset.

Suppose we have a distribution z and we want to generate the observation x from it. In other words we
want to calculate p(z∣x) p(z∣x)
We can do it by following way:

But, the calculation of p(x) can be quite difficult:

 This usually makes it an intractable distribution. Hence, we need to approximate p(z|x) to q(z|x) to make
it a tractable distribution. To better approximate p(z|x) to q(z|x), we will minimize the KL-divergence loss
which calculates how similar two distributions are:

By simplifying, the above minimization problem is equivalent to the following maximization problem :

Implementing Variational Autoencoder

In this implementation we will be using the Fashion-MNIST dataset this dataset is already available
in keras. datasets API so we don’t need to add or upload manually.

Adversarial Generative Networks:

GAN(Generative Adversarial Network) represents a cutting-edge approach to generative modeling within

deep learning, often leveraging architectures like convolutional neural networks. The goal of generative
modeling is to autonomously identify patterns in input data, enabling the model to produce new examples
that feasibly resemble the original dataset.

Generative Adversarial Networks (GANs) are a powerful class of neural networks that are used for
an unsupervised learning. GANs are made up of two neural networks, a discriminator and a
generator. They use adversarial training to produce artificial data that is identical to actual data.

• The Generator attempts to fool the Discriminator, which is tasked with accurately distinguishing
between produced and genuine data, by producing random noise samples.

• Realistic, high-quality samples are produced as a result of this competitive interaction, which drives
both networks toward advancement.

• GANs are proving to be highly versatile artificial intelligence tools, as evidenced by their extensive
use in image synthesis, style transfer, and text-to-image synthesis.

• They have also revolutionized generative modeling.

Through adversarial training, these models engage in a competitive interplay until the generator becomes
adept at creating realistic samples, fooling the discriminator approximately half the time.

Generative Adversarial Networks (GANs) can be broken down into three parts:

• Generative: To learn a generative model, which describes how data is generated in terms of a
probabilistic model.

• Adversarial: The word adversarial refers to setting one thing up against another. This means that, in
the context of GANs, the generative result is compared with the actual images in the data set. A
mechanism known as a discriminator is used to apply a model that attempts to distinguish between real
and fake images.

• Networks: Use deep neural networks as artificial intelligence (AI) algorithms for training purposes.

Types of GANs

1. Vanilla GAN: This is the simplest type of GAN. Here, the Generator and the Discriminator are simple
a basic multi-layer perceptrons. In vanilla GAN, the algorithm is really simple, it tries to optimize the
mathematical equation using stochastic gradient descent.

2. Conditional GAN (CGAN): CGAN can be described as a deep learning method in which some
conditional parameters are put into place.

• In CGAN, an additional parameter ‘y’ is added to the Generator for generating the
corresponding data.
 • Labels are also put into the input to the Discriminator in order for the Discriminator to help
distinguish the real data from the fake generated data.

3. Deep Convolutional GAN (DCGAN): DCGAN is one of the most popular and also the most
successful implementations of GAN. It is composed of ConvNets in place of multi-layer perceptrons.
The ConvNets are implemented without max pooling, which is in fact replaced by
convolutional stride.
Also, the layers are not fully connected.
4. Laplacian Pyramid GAN (LAPGAN): The Laplacian pyramid is a linear invertible image
representation consisting of a set of band-pass images, spaced an octave apart, plus a low-frequency
residual.
This approach uses multiple numbers of Generator and Discriminator networks and
different levels of the Laplacian Pyramid.
5. This approach is mainly used because it produces very high-quality images. The image is down-
sampled at first at each layer of the pyramid and then it is again up-scaled at each layer in a backward
pass where the image acquires some noise from the Conditional GAN at these layers until it reaches
its original size.
6. Super Resolution GAN (SRGAN): SRGAN as the name suggests is a way of designing a GAN in
which a deep neural network is used along with an adversarial network in order to produce higher-
resolution images. This type of GAN is particularly useful in optimally up-scaling native low-
resolution images to enhance their details minimizing errors while doing so.

Architecture of GANs
A Generative Adversarial Network (GAN) is composed of two primary parts, which are the Generator
and the Discriminator.

Generator Model
A key element responsible for creating fresh, accurate data in a Generative Adversarial Network
(GAN) is the generator model. The generator takes random noise as input and converts it into complex
data samples, such text or images. It is commonly depicted as a deep neural network.

The training data’s underlying distribution is captured by layers of learnable parameters in its design
through training. The generator adjusts its output to produce samples that closely mimic real data as it
is being trained by using backpropagation to fine-tune its parameters.
The generator’s ability to generate high-quality, varied samples that can fool the discriminator is what
makes it successful.

Generator Loss
The objective of the generator in a GAN is to produce synthetic samples that are realistic enough to
fool the discriminator. The generator achieves this by minimizing its loss function JG. The loss is
minimized when the log probability is maximized, i.e., when the discriminator is highly likely to
classify the generated samples as real. The following equation is given below:

JG. measure how well the generator is fooling the discriminator.

represents log probability of the discriminator being correct for generated samples.
The generator aims to minimize this loss, encouraging the production of samples that the discriminator

classifies as real

Discriminator Model
An artificial neural network called a discriminator model is used in Generative Adversarial Networks
(GANs) to differentiate between generated and actual input. By evaluating input samples and
allocating probability of authenticity, the discriminator functions as a binary [Link] time, the
discriminator learns to differentiate between genuine data from the dataset and artificial samples
created by the generator. This allows it to progressively hone its parameters and increase its level of
proficiency.
Convolutional layers or pertinent structures for other modalities are usually used in its architecture
when dealing with picture data. Maximizing the discriminator’s capacity to accurately identify
generated samples as fraudulent and real samples as authentic is the aim of the adversarial training
procedure. The discriminator grows increasingly discriminating as a result of the generator and
discriminator’s interaction, which helps the GAN produce extremely realistic-looking synthetic data
overall.
Discriminator Loss
The discriminator reduces the negative log likelihood of correctly classifying both produced and real
samples. This loss incentivizes the discriminator to accurately categorize generated samples as fake
and real samples with the following equation:

• JD assesses the discriminator’s ability to discern between produced and actual samples.
 • The log likelihood that the discriminator will accurately categorize real data is represented by
logD(xi).
• The log chance that the discriminator would correctly categorize generated samples as fake is

represented by
• The discriminator aims to reduce this loss by accurately identifying artificial and real samples.
MinMax Loss:
In a Generative Adversarial Network (GAN), the minimax loss formula is provided by:

Where,
• G is generator network and is D is the discriminator network
• Actual data samples obtained from the true data distribution Pdata (x) are represented by x.
• Random noise sampled from a previous distribution pz(z) (z)(usually a normal or uniform distribution)
is represented by z.
• D(x) represents the discriminator’s likelihood of correctly identifying actual data as real.
• D(G(z)) is the likelihood that the discriminator will identify generated data coming from the generator
as authentic.

Application Of Generative Adversarial Networks (GANs)

GANs, or Generative Adversarial Networks, have many uses in many different fields. Here are some
of the widely recognized uses of GANs:
Image Synthesis and Generation: GANs are often used for picture synthesis and generation tasks, They
may create fresh, lifelike pictures that mimic training data by learning the distribution that explains the
dataset. The development of lifelike avatars, high-resolution photographs, and fresh artwork have all
been facilitated by these types of generative networks.
Image-to-Image Translation: GANs may be used for problems involving image-to-image translation,
where the objective is to convert an input picture from one domain to another while maintaining its
key features. GANs may be used, for instance, to change pictures from day to night, transform
drawings into realistic images, or change the creative style of an image.
Text-to-Image Synthesis: GANs have been used to create visuals from descriptions in text. GANs
may produce pictures that translate to a description given a text input, such as a phrase or a caption.
This application might have an impact on how realistic visual material is produced using text-based
instructions.
Data Augmentation: GANs can augment present data and increase the robustness and generalizability
of machine-learning models by creating synthetic data samples.
Data Generation for Training: GANs can enhance the resolution and quality of low-resolution
images. By training on pairs of low-resolution and high-resolution images, GANs can generate high-
resolution images from low-resolution inputs, enabling improved image quality in various applications
such as medical imaging, satellite imaging, and video enhancement.
Advantages of GAN
The advantages of the GANs are as follows:
Synthetic data generation: GANs can generate new, synthetic data that resembles some known data
distribution, which can be useful for data augmentation, anomaly detection, or creative applications.
High-quality results: GANs can produce high-quality, photorealistic results in image synthesis, video
synthesis, music synthesis, and other tasks.
Unsupervised learning: GANs can be trained without labeled data, making them suitable for
unsupervised learning tasks, where labeled data is scarce or difficult to obtain.
Versatility: GANs can be applied to a wide range of tasks, including image synthesis, text-to-image
synthesis, image-to-image translation, anomaly detection, data augmentation, and others.
Disadvantages of GAN
The disadvantages of the GANs are as follows:
Training Instability: GANs can be difficult to train, with the risk of instability, mode collapse, or
failure to converge.
Computational Cost: GANs can require a lot of computational resources and can be slow to train,
especially for high-resolution images or large datasets.
Overfitting: GANs can overfit the training data, producing synthetic data that is too similar to the
training data and lacking diversity.
Bias and Fairness: GANs can reflect the biases and unfairness present in the training data, leading to
discriminatory or biased synthetic data.
Interpretability and Accountability: GANs can be opaque and difficult to interpret or explain,
making it challenging to ensure accountability, transparency, or fairness in their applications.
DEEP BOLTZMANN MACHINE (DBM)
DBMs are similar to DBNs except that apart from the connections within layers, the connections
between the layers are also undirected (unlike DBN in which the connections between layers are
directed). DBMs can extract more complex or sophisticated features and hence can be used for more
complex tasks.
What are Deep Boltzmann Machines (DBMs)?
Deep Boltzmann Machines (DBMs) are a kind of artificial neural network that belongs to the family
of generative models. They are designed to discover intricate structures within large datasets by
learning to recreate the input data they're given. Think of a DBM as an artist who, after studying a
collection of paintings, learns to create new artworks that could belong to the same collection.
Similarly, a DBM analyzes data and learns how to produce new examples that are similar to the original
data.
DBMs consist of multiple layers of hidden units, which are like the neurons in our brains. These units
work together to capture the probabilities of various patterns within the data. Unlike some other neural
networks, all units in a DBM are connected across layers, but not within the same layer, which allows
them to create a web of relationships between different features in the data. This structure helps DBMs
to be good at understanding complex data like images, text, or sound.
The 'deep' in the Deep Boltzmann Machine refers to the multiple layers in the network, which allow it
to build a deep understanding of the data. Each layer captures increasingly abstract representations of
the data. The first layer might detect edges in an image, the second layer might detect shapes, and the
third layer might detect whole objects like cars or trees.
How Deep Boltzmann Machines Work?
Deep Boltzmann Machines work by first learning about the data in an unsupervised way, which means
they look for patterns without being told what to look for. They do this using a process that involves
adjusting the connections between units based on the data they see. This process is similar to tuning a
radio to get a clear signal; the DBM 'tunes' itself to resonate with the structure of the data.
When a DBM is given a set of data, it uses a stochastic, or random, process to decide whether a hidden
unit should be turned on or off. This decision is based on the input data and the current state of other
units in the network. By doing this repeatedly, the DBM learns the probability distribution of the data—
basically, it gets an understanding of which patterns are likely and which are not.
After the learning phase, you can use a DBM to generate new data. When generating new data, the
DBM starts with a random pattern and refines it step by step, each time updating the pattern to be more
like the patterns it learned during training.
Concepts Related to Deep Boltzmann Machines (DBMs)
Several key concepts underpin Deep Boltzmann Machines:
• Energy-Based Models: DBMs are energy-based models, which means they assign an 'energy' level
to each possible state of the network. States that are more likely have lower energy. The network learns
by finding states that minimize this energy.
• Stochastic Neurons: Neurons in a DBM are stochastic. Unlike in other types of neural networks,
where neurons output a deterministic value based on their input, DBM neurons make random decisions
about whether to activate.
• Unsupervised Learning: DBMs learn without labels. They look at the data and try to understand the
underlying structure without any guidance on what features are important.
• Pre-training: DBMs often go through a pre-training phase where they learn one layer at a time. This
step-by-step learning helps in stabilizing the learning process before fine-tuning the entire network
together.
• Fine-Tuning: After pre-training, DBMs are fine-tuned, which means they adjust all their parameters
at once to better model the data.
Mathematical concepts
Deep Boltzmann Machines (DBMs) are grounded in some fascinating mathematical concepts, with
probability playing a starring role. At the heart of DBMs is the idea of modeling the data using a
probability distribution, which is mathematically defined by an energy function. The energy function
(ℎ)E(v,h) captures the relationship between visible units v (data) and hidden units h (features).

The probability of a certain state (a combination of visible and hidden units) is given by the Boltzmann
distribution:

where Z is the partition function, a normalization factor that ensures all probabilities sum up to one.
It's calculated as the sum of e^{-E(v,h)} over all possible states.
Learning in DBMs involves finding the weights that minimize the energy function, which in turn
maximizes the probability of the observed data. This is typically done using a learning algorithm
like Contrastive Divergence (CD) or Stochastic Gradient Descent (SGD), which adjust the weights to
lower the energy of data states and increase their probability.
During this process, a DBM learns the weights through repeated sampling. The sampling uses a
Markov Chain Monte Carlo (MCMC) method, allowing the model to explore different states based on
their probabilities.
In essence, DBMs use the language of statistical mechanics to model data in a probabilistic framework,
balancing complex interactions between layers to capture the essence of the data in a way that can be
intuitively visualized as a landscape of hills and valleys, where the data points naturally settle into the
lowest points, or the states of lowest energy.
 Implementation of Deep Boltzmann Machines (DBMs)
This code defines a Deep Boltzmann Machine (DBM) using Python, a type of generative neural
network useful for unsupervised learning tasks.
Class RBM:
Represents a Restricted Boltzmann Machine, a fundamental building block of a DBM. An RBM has
visible and hidden units and learns a probability distribution over inputs.
• __init__: Initializes weights and biases randomly.
• sample_hidden: Given visible units, it computes the activations of the hidden units.
• sample_visible: Given hidden units, it calculates the activations of the visible units.
• train: Trains the RBM using Contrastive Divergence, adjusting weights and biases to learn the data
distribution.
Types of Boltzmann Machines:
• Restricted Boltzmann Machines (RBMs)
• Deep Belief Networks (DBNs)
• Deep Boltzmann Machines (DBMs)

Restricted Boltzmann Machines (RBMs):

In a full Boltzmann machine, each node is connected to every other node and hence the connections
grow exponentially. This is the reason we use RBMs. The restrictions in the node connections in
RBMs are as follows –
• Hidden nodes cannot be connected to one another.
• Visible nodes connected to one another.

Suppose that we are using our RBM for building a recommender system that works on six (6) movies.
RBM learns how to allocate the hidden nodes to certain features. By the process of Contrastive
Divergence, we make the RBM close to our set of movies that is our case or scenario. RBM identifies
which features are important by the training process. The training data is either 0 or 1 or missing data
based on whether a user liked that movie (1), disliked that movie (0) or did not watch the movie
(missing data). RBM automatically identifies important features.
Contrastive Divergence:
RBM adjusts its weights by this method. Using some randomly assigned initial weights, RBM
calculates the hidden nodes, which in turn use the same weights to reconstruct the input nodes. Each
hidden node is constructed from all the visible nodes and each visible node is reconstructed from all
the hidden node and hence, the input is different from the reconstructed input, though the weights are
the same. The process continues until the reconstructed input matches the previous input. The process
is said to be converged at this stage. This entire procedure is known as Gibbs Sampling.
The Gradient Formula gives the gradient of the log probability of the certain state of the system with
respect to the weights of the system.

The above equations tell us – how the change in weights of the system will change the log probability
of the system to be a particular state. The system tries to end up in the lowest possible energy state
(most stable). Instead of continuing the adjusting of weights process until the current input matches
the previous one, we can also consider the first few pauses only. It is sufficient to understand how to
adjust our curve so as to get the lowest energy state. Therefore, we adjust the weights, redesign the
system and energy curve such that we get the lowest energy for the current position. This is known as
the Hinton’s shortcut.
 Working of RBM – Illustrative Example –
Consider – Mary watches four movies out of the six available movies and rates four of them. Say, she
watched m1, m3, m4 and m5 and likes m3, m5 (rated 1) and dislikes the other two, that is m1, m4 (rated
0) whereas the other two movies – m2, m6 are unrated. Now, using our RBM, we will recommend one
of these movies for her to watch next. Say –
• m3, m5 are of ‘Drama’ genre.
• m1, m4 are of ‘Action’ genre.
• ‘Dicaprio’ played a role in m5.
• m3, m5 have won ‘Oscar.’
• ‘Tarantino’ directed m4.
• m2 is of the ‘Action’ genre.
• m6 is of both the genres ‘Action’ and ‘Drama’, ‘Dicaprio’ acted in it and it has won an ‘Oscar’.
We have the following observations –
• Mary likes m3, m5 and they are of genre ‘Drama,’ she probably likes ‘Drama’ movies.
• Mary dislikes m1, m4 and they are of action genre, she probably dislikes ‘Action’ movies.
• Mary likes m3, m5 and they have won an ‘Oscar’, she probably likes an ‘Oscar’ movie.
• Since ‘Dicaprio’ acted in m5 and Mary likes it, she will probably like a movie in
which ‘Dicaprio’ acted.
• Mary does not like m4 which is directed by Tarantino, she probably dislikes any movie directed
by ‘Tarantino’.
Therefore, based on the observations and the details of m2, m6; our RBM recommends m6 to Mary
(‘Drama’, ‘Dicaprio’ and ‘Oscar’ matches both Mary’s interests and m6). This is how an RBM works
and hence is used in recommender systems.
 Deep Belief Networks (DBNs):
Suppose we stack several RBMs on top of each other so that the first RBM outputs are the input to the
second RBM and so on. Such networks are known as Deep Belief Networks. The connections within
each layer are undirected (since each layer is an RBM). Simultaneously, those in between the layers
are directed (except the top two layers – the connection between the top two layers is undirected).
There are two ways to train the DBNs-
1. Greedy Layer-wise Training Algorithm – The RBMs are trained layer by layer. Once the individual
RBMs are trained (that is, the parameters – weights, biases are set), the direction is set up between the
DBN layers.
2. Wake-Sleep Algorithm – The DBN is trained all the way up (connections going up – wake) and then
down the network (connections going down — sleep).