Module 3

Kohonen Self- Organizing Feature Map(SOM)

Kohonen Self-Organizing feature map (SOM)
refers to a neural network, which is trained
using competitive learning. Basic competitive
learning implies that the competition process
takes place before the cycle of learning. The
competition process suggests that some
criteria select a winning processing element.
After the winning processing element is
selected, its weight vector is adjusted
according to the used learning law (Hecht
Nielsen 1990).
The self-organizing map makes topologically
ordered mappings between input data and
processing elements of the map. Topological
ordered implies that if two inputs are of
similar characteristics, the most active
processing elements answering to inputs that
are located closed to each other on the map.
The weight vectors of the processing elements
are organized in ascending to descending
order. Wi < Wi+1 for all values of i or Wi+1 for
all values of i (this definition is valid for one-
dimensional self-organizing map only).
The self-organizing map is typically
represented as a two-dimensional sheet of
processing elements described in the figure
given below. Each processing element has its
own weight vector, and learning of SOM (self-
organizing map) depends on the adaptation of
these vectors. The processing elements of the
network are made competitive in a self-
organizing process, and specific criteria pick
the winning processing element whose
weights are updated. Generally, these criteria
are used to limit the Euclidean distance
between the input vector and the weight
vector. SOM (self-organizing map) varies
from basic competitive learning so that instead
of adjusting only the weight vector of the
winning processing element also weight
vectors of neighboring processing elements
are adjusted. First, the size of the
neighborhood is largely making the rough
ordering of SOM and size is diminished as
time goes on. At last, only a winning
processing element is adjusted, making the
fine-tuning of SOM possible. The use of
neighborhood makes topologically ordering
procedure possible, and together with
competitive learning makes process non-
linear.
It is discovered by Finnish professor and
researcher Dr. Teuvo Kohonen in 1982. The
self-organizing map refers to an unsupervised
learning model proposed for applications in
which maintaining a topology between input
and output spaces. The notable attribute of this
algorithm is that the input vectors that are
close and similar in high dimensional space
are also mapped to close by nodes in the 2D
space. It is fundamentally a method for
dimensionality reduction, as it maps high-
dimension inputs to a low dimensional
discretized representation and preserves the
basic structure of its input space.
All the entire learning process occurs without
supervision because the nodes are self-
organizing. They are also known as feature
maps, as they are basically retraining the
features of the input data, and simply grouping
themselves as indicated by the similarity
between each other. It has practical value for
visualizing complex or huge quantities of high
dimensional data and showing the relationship
between them into a low, usually two-
dimensional field to check whether the given
unlabeled data have any structure to it.
A self-Organizing Map (SOM) varies from
typical artificial neural networks (ANNs) both
in its architecture and algorithmic properties.
Its structure consists of a single layer linear
2D grid of neurons, rather than a series of
layers. All the nodes on this lattice are
associated directly to the input vector, but not
to each other. It means the nodes don't know
the values of their neighbors, and only update
the weight of their associations as a function
of the given input. The grid itself is the map
that coordinates itself at each iteration as a
function of the input data. As such, after
clustering, each node has its own coordinate
(i.j), which enables one to calculate Euclidean
distance between two nodes by means of the
Pythagoras theorem.
A Self-Organizing Map utilizes competitive
learning instead of error-correction learning,
to modify its weights. It implies that only an
individual node is activated at each cycle in
which the features of an occurrence of the
input vector are introduced to the neural
network, as all nodes compete for the privilege
to respond to the input.
The selected node- the Best Matching Unit
(BMU) is selected according to the similarity
between the current input values and all the
other nodes in the network. The node with the
fractional Euclidean difference between the
input vector, all nodes, and its neighboring
nodes is selected and within a specific radius,
to have their position slightly adjusted to
coordinate the input vector. By experiencing
all the nodes present on the grid, the whole
grid eventually matches the entire input
dataset with connected nodes gathered towards
one area, and dissimilar ones are isolated.
Algorithm:
Step:1
Each node weight w_ij initialize to a random
value.
Step:2
Choose a random input vector x_k.
Step:3
Repeat steps 4 and 5 for all nodes on the map.
Step:4
Calculate the Euclidean distance between
weight vector wij and the input vector x(t)
connected with the first node, where t, i, j =0.
Step:5
Track the node that generates the smallest
distance t.
Step:6
Calculate the overall Best Matching Unit
(BMU). It means the node with the smallest
distance from all calculated ones.
Step:7
Discover topological neighborhood βij(t) its
radius σ(t) of BMU in Kohonen Map.
Step:8
Repeat for all nodes in the BMU
neighborhood: Update the weight vector w_ij
of the first node in the neighborhood of the
BMU by including a fraction of the difference
between the input vector x(t) and the weight
w(t) of the neuron.
Step:9
Repeat the complete iteration until reaching
the selected iteration limit t=n.
Here, step 1 represents initialization phase,
while step 2 to 9 represents the training phase.
Where;
t = current iteration.
i = row coordinate of the nodes grid.
J = column coordinate of the nodes grid.
W= weight vector
w_ij = association weight between the nodes
i,j in the grid.
X = input vector
X(t)= the input vector instance at iteration t
β_ij = the neighborhood function, decreasing
and representing node i,j distance from the
BMU.
σ(t) = The radius of the neighborhood
function, which calculates how far neighbor
nodes are examined in the 2D grid when
updating vectors. It gradually decreases over
time

Support Vector Machine Algorithm

Support Vector Machine or SVM is one of the
most popular Supervised Learning algorithms,
which is used for Classification as well as
Regression problems. However, primarily, it is
used for Classification problems in Machine
Learning.
The goal of the SVM algorithm is to create the
best line or decision boundary that can
segregate n-dimensional space into classes so
that we can easily put the new data point in the
correct category in the future. This best
decision boundary is called a hyperplane.
SVM chooses the extreme points/vectors that
help in creating the hyperplane. These extreme
cases are called as support vectors, and hence
algorithm is termed as Support Vector
Machine. Consider the below diagram in
which there are two different categories that
are classified using a decision boundary or
hyperplane:
4.5M
541
Features of Java - Javatpoint

Example: SVM can be understood with the

example that we have used in the KNN
classifier. Suppose we see a strange cat that
also has some features of dogs, so if we want a
model that can accurately identify whether it is
a cat or dog, so such a model can be created
by using the SVM algorithm. We will first
train our model with lots of images of cats and
dogs so that it can learn about different
features of cats and dogs, and then we test it
with this strange creature. So as support vector
creates a decision boundary between these two
data (cat and dog) and choose extreme cases
(support vectors), it will see the extreme case
of cat and dog. On the basis of the support
vectors, it will classify it as a cat. Consider the
below diagram:

SVM algorithm can be used for Face

detection, image classification, text
categorization, etc.
Types of SVM
SVM can be of two types:
o Linear SVM: Linear SVM is used for
linearly separable data, which means if a
dataset can be classified into two classes
by using a single straight line, then such
data is termed as linearly separable data,
and classifier is used called as Linear SVM
classifier.
o Non-linear SVM: Non-Linear SVM is
used for non-linearly separated data, which
means if a dataset cannot be classified by
using a straight line, then such data is
termed as non-linear data and classifier
used is called as Non-linear SVM
classifier.
Hyperplane and Support Vectors in the SVM
algorithm:
Hyperplane: There can be multiple
lines/decision boundaries to segregate the
classes in n-dimensional space, but we need to
find out the best decision boundary that helps
to classify the data points. This best boundary
is known as the hyperplane of SVM.
The dimensions of the hyperplane depend on
the features present in the dataset, which
means if there are 2 features (as shown in
image), then hyperplane will be a straight line.
And if there are 3 features, then hyperplane
will be a 2-dimension plane.
We always create a hyperplane that has a
maximum margin, which means the maximum
distance between the data points.
Support Vectors:
The data points or vectors that are the closest
to the hyperplane and which affect the position
of the hyperplane are termed as Support
Vector. Since these vectors support the
hyperplane, hence called a Support vector.
How does SVM works?
Linear SVM:
The working of the SVM algorithm can be
understood by using an example. Suppose we
have a dataset that has two tags (green and
blue), and the dataset has two features x1 and
x2. We want a classifier that can classify the
pair(x1, x2) of coordinates in either green or
blue. Consider the below image:

So as it is 2-d space so by just using a straight

line, we can easily separate these two classes.
But there can be multiple lines that can
separate these classes. Consider the below
image:
Hence, the SVM algorithm helps to find the
best line or decision boundary; this best
boundary or region is called as a hyperplane.
SVM algorithm finds the closest point of the
lines from both the classes. These points are
called support vectors. The distance between
the vectors and the hyperplane is called
as margin. And the goal of SVM is to
maximize this margin. The hyperplane with
maximum margin is called the optimal
hyperplane.
Non-Linear SVM:
If data is linearly arranged, then we can
separate it by using a straight line, but for non-
linear data, we cannot draw a single straight
line. Consider the below image:

So to separate these data points, we need to

add one more dimension. For linear data, we
have used two dimensions x and y, so for non-
linear data, we will add a third dimension z. It
can be calculated as:
z=x2 +y2
By adding the third dimension, the sample
space will become as below image:

So now, SVM will divide the datasets into

classes in the following way. Consider the
below image:
Since we are in 3-d Space, hence it is looking
like a plane parallel to the x-axis. If we
convert it in 2d space with z=1, then it will
become as:
Hence we get a circumference of radius 1 in
case of non-linear data.
Principal Component Analysis
Principal Component Analysis is an
unsupervised learning algorithm that is used
for the dimensionality reduction in machine
learning. It is a statistical process that converts
the observations of correlated features into a
set of linearly uncorrelated features with the
help of orthogonal transformation. These new
transformed features are called the Principal
Components. It is one of the popular tools
that is used for exploratory data analysis and
predictive modeling. It is a technique to draw
strong patterns from the given dataset by
reducing the variances.
PCA generally tries to find the lower-
dimensional surface to project the high-
dimensional data.
PCA works by considering the variance of
each attribute because the high attribute shows
the good split between the classes, and hence
it reduces the dimensionality. Some real-world
applications of PCA are image processing,
movie recommendation system, optimizing
the power allocation in various
communication channels. It is a feature
extraction technique, so it contains the
important variables and drops the least
important variable.
The PCA algorithm is based on some
mathematical concepts such as:
o Variance and Covariance
o Eigenvalues and Eigen factors
Some common terms used in PCA algorithm:
o Dimensionality: It is the number of
features or variables present in the given
dataset. More easily, it is the number of
columns present in the dataset.
o Correlation: It signifies that how strongly
two variables are related to each other.
Such as if one changes, the other variable
also gets changed. The correlation value
ranges from -1 to +1. Here, -1 occurs if
variables are inversely proportional to each
other, and +1 indicates that variables are
directly proportional to each other.
o Orthogonal: It defines that variables are
not correlated to each other, and hence the
correlation between the pair of variables is
zero.
o Eigenvectors: If there is a square matrix
M, and a non-zero vector v is given. Then
v will be eigenvector if Av is the scalar
multiple of v.
o Covariance Matrix: A matrix containing
the covariance between the pair of
variables is called the Covariance Matrix.
Principal Components in PCA
As described above, the transformed new
features or the output of PCA are the Principal
Components. The number of these PCs are
either equal to or less than the original features
present in the dataset. Some properties of
these principal components are given below:
o The principal component must be the
linear combination of the original features.
o These components are orthogonal, i.e., the
correlation between a pair of variables is
zero.
o The importance of each component
decreases when going to 1 to n, it means
the 1 PC has the most importance, and n
PC will have the least importance.
Steps for PCA algorithm
[Link] the dataset
Firstly, we need to take the input dataset
and divide it into two subparts X and Y,
where X is the training set, and Y is the
validation set.
[Link] data into a structure
Now we will represent our dataset into a
structure. Such as we will represent the
two-dimensional matrix of independent
variable X. Here each row corresponds to
the data items, and the column corresponds
to the Features. The number of columns is
the dimensions of the dataset.
[Link] the data
In this step, we will standardize our
dataset. Such as in a particular column, the
features with high variance are more
important compared to the features with
lower variance.
If the importance of features is
independent of the variance of the feature,
then we will divide each data item in a
column with the standard deviation of the
column. Here we will name the matrix as
Z.
[Link] the Covariance of Z
To calculate the covariance of Z, we will
take the matrix Z, and will transpose it.
After transpose, we will multiply it by Z.
 The output matrix will be the Covariance
matrix of Z.
[Link] the Eigen Values and Eigen
Vectors
Now we need to calculate the eigenvalues
and eigenvectors for the resultant
covariance matrix Z. Eigenvectors or the
covariance matrix are the directions of the
axes with high information. And the
coefficients of these eigenvectors are
defined as the eigenvalues.
[Link] the Eigen Vectors
In this step, we will take all the
eigenvalues and will sort them in
decreasing order, which means from
largest to smallest. And simultaneously
sort the eigenvectors accordingly in matrix
P of eigenvalues. The resultant matrix will
be named as P*.
[Link] the new features Or
Principal Components
Here we will calculate the new features. To
do this, we will multiply the P* matrix to
the Z. In the resultant matrix Z*, each
 observation is the linear combination of
original features. Each column of the Z*
matrix is independent of each other.
[Link] less or unimportant features
from the new dataset.
The new feature set has occurred, so we
will decide here what to keep and what to
remove. It means, we will only keep the
relevant or important features in the new
dataset, and unimportant features will be
removed out.
Applications of Principal Component Analysis
o PCA is mainly used as the dimensionality
reduction technique in various AI
applications such as computer vision,
image compression, etc.
o It can also be used for finding hidden
patterns if data has high dimensions. Some
fields where PCA is used are Finance, data
mining, Psychology, etc.
Hopfield Network
Hopfield network is a special kind of neural
network whose response is different from
other neural networks. It is calculated by
converging iterative process. It has just one
layer of neurons relating to the size of the
input and output, which must be the same.
When such a network recognizes, for example,
digits, we present a list of correctly rendered
digits to the network. Subsequently, the
network can transform a noise input to the
relating perfect output.
In 1982, John Hopfield introduced an
artificial neural network to collect and retrieve
memory like the human brain. Here, a neuron
is either on or off the situation. The state of a
neuron(on +1 or off 0) will be restored,
relying on the input it receives from the other
neuron. A Hopfield network is at first
prepared to store various patterns or
memories. Afterward, it is ready to recognize
any of the learned patterns by uncovering
partial or even some corrupted data about that
pattern, i.e., it eventually settles down and
restores the closest pattern. Thus, similar to
the human brain, the Hopfield model has
stability in pattern recognition.
A Hopfield network is a single-layered and
recurrent network in which the neurons are
entirely connected, i.e., each neuron is
associated with other neurons. If there are two
neurons i and j, then there is a connectivity
weight wij lies between them which is
symmetric wij = wji .
3.8M
431
Exception Handling in Java - Javatpoint
Next
Stay
With zero self-connectivity, Wii =0 is given
below. Here, the given three neurons having
values i = 1, 2, 3 with values Xi=±1 have
connectivity weight Wij.
Updating rule:
Consider N neurons = 1, … , N with values Xi
= +1, -1.

The update rule is applied to the node i is

given by:
If hi ≥ 0 then xi → 1 otherwise xi → -1
Where hi = is called field at i, with b£ R
a bias.
Thus, xi → sgn(hi), where the value
of sgn(r)=1, if r ≥ 0, and the value of sgn(r)=-
1, if r < 0.
We need to put bi=0 so that it makes no
difference in training the network with random
patterns.
We, therefore, consider hi= .
We have two different approaches to update
the nodes:
Synchronously:
In this approach, the update of all the nodes
taking place simultaneously at each time.
Asynchronously:
In this approach, at each point of time, update
one node chosen randomly or according to
some rule. Asynchronous updating is more
biologically realistic.
Hopfield Network as a Dynamical system:
Consider, K = {-1, 1} N so that each state x £
X is given by xi £ { -1,1 } for 1 ≤ I ≤ N
Here, we get 2N possible states or
configurations of the network.
We can describe a metric on X by using the
Hamming distance between any two states:
P(x, y) = # {i: xi≠yi}
N Here, P is a metric with 0≤H(x,y)≤ N. It is
clearly symmetric and reflexive.
With any of the asynchronous or synchronous
updating rules, we get a discrete-time
dynamical system.
The updating rule up: X → X describes a map.
And Up: X → X is trivially continuous.
Example:
Suppose we have only two neurons: N = 2
There are two non-trivial choices for
connectivities:
w12 = w21 = 1
w12= w21 = -1
Asynchronous updating:
In the first case, there are two attracting fixed
points termed as [-1,-1] and [-1,-1]. All orbit
converges to one of these. For a second, the
fixed points are [-1,1] and [1,-1], and all orbits
are joined through one of these. For any fixed
point, swapping all the signs gives another
fixed point.
Synchronous updating:
In the first and second cases, although there
are fixed points, none can be attracted to
nearby points, i.e., they are not attracting fixed
points. Some orbits oscillate forever.
Energy function evaluation:
Hopfield networks have an energy function
that diminishes or is unchanged with
asynchronous updating.
For a given state X ∈ {−1, 1} N of the network
and for any set of association
weights Wij with Wij = wji and wii =0 let,

Here, we need to update Xm to X'm and denote

the new energy by E' and show that.
E'-E = (Xm-X'm ) ∑i≠mWmiXi.
Using the above equation, if Xm = Xm' then we
have E' = E
If Xm = -1 and Xm' = 1 , then Xm - Xm' = 2 and
hm= ∑iWmiXi ? 0
Thus, E' - E ≤ 0
Similarly if Xm =1 and Xm'= -1 then Xm - Xm'
= 2 and hm= ∑iWmiXi < 0
Thus, E - E' < 0.
Note:
If Xm flips, then E' - E = 2Xmhm
Neurons pull in or push away from each other:
Suppose the connection weight Wij =
Wji between two neurons I and j.
If Wij > 0, the updating rule implies:
o If Xj = 1, then the contribution of j in the
weighted sum, i.e., WijXj, is positive. Thus
the value of Xi is pulled by j towards its
value Xj= 1
o If Xj= -1 then WijXj , is negative, and Xi is
again pulled by j towards its value Xj = -1
Thus, if Wij > 0 , then the value of i is pulled
by the value of j. By symmetry, the value
of j is also pulled by the value of i.

If Wij < 0, then the value of i is pushed away

by the value of j.
It follows that for a particular set of
values Xi ∈ { -1 , 1 } for;
1 ≤ i ≤ N, the selection of weights taken
as Wij = XiXj for;
1 ≤ i ≤ N correlates to the Hebbian rule.
Training the network: One pattern (Ki=0)
Suppose the vector x→ = (x1,…,xi,…,xN) ∈ {-
1,1}N is a pattern that we like to store in the
Hopfield network.
To build a Hopfield network that
recognizes x→, we need to select connection
weight Wij accordingly.
If we select Wij =ɳ XiXj for 1 ≤ i , j ≤ N (Here,
i≠j), where ɳ > 0 is the learning rate, then the
value of Xi will not change under updating
condition as we illustrate below.
We have

It implies that the value of Xi, whether 1 or -

1 will not change, so that x→ is a fixed point.
Note that - x→ also becomes a fixed point
when we train the network with x→ validating
that Hopfield networks are sign blind.
Storage capacity
Therefore if we store p patterns in a Hopfield
network with a large number of N nodes, then
the probability of error, i.e., the probability
that C k i > 1, is: Perror = P(C k i > 1) ≈ 1 √
2πσ Z ∞ 1 exp(−x 2 /2σ 2 ) dx = 1 2 (1 − erf(1/
√ 2σ 2)) = 1 2 (1 − erf( p N/2p)) where the
error function erf is given by: erf(x) = 2 √ π Z
x 0 exp(−s 2 ) ds. I Therefore, given N and p
we can find out the probability Perror of error
for a single neuron of a stored pattern.
[Link]
[Link]
Optimization using Hopfield Network
Optimization is the process of making
something such as the design, environment,
resource, and system as efficient as possible
by utilizing a resemblance of the cost function
and the energy function. One such type of
neural network is called a Hopfield network,
which is composed of a single layer that
contains at least one of the fully connected
recurrent neural networks. It can be utilized to
optimize.
Things to keep in mind when using the
Hopfield network to optimize -
o The energy function should be minimal in
the network.
o It will be able to find a satisfactory
solution instead of choosing one of the
stored patterns.
o The effectiveness of the solution
uncovered by the Hopfield network is
 mainly dependent on the state that was
initially present in the network.
Travelling Salesman Problem
Finding the most efficient route the
salesperson takes is among the computational
challenges that can be improved by using
Hopfield neural networks.
Basic Concept of TSP
The Travelling Salesman Problem the TSP is a
classic optimization problem where
salespeople have to travel through a number of
cities connected to one another and keep the
costs. The distance travelled is at a minimum.
For instance, a salesperson must travel through
a set of four cities, A, B, C, and D. The
objective is to determine the most efficient
circular route, A-B-C-D, in order to reduce the
cost, which includes the cost of traveling from
city D to the next one city A.
Matrix Representation
Each tour of the TSP for n cities could be
described in a matrix n x N which's ith row
identifies the location of each city. The matrix
M, for four cities, A, B, C, and D, could be
expressed in the following manner: as follows:

Solution by Hopfield Network

When analysing the solution to the TSP using
a Hopfield network, each node in the network
is linked to a specific element of the matrix.
Energy Function Calculation
In order to be the optimal solution for optimal
results, the energy function should be as low
as possible. Based on the above constraints,
we are able to determine the function of
energy using the following formula: --
Constraint-I
The first constraint, on that basis, we calculate
the energy function, is that one component
must be at least 1 in every row on the matrix
M and all other elements in each row have not
to be greater than the value 0 because each
city may occur in only one location within the
TSP tour. This constraint can be
mathematically expressed as follows:

Based on the above constraint, now, we will

minimize the energy function, that will contain
the term proportional to:
Constraint-II
As we all know, in TSP, one city may be
found at any location within the tour.
Therefore, in every column of M, each
element has to be equal to 1, and the other
elements have to be equal to the value 0. This
restriction can mathematically be expressed as
follows:

Based on the above constraint, now, we will

minimize the energy function, that will contain
the term proportional to:

Cost Function Calculation

Let's say that a square matrix ( n x n) that is
represented with C is the cost matrix for TSP
for the n cities in which there is a value of n
greater than. Here are some of the parameters
to consider when calculating the cost function
o Cx,y: The component of the cost matrix
refers to the cost of travel from one city to
another city.
o The following relationship demonstrates
the relationship between the two elements,
A and B.

We know that within the Matrix, it is possible

to determine the value of output for each node
could be one or zero, so for each city pair A
and B, we can include the following words for
the power function.

Based on the cost function above, along with

the value for constraint end value of the
energy-related calculation function, E can be
described as follows:
Here, γ1 and γ2 are two weighing constants.
(assignment)
[Link]
4/issue_1/EL_14_1_23.pdf (linear eqtn
(seminar))

[Link]
[Link] -----character retrival

What is a Boltzmann machine?

A Boltzmann machine is an unsupervised deep
learning model in which every node is
connected to every other node. It is a type of
recurrent neural network, and the nodes make
binary decisions with some level of bias.
These machines are not deterministic deep
learning models, they are stochastic
or generative deep learning models. They are
representations of a system.
A Boltzmann machine has two kinds of nodes
• Visible nodes:
These are nodes that can be measured and
are measured.
• Hidden nodes:
These are nodes that cannot be measured
or are not measured.
According to some experts, a Boltzmann
machine can be called a stochastic Hopfield
network which has hidden units. It has a
network of units with an ‘energy’ defined for
the overall network.
Boltzmann machines seek to reach thermal
equilibrium. It essentially looks to optimize
global distribution of energy. But the
temperature and energy of the system are
relative to laws of thermodynamics and are not
literal.
A Boltzmann machine is made up of a
learning algorithm that enables it to discover
interesting features in datasets composed of
binary vectors. The learning algorithm tends to
be slow in networks that have many layers of
feature detectors but it is possible to make it
faster by implementing a learning layer of
feature detectors.
They use stochastic binary units to reach
probability distribution equilibrium (to
minimize energy). It is possible to get multiple
Boltzmann machines to collaborate together to
form far more sophisticated systems like deep
belief networks.
The Boltzmann machine is named
after Ludwig Boltzmann, an Austrian scientist
who came up with the Boltzmann distribution.
However, this type of network was first
developed by Geoff Hinton, a Stanford
Scientist.
Source: Wikipedia

What is the Boltzmann distribution?

The Boltzmann distribution is a probability
distribution that gives the probability of a
system being in a certain state as a function of
that state's energy and the temperature of the
system.
It was formulated by Ludwig Boltzmann in
1868 and is also known as the Gibbs
distribution.
What are Boltzmann machines used for?
The main aim of a Boltzmann machine is to
optimize the solution of a problem. To do this,
it optimizes the weights and quantities related
to the specific problem that is assigned to it.
This technique is employed when the main
aim is to create mapping and to learn from the
attributes and target variables in the data. If
you seek to identify an underlying structure or
the pattern within the data, unsupervised
learning methods for this model are regarded
to be more useful. Some of the most widely
used unsupervised learning methods are
clustering, dimensionality reduction, anomaly
detection and creating generative models.
All of these techniques have a different
objective of detecting patterns like identifying
latent grouping, finding irregularities in the
data, or even generating new samples from the
data that is available. You can even stack these
networks in layers to build deep neural
networks that capture highly complicated
statistics. Restricted Boltzmann machines are
widely used in the domain of imaging and
image processing as well because they have
the ability to model continuous data that are
common to natural images. They are even
used to solve complicated quantum
mechanical many-particle problems or
classical statistical physics problems like the
Ising and Potts classes of models.

How does a Boltzmann machine work?

Boltzmann machines are non-deterministic
(stochastic) generative Deep Learning models
that only have two kinds of nodes - hidden and
visible nodes. They don’t have any output
nodes, and that’s what gives them the non-
deterministic feature. They learn patterns
without the typical 1 or 0 type output through
which patterns are learned and optimized
using Stochastic Gradient Descent.
A major difference is that unlike other
traditional networks (A/C/R) which don’t have
any connections between the input nodes,
Boltzmann Machines have connections among
the input nodes. Every node is connected to all
other nodes irrespective of whether they are
input or hidden nodes. This enables them to
share information among themselves and self-
generate subsequent data. You’d only measure
what’s on the visible nodes and not what’s on
the hidden nodes. After the input is provided,
the Boltzmann machines are able to capture all
the parameters, patterns and correlations
among the data. It is because of this that they
are known as deep generative models and they
fall into the class of Unsupervised Deep
Learning.

What are the types of Boltzmann

machines?
There are three types of Boltzmann machines.
These are:
• Restricted Boltzmann Machines (RBMs)
• Deep Belief Networks (DBNs)
• Deep Boltzmann Machines (DBMs)

1. Restricted Boltzmann Machines (RBMs)

While in a full Boltzmann machine all the
nodes are connected to each other and the
connections grow exponentially, an RBM has
certain restrictions with respect to node
connections.
In a Restricted Boltzmann Machine, hidden
nodes cannot be connected to each other while
visible nodes are connected to each other.
2. Deep Belief Networks (DBNs)
In a Deep Belief Network, you could say that
multiple Restricted Boltzmann Machines are
stacked, such that the outputs of the first RBM
are the inputs of the subsequent RBM. The
connections within individual layers are
undirected, while the connections between
layers are directed. However, there is an
exception here. The connection between the
top two layers is undirected.
A deep belief network can either be trained
using a Greedy Layer-wise Training
Algorithm or a Wake-Sleep Algorithm.
3. Deep Boltzmann Machines (DBMs)
Deep Boltzmann Machines are very similar to
Deep Belief Networks. The difference
between these two types of Boltzmann
machines is that while connections between
layers in DBNs are directed, in DBMs, the
connections within layers, as well as the
connections between the layers, are all
undirected.
Boltzmann Machines
Boltzmann machine refers to an association of
uniformly associated neuron-like structure that
make hypothetical decisions about whether to
be on or off. Boltzmann Machine was
invented by renowned scientist Geoffrey
Hinton and Terry Sejnowski in 1985.
Boltzmann Machines have a fundamental
learning algorithm that permits them to find
exciting features that represent complex
regularities in the training data. The learning
algorithm is usually slow in networks with
various layers of feature detectors, but it is
quick in "Restricted Boltzmann Machines"
that has a single layer of feature detectors.
Many hidden layers can be adapted efficiently
by comprising Boltzmann Machines, utilizing
the feature activations of one as the training
data for the next.
Boltzmann Machines are utilized to resolve
two different computational issues. First, for a
search problem, the weight on the
associations are fixed and are used to represent
a cost function. The stochastic dynamics of a
Boltzmann Machine permit it to sample binary
state vectors that have minimum values of the
cost function. Second, for a learning issue, the
Boltzmann Machine has indicated a set of
binary data vectors, and this must figure out
how to generate these vectors with high
probability. To solve this, it must discover
weights on the associations so that relative to
other possible binary vectors, the data vectors
have minimum values of the cost function. For
solving a learning issue, Boltzmann machines
make numerous small updates to their weights,
and each update expects them to tackle a wide
range of search issues.
The Stochastic Dynamics of a Boltzmann
Machine:
When unit i is given a chance to update its
binary state, it initially computes its absolute
input, pi , which is the sum of its own bias, qi ,
and the weights on associations coming from
other active units:
Pi = qi + ?jmj wij
Where,
wij = It is the weight on the association
between i and j, and mj is 1 when unit j is on.
Unit i turns on with a probability given by the
logistic function:
If the units are updated successively in any
order that does not rely on their total inputs,
the network will eventually reach a Boltzmann
distribution (also known as equilibrium or
stationary distribution) in which the
probability of the given state vector k is
determined exclusively by the "energy" of that
state vector compared to the energies of all
possible binary state vectors:

As in Hopfield networks, the energy of state

vector k is defined as

Where, sik is the binary state appointed to

unit i by state vector k. If the weights on the
associations are chosen so that the energies of
the state vectors represent the cost of those
state vector, the stochastic dynamics of a
Boltzmann machine can be seen as a method
for getting away from poor local optima while
looking for low-cost solutions. The total input
of unit i , pi , represents the difference in
energy relying upon whether the units are off
or on, and the way that unit i sometimes turns
on even if pi is negative implies that the
energy can occasionally increase during the
search, therefore permitting the search to jump
over energy barriers. The search can be
upgraded by using simulated annealing. It
scales down all of the weights and energies by
a factor T, which is equivalent to the
temperature of a physical network. By
minimizing T from a considerable initial value
to small final value, it is possible to benefit
from the fast equilibrium at high temperature
and still have a final equilibrium distribution
that makes minimal solutions considerably
more probable than high-cost ones. At a zero
temperature, the update rule becomes
deterministic, and a Boltzmann Machines
transforms into a Hopefield network.
Different types of Boltzmann Machine
The learning rule can hold more complex
energy functions. For example, the quadratic
energy function can be replaced by an energy
function that has a common term si sj
sk wijk. The total input i is utilized to update
rule must be replaced by

The significant change in the learning rule is

that si sj is replaced by si sj sk. Boltzmann
machines model the dispersion of the data
vectors. However, there is a basic extension,
the "conditional Boltzmann machine" for
modeling conditional distributions. The
significant difference between the visible and
the hidden units is that when sampling (si sj)
data, the visible units are clamped, and the
hidden units are possibly not. If a subset of the
visible units is clamped when sampling ) (si
sj) model, this subset acts as "input" units, and
the remaining visible units serve as "output"
units.
The Speed of Learning
Learning is commonly very slow in
Boltzmann machines with various hidden
layers because the enormous networks can
take quite a long time to approach their
equilibrium distribution, particularly when the
weights are huge and the equilibrium
distribution is highly multimodal. When
samples from the equilibrium distribution can
be acquired, the learning signal is very noisy
because it is the difference between the two
sampled expectations. These issues can be
overcome by confining the network,
simplifying the learning algorithm, and
learning one hidden layer at a time.
Restricted Boltzmann Machine:
The restricted Boltzmann machine invented
by Smolensky in 1986. It comprises of a layer
of visible units and a layer of hidden units
with no visible-visible or hidden-hidden
associations. With these restrictions, the
hidden units are provisionally autonomous
given a visible vector, so unbiased sample
form (si sj) data can be obtained on one parallel
step. In order to sample form (si sj)model still
requires different iterations that substitute
between restoring all the hidden units in
parallel and restoring all the visible units in
parallel. However, learning still functions well
if (si sj)modelis replaced by (si sj)reconstruction ,
which is obtained as follows:
Beginning with a data vector on the visible
units, restore all of the hidden units in parallel.
Update all of the visible units in parallel to
obtain a reconstruction.
Update all of the hidden units once again.
Simulated annealing