UNIT II DEEP LEARNING ALGORITHMS FOR AI

Introduction to Artificial Neural Networks – Linear Associative Networks – Perceptrons -The

Back propagation Algorithm - Hopfield Nets - Boltzmann Machines - Deep Restricted
Boltzmann Machines (RBMs) - Variational Auto encoders – Deep Back prop Networks-
Radial Basis Function Networks (RBFNs)-Self Organizing Maps (SOMs).

1. Artificial Neural Network

An Artificial Neural Network in the field of Artificial intelligence where it attempts to mimic the
network of neurons makes up a human brain so that computers will have an option to understand
things and make decisions in a human-like manner. The artificial neural network is designed by
programming computers to behave simply like interconnected brain cells. Each neuron receives a
multiplied version of inputs and random weights, which is then added with a static bias value (unique
to each neuron layer); this is then passed to an appropriate activation function which decides the final
value to be given out of the neuron. There are various activation functions available as per the nature
of input values.
Once the output is generated from the final neural net layer, loss function (input vs output)is
calculated, and backpropagation is performed where the weights are adjusted to make the loss
minimum. Finding optimal values of weights is what the overall operation focuses around. Please
refer to the following for better understanding.

Weights are numeric values that are multiplied by inputs. In backpropagation, they are modified to
reduce the loss. In simple words, weights are machine learned values from NeuralNetworks. They
self-adjust depending on the difference between predicted outputs vs traininginputs. Activation
Function is a mathematical formula that helps the neuron to switch ON/OFF.

 Input layer represents dimensions of the input vector.

 Hidden layer represents the intermediary nodes that divide the input space into regions
with (soft) boundaries. It takes in a set of weighted input and produces output through
an activation function.
 Output layer represents the output of the neural network.
Advantages of Artificial Neural Network (ANN)
Parallel processing capability
 Artificial neural networks have a numerical value that can perform more than one task
simultaneously.
Storing Data on the Entire Network
 Data that is used in traditional programming is stored on the whole network, not on a
database.
Capability to work with incomplete knowledge
 After ANN training, the information may produce output even with inadequate data. The
loss of performance here relies upon the significance of missing data.
Having a memory distribution
 The succession of the network is directly proportional to the chosen instances, and if the
event can't appear to the network in all its aspects, it can produce false output.
Having fault tolerance
 Extortion of one or more cells of ANN does not prohibit it from generating output, and
this feature makes the network fault-tolerance.
Linear Associative Networks
Linear associator is the simplest and most widely used associative memory models. It is a
collection of simple processing units which have a quite complex collective computational
capability and behavior.
Linear Associator model (two layers):
The linear associator model is a feed-forward type network where produced output is in the form
of single feed-forward computation. The model comprises of two layers of processing units, one
work as an input layer while the other work as an output layer. The input is directly associated
with the outputs, through a series of weights. The connections carrying weights link each input
to every output. The addition of the products of the weights and the input is determined in each
Network architectures of Associate Memory Models:
The neural associative memory models pursue various neural network architectures to memorize
data. The network comprises either a single layer or two layers. The linear associator model refers
to a feed-forward type network, comprises of two layers of different processing units- The first
layer serving as the input layer while the other layer as an output layer. The Hopfield model refers
to a single layer of processing elements where each unit is associated with every other unit in the
given [Link] neural network architectures of these given models and the structure of the
corresponding association weight matrix w of the associative memory are depicted.
neuron node. The architecture of the linear associator is given below.

All p inputs units are associated to all q output units via associated weight matrix
W = [wij]p * q
where wij describes the strength of the unidirectional association of the ith inputunit to the jth
output [Link] connection weight matrix stores the z different associated pattern pairs {(Xk,Yk);
k=1,2,3,…,z}. Constructing an associative memory is building the connection weight matrix w
suchthat if an input pattern is presented, the stored pattern associated with the input pattern is
recovered.

WHAT IS A PERCEPTRON?

A perceptron is a binary classification algorithm modeled after the functioning of the human
brain—it was intended to emulate the neuron. The perceptron, while it has a simple structure,
has the ability to learn and solve very complex problems.

What is Multilayer Perceptron?

A multilayer perceptron (MLP) is a group of perceptrons, organized in multiple layers, that
canaccurately answer complex questions. Each perceptron in the first layer (on the left) sends
signals to all the perceptrons in the second layer, and so on. An MLP contains an input layer,at
least one hidden layer and an output layer.
The perceptron learns as follows:

1. Takes the inputs which are fed into the perceptrons in the input layer, multiplies them by their
weights, and computes the sum.
2. Adds the number one, multiplied by a “bias weight”. This is a technical step that makes it
possible to move the output function of each perceptron (the activation function) up,down, left and
right on the number graph.
3. Feeds the sum through the activation function—in a simple perceptron system, the activation
function is a step function.
4. The result of the step function is the output. A multilayer perceptron is quite similar to a modern
neural network. By adding a few ingredients, the perceptron architecture becomes a full-fledged
deep learning system:Activation functions and other hyper parameters: a full neural network uses a
variety of activation functions which output real values, not boolean values like in the classic
[Link] is more flexible in terms of other details of the learning process, such as the number
oftraining iterations (iterations and epochs), weight initialization schemes, regularization, and so
on. All these can be tuned as hyper parameters. Backpropagation: a full neural network uses the
backpropagation algorithm, to performiterative backward passes which try to find the optimal
values of perceptron weights, to generate the most accurate prediction.

Advanced architectures:
a full neural networks can have a variety of architectures that
can help solve specific problems. A few examples are Recurrent Neural Networks (RNN),
Convolutional Neural Networks (CNN), and Generative Adversarial Networks (GAN).

WHAT IS BACKPROPAGATION AND WHY IS IT IMPORTANT?

After a neural network is defined with initial weights, and a forward pass is performed togenerate
the initial prediction, there is an error function which defines how far away the mode lis from the
true prediction. There are many possible algorithms that can minimize the errorfunction—for
example, one could do a brute force search to find the weights that generate the smallest error.
However, for large neural networks, a training algorithm is needed that is very computationally
efficient. Backpropagation is that algorithm—it can discover the optimal weights relatively
quickly, even for a network with millions of weights.
HOW BACKPROPAGATION WORKS?
1. Forward pass — weights are initialized and inputs from the training set are fed into the network.
The forward pass is carried out and the model generates its initial prediction.
2. Error function — the error function is computed by checking how far away the prediction is
from the known true value.
3. Backpropagation with gradient descent — the backpropagation algorithm calculates how much
the output values are affected by each of the weights in the model. To do this, it calculates partial
derivatives, going back from the error function to a specific neuron and its weight. This provides
complete traceability from total errors, back to a specific weight which contributed to that error.
The result of backpropagation is a set of weights that minimize the error function.
4. Weight update — weights can be updated after every sample in the training set, but this is
usually not practical. Typically, a batch of samples is run in one big forward pass, and then
backpropagation performed on the aggregate result. The batch size and number of batches used in
training, called iterations, are important hyperparameters that are tuned to get the best results.
Running the entire training set through the backpropagation process is called an epoch.

Training algorithm of BPNN:

1. Inputs X, arrive through the pre connected path
2. Input is modeled using real weights W. The weights are usually randomly selected.
3. Calculate the output for every neuron from the input layer, to the hidden layers, to the output
layer.
4. Calculate the error in the outputs ErrorB= Actual Output – Desired Output
5. Travel back from the output layer to the hidden layer to adjust the weights such that the error is
decreased. Keep repeating the process until the desired output is achieved Architecture of back
propagation network: As shown in the diagram, the architecture of BPN has three interconnected
layers having weights on them. The hidden layer as well as the output layer also has bias, whose
weight is always 1, on them. As is clear from the diagram, the working of BPN is in two phases.
One phase sends the signal from the input layer to the output layer, and the other phase
backpropagates the error from the output layer to the input layer.
Introduction to Hopfield Network
A Hopfield network is a particular type of single-layered neuron network. Dr. John J. Hopfield
invented it in 1982. These networks were introduced to collect and retrieve memory and store
various patterns. Also, auto-association and optimization of the task can be done using these
networks. In this network, each node is fully connected(recurrent) to other nodes. These nodes
exist only in two states: ON (1) or OFF (0). These states can be restored based on the input
received from other nodes. Unlike other neural networks, the output of the Hopfield network is
finite. Also, the input and output sizes must be the same in these networks.

The Hopfield network consists of associative memory. This memory allows the system to retrieve
the memory using an incomplete portion. The network can restore the closest pattern using the data
captured in associative memory. This feature of Hopfield The Hopfield networks are categorized
into two categories. These are: Discrete Networks These networks give any of the two discrete
outputs. Based on the output received, further two types: networks makes it a good candidate for
pattern recognition. Associative memory is a content addressable memory that establishes a
relation between the input vector and the output target vector. It enables the reallocation of data
stored in the memory based on its similarity with the input vector. The Hopfield networks are
categorized into two categories. These are: Discrete Networks These networks give any of the two
discrete outputs. Based on the output received, further two types:
1. Binary: In this type, the output is either 0 or 1.
2. Bipolar: In bipolar networks, the output is either -1 (When output < 0) or 1 (When
output > 0)
Continuous Networks
Instead of receiving binary or bipolar output, the output value lies between 0 and 1. The
Architecture of Hopfield Network The architecture of the Hopfield network consists of the
following elements:
 Individual nodes preserve their states until required an update.
The node to be updated is selected randomly.
 Each node is connected to all other nodes except itself.
 The state of each node is either 0/1 or 1/-1.
 The Hopfield network structure is symmetric, i.e., Wij = Wji for all i's and j's.

In the above diagram, each symbol represents:

x1, x2, x3 - represents the input.
y1, y2, y3 - represents output obtained from each node.
Wij - represents the weight associated with the connection from i to j. Energy Function in Hopfield
Network In Hopfield networks, there are two different types of updations. Synchronous: Updating
all the nodes simultaneously each time. Asynchronous: Updating only one node at a time. That
node is selected randomly or based on specific rules. In asynchronous updation, each state of
Hopfield networks is associated with an energy value. The value is obtained from a function, and
that function is named an energy function. This function can decrease or remain unchanged during
updation. This energy function of the Hopfield network is defined as:

Where Wij = weight of the connection between i to j

Xi = value of input i
Xj= value of input j
A network is considered to be in a stable state if the energy function tends to be minimum
Training Model with Hopfield Network Training a Hopfield network refers to lowering of energy
of each state. The training model consists of a training and testing algorithm. Let's discuss each of
them one by one. Training Algorithm It is based on the Hebbian principle, which Donald Hebb
gave in 1949. In training, algorithm weights are updated using a specific rule. The rule is different
for binary and bipolar. For storing a set of 'N' input patterns [ X(n) where n = 1 to N ], the rule
followed is as follows:

Testing Algorithm
The testing algorithm involves several steps. These steps are:
 Initialize the weights using the above training algorithm rules.
 Follow steps 3 to 7 for each input vector 'Xi'.
 Assign the value of external input vector 'Xi' to initial activators Yi', for all i = 1 to N.
 Follow steps 5 to 7 for each initial activator 'Yi'.
 Calculate the network's net input 'Yin' using the below rule:

 Based on the calculated net input and threshold value, calculate the output. There
are three different conditions for the output. These are:

 Broadcast the obtained output 'Yi' to all other units. And update the activation
vector.
 Test the network.
Boltzmann Machine These are stochastic learning processes having recurrent structure and are the
basis of the early optimization techniques used in ANN. Boltzmann Machine was invented by
Geoffrey Hinton and Terry Sejnowski in 1985. More clarity can be observed in the words of
Hinton on Boltzmann Machine. “A surprising feature of this network is that it uses only locally
available information. The change of weight depends only on the behavior of the two units it
connects, even though the change optimizes a global measure” - Ackley, Hinton 1985.
Some important points about Boltzmann Machine −
 They use recurrent structure.
 They consist of stochastic neurons, which have one of the two possible states, either 1 or 0.
 Some of the neurons in this are adaptive freestate and some are clamped frozenstate.
 If we apply simulated annealing on discrete Hopfield network, then it would become Boltzmann
Machine.
Objective of Boltzmann Machine
The main purpose of Boltzmann Machine is to optimize the solution of a problem. It is the work of
Boltzmann Machine to optimize the weights and quantity related to that particular problem.
Architecture The following diagram shows the architecture of Boltzmann machine. It is clear from
the diagram, that it is a two-dimensional array of units. Here, weights on interconnections between
units are –p where p > 0. The weights of self-connections are given by b where b > 0.

Training Algorithm
As we know that Boltzmann machines have fixed weights, hence there will be no training
algorithm as we do not need to update the weights in the network. However, to test the network we
have to set the weights as well as to find the Consensus function CF. Boltzmann machine has a set
of units Ui and Uj and has bi-directional connections on them.
 We are considering the fixed weight say wij.
 wij ≠ 0 if Ui and Uj are connected. There also exists a symmetry in weighted interconnection,
i.e. wij = wji.
 wii also exists, i.e. there would be the self-connection between units.
 For any unit Ui, its state ui would be either 1 or 0. The main objective of Boltzmann Machine is
to maximize the Consensus Function CF which can be given by the following relation
Now, when the state changes from either 1 to 0 or from 0 to 1, then the change in consensus can be
given by the following relation

Here ui is the current state of Ui. The variation in coefficient (1 - 2ui) is given by the following
relation – Generally, unit Ui does not change its state, but if it does then the information would be
residing local to the unit. With that change, there would also be an increase in the consensus of the
network. Probability of the network to accept the change in the state of the unit is given by the
following relation –

Here, T is the controlling parameter. It will decrease as CF reaches the maximum value. Testing
Algorithm
Step 1 − Initialize the following to start the training
 Weights representing the constraint of the problem
 Control Parameter T
Step 2 − Continue steps 3-8, when the stopping condition is not true.
Step 3 − Perform steps 4-7.
Step 4 − Assume that one of the state has changed the weight and choose the
integer I, J as random values between 1 and n.
Step 5 − Calculate the change in consensus as follows
Step 6 − Calculate the probability that this network would accept the change in state
Step 7 − Accept or reject this change as follows −
Case I − if R < AF, accept the change.
Case II − if R ≥ AF, reject the change. Here, R is the random number between 0 and 1.
Step 8 − Reduce the control parameter temperature as follows − Tnew = 0.95Told
Step 9 − Test for the stopping conditions which may be as follows – Temperature reaches a
specified value
 There is no change in state for a specified number of iterations Restricted Boltzmann Machines
(RBM) A restricted term refers to that we are not allowed to connect the same type layer to each
other. In other words, the two neurons of the input layer or hidden layer can’t connect to each
other. Although the hidden layer and visible layer can be connected to each other. As in this
machine, there is no output layer so the question arises how we are going to identify, adjust the
weights and how to measure the that our prediction is accurate or not. All the questions have one
answer, that is Restricted Boltzmann Machine.
The RBM algorithm was proposed by Geoffrey Hinton (2007), which learns probability
distribution over its sample training data inputs. It has seen wide applications in different areas of
supervised/unsupervised machine learning such as feature learning, dimensionality reduction,
classification, collaborative filtering, and topic modeling. Consider the example movie rating.
Movies like Avengers, Avatar, and Interstellar have strong associations with the latest fantasy and
science fiction factor. Based on the user
 Temperature reaches a specified value
 There is no change in state for a specified number of iterations
Restricted Boltzmann Machines (RBM)
A restricted term refers to that we are not allowed to connect the same type layer to each other. In
other words, the two neurons of the input layer or hidden layer can’t connect to each other.
Although the hidden layer and visible layer can be connected to each [Link] in this machine,
there is no output layer so the question arises how we are going to identify, adjust the weights and
how to measure the that our prediction is accurate or not. All the questions have one answer, that is
Restricted Boltzmann [Link] RBM will discover latent factors that can explain the
activation of movie choices. In short, RBM describes variability among correlated variables of
input dataset in terms of a potentially lower number of unobserved [Link] energy function is
given by

How do Restricted Boltzmann Machines work?

In RBM there are two phases through which the entire RBM works: 1st Phase: In this phase, we
take the input layer and using the concept of weights and biased we are going to activate the hidden
layer. This process is said to be Feed Forward Pass. In Feed Forward Pass we are identifying the
positive association and negative association.
Feed Forward Equation:
 Positive Association — When the association between the visible unit and the
hidden unit is positive.
 Negative Association — When the association between the visible unit and the

2nd Phase: As we don’t have any output layer. Instead of calculating the output layer, we are
reconstructing the input layer through the activated hidden state. This process issaid to be Feed
Backward Pass. We are just backtracking the input layer through the activated hidden neurons.
After performing this we have reconstructed Input through the activated hidden state. So, we can
calculate the error and adjust weight in this way: Feed Backward Equation:
How do Restricted Boltzmann Machines work?
In RBM there are two phases through which the entire RBM works: 1st Phase: In this phase, we
take the input layer and using the concept of weights and biased we are going to activate the hidden
layer. This process is said to be Feed Forward Pass. In Feed Forward Pass we are identifying the
positive association and negative association.
Feed Forward Equation:
 Positive Association — When the association between the visible unit and the hidden unit is
positive.
 Negative Association — When the association between the visible unit and the hidden unit is
negative.
2nd Phase: As we don’t have any output layer. Instead of calculating the output layer,
 Error = Reconstructed Input Layer-Actual Input layer
 Adjust Weight = Input*error*learning rate (0.1)
After doing all the steps we get the pattern that is responsible to activate the hidden neurons. To
understand how it works: Let us consider an example in which we have some assumption that V1
visible unit activates the h1 and h2 hidden unit and V2 visible unit activates the h2 and h3 hidden.
Now when any new visible unit let V5 has come into the machine and it also activates the h1 and
h2 unit. So, we can back trace the hidden units easily and also identify that the characteristics of
the new V5 neuron is matching with that of V1. This is because V1 also activated the same hidden
unit earlier.

Types of RBM :
There are mainly two types of Restricted Boltzmann Machine (RBM) based on the types of
variables they use:
1. Binary RBM: In a binary RBM, the input and hidden units are binary variables.
Binary RBMs are often used in modeling binary data such as images or text.
2. Gaussian RBM: In a Gaussian RBM, the input and hidden units are continuous
variables that follow a Gaussian distribution. Gaussian RBMs are often used in
Variational Autoencoders
Variational Autoencoders (VAEs) are the most effective and useful process for Generative Models.
Generative models are used for generating new synthetic or artificial data from the real data. For
example, new music composition from currently composed music.
With the help of neural networks with the correct weight, autoencoders get trained to provide the
desired result. The neural network has an important role in Artificial Intelligence because of its
self-learning infrastructure capacity, and encoders can compress the data. Here is a simple visual
explanation of Autoencoder.
If we input image X and the encoder compresses data, which is also called dimension reductions
(you may be familiar with PCA or the common dimension reduction process), the encoder chooses
the best features (colour, size, shades, shape etc.) and stores highly compressed data in a space
called a bottleneck or latent space, this is called encoding process.
Similarly, the latent vector or bottleneck pushes data to the decoder, and itproduces output image
X’. With Loss = L(X, X’), we train the model to minimise the loss. And this process gets
automated, which is known as Autoencoder. Here, we have a bottleneck with discrete values. Thus,
this model tries to provide data which is close to original data. This is very useful in compression
and denoise the data. Autoencoder helps us store a lot of high data or a new image. Let us
understand how we are generating new data. Let’s say we have the image of a celebrity face from
which our encoder model has to recognize important features mentioned below. With every
feature, we have a probability distribution. Our goal is to produce new data from the current data or
a new face from the current face. How do faces differ? Skin tone, eye colour, hair colour, and
many other features are different. But overall, the list of the features remains the same.
Since we have a facility with two probability distributions: mean and standard deviations, we have
datasets of two new ranges to provide to the decoder. volume data and also helps dimension
reductions. If the Autoencoder is capable of handling this, why do we use Variational Autoencoder
(VAEs)?
The basic difference between autoencoder and variational encoder is its ability to provide
continuous data or a range of data in the latent space which is helping us to generate new data or a
new image. Let us understand how we are generating new data. Let’s say we have the image of a
celebrity face from which our encoder model has to recognize important features
mentioned below. With every feature, we have a probability distribution. Our goal is to produce
new data from the current data or a new face from the current face. How do faces differ? Skin tone,
eye colour, hair colour, and many other features are different. But overall, the list of the features
remains the same. Since we have a facility with two probability distributions: mean and standard
deviations, we have datasets of two new ranges to provide to the decoder. Lets see how this
actually gets processed in the Neural network. As our input data follows a normal distribution, we
will be able to provide two variables: mean and variance in the latent space. We want to build a
multivariate Gaussian model with the assumption of non-correlation in data which helps us result
in a simple vector. Now, provide a set of random samples from mean and variance distributions
from latent space to the decoder for the reproduction of data (image). Still, we do not get the
desired result unless we train this model to improvise with new samples every time. Since this is
not a one-time activity, we need to train the model. Backpropagation is oneof the important
processes to train the model. Since we have random sampling, we cannot perform
backpropagation, but we perform a reparameterization trick. We can randomly sample ε from a
unit Gaussian, and then shift the randomly sampled ε by the μ and scale it by σ.
Now we can backpropagate, and the autoencoder can learn to improvise. Let us now see

Now the most important part of the process is to identify the Loss function that helps to
train the model and to minimise the loss. In our case, VAEs loss functions consist of two
values.
Let’s us say encoding process as recognition model loss in recognition model will be
calculated with the sum of the square of means which will be:
L(x,x’)

Let’s say the decoding process is generation model and error will be the difference between
two distributions and which can be measured with KL divergence:

KL(q(z|x)||p(z))
Loss function of VAEs is:
L(x,x’) + ΣKL(q(z|x)||p(z))
We can conclude with a conceptual understanding of VAEs. This process is widely used to
generate new data for driverless vehicles, data transfer, new synthetic music and images. Deep
Backprop Networks
We would be seeing different kinds of Convolutional Neural Networks and how they differ from
each other. These are some groundbreaking CNN architectures that were proposed toachieve a
better accuracy and to reduce the computational cost .

1. LeNet-5
This is also known as the Classic Neural Network that was designed by Yann LeCun, Leon Bottou,
Yosuha Bengio and Patrick Haffner for handwritten and machine-printed character recognition in
1990’s which they called LeNet-5. The architecture was designed to identify handwritten digits in
the MNIST data-set. The architecture is pretty straightforward and simple to understand. The input
images were gray scale with dimension of 32*32*1 followed by two pairs of Convolution layer
with stride 2 and Average pooling layer with stride 1. Finally, fully connected layers with Softmax
activation in the output layer. Traditionally, this network had 60,000 parameters in total.

2. AlexNet
This network was very similar to LeNet-5 but was deeper with 8 layers, with more filters, stacked
convolutional layers, max pooling, dropout, data augmentation, ReLU and SGD. AlexNet was the
winner of the ImageNet ILSVRC-2012 competition, designed by Alex Krizhevsky, Ilya Sutskever
and Geoffery E. Hinton. It was trained on two Nvidia Geforce GTX 580 GPUs, therefore, the
network was split into two pipelines. AlexNet has 5 Convolution layers and 3 fully connected
layers. AlexNet consists of approximately 60 M parameters. A major drawback of this network
was that it comprises of too many hyperparameters.
3. ResNet
ResNet, the winner of ILSVRC-2015 competition are deep networks of over 100 layers. Residual
networks are similar to VGG nets however with a sequential approach they also use “Skip
connections” and “batch normalization” that helps to train deep layers without hampering the
performance. After VGG Nets, as CNNs were going deep, it was becoming hard to train them
because of vanishing gradients problem that makes the derivate infinitely small. Therefore, the
overall performance saturates or even degrades. The idea of skips connection came from highway
network where gated shortcut connections were used.

Normal Deep Networks vs Networks with skip connections For the above figure for network with
skip connection,
a[l+2]=g(w[l+2]a[l+1]+ a[l])
Lets say for some reason, due to weight decay w[l+2] becomes 0, therefore, a[l+2]=g(a[l]) Hence,
the layer that is introduced doesnot hurt the performance of the neural network. This the reason,
increasing layers doesn’t decrease the training accuracy as some layers may make the result worse.
The concept of skip connections can also be seen in LSTMs.

4. GoogleLe Net
Normal Deep Networks vs Networks with skip connections For the above figure for network with
skip connection,
a[l+2]=g(w[l+2]a[l+1]+ a[l])
Lets say for some reason, due to weight decay w[l+2] becomes 0, therefore, a[l+2]=g(a[l])
Inception network also known as GoogleLe Net was proposed by developers at google in “Going
Deeper with Convolutions” in 2014. The motivation of InceptionNet comes from the presence of
sparse features Salient parts in the image that can have a large variation in size. Due to this, the
selection of right kernel size becomes extremely difficult as big kernels are selected for global
features and small kernels when the features are locally located. The InceptionNets resolves this by
stacking multiple kernels at the same level. Typically it uses 5*5, 3*3 and 1*1 filters in one go. For
better understanding refer to the image below:

Note: Same padding is used to preserve the dimension of the image. As we can see in the image,
three different filters are applied in the same level and the output is combined and fed to the next
layer. The combination increases the overall number of channels in the output. The problem with
this structure was the number of parameter (120M approx.) that increases the computational cost.
Therefore, 1*1 filters were used before feeding the image directly to these filters that act as a
bottleneck and reduces the number of channels. Using 1*1 filters, the parameter were reduced to
1/10 of the actual. GoogLeNet has 9 such inception modules stacked linearly. It is 22 layers deep
(27, including Inception Module of GoogleLe Net
Note: Same padding is used to preserve the dimension of the image. As we can see in the image,
three different filters are applied in the same level and the the pooling layers). It uses global
average pooling at the end of the last inception module. Inception v2 and v3 were also mentioned
in the same paper that further increased the accuracy and decreasing computational cost.
Side branches can be seen in the network which predicts output in order to check the shallow
network performance at lower levels. Autoencoders Autoencoders are Neural Networks which are
commonly used for feature selection and extraction. However, when there are more nodes in the
hidden layer than there are inputs, the Network is risking to learn the so-called “Identity Function”,
also called “Null Function”, meaning that the output equals the input, marking the Autoencoder
useless. Denoising Autoencoders solve this problem by corrupting the data on purpose by
randomly turning some of the input values to zero. In general, the percentage of input nodes which
are being set to zero is about 50%. Other sources suggest a lower count, such as 30%. It depends
on the amount of data and input nodes you have.

When calculating the Loss function, it is important to compare the output values with the original
input, not with the corrupted input. That way, the risk of learning the identity function instead of
extracting features is eliminated.