Autoencoders
Autoencoders
Autoencoder is an unsupervised artificial neural
network that learns how to efficiently compress and
encode data then learns how to reconstruct the data
back from the reduced encoded representation to a
representation that is as close to the original input as
possible.
Autoencoder, by design, reduces data dimensions by
learning how to ignore the noise in the data.
Continue..
An autoencoder is a type of artificial neural network
used to learn efficient data codings in an
unsupervised manner.
The goal of an autoencoder is to: learn a
representation for a set of data, usually for
dimensionality reduction by training the network to
ignore signal noise.
Continue..
An autoencoder where dim(h) < dim(xi) is called an
under complete autoencoder.
An autoencoder where dim(h) ≥ dim(xi) is called an
over complete autoencoder. This means the
dimension of the latent representation is greater than
the input.
Types of Autoencoders
There are following types of autoencoders:
1. Denoising autoencoder
2. Sparse Autoencoder
3. Contractive Autoencoder
4. Convolutional Autoencoder
1. Denoising Autoencoder
Denoising autoencoders create a corrupted copy of the
input by introducing some noise. This helps to avoid
the autoencoders to copy the input to the output
without learning features about the data. These
autoencoders take a partially corrupted input while
training to recover the original undistorted input.
The model learns a vector field for mapping the input
data towards a lower dimensional manifold which
describes the natural data to cancel out the added
noise.
Advantages-
i. It was introduced to achieve good representation. Such
a representation is one that can be obtained robustly
from a corrupted input and that will be useful for
recovering the corresponding clean input.
ii. Corruption of the input can be done randomly by
making some of the input as zero. Remaining nodes
copy the input to the noised input.
iii. Minimizes the loss function between the output node
and the corrupted input.
iv. Setting up a single-thread denoising autoencoder is
easy.
Drawbacks-
i. To train an autoencoder to denoise data, it is necessary to
perform preliminary stochastic mapping in order to corrupt
the data and use as input.
ii. This model isn't able to develop a mapping which
memorizes the training data because our input and target
output are no longer the same.
2. Sparse autoencoders
Sparse autoencoders have hidden nodes greater than
input nodes. They can still discover important
features from the data.
A generic sparse autoencoder is visualized where the
obscurity of a node corresponds with the level of
activation. Sparsity constraint is introduced on the
hidden layer. This is to prevent output layer copy
input data.
Advantages-
i. Sparse autoencoders have a sparsity penalty, a value
close to zero but not exactly zero. Sparsity penalty is
applied on the hidden layer in addition to the
reconstruction error. This prevents overfitting.
ii. They take the highest activation values in the hidden
layer and zero out the rest of the hidden nodes. This
prevents autoencoders to use all of the hidden nodes at
a time and forcing only a reduced number of hidden
nodes to be used.
Drawbacks-
i. For it to be working, it's essential that the
individual nodes of a trained model which activate
are data dependent, and that different inputs will
result in activations of different nodes through the
network.
3. Contractive Autoencoder
The objective of a contractive autoencoder is to have a robust
learned representation which is less sensitive to small
variation in the data.
Robustness of the representation for the data is done by
applying a penalty term to the loss function.
Contractive autoencoder is another regularization technique
just like sparse and denoising autoencoders.
Advantages-
i. Contractive autoencoder is a better choice than
denoising autoencoder to learn useful feature
extraction.
ii. This model learns an encoding in which similar
inputs have similar encodings. Hence, we're forcing
the model to learn how to contract a neighborhood
of inputs into a smaller neighborhood of outputs.
4. Convolutional
Autoencoders
Autoencoders in their traditional formulation does
not take into account the fact that a signal can be seen
as a sum of other signals.
Convolutional Autoencoders use the convolution
operator to exploit this observation. They learn to
encode the input in a set of simple signals and then
try to reconstruct the input from them, modify the
geometry or the reflectance of the image.
Pros &Cons
Advantages-
i. Due to their convolutional nature, they scale well to
realistic-sized high dimensional images.
ii. Can remove noise from picture or reconstruct missing parts.
Drawbacks-
i. The reconstruction of the input image is often blurry and of
lower quality due to compression during which information
is lost.