CSC321 Lecture 20: Autoencoders
Roger Grosse
Roger Grosse CSC321 Lecture 20: Autoencoders 1 / 16
Overview
Latent variable models so far:
mixture models
Boltzmann machines
Both of these involve discrete latent variables. Now let’s talk about
continuous ones.
One use of continuous latent variables is dimensionality reduction
Roger Grosse CSC321 Lecture 20: Autoencoders 2 / 16
Autoencoders
An autoencoder is a feed-forward neural net whose job it is to take an
input x and predict x.
To make this non-trivial, we need to add a bottleneck layer whose
dimension is much smaller than the input.
Roger Grosse CSC321 Lecture 20: Autoencoders 3 / 16
Autoencoders
Why autoencoders?
Map high-dimensional data to two dimensions for visualization
Compression (i.e. reducing the file size)
Note: autoencoders don’t do this for free — it requires other ideas as
well.
Learn abstract features in an unsupervised way so you can apply them
to a supervised task
Unlabled data can be much more plentiful than labeled data
Roger Grosse CSC321 Lecture 20: Autoencoders 4 / 16
Principal Component Analysis
The simplest kind of autoencoder has one
hidden layer, linear activations, and squared
error loss.
L(x, x̃) = kx − x̃k2
This network computes x̃ = UVx, which is a
linear function.
If K ≥ D, we can choose U and V such that
UV is the identity. This isn’t very interesting.
But suppose K < D:
V maps x to a K -dimensional space, so it’s doing dimensionality
reduction.
The output must lie in a K -dimensional subspace, namely the column
space of U.
Roger Grosse CSC321 Lecture 20: Autoencoders 5 / 16
Principal Component Analysis
We just saw that a linear autoencoder has to map D-dimensional
inputs to a K -dimensional subspace S.
Knowing this, what is the best possible mapping it can choose?
Roger Grosse CSC321 Lecture 20: Autoencoders 6 / 16
Principal Component Analysis
We just saw that a linear autoencoder has to map D-dimensional
inputs to a K -dimensional subspace S.
Knowing this, what is the best possible mapping it can choose?
By definition, the projection of x onto S is the point in S which
minimizes the distance to x.
Fortunately, the linear autoencoder can represent projection onto S:
pick U = Q and V = Q> , where Q is an orthonormal basis for S.
Roger Grosse CSC321 Lecture 20: Autoencoders 6 / 16
Principal Component Analysis
The autoencoder should learn to choose the subspace which minimizes the
squared distance from the data to the projections.
This is equivalent to the subspace which maximizes the variance of the
projections.
By the Pythagorean Theorem,
N N
1 X (i) 1 X (i)
kx̃ − µk2 + kx − x̃(i) k2
N i=1 N i=1
| {z } | {z }
projected variance reconstruction error
N
1 X (i)
= kx − µk2
N i=1
| {z }
constant
You wouldn’t actually sove this problem by training a neural net. There’s a
closed-form solution, which you learn about in CSC 411.
The algorithm is called principal component analysis (PCA).
Roger Grosse CSC321 Lecture 20: Autoencoders 7 / 16
Principal Component Analysis
PCA for faces (“Eigenfaces”)
Roger Grosse CSC321 Lecture 20: Autoencoders 8 / 16
Principal Component Analysis
PCA for digits
Roger Grosse CSC321 Lecture 20: Autoencoders 9 / 16
Deep Autoencoders
Deep nonlinear autoencoders learn to project the data, not onto a
subspace, but onto a nonlinear manifold
This manifold is the image of the decoder.
This is a kind of nonlinear dimensionality reduction.
Roger Grosse CSC321 Lecture 20: Autoencoders 10 / 16
Deep Autoencoders
Nonlinear autoencoders can learn more powerful codes for a given
dimensionality, compared with linear autoencoders (PCA)
Roger Grosse CSC321 Lecture 20: Autoencoders 11 / 16
Layerwise Training
There’s a neat connection between autoencoders and RBMs.
An RBM is like an autoencoder with tied weights, except that the
units are sampled stochastically.
Roger Grosse CSC321 Lecture 20: Autoencoders 12 / 16
Layerwise Training
Suppose we’ve already trained an RBM with weights W(1) .
Let’s compute its hidden features on the training set, and feed that in
as data to another RBM:
Note that now W(1) is held fixed, but W(2) is being trained using
contrastive divergence.
Roger Grosse CSC321 Lecture 20: Autoencoders 13 / 16
Layerwise Training
A stack of two RBMs can be thought of as an autoencoder with three
hidden layers:
This gives a good initialization for the deep autoencoder. You can
then fine-tune the autoencoder weights using backprop.
This strategy is known as layerwise pre-training.
Roger Grosse CSC321 Lecture 20: Autoencoders 14 / 16
Autoencoders are not a probabilistic model.
However, there is an autoencoder-like probabilistic model called a
variational autoencoder (VAE). These are beyond the scope of the
course, and require some more advanced math.
Check out David Duvenaud’s excellent course “Differentiable
Inference and Generative Models”: [Link]
~duvenaud/courses/csc2541/[Link]
Roger Grosse CSC321 Lecture 20: Autoencoders 15 / 16
Deep Autoencoders
(Professor Hinton’s slides)
Roger Grosse CSC321 Lecture 20: Autoencoders 16 / 16