Empirical comparison between autoencoders and

traditional dimensionality reduction methods

Quentin Fournier Daniel Aloise
Ecole Polytechnique Montreal Ecole Polytechnique Montreal
Montreal, Quebec H3T 1J4 Montreal, Quebec H3T 1J4
[Link]@[Link] [Link]@[Link]

Therefore, neural networks may not be suitable for applica-

arXiv:2103.04874v1 [[Link]] 8 Mar 2021

Abstract—In order to process efficiently ever-higher dimen- tions whose resources are limited such as prototyping.
sional data such as images, sentences, or audio recordings, one
In this paper, we provide an analysis on how SVD-based
needs to find a proper way to reduce the dimensionality of such
data. In this regard, SVD-based methods including PCA and methods compare with neural networks in image classification
Isomap have been extensively used. Recently, a neural network tasks. Three reference image datasets were used, namely
alternative called autoencoder has been proposed and is often MNIST, Fashion-MNIST, and CIFAR-10. Two SVD-based
preferred for its higher flexibility. This work aims to show that methods and two autoencoders were applied to reduce the
PCA is still a relevant technique for dimensionality reduction in
dimension of each dataset and a k-NN classifier was trained
the context of classification. To this purpose, we evaluated the
performance of PCA compared to Isomap, a deep autoencoder, on each projection. This work brings an insight on how
and a variational autoencoder. Experiments were conducted on much autoencoders trade computation time off for projection’s
three commonly used image datasets: MNIST, Fashion-MNIST, quality as evaluated by the accuracy of a k-NN classifier.
and CIFAR-10. The four different dimensionality reduction The same analysis as evaluated by the logistic regression
techniques were separately employed on each dataset to project
and the quadratic discriminant analysis can be found in the
data into a low-dimensional space. Then a k-NN classifier was
trained on each projection with a cross-validated random search supplementary material1 . To the best of our knowledge, this is
over the number of neighbours. Interestingly, our experiments the first work which addresses the relevancy of PCA compared
revealed that k-NN achieved comparable accuracy on PCA and to autoencoders in the context of image classification.
both autoencoders’ projections provided a big enough dimension. The rest of this article is organized as follows: Section II
However, PCA computation time was two orders of magnitude
describes our proposed approach in more detail. Section III
faster than its neural network counterparts.
Index Terms—Machine learning, performance, classification, presents the experimental results. Finally, Section IV summa-
dimensionality reduction, PCA, autoencoder, k-NN rizes this work.

I. I NTRODUCTION II. P ROJECTION M ETHODS

In recent years, large sets of ever-higher dimensional data SVD-based methods and neural networks are two funda-
such as images [1]–[3], sentences, or audio recordings have mentally different approaches to dimensionality reduction. The
become the norm. One of the main problems that arise with former are exact and deterministic, whereas the latter settle
such data is called the curse of dimensionality. Since the for a small value of an objective function and are non-
volume of the data space increase exponentially fast with the deterministic between training runs.
dimension, the data becomes sparse. Another problem is that
distances computation time generally grows linearly with the A. SVD-based
dimension. In order to have a fair comparison, a simple linear method
In order to process such high-dimensional data, one can called PCA and a more complex non-linear one called Isomap
learn a projection into a lower-dimensional space. Singular were used.
value decomposition (SVD) based methods such as principal PCA finds the linear projection that best preserves the
component analysis (PCA) have been extensively used in that variance measured in the input space [5]. This is done by
regard. constructing the set of data’s eigenvectors and sorting them
In 2006, Hinton and Salakhutdinov [4] noted that a deep by their eigenvalues. Dimensionality reduction is done by
learning approach could be applied to dimensionality reduction projecting the data along the first k eigenvectors, where k
using neural networks that learn to predict their input. Such is the dimension of the reduced space. Although PCA is both
networks are called autoencoders and, once trained, yield a easy to use and very efficient, its effectiveness is limited when
non-linear dimensionality reduction that outperforms SVD- data is not linearly correlated.
based methods.
However, neural networks require more computation time 1 [Link] reduction/blob/master/
or resources to be trained than most SVD-based methods. supplementary [Link]

©2019 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including
reprinting/republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or
reuse of any copyrighted component of this work in other works. DOI: 10.1109/AIKE.2019.00044
Fig. 1. An example of a deep autoencoder. The encoder is in orange and Fig. 2. An example of a variational autoencoder. The encoder is in orange
the decoder in blue. The 1-dimension hidden representation is the bicoloured and the decoder in blue. Note that the hidden representation z is a density
node. from which a value s is sampled.

The isometric feature mapping (Isomap) [6] is based on When applied to dimensionality reduction, the autoencoder
classical multidimensional scaling (MDS), which finds a non- is trained with both the encoder and decoder. Then the decoder
linear embedding that best preserve inter-point distances. How- is discarded and the output of the encoder is treated as the
ever, instead of computing the matrix of inter-point distances data’s projection. This approach yields a non-linear general-
in one step, Isomap first compute the distance between each ization of PCA provided that both the encoder and decoder
point and its k-nearest neighbours, and constructs a weighted have at least one hidden layer [4]. Note that for the VAE, the
graph G. Inter-point distances are then evaluated by finding sampled vector s is used.
the shortest path in G. Finally, classical MDS is applied to the
distance matrix. By computing interpoint distances iteratively, III. E XPERIMENTS
Isomap is able to learn a non-linear embedding that preserve Let us describe the experimental framework: each dataset
the intrinsic geometry of the data. has already been fairly split into a training set and a test set.
B. Neural Networks In the case of autoencoders, a tenth of the training set is used
as a validation set to tune the hyperparameters and to evaluate
Let us first introduce the framework of an autoencoder.
the stopping criterion during training. A k-nearest neighbours
Given an input x, its projection z, and its reconstruction x0 ,
(k-NN) classifier is trained on each learned projection of the
an autoencoder is composed of two networks:
training set, with a cross-validated random search over k.
– An encoder defined by the function f (x) = z such that Finally, the test set is projected by each method and classified
x is the input and z is the output of the network. by the associated k-NN. We investigate the accuracy of the
– A decoder defined by the function g(z) = x0 such that z k-NN algorithm for different projection dimensions.
is the input and x0 is the output of the network.
The training objective is to minimize the distance between A. Datasets
the input x and its reconstruction g(f (x)) = x0 . It is necessary Methods described in section II can be applied to any vector-
to limit the capacity of the model to copy its input on its output represented data. In this study, we focus on small images. We
in order to force the autoencoder to extract useful properties. used the well-known database of handwritten digits MNIST [2]
One can impose a regularization term or limit the dimension and its modern fashion analogue called Fashion-MNIST [3].
of the projection z. Both datasets consist of (28 × 28) grayscale images divided as
Hereafter, we will use two autoencoders: a deep autoencoder 60,000 training and 10,000 testing examples. In addition, we
(DAE, Fig.1) and a variational autoencoder (VAE, Fig.2). The used CIFAR-10 [1] which is composed of (32 × 32) coloured
former is a standard network whose encoder and decoder are images divided as 50,000 training examples and 10,000 test
multilayer perceptrons. The latter is a generative model first examples. All three datasets contain ten equally distributed
introduced in 2013 by Kingma et al. [7] and differs by its latent classes. Images were normalized to speed up the training and
space z being non-deterministic. Hence, VAEs can both gener- were treated as 1-dimensional vectors of sizes 784 and 3072,
ate new samples and provide a probabilistic low-dimensional respectively.
projection. Given the model’s parameters θ, the projection
of x into z is a Gaussian probability density noted qθ (z|x). B. Detailed Architectures and Parameter Settings
This prior knowledge is embedded into the objective function Experiments were implemented in Python using Keras and
through the following regularization term KL(qθ (z|x)||p(z)) scikit-learn libraries. Training times were computed on a
which is Kullback-Leibler divergence between the encoder remote server equipped with two 16-core processors and two
distribution and the expected distribution p(z) ∼ N (0, 1). Titan Xp. Please note that neural networks training times
Although autoencoders can be specialized to some type of depend highly on the graphics card used. The code is available
input data (e.g., by adding convolutions for the processing of on Github2 .
images), we decided against to keep the comparison as fair as
possible. 2 [Link] reduction
 TABLE I 1.0
H IGHEST k-NN ACCURACY ± STANDARD DEVIATION ( IN %). 0.9
MNIST Fashion-MNIST CIFAR-10 0.8
PCA 97.48 ± 0.08 85.56 ± 0.05 42.35 ± 0.23
94.87 ± 0.16 81.17 ± 0.22 33.61 ± 0.52 0.7

Accuracy
ISOMAP
DAE 97.82 ± 0.08 87.62 ± 0.11 45.77 ± 0.37
VAE 97.78 ± 0.08 86.90 ± 0.17 44.39 ± 0.37 0.6
0.5 PCA
ISOMAP
0.4 DAE
Encoders were five layers deep with each layer half the size VAE
0.3 Baseline (100%)
of the previous one. The projection’s size is a parameter that
ranges from 1 to 99. Batch normalization was applied before 0.0% 1.0% 2.0% 3.0% 4.0% 5.0%
Dimension
each activation function. The non-linearity of choice was the 0.9
Rectified Linear Unit (ReLU) except for the last layer which
0.8
used a linear activation function. Decoders were symmetric
with the exception that the output layer’s activation function 0.7
was the sigmoid, providing better results due to the input being

Accuracy
normalized. Adam [8] optimizer was used as it offers both fast 0.6
training and good generalization performance. Finally, early 0.5
stopping was applied after a patience of 10 iterations. Weights PCA
ISOMAP
were initialized based on the uniform distribution suggested by 0.4 DAE
VAE
Glorot and Bengio [9]. Baseline (100%)
0.3
√ k was subject to a random search
The number of neighbours 0.0% 1.0% 2.0% 3.0% 4.0% 5.0%
over 60 different k ∈ [1, n] with n the size of the training set. Dimension
The test accuracy was estimated with a 5-fold cross-validation. 0.45
Isomap complexity is O(n3 ) [10] with n is the number
of examples. In order to make the experiments feasible, this 0.40
method was trained on a sample of size 10,0003 . Empirically, 0.35
Accuracy

we found that more training examples didn’t offer any sub-

stantial gains compared to the computational cost. 0.30

0.25 PCA
C. Results ISOMAP
0.20 DAE
Neural networks are non-deterministic as they are trained VAE
0.15 Baseline (100%)
with random batches of examples and their weights are ran-
domly initialized. In addition, all projection methods are eval- 0.0% 0.5% 1.0% 1.5% 2.0% 2.5% 3.0%
Dimension
uated with a k-NN classifier whose hyperparameter is selected
through a random search. Finally, computation times depend Fig. 3. k-NN mean accuracy with regards to the relative dimension of the
projection. The shade indicates the standard deviation. The baseline is obtained
to some degree on various factors out of our control. In order by training k-NN on the original space (100% of the dimensions). From top
to take these random factors into account, experiments were to bottom row: MNIST, Fashion-MNIST, and CIFAR-10.
repeated 5 times and the mean with the standard deviation is
reported.
1) Accuracy: Let us first consider the dimension’s impact The difference of accuracy between PCA and autoencoder
on k-NN mean accuracy (Fig.3). As expected, for small best projection is only 0.34 ± 0.08%, 2.06 ± 0.09% and
dimensions, PCA yielded the lowest accuracy as it is not 3.42 ± 0.31% on MNIST, Fashion-MNIST, and CIFAR-10
able to grasp non-linearity, and consequently, to learn complex respectively. Asymptotically, Isomap consistently yielded the
transformations. Isomap yielded slightly better accuracy than lowest accuracy.
PCA as it is able to learn a projection that preserves the in- On CIFAR-10, there seems to be an optimal projection’s
trinsic structure of the data. Both neural networks consistently size for PCA. While smaller dimensions do not hold sufficient
yielded the best accuracy as they are able to learn highly non- information to correctly classify the data, it is likely that higher
linear transformations. dimensions bring more noise than information, making the
classification task harder. Note that k-NN is not well suited for
As we allow the projection’s dimension to grow, the differ-
CIFAR-10 – hence its poor accuracy – and should be replaced
ence between each method reduces. PCA and neural networks
in practice.
yielded comparable accuracy as it became simpler to learn a
Note that accuracy’s standard deviation is very low – barely
projection that preserves enough information for classification.
visible on the graphs – which indicate that all methods
3 Note that it would take around 63 = 216 and 53 = 125 longer to train repeatedly learned a similar projection in the context of
Isomap with MNIST and CIFAR full training set, respectively. classification.
 The average standard deviation of DAE and VAE computation
time is 36.2s (12.8%) and 67.1s (18.2%), respectively. It can
102 be challenging to estimate the computational time required
to train a neural network, whereas PCA average standard
Training Time (s)

PCA
ISOMAP deviation is only 0.04s (3.8%).
DAE Even when there is an optimal dimension and that a random
101 VAE
search over the size of PCA projection is needed, PCA is still
faster. On CIFAR-10, it took 43 ± 2s to compute all PCA
100 projections, whereas it took in average 249±32s and 259±45s
to train a deep and a variational autoencoder, respectively.
0.0% 1.0% 2.0% 3.0% 4.0% 5.0% One can reduce the computation time of neural networks by
Dimension
103 using more powerful graphic cards. Note that the Titan Xps
used are high-end GPUs worth US$ 1,200 each in 2018.
IV. C ONCLUSION
102
On three commonly used image datasets, and in a context
Training Time (s)

PCA
ISOMAP of k-NN classification, PCA allows for a comparable accuracy
DAE as autoencoders at a fraction of the computation time and re-
101 VAE
sources. Supplementary material shows the same trend on both
logistic regression and quadratic discriminant analysis. This
100 observation holds true even if one needs to do a random search
on PCA projection size. Furthermore, SVD-based methods do
0.0% 1.0% 2.0% 3.0% 4.0% 5.0% not require an expensive GPU.
Dimension
Our advice would be to use PCA while prototyping on
large datasets to speed up computations, after which it can be
replaced by a more flexible method such as an autoencoder to
improve performance.
102
Training Time (s)

PCA Following work should include a broader range of datasets.

ISOMAP
DAE ACKNOWLEDGMENT
VAE
101 This research was financed by the Natural Sciences and
Engineering Research Council of Canada (NSERC) under
grant 2017-05617. This research was enabled in part by
support provided by Calcul Quebec and Compute Canada.
0.0% 0.5% 1.0% 1.5% 2.0% 2.5% 3.0%
Dimension R EFERENCES
[1] A. Krizhevsky, “Learning multiple layers of features from tiny images,”
Fig. 4. Computation time in second with regards to the relative dimension tech. rep., 2009.
of the projection. From top to bottom row: MNIST, Fashion-MNIST, and [2] Y. LeCun and C. Cortes, “MNIST handwritten digit database,” 2010.
CIFAR-10. [3] H. Xiao, K. Rasul, and R. Vollgraf, “Fashion-mnist: a novel image
dataset for benchmarking machine learning algorithms,” 2017.
[4] G. E. Hinton and R. R. Salakhutdinov, “Reducing the dimensionality of
2) Computation Time: Figure 4 reports the average com- data with neural networks,” Science, vol. 313, pp. 504–507, July 2006.
[5] J. Shlens, “A tutorial on principal component analysis,” CoRR,
putation time of each projection method. Note that it does not vol. abs/1404.1100, 2014.
include the k-NN classifier and that the scale is logarithmic. [6] J. B. Tenenbaum, V. de Silva, and J. C. Langford, “A global geometric
The computation time depends very little on the dimension framework for nonlinear dimensionality reduction,” Science, vol. 290,
no. 5500, p. 2319, 2000.
as SVD-based methods usually need to find every eigenvector4 [7] D. P. Kingma and M. Welling, “Auto-encoding variational bayes.,”
and autoencoder’s number of parameters vary very little. PCA, CoRR, vol. abs/1312.6114, 2013.
which is the simplest method to compute, was the fastest by [8] D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,”
CoRR, vol. abs/1412.6980, 2014.
two orders of magnitude. Isomap, whose number of training [9] X. Glorot and Y. Bengio, “Understanding the difficulty of training
examples was capped at 10,000, took around 100 seconds to deep feedforward neural networks,” in Proceedings of the Thirteenth
train. Compared to Isomap, DAE and VAE were about twice International Conference on Artificial Intelligence and Statistics (Y. W.
Teh and M. Titterington, eds.), vol. 9 of Proceedings of Machine
and four times as long to train, respectively. Learning Research, (Chia Laguna Resort, Sardinia, Italy), pp. 249–256,
Neural networks’ training time depends to some extent on PMLR, 13–15 May 2010.
[10] C. Ou, D. Sun, Z. Wang, X. Zhou, and W. Cheng, “Manifold learning
the number of epochs, which is controlled by early stopping. towards masking implementations: A first study,” IACR Cryptology
ePrint Archive, vol. 2017, p. 1112, 2017.
4 Scikit-learn implementation of PCA use randomized SVD introduced by [11] N. Halko, P.-G. Martinsson, and J. A. Tropp, “Finding structure with
Halko et al. [11] which does not require to compute the complete eigende- randomness: Probabilistic algorithms for constructing approximate ma-
composition, hence the increase in computation time with the dimension. trix decompositions,” arXiv e-prints, p. arXiv:0909.4061, Sep 2009.