PROCEEDINGS OF THE IEEE 1

Tensor Methods in Computer Vision and

Deep Learning
Yannis Panagakis∗ , Jean Kossaifi∗ , Grigorios G. Chrysos, James Oldfield, Mihalis A. Nicolaou,
Anima Anandkumar and Stefanos Zafeiriou

Abstract—Tensors, or multidimensional arrays, are data struc- body quantum systems [1] to algebraic geometry and the-
tures that can naturally represent visual data of multiple dimen- oretical computer science [2], to mention a few. In data
sions. Inherently able to efficiently capture structured, latent science and related fields such as machine learning, signal
semantic spaces and high-order interactions, tensors have a
long history of applications in a wide span of computer vision processing, computer vision, and statistics, tensors have been
arXiv:2107.03436v1 [[Link]] 7 Jul 2021

problems. With the advent of the deep learning paradigm shift used to represent and analyze information hidden in multi-
in computer vision, tensors have become even more funda- dimensional data, such as images and videos, or to capture
mental. Indeed, essential ingredients in modern deep learning and exploit higher-order similarities or dependencies among
architectures, such as convolutions and attention mechanisms, vector-valued variables. Just as pair-wise (i.e., second-order)
can readily be considered as tensor mappings. In effect, tensor
methods are increasingly finding significant applications in deep similarities of vector samples are represented by covariance
learning, including the design of memory and compute efficient or correlation matrices, a third- (or higher)-order tensor can
network architectures, improving robustness to random noise and represent similarities among three (or more) samples in a set.
adversarial attacks, and aiding the theoretical understanding of Such tensors are often referred to as higher-order statistics.
deep networks. In this context, tensor methods mainly focus on extending
This article provides an in-depth and practical review of
tensors and tensor methods in the context of representation matrix-based learning models such as component analysis [3],
learning and deep learning, with a particular focus on visual data dictionary learning e.g., [4], and regression models e.g., [5], to
analysis and computer vision applications. Concretely, besides handle data represented by higher-order tensors or estimating
fundamental work in tensor-based visual data analysis methods, parameters of latent variable models by analyzing higher-
we focus on recent developments that have brought on a gradual order statistics [6]. Such developments were for a long time
increase of tensor methods, especially in deep learning architec-
tures, and their implications in computer vision applications. To independent of the rise of deep learning, where the concept of
further enable the newcomer to grasp such concepts quickly, we a tensor is central. Concretely, besides representing data and
provide companion Python notebooks, covering key aspects of statistics, tensors can be viewed as multilinear mappings (i.e.,
the paper and implementing them, step-by-step with TensorLy. functions), which are higher-order generalizations of linear
Index Terms—Tensor Methods, Computer Vision, Deep Learn- mappings represented by matrices. For instance, multichannel
ing convolutional kernels, which are the fundamental building
blocks of deep convolutional networks, can be readily de-
I. I NTRODUCTION scribed using the language of tensors.
This article provides an overview of tensors and tensor
Tensors are a multidimensional generalization of matrices, methods in the context of representation learning and deep
and a core mathematical object in multilinear algebra, just learning, with a particular focus on visual data analysis and
like vectors and matrices are in linear algebra. The order computer vision applications.
of a tensor is the number of indices needed to address its Tensor structures in visual data. Visual data is a
elements. A matrix is a second-order tensor of two modes, prominent example of multi-dimensional data where a tensor
namely rows and columns, requiring two indices to access its structure is inherent in two main forms:
elements. In analogy, an N th order tensor has N number of • Tensor structure of visual measurements: Visual data
modes, and N indices index it. consist of physically meaningful modes that describe
Tensors have found tremendous applications in a wide spatial, color, and temporal aspects of the data. Indeed,
range of disciplines in science and engineering; from many- visual sensors readily generate tensor data: A grayscale
Yannis Panagakis is with the Department of Informatics and Telecommu- image is represented by a matrix whose elements capture
nications, National and Kapodistrian University of Athens, Greece. the intensity of light along with the spatial coordinates
Jean Kossaifi is with NVIDIA. of the image. Hence, a batch of grayscale images is
Grigorios G. Chrysos is with the Department of Electrical Engineering,
Ecole Polytechnique Federale de Lausanne (EPFL), Switzerland. conveniently represented by a third-order tensor with two
James Oldfield is with Queen Mary University of London spatial modes and one mode indexing different images
Mihalis A. Nicolaou is with the Computation-based Science and Technol- in the batch. A third-order tensor is also a natural way
ogy Re-search Center at The Cyprus Institute.
Anima Anandkumar is with Caltech and NVIDIA. to represent a color image where color channels (e.g.,
Stefanos Zafeiriou is with the Department of Computing, Imperial College RGB) are stacked along a third mode. Similarly, in
London. tensor representations of hyperspectral and multispectral
∗ Corresponding authors: Yannis Panagakis, email: yannisp@[Link], Jean
Kossaifi: email: [Link]@[Link] images, different channels capturing different spectral
bands of light are stacked along with the third mode.
PROCEEDINGS OF THE IEEE 2

Tensors of fourth-order are also being used to represent in representations that correspond to local parts of the visual
visual data that includes a time mode, in addition to object [9]. Decomposing visual data into shape and motion
the spatial and spectral/color modes. Higher-order tensor factors, or surface normal and reflectance factors, facilitates
structure also arises in medical imaging. For instance, the recovery of structure from motion and provides solutions
magnetic resonance imaging (MRI) and functional mag- to photometric stereo, e.g., [10]. Tensor decompositions with
netic resonance imaging (fMRI), are naturally stored in sparse regularization [11], [4] have proven to be very useful in
a multidimensional array such as a matrix (e.g., a two- inverse problems in imaging, such as denoising and inpainting.
dimensional (2D) MRI image), a third-order tensor (e.g., Recently, deep learning models have led to qualitative
a three-dimensional (3D) MRI image), or a fourth-order breakthroughs on a wide variety of computer vision
tensor (e.g., a 3D fMRI image over time). applications, among many other machine learning tasks
• Latent tensor structure in image formation: Image forma- [12]. Deep neural networks act as learnable approximations
tion relies upon the interactions of multiple latent factors of high-dimensional, non-linear functions. For example, in
of variation related to appearance (e.g., illumination, image classification, one seeks to learn an unknown high-
pose) or even image semantics (such as gender and age in dimensional target function that maps an input image to the
human faces). Such factors are often entangled through output label. The key ingredient of their success is that deep
a multilinear (tensor) mapping in an unobserved, latent learning models leverage statistical properties of data such as
space that gives rise to the rich visual data structure. This stationarity (e.g., shift-invariance) and compositionality (e.g.,
tensor mapping assumes that visual variation is linear if the hierarchical structure of images) through local statistics,
we keep all but one factor constant. For example, in facial which are present in visual data [13], [14], [15]. These
images, changing the illumination does not change the properties are exploited by convolutional architectures [13],
depicted person’s identity. [16], which are built of alternating multidimensional convo-
lutional layers, point-wise nonlinear functions (e.g., ReLU),
Typical computer vision and visual data analysis appli- downsampling (pooling) layers, while also containing tensor-
cations, such as detecting objects and recognizing faces in structured fully-connected layers. The use of multidimensional
images and videos, understanding human behavior in videos convolutions in deep neural networks allows extracting local
(e.g., action and activity recognition), enhancing the quality features shared across the image domain. In turn, this greatly
of images (i.e., image denoising and inpainting), and gener- reduces the number of learnable parameters and consequently
ating novel visual content (i.e., image synthesis), require the alleviates the effect of the curse of dimensionality–without
extraction of physically–or semantically–meaningful represen- sacrificing the ability to approximate the target function.
tations of visual data. Traditional machine learning models Tensor methods in deep learning architectures for com-
employed towards this end, such as component analysis, sparse puter vision: Despite the compositional structure of deep neu-
coding, and regression functions, treat data points as vectors ral networks that mitigates the curse of dimensionality, deep
and datasets as matrices. Hence, multi-dimensional visual learning models are usually over-parameterized involving an
data samples need to be flattened into very high-dimensional enormous number (typically tens of millions or even billions)
vectors, where natural topological structures and dependencies of unknown parameters. Even though over-parametrization
among different modes (e.g., spatial and temporal) of measure- of deep neural networks allows finding good local minima,
ments are lost. Besides structure loss, when high-dimensional particularly when they are trained with gradient descent [17],
vectors are employed in training matrix-based machine learn- [18], such a large number of trainable parameters makes deep
ing models, the number of data samples needed to estimate networks prone to overfitting and noise [19], hindering the
an arbitrary function (or parameters of a machine learning analysis of their generalization properties (i.e., how well they
model) within a given level of accuracy grows exponentially perform on unseen data) [20]. Tensor decompositions can
with data dimensionality. This phenomenon is known as the significantly reduce the number of unknown parameters of
curse of dimensionality [7]. deep models and further mitigate the curse of dimensionality.
Tensor methods in representation learning for computer Indeed, tensor decompositions can be applied to the weights
vision: Tensor methods have emerged as a powerful tool of neural network layers to compress them, and in some cases,
for learning representations from multidimensional data by speed them up [21], [22], [23]. In addition, using tensor-based
mitigating the curse of dimensionality without discarding operations within deep neural networks allows to preserve and
their structure, which carries useful information in the case leverage the topological structure in the data while resulting
of visual data. Indeed, being able to both recover latent in parsimonious models [24], [25]. In convolutional neural
factors in visual data and flexible enough to accommodate networks (CNNs), (non-separable) multichannel convolution
a large set of structural constraints and regularizations, tensor kernels can be decomposed into a sum of mode-separable
decomposition and tensor component analysis methods have convolutions using low-rank tensor decompositions, resulting
had a transformative impact in a wide range of computer vision in network compression and a reduction in the number of
applications, ranging from human sensing (face and body parameters [26], [27]. Deep network compression is significant
analysis) to medical and hyperspectral imaging. In this context, in the deployment of deep learning models in resource-
dimensionality reduction, clustering, and data compression limited devices, while reducing the number of parameters acts
rely on decompositions of tensors representing visual data as implicit regularization that improves generalization across
[8]. Non-negative factorization of such data tensors results tasks and domains [28], [29], [30]. Furthermore, randomized
PROCEEDINGS OF THE IEEE 3

tensor decomposition of neural networks’ building blocks has To ensure consistency in terminology that would be familiar
been proved effective in robustifying neural networks against to machine learning and computer vision researchers we have
adversarial attacks [31], [32] and to various types of random adopted the terminology employed in relevant overview papers
noise–such as noise occurring naturally when capturing data [36], [34], [6] and recent publications on these topics. The
such as MRI [33]. notation used here is very similar to that adopted in [36], [34],
Surveys that approach tensors from varying perspectives [43]. However, we differentiate on one point: we employ a
have emerged over the past years. In [34], one of the most different ordering of elements when tensors are flattened to
prominent and influential overviews of tensor decomposi- matrices than that used in [34] and may be familiar to some
tions is presented, while in [35], a mathematical account of the readers (cf. Definition II.1).
for multilinear mappings is provided. While [34], [35] target
audiences mostly from an applied mathematics perspective, A. Notation
other surveys focus on applications of tensors in scientific
and engineering fields; indicatively, in areas such as signal Scalars are denoted by plain letters i.e., x, I, J. First order
processing and machine learning [36], [37], data mining and tensors are vectors, denoted as x. Second order tensors are
fusion [38], multidimensional data analysis [39], [40], blind matrices, denoted as X. I denotes the identity matrix of
source separation [41], computer vision [3], [8], and spectral compatible dimensions. The transpose of X is denoted X> .
learning [6], [42]. The ith column of X is denoted as xi .
While the surveys mentioned above cover an impressive The sets of real and integer numbers are denoted by R and
spectrum of theoretical and algorithmic developments and Z, respectively. A set of M real matrices (vectors) of varying
applications, one can identify a gap in the literature concerning dimensions is denoted by {X(m) ∈ RIm ×N }M m=1 ({x
(m)
∈
Im M
recent advancements in the use of tensor methods in the con- R }m=1 ). Finally, for any i, j ∈ Z, with i < j, [i . . j]
text of representation learning and deep learning for computer denotes the set of integers {i, i + 1, · · · , j − 1, j}.
vision. In this paper, besides fundamental work in tensor learn- Tensors of order three or higher are denoted by X . The
ing for computer vision, we focus on recent developments that reader is reminded that the order of a tensor is the number
have brought on a gradual increase of tensor-related methods, of indices (dimensions) needed to address its elements. Each
especially in deep learning architectures, and their implications dimension is called a mode. Specifically, an N th order tensor
in vision applications. To further enable the newcomer to has N indices, with each index addressing a mode of X .
grasp such concepts quickly, we provide thorough tutorial- Assuming X is real-valued, it is defined over the tensor space
style companion notebooks in Python, covering every section RI1 ×I2 ×···×IN , where In ∈ Z for n = 1, 2, . . . , N . An element
of the paper1 . (i1 , i2 , · · · , iN ) of tensor X ∈ RI1 ×I2 ×···×IN is accessed as:
Structure of the paper. The remainder of the paper is Xi1 ,i2 ,··· ,iN or X (i1 , i2 , · · · , iN ). This corresponds to view-
organized as follows. In Section II, tensor and matrix algebra ing a tensor as a multi-dimensional array in RI1 ×I2 ×···×IN .
fundamentals are presented. In Section III, an overview of Furthermore, given a set of N vectors (or matrices) that
existing computational tools and the necessary infrastructure correspond to each mode of X , the nth vector (or matrix) is
is discussed, focusing on the TensorLy library on which the denoted as u(n) (or U(n) ).
companion notebooks of this paper are based. Section IV pro- Fibers are a generalization of the concept of rows and
vides an introduction to representation learning with tensors. columns of matrices to tensors. They are obtained by fixing
In Section V, recent research that utilizes tensor methods in all indices but one. A colon is used to denote all elements of
deep learning architectures are detailed, while related applica- a mode e.g., the mode-1 fibers of X are denoted as X:,i2 ,i3 .
tions in computer vision are discussed in Section VI. Finally, Slices (Figure 1) are obtained, for a third-order tensor, by
a summary of practical challenges is presented in Section VII, fixing one of the indices.
along with advice to practitioners.
B. Transforming tensors into matrices and vectors
II. P RELIMINARIES ON M ATRICES AND T ENSORS Definition II.1 (Tensor unfolding): Given a tensor, X ∈
Notebook: tensor [Link] RI1 ×I2 ×···×IN , its mode-n unfolding is a matrix X[n] ∈
QN
RIn ×IM , with M = k=1, Ik and is defined by the map-
k6=n
Multilinear algebra is much richer than its linear counterpart
and hence tensor algebra notation and manipulation of tensors PN from elementQ(iN1 , i2 , · · · , iN ) to (in , j), with j = 1 +
ping
k=1, (ik − 1) × m=k+1, Im .
may appear complicated to newcomers in the field. To make k6=n m6=n
the paper self-contained and give the reader an understanding Tensor unfolding corresponds to a reordering of the fibers
of the mathematical tools employed in tensor methods for of the tensor as the columns of a matrix. There are more than
learning, we first introduce the notation conventions used in one definition of tensor unfolding, which vary by the ordering
this paper and then review some of the fundamental concepts of the fibers. In particular, our definition corresponds to a row-
of linear and mutlilinear algebra. wise underlying ordering (or C-ordering ) of the elements. The
1 The GitHub repository containing the notebooks can be found here: https://
main other definition, popularised by Kolda and Bader [34],
[Link]/tensorly/Proceedings IEEE companion notebooks. Specific note- corresponds to a column-wise (or Fortran) ordering of the ele-
books are linked in a gray box under the relevant section title. ments, which is more efficient for libraries using that ordering,
PROCEEDINGS OF THE IEEE 4

(a) Horizontal Slices (b) Lateral Slices (c) Frontal Slices

Fig. 1: Illustration of the slices of a third order tensor of size 3 × 4 × 2

mode-0 fibers mode-1 fibers mode-2 fibers

Fibers

mode-0 unfolding mode-1 unfolding mode-2 unfolding

TensorLy

mode-0 unfolding mode-1 unfolding mode-2 unfolding

Matlab

TABLE I: Illustration of the fibers and unfolding of a third order tensor.

such as MATLAB. Conversely, our definition naturally leads to columns, their Khatri-Rao product, also known as column-
better performance for implementation with C-ordering of the wise Kronecker product, is defined as A B ∈ RI·J×R :
elements, which is the case with most Python/C++ software
libraries as well as most GPU libraries [44]. Table I illustrates
h i
A B = A:,1 ⊗ B:,1 , A:,2 ⊗ B:,2 , · · · , A:,R ⊗ B:,R
the link between the fibers and the unfoldings, as well as the
difference between our definition (row-wise, or C-ordering)
and the column-wise, or Fortran ordering [34]. Furthermore, the Khatri-Rao product of a set of M matrices
Definition II.2 (Tensor vectorization): Given a tensor, {X(m) ∈ RIm ×N }M m=1 is denoted by X
(1)
X(2) · · · X(M ) .
X ∈ RI1 ×I2 ×···×IN , we can flatten it into a vector vec(X ) Definition II.5 (Hadamard product): The Hadamard prod-
of size I1 · I2 · . . . · IN defined by the mapping from el- uct of A, B ∈ RI×J is the element-wise product is symbolized
ement (iP1 , i2 , · · · , iN ) of Q
X to element j of vec(X ), with as A ∗ B and is defined element-wise as A(i,j) B(i,j) .
N N
j = 1 + k=1 (ik − 1) × m=k+1 Im .
Definition II.6 (Outer product): ◦ denotes the vector outer
product. Given a set of N vectors {x(n) ∈ RIn }N n=1 their
C. Matrix and tensor products outer product is denoted as X = x(1) ◦ x(2) ◦ · · · ◦ x(N ) ∈
Here, we will define several matrix and tensor products RI1 ×I2 ×···×IN and defines a rank-one N th -order tensor.
employed by the surveyed methods.
Definition II.7 (n-mode product): For a tensor
Definition II.3 (Kronecker product): Given two matrices X ∈ RI1 ×I2 ×···×IN and a matrix M ∈ RR×In ,
A ∈ RI1 ×I2 and B ∈ RJ1 ×J2 , their Kronecker product is the n-mode product of a tensor is a tensor of size
defined as: (I1 × · · · × In−1 × R × In+1 × · · · × IN ) and can be
  expressed using unfolding of X and the classical dot product
a11 B · · · a1I2 B as:
 . .. 
 
..
A ⊗ B =  .. . .  ∈ RI1 ·J1 ×I2 ·J2
  X ×n M = MX[n] ∈ RI1 ×···×In−1 ×R×In+1 ×···×IN
aI1 1 B · · · aI1 I2 B

Definition II.4 (Khatri-Rao product): Given two matrices The n- mode vector product of a tensor X ∈ RI1 ×I2 ×...×IN
A ∈ RI×R and B ∈ RJ×R with the same number of with a vector x ∈ RIn is denoted by X ×n x ∈
PROCEEDINGS OF THE IEEE 5

RI1 ×I2 ×···×In−1 ×In+1 ×···×IN . The resulting tensor is of order

J
N − 1 and is defined element-wise as
In v M
X
(X ×n x)i1 ,...,in−1 ,in+1 ,...,iN = xi1 ,i2 ,...,iN xin . (1)
in =1 I I

In order to simplify the notation,

QN we denote X ×1 x(1) ×2 (a) vector (b) matrix
(2) (N )
x ×3 · · · ×N x = X n=1 ×n x(n) . K K

𝒳̂ 𝒳̂
Definition II.8 (Generalized inner-product): J L J
For two tensors X , Y ∈ RI1 ×I2 ×···×IN of
same size, Ptheir Pinner P product is defined as
hX , Yi =
I1 I2 IN I I
i1 =1 i2 =1 · · · in =1 Xi1 ,i2 ,··· ,in Yi1 ,i2 ,··· ,in
For two tensors X ∈ RDx ×I1 ×I2 ×···×IN and
I1 ×I2 ×···×IN ×Dy (c) 3rd order tensor (d) 4th order tensor
Y ∈ R sharing N modes of same size,
we similarly define the generalized inner product along the Fig. 2: Representation of tensors of various orders using tensor
N last (respectively first) modes of X (respectively Y) as diagrams. The vertices represent tensors and their degree the
order of the tensors.
I1 X
X I1 IN
X
hX , YiN = ··· X:,i2 ,i3 ,··· ,in Yi1 ,i2 ,··· ,in−1 ,:
i1 =1 i2 =1 in =1 I

with hX , YiN ∈ R Ix ×Iy

. I A v
Definition II.9 (Convolution): We denote a regular con- C = J
I
volution of X with W as X ? W. For 1–D convolutions, K B
we write the convolution of a tensor X ∈ RI1 ×I2 ×···×IN
K
v
with a vector v ∈ RK along the nth –mode as X ?n v. In
practice, as done in current deep learning frameworks [45], (a) matrix product (b) inner-product (vectors)
we use cross-correlation, which differs from a convolution I
by a flip of the kernel. This does not impact the results K
since the weights arePKlearned end-to-end. In other words, M
𝒳̂ 𝒳̂
J
(X ?n v)i1 ,··· ,iN = k=1 vk Xi1 ,··· ,in−1 ,in +k,in+1 ,··· ,IN .
I I

D. Tensor diagrams (c) matrix trace (d) inner-product (tensors)

Fig. 3: Representation of some tensor contractions using tensor
Most tensor methods involve a series of tensor operators,
diagrams
such as tensor contractions2 , resulting in several sums over
various modes and tensors, with as many indices, which can
be cumbersome to read and write.
Tensor diagrams are pictorial representations of tensor al- E. Matrix rank
gebraic operations, in the form of undirected graphs offering
The rank of a matrix can be defined in several equivalent
a more intuitive way to read and write tensor operators and
ways. Hereby we give a simple definition:
design tensor methods. The vertices (circles) represent tensors,
and its edge the modes of that tensor. The degree of each vertex Definition II.10 (Matrix rank): Let X ∈ RI×J an I × J
(the number of edges coming out of it) thus represents the matrix of real numbers. The rank of X is denoted as rank(X).
order of the corresponding tensor. Consequently, using tensor It is equivalently defined as:
diagrams a scalar (Fig. 2a) would simply be represented by
a vertex while a vector (i.e., a tensor with one index) would • The number of linearly independent columns of X
have one edge. A matrix (Fig. 2b) would be a vertex with two • The number of linearly independent rows of X
edges, and so on for third- (Fig. 2c) and fourth-order (Fig. 2d) The above definition immediately implies that if X ∈ RI×J ,
tensors. then rank(X) ≤ min(I, J). If rank(X) = min(I, J) then X
By employing tensor diagrams, tensor contraction over a is full-rank.
common dimension of two tensors is represented by connect-
ing the two corresponding edges. For instance, given matrices
A ∈ RI×J , B P∈J R
J×K
, we can express simply their product
>
C = AB = j=1 a:,j bj,: , as illustrated in Fig. 3a. F. Norms

2 The n-mode product of a tensor with a matrix is a tensor contraction One of the major families of tensor norms, of any order, is
between the n-th mode of the tensor and the second mode of the matrix. element-wise norms.
PROCEEDINGS OF THE IEEE 6

Definition II.11 (`p norm): The element-wise `p of X is: personal computers. However, with the advent of deep learning
 1/p and large bodies of annotated images, such as ImageNet [14],
X it became crucial to have large amounts of memory. Most
kX kp = kvec(X )kp =  |Xi1 ,i2 ,··· ,iN |p  importantly, to be able to process the information efficiently,
i1 ,i2 ,··· ,iN it became paramount to be able to process multiple bits
For p = 0, the `0 (pseudo)-norm, denoted kX k0 returns the of information in parallel. However, central processing units
number of non-zero elements of a tensor, acting as a measure (CPUs) are inherently sequential. First proposed in [48], then
of the density (and conversely of the sparsity) of the tensor. successfully used on MNIST [49] and more prominently, on
However, it is not exactly a norm in the mathematical sense (it ImageNet [14], it is the use of GPUs that revolutionized
is positive definite and satisfies the triangle inequality but is deep learning, making it feasible to train a very large model
not absolutely homogeneous), hence the name pseudo-norm. on millions of samples. While CPUs perform operations on
When p = 1 the `1 -norm is denoted by kX k1 and is defined tensors in a mostly sequential way and prioritize low latency,
as the sum of the absolute values of the tensor’s elements. The GPUs favor high throughput and accelerate operations by
`1 -norm is the tightest convex surrogate of `0 -norm [46] and is running them efficiently in parallel: modern CPUs typically
used as a convenient measure of sparsity in practice. For p = contain up to 16 cores, while a GPU has thousands of them,
2, the Frobenius norm is denoted as kXkF for matrices and allowing for the performance of hundreds of TFLOPS.
it is generalized to higher-order tensors by the tensor norm, Efficient libraries then enable to efficient use of the tensor
denoted kX k. cores composing GPUs. In particular, extending Basic Linear
The family of Schatten-p norms are defined for matrices and Algebra Subprograms (BLAS) primitives allows to signifi-
act as functions of the singular values (i.e., spectrum) of the cantly speed up computation by leveraging hardware accel-
matrix. Therefore, Schatten-p norms can be used to control eration [50]. The recent cuTensor [51] is a high-performance
the spectral properties of matrices. CUDA library that leverages GPU-acceleration for efficient
tensor operations. In particular, it obviates the need for costly
Definition II.12 (Schatten-p norm): Let reordering of the elements and reshaping when performing
[σ1 , σ2 , . . . , σmin(I,J) ]> be a vector of the singular values of operations such as tensor contraction for which it provides
X ∈ RI×J . A Schatten-p norm is obtained by taking the p native implementations.
norm of the singular values: Further, large models typically need to be trained on

min(I,J)
1/p multiple GPUs distributed in various machines. However,
kXkp = 
X p
σi  distributed training and inference across multiple machines
i
introduces new challenges and can be incredibly complex
to implement correctly. Fortunately, modern deep learning
For p = 1, the Schatten-1 norm is referred to as the nuclear libraries abstract such technicalities and allow the end-user to
norm and denoted by kXk∗ . It is defined as the sum of the focus on the model’s logic. Notable such frameworks include
singular values of X and is the tightest convex envelope of PyTorch [45], which offers a functional approach to deep
the rank function [47]. learning, and TensorFlow [52], which uses a symbolic one.
Others such as MXNet [53] enable a hybrid approach.
III. C OMPUTATIONAL I NFRASTRUCTURE AND T OOLS While CPU and GPU are the most widely used mediums
In this section, we discuss the necessary infrastructure for for training and inference in computer vision and deep learn-
computer vision and deep learning. Also, to accelerate the ing tasks, other hardware types can be used towards this
learning curve and enable researchers and practitioners to end. Examples include application-specific integrated circuits
get started in the field of tensor methods quickly, we review (ASICs) designed specifically for running a specialized set of
available software packages for tensor algebra and introduce instructions (e.g., TPUs [54]), or field-programmable gate ar-
the TensorLy library [44], with which we implement ready- rays (FPGAs) that conversely can be dynamically programmed
to-use examples that accompany the paper. depending on the application.

A. Hardware B. Software
Enabling deep learning, computer vision, and machine In the previous section, we covered hardware support,
learning start in general down at the hardware level. Indeed, however, there is a considerable interplay between hardware
modern computer vision is enabled by the “trinity of AI”: and software, and one cannot function well without the other.
data, with the introduction of large bodies of annotated data; Good software without optimized hardware support would
algorithms, with the introduction of convolutional neural result in slow training, while sub-par software with poor API,
networks, recurrent units, etc. and finally hardware, with documentation, and tests is unusable in practice. An example
the use of graphical processing units (GPUs). The latter was of this interplay is the organization of elements in memory.
crucial in scaling up computation and enabling training on Memory can be over-simplified as one long vector of numbers.
big (visual) data consisting of millions of images. The way these elements are ordered in memory influences how
Historically, datasets used in computer vision were relatively fast operations are executed. For instance, to store a matrix,
small and could easily fit in the memory of most commodity its elements can be arranged row-after-row (C-ordering) or
PROCEEDINGS OF THE IEEE 7

column-after-column (also called Fortran ordering). In NumPy, position. In C++, ITensor [62] provides efficient support for
elements are organized by default in row-order–the same tensor network calculations.
for PyTorch. This, in turn, influences higher-level operations: Finally, efficient operations on sparse tensors require spe-
unfolding, for instance, needs to be adapted to appropriate cialized implementations. Indeed, while libraries such as
ordering to avoid expensive reordering of the data. As a NumPy are highly-optimized for matrix and tensor contrac-
result, much effort has been spent developing user-friendly tions, they do not support operations on sparse tensors. Ex-
libraries that provide state-of-the-art algorithms, neatly coded plicitly representing sparse tensors as dense arrays is very
and wrapped in an intuitive, well-tested API. One notable such inefficient in terms of memory use and prohibitive for very
library is TensorLy [44]. large tensors. Instead, these can be represented efficiently,
e.g., using the COOrdinate format which instead of storing
all the elements only stores the coordinates and values of
non-zero elements. As a result, implementing them efficiently
Tensor Learning Deep
in Tensorized can be tricky. While there is a large body of work focusing
Networks
CP, Tucker, TTrain, on sparse matrix support, support for sparse tensor algebra
Ridge Regression…
is more scant. Libraries such as the PyData Sparse4 for
Tensor Decomposition CPU and the Minkowski Engine [63] for GPU are being
& Regression
Folding, unfolding, developed and provide an efficient way to leverage large,
n-mode-product, …
sparse multi-dimensional data. TensorLy also supports sparse
Core Tensor Operations tensor operations on CPU through the PyData Sparse library.
Write Once,
Execute Anywhere
Flexible Backend System IV. R EPRESENTATION L EARNING WITH T ENSOR
D ECOMPOSITIONS
Fig. 4: Overview of the TensorLy and TensorLy-Torch libraries A. Matrix decomposition and representation learning
Real-world visual data exhibit complex variability due to
TensorLy is a Python library that aims at making tensor many factors related to visual objects’ appearance (e.g., illu-
learning simple and accessible. It provides a high-level API mination and pose changes, rigid and non-rigid deformations),
for tensor methods, including core tensor operations, tensor image structure, semantics, and noise. Hence, useful infor-
decomposition, regression, and deep tensorized architectures. mation is hidden in high-dimensional visual measurements.
It has a flexible backend that allows running operations seam- Representation learning methods seek to extract useful in-
lessly using NumPy, PyTorch, TensorFlow, etc. The project is formation from such high-dimensional data through simple,
also open-sourced under the BSD license, making it suitable low-dimensional representations (i.e., data are expressed as a
for both academic and commercial use. TensorLy is also linear combination of basic yet fundamental structures) that
optimized for Python and its ecosystem. For instance, the simultaneously exhibit a set of desirable properties such as
unfolding is redefined as introduced in Section II to match the invariance to nuisance variability while at the same time
underlying C-ordering of the elements in memory, resulting recovering factors of variation in a meaningful manner for
in better performance. An overview of the functionalities a specific task. In practice, low-rank and sparsity are popular
of TensorLy can be seen in Figure 4. By default, it uses simplicity measures used in representation learning models.
NumPy as its backend. However, this can be set to any of Matrix decomposition forms the mathematical backbone in
the deep learning frameworks, optimized for running large- learning parsimonious, low-dimensional representations from
scale methods, and in the companion, we also use the popular high-dimensional vector data samples, such as images flattened
PyTorch library. Once PyTorch has been installed, it can into vectors. Component analysis, e.g., [64] and sparse coding,
be easily used as a backend for TensorLy, allowing all the e.g., [65], [66] are prominent examples of representation
operations to run transparently on multiple machines and learning methods that extract simple data representations by
GPU. Finally, TensorLy-Torch3 [55] builds on top of TensorLy means of low-rank matrix decompositions and sparse, possibly
and PyTorch and implements out-of-the-box layers for tensor overcomplete representations, respectively. Given M vector
operations within deep neural networks. data samples {xm }M m=1 , where each sample xm is an I-
Other libraries exist for tensor operations in Python. In dimensional vector, that is xm ∈ RIm , such a dataset can
particular, t3f [56] is a TensorFlow library for working with the be represented by a data matrix X ∈ RI×J which contains
tensor-train decomposition on CPU and GPU. It also supports in its columns data vectors. Matrix decompositions seek to
Riemannian optimization operations. Tntorch [57] is a library factorize X as a product of two factor matrices U(1) ∈ RI×R
for tensor networks in PyTorch, while TensorNetwork [58] and U(2) ∈ RJ×R , namely
additionally supports JAX, TensorFlow and NumPy. Scikit-
>
Tensor [59] is a NumPy library for tensor learning, supporting X = U(1) U(2) . (2)
several tensor decompositions.
In MATLAB, packages such as the Tensor Toolbox [60] or Assuming X is of low-rank R < min{I, J}, Eq. (2)
TensorLab [61] provide extensive support for tensor decom- is referred to as low-rank matrix decomposition. Hence, X
3 [Link] 4 [Link]
PROCEEDINGS OF THE IEEE 8

can be expressed as a sum of R rank-one matrices: X =

> PR (1) (2) PR (1) (2) >
U(1) U(2) = r=1 ur ◦ ur = r=1 ur ur . The
concept of low-rank matrix decomposition is generalized for
higher-order tensors by low-rank tensor decomposition which
will be discussed next in Section IV-B. Interestingly, as op-
posed to matrix decomposition which is not unique5 low-rank Fig. 5: Illustration of a CP decomposition of third-order
tensor decomposition, such as the Canonical-Polyadic decom- tensor X into a sum of rank-1 tensors.
position, also known as Parallel Factor Analysis (PARAFAC)
[67], [68], [69] is unique under mild algebraic assump-
tions (see [36] and references therein). are well suited to classification, and restoration tasks [65],
In practice, data matrices representing visual data sets are [66]. In Section IV-E we discuss tensor-structured dictionary
not exactly low-rank. The observed matrix can deviate from learning that generalizes the concept of dictionary learning to
the low-rank structure for several reasons, including noise, higher-order tensors.
outliers, and non-linear structures underlying the data. In such In multivariate linear regression models, enforcing low-
cases, we are interested in approximating the observed matrix rank constraints on the coefficient matrix offers an effective
with a low-rank matrix expressed as a product of two factor reduction of unknown parameters, which facilitates reliable
matrices. Depending on the application, these elements can parameter estimation and model interpretation. Likewise,
be interpreted either as a basis of a low-dimensional subspace low-rank structures can be imposed in tensor regression
or as a mixing operator (i.e., mixing matrix) that combines models such as those presented in Section IV-F.
hidden factors of variation, giving rise to the observed data.
Such components comprise a low-dimensional, and hence a B. Tensor decomposition
simplified representation of the data in the subspace spanned
Tensor decomposition extends the concept of matrix de-
by the basis elements. To ensure uniqueness and make basis
composition to higher-order tensors. Here, we give a brief
elements and/or components physically or semantically in-
overview of the three most fundamental tensor decomposition
terpretable, constraints such as orthogonality, nonnegativity,
models that are most useful in the context of this paper. For
statistical independence, smoothness, or sparsity are usually
more details on these models, the interested reader is referred
imposed on components. For instance, in principal component
to [38], [34], [36], [37], where algorithmic matters are also
analysis (PCA) [70], [71] basis elements are imposed to
covered in detail. We introduce all three decompositions for
be column orthonormal. In independent component analysis
an N th order tensor X ∈ RI1 ×I2 ×···×IN .
(ICA) [72] latent components should be statistically indepen-
1) Canonical-Polyadic (CP) decomposition: The CP de-
dent, while in non-negative matrix factorization (NMF) [73]
composition, also referred to as PARAFAC, [67], [68], [69],
they are imposed to be non-negative. Furthermore, additional
decomposes X into a sum of R rank-one tensors. The rank
constraints can be used to reflect topological or class-specific
of a tensor X is the minimum number of rank-one tensors
properties of the data [74], as well as spectral [75] or temporal
that sum to X and generalizes the notion of matrix rank (cf.
patterns [76].
Definition II.10) to high-order tensors. However, computing
The vast majority of the component analysis models men-
the tensor rank is NP-hard [78], [79]. Hence, a predefined
tioned above employ the least-squares criterion [64] which
rank should be known in advance or estimated from data
assumes that data are contaminated by Gaussian noise of small
using Bayesian approaches, e.g., [80], [81] or deep neural
variance. However, such an assumption is often not valid for
networks [82]. When directly learning the decomposition end-
visual data, as we will see in Section IV-C, where robust to
to-end with stochastic gradient descent, the rank parameter is
non-Gaussian noise tensor decomposition is discussed. Exten-
validated along with other parameters.
sions of component analysis methods to high-order tensors are Formally, the CP decomposition seeks to find the vectors
presented in Section IV-D. (1) (2) (N )
ur , ur , · · · , ur , such that:
Apart from low-rank, sparsity is another widely employed
R
low-dimensional structure in representation learning. Sparse X
coding, in particular, aims at finding a sparse representation X = u(1) (2) (N )
r ◦ ur ◦ · · · ◦ ur . (3)
r=1
| {z }
of the input vector data in the form of a sparse linear rank-1 components (tensors)
combination of basic elements. These elements are called These vectors can be collected into N matrices, {U(n) ∈
atoms, and they compose a dictionary. Atoms in the dictionary RIn ×R }N
n=1 , with each matrix defined as:
(i.e., columns of U(1) ) are not required to be orthogonal, and h i
they may comprise an over-complete spanning set. Learning U(n) = u(n)
1 , u
(n)
2 , · · · , u
(n)
R
. (4)
the dictionary and the linear combination coefficients (i.e., the
>
sparse code represented in columns of U(2) ) is referred to as The CP decomposition is denoted more compactly as X =
dictionary learning [65], [66], [77]. For signals such as natural JU(1) , U(2) , · · · , U(N ) K [34]. Representing a tensor using its
images, it is now well established that sparse representations CP decomposition is sometimes referred to as Kruskal format.
A pictorial representation of the CP decomposition is shown
5 To verify this consider any invertible matrix Q ∈ RR×R . Then we have in Fig. 5.
> >
X = U(1) U(2) = U(1) QQ−1 U(2) .
PROCEEDINGS OF THE IEEE 9

Fig. 7: Illustration of a Tensor-Train decomposition of a

third-order tensor X into a series of third-order cores. The
boundary conditions dictate R1 = RN +1 = 1
Fig. 6: Illustration of a Tucker decomposition of a third-
order tensor X into a core tensor and three matrices.

2) Tucker decomposition: The Tucker decomposition

[83] decomposes, non-uniquely, X into a core tensor
G ∈ RR1 ×R2 ×···×R
N and a set of factor matrices
U(1) , U(2) , · · · , U(N ) , with U(n) ∈ RRn ×In , n =
1, 2, . . . , N which are multiplied with G along its modes as
follows:
X = G ×1 U(1) ×2 U(2) ×3 · · · ×N U(N ) . (5)
The core tensor captures interactions between the columns of
factor matrices. If Rn  In , ∀n, then the core tensor can Fig. 8: Representation of a CP decomposition using tensor
be viewed as a compressed version of X . It is also worth diagrams.
noting that the CP decomposition can be expressed as Tucker
decomposition, with Rn = R, ∀n ∈ {1, 2, . . . , N } and the
core tensor being superdiagonal. The Tucker decomposition is
denoted compactly as JG; U(1) , U(2) , · · · , U(N ) K [34], and it conditions can be replaced by periodic boundary conditions
is shown pictorially in Fig. 6. By imposing the factor matrices (the Tensor-Ring) [88], [89] which instead contracts together
to be orthonormal, the Tucker model is known as higher-order the first and last cores along R1 and RN +1 .
singular value decomposition (HOSVD) [84]. Let us now depict the aforementioned tensor decompositions
In practice, data are corrupted by noise, and as in the matrix using tensor diagrams. While representing tensor decompo-
case, both the Tucker and CP decompositions are not exact. sitions visually (e.g., Figures 5 and 6), we are inherently
Therefore, they are approximated by optimizing a suitable limited to a maximum of third-order tensors, tensor diagrams
criterion representing fitting loss. Typically, the least-squares conveniently represent decompositions of higher-order tensors.
criterion is employed. In this case, assuming that {Rn }N n=1
For instance, let X ∈ RI1 ×I2 ×I3 ×I4 ×I5 be a 5th -order tensor,
are known, the Tucker decomposition is computed by solving its CP decomposition is shown in Fig. 8, Tucker decomposition
the following non-convex minimization problem: is shown in Fig. 9, and Fig. 10 depicts its TT decomposition.

min kX − JG; U(1) , U(2) , · · · , U(N ) Kk2 . (6)

G,{U(n) }N
n=1

The most common optimization algorithm for the Tucker

and CP decomposition is the Alternating Least Square
(ALS) [85], where the unknowns are estimated iteratively.
More specifically, the ALS fixes, sequentially, all but one
factor, which is then updated by solving a linear least-squares
problem. However, the ALS algorithm is not guaranteed to
convergence towards a minimum; see [35] for example.
3) Tensor-Train: The Tensor-Train (TT) decomposi-
tion [86], also known as the Matrix-Product-State [87] in
Fig. 9: Representation of a Tucker decomposition using tensor
quantum physics, expresses X as a product of third order
diagrams.
tensors G1 ∈ RR1 ×I1 ×R2 , · · · , GN ∈ RRN ×IN ×RN +1 (the
cores of the decomposition), such that:
X (i1 , i2 , · · · , iN ) = G1 [i1 ] × G2 [i2 ] × · · · × GN [iN ] .
| {z } | {z } | {z }
R1 ×R2 R2 ×R3 RN ×RN +1

The boundary conditions of the tensor-train (open boundary

conditions) decomposition dictate R1 = RN +1 = 1. The Fig. 10: Representation of a Tensor-Train decomposition using
decomposition is illustrated in Fig. 7. The open boundary tensor diagrams.
PROCEEDINGS OF THE IEEE 10

C. Robust tensor decomposition certain conditions (9) is guaranteed to exactly recover the low-
rank component [98]. Similar models has been introduced for
Notebook: [Link]
the problems of tensor completion [99], [100], [101], [98],
[92].
Visual data of interest, such as facial images captured in
As mentioned in Section IV-B, the CP and Tucker decompo-
unconstrained, real-world conditions, are often severely cor-
sitions are approximated in practice by minimizing the least-
rupted by nuisance factors of variation. Such factors include,
squares criterion, resulting in algorithms sensitive to sparse,
but are not limited to, significant pose variations, illumination
non-Gaussian noise. To alleviate this problem and obtain more
changes, and occlusions, as well as artifacts or missing data
robust approximations of tensor decompositions [102], [103],
introduced during the acquisition process. Collectively, these
[94] replace least-squares loss with the `1 -norm.
can be characterized as gross errors. From a statistical point
The vast majority of the aforementioned models are opti-
of view, gross errors and outliers do not follow a Gaussian
mized iteratively by employing alternating direction method
distribution, as is assumed in the vast majority of learning
of multipliers (ADMM) type of algorithms [104].
models employed in computer vision–including the tensor
decompositions discussed above. Indeed, learning models that
rely on optimization of algebraic criteria induced by second- D. Tensor component analysis
order statistics of observations (e.g., covariance matrix), or
Notebooks: [Link], [Link]
goodness of fit measures such as the least-squares criterion,
are unable to cope with gross errors and outliers [90]. As a
result, the estimated components can be arbitrarily far from Tensor component analysis (also referred to as multilinear
the true ones. Hence, their applicability in analyzing visual subspace learning [3]) extends the concept of component
data captured in unconstrained conditions is rather limited. analysis to data represented by higher-order tensors (please
Robust tensor decomposition methods aim to recover a low- refer to [3], [8] for an overview on the topic).
rank tensor from noisy measurements, assuming the presence Let us assume a set of M tensor data samples, that is
of non-Gaussian noise of possibly large magnitude. Those {Xm }M m=1 , where each data sample is a N − 1-order tensor

outliers only affect a small fraction of the measurements, and Xm ∈ RI1 ×I2 ×···×IN −1 . Therefore, as mentioned previously,
consequently, noise is considered sparse. Concretely, robust a set of such data is represented by an N th order ten-
tensor modeling seeks a decomposition of the form: sor X ∈ RI1 ×I2 ×···×IN . Tensor component analysis meth-
ods seek to estimate a set of N − 1 projection matrices,
X = L + S, (7) {U(n) ∈ RRn ×In }N −1
n=1 with Rn  In ∀n ∈ {1, 2, . . . , N − 1}
such that the projection of tensor data samples onto the
where X is an N th -order data tensor usually composed by a low-dimensional multilinear subspace, Q namely {Xm ×1
arranging a set of tensorial data samples of N − 1th order N −1
U(1) ×2 U(2) ×3 · · · ×N −1 U(N −1) = Xm n=1 ×n U(n) ∈
along the N th mode of the tensor. L and S are unknown RR1 ×R2 ×···×RN −1 }Mm=1 , yields a low-dimensional tensor rep-
N th -order tensors accounting for the low-rank tensor and resentation that satisfy some optimality criterion, such as
sparse noise, respectively. minimizing reconstruction error e.g., [9] or maximizing tensor
However, as mentioned previously optimizing tensor rank is scatter [105]. Structural constraints, such as orthogonality in
intractable [78] in general. To address this [91], [92], [93], [94] Multilinear PCA (MPCA) [105] or non-negativity in Non-
rely on approximating tensor Prank by a convex combination Negative Tensor Factorization (NTF) [9] and its variants [106]
N
of the n-ranks of L. That is, n=1 αn rankn (L), where αn ≥ are also imposed often on multilinear projections.
PN
0, n=1 αn = 1 and rankn (L) denotes the column rank of Different tensor component analysis methods can be
the mode-n matricization of L. Therefore, the low-rank tensor derived as different solutions to constrained optimization
is recovered by solving: problems. For example, unsupervised tensor component
N
X analysis can be formulated as a constrained low-rank tensor
min αn rankn (L) + λkSk0 s.t. X = L + S, (8) approximation problem:
L,S
n=1 > > >
min kX − G ×1 U(1) ×2 U(2) ×3 · · · ×N U(N ) k2
where λ is a positive parameter. However, both the rankn {U(n) }N
n=1
function and the `0 -norm are discrete functions and hence hard N
X  
to optimize [95], [96]. To make (8) tractable, discrete norms + λn rn U(n) s.t. {U(n) ∈ Cn }N
n=1 ,
are replaced by their convex surrogates [47], [46]. That is, the n=1
rank function is replaced by the nuclear norm and the `0 -norm (10)
by the `1 -norm, resulting into a convex optimization problem:
where rn (·) is a regularization function (e.g., `1 -norm for
[97], [94]:
inducing sparsity), λn is a positive regularization parameter,
N
X and Cn denotes a set of structural constraints imposed onto
min αn kL(n) k∗ + λkSk1 s.t. X = L + S, (9) the low-dimensional factors. For instance, MPCA [105] is
L,S
n=1 derived by (10) by setting G = X ×1 U(1) ×2 U(2) ×3
which is an extension of principal components pursuit (PCP) · · · ×N −1 U(N −1) ×N I, requiring U(N −1) = I, and enforcing
[75] to higher-order tensors. It is also worth noting that under all the N − 1 projection matrices to be column orthonormal.
PROCEEDINGS OF THE IEEE 11

>
That is, {Cn = U(n) U(n) = I}N −1
n=1 . Furthermore, by where the regularizers g(·) and r(·) promote sparsity in the
requiring factor matrices to be orthogonal and nonnegative representations, and low mutual-coherence of the dictionary
(10) yields the non-negative MPCA [107]. By neglecting D = U(2) ⊗ U(1) , respectively. Here, D is constrained to
orthogonality constraints and requiring factor matrices to be have orthogonal columns.
only nonnegative, i.e., {Cn = U(n) ≥ 0}N −1
n=1 the NTF [9] is A different approach is taken in [111], where a separable
obtained by (10) when G is a superdiagonal tensor, i.e., G = I. 2D dictionary is learned following a two-step strategy similar
Supervised tensor component analysis exploits available to that of the K-SVD. Each observation matrix Xm is ap-
>
side information such as class labels or neighborhood relation- proximated by U(1) Rm U(2) , where Rm are slices of a core
ships among samples to estimate a multilinear projection. Such tensor R. In the first step, the sparse representations Rm are
models can be unified under the graph embedding framework obtained by 2D Orthogonal Matching Pursuit (OMP) [112]. In
[74] by solving: the second step, a CP decomposition is performed on a tensor
M
X N
Y −1 N
Y −1 of residuals.
min Wm,k kXm ×n U(n) − Xk ×n U(n) k2 , Theoretical analysis suggests that the sample complexity
−1
{U(n) }N
n=1 m,k
m6=k
n=1 n=1 of tensor-structured dictionary learning can be significantly
M N −1 lower than that for unstructured, vector data; see [113] for 2D
data, and [114] for N -order tensor data. This suggests better
X Y
s.t Bm,k kXm ×n U(n) k2 = 1,
m,k n=1 performance is achievable with separable dictionary learning
(11) from tensor data compared to vector-based dictionary learning
methods.
where W denotes the affinity matrix capturing some sort of
However, the aforementioned models learn dictionaries on
within-class neighborhood relationship among tensor samples
many small image patches whose number can easily become
Xm and B is an affinity matrix representing between-class
prohibitively large, undermining the scalability benefits of
neighborhood relationship. Multilinear extensions to Linear
learning separable dictionaries. Moreover, it is worth noting
Discriminant Analysis (LDA), such as [108], [109], [110] for
that none of the models mentioned above are robust to
example, are obtained by setting Wmk = δcm ,ck /ncm and
1 gross errors due to the loss function used. Such limitations
B = I− M ee> , where δ denotes the Kronecker delta function,
are alliviated by the Robust Kronecker Component Analysis
e is the standard basis and nc denotes the number of the
(RKCA) [115], [4].
samples belonging to the cth class.
The RKCA combines ideas from separable dictionary learn-
E. Tensor-structured dictionary learning ing and robust tensor decomposition. Instead of seeking over-
completeness, low-rank is promoted on the pair of dictionaries,
Sparse coding (or dictionary learning) methods are cen-
and sparsity is imposed in the code yielding a sparse represen-
tral in numerous visual information processing tasks. In this
tation of the input tensor samples. Specifically, RKCA solves:
context, a data matrix X ∈ RI×M √ contains
√ in its columns
vectorized image patches of size I × I, rather than data M
X >
samples. Assuming D is over-complete and requiring R = min kXm − U(1) Rm U(2) − Sm k2F
U(1) ,U(2) ,R,S
[r1 , r2 , . . . , rM ] to be sparse, sparse dictionary learning solves m=1
M M
the following, non-convex, optimization problem: X X
+λ kRm k1 + λ kSm k1 + kU(2) ⊗ U(1) kF , (14)
m=1 m=1
M
X
min kX − DRk2F + λ krm k0 , (12) where matrices {Sm }M m=1 account for sparse noise and out-
R,D
m=1 liers. Equivalently, enforcing the equality constraints and con-
However, extracting patches from images and flattening catenating the matrices Xm , Rm , and Sm as the frontal slices
them into vectors makes (12) suboptimal for multidimensional of a third-order tensor, (14) is written as:
tensor data. Besides structure loss, typical solvers for (12),
such as the K-SVD family of algorithms [65], suffer from a
min αkRk1 + λkSk1 + kU(2) ⊗ U(1) kF
high computational burden, preventing their applicability to U(1) ,U(2) ,R,S
massive amounts of tensor data. s.t X = R ×1 U(1) ×2 U(2) + S. (15)
To preserve multilinear structures in dictionary learning, a
separable structure on the dictionary can be enforced. For Consequently, (15) indicate that separable dictionary can be
instance, Separable Dictionary Learning (SeDiL) [11] con- expressed as a regularized tensor decomposition. As opposed
siders a set of samples in matrix form, namely, {Xm }M m=1 , to classic dictionary learning models, it is worth mentioning
admitting sparse representations on a pair of bases U(1) , U(2) , that following [116], [117], the global optimality of RKCA is
of which the Kronecker product constructs the dictionary. ensured.
More specifically, SeDiL solves:
M F. Tensor regression
1 X >
min kXm − U(1) Rm U(2) k2F Besides learning representations, low-rank tensor decompo-
U ,U ,R 2
(1) (2)
m=1
sition facilitates reducing the parameters of tensor regression
+ λ1 g(R) + λ2 r(U(1) ) + λ3 r(U(2) ), (13) models and prevents overfitting. Tensor regression models
PROCEEDINGS OF THE IEEE 12

generalize linear regression to higher-order tensors. More A. Parameterizing fully-connected layers

specifically, in tensor regression a regression output y ∈ R Notebook: [Link]
is expressed as the inner product between the observation
tensor X and a weight tensor W with the same dimension Leveraging tensor decompositions within neural networks
as X : y = hX , Wi + b. Several low-rank tensor regression by re-parametrizing fully-connected layers has been first pro-
models have been proposed [5], [118], [119], [120], [121], posed in [21]. Since the weights of fully-connected layers
[122]. These methods share the assumption that the unknown are represented as matrices, tensor decompositions cannot be
regression weight tensor is low-rank. This is enforced by applied directly. Therefore, the weight matrix needs to be
expressing the weights tensor in CP [5], [119] or Tucker reshaped to obtain a higher-order tensor.
form [123], [121], [124]. Alternatively, low-rank constraints Specifically, consider an input matrix W of size I × J,
on the weight tensor are enforced by spectral regularization, whose dimensions can be expressed as I = I1 × I2 × · · · × IN
such as applying the nuclear norm on the unfoldings of the and J = J1 × J2 × · · · × JN . Hence, W is tensorized first by
weight tensor along its modes [123], in a similar way to reshaping it to a higher-order tensor of size I1 × I2 × · · · ×
robust tensor decomposition described in Section IV-C. To IN × J1 × J2 × · · · × JN . Next, by permuting the dimensions
further prevent overfitting, regularization is incorporated into and reshaping it again, an N th order tensor of I1 J1 × I2 J2 ×
low-rank tensor regression models. For instance, [120], [5], · · · × IN JN is obtained. This tensor is then compressed in the
[118] employ tensor norm and hence extend the low-rank TT format. The approximated weights are reconstructed from
ridge regression to high-order tensors. `1 -norm and elastic-net that low-rank TT factorization during inference, and are then
regularization is adopted in [119], while a combination of `1 - reshaped back into a matrix.
norm and nuclear norm regularizes the weight tensor in [122] In this setting, other types of decompositions have also
forcing it to be simultaneously low-rank and sparse. It is worth been explored, including Tensor-Ring [131] and Block-Term
mentioning that low-rank regression models extend support Decomposition [132]. This strategy has also been extended to
vector machines to higher-order tensors, e.g., [5], [122]. parametrize other types of layers [133].
Furthermore, higher-order generalizations of Partial-Least In the context of two-layer fully-connected neural net-
Squares were proposed [125], [126]. The idea of Higher Order works, tensors have also been used for deriving alternative
Partial Least Squares (HO-PSL) is to find a latent correlated to (stochastic) gradient descent methods for training, with
subspace between the observation tensor and the response guaranteed generalization bounds. Specifically, in [134], the
tensors. Alternatively, it is possible to perform regression CP decomposition is applied on the cross-moment tensor,
in a subspace obtained with Tensor PCA [127]. Finally, which encodes the correlation between the third-order score
quantile regression has also recently been generalized to function (i.e., the normalized derivative of the input pdf) to
higher-order [128], using a Tucker structure on the regression obtain the parameters of the first layer. The parameters of
weights. the second layer are obtained next via regression. Under mild
assumptions, this approach comes with guaranteed risk bounds
V. T ENSOR M ETHODS IN D EEP N EURAL N ETWORKS in both the realizable setting (i.e., the target function can be
A RCHITECTURES approximated with zero error from the neural network under
consideration) and the non-realizable one, where the target
Tensors play a central role in modern deep learning architec- function is arbitrary.
tures. The basic building blocks of deep neural networks, such
as multichannel convolutional kernels and attention blocks, B. Tensor contraction layers
are essentially tensor mappings, represented by tensors. Deep In modern deep neural networks such as CNNs, each layer’s
neural networks, as mentioned earlier, operate in a regime output is an (activation) tensor. However, activation tensors
where the number of parameters is typically tens of millions are usually flattened and passed to subsequent fully-connected
or even billions, and is thus much larger than the number of layers resulting in structural information loss. A natural way
training data. In this over-parameterized regime, incorporating to preserve the multilinear structure of activation tensors is to
tensor decompositions into deep learning architectures leads to incorporate tensor operations into deep networks. For instance,
a significant reduction in the number of unknown parameters. tensor contraction can be applied to an activation tensor to
This allows the design of networks that further preserve the obtain a low-dimensional representation of it [24]. In other
topological structure in the data, being more parsimonious words, an input activation tensor is contracted along each
in terms of parameters, more data-efficient, and more robust mode with a projection matrix (an n-mode product as defined
to various types of noise and domain shifts. Besides the earlier).
design of deep networks, tensor analysis can be beneficial Let us assume a network involves an input activation tensor
in demystifying the success behind neural networks, having X of size S × I1 × · · · × IN , where S is the batch size. A
already been used to prove universal approximation properties Tensor Contraction Layer (TCL) contracts the input activation
of neural networks [129], or to understand the inductive bias tensor, along each of its dimensions except the batch-size, with
leveraged by networks in computer vision [130]. This section a series of factor matrices {V(n) }Nn=2 .The result is a compact
overviews recent developments in the use of tensor methods core tensor X 0 of smaller size S × R1 × · · · × RN defined as
to design efficient and robust deep neural networks and the
theoretical understanding of their properties. X 0 = X ×2 V(2) ×3 V(3) × · · · ×N V(N ) , (16)
PROCEEDINGS OF THE IEEE 13

by assuming a Tucker structure on the regression weights. That

is, for a rank–(R1 , · · · , RN ) Tucker decomposition, we have:
y (s) = hX (s) , Wi + b
with W = G ×1 U(1) ×2 U(2) · · · ×N U(N ) (17)
with G ∈ RR1 ×···×RN , U(n) ∈ RIn ×Rn for each n in [1 . . N ]
Fig. 11: An input tensor X is contracted into a smaller tensor and U(N ) ∈ R1×RN . For simplicity, here, we introduce the
X 0 through a Tensor Contraction Layer [24]. problem for a scalar output, but it is established, in general,
for a tensor-valued response. The inner product between the
activation tensor and the regression weight is then replaced
with V(n) ∈ RRn ×In , n ∈ [2 . . N ]. Note that the projections by a tensor contraction along matching modes.
start at the second mode as we do not contract along the first In addition to the implicit regularization induced by the low-
mode S, which corresponds to the mini-batch size. These rank structure of the regression weights, TRLs result in a much
factors are learned in an end-to-end manner, concurrently with smaller number of parameters. For an input activation tensor
all the other network parameters by gradient backpropagation. X ∈ RS×I1 ×I2 ×···×IN , a rank-(R1 , R2 , · · · , RN , RN +1 ) TRL
A TCL that produces a compact core of smaller size is and an d-dimensional output. A fully connected layer taking
denoted as size–(R2 , · · · , RN ) TCL, or TCL–(R2 , · · · , RN ). the same input X (after a flattening QN layer) would have nFC
parameters, with nFC = d × n=1 In . By comparison, a
In addition to preserving the multi-linear structure in the rank-(R1 , R2 , · · · , RN , RN +1 ) TRL taking X as input has a
input, TCLs have much less parameters than a corresponding number of parameters nTRL , with:
fully-connected layer. We can see this by establishing the link
N +1 N
between a fully-connected layer with structured weights and Y X
a tensor regression layer. Let us assume an input activation nTRL = Rn + Rn × In + RN +1 × d.
n=1 n=1
tensor X of size S1 × I1 × · · · × IN . A size–(R2 , · · · , RN )
TCL parameterized by weight factors V(2) , · · · , V(N ) has a As previously observed for the TCL, TRL is much more
PN
parsimonious in terms of the number of parameters. Note that,
total of n=2 In × Rn parameters. By unfolding the input
tensor (i.e., vectorizing each sample of the batch), TCL can be while we introduced the TRL here with a Tucker structure on
equivalently expressed as a fully-connected layer parametrized the regression weight tensor, in general, any low-rank format

> can be used, including CP and TT [135].
by the weight matrix W = V(2) ⊗ · · · ⊗ V(N ) , which
computes
 > D. Parametrizing convolutional layers
X[1] W = X[1] V(2) ⊗ · · · ⊗ V(N ) . The weights of convolutional layers are naturally repre-
However, since fully-connected layers have no structure sented as tensors. For instance, a 2D convolution is represented
on their weights, the corresponding fully-connected layer by a 4th –order tensor. Hence convolutional layers can directly
be compressed by employing tensor decomposition. In fact,
QN matrix W of the same size as above
parametrized by a weight
there is a close link between deep neural networks and efficient
would have a total of n=2 In × Rn parameters, compared to
PN convolutional blocks, which we will highlight in this section.
the n=2 In × Rn parameters of the TCL.
1) 1 × 1 convolutions: The first thing to notice is that 1 × 1
convolutions are tensor contractions. This type of convolution
C. Tensor regression layers is frequently used within deep neural networks in order to
Notebook: tensor regression [Link] create a data bottleneck. For a 1 × 1 convolution Φ, defined
by kernel W ∈ RT ×C×1×1 and applied to an activation tensor
In CNNs, after an activation tensor has been obtained X ∈ RC×H×W . We denote the squeezed version of W along
through a series of convolutional layers, predictions (outputs) the first mode as W ∈ RT ×C . We then have:
are typically generated by flattening this activation tensor or C
X
applying a spatial pooling, before using a fully-connected Φ(X )t,y,x = X ? W = Wt,k,y,x Xk,y,x = X ×1 W.
output layer. However, both approaches discard the multilinear k=1

structure of the activation tensor. To alleviate this, tensor Note that, here, we have x = y = 1 as we are considering a
regression models can be incorporated within deep neural net- 1 × 1 convolution.
works and trained end-to-end jointly with the other parameters 2) Kruskal convolutions: Let us now focus on how tensor
of the network [25]. decomposition can be applied to the convolutional kernels of
Specifically, let us assume we have a set of S input CNNs and how this results not only in fewer parameters but
activation tensors X (s) ∈ RI1 ×I2 ×···×IN and corresponding also faster operations. The CP decomposition, for instance, al-
target labels y (s) ∈ R, with s ∈ [1 . . S]. A Tensor lows separating the modes of a convolutional kernel, resulting
Regression Layer (TRL) estimates the regression weight in a Kruskal form. Albeit the application of tensor decomposi-
tensor W ∈ RI1 ×I2 ···×IN , by further assuming that it admits a tion in the context of CNNs is relatively new and of practical
low-rank decomposition. Originally, TRL has been introduced interest nowadays, the idea of separable representation of
PROCEEDINGS OF THE IEEE 14

Fig. 12: Illustration of a convolutional deep network where the activation tensor is processed through a TCL followed by a
TRL to produce the outputs while preserving the topological structure [25]. Note that y, here, is a scalar.

linear operators by means of tensor decomposition is not new

(see [136]) and in fact was one of the original motivations for
tensor rank decomposition of the matrix-multiplication tensor
(see [137] for example). Fig. 13: Illustration of a 2D Kruskal convolution, as
In computer vision, the use of separable convolutions was expressed Eq. (19), with matching colours. First, a 1 × 1
proposed in [138], in the context of learnable filter banks. convolution reduces the number of input channels to the rank
In the context of deep learning, [139] proposed to leverage (blue). Two depthwise convolutions are then applied on the
redundancies across channels by exploiting separable convo- height and width of the activation tensor (red and green).
lutions. Furthermore, it is possible to start from a pre-trained Finally, a second 1 × 1 convolution restores the number of
convolutional kernel and apply CP decomposition to it in channels from the rank of the CP decomposition to the desired
order to obtain a separable convolution. This was proposed number of output channels (black).
for 2D convolutions in [22], where a CP decomposition of the
kernel was achieved by minimizing the reconstruction error
between the pre-trained weights and the corresponding CP 3) Tucker convolutions: As previously, we consider the
approximation. The authors demonstrated both space savings convolution F = X ? W described in (18). However, instead
and computational speedups. Similar results can be obtained of a Kruskal structure, the convolution kernel W is assumed
using different optimization strategies such as the tensor power to admit a Tucker decomposition as follows:
method [23]. R1 X
R2 X
R3 X
R4
(T ) (H) (W )
X
An advantage of factorizing the kernel using a CP decompo- W(t, s, j, i) = Gr1 ,r2 ,r3 ,r4 Ut,r1 U(C)
s,r2 Uj,r3 Ui,r4 .
sition is that the resulting factors can be used to parametrize a r1 =0 r2 =0 r3 =0 r4 =0

series of efficient depthwise separable convolutions, replacing This approach allows for an efficient reformulation [140]:
the original convolution [22], as illustrated in Fig. 13. This can first, the factors along the spatial dimensions are absorbed
be seen by considering the expression of a regular convolution: (H) (W )
into the core by writing H = G ×3 Uj,r3 ×4 Ui,r4 . By
reordering the term, we can now see that a Tucker convolution
C X W
H X
X is equivalent to first transforming the number of channels, then
Ft,y,x = W(t, k, j, i)X (k, j + y, i + x). (18)
applying a (small) convolution before restoring the channel
k=1 j=1 i=1
dimension from the rank to the target number of channels:
 
A Kruskal convolution is obtained by expanding the ker-  
nel W in the CP form as introduced in Section IV-B1.

 
R1
 R2 
H X W X
 C
X
By reordering the terms, we can then obtain an efficient X X
(T )  (C) 
Ft,y,x = Ut,r1  Hr1 ,r2 ,j,i  Uk,r2 X (k, j + y, i + x) .


reparametrization [22]: r1 =1

 j=1 i=1 r2 =1

k=1 
 | {z } 

   1×1 conv 
| {z }
  H×W conv
 | {z }
1×1 conv
 
R
 W H
" C
#
(20)
X 
X (T )  (W ) 
X (H)
X (C) 
Ft,y,x = Ut,r  Ui,r  Uj,r Uk,r X (k, j + y, i + x) 

 
Besides the aforementioned types of convolutions (i.e.,
 
r=1  i=1  j=1 k=1 
 | {z }  

| {z
1×1 conv
}
 bottleneck layers, separable convolutions), [141] identify max-
|
depthwise conv
{z } imally resource-efficient convolutional layers utilizing tensor
|
depthwise conv
{z } networks and decompositions. Tensor decompositions have
1×1 convolution also been used for unifying different successful architectures.
(19) Particularly, in [29], the block term decomposition [142] is
used to unify different architectures that employ residual
PROCEEDINGS OF THE IEEE 15

where ρ applies the 1D spatial convolutions:

 
ρ(X ) = X ?1 U(K1 ) ?2 U(K2 ) ? · · · ?N U(KN ) . (22)
A factorized separable convolution can be extended from N -
Fig. 14: Illustration of a Tucker convolution expressed dimensions to N + K (N, K ∈ N) by means of transduction
as a series of small, efficient convolutions, as illustrated in [27]. Since the convolution is fully separable, it is possible
Eq. (20). Note that this is the approach taken by ResNet for the to add additional factors. In particular, to apply transduction
Bottleneck blocks. After the full kernel has been decomposed, from N to N + 1, it is sufficient to add an additional factor
the factors along the input and output channels are used to UKN +1 and update ρ in equation (22) to:
parametrize 1×1 convolutions, applied respectively first (blue)  
and last (green). The remaining two factors are absorbed into ρ(X ) = X ?1 U(K1 ) ?2 U(K2 ) ? · · · ?N U(KN ) ?N +1 U(KN +1 ) .
the core and used to parametrize a regular 2D convolution
6) Parametrizing full networks: So far, we reviewed meth-
(red).
ods focusing on parametrizing a single layer. It is also possible
to jointly parametrize a whole convolutional neural network
connections, such as ResNet [15], while proposing a new with a single low-rank tensor. This is the idea of “T-Net” [26],
architecture that improves parameter efficiency. This implies where all the convolutional layers of a stacked-hourglass [146]
that searching over the space of tensor decompositions and (a type of U-Net [147]) are parametrized by a single 8th –order
their parameters might be an excellent proxy task to limit the tensor. Each of the modes of this tensor models a modality
search space for architecture search. of the network, allowing to jointly model and regularize
From a theoretical point of view, by using the CP decom- all the layers. Hence correlations across channels, layers,
position to replace convolutional with low-rank layers results convolutional blocks, and subnetworks of the model can be
in deriving the upper bound for the generalization error of leveraged. Imposing low-rank constraints on this model tensor
the compressed network [143]. This bound improves previous results in large parameter space savings with little to no impact
results on the compressibility and generalizability of neural on performance, or even improved performance, depending
networks [144]. It is also related to the layers’ rank, providing on the compression ratio. That is, small compression ratios
insights on designing neural networks (with low-rank tensors result in improved performance, while it is possible to reach
as layers). substantial compression ratios with only minor decreases in
4) Multi-stage compression: The factorization strategies performance. In addition, it is possible to partially reconstruct
mentioned above allow for efficient compression of pre- the full tensor to obtain a Tucker factorization of the kernel
trained fully-connected and convolutional layers of deep neural of each layer, thus allowing for speedups as described in
networks by applying tensor decomposition to the learned section V-D3.
weights. Fine-tuning allows recovery for loss of performance 7) Preventing degeneracies in factorized convolutions: As
when increasing the compression ratio. However, it is also we discussed above, when parametrizing DCNNs, one can
possible to apply multi-state compression [145] to further either train the factorized layers from scratch in an end-to-end
compress without loss of performance. Instead of compressing fashion via stochastic gradient descent, or decompose existing
then fine-tuning, the idea is to adopt an iterative approach by convolutional kernels to obtain the factorized version. In the
alternating low-rank factorization with rank selection and fine- latter case, the authors of [148] showed that when applying
tuning. decomposition to obtain a factorized convolution, degeneracy
5) General N-D separable convolutions and transduction: can appear in the decomposition due to diverging components.
Design of general (N-dimensional) fully separable, convolu- They tackle this issue and enable stable factorization of
tion has been introduced in the context of deep neural networks convolutional layers through sensitivity minimization.
in [27]. Let us assume an (N +1)th –order input activation ten-
sor X ∈ RC×D1 ×···×DN corresponding to N dimensions with E. Robustness of deep networks
C channels. The higher-order separable convolution, which Deep neural networks produce remarkable results in com-
we denote Φ, is defined by a kernel W ∈ RT ×C×K1 ×···×KN . puter vision applications but are typically very vulnerable to
By expressing this kernel in a factorized Kruskal form as noise, such as capture noise, corruption, adversarial attacks
W = Jλ; U(T ) , U(C) , U(K1 ) , · · · , U(KN ) K, we obtain: of domain shift. Below, we provide a brief review of tensor
R X
X K1
C X methods that robustify deep networks.
Φ(X )t,i1 ,··· ,iN = ··· 1) Tensor Dropout: One known way to increase robust-
r=1 s=1 i1 =1 ness is to incorporate randomization during training. For
KN
X  (T ) (K1 ) (KN ) example, Dropout [149] randomly drops activations, while
λr Ut,r U(C)

··· s,r Ui1 ,r · · · UiN ,r Xs,i1 ,··· ,iN . DropConnect [150] applies the randomization to the weights of
iN =1
fully-connected layers. However, randomizing directly in the
The separable higher-order convolution can be obtained by original space results in sparse weights or activations (with
reordering the term into: zero-ed values), which affects the high-order statistics.
Tensor dropout [33] is a principled way to incorporate
    
F = ρ X ×1 U(T ) ×1 diag(λ)U(C) , (21)
randomization to robustify the model without altering its
PROCEEDINGS OF THE IEEE 16

(2) (2)
u0
<latexit sha1_base64="CiIHBPBabcTmqIUGs6WhikVvWNs=">AAAB+3icbVC7TsMwFL0pr1JeoYwsFhVSWaqkC4wVLIxFog+pDZXjOq1Vx4lsB1FF+RUWBhBi5UfY+BucNgO0HMnS0Tn36h4fP+ZMacf5tkobm1vbO+Xdyt7+weGRfVztqiiRhHZIxCPZ97GinAna0Uxz2o8lxaHPac+f3eR+75FKxSJxr+cx9UI8ESxgBGsjjezqMMR66gdpko2ch7TevMhGds1pOAugdeIWpAYF2iP7aziOSBJSoQnHSg1cJ9ZeiqVmhNOsMkwUjTGZ4QkdGCpwSJWXLrJn6NwoYxRE0jyh0UL9vZHiUKl56JvJPKla9XLxP2+Q6ODKS5mIE00FWR4KEo50hPIi0JhJSjSfG4KJZCYrIlMsMdGmroopwV398jrpNhuu03DvnFrruqijDKdwBnVw4RJacAtt6ACBJ3iGV3izMuvFerc+lqMlq9g5gT+wPn8AZ7aT/g==</latexit>
sha1_base64="VNZRZc9WBAsRR+5q1FqmimbU2UY=">AAAB+3icbVC7TsMwFL3hWcorlJHFokIqS5V0AYmlgoWxSPQhtSU4rtNadZzIdhBVlG/gD1gYQIiVH2Hjb3DaDtByJEtH59yre3z8mDOlHefbWlldW9/YLGwVt3d29/btg1JLRYkktEkiHsmOjxXlTNCmZprTTiwpDn1O2/74KvfbD1QqFolbPYlpP8RDwQJGsDaSZ5d6IdYjP0iTzHPu0krtNPPsslN1pkDLxJ2Tcr38dHEPAA3P/uoNIpKEVGjCsVJd14l1P8VSM8JpVuwlisaYjPGQdg0VOKSqn06zZ+jEKAMURNI8odFU/b2R4lCpSeibyTypWvRy8T+vm+jgvJ8yESeaCjI7FCQc6QjlRaABk5RoPjEEE8lMVkRGWGKiTV1FU4K7+OVl0qpVXafq3pg2LmGGAhzBMVTAhTOowzU0oAkEHuEZXuHNyqwX6936mI2uWPOdQ/gD6/MHgA6Viw==</latexit>
sha1_base64="/v3/B+5uZInNKo75DJZ4pjlyKwM=">AAAB+3icbVDLSsNAFL3xWesr1qUugkWom5J0o+Cm6MZlBfvANobJdNIOnUzCzEQsId/gH7hxoUi3/og7/8ZJ24W2Hhg4nHMv98zxY0alsu1vY2V1bX1js7BV3N7Z3ds3D0otGSUCkyaOWCQ6PpKEUU6aiipGOrEgKPQZafuj69xvPxIhacTv1DgmbogGnAYUI6Ulzyz1QqSGfpAmmWc/pJXaWeaZZbtqT2EtE2dOyvXy82X//njS8MyvXj/CSUi4wgxJ2XXsWLkpEopiRrJiL5EkRniEBqSrKUchkW46zZ5Zp1rpW0Ek9OPKmqq/N1IUSjkOfT2ZJ5WLXi7+53UTFVy4KeVxogjHs0NBwiwVWXkRVp8KghUba4KwoDqrhYdIIKx0XUVdgrP45WXSqlUdu+rc6jauYIYCHMEJVMCBc6jDDTSgCRie4AXe4N3IjFfjw5jMRleM+c4h/IHx+QMIppax</latexit>
u1
<latexit sha1_base64="aeRZ/b+8C5goGU7ro8F91ggOxiU=">AAAB+3icbVDLSsNAFL3xWesr1qWbwSLUTUmKoMuiG5cV7APaWCbTSTt0MgkzE7GE/IobF4q49Ufc+TdO2iy09cDA4Zx7uWeOH3OmtON8W2vrG5tb26Wd8u7e/sGhfVTpqCiRhLZJxCPZ87GinAna1kxz2oslxaHPadef3uR+95FKxSJxr2cx9UI8FixgBGsjDe3KIMR64gdpkj2ktcZ5NnSHdtWpO3OgVeIWpAoFWkP7azCKSBJSoQnHSvVdJ9ZeiqVmhNOsPEgUjTGZ4jHtGypwSJWXzrNn6MwoIxRE0jyh0Vz9vZHiUKlZ6JvJPKla9nLxP6+f6ODKS5mIE00FWRwKEo50hPIi0IhJSjSfGYKJZCYrIhMsMdGmrrIpwV3+8irpNOquU3fvLqrN66KOEpzAKdTAhUtowi20oA0EnuAZXuHNyqwX6936WIyuWcXOMfyB9fkDatKUAw==</latexit>
uR
<latexit sha1_base64="Z6SxlRv5tTM4OtAa3m1Peud0FBM=">AAAB+3icbVDLSsNAFL2pr1pfsS7dBItQNyUpgi6LblxWsQ9oY5hMJ+3QySTMTMQS8ituXCji1h9x5984abPQ1gMDh3Pu5Z45fsyoVLb9bZTW1jc2t8rblZ3dvf0D87DalVEiMOngiEWi7yNJGOWko6hipB8LgkKfkZ4/vc793iMRkkb8Xs1i4oZozGlAMVJa8szqMERq4gdpkj2k9eZZ5t15Zs1u2HNYq8QpSA0KtD3zaziKcBISrjBDUg4cO1ZuioSimJGsMkwkiRGeojEZaMpRSKSbzrNn1qlWRlYQCf24subq740UhVLOQl9P5knlspeL/3mDRAWXbkp5nCjC8eJQkDBLRVZehDWigmDFZpogLKjOauEJEggrXVdFl+Asf3mVdJsNx244t+e11lVRRxmO4QTq4MAFtOAG2tABDE/wDK/wZmTGi/FufCxGS0axcwR/YHz+AJzWlCQ=</latexit>
(2)
F Tensor s ruc ures n po ynom a ne works and a en on
mechan sms
<latexit sha1_base64="Vi+K8ELYmTf+ipNToABgESmitc8=">AAAB8nicbVDLSgMxFM3UVx1fVZdugkVwVWZE1GXRjcsK9gHtUDKZTBuaSYbkjlCGfoYbF4q49Wvc+Tdm2llo64HA4Zxzyb0nTAU34HnfTmVtfWNzq7rt7uzu7R/UDo86RmWasjZVQuleSAwTXLI2cBCsl2pGklCwbji5K/zuE9OGK/kI05QFCRlJHnNKwEr9gbDRiAx91x3W6l7DmwOvEr8kdVSiNax9DSJFs4RJoIIY0/e9FIKcaOBUsJk7yAxLCZ2QEetbKknCTJDPV57hM6tEOFbaPgl4rv6eyElizDQJbTIhMDbLXiH+5/UziG+CnMs0Aybp4qM4ExgULu7HEdeMgphaQqjmdldMx0QTCralogR/+eRV0rlo+FcN7+Gy3rwt66iiE3SKzpGPrlET3aMWaiOKFHpGr+jNAefFeXc+FtGKU84coz9wPn8A11OQTg==</latexit>

<latexit sha1_base64="2i6RfKeIHze8bGwySG8gtzy4dKA=">AAAB8nicbVBNSwMxFHxbv+r6VfXoJVgET2VXRD0WvXisYm1hu5RsNtuGZpMlyQql9Gd48aCIV3+NN/+N2XYP2joQGGbmkfcmyjjTxvO+ncrK6tr6RnXT3dre2d2r7R88apkrQttEcqm6EdaUM0HbhhlOu5miOI047USjm8LvPFGlmRQPZpzRMMUDwRJGsLFS0OM2GuP+vev2a3Wv4c2AlolfkjqUaPVrX71YkjylwhCOtQ58LzPhBCvDCKdTt5drmmEywgMaWCpwSnU4ma08RSdWiVEilX3CoJn6e2KCU63HaWSTKTZDvegV4n9ekJvkKpwwkeWGCjL/KMk5MhIV96OYKUoMH1uCiWJ2V0SGWGFibEtFCf7iycvk8azhXzS8u/N687qsowpHcAyn4MMlNOEWWtAGAhKe4RXeHOO8OO/OxzxaccqZQ/gD5/MHCaiQbw==</latexit>

<latexit sha1_base64="i8KoeG9ZJF8f+g6WMhfeieJBP8A=">AAAB8XicbVBNS8NAFHypX7V+VT16WSyCp5KIqMeiF48VbC22oWw2L+3SzSbsboRS+i+8eFDEq//Gm//GTZuDtg4sDDPz2PcmSAXXxnW/ndLK6tr6RnmzsrW9s7tX3T9o6yRTDFssEYnqBFSj4BJbhhuBnVQhjQOBD8HoJvcfnlBpnsh7M07Rj+lA8ogzaqz02BM2GtK+W+lXa27dnYEsE68gNSjQ7Fe/emHCshilYYJq3fXc1PgTqgxnAqeVXqYxpWxEB9i1VNIYtT+ZbTwlJ1YJSZQo+6QhM/X3xITGWo/jwCZjaoZ60cvF/7xuZqIrf8JlmhmUbP5RlAliEpKfT0KukBkxtoQyxe2uhA2poszYkvISvMWTl0n7rO5d1N2781rjuqijDEdwDKfgwSU04Baa0AIGEp7hFd4c7bw4787HPFpyiplD+APn8weeWZA5</latexit>

0 1 R
⇡
<latexit sha1_base64="1q/1tZx1S5Cts5qr+pV12vBHIps=">AAAB7nicbVBNSwMxEJ2tX7V+VT16CRbBU9kVQY9FLx4r2A9ol5JNs21oNglJVixLf4QXD4p49fd489+YbfegrQ8GHu/NMDMvUpwZ6/vfXmltfWNzq7xd2dnd2z+oHh61jUw1oS0iudTdCBvKmaAtyyynXaUpTiJOO9HkNvc7j1QbJsWDnSoaJngkWMwItk7q9LFSWj4NqjW/7s+BVklQkBoUaA6qX/2hJGlChSUcG9MLfGXDDGvLCKezSj81VGEywSPac1TghJowm587Q2dOGaJYalfCorn6eyLDiTHTJHKdCbZjs+zl4n9eL7XxdZgxoVJLBVksilOOrET572jINCWWTx3BRDN3KyJjrDGxLqGKCyFYfnmVtC/qgV8P7i9rjZsijjKcwCmcQwBX0IA7aEILCEzgGV7hzVPei/fufSxaS14xcwx/4H3+AJNij7Y=</latexit>
<latexit

u0
(1)
+
<latexit sha1_base64="aWOcHJrcbrsxJSLO3n80z068CGE=">AAAB6HicbVBNS8NAEJ3Ur1q/qh69LBZBEEoigh6LXjy2YD+gDWWznbRrN5uwuxFK6C/w4kERr/4kb/4bt20O2vpg4PHeDDPzgkRwbVz32ymsrW9sbhW3Szu7e/sH5cOjlo5TxbDJYhGrTkA1Ci6xabgR2EkU0igQ2A7GdzO//YRK81g+mEmCfkSHkoecUWOlxkW/XHGr7hxklXg5qUCOer/81RvELI1QGiao1l3PTYyfUWU4Ezgt9VKNCWVjOsSupZJGqP1sfuiUnFllQMJY2ZKGzNXfExmNtJ5Ege2MqBnpZW8m/ud1UxPe+BmXSWpQssWiMBXExGT2NRlwhcyIiSWUKW5vJWxEFWXGZlOyIXjLL6+S1mXVc6te46pSu83jKMIJnMI5eHANNbiHOjSBAcIzvMKb8+i8OO/Ox6K14OQzx/AHzucPcYWMrw==</latexit>
<latexit

u1
(1)
+···+
<latexit sha1_base64="gELhB3ha2w5Sm2A6DDXvXJWdsiM=">AAAB8XicbVBNS8NAEJ3Ur1q/qh69LBZBKJREBD0WvXisYD+wDWWz2bRLN5uwOxFK6b/w4kERr/4bb/4bt20O2vpg4PHeDDPzglQKg6777RTW1jc2t4rbpZ3dvf2D8uFRyySZZrzJEpnoTkANl0LxJgqUvJNqTuNA8nYwup357SeujUjUA45T7sd0oEQkGEUrPVZJj4UJGlLtlytuzZ2DrBIvJxXI0eiXv3phwrKYK2SSGtP13BT9CdUomOTTUi8zPKVsRAe8a6miMTf+ZH7xlJxZJSRRom0pJHP198SExsaM48B2xhSHZtmbif953Qyja38iVJohV2yxKMokwYTM3ieh0JyhHFtCmRb2VsKGVFOGNqSSDcFbfnmVtC5qnlvz7i8r9Zs8jiKcwCmcgwdXUIc7aEATGCh4hld4c4zz4rw7H4vWgpPPHMMfOJ8/LSaP7Q==</latexit>

(1)
uR
W
<latexit sha1_base64="rth1M7vaa+nNvQmshDPDRtiZUkc=">AAAB8nicbVDLSsNAFL2pr1pfVZdugkVwVRIRdVl047KCfUAbymQ6aYdOZsLMjVBCP8ONC0Xc+jXu/BsnbRbaemDgcM69zLknTAQ36HnfTmltfWNzq7xd2dnd2z+oHh61jUo1ZS2qhNLdkBgmuGQt5ChYN9GMxKFgnXByl/udJ6YNV/IRpwkLYjKSPOKUoJV6/ZjgmBKRdWaDas2re3O4q8QvSA0KNAfVr/5Q0TRmEqkgxvR8L8EgIxo5FWxW6aeGJYROyIj1LJUkZibI5pFn7plVhm6ktH0S3bn6eyMjsTHTOLSTeUSz7OXif14vxegmyLhMUmSSLj6KUuGicvP73SHXjKKYWkKo5jarS8dEE4q2pYotwV8+eZW0L+r+Vd17uKw1bos6ynACp3AOPlxDA+6hCS2goOAZXuHNQefFeXc+FqMlp9g5hj9wPn8Ak2KRcg==</latexit>

<latexit sha1_base64="zplhfoNRrn5/uFyHq2VvCZn32Ls=">AAAB+3icbVA9T8MwFHwpX6V8hTKyRFRIZakcFhgrWBiLRFukNkSO67RWHSeyHUQV5a+wMIAQK3+EjX+D02aAlpMsne7e0ztfkHCmNELfVmVtfWNzq7pd29nd2z+wD+s9FaeS0C6JeSzvA6woZ4J2NdOc3ieS4ijgtB9Mrwu//0ilYrG407OEehEeCxYygrWRfLs+jLCeBGGW5j56yJruWe7bDdRCczirxC1JA0p0fPtrOIpJGlGhCcdKDVyUaC/DUjPCaV4bpoommEzxmA4MFTiiysvm2XPn1CgjJ4yleUI7c/X3RoYjpWZRYCaLpGrZK8T/vEGqw0svYyJJNRVkcShMuaNjpyjCGTFJieYzQzCRzGR1yARLTLSpq2ZKcJe/vEp65y0Xtdxb1GhflXVU4RhOoAkuXEAbbqADXSDwBM/wCm9Wbr1Y79bHYrRilTtH8AfW5w9mMJP9</latexit>
sha1_base64="BEK/Im4/pWDgHDv6+QyrbZyurpQ=">AAAB+3icbVC7TsMwFL0pr1JeoYwsFhVSWaqEBSSWChbGItGH1JbguE5r1XEi20FUUb6BP2BhACFWfoSNv8FpO0DLkSwdnXOv7vHxY86Udpxvq7Cyura+UdwsbW3v7O7Z++WWihJJaJNEPJIdHyvKmaBNzTSnnVhSHPqctv3xVe63H6hULBK3ehLTfoiHggWMYG0kzy73QqxHfpAmmefcpVX3JPPsilNzpkDLxJ2TSr3ydHEPAA3P/uoNIpKEVGjCsVJd14l1P8VSM8JpVuolisaYjPGQdg0VOKSqn06zZ+jYKAMURNI8odFU/b2R4lCpSeibyTypWvRy8T+vm+jgvJ8yESeaCjI7FCQc6QjlRaABk5RoPjEEE8lMVkRGWGKiTV0lU4K7+OVl0jqtuU7NvTFtXMIMRTiEI6iCC2dQh2toQBMIPMIzvMKblVkv1rv1MRstWPOdA/gD6/MHfoiVig==</latexit>
sha1_base64="S63xG3fBWJCrFG2mNBAMIAY992E=">AAAB+3icbVDLSsNAFL2pr1pfsS51ESxC3ZTEjYKbohuXFewD2xgm00k7dDIJMxOxhHyDf+DGhSLd+iPu/BsnbRfaemDgcM693DPHjxmVyra/jcLK6tr6RnGztLW9s7tn7pdbMkoEJk0csUh0fCQJo5w0FVWMdGJBUOgz0vZH17nffiRC0ojfqXFM3BANOA0oRkpLnlnuhUgN/SBNMs9+SKvOaeaZFbtmT2EtE2dOKvXK82X//mjS8MyvXj/CSUi4wgxJ2XXsWLkpEopiRrJSL5EkRniEBqSrKUchkW46zZ5ZJ1rpW0Ek9OPKmqq/N1IUSjkOfT2ZJ5WLXi7+53UTFVy4KeVxogjHs0NBwiwVWXkRVp8KghUba4KwoDqrhYdIIKx0XSVdgrP45WXSOqs5ds251W1cwQxFOIRjqIID51CHG2hAEzA8wQu8wbuRGa/GhzGZjRaM+c4B/IHx+QMHIJaw</latexit> <latexit sha1_base64="VDVtgmyCVma59XnNKQsl8+Wzync=">AAAB+3icbVDLSsNAFL2pr1pfsS7dBItQNyUjgi6LblxWsA9oY5lMJ+3QySTMTMQS8ituXCji1h9x5984abPQ1gMDh3Pu5Z45fsyZ0q77bZXW1jc2t8rblZ3dvf0D+7DaUVEiCW2TiEey52NFORO0rZnmtBdLikOf064/vcn97iOVikXiXs9i6oV4LFjACNZGGtrVQYj1xA/SJHtI6+gsG6KhXXMb7hzOKkEFqUGB1tD+GowikoRUaMKxUn3kxtpLsdSMcJpVBomiMSZTPKZ9QwUOqfLSefbMOTXKyAkiaZ7Qzlz9vZHiUKlZ6JvJPKla9nLxP6+f6ODKS5mIE00FWRwKEu7oyMmLcEZMUqL5zBBMJDNZHTLBEhNt6qqYEtDyl1dJ57yB3Aa6u6g1r4s6ynAMJ1AHBJfQhFtoQRsIPMEzvMKblVkv1rv1sRgtWcXOEfyB9fkDaUqUAg==</latexit>

<latexit sha1_base64="jVarAtMrBBRu7q3hVtT/KqbDK7M=">AAAB+3icbVDLSsNAFL3xWesr1qWbwSLUTUlE0GXRjcsq9gFtLJPppB06mYSZiVhCfsWNC0Xc+iPu/BsnbRbaemDgcM693DPHjzlT2nG+rZXVtfWNzdJWeXtnd2/fPqi0VZRIQlsk4pHs+lhRzgRtaaY57caS4tDntONPrnO/80ilYpG419OYeiEeCRYwgrWRBnalH2I99oM0yR7SmnuaDe4GdtWpOzOgZeIWpAoFmgP7qz+MSBJSoQnHSvVcJ9ZeiqVmhNOs3E8UjTGZ4BHtGSpwSJWXzrJn6MQoQxRE0jyh0Uz9vZHiUKlp6JvJPKla9HLxP6+X6ODSS5mIE00FmR8KEo50hPIi0JBJSjSfGoKJZCYrImMsMdGmrrIpwV388jJpn9Vdp+7enlcbV0UdJTiCY6iBCxfQgBtoQgsIPMEzvMKblVkv1rv1MR9dsYqdQ/gD6/MHm06UIw==</latexit>

No ebook po ynom a neu a ne wo k as o d gene a on pynb

(0) (0) (0)
u0
<latexit sha1_base64="CAuGRcia7Gz/Xp6jZvwVYGUuoTc=">AAAB+3icbVA9T8MwFHwpX6V8hTKyRFRIZakcFhgrWBiLRFukNkSO67RWHSeyHUQV5a+wMIAQK3+EjX+D02aAlpMsne7e0ztfkHCmNELfVmVtfWNzq7pd29nd2z+wD+s9FaeS0C6JeSzvA6woZ4J2NdOc3ieS4ijgtB9Mrwu//0ilYrG407OEehEeCxYygrWRfLs+jLCeBGGW5j56yJroLPftBmqhOZxV4pakASU6vv01HMUkjajQhGOlBi5KtJdhqRnhNK8NU0UTTKZ4TAeGChxR5WXz7LlzapSRE8bSPKGdufp7I8ORUrMoMJNFUrXsFeJ/3iDV4aWXMZGkmgqyOBSm3NGxUxThjJikRPOZIZhIZrI6ZIIlJtrUVTMluMtfXiW985aLWu4tarSvyjqqcAwn0AQXLqANN9CBLhB4gmd4hTcrt16sd+tjMVqxyp0j+APr8wdkqpP8</latexit>
sha1_base64="zKIgcI4HIXvjCk8X7vwM7EmYvKA=">AAAB+3icbVDLSgMxFL1TX7W+xrp0EyxC3ZSMGwU3RTcuK9gHtHXMpJk2NPMgyYhlmG/wD9y4UMStP+LOvzHTdqGtBwKHc+7lnhwvFlxpjL+twsrq2vpGcbO0tb2zu2fvl1sqSiRlTRqJSHY8opjgIWtqrgXrxJKRwBOs7Y2vcr/9wKTiUXirJzHrB2QYcp9Too3k2uVeQPTI89Mkc/FdWsUnmWtXcA1PgZaJMyeVeuXp4h4AGq791RtENAlYqKkgSnUdHOt+SqTmVLCs1EsUiwkdkyHrGhqSgKl+Os2eoWOjDJAfSfNCjabq742UBEpNAs9M5knVopeL/3ndRPvn/ZSHcaJZSGeH/EQgHaG8CDTgklEtJoYQKrnJiuiISEK1qatkSnAWv7xMWqc1B9ecG9PGJcxQhEM4gio4cAZ1uIYGNIHCIzzDK7xZmfVivVsfs9GCNd85gD+wPn8AfQKViQ==</latexit>
sha1_base64="ts80tZhOjwU8DKJf8kIGi9Ny0fI=">AAAB+3icbVC7TsMwFHXKq5RXKCMMFhVSWSqHBSSWChbGItGHaEPkuE5r1XEi20FUUb6BP2BhAKGu/Agbf4PTdoCWI1k6Oude3ePjx5wpjdC3VVhZXVvfKG6WtrZ3dvfs/XJLRYkktEkiHsmOjxXlTNCmZprTTiwpDn1O2/7oOvfbj1QqFok7PY6pG+KBYAEjWBvJs8u9EOuhH6RJ5qGHtIpOM8+uoBqaAi4TZ04q9crzZf/+aNLw7K9ePyJJSIUmHCvVdVCs3RRLzQinWamXKBpjMsID2jVU4JAqN51mz+CJUfowiKR5QsOp+nsjxaFS49A3k3lStejl4n9eN9HBhZsyESeaCjI7FCQc6gjmRcA+k5RoPjYEE8lMVkiGWGKiTV0lU4Kz+OVl0jqrOajm3Jo2rsAMRXAIjkEVOOAc1MENaIAmIOAJvIA38G5l1qv1YU1mowVrvnMA/sD6/AEFmpav</latexit>
u1
<latexit sha1_base64="4RJHkPCKOdHBKrIU9Gop6wfiZpE=">AAAB+3icbVA9T8MwFHwpX6V8hTKyRFRIZakcFhgrWBiLRFukNkSO67RWHSeyHUQV5a+wMIAQK3+EjX+D02aAlpMsne7e0ztfkHCmNELfVmVtfWNzq7pd29nd2z+wD+s9FaeS0C6JeSzvA6woZ4J2NdOc3ieS4ijgtB9Mrwu//0ilYrG407OEehEeCxYygrWRfLs+jLCeBGGW5r77kDXRWe7bDdRCczirxC1JA0p0fPtrOIpJGlGhCcdKDVyUaC/DUjPCaV4bpoommEzxmA4MFTiiysvm2XPn1CgjJ4yleUI7c/X3RoYjpWZRYCaLpGrZK8T/vEGqw0svYyJJNRVkcShMuaNjpyjCGTFJieYzQzCRzGR1yARLTLSpq2ZKcJe/vEp65y0Xtdxb1GhflXVU4RhOoAkuXEAbbqADXSDwBM/wCm9Wbr1Y79bHYrRilTtH8AfW5w9mNJP9</latexit>
sha1_base64="D+hn/TPbRffsrBwFza/aixnsEEU=">AAAB+3icbVC7TsMwFL0pr1JeoYwsFhVSWaqEBSSWChbGItGH1JbguE5r1XEi20FUUb6BP2BhACFWfoSNv8FpO0DLkSwdnXOv7vHxY86Udpxvq7Cyura+UdwsbW3v7O7Z++WWihJJaJNEPJIdHyvKmaBNzTSnnVhSHPqctv3xVe63H6hULBK3ehLTfoiHggWMYG0kzy73QqxHfpAmmefepVXnJPPsilNzpkDLxJ2TSr3ydHEPAA3P/uoNIpKEVGjCsVJd14l1P8VSM8JpVuolisaYjPGQdg0VOKSqn06zZ+jYKAMURNI8odFU/b2R4lCpSeibyTypWvRy8T+vm+jgvJ8yESeaCjI7FCQc6QjlRaABk5RoPjEEE8lMVkRGWGKiTV0lU4K7+OVl0jqtuU7NvTFtXMIMRTiEI6iCC2dQh2toQBMIPMIzvMKblVkv1rv1MRstWPOdA/gD6/MHfoyVig==</latexit>
sha1_base64="nnuVeggnWbedb11xLyugyPFH7aU=">AAAB+3icbVDLSsNAFL2pr1pfsS51ESxC3ZTEjYKbohuXFewD2xgm00k7dDIJMxOxhHyDf+DGhSLd+iPu/BsnbRfaemDgcM693DPHjxmVyra/jcLK6tr6RnGztLW9s7tn7pdbMkoEJk0csUh0fCQJo5w0FVWMdGJBUOgz0vZH17nffiRC0ojfqXFM3BANOA0oRkpLnlnuhUgN/SBNMs95SKv2aeaZFbtmT2EtE2dOKvXK82X//mjS8MyvXj/CSUi4wgxJ2XXsWLkpEopiRrJSL5EkRniEBqSrKUchkW46zZ5ZJ1rpW0Ek9OPKmqq/N1IUSjkOfT2ZJ5WLXi7+53UTFVy4KeVxogjHs0NBwiwVWXkRVp8KghUba4KwoDqrhYdIIKx0XSVdgrP45WXSOqs5ds251W1cwQxFOIRjqIID51CHG2hAEzA8wQu8wbuRGa/GhzGZjRaM+c4B/IHx+QMHJJaw</latexit>
uR
<latexit sha1_base64="vTsTltRZCKKWV3XpQVVZ8gbGGvQ=">AAAB+3icbVC7TsMwFL0pr1JepYwsFhVSWSqHBcYKFsaC6ENqQ+S4TmvVech2EFWUX2FhACFWfoSNv8FpM0DLkSwdnXOv7vHxYsGVxvjbKq2tb2xulbcrO7t7+wfVw1pXRYmkrEMjEcm+RxQTPGQdzbVg/VgyEniC9bzpde73HplUPArv9SxmTkDGIfc5JdpIbrU2DIieeH6aZO7dQ9rAZ5lbreMmngOtErsgdSjQdqtfw1FEk4CFmgqi1MDGsXZSIjWngmWVYaJYTOiUjNnA0JAETDnpPHuGTo0yQn4kzQs1mqu/N1ISKDULPDOZJ1XLXi7+5w0S7V86KQ/jRLOQLg75iUA6QnkRaMQlo1rMDCFUcpMV0QmRhGpTV8WUYC9/eZV0z5s2btq3uN66KuoowzGcQANsuIAW3EAbOkDhCZ7hFd6szHqx3q2PxWjJKnaO4A+szx+Y/pQe</latexit>
sha1_base64="Gek23j3w3hKXy7z3mDu6Rdml72E=">AAAB+3icbVC7TsMwFL2hPEp5hTKyWFSVylIlLDBWsDAWRB9SGyLHdVqrjhPZDqKKMvIbLAwgxMqPsPE3uI8BWo5k6eice3WPT5BwprTjfFtrhfWNza3idmlnd2//wD4st1WcSkJbJOax7AZYUc4EbWmmOe0mkuIo4LQTjK+mfueBSsVicacnCfUiPBQsZARrI/l2uR9hPQrCLM392/us5pzmvl1x6s4MaJW4C1JpVJ8KDQBo+vZXfxCTNKJCE46V6rlOor0MS80Ip3mpnyqaYDLGQ9ozVOCIKi+bZc9R1SgDFMbSPKHRTP29keFIqUkUmMlpUrXsTcX/vF6qwwsvYyJJNRVkfihMOdIxmhaBBkxSovnEEEwkM1kRGWGJiTZ1lUwJ7vKXV0n7rO46dffGtHEJcxThGE6gBi6cQwOuoQktIPAIz/AKb1ZuvVjv1sd8dM1a7BzBH1ifPzAQlUs=</latexit>
sha1_base64="rnITZ1oiFnf3doGLkv40ntoT4lM=">AAAB+3icbVC7TgJBFL0LPhBfK5Y2EwkJNmTXRkuijSUaeSSAm9lhFibMPjIzaySbLf0NGwuMsfVH7IylP+IsUCh4kklOzrk398xxI86ksqxPI5dfW9/YLGwVt3d29/bNg1JLhrEgtElCHoqOiyXlLKBNxRSnnUhQ7Luctt3xZea376mQLAxu1SSifR8PA+YxgpWWHLPU87EauV4Sp87NXVK1TlLHLFs1awa0SuwFKdcrj3n7e/rVcMyP3iAksU8DRTiWsmtbkeonWChGOE2LvVjSCJMxHtKupgH2qewns+wpqmhlgLxQ6BcoNFN/byTYl3Liu3oySyqXvUz8z+vGyjvvJyyIYkUDMj/kxRypEGVFoAETlCg+0QQTwXRWREZYYKJ0XUVdgr385VXSOq3ZVs2+1m1cwBwFOIJjqIINZ1CHK2hAEwg8wBNM4cVIjWfj1Xibj+aMxc4h/IHx/gNPC5eh</latexit>

Apar from CNNs ensor s ruc ures ar se n po ynom a

··· ⇠ Bernoulli(✓)
<latexit sha1_base64="9L8q94v0h+NaL5zP9PP9a02a6pU=">AAACD3icbVC7SgNBFJ31GeMramkzGJTYhF0RtQyxsYxgHpANYXZykwyZnV1m7ophyR/Y+Cs2ForY2tr5N04ehSYeGDiccy537gliKQy67reztLyyurae2chubm3v7Ob29msmSjSHKo9kpBsBMyCFgioKlNCINbAwkFAPBtdjv34P2ohI3eEwhlbIekp0BWdopXbuxJc23GHUNyKkPsIDpmXQKkqkFKOCj31Adppt5/Ju0Z2ALhJvRvJkhko79+V3Ip6EoJBLZkzTc2NspUyj4BJGWT8xEDM+YD1oWqpYCKaVTu4Z0WOrdGg30vYppBP190TKQmOGYWCTIcO+mffG4n9eM8HuVSsVKk4QFJ8u6iaSYkTH5dCO0MBRDi1hXAv7V8r7TDOOtsJxCd78yYukdlb0Loru7Xm+VJ7VkSGH5IgUiEcuSYnckAqpEk4eyTN5JW/Ok/PivDsf0+iSM5s5IH/gfP4Am3ucYA==</latexit>

<latexit sha1_base64="Vi+K8ELYmTf+ipNToABgESmitc8=">AAAB8nicbVDLSgMxFM3UVx1fVZdugkVwVWZE1GXRjcsK9gHtUDKZTBuaSYbkjlCGfoYbF4q49Wvc+Tdm2llo64HA4Zxzyb0nTAU34HnfTmVtfWNzq7rt7uzu7R/UDo86RmWasjZVQuleSAwTXLI2cBCsl2pGklCwbji5K/zuE9OGK/kI05QFCRlJHnNKwEr9gbDRiAx91x3W6l7DmwOvEr8kdVSiNax9DSJFs4RJoIIY0/e9FIKcaOBUsJk7yAxLCZ2QEetbKknCTJDPV57hM6tEOFbaPgl4rv6eyElizDQJbTIhMDbLXiH+5/UziG+CnMs0Aybp4qM4ExgULu7HEdeMgphaQqjmdldMx0QTCralogR/+eRV0rlo+FcN7+Gy3rwt66iiE3SKzpGPrlET3aMWaiOKFHpGr+jNAefFeXc+FtGKU84coz9wPn8A11OQTg==</latexit> <latexit sha1_base64="fj3n9bRRIMbETcWokO4cEM+/qug=">AAAB8XicbVDLSgMxFL1TX7W+qi7dBIvgqswUUZdFNy4r2Ae2Q8lkMm1oJhmSjFCG/oUbF4q49W/c+Tdm2llo64HA4Zxzyb0nSDjTxnW/ndLa+sbmVnm7srO7t39QPTzqaJkqQttEcql6AdaUM0HbhhlOe4miOA447QaT29zvPlGlmRQPZppQP8YjwSJGsLHS44DbaIiHjcqwWnPr7hxolXgFqUGB1rD6NQglSWMqDOFY677nJsbPsDKMcDqrDFJNE0wmeET7lgocU+1n841n6MwqIYqksk8YNFd/T2Q41noaBzYZYzPWy14u/uf1UxNd+xkTSWqoIIuPopQjI1F+PgqZosTwqSWYKGZ3RWSMFSbGlpSX4C2fvEo6jbp3WXfvL2rNm6KOMpzAKZyDB1fQhDtoQRsICHiGV3hztPPivDsfi2jJKWaO4Q+czx+hY5A7</latexit> <latexit sha1_base64="2i6RfKeIHze8bGwySG8gtzy4dKA=">AAAB8nicbVBNSwMxFHxbv+r6VfXoJVgET2VXRD0WvXisYm1hu5RsNtuGZpMlyQql9Gd48aCIV3+NN/+N2XYP2joQGGbmkfcmyjjTxvO+ncrK6tr6RnXT3dre2d2r7R88apkrQttEcqm6EdaUM0HbhhlOu5miOI047USjm8LvPFGlmRQPZpzRMMUDwRJGsLFS0OM2GuP+vev2a3Wv4c2AlolfkjqUaPVrX71YkjylwhCOtQ58LzPhBCvDCKdTt5drmmEywgMaWCpwSnU4ma08RSdWiVEilX3CoJn6e2KCU63HaWSTKTZDvegV4n9ekJvkKpwwkeWGCjL/KMk5MhIV96OYKUoMH1uCiWJ2V0SGWGFibEtFCf7iycvk8azhXzS8u/N687qsowpHcAyn4MMlNOEWWtAGAhKe4RXeHOO8OO/OxzxaccqZQ/gD5/MHCaiQbw==</latexit>

<latexit sha1_base64="i8KoeG9ZJF8f+g6WMhfeieJBP8A=">AAAB8XicbVBNS8NAFHypX7V+VT16WSyCp5KIqMeiF48VbC22oWw2L+3SzSbsboRS+i+8eFDEq//Gm//GTZuDtg4sDDPz2PcmSAXXxnW/ndLK6tr6RnmzsrW9s7tX3T9o6yRTDFssEYnqBFSj4BJbhhuBnVQhjQOBD8HoJvcfnlBpnsh7M07Rj+lA8ogzaqz02BM2GtK+W+lXa27dnYEsE68gNSjQ7Fe/emHCshilYYJq3fXc1PgTqgxnAqeVXqYxpWxEB9i1VNIYtT+ZbTwlJ1YJSZQo+6QhM/X3xITGWo/jwCZjaoZ60cvF/7xuZqIrf8JlmhmUbP5RlAliEpKfT0KukBkxtoQyxe2uhA2poszYkvISvMWTl0n7rO5d1N2781rjuqijDEdwDKfgwSU04Baa0AIGEp7hFd4c7bw4787HPFpyiplD+APn8weeWZA5</latexit>

R
<latexit sha1_base64="Wui+5Kue34lfzK7VwB4qmvb1Hb0=">AAAB7nicbVBNS8NAEJ3Ur1q/qh69BIvgqSQi6rHoxWMF+wFtKJvNpl262Q27E6GU/ggvHhTx6u/x5r9x0+agrQ8GHu/NMDMvTAU36HnfTmltfWNzq7xd2dnd2z+oHh61jco0ZS2qhNLdkBgmuGQt5ChYN9WMJKFgnXB8l/udJ6YNV/IRJykLEjKUPOaUoJU6fRopNJVBtebVvTncVeIXpAYFmoPqVz9SNEuYRCqIMT3fSzGYEo2cCjar9DPDUkLHZMh6lkqSMBNM5+fO3DOrRG6stC2J7lz9PTEliTGTJLSdCcGRWfZy8T+vl2F8E0y5TDNkki4WxZlwUbn5727ENaMoJpYQqrm91aUjoglFm1Aegr/88ippX9T9q7r3cFlr3BZxlOEETuEcfLiGBtxDE1pAYQzP8ApvTuq8OO/Ox6K15BQzx/AHzucP5oyPSA==</latexit>

0 1 2
ne works and a en on mechan sms cap ur ng h gher-order
Fig. 15: Tensor Dropout on a CP decomposition. Visualiza- (e g mu p ca ve) n erac ons among da a or parame ers
tion of tensor dropout applied to a CP factorization. Here, a Here we prov de an overv ew of how ensor me hods are
Bernouilli dropout is applied by first sampling a multivariate ncorpora ed w h n such mode s
Bernoulli random vector λ. Its elements have value 1 (red) Mu p ca ve n erac ons and po ynom a func on approx-
with probability θ and 0 (white) with probability 1 − θ. Each ma on have ong been used n he doma n of mach ne
rank-1 CP component is kept or discarded according to the earn ng Group Me hod of Da a Hand ng (GMDH) [154]
corresponding Bernoulli variable, thus inducing stochasticity proposes bu d ng quadra c po ynom a s from npu e emen s
on the rank of the decomposition. The e emen s n each quadra c po ynom a are prede erm ned
wh ch does no a ow he me hod o cap ure da a-dr ven
corre a ons among d fferen e emen s The me hod was a er
ex ended n h gher-order po ynom a s [155] Low-order po y-
nom a ne works were a so used n p -s gma [156] and s gma-
I
<latexit sha1_base64="nW+xfhCanliMCGrQ8ApFDvo4sm0=">AAAB6HicbVA9SwNBEJ2LXzF+RS1tFoNgFe5sTBm00S4B8wHJEfY2c8mavb1jd08IR36BjYUitv4kO/+Nm+QKTXww8Hhvhpl5QSK4Nq777RQ2Nre2d4q7pb39g8Oj8vFJW8epYthisYhVN6AaBZfYMtwI7CYKaRQI7AST27nfeUKleSwfzDRBP6IjyUPOqLFS835QrrhVdwGyTrycVCBHY1D+6g9jlkYoDRNU657nJsbPqDKcCZyV+qnGhLIJHWHPUkkj1H62OHRGLqwyJGGsbElDFurviYxGWk+jwHZG1Iz1qjcX//N6qQlrfsZlkhqUbLkoTAUxMZl/TYZcITNiagllittbCRtTRZmx2ZRsCN7qy+ukfVX13KrXdCv1mzyOIpzBOVyCB9dQhztoQAsYIDzDK7w5j86L8+58LFsLTj5zCn/gfP4Anb2MyQ==</latexit>
<latexit sha1_base64="/u/o/sfz0iFL53NGkbEFsJokk8c=">AAAB6HicbZC7SwNBEMbnfMbzFbW0WQyCVbiz0UYM2miXgHlAcoS9zVyyZm/v2N0TQgjY21goYus/Y2/nf+PmUWjiBws/vm+GnZkwFVwbz/t2lpZXVtfWcxvu5tb2zm5+b7+mk0wxrLJEJKoRUo2CS6wabgQ2UoU0DgXWw/71OK8/oNI8kXdmkGIQ067kEWfUWKty284XvKI3EVkEfwaFy0/34hEAyu38V6uTsCxGaZigWjd9LzXBkCrDmcCR28o0ppT1aRebFiWNUQfDyaAjcmydDokSZZ80ZOL+7hjSWOtBHNrKmJqens/G5n9ZMzPReTDkMs0MSjb9KMoEMQkZb006XCEzYmCBMsXtrIT1qKLM2Nu49gj+/MqLUDst+l7Rr3iF0hVMlYNDOIIT8OEMSnADZagCA4QneIFX5955dt6c92npkjPrOYA/cj5+AA0EjpY=</latexit>
sha1_base64="hqmPq/tZDr2D6CBx3rx2YUakZik=">AAAB6HicbZC7SgNBFIbPxltcb1FLm8UgWIVdG23EoI12CZgLJEuYnZxNxszOLjOzQljyBDYWitjqw9jbiG/j5FJo4g8DH/9/DnPOCRLOlHbdbyu3tLyyupZftzc2t7Z3Crt7dRWnkmKNxjyWzYAo5ExgTTPNsZlIJFHAsREMrsZ54x6lYrG41cME/Yj0BAsZJdpY1ZtOoeiW3ImcRfBmULz4sM+T9y+70il8trsxTSMUmnKiVMtzE+1nRGpGOY7sdqowIXRAetgyKEiEys8mg46cI+N0nTCW5gntTNzfHRmJlBpGgamMiO6r+Wxs/pe1Uh2e+RkTSapR0OlHYcodHTvjrZ0uk0g1HxogVDIzq0P7RBKqzW1scwRvfuVFqJ+UPLfkVd1i+RKmysMBHMIxeHAKZbiGCtSAAsIDPMGzdWc9Wi/W67Q0Z8169uGPrLcf/oSQCg==</latexit>
U <latexit sha1_base64="i8K

<latexit sha1_base64="Vi+K8ELYmTf+ipNToABgESmitc8=">AAAB8nicbVDLSgMxFM3UVx1fVZdugkVwVWZE1GXRjcsK9gHtUDKZTBuaSYbkjlCGfoYbF4q49Wvc+Tdm2llo64HA4Zxzyb0nTAU34HnfTmVtfWNzq7rt7uzu7R/UDo86RmWasjZVQuleSAwTXLI2cBCsl2pGklCwbji5K/zuE9OGK/kI05QFCRlJHnNKwEr9gbDRiAx91x3W6l7DmwOvEr8kdVSiNax9DSJFs4RJoIIY0/e9FIKcaOBUsJk7yAxLCZ2QEetbKknCTJDPV57hM6tEOFbaPgl4rv6eyElizDQJbTIhMDbLXiH+5/UziG+CnMs0Aybp4qM4ExgULu7HEdeMgphaQqjmdldMx0QTCralogR/+eRV0rlo+FcN7+Gy3rwt66iiE3SKzpGPrlET3aMWaiOKFHpGr+jNAefFeXc+FtGKU84coz9wPn8A11OQTg==</latexit>

p -s gma [157] The p -s gma ne work nc udes a s ng e h dden

1

⇡
<latexit sha1_base64="1q/1tZx1S5Cts5qr+pV12vBHIps=">AAAB7nicbVBNSwMxEJ2tX7V+VT16CRbBU9kVQY9FLx4r2A9ol5JNs21oNglJVixLf4QXD4p49fd489+YbfegrQ8GHu/NMDMvUpwZ6/vfXmltfWNzq7xd2dnd2z+oHh61jUw1oS0iudTdCBvKmaAtyyynXaUpTiJOO9HkNvc7j1QbJsWDnSoaJngkWMwItk7q9LFSWj4NqjW/7s+BVklQkBoUaA6qX/2hJGlChSUcG9MLfGXDDGvLCKezSj81VGEywSPac1TghJowm587Q2dOGaJYalfCorn6eyLDiTHTJHKdCbZjs+zl4n9eL7XxdZgxoVJLBVksilOOrET572jINCWWTx3BRDN3KyJjrDGxLqGKCyFYfnmVtC/qgV8P7i9rjZsijjKcwCmcQwBX0IA7aEILCEzgGV7hzVPei/fufSxaS14xcwx/4H3+AJNij7Y=</latexit>
<latexit
G M
U K ayer and he ou pu s he produc of a h dden represen a-
M
<latexit sha1_base64="X1Jlvi+WH1kXuzvyGeTdBmwNrOw=">AAAB6HicbVA9SwNBEJ2LXzF+RS1tFoNgFe5sTBm0EWwSMB+QHGFvM5es2ds7dveEcOQX2FgoYutPsvPfuEmu0MQHA4/3ZpiZFySCa+O6305hY3Nre6e4W9rbPzg8Kh+ftHWcKoYtFotYdQOqUXCJLcONwG6ikEaBwE4wuZ37nSdUmsfywUwT9CM6kjzkjBorNe8H5YpbdRcg68TLSQVyNAblr/4wZmmE0jBBte55bmL8jCrDmcBZqZ9qTCib0BH2LJU0Qu1ni0Nn5MIqQxLGypY0ZKH+nshopPU0CmxnRM1Yr3pz8T+vl5qw5mdcJqlByZaLwlQQE5P512TIFTIjppZQpri9lbAxVZQZm03JhuCtvrxO2ldVz616TbdSv8njKMIZnMMleHANdbiDBrSAAcIzvMKb8+i8OO/Ox7K14OQzp/AHzucPoMWMyw==</latexit>
<latexit sha1_base64="Cu+q0FVT/ACPKbJkVwRmHZdE7oU=">AAAB6HicbZC7SwNBEMbnfMbzFbW0WQyCVbiz0UYM2gg2CZgHJEfY28wla/b2jt09IYSAvY2FIrb+M/Z2/jduHoUmfrDw4/tm2JkJU8G18bxvZ2l5ZXVtPbfhbm5t7+zm9/ZrOskUwypLRKIaIdUouMSq4UZgI1VI41BgPexfj/P6AyrNE3lnBikGMe1KHnFGjbUqt+18wSt6E5FF8GdQuPx0Lx4BoNzOf7U6CctilIYJqnXT91ITDKkynAkcua1MY0pZn3axaVHSGHUwnAw6IsfW6ZAoUfZJQybu744hjbUexKGtjKnp6flsbP6XNTMTnQdDLtPMoGTTj6JMEJOQ8dakwxUyIwYWKFPczkpYjyrKjL2Na4/gz6+8CLXTou8V/YpXKF3BVDk4hCM4AR/OoAQ3UIYqMEB4ghd4de6dZ+fNeZ+WLjmzngP4I+fjBxAMjpg=</latexit>
sha1_base64="+JGHOBgUMtqFjzrGlpGMgb8trkE=">AAAB6HicbZC7SgNBFIbPxltcb1FLm8UgWIVdG23EoI1gk4C5QLKE2cnZZMzs7DIzK4QlT2BjoYitPoy9jfg2Ti6FJv4w8PH/5zDnnCDhTGnX/bZyS8srq2v5dXtjc2t7p7C7V1dxKinWaMxj2QyIQs4E1jTTHJuJRBIFHBvB4GqcN+5RKhaLWz1M0I9IT7CQUaKNVb3pFIpuyZ3IWQRvBsWLD/s8ef+yK53CZ7sb0zRCoSknSrU8N9F+RqRmlOPIbqcKE0IHpIctg4JEqPxsMujIOTJO1wljaZ7QzsT93ZGRSKlhFJjKiOi+ms/G5n9ZK9XhmZ8xkaQaBZ1+FKbc0bEz3trpMolU86EBQiUzszq0TySh2tzGNkfw5ldehPpJyXNLXtUtli9hqjwcwCEcgwenUIZrqEANKCA8wBM8W3fWo/VivU5Lc9asZx/+yHr7AQGbkAw=</latexit>

W ons w h fixed we gh s equa o 1 The dea of s gma-p -

<latexit sha1_base64="2i6RfKeIHze8bGwySG8gtzy4dKA=">AAAB8nicbVBNSwMxFHxbv+r6VfXoJVgET2VXRD0WvXisYm1hu5RsNtuGZpMlyQql9Gd48aCIV3+NN/+N2XYP2joQGGbmkfcmyjjTxvO+ncrK6tr6RnXT3dre2d2r7R88apkrQttEcqm6EdaUM0HbhhlOu5miOI047USjm8LvPFGlmRQPZpzRMMUDwRJGsLFS0OM2GuP+vev2a3Wv4c2AlolfkjqUaPVrX71YkjylwhCOtQ58LzPhBCvDCKdTt5drmmEywgMaWCpwSnU4ma08RSdWiVEilX3CoJn6e2KCU63HaWSTKTZDvegV4n9ekJvkKpwwkeWGCjL/KMk5MhIV96OYKUoMH1uCiWJ2V0SGWGFibEtFCf7iycvk8azhXzS8u/N687qsowpHcAyn4MMlNOEWWtAGAhKe4RXeHOO8OO/OxzxaccqZQ/gD5/MHCaiQbw==</latexit>

I
<latexit sha1_base64="nW+xfhCanliMCGrQ8ApFDvo4sm0=">AAAB6HicbVA9SwNBEJ2LXzF+RS1tFoNgFe5sTBm00S4B8wHJEfY2c8mavb1jd08IR36BjYUitv4kO/+Nm+QKTXww8Hhvhpl5QSK4Nq777RQ2Nre2d4q7pb39g8Oj8vFJW8epYthisYhVN6AaBZfYMtwI7CYKaRQI7AST27nfeUKleSwfzDRBP6IjyUPOqLFS835QrrhVdwGyTrycVCBHY1D+6g9jlkYoDRNU657nJsbPqDKcCZyV+qnGhLIJHWHPUkkj1H62OHRGLqwyJGGsbElDFurviYxGWk+jwHZG1Iz1qjcX//N6qQlrfsZlkhqUbLkoTAUxMZl/TYZcITNiagllittbCRtTRZmx2ZRsCN7qy+ukfVX13KrXdCv1mzyOIpzBOVyCB9dQhztoQAsYIDzDK7w5j86L8+58LFsLTj5zCn/gfP4Anb2MyQ==</latexit>
<latexit sha1_base64="/u/o/sfz0iFL53NGkbEFsJokk8c=">AAAB6HicbZC7SwNBEMbnfMbzFbW0WQyCVbiz0UYM2miXgHlAcoS9zVyyZm/v2N0TQgjY21goYus/Y2/nf+PmUWjiBws/vm+GnZkwFVwbz/t2lpZXVtfWcxvu5tb2zm5+b7+mk0wxrLJEJKoRUo2CS6wabgQ2UoU0DgXWw/71OK8/oNI8kXdmkGIQ067kEWfUWKty284XvKI3EVkEfwaFy0/34hEAyu38V6uTsCxGaZigWjd9LzXBkCrDmcCR28o0ppT1aRebFiWNUQfDyaAjcmydDokSZZ80ZOL+7hjSWOtBHNrKmJqens/G5n9ZMzPReTDkMs0MSjb9KMoEMQkZb006XCEzYmCBMsXtrIT1qKLM2Nu49gj+/MqLUDst+l7Rr3iF0hVMlYNDOIIT8OEMSnADZagCA4QneIFX5955dt6c92npkjPrOYA/cj5+AA0EjpY=</latexit>
sha1_base64="hqmPq/tZDr2D6CBx3rx2YUakZik=">AAAB6HicbZC7SgNBFIbPxltcb1FLm8UgWIVdG23EoI12CZgLJEuYnZxNxszOLjOzQljyBDYWitjqw9jbiG/j5FJo4g8DH/9/DnPOCRLOlHbdbyu3tLyyupZftzc2t7Z3Crt7dRWnkmKNxjyWzYAo5ExgTTPNsZlIJFHAsREMrsZ54x6lYrG41cME/Yj0BAsZJdpY1ZtOoeiW3ImcRfBmULz4sM+T9y+70il8trsxTSMUmnKiVMtzE+1nRGpGOY7sdqowIXRAetgyKEiEys8mg46cI+N0nTCW5gntTNzfHRmJlBpGgamMiO6r+Wxs/pe1Uh2e+RkTSapR0OlHYcodHTvjrZ0uk0g1HxogVDIzq0P7RBKqzW1scwRvfuVFqJ+UPLfkVd1i+RKmysMBHMIxeHAKZbiGCtSAAsIDPMGzdWc9Wi/W67Q0Z8169uGPrLcf/oSQCg==</latexit>

R
R

R2
<latexit sha1_base64="AeMAurJxJyiv14HrOLqcxtmTLjc=">AAAB63icbVBNS8NAEJ3Ur1q/qh69LBbBU0l60WPRi8cq9gPaUDbbTbt0dxN2J0IJ/QtePCji1T/kzX9j0uagrQ8GHu/NMDMviKWw6LrfTmljc2t7p7xb2ds/ODyqHp90bJQYxtsskpHpBdRyKTRvo0DJe7HhVAWSd4Ppbe53n7ixItKPOIu5r+hYi1Awirn0MGxUhtWaW3cXIOvEK0gNCrSG1a/BKGKJ4hqZpNb2PTdGP6UGBZN8XhkklseUTemY9zOqqeLWTxe3zslFpoxIGJmsNJKF+nsipcramQqyTkVxYle9XPzP6ycYXvup0HGCXLPlojCRBCOSP05GwnCGcpYRyozIbiVsQg1lmMWTh+CtvrxOOo2659a9e7fWvCniKMMZnMMleHAFTbiDFrSBwQSe4RXeHOW8OO/Ox7K15BQzp/AHzucPBwONiw==</latexit>
sha1_base64="TxZOu0H6CFRUjeABrWpAEq33nII=">AAAB63icbVA9TwJBEJ3DL8Qv1MTGZiMxsSJ3NFoSbSzBeEACF9xb9mDD7t5ld8+EXPgLNhYaY+sfsrPxt7gHFAq+ZJKX92YyMy9MONPGdb+cwtr6xuZWcbu0s7u3f1A+PGrpOFWE+iTmseqEWFPOJPUNM5x2EkWxCDlth+Ob3G8/UqVZLO/NJKGBwEPJIkawyaW7fq3UL1fcqjsDWiXeglTqJ83vBwBo9MufvUFMUkGlIRxr3fXcxAQZVoYRTqelXqppgskYD2nXUokF1UE2u3WKzq0yQFGsbEmDZurviQwLrScitJ0Cm5Fe9nLxP6+bmugqyJhMUkMlmS+KUo5MjPLH0YApSgyfWIKJYvZWREZYYWJsPHkI3vLLq6RVq3pu1WvaNK5hjiKcwhlcgAeXUIdbaIAPBEbwBC/w6gjn2Xlz3uetBWcxcwx/4Hz8AJWbj3A=</latexit>
sha1_base64="YXW80RRDGsashW7c3kJ+JLp1Su8=">AAAB63icbVC7SgNBFL0bXzG+ooKNzWAQrMJuGi1DbCwTMQ9IljA7mU2GzMwuM7NCWPILNhaK2Fr6F36BnY3f4mySQhMPXDiccy/33hPEnGnjul9Obm19Y3Mrv13Y2d3bPygeHrV0lChCmyTikeoEWFPOJG0aZjjtxIpiEXDaDsbXmd++p0qzSN6ZSUx9gYeShYxgk0m3/UqhXyy5ZXcGtEq8BSlVTxrf7L32Ue8XP3uDiCSCSkM41rrrubHxU6wMI5xOC71E0xiTMR7SrqUSC6r9dHbrFJ1bZYDCSNmSBs3U3xMpFlpPRGA7BTYjvexl4n9eNzHhlZ8yGSeGSjJfFCYcmQhlj6MBU5QYPrEEE8XsrYiMsMLE2HiyELzll1dJq1L23LLXsGnUYI48nMIZXIAHl1CFG6hDEwiM4AGe4NkRzqPz4rzOW3POYuYY/sB5+wHm3JEs</latexit>

M s gma s o sum mu p e p -s gma ne works In he con ex

K
of kerne me hods Fac or za on Mach nes [158] and he r
<latexit sha1_base64="X1Jlvi+WH1kXuzvyGeTdBmwNrOw=">AAAB6HicbVA9SwNBEJ2LXzF+RS1tFoNgFe5sTBm0EWwSMB+QHGFvM5es2ds7dveEcOQX2FgoYutPsvPfuEmu0MQHA4/3ZpiZFySCa+O6305hY3Nre6e4W9rbPzg8Kh+ftHWcKoYtFotYdQOqUXCJLcONwG6ikEaBwE4wuZ37nSdUmsfywUwT9CM6kjzkjBorNe8H5YpbdRcg68TLSQVyNAblr/4wZmmE0jBBte55bmL8jCrDmcBZqZ9qTCib0BH2LJU0Qu1ni0Nn5MIqQxLGypY0ZKH+nshopPU0CmxnRM1Yr3pz8T+vl5qw5mdcJqlByZaLwlQQE5P512TIFTIjppZQpri9lbAxVZQZm03JhuCtvrxO2ldVz616TbdSv8njKMIZnMMleHANdbiDBrSAAcIzvMKb8+i8OO/Ox7K14OQzp/AHzucPoMWMyw==</latexit>
<latexit sha1_base64="Cu+q0FVT/ACPKbJkVwRmHZdE7oU=">AAAB6HicbZC7SwNBEMbnfMbzFbW0WQyCVbiz0UYM2gg2CZgHJEfY28wla/b2jt09IYSAvY2FIrb+M/Z2/jduHoUmfrDw4/tm2JkJU8G18bxvZ2l5ZXVtPbfhbm5t7+zm9/ZrOskUwypLRKIaIdUouMSq4UZgI1VI41BgPexfj/P6AyrNE3lnBikGMe1KHnFGjbUqt+18wSt6E5FF8GdQuPx0Lx4BoNzOf7U6CctilIYJqnXT91ITDKkynAkcua1MY0pZn3axaVHSGHUwnAw6IsfW6ZAoUfZJQybu744hjbUexKGtjKnp6flsbP6XNTMTnQdDLtPMoGTTj6JMEJOQ8dakwxUyIwYWKFPczkpYjyrKjL2Na4/gz6+8CLXTou8V/YpXKF3BVDk4hCM4AR/OoAQ3UIYqMEB4ghd4de6dZ+fNeZ+WLjmzngP4I+fjBxAMjpg=</latexit>
sha1_base64="+JGHOBgUMtqFjzrGlpGMgb8trkE=">AAAB6HicbZC7SgNBFIbPxltcb1FLm8UgWIVdG23EoI1gk4C5QLKE2cnZZMzs7DIzK4QlT2BjoYitPoy9jfg2Ti6FJv4w8PH/5zDnnCDhTGnX/bZyS8srq2v5dXtjc2t7p7C7V1dxKinWaMxj2QyIQs4E1jTTHJuJRBIFHBvB4GqcN+5RKhaLWz1M0I9IT7CQUaKNVb3pFIpuyZ3IWQRvBsWLD/s8ef+yK53CZ7sb0zRCoSknSrU8N9F+RqRmlOPIbqcKE0IHpIctg4JEqPxsMujIOTJO1wljaZ7QzsT93ZGRSKlhFJjKiOi+ms/G5n9ZK9XhmZ8xkaQaBZ1+FKbc0bEz3trpMolU86EBQiUzszq0TySh2tzGNkfw5ldehPpJyXNLXtUtli9hqjwcwCEcgwenUIZrqEANKCA8wBM8W3fWo/VivU5Lc9asZx/+yHr7AQGbkAw=</latexit>

J U J
h gher-order ex ens on [159] mode bo h ow- and h gher-order
<latexit sha1_base64="gw8Pa1ixMp2vvNZp4PrpWM+AVAM=">AAAB6HicbVA9SwNBEJ2LXzF+RS1tFoNgFe5sTBm0EasEzAckR9jbzCVr9vaO3T0hHPkFNhaK2PqT7Pw3bpIrNPHBwOO9GWbmBYng2rjut1PY2Nza3inulvb2Dw6PyscnbR2nimGLxSJW3YBqFFxiy3AjsJsopFEgsBNMbud+5wmV5rF8MNME/YiOJA85o8ZKzftBueJW3QXIOvFyUoEcjUH5qz+MWRqhNExQrXuemxg/o8pwJnBW6qcaE8omdIQ9SyWNUPvZ4tAZubDKkISxsiUNWai/JzIaaT2NAtsZUTPWq95c/M/rpSas+RmXSWpQsuWiMBXExGT+NRlyhcyIqSWUKW5vJWxMFWXGZlOyIXirL6+T9lXVc6te063Ub/I4inAG53AJHlxDHe6gAS1ggPAMr/DmPDovzrvzsWwtOPnMKfyB8/kDn0GMyg==</latexit>
<latexit sha1_base64="wLxvveqLoBZWl9mm6jK0lxSafvE=">AAAB6HicbZC7SwNBEMbnfMbzFbW0WQyCVbiz0UYM2ohVAuYByRH2NnPJmr29Y3dPCCFgb2OhiK3/jL2d/42bR6GJHyz8+L4ZdmbCVHBtPO/bWVpeWV1bz224m1vbO7v5vf2aTjLFsMoSkahGSDUKLrFquBHYSBXSOBRYD/vX47z+gErzRN6ZQYpBTLuSR5xRY63KbTtf8IreRGQR/BkULj/di0cAKLfzX61OwrIYpWGCat30vdQEQ6oMZwJHbivTmFLWp11sWpQ0Rh0MJ4OOyLF1OiRKlH3SkIn7u2NIY60HcWgrY2p6ej4bm/9lzcxE58GQyzQzKNn0oygTxCRkvDXpcIXMiIEFyhS3sxLWo4oyY2/j2iP48ysvQu206HtFv+IVSlcwVQ4O4QhOwIczKMENlKEKDBCe4AVenXvn2Xlz3qelS86s5wD+yPn4AQ6Ijpc=</latexit>
sha1_base64="Mm4oe/6THoVh9Di/83XU7Ifr3UA=">AAAB6HicbZC7SgNBFIbPxltcb1FLm8UgWIVdG23EoI1YJWAukCxhdnI2GTM7u8zMCmHJE9hYKGKrD2NvI76Nk0uhiT8MfPz/Ocw5J0g4U9p1v63c0vLK6lp+3d7Y3NreKezu1VWcSoo1GvNYNgOikDOBNc00x2YikUQBx0YwuBrnjXuUisXiVg8T9CPSEyxklGhjVW86haJbcidyFsGbQfHiwz5P3r/sSqfw2e7GNI1QaMqJUi3PTbSfEakZ5Tiy26nChNAB6WHLoCARKj+bDDpyjozTdcJYmie0M3F/d2QkUmoYBaYyIrqv5rOx+V/WSnV45mdMJKlGQacfhSl3dOyMt3a6TCLVfGiAUMnMrA7tE0moNrexzRG8+ZUXoX5S8tySV3WL5UuYKg8HcAjH4MEplOEaKlADCggP8ATP1p31aL1Yr9PSnDXr2Yc/st5+AAAXkAs=</latexit>

<latexit sha1_base64="gw8Pa1ixMp2vvNZp4PrpWM+AVAM=">AAAB6HicbVA9SwNBEJ2LXzF+RS1tFoNgFe5sTBm0EasEzAckR9jbzCVr9vaO3T0hHPkFNhaK2PqT7Pw3bpIrNPHBwOO9GWbmBYng2rjut1PY2Nza3inulvb2Dw6PyscnbR2nimGLxSJW3YBqFFxiy3AjsJsopFEgsBNMbud+5wmV5rF8MNME/YiOJA85o8ZKzftBueJW3QXIOvFyUoEcjUH5qz+MWRqhNExQrXuemxg/o8pwJnBW6qcaE8omdIQ9SyWNUPvZ4tAZubDKkISxsiUNWai/JzIaaT2NAtsZUTPWq95c/M/rpSas+RmXSWpQsuWiMBXExGT+NRlyhcyIqSWUKW5vJWxMFWXGZlOyIXirL6+T9lXVc6te063Ub/I4inAG53AJHlxDHe6gAS1ggPAMr/DmPDovzrvzsWwtOPnMKfyB8/kDn0GMyg==</latexit>
<latexit sha1_base64="wLxvveqLoBZWl9mm6jK0lxSafvE=">AAAB6HicbZC7SwNBEMbnfMbzFbW0WQyCVbiz0UYM2ohVAuYByRH2NnPJmr29Y3dPCCFgb2OhiK3/jL2d/42bR6GJHyz8+L4ZdmbCVHBtPO/bWVpeWV1bz224m1vbO7v5vf2aTjLFsMoSkahGSDUKLrFquBHYSBXSOBRYD/vX47z+gErzRN6ZQYpBTLuSR5xRY63KbTtf8IreRGQR/BkULj/di0cAKLfzX61OwrIYpWGCat30vdQEQ6oMZwJHbivTmFLWp11sWpQ0Rh0MJ4OOyLF1OiRKlH3SkIn7u2NIY60HcWgrY2p6ej4bm/9lzcxE58GQyzQzKNn0oygTxCRkvDXpcIXMiIEFyhS3sxLWo4oyY2/j2iP48ysvQu206HtFv+IVSlcwVQ4O4QhOwIczKMENlKEKDBCe4AVenXvn2Xlz3qelS86s5wD+yPn4AQ6Ijpc=</latexit>
sha1_base64="Mm4oe/6THoVh9Di/83XU7Ifr3UA=">AAAB6HicbZC7SgNBFIbPxltcb1FLm8UgWIVdG23EoI1YJWAukCxhdnI2GTM7u8zMCmHJE9hYKGKrD2NvI76Nk0uhiT8MfPz/Ocw5J0g4U9p1v63c0vLK6lp+3d7Y3NreKezu1VWcSoo1GvNYNgOikDOBNc00x2YikUQBx0YwuBrnjXuUisXiVg8T9CPSEyxklGhjVW86haJbcidyFsGbQfHiwz5P3r/sSqfw2e7GNI1QaMqJUi3PTbSfEakZ5Tiy26nChNAB6WHLoCARKj+bDDpyjozTdcJYmie0M3F/d2QkUmoYBaYyIrqv5rOx+V/WSnV45mdMJKlGQacfhSl3dOyMt3a6TCLVfGiAUMnMrA7tE0moNrexzRG8+ZUXoX5S8tySV3WL5UuYKg8HcAjH4MEplOEaKlADCggP8ATP1p31aL1Yr9PSnDXr2Yc/st5+AAAXkAs=</latexit>

R1
<latexit sha1_base64="AoPwX7MC6DZy2+2i9wxHyINtHgI=">AAAB6nicbVC7TsNAEFyHVwgvAyXNiQiJKrJpoIygoQyPPKTEis6XdXLK+WzdnZEiK59AQwFCtHwRHX/DJXEBCSOtNJrZ1e5OmAqujed9O6W19Y3NrfJ2ZWd3b//APTxq6SRTDJssEYnqhFSj4BKbhhuBnVQhjUOB7XB8M/PbT6g0T+SjmaQYxHQoecQZNVZ6uO/7fbfq1bw5yCrxC1KFAo2++9UbJCyLURomqNZd30tNkFNlOBM4rfQyjSllYzrErqWSxqiDfH7qlJxZZUCiRNmShszV3xM5jbWexKHtjKkZ6WVvJv7ndTMTXQU5l2lmULLFoigTxCRk9jcZcIXMiIkllClubyVsRBVlxqZTsSH4yy+vktZFzfdq/p1XrV8XcZThBE7hHHy4hDrcQgOawGAIz/AKb45wXpx352PRWnKKmWP4A+fzB9B9jXY=</latexit>
sha1_base64="KLgvIkTwGb+XA0Z1huQkX3gus00=">AAAB6nicbVA9SwNBEJ2LXzF+RQUbm8UgWIU7Gy2DNpaJmg9Ijri3mUuW7O0du3tCOPITbCwUsfUX2dn4W9x8FJr4YODx3gwz84JEcG1c98vJrayurW/kNwtb2zu7e8X9g4aOU8WwzmIRq1ZANQousW64EdhKFNIoENgMhtcTv/mISvNY3ptRgn5E+5KHnFFjpbvbrtctltyyOwVZJt6clCpHte8HAKh2i5+dXszSCKVhgmrd9tzE+BlVhjOB40In1ZhQNqR9bFsqaYTaz6anjsmpVXokjJUtachU/T2R0UjrURTYzoiagV70JuJ/Xjs14aWfcZmkBiWbLQpTQUxMJn+THlfIjBhZQpni9lbCBlRRZmw6BRuCt/jyMmmclz237NVsGlcwQx6O4QTOwIMLqMANVKEODPrwBC/w6gjn2Xlz3metOWc+cwh/4Hz8AF8kj1s=</latexit>
sha1_base64="R72cnmglGFVXLhxPb5adsQXXvW0=">AAAB6nicbVC7SgNBFL0bXzG+ooKNzWAQrMKujZYhNpaJmgckS5id3CRDZmeXmVkhLPkEGwtFbG39C7/AzsZvcfIoNPHAhcM593LvPUEsuDau++VkVlbX1jeym7mt7Z3dvfz+QV1HiWJYY5GIVDOgGgWXWDPcCGzGCmkYCGwEw6uJ37hHpXkk78woRj+kfcl7nFFjpdubjtfJF9yiOwVZJt6cFEpH1W/+Xv6odPKf7W7EkhClYYJq3fLc2PgpVYYzgeNcO9EYUzakfWxZKmmI2k+np47JqVW6pBcpW9KQqfp7IqWh1qMwsJ0hNQO96E3E/7xWYnqXfsplnBiUbLaolwhiIjL5m3S5QmbEyBLKFLe3EjagijJj08nZELzFl5dJ/bzouUWvatMowwxZOIYTOAMPLqAE11CBGjDowwM8wbMjnEfnxXmdtWac+cwh/IHz9gOwZZEX</latexit>

F g 16 Tensor Dropout on a Tucker decompos t on V su- n erac ons be ween var ab es us ng fac or zed parame ers
a za on of ensor dropou app ed o a Tucker decompos on v a ma r x ensor decompos ons By emp oy ng fac or zed
Severa ypes of random za on can be used Here a Bernou parame ers such mode s are ab e o es ma e h gher-order
dropou s represen ed app ed n he a en subspace of he n erac ons even n asks where da a are sparse (e g recom-
decompos on by con rac ng he core a ong each mode mender sys ems) un ke o her kerne me hods such as SVMs
w h a ser es of (d agona ) Bernou ske ch ng ma r ces More recen y deeper neura ne works have been proposed
M(k) = d ag(λ(k) ) o cap ure mu p ca ve n erac ons [160] [161] For n-
s ance H ghway ne works [160] n roduce an arch ec ure
ha enab es second-order n erac ons be ween he npu s In
[162] he au hors augmen he or g na convo u on w h a
mu p ca ve n erac on o ncrease represen a ona power
s a s cs I cons s s of app y ng ske ch ng ma r ces n he
In [161] he au hors group severa such mu p ca ve n erac-
a en subspace spanned by a ensor decompos on whe her
ons as a second-order po ynom a and prove ha he second-
ha ’s CP (see F g 15) or Tucker (see F g 16) ead ng o
order po ynom a s ex end he c ass of func ons ha can be
s gn fican y mproved robus ness o random no se (e g dur ng
represen ed w h zero error
cap ure) or adversar a a acks
Recen emp r ca [163] [164] [165] and heore ca [161]
2) De ens ve Tensor za on: By comb n ng he resu s nd ca e ha mu var a e h gher-order po ynom a s
reparame r za on approach descr bed n sec on V-D5 w h expand he c asses of func ons ha can be approx ma ed
ensor dropou one can dras ca y mprove he robus ness w h deep neura ne works Approx ma ng a func on us ng
of a DCNN o adversar a a acks When comb ned w h a mu var a e h gher-order po ynom a nvo ves earn ng he
adversar a ra n ng h s approach proved successfu aga ns mu var a e po ynom a expans on parame ers us ng npu -
a w de range of a acks [151] ou pu ra n ng da a H gher-order ensors na ura y repre-
sen he parame ers of such a mu var a e po ynom a [166]
3) Tensor-Sh e d: In add on o he ne of research ha whose d mens ona y ncreases exponen a y o he order of
cons s s n mod fy ng he we gh s or ac va ons of he ne work he po ynom a approx ma on Th s s ano her ns ance of
o ncrease robus ness o adversar a a acks s poss b e o he curse of d mens ona y ha ensor decompos ons can
mod fy he npu magee d rec y For ns ance Sh e d [152] m ga e Moreover n h s se ng choos ng d fferen ensor
uses a JPEG encod ng o remove any po en a adversar a decompos ons resu s n d fferen recurs ve equa ons ha
no se However he defens ve capac y of h s approach s compose he po ynom a func on approx ma on and are hence
m ed Ins ead of a JPEG encod ng n [153] ensor decom- nked o d fferen neura ne work arch ec ures
pos on echn ques are u zed as a preprocess ng s ep o find Par cu ar y e z ∈ Rd be he npu e g he a en
a ow-rank approx ma on of mages wh ch can s gn fican y no se n a genera ve mode or an mage n a d scr m na ve
d scard h gh-frequency per urba ons mode and e x ∈ Ro be he arge e g an ou pu mage
PROCEEDINGS OF THE IEEE 17

or a class-label. A polynomial expansion6 of the input z polynomials has recently been studied using tensor decom-
can be considered for approximating the target x. That is, positions [172]. The work of [173] demonstrates that the
a vector-valued function G(z) : Rd → Ro expresses the aforementioned factorization of multivariate polynomials is
high-order multivariate polynomial expansion: related to the joint CP decomposition.
In deep polynomial networks, instead of using a single
N 
X n+1
Y  polynomial, also products of polynomials can be used [164].
x = G(z) = W [n] ×j z + β, (23) That is, the higher-order multivariate polynomial function is
n=1 j=2 expressed as a product of low-order multivariate polynomials.
Qn N
This approach enforces variable separability in two levels.
where β ∈ Ro and W [n] ∈ Ro× m=1 ×m d n=1 are the

First, in the parameters of low-order polynomials via tensor
learnable parameters. The form of (23) can approximate decomposition as aforementioned, and second by allowing ap-
any smooth function (for large N ). As aforementioned the proximation as a product of simpler functions. The product of
number of parameters required to accommodate all higher- polynomials increases the expansion order without increasing
order interactions of the input increases exponentially with the the number of layers significantly.
desired order of the polynomial which is impractical for high-
dimensional data. To make this model practical, its parameters
can be reduced by making a) low-rank assumptions, and b) U[2] U[3] C β
sharing factors between the decompositions of each parameter
tensor, which accounts to jointly factorize all the parameter ∗ ∗
z U[1] + ∗ + + G(z)
tensors W [n] . To illustrate this, a certain form of CP decom-
position with shared factors among low-rank decompositions Fig. 17: Schematic illustration of the polynomial expansion
of weight tensors [164] is selected. Concretely, the parameter for third order approximation.
tensor W [n] is factorized using n factor matrices. The first and
the last factor matrices (called U[1] and C respectively) are Furthermore, the original attention mechanism [174] is
shared across all parameter tensors. Then, depending on the focused on neural machine translation, yet it expresses mul-
interactions between the layers we want to forge, we can share tiplicative interactions. The idea is to capture long-range
the corresponding factor matrices. That results (see [164], dependencies that might exist in a temporal sequence. This
[169] for detailed derivation) in a simple recursive relationship, was later extended to attention mechanisms for image and
that can be expressed as: video data [175]. Multi-head attention learns different matrices
per head; concatenating those matrices, a third-order tensor
 
T
xn = U[n] z ∗ xn−1 + xn−1 , (24)
is obtained. The authors utilize a CP decomposition to both
for n = 2, . . . , N with x1 = U[1]T
z and x = CxN + β. The reduce the parameters and reduce the redundancy in the
o×k learned matrices [176]. In [177], where the authors extend
parameters C ∈ R , U[n] ∈ Rd×k for n = 1, . . . , N are
self-attention to capture not only inter- but also intra-channel
learnable with k ∈ N the rank of the CP decomposition.
correlations by constructing an appropriate tensor such that the
Each recursive call in (24) expands the approximation order
variance of spatial-and channel-wise features is regularized.
by one. Then, the final output x is a polynomial of N th order.
The recursive relationship of (24) enables us to implement
VI. A PPLICATIONS IN C OMPUTER V ISION
the polynomial using a hierarchical neural network, where
the approximation order defines the depth of the network. In this section, we firstly discuss applications related to
A schematic assuming a third-order expansion (N = 3) is extracting multiple latent factors of variation from visual
illustrated in Fig. 17. The tensor decomposition selected and data (Section VI-A). An overview of applications that focus
the sharing of the factors have a crucial role in the resulting on feature extraction with multilinear subspace learning is
architecture. As aforementioned, by selecting different decom- presented in Section VI-B, while applications in inverse
positions, new architectures can emerge [164], while additional problems such as denoising and completion are presented in
constraints can be placed on the structure, e.g., shift invariance Section VI-C. In Section VI-D, tensor regression applications
by allowing factor matrices to implement convolutions (e.g., in visual classification are discussed. Finally, in Section VI-E,
via im2col operator or a circulant operator). some recent applications of tensor-parametrized deep learning
Higher-order polynomials can be represented as a product of architectures are briefly presented
lower-order polynomials, which have been extensively studied
for both univariate and multivariate cases. Berlekamp’s algo- A. Multiple factors analysis in computer vision
rithm [170] was one of the first for successfully factoring uni- Notebook: [Link]
variate polynomials over finite fields; the Cantor–Zassenhaus
algorithm [171] has since become more popular. Factorizing Extracting multiple hidden factors of variation from visual
multivariate polynomials as a linear combination of univariate data is a first fundamental step in a large number of computer
6 The Stone-Weierstrass theorem [167] guarantees that any smooth function
vision applications. Consider the problem of view- and pose-
can be approximated by a polynomial. Multivariate function approximation is invariant visual object classification, where one needs to clas-
covered by an extension of the Weierstrass theorem, e.g., in [168] (pg 19). sify visual objects captured across different perspectives and
PROCEEDINGS OF THE IEEE 18

poses. To achieve this, it is required to design or learn data rep- assumed to be formed by the entanglement of M −1 different,
resentations that are invariant to view and pose changes, and at yet unknown sources of variation as:
the same time, capture discriminant information regarding the (2) (u) (M )
object’s category. Structure from motion (which aims to esti- xi = B ×2 ui ×3 ai ×4 · · · ×m ui . (26)
mate 3D structures from two-dimensional image sequences) (m)
Each factor ui accounts for a source of variability in the
is another computer vision task where disentanglement of
data, while the multiplicative interactions of these factors
multiple factors is required. More specifically, one needs to
through tensor B ∈ Rd×K2 ×···×KM emulate the entangled
disentangle shape and motion factors and subsequently exploit
variability giving rise to the rich structure of image xi . For
them in reconstructing 3D visual objects. Face editing [178],
a set of images represented by the observation matrix X, the
[179] and talking faces synthesis [180] are further problems
above model is written in matrix form as [10], [198]:
requiring the manipulation of multiple latent variables.
X = B(1) (U(2) U(3) · · · U(M ) ), (27)
Multiple factors analysis via tensor methods is achieved
using two different approaches. The first and historically where B(1) Rd×K2 ·K3 ···KM is the mode-1 unfolding of B and
oldest approach flattens visual data to vectors, which are then {U (m) }M m=2 ∈ R
Km ×N
gathers the variation coefficients for
arranged into a tensor according to the factors involved in all images across M − 1 modes of variation. Observe that this
their formation to prepare the data for subsequent analysis. model is different from the Tucker decomposition and does
The second, more recent approach assumes a tensor-structured not require labels for the data tensor formation, in contrast
latent space in deep autoencoder-based architectures [181]). to TensorFaces. To find the unknown latent factors and the
The TensorFaces [182] is probably the most popular method mixing operator, the reconstruction error kX − B(1) (U(2)
in the first category and has been widely employed to ana- U(3) · · · U(M ) )k2F is minimized. Having found the disen-
lyze visual data of known sources of variation. For instance tangled modes of variation, they can subsequently be used to
PIE [183] and Multi-PIE [184] databases contain a number of drive shape from shading, face editing (expression transfer),
people (i.e., multiple identities) captured under different poses and estimation of surface normals [10], [198] and viewpoint
and illuminations, displaying a variety of facial expressions. and illumination invariant face recognition [199].
Likewise, the Weizmann face dataset [185] consists of 28 In the second family of methods, the multilinear structure is
subjects in 5 viewpoints, 4 illuminations, and 3 expressions. imposed on the latent space of deep autoencoder-based mod-
Following the TensorFaces framework, such datasets are rep- els. More specifically, [178], [179], [202], employ CNN-based
resented by a tensor with order equal to the number of sources encoders to obtain a set of M − 1 latent vector representations
of variation, and vectorized images are arranged into its fibers (m)
{ui }M m=2 for each image using label information. Each
according to the labels that describe the sources of variation. factor accounts for a source of variability in the data, while
The Weizmann dataset, for example, would represented by a the multiplicative interactions of these factors emulate the
28 × 5 × 4 × 3 × 7943 tensor X , where each image consists of entangled variability using (26). The entangled latent variable
7943 pixels. TensorFaces uncover the latent factors of variation is fed into a decoder network that reconstructs each image. In
by seeking a decompostion of the form: this setting, the autoencoder parameters and the tensor B are
X = B ×1 Upeople ×2 Uviews ×3 Uillumns ×4 Uexpres ×5 Upixels , trained end-to-end. Such models can be used then for image
(25) editing (e.g., face editing) via manipulating the latent factors
of variation. A similar parameterization of the latent space of
where matrices U capture different latent factors of variation conditional deep variational autoencoders has been employed
and B corresponds to multilinear mixing tensor that entangles in [203] allowing to generate images formed by combinations
the factors of variation. Such factor matrices are employed of sources of variation that have not been observed in training
next for extraction of invariant features. Similar analysis has data.
also been done in analysis of motion signatures [186] and gait
sequences [187], disentangling pose and body shape [188],
[189], 3D head pose estimation [190], face transfer [191], B. Dimensionality reduction via tensor component analysis
appearance-based tracking [192], [193], structure from motion, Tensor component analysis such as tensor extensions
and optical flow [194], [195]. A compositional variant of of PCA and LDA (cf. Section IV-D) and decompositions
TensorFaces has also been proposed [196]. TensorFaces and have found a surge of applications in dimensionality
related approaches [197], [196] are supervised learning models reduction and features extraction from tensor-valued visual
and hence suitable for analysis of visual data collected in data. Subsequently, the extracted features are utilized for
tightly controlled conditions. downstream tasks such as classification, using learning
Unsupervised variants of TensorFaces have been proposed models such as nearest-neighbors, SVMs, or neural networks.
in [10], [198],[199] (extending [200] and [201]). Such methods In human sensing applications, tensor component analysis
uncover multiple latent factors of variation from vector data methods are applied directly on face or body silhouette images,
samples in an unsupervised setting (i.e., without the presence and the extracted features are used for face recognition, and
of labels). It is only assumed that the number of sources body/gait analysis [105], [109], [204], [205], [206], [207].
of variation is known. More specifically, let X ∈ Rd×N Besides modeling variation on texture, tensor component
be a matrix of observations, where each of the N columns analysis models, such as the HOSVD, have been used to build
represents a vectorized image of d pixels. Each image is statistical shape models in the context of Active Appearance
PROCEEDINGS OF THE IEEE 19

Models (AAM) [208], [209] instead of the linear model in [128] and applied for crowd counting on image datasets.
typically used [210]. Furthermore, the Tucker decomposition In [226], tensor logistic regression is employed towards land-
has been used for image compression [211], [212]. cover classification from hyperspectral data. It effectively
In the domain of hyperspectral image analysis, tensor com- reduces the number of parameters compared toQ methods
N
ponent analysis methods can be used to reduce the statistical that treat hyperspectral images as vectors from C i=1 Ii to
PN
redundancy resulting from high interband correlations [213]. C i=1 Ii , where C indicates the number of classes and N
For instance, in [214], discriminant features are extracted for is the order of the tensor describing the data. Classification
classification by applying the tensor discriminative locality of fMRI images may relate to several target variables, for
alignment (TDLA) method onto a data tensor consisting of example, detecting and identifying disease and brain damage
hyperspectral image patches. and human brain activity that can be associated with stimuli
Variants of tensor networks and decompositions have been context. A variety of tensor methods has been used to this
further employed for tasks such as tumor and legion detection end, such as tensor extensions to SVM [227] and regularized
from visual data [215], [216], showing benefits both in terms low-rank tensor regression models [119], [122], [120].
of performance and computational cost. Unsupervised coupled
matrix-tensor factorization methods have also been proposed E. Deep networks parametrization using tensor decomposition
towards identifying coherent brain regions across subjects that
activate when similar stimuli are provided [217]. The vast majority of the deep learning models discussed
in Section V have been introduced in the context of image
classification, and their effectiveness have been studied in
C. Tensor methods in inverse problems large-scale standard benchmark datasets such as ImageNet.
Inverse problems in imaging span a wide range of tasks, Neural network compression using CP decomposition has also
such as image restoration via denoising and completion of been applied on land-cover classification [226]. Furthermore,
missing visual data (imputation). By representing visual as CNNs with tensor regression layers have also been used for
tensors, models addressing inverse problems exploit low-rank various applications such as affect estimation from faces [228],
tensor decompositions (Section IV-B), or rely on robust [27]. Randomized TRL has been employed in age prediction
variants (Sections IV-C IV-E) to estimate a low-rank tensor from MRI scans [33], significantly improving the results with
which forms a basis from which missing data can be imputed respect to state-of-the-art algorithms.
or denoised. Along this line of research, several methods Mobilenet V1 [229] and Mobilenet V2 [230] are seminal
for image and video inpainting and/or denoising have been light-weight architectures allowing mobile and embedded vi-
developed by estimating the low-rank tensor using either sion applications. Interestingly, these architectures can be in-
nuclear norm-based regularizers, or by assuming an explicit terpreted as special cases of Kruskal and Tucker convolutions
low-rank tensor structure described by decompositions such described in Section V-D. The interested reader is referred
CP, Tucker, Tensor Ring, or TT decomposition [218], [219], to [27], [141] for details.
[220], [221], [92], [99], [100], [101], [98]. For the same Parametrizing one or several layers with a low-rank tensor
task, methods based on tensor-structured separable dictionary factorization has several advantages, namely robustness to
learning (Section IV-E) have also been proposed [4], [115], noise, regularizing effect, computational speedups, and re-
[11]. In the context of hyperspectral image analysis, data are duction in the number of parameters. Another advantage is
corrupted with more complex noise distributions specific to conceptual: the structure induced by the factorization can be
the domain, such as impulse noise and stripings. To denoise, used to adapt a model to different domains or efficiently share
such data, regularized variants of the CP decomposition information between tasks. For instance, when applying a
have been employed [222], [223], [224]. Experimental results Tucker decomposition to a tensor, we obtain a latent subspace
indicate increased stability and accuracy compared to methods represented by the core tensor. We can project to and from
that do not exploit the tensor structure of the data. that space using the factors of the decomposition, which
In [225], several extensions of multilinear component anal- are typically projection matrices. This enables various ap-
ysis methods are presented, emphasizing the discovery of plications, including cross-task and cross-domain incremental
common features shared by multiple data views - such as learning, typically by allowing weight sharing through joint
common and individual feature analysis, with applications in factorization. [28] propose weight sharing in multitask learning
MRI denoising and completion. and [29] propose sharing residual units. Most recently, [30]
proposed to learn a task agnostic core, which is shared between
D. Tensor regression for visual classification tasks and domains. This shared subspace can be specialized to
Low-rank tensorregression models are well-suited for the new task or domain by learning a new set of task-specific
classification of large-scale visual data, such as face and body projection matrices.
images and videos, as well as hyperspectral and medical
images. In [5], low-rank tensor regression models with VII. C HALLENGES AND P RACTITIONER ’ S G UIDE
automatic rank selection are applied on problems such as face Tensor methods are handy tools for learning from multi-
classification and pose estimation, while in [123], a multilinear dimensional data and a convenient framework for infusing
multitask learning framework is used for pain estimation from more structure in deep neural networks, leveraging redundancy
videos. Low-rank quantile tensor regression is also utilized to obtain more compact models, and increasing robustness.
PROCEEDINGS OF THE IEEE 20

However, some challenges may currently prevent their wider on the tensor structure, and therefore the two problems are
adoption, especially in the context of deep learning. This intertwined. However, there is currently a lack of theory to
section highlights such significant challenges and guides prac- guide optimal choices and enable efficient search over model
titioners on how to tackle them in practice. architectures. Furthermore, auto-tensorization is of particular
Rank selection. Most tensor methods discussed in this interest. In other words, given a (pre-trained) existing network
paper require knowledge or estimation of the rank of the architecture, an interesting question is how to automatically
decomposition. However, determining the rank of a tensor choose the decomposition and its rank to approximate the
is, in general, NP-hard [78], [79] and usually heuristics are network in a compressed form optimally.
employed in practice. While the choice of exact rank within Training tensorized deep neural networks. Training ten-
a deep framework is less crucial as all the parameters are sorized deep neural networks remains a challenging task due
trained end-to-end jointly; rank selection remains a significant to mainly numerical issues. For instance, composing several
challenge. tensor contractions can lead to instability, with the gradient
In practice, one can start with selecting a rank such that vanishing or exploding. This is particularly true when using
90% of the parameters are preserved and incrementally reduce lower precisions (e.g., int8) motivating the need for principled
the rank, e.g., through multi-stage compression [145]. When ways to initialize the factors of tensor decompositions, and
starting from a pre-trained networks, it is also possible to the need for better normalization methods that can take into
approximate the rank by applying Bayesian matrix factoriza- account the structure.
tion [231] to the unfolding of the weight tensor to get an In practice, using full precision (i.e., ‘float64’) can alleviate
approximation of the rank [140], [145]. Alternatively, a Lasso- most of such numerical issues. However, in order to speed up
type penalty can be added to the loss function to automatically learning and inference, modern deep learning systems often
set some of the components to zero [27]. rely on reduced precision, e.g., float32, or even adopt quanti-
Which decomposition to use? Another significant open zation of the deep models to int16, int8, or just binary, which
question in tensor methods is the choice of decomposition. in turn can cause numerical stability issues. To enable robust
While there are heuristics developed through intuition and and stable learning in such settings, one can use automatic
experimental work, there is no principled way to choose the mixed-precision software tools (available for instance through
most appropriate decomposition. NVIDIA’s Amp). Automatic mixed-precision relies on two
In practice, as a rule of thumb, the Tucker decomposition parts: using reduced precision when this can be done without
is well suited for subspace learning as it allows to project to affecting performance and automatic loss scaling to preserve
and from the latent subspace represented by its core. The CP small gradient values. On the hardware side, better handling of
decomposition is well suited for learning interpretable latent mathematical operations allows for striking a balance between
factors; however, it is less suited for decomposing tensors speed and precision, for example by using tensor floats. Tensor
which have some very large modes and some smaller ones, floats on 32 bits (TF32), for instance, use the same mantissa
since the same rank is used for all dimensions. The Tensor- as half-precision (Float16) but still use an 8-bit exponent to
Train decomposition is best when the goal is to achieve very support a larger numerical range. Alternatively, it is possible to
high compression ratios, as it disentangles the dimensions. In perform the calculations in log-space. While such an approach
fact, it may be desirable to jointly model dimensions if we improves numerical stability, it limits the range to positive
want to preserve the correlations between them [21]. values only.
Similar intuition applies to the choice of tensor decom- Finally, it should be noted that a careful initialization of
position in the context of deep learning, albeit with some the factors of the decomposition is crucial to ensure that the
additional considerations. A CP factorization can be seen as values of the reconstruction are in the expected range. It is
a MobileNet-v2 block, where the rank is equal to the product well known that deep convolutional neural networks need to
of the expansion factor with the number of input channels. be carefully initialized, e.g., using the He [233] or Glorot [234]
This means that for achieving speedups, one should choose initialization. These typically use a Gaussian distribution with
the rank accordingly, e.g., to be a power of 2. CP is a a zero-mean and small variance. It is possible to perform
beneficial choice when combined with transduction to train, variance analysis and initialize the factors directly, so the
e.g., on a static domain such as images, and then generalize to reconstruction has zero-mean and a given variance.
videos [27] during inference. A Tucker convolution gives more
flexibility by allowing each mode to have a different rank, but R EFERENCES
the rank chosen along the input and output channels should
also be chosen with speed in mind. For efficient learning of [1] S. Montangero, S. Montangero, and Evenson, Introduction to Tensor
Network Methods. Springer, 2018.
convolutional LSTMs in higher-orders, convolutional tensor- [2] J. M. Landsberg, “Tensors: geometry and applications,” Representation
train decompositions have proven successful [232]. theory, vol. 381, no. 402, p. 3, 2012.
As we established in Section V, there is an equivalence [3] H. Lu, K. N. Plataniotis, and A. Venetsanopoulos, Multilinear Subspace
Learning: Dimensionality Reduction of Multidimensional Data, 1st ed.
between choosing the tensor decomposition applied to con- Chapman & Hall/CRC, 2013.
volutional kernels and choosing an efficient convolutional [4] M. Bahri, Y. Panagakis, and S. Zafeiriou, “Robust Kronecker compo-
block, and an equivalence between finding the decomposition’s nent analysis,” IEEE Trans. Pattern Anal. Mach. Intell., vol. 41, pp.
2365–2379, 2019.
rank and finding the block’s parameters. Clearly, choosing [5] W. Guo, I. Kotsia, and I. Patras, “Tensor learning for regression,” IEEE
the appropriate tensor decomposition and its rank depends Trans. Image Process., vol. 21, no. 2, pp. 816–827, Feb 2012.
PROCEEDINGS OF THE IEEE 21

[6] A. Anandkumar, R. Ge, D. Hsu, S. M. Kakade, and M. Telgarsky, [31] I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley,
“Tensor decompositions for learning latent variable models,” J. Mach. S. Ozair, A. Courville, and Y. Bengio, “Generative adversarial nets,”
Learn. Res., vol. 15, pp. 2773–2832, 2014. in Advances in Neural Inf. Process. Syst. (NeurIPS), 2014.
[7] R. Bellman, Adaptive Control Processes. Princeton University Press, [32] A. Madry, A. Makelov, L. Schmidt, D. Tsipras, and A. Vladu, “Towards
1961. deep learning models resistant to adversarial attacks,” in Proc. Int. Conf.
[8] H. Lu, K. N. Plataniotis, and A. N. Venetsanopoulos, “A survey Learn. Representations (ICLR), 2018.
of multilinear subspace learning for tensor data,” Pattern Recognit., [33] A. Kolbeinsson, J. Kossaifi, Y. Panagakis, A. Anandkumar, I. Tzoulaki,
vol. 44, no. 7, pp. 1540–1551, 2011. and P. Matthews, “Tensor dropout for robust learning,” CoRR, vol.
[9] A. Shashua and T. Hazan, “Non-negative tensor factorization with abs/1902.10758, 2020.
applications to statistics and computer vision,” in Proc. Int. Conf. Mach. [34] T. G. Kolda and B. W. Bader, “Tensor decompositions and applica-
Learn. (ICML), 2005, pp. 792–799. tions,” SIAM Rev., vol. 51, no. 3, pp. 455–500, 2009.
[10] M. Wang, Y. Panagakis, P. Snape, and S. P. Zafeiriou, “Disentangling [35] P. Comon, “Tensors: a brief introduction,” IEEE Signal Process. Mag.,
the modes of variation in unlabelled data,” IEEE Trans. Pattern Anal. vol. 31, pp. 44–53, 05 2014.
Mach. Intell., vol. 40, no. 11, pp. 2682–2695, 2018. [36] N. D. Sidiropoulos, L. De Lathauwer, X. Fu, K. Huang, E. E. Papalex-
[11] S. Hawe, M. Seibert, and M. Kleinsteuber, “Separable dictionary akis, and C. Faloutsos, “Tensor decomposition for signal processing
learning,” Proc. IEEE Conf. Comput. Vis. Pattern Recognit. (CVPR), and machine learning,” IEEE Trans. Signal Process., vol. 65, no. 13,
Jun 2013. pp. 3551–3582, 2017.
[12] Y. LeCun, Y. Bengio, and G. Hinton, “Deep learning,” Nature, vol. [37] A. Cichocki, D. Mandic, L. De Lathauwer, G. Zhou, Q. Zhao, C. Ca-
521, no. 7553, p. 436–444, May 2015. iafa, and H. A. PHAN, “Tensor decompositions for signal processing
[13] Y. LeCun, Y. Bengio et al., “Convolutional networks for images, applications: From two-way to multiway component analysis,” IEEE
speech, and time series,” The Handbook of Brain Theory and Neural Signal Process. Mag., vol. 32, no. 2, pp. 145–163, 2015.
Netw., vol. 3361, no. 10, p. 1995, 1995. [38] E. E. Papalexakis, C. Faloutsos, and N. D. Sidiropoulos, “Tensors
[14] A. Krizhevsky, I. Sutskever, and G. E. Hinton, “ImageNet classification for data mining and data fusion: Models, applications, and scalable
with deep convolutional neural networks,” in Advances in Neural Inf. algorithms,” ACM Trans. Intell. Syst. and Technol. (TIST), vol. 8, no. 2,
Process. Syst. (NeurIPS), 2012, pp. 1097–1105. pp. 1–44, 2016.
[15] K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for [39] A. Smilde, R. Bro, and P. Geladi, “Multi-way analysis with applications
image recognition,” in Proc. IEEE Conf. Comput. Vis. Pattern Recognit. in the chemical sciences,” J. Chemometrics, vol. 19, no. 2, pp. 119–120,
(CVPR), 2016. 2005.
[16] Y. LeCun, L. Bottou, Y. Bengio, and P. Haffner, “Gradient-based [40] E. Acar and B. Yener, “Unsupervised multiway data analysis: A
learning applied to document recognition,” Proc. IEEE, vol. 86, no. 11, literature survey,” IEEE Trans. Know. Data Eng., vol. 21, no. 1, pp.
pp. 2278–2324, 1998. 6–20, 2008.
[17] S. S. Du and J. D. Lee, “On the power of over-parametrization in neural
[41] A. Cichocki, R. Zdunek, A. H. Phan, and S.-i. Amari, Nonnegative
networks with quadratic activation,” in Proc. Int. Conf. Mach. Learn.
matrix and tensor factorizations: applications to exploratory multi-way
(ICML), 2018.
data analysis and blind source separation. John Wiley & Sons, 2009.
[18] M. Soltanolkotabi, A. Javanmard, and J. D. Lee, “Theoretical insights
[42] M. Janzamin, R. Ge, J. Kossaifi, A. Anandkumar et al., “Spectral
into the optimization landscape of over-parameterized shallow neural
learning on matrices and tensors,” Found. and Trends® in Mach.
networks,” IEEE Trans. Inf. Theory, 2018.
Learn., vol. 12, no. 5-6, pp. 393–536, 2019.
[19] R. Caruana, S. Lawrence, and L. Giles, “Overfitting in neural nets:
[43] H. A. Kiers, “Towards a standardized notation and terminology in
Backpropagation, conjugate gradient, and early stopping,” in Advances
multiway analysis,” J. Chemometrics, vol. 14, no. 3, pp. 105–122, 2000.
in Neural Inf. Process. Syst. (NeurIPS), 2001.
[20] H. Mhaskar, Q. Liao, and T. Poggio, “Learning functions: when is deep [44] J. Kossaifi, Y. Panagakis, A. Anandkumar, and M. Pantic, “Tensorly:
better than shallow,” arXiv preprint arXiv:1603.00988, 2016. Tensor learning in python,” J. Mach. Learn. Res., vol. 20, no. 26, pp.
[21] A. Novikov, D. Podoprikhin, A. Osokin, and D. Vetrov, “Tensorizing 1–6, 2019.
neural networks,” in Advances in Neural Inf. Process. Syst. (NeurIPS), [45] A. Paszke, S. Gross, S. Chintala, G. Chanan, E. Yang, Z. DeVito,
2015, pp. 442–450. Z. Lin, A. Desmaison, L. Antiga, and A. Lerer, “Automatic differenti-
[22] V. Lebedev, Y. Ganin, M. Rakhuba, I. V. Oseledets, and V. S. Lem- ation in PyTorch,” in Advances in Neural Inf. Process. Syst. (NeurIPS),
pitsky, “Speeding-up convolutional neural networks using fine-tuned 2017.
CP-decomposition,” in Proc. Int. Conf. Learn. Representations (ICLR), [46] D. L. Donoho, “For most large underdetermined systems of linear
2015. equations the minimal L1-norm solution is also the sparsest solution,”
[23] M. Astrid and S. Lee, “CP-decomposition with tensor power Commun. on Pure and Appl. Math., vol. 59, no. 6, pp. 797–829, 2006.
method for convolutional neural networks compression,” CoRR, vol. [47] F. Maryam, “Matrix rank minimization with applications,” Ph.D. dis-
abs/1701.07148, 2017. sertation, Stanford University, 2002.
[24] J. Kossaifi, A. Khanna, Z. Lipton, T. Furlanello, and A. Anandkumar, [48] R. Raina, A. Madhavan, and A. Y. Ng, “Large-scale deep unsupervised
“Tensor contraction layers for parsimonious deep nets,” in Proc. IEEE learning using graphics processors,” in Proc. Int. Conf. Mach. Learn.
Conf. Comput. Vis. Pattern Recognit. (CVPR), 2017, pp. 1940–1946. (ICML). New York, NY, USA: Assoc. for Comput. Machinery, 2009,
[25] J. Kossaifi, Z. C. Lipton, A. Kolbeinsson, A. Khanna, T. Furlanello, p. 873–880.
and A. Anandkumar, “Tensor regression networks,” J. Mach. Learn. [49] D. C. Cireşan, U. Meier, J. Masci, L. M. Gambardella, and J. Schmid-
Res., vol. 21, no. 123, pp. 1–21, 2020. huber, “Flexible, high performance convolutional neural networks for
[26] J. Kossaifi, A. Bulat, G. Tzimiropoulos, and M. Pantic, “T-Net: image classification,” in Proc. Twenty-Second Int. Joint Conf. Artif.
Parametrizing fully convolutional nets with a single high-order tensor,” Intell., ser. IJCAI’11. AAAI Press, 2011, p. 1237–1242.
in Proc. IEEE Conf. Comput. Vis. Pattern Recognit. (CVPR), 2019, pp. [50] Y. Shi, U. N. Niranjan, A. Anandkumar, and C. Cecka, “Tensor
7814–7823. contractions with extended BLAS kernels on CPU and GPU,” in IEEE
[27] J. Kossaifi, A. Toisoul, A. Bulat, Y. Panagakis, T. M. Hospedales, Int. Conf. High Perform. Comput. (HiPC), Dec 2016, pp. 193–202.
and M. Pantic, “Factorized higher-order CNNs with an application to [51] P. Springer and M. Hoehnerbach. (2019) cuTENSOR: A high-
spatio-temporal emotion estimation,” in Proc. IEEE Conf. Comput. Vis. performance CUDA library for tensor primitives. [Online]. Available:
Pattern Recognit. (CVPR), 2020, pp. 6060–6069. [Link]
[28] Y. Yang and T. M. Hospedales, “Deep multi-task representation [52] M. Abadi, A. Agarwal, P. Barham, E. Brevdo, Z. Chen, C. Citro, G. S.
learning: A tensor factorisation approach,” in Proc. Int. Conf. Learn. Corrado, A. Davis, J. Dean, M. Devin, S. Ghemawat, I. Goodfellow,
Representations (ICLR), 2017. A. Harp, G. Irving, M. Isard, Y. Jia, R. Jozefowicz, L. Kaiser, M. Kud-
[29] Y. Chen, X. Jin, B. Kang, J. Feng, and S. Yan, “Sharing residual lur, J. Levenberg, D. Mané, R. Monga, S. Moore, D. Murray, C. Olah,
units through collective tensor factorization to improve deep neural M. Schuster, J. Shlens, B. Steiner, I. Sutskever, K. Talwar, P. Tucker,
networks,” in Proc. of the Twenty-Seventh Int. Joint Conf. on Artif. V. Vanhoucke, V. Vasudevan, F. Viégas, O. Vinyals, P. Warden, M. Wat-
Intell., IJCAI-18. Int. Joint Conf. on Artif. Intell. Org., 7 2018, pp. tenberg, M. Wicke, Y. Yu, and X. Zheng, “TensorFlow: Large-scale
635–641. machine learning on heterogeneous systems,” 2015, software available
[30] A. Bulat, J. Kossaifi, G. Tzimiropoulos, and M. Pantic, “Incremental from [Link].
multi-domain learning with network latent tensor factorization,” Proc. [53] T. Chen, M. Li, Y. Li, M. Lin, N. Wang, M. Wang, T. Xiao, B. Xu,
AAAI Conf. Artif. Intell. (AAAI), 2020. C. Zhang, and Z. Zhang, “MXNet: A flexible and efficient machine
PROCEEDINGS OF THE IEEE 22

learning library for heterogeneous distributed systems,” CoRR, vol. [81] P. Rai, Y. Wang, S. Guo, G. Chen, D. Dunson, and L. Carin, “Scalable
abs/1512.01274, 2015. bayesian low-rank decomposition of incomplete multiway tensors,” in
[54] N. P. Jouppi, C. Young, N. Patil, D. Patterson, G. Agrawal, R. Bajwa, Proc. Int. Conf. Mach. Learn. (ICML), vol. 32, no. 2. Bejing, China:
S. Bates, S. Bhatia, N. Boden, A. Borchers et al., “In-datacenter PMLR, 22–24 Jun 2014, pp. 1800–1808.
performance analysis of a tensor processing unit,” in Proc. Annu. Int. [82] M. Zhou, Y. Liu, Z. Long, L. Chen, and C. Zhu, “Tensor rank
Symp. Comput. Architecture, 2017, pp. 1–12. learning in CP decomposition via convolutional neural network,” Signal
[55] (2020) Tensorly-torch. [Online]. Available: [Link] Process.: Image Commun., vol. 73, pp. 12 – 21, 2019.
torch [83] L. R. Tucker, “Some mathematical notes on three-mode factor analy-
[56] A. Novikov, P. Izmailov, V. Khrulkov, M. Figurnov, and I. Oseledets, sis,” Psychometrika, vol. 31, pp. 279–311, 1966.
“Tensor train decomposition on tensorflow (t3f),” J. Mach. Learn. Res., [84] L. De Lathauwer, B. De Moor, and J. Vandewalle, “A multilinear
vol. 21, no. 30, pp. 1–7, 2020. singular value decomposition,” SIAM J. Matrix Anal. and Appl., vol. 21,
[57] R. Ballester-Ripoll. (2018) tntorch – tensor network learning with no. 4, pp. 1253–1278, 2000.
pytorch. [Online]. Available: [Link] [85] M. J. Mohlenkamp, “Musings on multilinear fitting,” Linear Algebra
[58] (2019) Tensor network. [Online]. Available: [Link] and its Appl., vol. 438, no. 2, pp. 834 – 852, 2013, tensors and
TensorNetwork Multilinear Algebra.
[59] M. Nickel. (2007) scikit-tensor. [Online]. Available: [Link] [86] I. V. Oseledets, “Tensor-train decomposition,” SIAM J. Sci. Comput.,
com/mnick/scikit-tensor/ vol. 33, no. 5, pp. 2295–2317, Sep. 2011.
[60] B. W. Bader and T. G. Kolda, “MATLAB tensor toolbox version 2.6,” [87] S. Rommer and S. Östlund, “Class of ansatz wave functions for one-
February 2015. [Online]. Available: [Link] dimensional spin systems and their relation to the density matrix
TensorToolbox/ renormalization group,” Phys. Rev. B, vol. 55, pp. 2164–2181, Jan 1997.
[61] N. Vervliet, O. Debals, L. Sorber, M. Van Barel, and L. De Lathauwer, [88] M. Espig, K. K. Naraparaju, and J. Schneider, “A note on tensor chain
“Tensorlab 3.0,” Mar. 2016. [Online]. Available: [Link] approximation,” Comput. and Visualization in Sci., vol. 15, no. 6, pp.
net 331–344, Dec. 2012.
[62] M. Fishman, S. White, and E. M. Stoudenmire, “The itensor software [89] Q. Zhao, G. Zhou, S. Xie, L. Zhang, and A. Cichocki, “Tensor ring
library for tensor network calculations,” ArXiv, vol. abs/2007.14822, decomposition,” CoRR, vol. abs/1606.05535, 2016.
2020. [90] P. Huber, “Robust statistics (pp. 1248-1251),” Springer Berlin Heidel-
[63] C. Choy, J. Gwak, and S. Savarese, “4D spatio-temporal convnets: berg. Rehabilitation Psychology, vol. 46, pp. 382–399, 2011.
Minkowski convolutional neural networks,” in Proc. IEEE Conf. Com- [91] S. Gandy, B. Recht, and I. Yamada, “Tensor completion and low-n-rank
put. Vis. Pattern Recognit. (CVPR), 2019, pp. 3075–3084. tensor recovery via convex optimization,” Inverse Problems, vol. 27,
[64] F. De la Torre, “A least-squares framework for component analysis,” no. 2, p. 025010, 2011.
IEEE Trans. Pattern Anal. Mach. Intell., vol. 34, no. 6, pp. 1041–1055, [92] J. Liu, P. Musialski, P. Wonka, and J. Ye, “Tensor completion for
2012. estimating missing values in visual data,” IEEE Trans. Pattern Anal.
[65] Z. Zhang, Y. Xu, J. Yang, X. Li, and D. Zhang, “A survey of sparse Mach. Intell., vol. 35, no. 1, pp. 208–220, 2012.
representation: Algorithms and applications,” IEEE Access, vol. 3, p. [93] L. Yang, Z.-H. Huang, and X. Shi, “A fixed point iterative method for
490–530, 2015. low n-rank tensor pursuit,” IEEE Signal Process. Lett., vol. 61, no. 11,
[66] J. Mairal, F. Bach, J. Ponce, and G. Sapiro, “Online learning for matrix pp. 2952–2962, 2013.
factorization and sparse coding,” J. Mach. Learn. Res., vol. 11, p. [94] D. Goldfarb and Z. T. Qin, “Robust low-rank tensor recovery: Models
19–60, Mar. 2010. and algorithms,” SIAM J. Matrix Anal. and Appl., vol. 35, no. 1, pp.
[67] F. L. Hitchcock, “The expression of a tensor or a polyadic as a sum of 225–253, 2014.
products,” J. of Math. and Phys., vol. 6, no. 1-4, pp. 164–189, 1927. [95] L. Vandenberghe and S. Boyd, “Semidefinite programming,” SIAM
[68] J. D. Carroll and J.-J. Chang, “Analysis of individual differences Rev., vol. 38, no. 1, pp. 49–95, 1996.
in multidimensional scaling via an n-way generalization of “Eckart- [96] B. K. Natarajan, “Sparse approximate solutions to linear systems,”
Young” decomposition,” Psychometrika, vol. 35, pp. 283–319, 1970. SIAM J. Comput., vol. 24, no. 2, pp. 227–234, 1995.
[69] R. A. Harshman, “Foundations of the PARAFAC procedure: Models [97] Y. Li, J. Yan, Z. Yue, and J. Yang, “Optimum subspace learning and
and conditions for an ”explanatory” multi-modal factor analysis,” error correction for tensors,” in Proc. Eur. Conf. Comput. Vis. (ECCV),
UCLA Working Papers in Phonetics, vol. 16, pp. 1–84, 1970. 09 2010, pp. 790–803.
[70] K. P. F.R.S., “LIII. on lines and planes of closest fit to systems of points [98] B. Huang, C. Mu, D. Goldfarb, and J. Wright, “Provable models for
in space,” The London, Edinburgh, and Dublin Philos. Mag. and J. of robust low-rank tensor completion,” Pacific J. Optim., vol. 11, pp. 339–
Sci., vol. 2, no. 11, pp. 559–572, 1901. 364, 04 2015.
[71] H. Hotelling, “Analysis of a complex of statistical variables into [99] W. Hu, D. Tao, W. Zhang, Y. Xie, and Y. Yang, “A new low-rank tensor
principal components.” J. Educ. Psychol., vol. 24, no. 6, p. 417–441, model for video completion,” ArXiv, vol. abs/1509.02027, 2015.
1933. [100] S. Gandy, B. Recht, and I. Yamada, “Tensor completion and low-n-rank
[72] A. Hyvärinen and E. Oja, “Independent component analysis: algorithms tensor recovery via convex optimization,” Inverse Problems, vol. 27,
and applications,” Neural Netw., vol. 13, no. 4, pp. 411 – 430, 2000. no. 2, p. 025010, Jan 2011.
[73] D. Lee and H. Seung, “Learning the parts of objects by non-negative [101] Y. Liu, F. Shang, W. Fan, J. Cheng, and H. Cheng, “Generalized higher-
matrix factorization,” Nature, vol. 401, pp. 788–91, 11 1999. order orthogonal iteration for tensor decomposition and completion,”
[74] S. Yan, D. Xu, B. Zhang, H. Zhang, Q. Yang, and S. Lin, “Graph in Advances in Neural Inf. Process. Syst. (NeurIPS), 2014, pp. 1763–
embedding and extensions: A general framework for dimensionality 1771.
reduction,” IEEE Trans. Pattern Anal. Mach. Intell., vol. 29, no. 1, pp. [102] D. G. Chachlakis, A. Prater-Bennette, and P. P. Markopoulos, “L1-
40–51, 2007. norm Tucker tensor decomposition,” IEEE Access, vol. 7, pp. 178 454–
[75] E. J. Candès, X. Li, Y. Ma, and J. Wright, “Robust principal component 178 465, 2019.
analysis?” J. ACM, vol. 58, no. 3, Jun. 2011. [103] D. Chachlakis, A. Prater-Bennette, and P. Markopoulos, “L1-norm
[76] Z. Zhang and D. Tao, “Slow feature analysis for human action higher-order orthogonal iterations for robust tensor analysis,” in IEEE
recognition,” IEEE Trans. Pattern Anal. Mach. Intell., vol. 34, no. 3, Int. Conf. Acoustics, Speech, Signal Process. (ICASSP), 2020, pp.
pp. 436–450, 2012. 4826–4830.
[77] A. Mensch, J. Mairal, B. Thirion, and G. Varoquaux, “Dictionary [104] D. P. Bertsekas, Constrained Optimization and Lagrange Multiplier
learning for massive matrix factorization,” in Proc. Int. Conf. Mach. Methods (Optimization and Neural Computation Series), 1st ed.
Learn. (ICML), ser. ICML’16. [Link], 2016, p. 1737–1746. Athena Scientific, 1996.
[78] J. Håstad, “Tensor rank is NP-complete,” J. Algorithms (Print), vol. 11, [105] H. Lu, K. N. Plataniotis, and A. N. Venetsanopoulos, “MPCA: Mul-
no. 4, pp. 644–654, 1990. tilinear principal component analysis of tensor objects,” IEEE Trans.
[79] C. J. Hillar and L.-H. Lim, “Most tensor problems are NP-hard,” J. Neural Netw., vol. 19, no. 1, pp. 18–39, 2008.
ACM, vol. 60, no. 6, Nov. 2013. [106] Y. Panagakis, C. Kotropoulos, and G. Arce, “Music genre classification
[80] Q. Zhao, L. Zhang, and A. Cichocki, “Bayesian CP factorization of using locality preserving non-negative tensor factorization and sparse
incomplete tensors with automatic rank determination,” IEEE Trans. representations.” in Proc. Int. Soc. Music Information Retrieval. (IS-
Pattern Anal. Mach. Intell., vol. 37, no. 9, pp. 1751–1763, 2015. MIR), 01 2009, pp. 249–254.
PROCEEDINGS OF THE IEEE 23

[107] Y. Panagakis, C. Kotropoulos, and G. R. Arce, “Non-negative multilin- network design,” in Proc. Int. Conf. Learn. Representations (ICLR),
ear principal component analysis of auditory temporal modulations for 2018.
music genre classification,” IEEE Trans. Audio Speech Lang. Process., [131] Q. Zhao, M. Sugiyama, L. Yuan, and A. Cichocki, “Learning efficient
vol. 18, no. 3, pp. 576–588, 2010. tensor representations with ring structure networks,” in ICLR work-
[108] S. Yan, D. Xu, Q. Yang, L. Zhang, X. Tang, and H.-J. Zhang, shops, 2018.
“Multilinear discriminant analysis for face recognition,” IEEE Trans. [132] J. Ye, G. Li, D. Chen, H. Yang, S. Zhe, and Z. Xu, “Block-term tensor
Image Process., vol. 16, no. 1, pp. 212–220, 2006. neural networks,” Neural Netw., vol. 130, pp. 11 – 21, 2020.
[109] D. Tao, X. Li, X. Wu, and S. J. Maybank, “General tensor discriminant [133] T. Garipov, D. Podoprikhin, A. Novikov, and D. Vetrov, “Ultimate
analysis and gabor features for gait recognition,” IEEE Trans. Pattern tensorization: compressing convolutional and FC layers alike,” NeurIPS
Anal. Mach. Intell., vol. 29, no. 10, pp. 1700–1715, 2007. workshop: Learning with Tensors: Why Now and How?, 2016.
[110] Q. Li and D. Schonfeld, “Multilinear discriminant analysis for higher- [134] M. Janzamin, H. Sedghi, and A. Anandkumar, “Beating the perils of
order tensor data classification,” IEEE Trans. Pattern Anal. Mach. non-convexity: Guaranteed training of neural networks using tensor
Intell., vol. 36, no. 12, pp. 2524–2537, 2014. methods,” arXiv preprint arXiv:1506.08473, 2015.
[111] S. Hsieh, C. Lu, and S. Pei, “2D sparse dictionary learning via [135] X. Cao, G. Rabusseau, and J. Pineau, “Tensor regression networks with
tensor decomposition,” in 2014 IEEE Global Conf. Signal Inf. Process. various low-rank tensor approximations,” CoRR, vol. abs/1712.09520,
(GlobalSIP), 2014, pp. 492–496. 2017.
[112] Y. Fang, B. Huang, and J. Wu, “2D sparse signal recovery via 2D [136] A. Cichocki, A.-H. Phan, Q. Zhao, N. Lee, I. Oseledets, M. Sugiyama,
orthogonal matching pursuit,” Sci. China Inf. Sci., vol. 55, 01 2011. and D. P. Mandic, “Tensor networks for dimensionality reduction and
[113] Z. Shakeri, W. U. Bajwa, and A. D. Sarwate, “Minimax lower bounds large-scale optimization: Part 2 applications and future perspectives,”
for Kronecker-structured dictionary learning,” 2016 IEEE Int. Symp. Foundations and Trends® in Machine Learning, vol. 9, no. 6, pp. 431–
Inf. Theory (ISIT), Jul 2016. 673, 2017.
[114] Z. Shakeri, W. U. Bajwa, and A. D. Sarwate, “Minimax lower bounds [137] G. O. Berger, P. Absil, L. D. Lathauwer, R. M. Jungers, and M. V.
on dictionary learning for tensor data,” IEEE Trans. Inf. Theory, vol. 64, Barel, “Equivalent polyadic decompositions of matrix multiplication
no. 4, pp. 2706–2726, 2018. tensors,” CoRR, vol. abs/1902.03950, 2019.
[115] M. Bahri, Y. Panagakis, and S. Zafeiriou, “Robust Kronecker- [138] R. Rigamonti, A. Sironi, V. Lepetit, and P. Fua, “Learning separable
decomposable component analysis for low-rank modeling,” in Proc. filters,” in Proc. IEEE Conf. Comput. Vis. Pattern Recognit. (CVPR),
IEEE Int. Conf. Comput. Vis. (ICCV), 2017, pp. 3372–3381. June 2013.
[116] B. Haeffele, E. Young, and R. Vidal, “Structured low-rank matrix [139] M. Jaderberg, A. Vedaldi, and A. Zisserman, “Speeding up convolu-
factorization: Optimality, algorithm, and applications to image process- tional neural networks with low rank expansions,” in Proc. Brit. Mach.
ing,” Proc. Int. Conf. Mach. Learn. (ICML), vol. 5, pp. 4108–4117, 01 Vis. Conf. (BMVC), 2014.
2014. [140] Y. Kim, E. Park, S. Yoo, T. Choi, L. Yang, and D. Shin, “Compression
[117] B. D. Haeffele and R. Vidal, “Global optimality in tensor factorization, of deep convolutional neural networks for fast and low power mobile
deep learning, and beyond,” ArXiv, vol. abs/1506.07540, 2015. applications,” Proc. Int. Conf. Learn. Representations (ICLR), 2016.
[141] K. Hayashi, T. Yamaguchi, Y. Sugawara, and S.-i. Maeda, “Exploring
[118] G. Rabusseau and H. Kadri, “Low-rank regression with tensor re-
unexplored tensor network decompositions for convolutional neural
sponses,” in Advances in Neural Inf. Process. Syst. (NeurIPS), D. D.
networks,” in Advances in Neural Inf. Process. Syst. (NeurIPS), 2019,
Lee, M. Sugiyama, U. V. Luxburg, I. Guyon, and R. Garnett, Eds.
pp. 5552–5562.
Curran Associates, Inc., 2016, pp. 1867–1875.
[142] L. De Lathauwer, “Decompositions of a higher-order tensor in block
[119] H. Zhou, L. Li, and H. Zhu, “Tensor regression with applications in
terms—part II: Definitions and uniqueness,” SIAM J. Matrix Anal.
neuroimaging data analysis,” J. Amer. Statist. Assoc., vol. 108, no. 502,
Appl., vol. 30, no. 3, pp. 1033–1066, 2008.
pp. 540–552, 2013.
[143] J. Li, Y. Sun, J. Su, T. Suzuki, and F. Huang, “Understanding general-
[120] N. Batmanghelich, A. Dong, B. Taskar, and C. Davatzikos, “Regu- ization in deep learning via tensor methods,” in Proc. Int. Conf. Artif.
larized tensor factorization for multi-modality medical image classifi- Intell. Statist. (AISTATS), 2020.
cation,” in Int. Conf. Medical Image Comput. and Comput.-Assisted [144] S. Arora, N. Cohen, W. Hu, and Y. Luo, “Implicit regularization in
Intervention. Springer, 2011, pp. 17–24. deep matrix factorization,” in Advances in Neural Inf. Process. Syst.
[121] R. Yu and Y. Liu, “Learning from multiway data: Simple and efficient (NeurIPS), H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc,
tensor regression,” in Proc. Int. Conf. Mach. Learn. (ICML), vol. 48, E. Fox, and R. Garnett, Eds., vol. 32, 2019, pp. 7413–7424.
20–22 Jun 2016, pp. 373–381. [145] J. Gusak, M. Kholiavchenko, E. Ponomarev, L. Markeeva,
[122] X. Song and H. Lu, “Multilinear regression for embedded feature P. Blagoveschensky, A. Cichocki, and I. Oseledets, “Automated
selection with application to fMRI analysis,” in Proc. AAAI Conf. Artif. multi-stage compression of neural networks,” in Proc. IEEE Int. Conf.
Intell. (AAAI), 2017. Comput. Vis. (ICCV), Oct 2019.
[123] B. Romera-paredes, H. Aung, N. Bianchi-berthouze, and M. Pontil, [146] A. Newell, K. Yang, and J. Deng, “Stacked hourglass networks for
“Multilinear multitask learning,” in Proc. Int. Conf. Mach. Learn. human pose estimation,” in Proc. Eur. Conf. Comput. Vis. (ECCV),
(ICML), S. Dasgupta and D. Mcallester, Eds., vol. 28, no. 3. JMLR 2016.
Workshop and Conf. Proc., May 2013, pp. 1444–1452. [147] O. Ronneberger, P. Fischer, and T. Brox, “U-Net: Convolutional net-
[124] X. Li, D. Xu, H. Zhou, and L. Li, “Tucker tensor regression and works for biomedical image segmentation,” Medical Image Comput.
neuroimaging analysis,” Statist. in Biosci., vol. 10, no. 3, pp. 520–545, Comput.-Assisted Intervention (MICCAI), p. 234–241, 2015.
Dec. 2018. [148] A.-H. Phan, K. Sobolev, K. Sozykin, D. Ermilov, J. Gusak,
[125] Q. Zhao, C. F. Caiafa, D. P. Mandic, L. Zhang, T. Ball, A. Schulze- P. Tichavský, V. Glukhov, I. Oseledets, and A. Cichocki, “Stable low-
bonhage, and A. S. Cichocki, “Multilinear subspace regression: An rank tensor decomposition for compression of convolutional neural
orthogonal tensor decomposition approach,” in Advances in Neural Inf. network,” in ECCV, A. Vedaldi, H. Bischof, T. Brox, and J.-M. Frahm,
Process. Syst. (NeurIPS), 2011, pp. 1269–1277. Eds. Cham: Springer International Publishing, 2020, pp. 522–539.
[126] Q. Zhao, C. F. Caiafa, D. P. Mandic, Z. C. Chao, Y. Nagasaka, [149] N. Srivastava, G. Hinton, A. Krizhevsky, I. Sutskever, and R. Salakhut-
N. Fujii, L. Zhang, and A. Cichocki, “Higher-order partial least squares dinov, “Dropout: A simple way to prevent neural networks from
(HOPLS): A generalized multi-linear regression method,” CoRR, vol. overfitting,” J. Mach. Learn. Res., vol. 15, no. 56, pp. 1929–1958,
abs/1207.1230, 2012. 2014.
[127] J. Wu and M. M. Trivedi, “A regression model in TensorPCA subspace [150] L. Wan, M. Zeiler, S. Zhang, Y. L. Cun, and R. Fergus, “Regularization
for face image super-resolution reconstruction,” in Proc. 18th Int. Conf. of neural networks using DropConnect,” in Proc. Int. Conf. Mach.
on Pattern Recognit. (ICPR), vol. 3. IEEE Comput. Soc., 2006, pp. Learn. (ICML), S. Dasgupta and D. McAllester, Eds., vol. 28, no. 3,
627–630. 2013, pp. 1058–1066.
[128] W. Lu, Z. Zhu, and H. Lian, “High-dimensional quantile tensor [151] A. Bulat, J. Kossaifi, S. Bhattacharya, Y. Panagakis, G. Tzimiropoulos,
regression,” J. Mach. Learn. Res., vol. 21, no. 250, pp. 1–31, 2020. N. D. Lane, and M. Pantic, “Defensive tensorization: Randomized
[129] N. Cohen and A. Shashua, “Convolutional rectifier networks as general- tensor parametrization for robust neural networks,” 2020. [Online].
ized tensor decompositions,” in Proc. Int. Conf. Mach. Learn. (ICML), Available: [Link]
2016, pp. 955–963. [152] N. Das, M. Shanbhogue, S.-T. Chen, F. Hohman, S. Li, L. Chen, M. E.
[130] Y. Levine, D. Yakira, N. Cohen, and A. Shashua, “Deep learning and Kounavis, and D. H. Chau, “SHIELD,” Proc. 24th ACM SIGKDD Int.
quantum entanglement: Fundamental connections with implications to Conf. Knowl. Discovery & Data Mining, Jul 2018.
PROCEEDINGS OF THE IEEE 24

[153] N. Entezari and E. E. Papalexakis, “TensorShield: Tensor-based defense [180] T. Kefalas, K. Vougioukas, Y. Panagakis, S. Petridis, J. Kossaifi, and
against adversarial attacks on images,” ArXiv, vol. abs/2002.10252, M. Pantic, “Speech-driven facial animation using polynomial fusion
2020. of features,” in IEEE Int. Conf. Acoustics, Speech, Signal Process.
[154] A. G. Ivakhnenko, “Polynomial theory of complex systems,” IEEE (ICASSP), 2020, pp. 3487–3491.
Trans. Syst., Man, Cybern. Syst., no. 4, pp. 364–378, 1971. [181] D. P. Kingma and M. Welling, “Auto-encoding variational bayes,”
[155] S.-K. Oh, W. Pedrycz, and B.-J. Park, “Polynomial neural networks ar- in Proc. Int. Conf. Learn. Representations (ICLR), Y. Bengio and
chitecture: analysis and design,” Computers & Electrical Engineering, Y. LeCun, Eds., 2014.
vol. 29, no. 6, pp. 703–725, 2003. [182] M. A. O. Vasilescu and D. Terzopoulos, “Multilinear analysis of image
[156] Y. Shin and J. Ghosh, “The pi-sigma network: An efficient higher-order ensembles: Tensorfaces,” in Proc. Eur. Conf. Comput. Vis. (ECCV),
neural network for pattern classification and function approximation,” 2002.
in Int. Joint Conf. Neural Netw., vol. 1, 1991, pp. 13–18. [183] T. Sim, S. Baker, and M. Bsat, “The CMU pose, illumination, and
[157] C.-K. Li, “A sigma-pi-sigma neural network (SPSNN),” Neural Pro- expression (PIE) database,” in Proc. IEEE Int. Conf. Automatic Face
cess. Lett., vol. 17, no. 1, pp. 1–19, 2003. and Gesture Recognit., 2002, pp. 53–58.
[158] S. Rendle, “Factorization machines,” in 2010 IEEE International Con- [184] R. Gross, I. Matthews, J. Cohn, T. Kanade, and S. Baker, “Multi-PIE,”
ference on Data Mining, 2010, pp. 995–1000. Image and Vision Comput., vol. 28, no. 5, pp. 807 – 813, 2010.
[159] M. Blondel, A. Fujino, N. Ueda, and M. Ishihata, “Higher-order [185] Y. Moses, S. Ullman, and S. Edelman, “Generalization to novel images
factorization machines,” in Advances in Neural Information Processing in upright and inverted faces,” Perception, vol. 25, no. 4, pp. 443–461,
Systems, vol. 29, 2016. 1996.
[160] R. K. Srivastava, K. Greff, and J. Schmidhuber, “Highway networks,” [186] M. A. O. Vasilescu, “Human motion signatures: Analysis, synthesis,
arXiv preprint arXiv:1505.00387, 2015. recognition,” in Proc. IEEE Int. Conf. Pattern Recognit. (ICPR), ser.
[161] S. M. Jayakumar, W. M. Czarnecki, J. Menick, J. Schwarz, J. Rae, ICPR ’02. USA: IEEE Comput. Soci., 2002, p. 30456.
S. Osindero, Y. W. Teh, T. Harley, and R. Pascanu, “Multiplicative [187] C.-S. Lee and A. Elgammal, “Towards scalable view-invariant gait
interactions and where to find them,” in Proc. Int. Conf. Learn. recognition: Multilinear analysis for gait,” in Audio- and Video-Based
Representations (ICLR), 2020. Biometric Person Authentication, T. Kanade, A. Jain, and N. K. Ratha,
[162] X. Dong, J. Huang, Y. Yang, and S. Yan, “More is less: A more Eds. Berlin, Heidelberg: Springer Berlin Heidelberg, 2005, pp. 395–
complicated network with less inference complexity,” in Proc. IEEE 405.
Conf. Comput. Vis. Pattern Recognit. (CVPR), 2017, pp. 5840–5848. [188] N. Hasler, H. Ackermann, B. Rosenhahn, T. Thormählen, and H.-P.
[163] T. Karras, S. Laine, and T. Aila, “A style-based generator architecture Seidel, “Multilinear pose and body shape estimation of dressed subjects
for generative adversarial networks,” in Proc. IEEE Conf. Comput. Vis. from image sets,” in Proc. IEEE Conf. Comput. Vis. Pattern Recognit.
Pattern Recognit. (CVPR), 2019. (CVPR). IEEE, 2010, pp. 1823–1830.
[164] G. Chrysos, S. Moschoglou, G. Bouritsas, Y. Panagakis, J. Deng, and [189] Y. Chen, Z. Liu, and Z. Zhang, “Tensor-based human body modeling,”
S. Zafeiriou, “π−nets: Deep polynomial neural networks,” in Proc. in Proc. IEEE Conf. Comput. Vis. Pattern Recognit. (CVPR), 2013, pp.
IEEE Conf. Comput. Vis. Pattern Recognit. (CVPR), 2020. 105–112.
[190] D. Derkach, A. Ruiz, and F. M. Sukno, “Tensor decomposition and
[165] G. G. Chrysos and Y. Panagakis, “NAPS: Non-adversarial polynomial
non-linear manifold modeling for 3D head pose estimation,” Int. J.
synthesis,” Pattern Recognit. Lett., vol. 140, pp. 318–324, 2020.
Comput. Vision (IJCV), vol. 127, no. 10, pp. 1565–1585, 2019.
[166] L. Sorber, M. V. Barel, and L. D. Lathauwer, “Numerical solution of
[191] D. Vlasic, M. Brand, H. Pfister, and J. Popovic, “Face transfer with
bivariate and polyanalytic polynomial systems,” SIAM J. Numer. Anal.,
multilinear models,” in ACM SIGGRAPH 2006 Courses, 2006, pp. 24–
vol. 52, no. 4, pp. 1551–1572, 2014.
es.
[167] M. H. Stone, “The generalized Weierstrass approximation theorem,”
[192] X. Zhang, X. Shi, W. Hu, X. Li, and S. Maybank, “Visual tracking via
Math. Mag., vol. 21, no. 5, pp. 237–254, 1948.
dynamic tensor analysis with mean update,” Neurocomput., vol. 74,
[168] S. Nikol’skii, Analysis III: Spaces of Differentiable Functions, ser. no. 17, pp. 3277–3285, 2011.
Encyclopaedia of Mathematical Sciences. Springer Berlin Heidelberg, [193] J. Shao, S. K. Zhou, and R. Chellappa, “Appearance-based tracking and
2013. recognition using the 3D trilinear tensor,” in IEEE Int. Conf. Acoustics,
[169] G. G. Chrysos, S. Moschoglou, G. Bouritsas, J. Deng, Y. Panagakis, and Speech, Signal Process. (ICASSP), vol. 3. IEEE, 2004, pp. iii–613.
S. P. Zafeiriou, “Deep polynomial neural networks,” IEEE Transactions [194] M. Ramachandran, A. Veeraraghavan, and R. Chellappa, “A fast
on Pattern Analysis and Machine Intelligence, 2021. bilinear structure from motion algorithm using a video sequence and
[170] E. R. Berlekamp, “Factoring polynomials over finite fields,” Bell System inertial sensors,” IEEE Trans. Pattern Anal. Mach. Intell., vol. 33, no. 1,
Technical Journal, vol. 46, no. 8, pp. 1853–1859, 1967. pp. 186–193, 2010.
[171] D. G. Cantor and H. Zassenhaus, “A new algorithm for factoring [195] H. Liu, R. Chellappa, and A. Rosenfeld, “Accurate dense optical flow
polynomials over finite fields,” Mathematics of Computation, pp. 587– estimation using adaptive structure tensors and a parametric model,”
592, 1981. IEEE Trans. Image Process., vol. 12, no. 10, pp. 1170–1180, 2003.
[172] P. Dreesen, M. Ishteva, and J. Schoukens, “Decoupling multivariate [196] M. A. O. Vasilescu and E. Kim, “Compositional hierarchical tensor
polynomials using first-order information and tensor decompositions,” factorization: Representing hierarchical intrinsic and extrinsic causal
SIAM J. Matrix Anal. and Appl., vol. 36, no. 2, pp. 864–879, 2015. factors,” in ACM SIGKDD Conf. on Knowl. Discovery and Data Mining
[173] P. Comon, Y. Qi, and K. Usevich, “A polynomial formulation for joint (KDD’19): Workshop on Tensor Methods for Emerging Data Science
decomposition of symmetric tensors of different orders,” in Int. Conf. Challenges, 2019.
Latent Variable Anal. Signal Separation. Springer, 2015, pp. 22–30. [197] Q. Qiu and R. Chellappa, “Compositional dictionaries for domain
[174] D. Bahdanau, K. Cho, and Y. Bengio, “Neural machine translation adaptive face recognition,” IEEE Trans. Image Process., vol. 24, no. 12,
by jointly learning to align and translate,” in Proc. Int. Conf. Learn. p. 5152–5165, Dec 2015.
Representations (ICLR), 2015. [198] M. Wang, Y. Panagakis, P. Snape, and S. Zafeiriou, “Learning the
[175] X. Wang, R. Girshick, A. Gupta, and K. He, “Non-local neural multilinear structure of visual data,” in Proc. IEEE Conf. Comput. Vis.
networks,” in Proc. IEEE Conf. Comput. Vis. Pattern Recognit. (CVPR), Pattern Recognit. (CVPR), 2017, pp. 6053–6061.
2018, pp. 7794–7803. [199] M. A. O. Vasilescu and D. Terzopoulos, “Multilinear projection for
[176] J.-B. Cordonnier, A. Loukas, and M. Jaggi, “Multi-head attention: appearance-based recognition in the tensor framework,” in Proc. IEEE
Collaborate instead of concatenate,” arXiv preprint arXiv:2006.16362, Int. Conf. Comput. Vis. (ICCV), 2007, pp. 1–8.
2020. [200] J. B. Tenenbaum and W. T. Freeman, “Separating style and content
[177] F. Babiloni, I. Marras, G. Slabaugh, and S. Zafeiriou, “TESA: Tensor with bilinear models,” Neural Comput., vol. 12, no. 6, pp. 1247–1283,
element self-attention via matricization,” in Proc. IEEE Conf. Comput. 2000.
Vis. Pattern Recognit. (CVPR), 2020, pp. 13 945–13 954. [201] Y. Tang, R. Salakhutdinov, and G. Hinton, “Tensor analyzers,” in Proc.
[178] M. Wang, Z. Shu, Y. Panagakis, D. Samaras, and S. Zafeiriou, “An Int. Conf. Mach. Learn. (ICML), S. Dasgupta and D. McAllester, Eds.,
adversarial neuro-tensorial approach for learning disentangled repre- vol. 28, no. 3. Atlanta, Georgia, USA: PMLR, 17–19 Jun 2013, pp.
sentations,” Int. J. Comput. Vision (IJCV), vol. 127, pp. 743–762, 2019. 163–171.
[179] V. F. Abrevaya, S. Wuhrer, and E. Boyer, “Multilinear autoencoder for [202] Y.-J. Hong, S. E. Choi, G. P. Nam, H. Choi, J. Cho, and I.-J.
3D face model learning,” in IEEE Winter Conf. Appl. Comput. Vision Kim, “Adaptive 3D model-based facial expression synthesis and pose
(WACV). IEEE, 2018, pp. 1–9. frontalization,” Sensors, vol. 20, no. 9, p. 2578, 2020.
PROCEEDINGS OF THE IEEE 25

[203] M. Georgopoulos, G. Chrysos, Y. Panagakis, and M. Pantic, “Multilin- [227] L. He, X. Kong, P. S. Yu, X. Yang, A. B. Ragin, and Z. Hao, “Dusk:
ear latent conditioning for generating unseen attribute combinations,” A dual structure-preserving kernel for supervised tensor learning with
Proc. Int. Conf. Mach. Learn. (ICML), 2020. applications to neuroimages,” in Proc. 2014 SIAM Int. Conf. Data
[204] H. Lu, K. N. Plataniotis, and A. N. Venetsanopoulos, “Uncorrelated Mining. SIAM, 2014, pp. 127–135.
multilinear discriminant analysis with regularization for gait recogni- [228] A. Mitenkova, J. Kossaifi, Y. Panagakis, and M. Pantic, “Valence and
tion,” in 2007 Biometrics Symp., 2007. arousal estimation in-the-wild with tensor methods,” in IEEE Int. Conf.
[205] H. Lu, K. N. Plataniotis, and A. N. Venetsanopoulos, “Boosting LDA on Autom. Face & Gesture Recognit. (FG). IEEE, 2019, pp. 1–7.
with regularization on MPCA features for gait recognition,” in 2007 [229] A. G. Howard, M. Zhu, B. Chen, D. Kalenichenko, W. Wang,
Biometrics Symp., 2007. T. Weyand, M. Andreetto, and H. Adam, “MobileNets: Efficient
[206] H. Lu, K. N. Plataniotis, and A. N. Venetsanopoulos, “Multilinear convolutional neural networks for mobile vision applications,” CoRR,
principal component analysis of tensor objects for recognition,” in Proc. vol. abs/1704.04861, 2017.
IEEE Int. Conf. Pattern Recognit. (ICPR), vol. 2, 01 2006, pp. 776–779. [230] M. Sandler, A. Howard, M. Zhu, A. Zhmoginov, and L.-C. Chen,
[207] H. Lu, K. N. Plataniotis, and A. N. Venetsanopoulos, “Boosting “MobileNetV2: Inverted residuals and linear bottlenecks,” in Proc.
discriminant learners for gait recognition using MPCA features,” IEEE Conf. Comput. Vis. Pattern Recognit. (CVPR), 2018.
EURASIP J. Image and Video Process., 2009. [231] S. Nakajima, R. Tomioka, M. Sugiyama, and S. Babacan, “Perfect
[208] I. Macêdo, E. V. Brazil, and L. Velho, “Expression transfer between dimensionality recovery by variational bayesian pca,” in Advances in
photographs through multilinear AAM’s,” in 2006 19th Brazilian Symp. Neural Information Processing Systems, F. Pereira, C. J. C. Burges,
Comput. Graphics and Image Process. IEEE, 2006, pp. 239–246. L. Bottou, and K. Q. Weinberger, Eds., vol. 25. Curran Associates,
Inc., 2012. [Online]. Available: [Link]
[209] Z.-H. Feng, J. Kittler, B. Christmas, and X.-J. Wu, “A unified tensor-
2012/file/[Link]
based active appearance model,” ACM Trans. Multimedia Comput.,
[232] J. Su, W. Byeon, J. Kossaifi, F. Huang, J. Kautz, and A. Anandkumar,
Commun., Appl. (TOMM), vol. 15, no. 3s, pp. 1–22, 2019.
“Convolutional Tensor-Train LSTM for spatio-temporal learning,” in
[210] J. Kossaifi, G. Tzimiropoulos, and M. Pantic, “Fast and exact newton Advances in Neural Information Processing Systems, vol. 33, 2020,
and bidirectional fitting of active appearance models,” IEEE transac- pp. 13 714–13 726.
tions on image processing, vol. 26, no. 2, pp. 1040–1053, 2016. [233] K. He, X. Zhang, S. Ren, and J. Sun, “Delving deep into rectifiers:
[211] K. Inoue and K. Urahama, “DSVD: a tensor-based image compression Surpassing human-level performance on imagenet classification,” in
and recognition method,” in IEEE Int. Symp. on Circuits and Syst., Proceedings of the IEEE International Conference on Computer Vision
2005, pp. 6308–6311 Vol. 6. (ICCV), December 2015.
[212] R. Ballester-Ripoll, P. Lindstrom, and R. Pajarola, “TTHRESH: Ten- [234] X. Glorot and Y. Bengio, “Understanding the difficulty of training
sor compression for multidimensional visual data,” IEEE Trans. Vis. deep feedforward neural networks,” in Proceedings of the Thirteenth
Comput. Graphics, vol. 26, no. 9, p. 2891–2903, Sep 2020. International Conference on Artificial Intelligence and Statistics, vol. 9.
[213] J. A. Richards, X. Jia, D. E. Ricken, and W. Gessner, Remote Sensing JMLR Workshop and Conference Proceedings, 2010, pp. 249–256.
Digital Image Analysis: An Introduction, 3rd ed. Berlin, Heidelberg:
Springer-Verlag, 1999.
[214] L. Zhang, L. Zhang, D. Tao, and X. Huang, “Tensor discriminative
locality alignment for hyperspectral image spectral–spatial feature
extraction,” IEEE Trans. Geosci. Remote Sens., vol. 51, no. 1, pp. 242–
256, 2012.
[215] R. Selvan and E. B. Dam, “Tensor networks for medical image
classification,” arXiv preprint arXiv:2004.10076, 2020.
[216] T. Papastergiou, E. I. Zacharaki, and V. Megalooikonomou, “Tensor de-
composition for multiple-instance classification of high-order medical
data,” Complexity, vol. 2018, 2018.
[217] E. E. Papalexakis, C. Faloutsos, T. M. Mitchell, P. P. Talukdar,
N. D. Sidiropoulos, and B. Murphy, “Turbo-SMT: Accelerating coupled
sparse matrix-tensor factorizations by 200x,” in Proc. 2014 SIAM Int.
Conf. Data Mining. SIAM, 2014, pp. 118–126.
[218] A. Ravindran, M. Baburaj, and S. N. George, “Video inpainting based
on re-weighted tensor decomposition,” in Proc. of 2nd Int. Conf.
Comput. Vision and Image Process. Springer, 2018, pp. 265–276.
[219] L. Yuan, C. Li, D. Mandic, J. Cao, and Q. Zhao, “Tensor ring
decomposition with rank minimization on latent space: An efficient
approach for tensor completion,” in Proc. of the AAAI Conf. on Artif.
Intell., vol. 33, 2019, pp. 9151–9158.
[220] R. Zdunek and T. Sadowski, “Image completion with hybrid interpola-
tion in tensor representation,” Appl. Sci., vol. 10, no. 3, p. 797, 2020.
[221] W. Hu, D. Tao, W. Zhang, Y. Xie, and Y. Yang, “The twist tensor
nuclear norm for video completion,” IEEE Trans. Neural Netw., vol. 28,
no. 12, pp. 2961–2973, 2016.
[222] J. Xue, Y. Zhao, W. Liao, and J. C.-W. Chan, “Nonlocal low-rank
regularized tensor decomposition for hyperspectral image denoising,”
IEEE Trans. Geosci. Remote Sens., vol. 57, no. 7, pp. 5174–5189, 2019.
[223] X. Liu, S. Bourennane, and C. Fossati, “Denoising of hyperspectral im-
ages using the PARAFAC model and statistical performance analysis,”
IEEE Trans. Geosci. Remote Sens., vol. 50, no. 10, pp. 3717–3724,
2012.
[224] H. Fan, Y. Chen, Y. Guo, H. Zhang, and G. Kuang, “Hyperspectral
image restoration using low-rank tensor recovery,” IEEE J. Sel. Topics
Appl. Earth Observ., vol. 10, no. 10, pp. 4589–4604, 2017.
[225] G. Zhou, Q. Zhao, Y. Zhang, T. Adalı, S. Xie, and A. Cichocki, “Linked
component analysis from matrices to high-order tensors: Applications
to biomedical data,” Proc. IEEE, vol. 104, no. 2, pp. 310–331, 2016.
[226] K. Makantasis, A. D. Doulamis, N. D. Doulamis, and A. Nikitakis,
“Tensor-based classification models for hyperspectral data analysis,”
IEEE Trans. Geosci. Remote Sens., vol. 56, no. 12, pp. 6884–6898,
2018.