MulT: An End-to-End Multitask Learning Transformer

Deblina Bhattacharjee, Tong Zhang, Sabine Süsstrunk and Mathieu Salzmann

School of Computer and Communication Sciences, EPFL, Switzerland
{[Link], [Link], [Link], [Link]}@[Link]

Abstract

We propose an end-to-end Multitask Learning

Transformer framework, named MulT, to simultane-
ously learn multiple high-level vision tasks, including depth
estimation, semantic segmentation, reshading, surface nor-
mal estimation, 2D keypoint detection, and edge detection.
Based on the Swin transformer model, our framework
encodes the input image into a shared representation and
makes predictions for each vision task using task-specific
transformer-based decoder heads. At the heart of our
approach is a shared attention mechanism modeling the
dependencies across the tasks. We evaluate our model
on several multitask benchmarks, showing that our MulT
framework outperforms both the state-of-the art multitask
Figure 1. Motivation for MulT. Our MulT model, which is a
convolutional neural network models and all the respective
transformer-based encoder-decoder model with shared attention to
single task transformer models. Our experiments further
learn task inter-dependencies, produces better results than both the
highlight the benefits of sharing attention across all the dedicated 1-task transformer models (1-task Swin [26]) and the
tasks, and demonstrate that our MulT model is robust and state-of-the-art multitask CNN baseline [41].
generalizes well to new domains. Our project website is at
[Link]
address the following questions: Can a transformer model
trained jointly across tasks improve the performance in each
task relative to single-task transformers? Can one explicitly
1. Introduction encode dependencies across tasks in a transformer-based
First proposed in [45], transformers have made great framework? Can a multitask transformer generalize to un-
strides in a wide range of domains. For instance, previ- seen domains?
ous works [13, 25, 35–37, 52] have demonstrated that trans- To the best of our knowledge, only [9, 30, 40] have
formers trained on large datasets learn strong representa- touched upon the problem of addressing multiple tasks with
tions for many downstream language tasks; and models transformers. However, none of these works aims to en-
based on transformers have achieved promising results on code strong dependencies across the tasks beyond the use
image classification, object detection, and panoptic segmen- of a shared backbone. Furthermore, IPT [9] handles solely
tation [5, 6, 14, 18, 34, 38, 44, 49, 58]. In contrast to these low-level vision tasks, such as denoising, super-resolution
works that focus on a single task, in this paper, we investi- and deraining, whereas [30] focuses uniquely on the tasks
gate the use of transformers for multitask learning. of object detection and semantic segmentation and [40]
Although a few works have studied the use of transform- on scene recognition and importance score prediction in
ers to handle multiple input modalities, such as images and videos. Here, we cover a much wider range of high-level
text, they typically focus on a single task, e.g., visual ques- vision tasks and explicitly model their dependencies.
tion answering [19, 20, 23, 28], with the exception of [19], To this end, we introduce MulT, which consists of a
which tackles several language tasks but a single vision one. transformer-based encoder to transform the input image into
By contrast, our goal is to connect multiple vision tasks cov- a latent representation shared by the tasks, and transformer
ering the 2D, 3D, and semantic domains. To this end we decoders with task-specific heads producing the final pre-

12031
dictions for each of the tasks. While the MulT encoder recurrent-based encoder-decoder architectures. Since then
mainly utilizes the self-attention mechanism [3, 33] to ex- they have been widely applied to a great range of prob-
tract intrinsic features, as most transformers, we equip the lems, including speech recognition [16] and language mod-
decoders with a shared attention mechanism across the dif- eling [12,13]. In the vision domain, transformers have been
ferent vision tasks, thus allowing the overall framework to used to extract visual features, replacing CNNs for object
encode task dependencies. Thus, we leverage the query and detection, image classification, segmentation and video rep-
key vectors from the encoder along with the task-specific resentation learning [2, 5, 6,14, 18, 34,58]. Recently, several
values in the decoder to predict the task-specific outputs. works, such as UniT [19] and VILBERT-MT [23], have
Our contributions can be summarized as follows: learned multiple tasks from multimodal domains, such as
vision and text. Here, however, we focus on a single input
• We propose an end-to-end multitask transformer archi- modality: images.
tecture that handles multiple high-level vision tasks in
a single model. Learning task inter-dependencies. Taskonomy [54]
studied the relationships between multiple visual tasks for
• We introduce a shared attention between the trans- transfer learning and introduced a dataset with 4 million
former decoders of the multiple tasks. This shared at- images and corresponding labels for 26 tasks. Following
tention mechanism further improves the performance this, a number of recent works have further studied tasks
of each vision task. relationships for transfer learning [1, 15, 32, 46]. However,
these works differ from multitask learning, in the sense that
• Our framework lets us learn the inter-dependencies they analyze a network trained on a source task and applied
across high-level vision tasks. to a different target task, whereas we study the effect of
• We show that our model generalizes and adapts to new leveraging multiple tasks during training. In [41], Stand-
domains with a lower average error on the different ley et al. found notable differences between transfer task
vision tasks than the existing multitask convolutional affinity and multi-task affinity and showed the benefits of
models [41, 53]. leveraging structural similarities between tasks at all lev-
els for multitask learning. In this work, we further study
Our exhaustive experiments and analyses across a variety the task inter-dependencies, but by designing a multitask
of tasks show our MulT model not only improves the perfor- transformer model instead of a CNN one. Our MulT model
mance over single-task architectures, but also outperforms lets us learn the inter-dependencies across high-level vision
the state-of-the-art multitask CNN-based models (as shown tasks and further improves the task inter-dependencies seen
in Figure 1) on standard benchmarks, such as Taskon- in CNN-based models.
omy [54], Replica [42], NYU [31] and CocoDoom [29] .
Attention mechanisms. While there have been a myriad
of attention mechanisms [8, 11, 47, 48, 50, 51] to exploit
2. Related Work long range dependencies using transformers, none of the
Multitasking. In its most conventional form, multi-task prior works utilize a cross-task shared attention for multi-
learning predicts multiple outputs out of a shared en- task learning. This is what we propose in this work to han-
coder/representation for an input [55]. Prior works [24, 41, dle multiple vision tasks.
43, 53, 54] follow this architecture to jointly learn multiple
vision tasks using a CNN. Leveraging this encoder-decoder 3. MulT: A Multitask Transformer
architecture, IPT [9] was the first transformer-based mul-
Our model, MulT, follows the principle of a trans-
titask network aiming to solve low-level vision tasks af-
former encoder-decoder architecture [45]. It consists of
ter fine-tuning a large pre-trained network. This was fol-
a transformer-based encoder to map the input image to
lowed by [30], which jointly addressed the tasks of object
a latent representation shared by the tasks, followed by
detection and semantic segmentation. Recently, [40] used a
transformer decoders with task-specific heads producing the
similar architecture for scene and action understanding and
predictions for the respective tasks. Figure 2 shows an
score prediction in videos. However, none of these works
overview of our MulT framework. For our transformer-
connect such a wide range of vision tasks as we do, includ-
based encoder, we use a pyramidal backbone, named the
ing 2D, 3D, and semantic domains. Furthermore, they do
Swin Transformer [26] to embed the visual features into a
not explicitly model the dependencies between the tasks,
list of hidden states that incorporates global contextual in-
which we achieve via our shared attention mechanism.
formation. We then apply the transformer decoders to pro-
Transformers. Transformers [45] were originally intro- gressively decode and upsample the tokenized maps from
duced for language tasks, in particular for machine trans- the encoded image. Finally, the representation from the
lation where they showed impressive improvements over transformer decoder is passed to a task-specific head, such

122032
Figure 2. Detailed overview of our MulT architecture. Our MulT model builds upon the Swin transformer [26] backbone and models the
dependencies between multiple vision tasks via a shared attention mechanism (shown in the bottom left), which we introduce in this work.
The encoder module (in green) embeds a shared representation of the input image, which is then decoded by the transformer decoders (in
blue) for the respective tasks. Note that the transformer decoders have the same architecture but different task heads. The overall model is
jointly trained in a supervised manner using a weighted loss [10] of all the tasks involved. For clarity, only three tasks are depicted here.

as a simple two layer classifier (in the case of segmentation), mension at every intermediate stage, matching the behavior
which outputs the final predictions. Given the simplicity of of typical fully-convolutional backbones and producing a
MulT, it can be extended easily to more tasks. We empiri- feature pyramid (with output sizes of 1/4, 1/8, 1/16, 1/32
cally show that our model can jointly learn 6 different tasks of the original resolution) compatible with most previous
and generalizes well to new domains. The following sec- architectures for vision tasks.
tions describe the details of each component in MulT.
Following [17], most of the computation is concentrated
3.1. Encoder Module in the third stage: Out of a total of N = 24 transformer en-
coders, 2 blocks are in the first, second and fourth stage and
For the encoder, we adopt Swin-L [26], which applies 18 are in the third stage. In each block, the self-attention
stacked transformers to features of gradually decreasing res- is repeated according to the number of heads used and de-
olution in a pyramidal manner, hence producing hierarchi- pending on the stage of the encoding process. This is done
cal multi-scale encoded features, as shown in Figure 2. In to match the increase in the channel dimensions, where the
particular, following the ResNet [17] structure and design dimensions M = {6, 12, 24, 48} in the first, second, third
rules, four stages are defined in succession: each of them and fourth stage, respectively. However, the high resolution
contains a patch embedding step, which reduces the spa- in the first two stages does not allow the use of global self-
tial resolution and increases the channel dimension, and a attention, due to its quadratic complexity with respect to the
columnar sequence of transformer blocks. The initial ba- token sequence length. To solve this issue, in all stages, the
sic patch embedding in the first stage is performed with tokens, that are reshaped in a 2D representation, are divided
square patches of size pH = pW = 4 and with chan- into non-overlapping square windows of size h = w = 7,
nel size C = 192, without the addition of the ‘class’ to- and the intra-window self-attention is independently com-
ken; the patch merging in all three subsequent stages takes puted for each of them. This means that each token attends
the output tokens of the previous stage, reshapes them in to only the tokens in its own window, both as a query and as
a 2D representation and aggregates neighboring tokens in a key/value. A possible downside of this approach could be
non-overlapping patches of size pH = pW = 2 through that the restriction to fixed local windows completely stops
channel-wise concatenation and a linear transformation that any type of global or long-range interaction. The adopted
halves the resulting number of channels (hence doubles the solution is to alternate regular window partitioning with an-
number of channels with respect to the input tokens). This other non-overlapping partitioning in which the windows
approach halves the resolution and doubles the channel di- are shifted by half their size, ⌊h/2⌋ = ⌊w/2⌋ = 3, both

132033
in the height and width dimensions. This has the effect of let xt denote the upsampled output of the previous stage,
gradually increasing the virtual receptive field of the subse- and xsa the output of the encoding stage operating at the
quent attention computations. same resolution. As illustrated in Figure 3, the decoder
stage takes both xt and xsa as input. The standard way
3.2. Decoder Module to compute self-attention for task t would be to obtain the
Inspired by the two CNN-based decoders proposed key, query and value vectors from its own decoder output
in [56], we develop corresponding conceptually simi- xt only. By contrast, for our shared attention, we use only
lar transformer-based versions. The general idea is to one of the task streams to calculate the attention. That is,
r r
replace convolutional layers with windowed transformer we compute a query qsa and a key ksa from xsa (coming
blocks. Specifically, our decoder architecture consists of from the encoder) by using the linear layers, shown in Fig-
four stages, each containing a sequence of 2 transformer ure 3, of the decoder of one particular reference task r. To
blocks for a total of 8. In each stage, the two sequential nonetheless reflect the fact that the output of the decoder for
transformer blocks allow us to leverage inter-window con- task t should be related to this particular task, we compute
nectivity by alternating regular and shifted window config- the values v t using the previous stage output xt for task t.
urations as in the encoder. Between consecutive stages, we Thus, we compute attention values from the reference task
use an upsampling layer to double the spatial resolution and r as
half the channel dimension; we therefore adjust the num-
ber of attention heads accordingly to 48, 24, 12, 6, in the \label {eq:method_shared_attention} \begin {aligned} A^r_{sa}=\text {softmax}\left (\frac {q^r_{sa}.{k^r_{sa}}^T}{\sqrt {C^r_{qkv}}}+B^r\right ),\\ \end {aligned} (1)
first, second, third and fourth stage, respectively. The spa-
tial/channel shape of the resulting feature maps matches the
where C r is the number of channels and B r is the bias.
outputs of the encoder stages, which are delivered to the cor-
For any task t, we then compute x̃t = Arsa v t . This x̃t is
responding decoder stages by skip connections. This yields
then used by the self-attention head headti (., .) to compute
an hourglass structure with mirrored encoder-decoder com-
headti (x̃ti , Wit ) = x̃ti · Wit , where Wit is the learnt atten-
munication: the lower-resolution stages of the decoder are
tion weight for task t and x̃ti is the ith channel, respectively.
guided by the higher-level deeper encoded features and the
Note that this formulation represents the ith instance of the
higher-resolutions stages of the decoder are guided by the
self-attention, which is repeated M times to obtain a multi-
lower-level shallower encoded features, allowing to gradu-
head attention as MHAt (., .) for task t. Following which,
ally recover information in a coarse to fine manner and to
we compute xtlinear by linearly projecting the output of
exploit the different semantic levels where they are more
MHAt (., .). Finally, we obtain y t as follows:
relevant. Note that the first transformer block in each stage
of the decoder uses a regular window partitioning while the
\label {eq:method_shared_attention-MHA} \begin {aligned} &\text {MHA}^t(\tilde {x}^t,W)=\text {Concat}(\text {head}^t_1,\cdots , \text {head}^t_M)\mathbf {W}\;,\\ & x^t_{linear}= \text {MHA}^t(.,.)\;,\\ & y^t = x^t + x^t_{linear}\;, \end {aligned}
second uses a shifted window partitioning; this can easily be
extended to using a longer sequence of transformer blocks, (2)
as long as the length is a multiple of 2, which makes it pos-
sible to alternate between the two configurations.
To perform multitask prediction, we share the encoder where W indicates the multi-head attention weight. Em-
across all tasks and use task-specific decoders with the same pirically, we have found that the attention from the surface
architecture but different parameter values. We then simply normal task stream benefits our 6-task MulT model, and we
append task-specific heads to the decoder. For instance, a thus take this task as reference task r, whose attention is
model jointly trained for semantic segmentation and depth shared across the tasks. As shown in Figure 3, xr is the
prediction will have two task-specific heads: one predicting upsampled output of the previous stage of a particular de-
K channels followed by a softmax for semantic segmen- coder for the reference task, taken here as surface normal
tation and one predicting a single channel followed by a prediction.
sigmoid for depth estimation. Note that our shared attention differs from the co-
attention introduced in prior works [7], where the value and
3.3. Shared Attention key are passed via a skip connection from the encoder lay-
To account for the task dependencies beyond sharing en- ers. Figure 4 shows the effect of adding our shared atten-
coder parameters, we develop a shared attention mechanism tion mechanism across the tasks, where our MulT with the
that integrates the information contained in the encoded fea- shared attention mechanism improves the results across all
tures into the decoding stream. Let us now describe how this the tasks in comparison with our MulT model without the
mechanism works for one particular decoder stage. Note shared attention.
that we apply the same procedure for all decoder stages. Task Heads and Loss. The feature maps from the trans-
Formally, for one task t and one particular decoder stage, former decoder modules are input to different task-specific

142034
 with jointly learning prominent, high-level vision tasks.

4.1. Datasets
We evaluate MulT using the following datasets:
Taskonomy [54] is used as our main training dataset.
It comprises 4 million real images of indoor scenes with
multi-task annotations for each image. The experiments
were performed using the following 6 tasks: semantic seg-
mentation (S), depth (zbuffer) (D), surface normals (N ),
2D keypoints (K), 2D (Sobel) texture edges (E) and reshad-
ing (R). The tasks were selected to cover 2D, 3D, and
semantic domains and have sensor-based/semantic ground
truth. We report results on the Taskonomy test set.
Replica [42] comprises high-resolution 3D ground truth
Figure 3. Overview of our shared attention mechanism. and enables more reliable evaluations of fine-grained de-
tails. We test all the networks on 1227 images from Replica
(with and without fine-tuning).
NYU [31] comprises 1449 images from 464 different in-
door scenes. We test all the networks on NYU (with and
without fine-tuning).
CocoDoom [29] contains synthetic images from the
Doom video game. We use it as an out-of-training-
distribution dataset.

4.2. Training Details

We jointly train MulT on multiple tasks, including, se-
mantic segmentation, depth estimation, 2D keypoint de-
Figure 4. Motivation of the shared attention mechanism on tection, 2D edge detection, surface normal estimation and
our MulT model. The shared attention mechanism learns the task reshading. In our implementation, we train with a batch
inter-dependencies and improves the prediction for each task. For size of 32 on 32 Nvidia V100-SXM2-32GB GPUs in a dis-
instance, the yellow circled region shows how our MulT model tributed fashion, using PyTorch. We use the weighted Adam
with shared attention across tasks improves the semantic segmen- optimizer [27] with a learning rate of 5e-5 and the warm-up
tation performance, where the chair mask is correctly classified
cosine learning rate schedule (using 2000 warm-up itera-
in our predictions as in the ground truth. However in the MulT
tions). The optimizer updates the model parameters based
model without the shared attention, the chair is predicted as a
couch mask. Best viewed on screen and when zoomed in. on gradients from the task losses.

4.3. Baselines
heads to make subsequent predictions. Each class head in-
cludes a single linear layer to output a H × W × 1 map, We compare our MulT model with the following state-
where H, W are the input image dimensions. We employ a of-the-art baselines.
weighted sum [10] based task-specific losses to jointly train Baseline UNet (for single-task or independent learning)
the network, where the losses are calculated between the constitutes our CNN-based baseline. We use it as a refer-
ground truth and final predictions for each task. In particu- ence for all the multitask models.
lar, we use cross-entropy for segmentation, rotate loss [54] Baseline Swin transformer [26] (for single-task or inde-
for depth, and L1 loss for surface normals, 2D keypoints, pendent learning) constitutes the single task transformer
2D edges and reshading, respectively. Note that we em- baseline. It is almost identical to our MulT model, except
ploy these losses to maintain consistency with the base- for not including shared attention and for being trained with
lines [41, 53, 54]. only one dedicated task. We use it to evaluate the benefits
of our multitask learning strategy
4. Experiments and Results Multi-task learning [22] (MTL) comprises a network
with one shared encoder and multiple decoders each ded-
To provide a thorough analysis of MulT and also com- icated to a task. This baseline further identifies if tasks are
pare it with well-established prior work, we experiment inter-dependent, such that a shared representation can give

152035
comparable performance across multiple tasks, without ex- increasing the size of the decoder modules. Altogether, re-
plicitly adding task constraints. porting results for all possible task combinations requires
Taskonomy [54] studies the relationships between multi- training (26 − 1) models. Here, we focus on the 6-task case,
ple visual tasks for transfer learning. but provide 3-task, 4-task, and 5-task results in the supple-
Taskgrouping [41] studies task compatibility in multi- mentary material. The results of our 6-task MulT model
task learning, thus providing a framework for determining and of the baselines are reported in Table 2 and Table 3
which tasks should be trained jointly and which tasks should for the Taskonomy test set [54], and the Replica [42] and
be trained separately. NYU [31] dataset, respectively. Our MulT model outper-
Cross-task consistency [53] presents a general and data- forms the multitask CNN baselines as well as the 1-task
driven framework for augmenting standard supervised CNN and Swin ones. Furthermore, as can be verified from
learning with cross-task consistency. It is inspired from the results in the supplementary material, increasing the
Taskonomy [54] but adds a consistency constraint to learn number of tasks improves the results of our MulT model,
multiple tasks jointly. e.g., a 6-task network outperforms a 5-task one, which in
Note that we do not compare our method with the con- turn outperforms a 4-task network.
temporary work [19] as it focuses on bimodal multitask
4.5. Qualitative Results
learning for vision- and language-related tasks. By con-
trast, in this work, we tackle unimodal multitask learn- We qualitatively compare the results of our MulT model
ing for high-level vision tasks. All the multitask base- with different CNN-based multitask baselines [22, 41, 53,
lines were trained using their best model configurations as 54], as well as with the single task dedicated Swin trans-
in [22, 41, 53, 54], respectively. former [26]. The results in Figure 5 show the performance
of the different networks on all six vision tasks. All the
Relative Performance On multitasking models are jointly trained on the six tasks on
S D N K E R
the Taskonomy benchmark, and the single task dedicated
S - +3.83% -1.42% -1.33% +33.9% -0.80%
D +4.83% - +2.77% -1.92% +35.2% +3.93% Swin models are trained on the respective tasks. Our MulT
N +11.3% +8.35% - +91.2% +77.1% +9.09% model yields higher-quality predictions than both the single
K +5.11% +0.57% -6.88% - +70.1% +0.21% task Swin baselines and the multitask CNN baselines. We
E +6.09% +4.33% -0.73% +4.75% - +5.11% provide additional qualitative results in the supplementary
R +8.61% +4.45% +5.91% +1.95% +33.9% - material.
Table 1. Quantitative comparison of our MulT model with
a single-task dedicated Swin transformer baseline [26]. Our 4.6. Generalization to New Domains
MulT model is jointly trained in a pairwise manner on the Taskon-
omy benchmark [54]. For instance, in the first row, second col- In this section, we demonstrate how well MulT general-
umn, we show the results of our MulT model trained with seman- izes to new domains without any fine-tuning, and how ef-
tic segmentation and depth in a pairwise manner, and tested on ficiently MulT can adapt to a new domain by fine-tuning
the task of depth estimation. The relative performance percentage on a small set of training examples from the new domain.
for each task is evaluated by taking the percentage increase or de- To this end, we compare our MulT model and the two
crease w.r.t. the single-task baseline. The results here are reported baselines of Taskgrouping (TG) [41] and Cross-task con-
on the Taskonomy test set. The columns show the task tested on, sistency (CT) [53] on two new domains, namely, Gaussian-
and the rows show the other task used for training. blurred images from Taskonomy [21] and images from the
Cocodoom [29] dataset. Note that all the networks were
4.4. Quantitative Results
trained on the vanilla Taskonomy dataset [54]. When fine-
The results in Table 1 show the relative performance of tuning the networks, we use either 16 or 128 images from
our MulT model when trained on pairs of tasks and tested on the new domain. The original training data (Taskonomy)
one of the two tasks. We observe that, out of the pairwise- is retained during fine-tuning to prevent the networks from
trained multitask models, surface normals help the other vi- forgetting the original domain.
sion tasks. However, the performance of normals tends to The results in Table 4 and Figure 6 show that our MulT
decrease w.r.t. its single task dedicated model, except when model yields better generalization and adaptation to new
used in conjunction with either depth predition or reshading. domains, both with and without fine-tuning. These find-
Note that the trends we observe are similar to those shown ings confirm the observations made in [4] for the single-
in [41] for the CNN case. This suggests that transformers task scenario. The cross-task consistency [53] model shows
follow a similar behavior to that of CNNs in the presence of improved performance in comparison to the Taskgroup-
multiple tasks. ing [41] baseline because of its explicitly enforced consis-
In cases of more than two tasks, we observe, as in [41], tency constraint, whereas the Taskgrouping model [41] suf-
that effectively leveraging between 3 and 6 tasks required fers due to the joint task pairings and the lack of an attention

162036
 Relative Performance On
Taskonomy Test Set [54]
S D N K E R
MTL [22] vs 1-task CNN [39] +2.05% +3.11% +4.38% -1.29 % +45.22% +2.99%
Taskonomy [54] vs 1-task CNN [39] +2.63% -3.82 % +2.95% +10.13 % +59.05% +4.52%
Taskgrouping [41] vs 1-task CNN [39] +6.24% +3.36% +4.23% +21.77% +73.6 % +5.79%
Cross-task [53] vs 1-task CNN [39] +9.01% +6.77% +5.61% +23.20% +75.8 % +11.1%
MulT vs 1-task Swin [26] +19.7% +10.2% +8.72% +94.75% +88.8% +16.4%
MulT vs 1-task CNN [39] +21.6% +11.5% +9.71% +97.04% +92.9% +21.0%
Table 2. Quantitative comparison of our MulT model with baselines when jointly trained for six tasks on the Taskonomy bench-
mark [54]. Our six-task MulT model consistently outperforms all the baselines, including the multitasking CNN baselines and the single-
task CNN and Swin baselines. The relative performance percentage for each task is evaluated by taking the percentage increase or decrease
w.r.t. the single-task baseline. The results here are reported on the Taskonomy test set. Bold and underlined values show the best and
second-best results, respectively.
Relative Performance On
Replica Dataset [42] NYU Dataset [31]
D N R S D
MTL [22] vs 1-task CNN [39] +2.53% +3.03% +1.87% +1.13% +2.72%
Taskonomy [54] vs 1-task CNN [39] -4.55% +1.99% +3.33% +2.05% -4.07%
Taskgrouping [41] vs 1-task CNN [39] +2.75% +4.09% +5.47% +6.01% +2.91%
Cross-task [53] vs 1-task CNN [39] +5.10% +4.33% +9.55% +8.10% +5.71%
MulT vs 1-task Swin [26] +8.33% +7.05% +14.2% +13.3% +8.54%
MulT vs 1-task CNN [39] +10.1% +8.59% +19.6% +15.7% +10.4%
Table 3. Quantitative comparison of our MulT model with baselines on the Replica benchmark and the NYU benchmark. We apply
our MulT model, jointly trained on 6 tasks on the Taskonomy dataset, to test the depth, normals and reshading prediction performances
on the Replica dataset [42], and the segmentation and depth prediction performance on the NYU dataset [31]. Our six-task MulT model
consistently outperforms all the baselines, including the multitasking CNN baselines and the single-task CNN and Swin baselines. The
relative performance percentage for each task is evaluated by taking the percentage increase or decrease w.r.t. the single-task baseline.
Bold and underlined values show the best and second-best results, respectively.

mechanism or an additional constraint. Nevertheless, our a single end-to-end encoder-decoder framework. Our MulT
MulT model outperforms both these baselines and shows model simultaneously addresses 6 different vision tasks,
better generalization. learning them in a single training step and outperforming an
independent single task model on each task with a compact
Generalization to New Domains set of shared parameters. This allows us to use a single net-
No. of Error (w/ Fine-tuning)↓ Error (w/o Fine-tuning)↓
Domains images MulT CT [53] TG [41] MulT CT [53] TG [41] work to handle multiple vision tasks instead of multiple sin-
Blur [21] 128 12.6 17.4 21.9 gle task networks, thereby reducing the computational cost,
27.0 46.2 55.1
(Taskonomy) 16 17.5 22.2 26.3 for both training and inference. Furthermore, our MulT
CocoDoom 128 13.3 18.5 25.3 model outperforms the state-of-the-art CNN-based multi-
39.3 54.3 67.7
[29] 16 20.9 27.1 39.9
tasking models, in terms of both performance in the original
Table 4. Domain generalization on Taskonomy blur data [21] domain and generalization/adaptation to new domains.
and CocoDoom [29]. Our MulT model shows better abilities to
Our current framework nonetheless suffers from some
generalize and adapt to new domains, both with and without fine-
limitations:
tuning. Bold and underlined values show the best and second-best
results, respectively.
Data dependency. Although we validated our findings
using various architectures and benchmarks, the results of
Supplementary Material. We defer additional discus-
our approach, as any deep learning one, are in principle data
sions and experiments, particularly analyzing the effect of
specific. In particular, MulT is a data intensive architecture,
the shared attention in our MulT model and the effect of
and thus when trained on a limited amount of data, it may
the network size for different task combinations, as well as
not achieve the same performance as reported in this work.
additional qualitative results to the supplementary material.
Note, however, that this is also the case for both single task
We also analyze the number of parameters required by each
transformers and the CNN-based multitask baselines.
model and the environmental impact of training such mod-
Unpaired Data. Our current framework, as the CNN-
els in the supplementary material.
based multitask baselines, requires paired training data. Ex-
5. Conclusion and Limitations tending our approach to unlabeled/unpaired data, as in [57],
appears feasible and remains open for future work.
In this work, we have shown that the transformer frame- Modeling efficient attention. Our current framework
work can be applied to jointly handle multiple tasks within makes use of shared attention across the visual tasks. Ex-

172037
Figure 5. Qualitative comparison on the six vision tasks of the Taskonomy benchmark [54]. From top to bottom, we show qualitative
results using MTL [22], Taskonomy [54], Taskgrouping [41], Cross-task consistency [53], the single-task dedicated Swin transformer [26]
and our six-task MulT model. We show, from left to right, the input image, the semantic segmentation results, the depth predictions, the
surface normal estimations, the 2D keypoint detections, the 2D edge detections and the reshading results for all the models. All models
are jointly trained on the six vision tasks, except for the Swin transformer baseline, which is trained on the independent single tasks. Our
MulT model outperforms both the single task Swin baselines and the multitask CNN based baselines. Best seen on screen and zoomed
within the yellow circled regions.

Figure 6. Generalization to new domains. Our MulT model generalizes better to new domains than the Cross-task [53] baseline,
both when fine-tuned and not fine-tuned, across the tasks of surface normal prediction and reshading. This shows the benefits of our
shared attention module. We test the models on two target domains, Gaussian blur applied to the Taskonomy images [54] and the out-of-
distribution CocoDoom dataset [29]. Best viewed on screen and when zoomed in the yellow circled regions.

tending this concept to incorporate local versus global at- tasks. In addition, it would be worthwhile to explore the ro-
tention, as in [51], appears feasible and remains open for bustness of large-scale multitask transformers to adversarial
future work. tasks, which could become increasingly problematic as the
number and variety of tasks grow.
Besides addressing these limitations, in the future, we
plan to extend our methodology to learning different types Acknowledgement. This work was supported
of tasks like edge occlusions, principal curvatures and unsu- in part by the Swiss National Science Foun-
pervised segmentation, and doing zero-shot learning on new dation via the Sinergia grant CRSII5−180359.

182038
 Mostafa Dehghani, Matthias Minderer, Georg Heigold, Syl-
vain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is
References worth 16x16 words: Transformers for image recognition at
scale. In International Conference on Learning Representa-
[1] Alessandro Achille, Michael Lam, Rahul Tewari, Avinash tions, 2021. 1, 2
Ravichandran, Subhransu Maji, Charless Fowlkes, Stefano [15] Kshitij Dwivedi and Gemma Roig. Representation similar-
Soatto, and Pietro Perona. Task2vec: Task embedding for ity analysis for efficient task taxonomy & transfer learning.
meta-learning. arXiv:1902.03545, [Link], 2019. 2 arXiv:1904.11740, [Link], 2019. 2
[2] Farhat Afza, Muhammad Sharif, Muhammad Attique Khan, [16] Anmol Gulati, James Qin, Chung-Cheng Chiu, Niki Par-
Usman Tariq, Hwan-Seung Yong, and Jaehyuk Cha. Multi- mar, Yu Zhang, Jiahui Yu, Wei Han, Shibo Wang, Zheng-
class skin lesion classification using hybrid deep features se- dong Zhang, Yonghui Wu, and Ruoming Pang. Conformer:
lection and extreme learning machine. Sensors, 22(3), 2022. Convolution-augmented transformer for speech recognition.
2 arXiv:2005.08100, [Link], 2020. 2
[3] Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. [17] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian
Neural machine translation by jointly learning to align and Sun. Deep residual learning for image recognition.
translate. arXiv: 1409.0473, [Link], 2016. 2 arXiv:1512.03385, 2015. 3
[4] Yutong Bai, Jieru Mei, Alan Yuille, and Cihang Xie. Are [18] Jie Hu, Li Shen, Samuel Albanie, Gang Sun, and Andrea
transformers more robust than cnns? arXiv: 2111.05464, Vedaldi. Gather-excite: Exploiting feature context in convo-
[Link], 2021. 6 lutional neural networks. In Proceedings of NeurIPS, 2018.
[5] Irwan Bello, Barret Zoph, Ashish Vaswani, Jonathon Shlens, 1, 2
and Quoc V. Le. Attention augmented convolutional net- [19] Ronghang Hu and Amanpreet Singh. Unit: Multimodal
works. In Proceedings of the IEEE/CVF International Con- multitask learning with a unified transformer. arXiv:
ference on Computer Vision (ICCV), 2019. 1, 2 2102.10772, [Link], 2021. 1, 2, 6
[6] Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas [20] Ronghang Hu, Amanpreet Singh, Trevor Darrell, and Mar-
Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to- cus Rohrbach. Iterative answer prediction with pointer-
end object detection with transformers. arXiv: 2005.1287, augmented multimodal transformers for textvqa. CoRR,
[Link], 2020. 1, 2 abs/1911.06258, 2019. 1
[7] Hila Chefer, Shir Gur, and Lior Wolf. Transformer inter- [21] Jason Jo and Yoshua Bengio. Measuring the ten-
pretability beyond attention visualization. In Proceedings of dency of cnns to learn surface statistical regularities.
the IEEE/CVF Conference on Computer Vision and Pattern arXiv:1711.11561, [Link], 2017. 6, 7
Recognition (CVPR), pages 782–791, June 2021. 4 [22] Iasonas Kokkinos. Ubernet: Training a ‘universal’ convo-
[8] Chun-Fu Chen, Rameswar Panda, and Quanfu Fan. Re- lutional neural network for low-, mid-, and high-level vi-
gionvit: Regional-to-local attention for vision transformers. sion using diverse datasets and limited memory. arXiv:
arXiv:2106.02689, [Link], 2021. 2 1609.02132, [Link], 2016. 5, 6, 7, 8
[9] Hanting Chen, Yunhe Wang, Tianyu Guo, Chang Xu, Yiping [23] Liunian Harold Li, Mark Yatskar, Da Yin, Cho-Jui Hsieh,
Deng, Zhenhua Liu, Siwei Ma, Chunjing Xu, Chao Xu, and and Kai-Wei Chang. Visualbert: A simple and performant
Wen Gao. Pre-trained image processing transformer. arXiv: baseline for vision and language. arXiv: 1908.03557, [Link],
2012.00364, [Link], 2021. 1, 2 2019. 1, 2
[10] Zhao Chen, Vijay Badrinarayanan, Chen-Yu Lee, and An- [24] Shikun Liu, Edward Johns, and Andrew J. Davison.
drew Rabinovich. GradNorm: Gradient normalization for End-to-end multi-task learning with attention. CoRR,
adaptive loss balancing in deep multitask networks. In Pro- abs/1803.10704, 2018. 2
ceedings of the 35th International Conference on Machine [25] Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar
Learning, volume 80 of Proceedings of Machine Learning Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettle-
Research, pages 794–803. PMLR, 2018. 3, 5 moyer, and Veselin Stoyanov. Roberta: A robustly optimized
[11] Xiangxiang Chu, Zhi Tian, Yuqing Wang, Bo Zhang, Haib- bert pretraining approach. arXiv preprint arXiv:1907.11692,
ing Ren, Xiaolin Wei, Huaxia Xia, and Chunhua Shen. 2019. 1
Twins: Revisiting the design of spatial attention in vision [26] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei,
transformers. In NeurIPS 2021, 2021. 2 Zheng Zhang, Stephen Lin, and Baining Guo. Swin trans-
[12] Zihang Dai*, Zhilin Yang*, Yiming Yang, William W. Co- former: Hierarchical vision transformer using shifted win-
hen, Jaime Carbonell, Quoc V. Le, and Ruslan Salakhutdi- dows. arXiv: 2103.14030, [Link], 2021. 1, 2, 3, 5, 6, 7,
nov. Transformer-XL: Language modeling with longer-term 8
dependency. arXiv, [Link], 2019. 2 [27] Ilya Loshchilov and Frank Hutter. Fixing weight decay reg-
[13] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina ularization in adam. Openreview, 2018. 5
Toutanova. Bert: Pre-training of deep bidirectional trans- [28] Jiasen Lu, Vedanuj Goswami, Marcus Rohrbach, Devi
formers for language understanding. arXiv:1810.04805, Parikh, and Stefan Lee. 12-in-1: Multi-task vision and lan-
[Link], 2019. 1, 2 guage representation learning. In The IEEE/CVF Conference
[14] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, on Computer Vision and Pattern Recognition (CVPR), June
Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, 2020. 1

192039
[29] A. Mahendran, H. Bilen, J. F. Henriques, and A. Vedaldi. Re- [44] Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco
searchdoom and cocodoom: Learning computer vision with Massa, Alexandre Sablayrolles, and Herve Jegou. Training
games, copyright(c) 2014, coco api. arXiv: 1610.02431, data-efficient image transformers; distillation through atten-
[Link], 2016. 2, 5, 6, 7, 8 tion. In Proceedings of the 38th International Conference
[30] Eslam Mohamed and Ahmed El-Sallab. Spatio-temporal on Machine Learning, volume 139, pages 10347–10357.
multi-task learning transformer for joint moving object de- PMLR, 2021. 1
tection and segmentation. arXiv: 2106.11401, [Link], 2021. [45] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszko-
1, 2 reit, Llion Jones, Aidan N Gomez, Ł ukasz Kaiser, and Illia
[31] Pushmeet Kohli Nathan Silberman, Derek Hoiem and Rob Polosukhin. Attention is all you need. In Advances in Neural
Fergus. Indoor segmentation and support inference from Information Processing Systems, volume 30, 2017. 1, 2
rgbd images. In ECCV, CC-BY 4.0, 2012. 2, 5, 6, 7 [46] Aria Wang, Michael Tarr, and Leila Wehbe. Neural taskon-
[32] Arghya Pal and Vineeth N. Balasubramanian. Zero-shot task omy: Inferring the similarity of task-derived representations
transfer. CoRR, abs/1903.01092, 2019. 2 from brain activity. In Advances in Neural Information Pro-
[33] Ankur Parikh, Oscar Täckström, Dipanjan Das, and Jakob cessing Systems, volume 32, 2019. 2
Uszkoreit. A decomposable attention model for natural [47] Wenhai Wang, Enze Xie, Xiang Li, Deng-Ping Fan, Kaitao
language inference. In Proceedings of the 2016 Confer- Song, Ding Liang, Tong Lu, Ping Luo, and Ling Shao.
ence on Empirical Methods in Natural Language Process- Pvtv2: Improved baselines with pyramid vision transformer.
ing, pages 2249–2255. Association for Computational Lin- arXiv:2106.13797, [Link], 2021. 2
guistics, 2016. 2 [48] Wenxiao Wang, Lu Yao, Long Chen, Deng Cai, Xiaofei He,
[34] Niki Parmar, Ashish Vaswani, Jakob Uszkoreit, Lukasz and Wei Liu. Crossformer: A versatile vision transformer
Kaiser, Noam Shazeer, Alexander Ku, and Dustin Tran. Im- based on cross-scale attention. arXiv:2108.00154, [Link],
age transformer. In Proceedings of the 35th International 2021. 2
Conference on Machine Learning, volume 80 of Proceedings [49] Xiaolong Wang, Ross Girshick, Abhinav Gupta, and Kaim-
of Machine Learning Research, pages 4055–4064. PMLR, ing He. Non-local neural networks. In Proceedings of the
2018. 1, 2 IEEE Conference on Computer Vision and Pattern Recogni-
[35] Alec Radford and Ilya Sutskever. Improving language un- tion (CVPR), June 2018. 1
derstanding by generative pre-training. In arxiv, 2018. 1 [50] Weijian Xu, Yifan Xu, Tyler Chang, and Zhuowen
[36] Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Tu. Co-scale conv-attentional image transformers.
Amodei, Ilya Sutskever, et al. Language models are unsu- arXiv:2104.06399, [Link], 2021. 2
pervised multitask learners. OpenAI blog, 1(8):9, 2019. 1 [51] Jianwei Yang, Chunyuan Li, Pengchuan Zhang, Xiyang Dai,
[37] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Bin Xiao, Lu Yuan, and Jianfeng Gao. Focal self-attention
Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and for local-global interactions in vision transformers. arXiv:
Peter J. Liu. Exploring the limits of transfer learning with a 2107.00641, [Link], 2021. 2, 8
unified text-to-text transformer. Journal of Machine Learn- [52] Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell,
ing Research, 21(140):1–67, 2020. 1 Russ R Salakhutdinov, and Quoc V Le. Xlnet: General-
[38] Prajit Ramachandran, Niki Parmar, Ashish Vaswani, Irwan ized autoregressive pretraining for language understanding.
Bello, Anselm Levskaya, and Jonathon Shlens. Stand-alone In Advances in Neural Information Processing Systems, vol-
self-attention in vision models. arXiv: 1906.05909, [Link], ume 32. Curran Associates, Inc., 2019. 1
2019. 1 [53] Amir Zamir, Alexander Sax, Teresa Yeo, Oğuzhan Kar,
[39] Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Nikhil Cheerla, Rohan Suri, Zhangjie Cao, Jitendra Malik,
Convolutional networks for biomedical image segmentation. and Leonidas Guibas. Robust learning through cross-task
arXiv:1505.04597, [Link], 2015. 7 consistency. arXiv, 2020. 2, 5, 6, 7, 8
[40] Hongje Seong, Junhyuk Hyun, and Euntai Kim. Video mul- [54] Amir R. Zamir, Alexander Sax, William B. Shen, Leonidas J.
titask transformer network. In 2019 IEEE/CVF International Guibas, Jitendra Malik, and Silvio Savarese. Taskonomy:
Conference on Computer Vision Workshop (ICCVW), pages Disentangling task transfer learning. In IEEE Conference
1553–1561, 2019. 1, 2 on Computer Vision and Pattern Recognition (CVPR). IEEE
[41] Trevor Standley, Amir R. Zamir, Dawn Chen, Leonidas (MIT License Copyright (c) 2017 Stanford Vision and Learn-
Guibas, Jitendra Malik, and Silvio Savarese. Which tasks ing Group), 2018. 2, 5, 6, 7, 8
should be learned together in multi-task learning? arXiv: [55] Yu Zhang and Qiang Yang. A survey on multi-task learning.
1905.07553, [Link], 2019. 1, 2, 5, 6, 7, 8 arXiv:1707.08114, [Link], 2021. 2
[42] Julian Straub, Thomas Whelan, Lingni Ma, Yufan Chen, and [56] Sixiao Zheng, Jiachen Lu, Hengshuang Zhao, Xiatian Zhu,
Erik Wijmans et. al. The Replica dataset: A digital replica Zekun Luo, Yabiao Wang, Yanwei Fu, Jianfeng Feng, Tao
of indoor spaces. arXiv preprint arXiv:1906.05797, CC-BY Xiang, Philip H.S. Torr, and Li Zhang. Rethinking semantic
4.0, 2019. 2, 5, 6, 7 segmentation from a sequence-to-sequence perspective with
[43] Gjorgji Strezoski, Nanne van Noord, and Marcel Worring. transformers. In CVPR, 2021. 4
Many task learning with task routing. arXiv:1903.12117, [57] Jun-Yan Zhu, Taesung Park, Phillip Isola, and Alexei A.
2019. 2 Efros. Unpaired image-to-image translation using cycle-

10
12040
 consistent adversarial networks. arXiv: 1703.10593, [Link],
2020. 7
[58] Xizhou Zhu, Weijie Su, Lewei Lu, Bin Li, Xiaogang Wang,
and Jifeng Dai. Deformable detr: Deformable transformers
for end-to-end object detection. arXiv: 2010.04159, [Link],
2021. 1, 2

11
12041