Toward a Diffusion-Based Generalist for Dense Vision Tasks

Yue Fan1,2 *, Yongqin Xian2 , Xiaohua Zhai3 , Alexander Kolesnikov3 ,

Muhammad Ferjad Naeem4 , Bernt Schiele1 , Federico Tombari2,5
1
Max Planck Institute for Informatics, Saarland Informatics,
2
Google, 3 Google DeepMind, 4 ETH Zurich, 5 TU Munich
arXiv:2407.00503v1 [[Link]] 29 Jun 2024

Prompt: Prompt: Prompt: Prompt:

“Perform “Perform “Perform “Perform Image
semantic Depth instance restoration
segmentation” estimation” segmentation” deraining”

Diffusion Generalist
step=0 step=25 step=50 step=0 step=25 step=50

Denoising Process Denoising Process

Figure 1. We present a diffusion-based vision generalist for dense vision tasks. Given an input image, the model performs the corresponding
task following the text instruction. We showcase the effectiveness of our model on depth estimation, semantic segmentation, panoptic
segmentation, and three types of image restoration tasks. The images are the actual output of our model.

Abstract 1. Introduction

Building generalized models that can solve many com- The field of artificial intelligence has made significant
puter vision tasks simultaneously is an intriguing direc- progress in building generalized model frameworks. In
tion. Recent works have shown image itself can be used particular, autoregressive transformers [27] have become a
as a natural interface for general-purpose visual percep- prominent unified approach in Natural Language Process-
tion and demonstrated inspiring results. In this paper, we ing (NLP), effectively addressing a wide range of tasks with
explore diffusion-based vision generalists, where we unify a singular model architecture [10, 19, 21, 25]. However,
different types of dense prediction tasks as conditional im- in computer vision (CV), building a unified framework re-
age generation and re-purpose pre-trained diffusion models mains challenging due to the inherent diversity of the tasks
for it. However, directly applying off-the-shelf latent diffu- and output formats. Consequently, state-of-the-art com-
sion models leads to a quantization issue. Thus, we pro- puter vision models still have many complex task-specific
pose to perform diffusion in pixel space and provide a recipe designs [3, 8, 9, 15, 31], making it difficult for feature shar-
for finetuning pre-trained text-to-image diffusion models for ing across tasks and, thus, limiting knowledge transfer.
dense vision tasks. In experiments, we evaluate our method The stark contrast between NLP and CV has given rise to
on four different types of tasks and show competitive per- a growing interest in developing unified approaches for vi-
formance to the other vision generalists. sion tasks [6, 7, 18, 29, 30, 35]. Recently, [29, 30] have
shown image itself can be used as a robust interface for
unifying different vision tasks and demonstrated good per-
* Intern at Google during the project. formance. In this paper, we propose a multi-task diffusion
generalist for dense vision tasks by reformulating the dense available image-text pairs. Painter [29] and SegGPT [30],
prediction tasks as conditional image generation, and re- on the other hand, reformulate different vision tasks as an
purpose pre-trained latent diffusion models for it. Fig. 1 image inpainting problem, and perform in-context learning
visualizes the output of our model on semantic segmenta- following [2]. Unlike the previous work, our method unifies
tion, panoptic segmentation, depth estimation, and image different vision tasks under a conditional image generation
restoration. Based on text prompts, our model can per- framework and introduces a diffusion-based vision general-
form different tasks with one set of parameters. However, ist for it.
directly finetuning the pre-trained latent diffusion models Unified framework & Task-specific model: Besides
(e.g. Stable Diffusion [22]) leads to quantization errors for the aforementioned literature, there is another line of re-
segmentation tasks (see Table 2). To this end, we propose lated works that pursue unified architecture but task-specific
to do pixel-space diffusion which effectively improves the models. UViM [14] addressed the high-dimensionality out-
generation quality and does not suffer from quantization er- put space of vision tasks via learned guiding code, where a
rors. Moreover, our exploration into training diffusion mod- short sequence modeled by an additional language model
els as vision generalists reveals a list of interesting findings to encode task-specific information guides the prediction
as follows: of the base model. Separate models are trained for differ-
• Diffusion-based generalists show superior performance ent tasks as the guiding code is task-specific. XDecoder
over the non-diffusion-based generalists on tasks involv- [36] unified pixel-level image segmentation, image-level re-
ing semantics or global understanding of the scene. trieval, and vision-language tasks with a generic decoding
• We find conditioning on the image feature extracted from procedure, which predicts pixel-level masks and token-level
powerful pre-trained image encoders results in better per- semantics, and different combinations of the two outputs are
formance than directly conditioning on the raw image. used for different tasks. Despite their good performance,
• Pixel diffusion is better than latent diffusion as it does not the task/modality-specific customization poses difficulty for
have the quantization issue while upsampling. knowledge sharing among different tasks and is also not
• We observe that text-to-image generation pre-training sta- friendly for supporting unseen tasks.
bilizes the training and leads to better performance.
In experiments, we demonstrate the model’s versatility 3. Toward a Diffusion-Based Generalist
across six different dense prediction tasks on depth esti-
mation, semantic segmentation, panoptic segmentation, im- 3.1. Unification with Conditional Image Generation
age denoising, image draining, and light enhancement. Our As the output of most vision tasks can be always visualized
method achieves competitive performance to the current as images, we redefine the output space of different vision
state-of-the-art in many settings. tasks as RGB images and unify them as conditional image
generation to tackle the inherent difference of output for-
2. Related Work mats of different vision tasks. Given a input image x and
Unified framework & Unified model: Efforts have been the corresponding ground-truth y, we first transform y into
made to unify various vision tasks with a single model, re- RGB images and then pair it with a task descriptor in text.
sulting in several vision generalists [6, 7, 14, 18, 29, 30]. In- By doing so, training sets of different tasks are combined
spired by the success of sequence-to-sequence modeling in into a holistic training set. And training the model jointly
Natural Language Processing (NLP), Pix2Seq [6, 7] lever- on it enables the knowledge transfer between tasks. At test
ages a plain autoregressive transformer and tackles many vi- time, given a new image, the model can perform different
sion tasks with next-token prediction. For example, bound- tasks following the text instructions (examples in Fig. 1).
ing boxes in object detection are cast as sequences of dis- In this paper, we consider four types of dense prediction
crete tokens, and masks in semantic segmentation are en- tasks: depth estimation, semantic segmentation, panoptic
coded with coordinates of object polygons [4]. The idea was segmentation, and image restoration.
further developed in Unified-IO [18], where dense predic- Depth estimation outputs real number depth value for each
tion such as segmentation, depth map, and image restora- pixel on x. Given the minimum and the maximum values,
tion are also unified as tokens by using the correspond- we map them into [0, 255] linearly and discretize them into
ing image features from a vector quantization variational integers, which is then repeated and stacked along the chan-
auto-encoder (VQ-VAE) [26]. On the output side, the pre- nel to form the ground-truth RGB label.
dicted image tokens are then decoded into masks and depth Semantic segmentation predicts a class label for each
maps as the final prediction. Similarly, OFA [28] unified pixel. We use a pre-defined injective class-to-color map-
a diverse set of cross-modal and unimodal tasks in a sim- ping to transform the segmentation mask into RGB im-
ple sequence-to-sequence learning framework and achieved ages. Given a task with C categories, we define C col-
competitive performance pretrained with only 20M publicly ors which are evenly distributed in the 3-dimensional RGB
 Data Reformation 𝑿𝟎 Diffusion Generalist

Target: 𝑿𝒕

Image
Input: concat. Denoising UNet MSE
Encoder

“Perform semantic segmentation.” “semantic segmentation”

Text
Instruction: “Perform panoptic segmentation.” “panoptic segmentation”
“depth estimation” Encoder
“Perform depth estimation.”

Figure 2. The training pipeline of the diffusion-based vision generalist consists of two parts: Left: Redefining the output space of different
vision tasks as RGB images so that they can be unified under a conditional image generation framework. Right: We finetune a pre-trained
diffusion model on the reformatted data from the first step. Diffusion is performed in the pixel space to mitigate the quantization error of
the latent diffusion (see Table 2). The image and text conditionings are fed into the model via the corresponding encoders, where only the
image encoder is tuned during the training.

space. Specifically, following [29], the class index is rep- RGB image format Xi and augment each task with a text
resented by a 3-digit number with b-base system, where instruction T i . At each training step, we randomly sample
1
b = ⌈C 3 ⌉. Thus, the margin between two colors is defined a subset of tasks and then sample data from each task. For
as int( b ). The color for the i-th class is then [int( bi2 ) × m,
256
each input data {I i , X i , T i }, we first compute the multi-
int( bi )%b × m, l%b × m]. At test time, we find the nearest scale feature map of the original image I i from the image
neighbor of the predicted color in the predefined class-to- encoder. Then, it is concatenated with the noised target im-
color mapping and predict the corresponding category. age Xti before being fed into the UNet for the reconstruc-
Panoptic segmentation is solved as a combination of se- tion loss. Note that the image feature can have a different
mantic and instance segmentation. Semantic segmentation spatial resolution than the target image Xti , in which case
labels are constructed as stated above. For instance seg- the concatenation will be performed on the interpolated im-
mentation, we set N as the maximum number of instances age feature. In experiments, we find both the image feature
a single training image can contain. Then, we define N col- resolution and the target resolution are important for the fi-
ors which are evenly distributed in the 3-dimensional RGB nal performance but target resolution matters more. The
space as in semantic segmentation. Finally, we assign col- text conditioning T i is fed into the UNet via cross-attention
ors to objects based on their spatial location to form the [22]. The whole pipeline is trained in an end-to-end manner
RGB ground-truth label. For example, the instance whose except for the text encoder, which is frozen throughout the
center is at the upper leftmost corner obtains the first color training. Compared to the standard diffusion model for con-
and the lower rightmost gets the last color. At test time, the ditional image generation, there are three main differences:
model makes predictions twice with different text instruc- • We propose to directly perform diffusion in the pixel
tions and merge the results for panoptic segmentation. space. As shown in Table 2, when mapping from the
latent space to the pixel space, visually uniform re-
Image restoration aims to predict the clean image from
gions actually have pixels of many different RGB values.
corrupted images. Thus, the output space is inherently RGB
This variance can lead to inaccurate class mappings, and
image and does not need further transformation to fit in the
consequently, suboptimal performance for semantic and
framework.
panoptic segmentation.
3.2. A Diffusion Multi-Task Generalist Framework • The image conditioning is provided via a feature extractor
(we use ConvNeXt [17]) and is concatenated to the target
By reformating the output space of different vision tasks image X0 . Compared to the widely adopted method of
into images, it is natural to solve them together under a directly concatenating the raw image as the condition, this
conditional image generation framework. To this end, we brings significant performance improvement, especially
leverage the powerful diffusion models pre-trained for im- for semantic and panoptic segmentation (see Table 3 for
age generation and re-purpose them in our use case. ablation).
Fig. 2 shows the overall pipeline of the method, which is • We remove the self-attention layers in the outermost lay-
a conditional image generation framework with pixel-space ers of UNet. This is because the pixel space diffusion at
diffusion. Given M tasks with datasets {Ii , Yi }Mi=1 , where large target image resolutions induces considerable mem-
Ii are the input images of task i and Yi are the correspond- ory costs. Removing them alleviates the issue without
ing ground-truth labels. We first transform the output into compromising the performance.
 Target Depth Estimation Semantic Seg. Panoptic Seg. Denoising Deraining Light Enhance.
image RMSE ↓ mIoU ↑ PQ ↑ SSIM ↑ SSIM ↑ SSIM ↑
resolution NYUv2 ADE-20K COCO SIDD 5 datasets LoL
Generalist framework, task-specific models
UViM [14] 512 × 512 0.467 - 45.8% - - -
Generalist models
Unified-IO [18] 256 × 256 0.385 25.7% - - - -
InstructCV [11] 256 × 256 0.297 47.2% - - - -
Painter [29] 448 × 448 0.288 49.9% 43.4% 0.954 0.868 0.872
Painter [29] 128 × 128 0.435† 28.4%† 22.6%† 0.922† 0.626† 0.773†
Ours 128 × 128 0.448 48.7% 40.3% 0.954 0.815 0.758

Table 1. Our method achieves competitive performance in most of the tasks while trained at a much smaller target resolution of 128 × 128.
When compared at the same resolution, our method shows superior performance over the previous best method (Painter [29]), especially
on semantic segmentation and panoptic segmentation. The best number is in bold and the second best number is underscored. †indicates
numbers from our reproduction.

Input Image Generated RGB Image Class Prediction tic segmentation. Image restoration tasks are evaluated on
several popular benchmarks, including SIDD [1] for image
denoising, LoL [32] for low-light image enhancement, and
5 merged datasets [33] for deraining.
Implementation details. As mentioned above, we take the
Stable Diffusion v1.4 [22] checkpoint and finetune it jointly
Semantic Seg. Panoptic Seg.
mIoU ↑ PQ ↑ on six tasks. The image feature extractor is an ImageNet-
ADE-20K COCO 21K [23] pre-trained ConvNeXt-Large [17]. The text en-
Latent Diffusion 17.1% 11.7% coder is Open-CLIP [20], which is used in Stable Diffu-
Pixel Diffusion 48.0% 35.5% sion [22]. We adopt uniform sampling for each tasks ex-
Table 2. Upper: Semantic segmentation output of the latent diffu- cept panoptic segmentation, whose weight is twice as much
sion model. The perceptually same colored regions have different as the other tasks (as it is a combination of semantic and
pixel values and, therefore, are mapped to different class labels, instance segmentation). Following [5], we also adjust the
leading to bad final performance. While the red box contains only input scaling factor by a constant factor b in the forward
one ground-truth class sky in generated RGB image, the final class noising processing of diffusion. We use AdamW optimizer
prediction has four classes after the quantization. Lower: Latent [13] with constant learning rate of 0.0001, linearly warmed
diffusion suffers from the quantization issue while pixel diffusion up in the first 20,000 iterations. The target image resolu-
achieves good performance. tion is 128 × 128 while the conditioning image resolution
is 512 × 512. We train our model for 180,000 steps in total
4. Experimental Results with a batch size of 1024.
Here, we first explain experimental settings in Section 4.1. 4.2. Recipes for Diffusion-Based Generalists
Then, we highlight important design choices in diffusion-
based multi-task generalists in Section 4.2 before compar- In this section, we analyze the design choices of our method
ing our method with previous approaches in Section 4.3. and show their importance through ablation experiments.
Specifically, we show the importance of diffusion by train-
4.1. Datasets and Implementation Details ing the same model as in Fig. 2 to directly generate target
images without using diffusion (non-diffusion). We study
Datasets: We evaluate our method on six different dense the significance of image generation pre-training and image
prediction tasks with various output formats. For depth es- encoder by training models without them (train from scratch
timation, we use NYUv2 [24] and report the Root Mean and direct concat.). If not specified, we train all models at a
Square Error (RMSE). For semantic segmentation, we eval- target resolution of 64 × 64 for 50,000 steps.
uate on ADE20K [34] and adopt the widely used metric We attribute the success of our method to four aspects.
of mean IoU (mIoU). For panoptic segmentation, we use (1) While having similar results on image restoration tasks,
MS-COCO [16] and report panoptic quality as the mea- diffusion-based generalist achieves better performance than
sure. During inference, the model is forwarded twice for non-diffusion models on segmentation tasks which requires
each validation image with different instructions to obtain a global understanding of the scene and the semantics. For
the results of semantic and instance segmentation respec- example, the diffusion model reaches 35.5% PQ for panop-
tively. The outputs are then merged together into the panop- tic segmentation while the non-diffusion model has only
 Depth Estimation Semantic Seg. Panoptic Seg. Denoising Deraining Light Enhance.
RMSE ↓ mIoU ↑ PQ ↑ SSIM ↑ SSIM ↑ SSIM ↑
NYUv2 ADE-20K COCO SIDD 5 datasets LoL
Ours 0.511 48.0% 35.5% 0.949 0.772 0.704
Non-diffusion 0.443 42.4% 19.8% 0.951 0.773 0.703
Train from scratch 0.528 46.6% 33.6% 0.948 0.764 0.704
Direct concat. 0.476 37.6% 27.1% 0.941 0.772 0.687

Table 3. We analyze the important design choices of our method and aim to provide a recipe for training diffusion-based generalists: 1.
diffusion models greatly outperform non-diffusion models on panoptic segmentation; 2. text-to-image generation pre-training leads to an
overall better performance; 3. conditioning on image features extracted from an encoder gives significant improvement over the raw image.

Original Output of Denoising Process Original Output of Denoising Process Original Output of Denoising Process
Image t=0 t=25 t=50 Image t=0 t=25 t=50 Image t=0 t=25 t=50

Original Output of Denoising Process Original Output of Denoising Process Original Output of Denoising Process
Image t=0 t=25 t=50 Image t=0 t=25 t=50 Image t=0 t=25 t=50

Figure 3. Qualitative results on images from the validation sets of ADE20K, MS-COCO, NYU-V2, SIDD, Deraining, and LOL. Following
a raster scan order, the text prompts are ”Performance semantic segmentation”, ”Performance instance segmentation”, ”Performance depth
estimation”, ”Performance image restoration denoising”, ”Performance image restoration deraining”, and ”Performance image restoration
light enhancement”, respectively. The images are not cherry-picked.
19.8% (Table 3 ours v.s. non-diffusion). (2) Image gen- different target image resolutions. Since our method per-
eration pre-training on large scale dataset transfers useful forms diffusion in the pixel space, increasing the target im-
knowledge to the many downstream tasks. The model fine- age resolution is important for good performance. Despite
tuned from Stable Diffusion v1.4 [22] achieves better results the increased memory cost, our method achieves its best
than the one trained from scratch across the tasks (Table performance at the resolution of 128 × 128 and can be fur-
3 ours vs train from scratch). (3) The image conditioning ther improved with even larger target images.
can take advantage of powerful pre-trained image encoders
by conditioning on the image features rather than the raw Depth Sem. Seg. Pan. Seg. Denoise Detrain Enhance.
RMSE ↓ mIoU ↑ PQ ↑ SSIM ↑ SSIM ↑ SSIM ↑
image, which is in contrast to the standard practice for im- NYUv2 ADE-20K COCO SIDD 5 datasets LoL
age generation tasks. On semantic segmentation and panop-
32x32 0.514 44.4% 32.1% 0.940 0.743 0.653
tic segmentation, extracting features gives 10.4% and 8.4% 64x64 0.511 48.0% 35.5% 0.949 0.772 0.704
performance improvement, respectively (Table 3 ours v.s. 128x128 0.467 49.2% 36.7% 0.953 0.810 0.762
direct concat.). (4) Pixel diffusion is better than latent diffu-
Table 5. Effect of output resolution. Increasing the target image
sion as it does not suffer from the quantization issue while resolution significantly improves the performance across tasks.
upsampling (see Table 2 for an example).
Importance of noise-signal ratio. In DDPM [12], the
√
4.3. Comparisons with Prior Art forward diffusion process is defined as xt = γt x0 +
√
1 − γt ϵ, where x0 is the input image, ϵ is a Gaussian
We compare our model with recent vision generalists in
noise, and t is the number of diffusion step. As shown
Table 1. With a much smaller target image resolution at
in [5], the denoising task at the same noise level (i.e. the
128 × 128, our method achieves competitive performance
same t) becomes simpler with the increase in the image
across the tasks. In particular, when compared with the pre-
size. In order to compensate for this, [5] proposed to scale
vious best model Painter [29] at the same target resolution,
the input with a constant b to explicitly control the noise-
our method has a significant margin over them, which high-
signal ratio, which results
√ in the forward diffusion process
lights the potential of our method at a higher resolution. √
as xt = γt bx0 + 1 − γt ϵ. As we reduce b, it increases
4.4. Qualitative Results the noise levels. Table 6 shows the effect of the noise-signal
ratio b where b = 0.5 gives the best performance.
In this section, we visualize the output of our method on six
different tasks in Fig. 3. We use DDIM at inference time Depth Sem. Seg. Pan. Seg. Denoise Detrain Enhance.
with 50 steps. Each figure shows the output of the denoising RMSE ↓ mIoU ↑ PQ ↑ SSIM ↑ SSIM ↑ SSIM ↑
process at the 0-th, 25-th, and 50-th steps. NYUv2 ADE-20K COCO SIDD 5 datasets LoL
0.1 0.497 46.9% 33.1% 0.948 0.770 0.702
4.5. Ablation Study 0.3 0.511 48.0% 35.5% 0.949 0.772 0.704
0.5 0.514 49.3% 35.9% 0.949 0.774 0.708
In this section, we analyze the effect of other important
0.7 0.533 48.2% 34.4% 0.949 0.773 0.707
hyper-parameters of our method, such as batch size, target 1.0 0.572 40.3% 31.1% 0.948 0.770 0.706
image resolution, and noise-signal ratio. Similar to Section
4.2, we train all models at a target resolution of 64 × 64 for Table 6. Importance of noise-signal
√ ratio b in the forward diffusion
√
50,000 steps by default. process xt = γt bx0 + 1 − γt ϵ.

Effect of batch size. Here, we discuss the effect of different

5. Conclusion and Limitations
batch sizes for our method. As shown in Table 4, the per-
formance of most of the tasks improves with the increase of In this work, we explore a diffusion-based vision gener-
the batch size. In particular, panoptic segmentation greatly alist, where different dense prediction tasks are unified as
benefits from the large batch size. conditional image generation and we re-purpose pre-trained
diffusion models for it. Furthermore, we analyze differ-
Depth Sem. Seg. Pan. Seg. Denoise Detrain Enhance. ent design choices of diffusion-based generalists and pro-
RMSE ↓ mIoU ↑ PQ ↑ SSIM ↑ SSIM ↑ SSIM ↑ vide a recipe for training such a model. In experiments,
NYUv2 ADE-20K COCO SIDD 5 datasets LoL
we demonstrate the model’s versatility across six different
128 0.548 35.5% 26.2% 0.941 0.754 0.701 dense prediction tasks and achieve competitive performance
256 0.495 44.3% 30.0% 0.945 0.766 0.703
512 0.491 47.1% 33.5% 0.948 0.770 0.702
to the current state-of-the-art. This work, however, is also
1024 0.511 48.0% 35.5% 0.949 0.772 0.704 subject to limitations. For example, full fine-tuning of the
pre-trained diffusion model at a larger target image resolu-
Table 4. Large batch size improves the performance for all the tion is memory intensive due to the pixel space diffusion.
tasks except depth estimation.
Thus, exploring parameter-efficient tuning for such a model
Effect of target resolution. Table 5 studies the effect of would be an interesting future direction.
References model for vision, language, and multi-modal tasks. In ICLR,
2022. 1, 2, 4
[1] Abdelrahman Abdelhamed, Stephen Lin, and Michael S
[19] Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario
Brown. A high-quality denoising dataset for smartphone
Amodei, Ilya Sutskever, et al. Language models are unsu-
cameras. In CVPR, 2018. 4
pervised multitask learners. OpenAI blog, 2019. 1
[2] Amir Bar, Yossi Gandelsman, Trevor Darrell, Amir Glober-
[20] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya
son, and Alexei Efros. Visual prompting via image inpaint-
Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry,
ing. NeurIPS, 2022. 2
Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learn-
[3] Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas ing transferable visual models from natural language super-
Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to- vision. In ICLR, 2021. 4
end object detection with transformers. In ECCV, 2020. 1
[21] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee,
[4] Lluis Castrejon, Kaustav Kundu, Raquel Urtasun, and Sanja Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and
Fidler. Annotating object instances with a polygon-rnn. In Peter J Liu. Exploring the limits of transfer learning with a
CVPR, 2017. 2 unified text-to-text transformer. JMLR, 2020. 1
[5] Ting Chen. On the importance of noise scheduling for diffu- [22] Robin Rombach, Andreas Blattmann, Dominik Lorenz,
sion models. arXiv preprint, 2023. 4, 6 Patrick Esser, and Björn Ommer. High-resolution image syn-
[6] Ting Chen, Saurabh Saxena, Lala Li, David J Fleet, and Ge- thesis with latent diffusion models. In CVPR, 2022. 2, 3, 4,
offrey Hinton. Pix2seq: A language modeling framework for 6
object detection. arXiv preprint, 2021. 1, 2 [23] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, San-
[7] Ting Chen, Saurabh Saxena, Lala Li, Tsung-Yi Lin, David J jeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy,
Fleet, and Geoffrey E Hinton. A unified sequence interface Aditya Khosla, Michael Bernstein, et al. Imagenet large
for vision tasks. NeurIPS, 2022. 1, 2 scale visual recognition challenge. IJCV, 2015. 4
[8] Bowen Cheng, Alex Schwing, and Alexander Kirillov. Per- [24] Nathan Silberman, Derek Hoiem, Pushmeet Kohli, and Rob
pixel classification is not all you need for semantic segmen- Fergus. Indoor segmentation and support inference from
tation. NeurIPS, 2021. 1 rgbd images. In ECCV, 2012. 4
[9] Bowen Cheng, Ishan Misra, Alexander G Schwing, Alexan- [25] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier
der Kirillov, and Rohit Girdhar. Masked-attention mask Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste
transformer for universal image segmentation. In CVPR, Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al.
2022. 1 Llama: Open and efficient foundation language models.
[10] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina arXiv preprint, 2023. 1
Toutanova. Bert: Pre-training of deep bidirectional trans- [26] Aaron Van Den Oord, Oriol Vinyals, et al. Neural discrete
formers for language understanding. arXiv preprint, 2018. representation learning. NeurIPS, 2017. 2
1 [27] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszko-
[11] Yulu Gan, Sungwoo Park, Alexander Schubert, Anthony reit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia
Philippakis, and Ahmed M Alaa. Instructcv: Instruction- Polosukhin. Attention is all you need. NeurIPS, 2017. 1
tuned text-to-image diffusion models as vision generalists. [28] Peng Wang, An Yang, Rui Men, Junyang Lin, Shuai Bai,
arXiv preprint, 2023. 4 Zhikang Li, Jianxin Ma, Chang Zhou, Jingren Zhou, and
[12] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffu- Hongxia Yang. Ofa: Unifying architectures, tasks, and
sion probabilistic models. NeurIPS, 2020. 6 modalities through a simple sequence-to-sequence learning
[13] Diederik P Kingma and Jimmy Ba. Adam: A method for framework. In ICML, 2022. 2
stochastic optimization. arXiv preprint, 2014. 4 [29] Xinlong Wang, Wen Wang, Yue Cao, Chunhua Shen, and
[14] Alexander Kolesnikov, André Susano Pinto, Lucas Beyer, Tiejun Huang. Images speak in images: A generalist painter
Xiaohua Zhai, Jeremiah Harmsen, and Neil Houlsby. Uvim: for in-context visual learning. In CVPR, 2023. 1, 2, 3, 4, 6
A unified modeling approach for vision with learned guiding [30] Xinlong Wang, Xiaosong Zhang, Yue Cao, Wen Wang,
codes. NeurIPS, 2022. 2, 4 Chunhua Shen, and Tiejun Huang. Seggpt: Segmenting ev-
[15] Zhenyu Li, Xuyang Wang, Xianming Liu, and Junjun Jiang. erything in context. arXiv preprint, 2023. 1, 2
Binsformer: Revisiting adaptive bins for monocular depth [31] Zhendong Wang, Xiaodong Cun, Jianmin Bao, Wengang
estimation. arXiv preprint, 2022. 1 Zhou, Jianzhuang Liu, and Houqiang Li. Uformer: A gen-
[16] Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, eral u-shaped transformer for image restoration. In CVPR,
Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence 2022. 1
Zitnick. Microsoft coco: Common objects in context. In [32] Chen Wei, Wenjing Wang, Wenhan Yang, and Jiaying
ECCV, 2014. 4 Liu. Deep retinex decomposition for low-light enhancement.
[17] Zhuang Liu, Hanzi Mao, Chao-Yuan Wu, Christoph Feicht- arXiv preprint, 2018. 4
enhofer, Trevor Darrell, and Saining Xie. A convnet for the [33] Syed Waqas Zamir, Aditya Arora, Salman Khan, Munawar
2020s. In CVPR, 2022. 3, 4 Hayat, Fahad Shahbaz Khan, Ming-Hsuan Yang, and Ling
[18] Jiasen Lu, Christopher Clark, Rowan Zellers, Roozbeh Mot- Shao. Learning enriched features for fast image restoration
taghi, and Aniruddha Kembhavi. Unified-io: A unified and enhancement. PAMI, 2022. 4
[34] Bolei Zhou, Hang Zhao, Xavier Puig, Sanja Fidler, Adela
Barriuso, and Antonio Torralba. Scene parsing through
ade20k dataset. In CVPR, 2017. 4
[35] Xizhou Zhu, Jinguo Zhu, Hao Li, Xiaoshi Wu, Hongsheng
Li, Xiaohua Wang, and Jifeng Dai. Uni-perceiver: Pre-
training unified architecture for generic perception for zero-
shot and few-shot tasks. In CVPR, 2022. 1
[36] Xueyan Zou, Zi-Yi Dou, Jianwei Yang, Zhe Gan, Linjie Li,
Chunyuan Li, Xiyang Dai, Harkirat Behl, Jianfeng Wang,
Lu Yuan, et al. Generalized decoding for pixel, image, and
language. In CVPR, 2023. 2