CVPR 2024 Conference Paper

AM-RADIO: Agglomerative Vision Foundation Model

Reduce All Domains Into One

Mike Ranzinger˚ , Greg Heinrich˚ , Jan Kautz, Pavlo Molchanov

NVIDIA
{mranzinger,gheinrich,jkautz,pmolchanov}@[Link]
arXiv:2312.06709v5 [[Link]] 30 Apr 2024

RADIO(ours) H-16
DINOv2 g-14
DFN CLIP H-14 IN1k k-NN
SAM H-16 86.06

Training IN1k Zero-Shot TextVQA

83.90 85.27 56.78
DINOv2 56.32
82.93
SAM 83.41
RADIO 47.18
etc.
etc.
Vision etc.
Foundation COCO 22.12 VQAv2
77.1876.17 10.87
Model
CLIP 78.78 79.28
76.23
57.65
Vision-Language
General Models 28.08
39.00 81.76
purpose User: What type
34.34 49.92
Feature Maps of radio? 48.68 85.62
70.29 85.91
System: Vintage
51.34 86.20
Pixel-level Text Semantic ADE20k POPE
61.73
61.88
tasks: grounding: segmentation:
Inference

A radio on 82.78
sandy
beach, with
beach ball
84.71 63.01
nearby PascalVOC GQA

Figure 1. AM-RADIO is a framework to distill multiple pretrained vision foundation models, such as CLIP [51], DINOv2[48], SAM [35],
into a single model that we call RADIO. As a result, a single vision foundation model agglomerates unique properties of the original models.
This unifying approach obtains state-of-the-art feature representations in a single forward pass while also enabling unique properties such
as zero-shot classification (CLIP) or open set instance segmentation (SAM) at negligible additional cost.
Image description: (left) PCA feature visualization of different models. Our proposed RADIO model can process any resolution and
aspect ratio, and produces semantically rich dense encodings; (middle) the overview of the AM-RADIO framework; (right) benchmarks
on classification, segmentation, and vision-language modeling tasks, see section 5.

Abstract efficient backbone, we evaluated numerous architectures in

A handful of visual foundation models (VFMs) have re- our multi-teacher distillation pipeline using the same train-
cently emerged as the backbones for numerous downstream ing recipe. This led to the development of a novel architec-
tasks. VFMs like CLIP, DINOv2, SAM are trained with dis- ture (E-RADIO) that exceeds the performance of its prede-
tinct objectives, exhibiting unique characteristics for vari- cessors and is at least 6x faster than the teacher models at
ous downstream tasks. We find that despite their conceptual matched resolution. Our comprehensive benchmarking pro-
differences, these models can be effectively merged into a cess covers downstream tasks including ImageNet classifi-
unified model through multi-teacher distillation. We name cation, semantic segmentation linear probing, COCO ob-
this approach AM-RADIO (Agglomerative Model – Reduce ject detection and integration into LLaVa-1.5.
All Domains Into One). This integrative approach not only Code: [Link]
surpasses the performance of individual teacher models
but also amalgamates their distinctive features, such as 1. Introduction
zero-shot vision-language comprehension, detailed pixel-
level understanding, and open vocabulary segmentation ca- Knowledge Distillation [26] has been a very successful
pabilities. Additionally, in pursuit of the most hardware- and popular technique for transferring the knowledge of a
“teacher” model (or ensemble of models) into a typically
* Equal contribution smaller “student” model. In the original formulation, both

1
 Params Resol- ImageNet1K Segmentation (linear) Vision-Language (LLaVa-1.5 [40]) SAM [35]
Model Throughput
(M) ution Zero-shot k-NN ADE20k VOC GQA POPE TextVQA VQAv2 COCO
OpenCLIP-H/14 [11] 632 224 503 77.19 81.10 40.04 68.03 57.94 83.61 50.48 72.24 -
MetaCLIP-H/14 [64] 632 224 486 80.51 82.12 35.39 62.62 60.57 84.76 53.65 75.71 -
SigLIP-L/14 [74] 428 384 241 82.61 85.16 40.53 70.31 57.70 84.85 56.65 71.94 -
Intern-ViT-6B [10] 5,902 224 63 83.20:: 78.43 47.20 76.85 60.18 84.02 52.45 76.75 -
::
5,537 448 14 68.64 42.78 74.43 61.19 87.23 60.36 78.83 -
*DFN CLIP-H/14 [19] 633 378 170 83.90 85.27 39.00 70.29 61.73 85.91 56.78 78.78 -
*OpenAI CLIP-L/14 [51] 305 336 414 75.54 79.80 36.51 67.04 62.20 86.09 57.92 78.49 -
*DINOv2-g/14-reg [14] 1,137 224 294: - 83.41 48.68 82.78 61.88 85.62 47.18 76.23 -
*SAM-H/16 [35] 637 1024 12 - 22.12 28.08 34.34 49.92 81.76 43.91 57.65 77.18
E-RADIO-L (Ours) 391 512 468 80.73 83.89 48.22 81.64 61.70 85.07 51.47 76.73 76.31
RADIO-ViT-H/16 (Ours) 653 432 158 82.93 86.06 51.34 84.71 63.01 86.20 56.32 79.28 76.23

Table 1. Comparison of vision foundation and RADIO models. “Zero-Shot” and k-NN are computed on ImageNet-1K. ADE20K [77] and
VOC (PascalVOC2012) refer to linear probe semantic segmentation mIOU. GQA, POPE (popular), TextVQA, and VQAv2 are obtained
via LLaVa 1.5 [40] by replacing the vision encoder. COCO is the instance segmentation metric introduced by [8] to evaluate SAM [35]
distillation. RADIO attains the best metrics on most benchmarks, and is competitive with the rest, while E-RADIO enables high quality
results in resource constrained settings. Note that Zero-Shot and COCO use teacher’s decoder head that is not finetuned. Throughput
computed using NVIDIA A100 GPU, stated resolution, and TensorRT v8601. *Denotes teachers used to train our final RADIO. : We failed
to export DINOv2-g-reg to TensorRT, so we report DINOv2-g here, which should be fairly close. :: We were unable to get zero shot working
using their model code.

Input: Frozen an iterative learning procedure with a high-capacity model

Image only, where a student of equal or greater capacity than the teacher
Trainable
no labels
is trained with heavy augmentation applied to the student.
Once trained, they expand the dataset by pseudo-labeling
(from scratch)
new data using the trained student. They then make the stu-
Input RADIO
conditioner: Vision dent become the teacher, and repeat the process. An im-
image and patch Foundation portant finding in this work is that the student is capable of
size etc model
surpassing the performance of the teacher.
Teacher:
The authors of [26] explore the concept of ensemble dis-
Summary

2
CLIP, DINOv2, 0 tillation, where there are multiple teachers, each of which
Features

2
SAM 4
5 having restricted domain knowledge. [78] provides an
overview of multi-teacher distillation, and proposes that in-
Summary

2
0
stead of matching the summary of an ensemble of teachers,
Teacher
head 1
Features

2
4
5 the student can match the features of each individual teacher
Teacher
head 2

Match via some learned non-shared mapping from the representa-

Summary token channels tion space of the student to each teacher. Of interest in their
distillation loss approach is that the student and teacher don’t need to share
Feature Match Feature map the same architecture, and also that treating teachers indi-
distillation loss spatial interpolation
vidually yields improved performance.
Repeat for every teacher before the RADIO update Recently, the concept of Foundation Models (FMs) [3]
has emerged, with the general understanding that these
Figure 2. AM-RADIO - is a multi-teacher distillation framework models are large, general, and expensive to train. Through
that efficiently trains new vision foundation models of arbitrary ar- training on very large datasets they are broadly applicable
chitecture. It unifies unique attributes (like zero-shot text ground- to numerous downstream tasks. A seminal example of such
ing, dense correspondence) of each teacher into a single model that
models is CLIP [51], which trains on web-scale weakly su-
even outperforms them on a majority of the tasks.
pervised (image, caption) pairs, and results in exceptional
zero-shot performances on a wide array of computer vision
the student and the teacher operate on the same in-domain benchmarks. While CLIP is firmly a FM, another model,
dataset, and the student simultaneously matches the logits DINOv2 [48] has emerged with broad capabilities, often
of the teacher, and the ground truth labels. Instead of us- surpassing CLIP on dense tasks that require strong spatial
ing labeled images, an alternative approach is to train the features, such as ADE20k [77] and Pascal VOC [18]. Sep-
student model to match the features of the teacher model arately, SAM (Segment Anything) [35] is gaining popular-
[1, 25, 28, 53, 56, 61, 72]. ity for its excellent open-vocabulary instance segmentation
Instead of using a smaller student model, [63] employ abilities, whose vision encoder we hypothesize has strong

2
dense feature representations. teachers before any task-specific processing stages.
We introduce AM-RADIO with the goal of learning from Multi-Teacher Distillation There is also a body of work
multiple foundational models simultaneously. We observe that studies distilling a student model jointly from multi-
that, when given a student model of sufficient capacity, ple teacher models simultaneously [2, 20, 26, 36, 42, 50,
it is often able to exceed any of its teachers on impor- 68, 69, 71, 75, 78]. Because of the heterogeneous domains
tant axes. In addition to performing well on representative that our teacher models cover, we don’t apply approaches
foundational benchmarks, by virtue of the training frame- that marginalize teachers into a unified label, and instead
work, our student models are able to mimic their teacher map students to each teacher independently using teacher-
models, and thus are able to perform downstream tasks specific projection heads from the unified student represen-
that are otherwise performed by the teachers. Examples tation. Although the reason behind this method in [78] is
of this include CLIP-ZeroShot applications, since the lan- different, we find the same overall strategy to be effective.
guage model trained by CLIP is compatible with our stu- While [61] doesn’t study matching the features of multi-
dent, and also Segment-Anything tasks, as the student is ple teachers simultaneously, we are able to extend their
able to replace the vision encoder and interface with the paradigm via the different projection heads. To preserve
already-trained mask decoders. drop-in compatibility with teacher frameworks, we elimi-
We also study the effect of using a more hardware- nate the feature normalization in the loss function.
efficient model architecture. Most works on efficiency Distilling Foundation Models Foundation Models [3] are
are not directly comparable as they use different training meant to be generalist models that are trained on massive
recipes, even when evaluated on the same dataset such as amounts of data, and are typically resource intensive to train
ImageNet-1k, and may be over-tuned. To this end, we eval- from scratch. In the vein of single-teacher distillation, [48]
uate more than 10 promising architectures under the same employ self-distillation to train their smaller variants from
training recipe for a direct comparison. We reveal that the larger teacher. [61] distills their model from a CLIP
CNN-like architectures are faster but struggle to distill ViT [51] teacher. Instead of focusing our energy on one teacher
VFMs. This led us to the development of a novel hybrid in particular, we instead grab high-quality versions of CLIP
architecture, E-RADIO, that exceeds the performance of its [51] (using OpenCLIP [30]), DINOv2 [48], and SAM [35].
predecessors and is at least 6x faster than teacher models at Concurrently with our work, [60] describe a methodology
matched resolution. for merging a CLIP model into a pretrained SAM model via
Our main contributions are as follows: distillation, which is, in spirit, quite similar to our approach.
• We describe a general methodology for distilling multi- In contrast to theirs, we include DINOv2 and also simplify
ple distinct foundation models into one, including models the objective to straightforward feature matching. Since we
with incompatible input resolutions. don’t rely on the student model to be pre-trained, it also
• We show that these student models are able to outperform gives us the flexibility to have the student be an architecture
their teachers on representative benchmarks. distinct from any teacher.
• We demonstrate that these student models can either
drop-in replace their teachers, or their features can be 3. Knowledge Agglomeration
used directly in downstream applications such as provid-
We propose a framework to train a vision foundation model
ing visual encoding for LLaVA [40, 41].
from scratch via multi-teacher distillation as shown in Fig-
• We benchmark a number of efficient architectures and
ure 2. We demonstrate that each teacher brings unique prop-
propose a new architecture (E-RADIO) that allows for
erties to the foundational vision model, and the resulting
similar model quality at significant speedups.
trained model will agglomerate these attributes.
2. Related Work 3.1. Overview
Knowledge Distillation The underpinning of our work is As an initial assumption, we expect that the teacher mod-
based on the method of Knowledge Distillation [4, 5, 26, els are capable of representing a broad swath of images
34, 47] which aims to train a “student” model using soft found on the internet, coming from datasets such as Ima-
targets produced by an already-trained “teacher” model, us- geNet (1k or 21k) [15], LAION-400M [54] or DataComp-
ing the the teacher’s output logits as “soft” labels. Alterna- 1B [21]. With this in mind, we choose to study 3 seminal
tively, distillation can be performed using intermediate net- teacher model families: CLIP [51], DINOv2 [48], and SAM
work activations [1, 25, 28, 53, 56, 61, 72]. In general, due [35] as they have demonstrated outstanding performance
to the heterogeneous nature of the different teacher founda- over a broad range of tasks (as in CLIP), or specifically
tion models that we employ, we ignore any potential labels strong performance on downstream dense tasks, such as se-
coming from the data, and we ignore the logits of teachers, mantic segmentation under linear probe (as in DINOv2), or
and simply opt to match the feature representations of the open-vocabulary segmentation (as in SAM). Because these

3
teacher models come from such diverse domains, we omit Dataset k-NN Zero Shot ADE20K
any form of supplemental ground truth guidance and treat ImageNet 1K 84.79 80.44 48.11
the aforementioned datasets simply as sources of images. ImageNet 21K 84.61 80.10 48.65
To assess the quality of our models, we adopt a set of repre- LAION-400M 83.77 77.46 48.6
DataComp-1B 83.91 78.51 49.01
sentative metrics across a few broad domains.
• Image level reasoning: (i) k-NN Top-1 accuracy on
Table 2. Ablation study on the choice of training dataset. We use
ImageNet-1K, and (ii) Zero-Shot accuracy using the MetaCLIP ViT-H/14 [16] and DINOv2 ViT-g/14 teachers, and a
CLIP teacher’s language model [51]. k-NN [9, 48, 62] ViT-L/14 student model with CPE [33]. Both “k-NN” and “Zero
embeds the model’s summary feature vector for every im- Shot” are for ImageNet-1k. ADE20k refers to mIOU linear probe
age in the training set, and then for each validation image, on ADE20k.
it uses a weighted sum of the k nearest training vectors to
elect a label. Teachers Zero Shot k-NN ADE20K
• Pixel-level visual tasks: segmentation mIOU on (i)
None 75.77 82.59 41.18
ADE20K and (ii) Pascal VOC - under the linear probe CLIP 75.64 82.60 44.42
setting, details in Section 5.3. DINOv2 74.68 83.02 47.05
• Large Vision-Language Models: we plug our frozen Both 74.85 82.96 48.13
vision encoder model into LLaVA-1.5 [40] and evaluate
it on a wide set of tasks including GQA [29], TextVQA Table 3. Ablation over which teachers we supervise the spatial fea-
[55], ScienceQA [46] and VQAv2 [23]. Details in Sec- tures. We use a ViT-L/14 student model and train on the LAION-
tion 5.4. 400M dataset. Adding this loss term is always beneficial. DINOv2
• SAM-COCO instance segmentation: From [8], we appears to provide better spatial features than CLIP, but training
the student to match both teachers produces the best results. We
adopt their COCO instance segmentation methodology to
don’t ablate SAM as we solely want it for its spatial features.
evaluate our ability to replicate SAM visual features.
Results on these tasks, both for teacher models and our AM-
RADIO variants, are summarized in Table 1. DINOv2, we use the “class token” as the summary feature
vector, and we don’t match a summary for SAM.
3.2. Adaptor Heads
Let f px|Θ0 q be the student vision encoder with parame-
We opt for simplicity in design of the adaptor heads, and psq psq
ters Θ0 , and yis “ hi px1 |Θi q be the learned student head
leave alternative architectures as future work. To this end, psq psq
matching teacher summary features zi “ ti px|Φi q with
we employ a simple 2-layer MLP, with a LayerNorm and psq
student adaptor parameters Θi and teacher parameters Φi .
GELU in between. The input dimension is the student em-
bedding dimension, the intermediate dimension is the max-
imum embedding dimension of all teachers, and the output \begin {aligned} x' &= f\left (x|\Theta _0\right ); & y_i^{(s)} &= h_i^{(s)}\left (x'| \Theta _i^{(s)}\right ); \\ z_i^{(s)} &= t_i^{(s)}\left (x |\Phi _i\right ), \end {aligned} \label {eq:model_outputs}
dimension matches the specific teacher. For each teacher, (1)
we employ two heads, one for the summary vector, and one
for the spatial features.
\begin {aligned} L_{\text {summary}}(x) &= \sum _i \lambda _i L_{\text {cos}}(y_i^{(s)}, z_i^{(s)}) \end {aligned} \label {eq:summary_loss} (2)
3.3. Distillation Dataset Choice
In table 2 we study the effect of different datasets on down- We found empirically that cosine distance loss produced
stream metrics. While the highest image classification met- better models compared to L1, MSE, Smooth-L1 [22]. Ad-
rics are achieved using ImageNet-1K as the training dataset, ditionally, supervising the spatial features of the model by
we argue that it doesn’t fairly measure “zero shot” perfor- matching the teacher was not only important for down-
mance as the student directly learns the teacher features stream dense tasks, but also improved the holistic quality
in the evaluation domain. For this reason, we opt for the of our model.
DataComp-1B dataset. For matching the spatial features, we employ a combi-
nation of cosine similarity and smooth L1. Similar to equa-
3.4. Loss Formulation tion (2) where we found that cosine similarity produced the
Because we don’t have ground truth data for each teacher best results, we found the same to be true for the spatial
for each image, we instead opt to match the features coming features. However, we want to allow our student model to
from each teacher’s vision encoder. In particular, we distin- be a drop-in replacement in the teacher frameworks, thus
guish between the summary feature vector and the spatial it’s important that we match the magnitude of the teacher
feature vectors for each teacher. The summary feature is vectors, and so we include smooth L1. In (3) we show the
pvq pvq
computed differently based on the model. For CLIP and formulation of this loss. Let hi px1 |Θi q be the learned

4
 Method Zero Shot k-NN ADE20K 224px, and SAM ViTDet-H 1024px. We apply random
Naive 70.63 79.50 44.71 scale + cropping to both student and teacher inputs. We
Uncertainty [12] 70.92 79.37 44.57 chose the DataComp-1B dataset due to it having the highest
AdaLoss [27] 71.31 79.77 44.36 quality results of the web-scale datasets we had access to.
We train in two stages, first with CLIP+DINOv2 for 300k
Table 4. Loss term balancing methods comparison. We use a steps at 256px, and second with CLIP+DINOv2 at 432px
ViT-B/14 student, and CLIP+DINOv2 teachers. We found that plus SAM at 1024px for 300k steps.
AdaLoss produces the best results on the ImageNet tasks, but the
Student architecture. We study two settings for student
worst on ADE20K.
model architecture:
• Standard ViT [16] architecture to match the architecture
student head for matching teacher feature vectors, and cor-
pvq pvq of teachers. Our best model is a ViT-H/16.
responding ti px|Φi q be the teacher feature vectors, with • Efficient architecture variants prioritizing high through-
1
x “ f px|Θ0 q, then the spatial feature loss is: put on GPUs. See Section 5.1.
Multi-scale Teachers. We choose ViT-H/16 architecture
\begin {aligned} L_{\text {match}}(x, y) &= \alpha L_{\text {cos}}(x, y) + \beta L_{\text {smooth-l1}}(x, y) \\ L_{\text {features}}(x) &= \sum _i \gamma _i L_{\text {match}}\left (h_i^{(v)}(x'|\Theta _i^{(v)}), t_i^{(v)}(x|\Phi _i^{(v}))\right ) \end {aligned} \label {eq:feature_distillation} for our student model. To match resolution of SAM fea-
tures, we feed the expected resolution of 10242 . Given that
our CLIP and DINOv2 teachers are patch-14 models, we
opt to feed the student 4322 inputs, as that is the same ef-
(3)
fective resolution as 3782 for patch-14. We found that in-
We choose α “ 0.9 and β “ 0.1 to mostly rely on the
terpolating DINOv2 features doesn’t degrade results, so the
empirically better cosine distance, but to also match vector
teacher operates at 224px and we upsample the outputs to
magnitudes.
match the student.
Rank/Teacher Partitioning. We group teacher models
3.4.1 Loss Balancing by (batch size, student resolution), and then distribute the
groups to different GPUs, such that each GPU processes a
Due to the number of possible combinations of loss weights
consistent batch size and input resolution. We also sample
between the different teachers, and even which teachers,
groups at different rates. For our training setups that in-
and possible formulations of loss functions, we mostly
clude SAM, we train with 64 GPUs, half of which get the
opted toward naive loss balancing with all teachers equally
CLIP+DINOv2 group with batch size 32 per GPU and in-
weighted for spatial features (γi “ 1). For summary fea-
put resolution 432, and the other half get SAM with batch
tures, we have λCLIP “ λDIN O “ 1 and λSAM “ 0.
size 2 per GPU and input resolution 1024. This results in an
We did experiment with automatic loss balancing using
effective batch size of 1,152. For CLIP+DINOv2 training,
predicted uncertainty [12], AdaLoss [27] (momentum 0.99)
we use 32 GPUs, resulting in batch size 1024.
and separately with AMTML-KD [42], as ways to learn the
balance of λi and γi . In the case of AMTML-KD, the model Multi-Resolution ViTs. Many of our student models use
would always collapse its entire weight around the CLIP ViT [16] as the base vision architecture. Traditionally, ViTs
teacher and would yield worse results than naive manual use a learned position embedding for each input patch in an
balancing. Based on the results in table 4, there is very little image, which in turn enforces that the model always oper-
advantage to the more exotic balancing schemes, so we opt ates at a constant resolution. We employ the Cropped Posi-
for the ”Naive” method throughout the rest of the paper. tion Embedding (CPE) [33] augmentation with the number
of positions being equal to 1282 . The position embeddings
are then randomly cropped and interpolated to match the
4. Implementation Details
number of input patches for the student model. Even when
Performing heterogeneous multi-teacher distillation is not training with CLIP+DINOv2 at 224 resolution, we found
trivial due to a mismatch in feature dimensions, input res- that this technique results in a negligible drop (Table 5) in
olutions, concepts for loss computation, and downsampling summary metrics, but improved semantic segmentation lin-
ratios, as well as challenges in fitting multiple teachers into ear probing mIOU. For heterogeneous-resolution students,
a single GPU. this is a seamless technique that allows ViT to operate at
General. We train all student models using the AdamW arbitrary resolutions within some envelope. In addition to
[45] optimizer, batch size 1024, cosine annealing learning enabling arbitrary resolutions, as shown in figure 3, CPE
rate schedule and base learning rate of 0.001. We train for reduces the noise artifacts in the position embeddings as
600k steps, resulting in 614M total examples seen. For our compared to other ViT models [6, 66, 67].
best student model, we train using DFN CLIP ViT-H/14 High-Resolution ViT Student. In SAM, they employ the
378px, OpenAI CLIP ViT-L/14 336px, DINOv2 ViT-g/14 ViTDet [37] architecture as a way to reduce the computa-

5
 Method k-NN ADE20K Zero Shot k-NN ADE20K VOC VQAv2
Non-CPE 82.96 47.30 CLS token 78.55 83.91 49.01 83.51 77.66
CPE 82.84 48.52 Avgpool 80.12 83.83 38.36 77.04 78.28

Table 5. Comparing identical ViT-L/14 student models, with and Table 6. Comparing identical ViT models, with CLS token and
without CPE [33] formulation. While the student only ever trains average pooling summarization.
at 2242 resolution, CPE allows us to generalize to 5182 resolu-
tion, not only improving over non-CPE, but even outperforming teachers a 162 feature map. For Lfeatures we bilinearly in-
DINOv2-g itself. terpolate the outputs to match the larger resolution between
the student and teacher features.
Feature Summarization. In 3.4 we explained how teacher
summary features are extracted using the “class token” of
their respective ViT models. We now turn our attention to
the summarization of student features. ViTs have 2 options:
(i) a separate summarization “CLS” token or (ii) average
pooling patch tokens. We evaluate both options in Table 6.
We observe that average pooling improves summary loss,
but has a more significant detrimental effect on the feature
(a) RADIO 2048px (b) DINOv2-g-reg 518px
loss. Given the importance of the latter we choose to use
separate CLS tokens.

5. Results
In this section, we analyze models obtained with the pro-
posed AM-RADIO framework. First, we touch upon back-
bone efficiency, then compare with the original teachers
(CLIP, DINOv2, SAM), and benchmark models under vi-
(c) DFN CLIP 378px (d) OpenAI CLIP 336px sion question answering in the LLaVa framework. We will
see that the proposed models outperform the original teach-
Figure 3. PCA visualization of the position embeddings for var- ers in multiple metrics, including throughput. Results are
ious models. The CPE method not only allows RADIO to learn shown in Figure 1 and Table 1.
an arbitrarily large absolution position embedding map, but also
goes a long way towards regularizing the space and eliminating 5.1. Efficient Students
high frequency artifacts. As seen with the other models, position
embeddings normally have regular frequency patterns, leading to We aim to find an efficient model architecture to speed up
undesirable output artifacts from the ViT [6, 66, 67]. the inference of VFM. There are a number of architectural
designs aimed at high throughput on GPU devices. We
use our distillation framework to evaluate several backbones
tional and memory burden of ViT models at high-resolution. with no change in training hyperparameters.
We reformulate this arch instead into a training augmenta- Upon reviewing the literature on efficient vision back-
tion, where we sample a window size from a set of possible bones focused for high GPU throughput, we pick the fol-
window sizes. This allows us to reduce the computational lowing list of architectures: EfficientNetV2 [58], ResNetv2
burden of training the student model with the SAM teacher, [57], RegNetY [52], FasterViT [24], EfficientViT [8], Con-
and, as we make the window size flexible, it provides an vNext [44], NFNet [7], SwinV2 [43], MaxViT [59], Pool-
additional throughput scaling mechanism during inference. formerV2 [70] and MViTV2 [38]. We train all the back-
Table 8 demonstrates our ability to replace SAM’s encoder. bones via distillation on the ImageNet-21k dataset, using
Separately, we found that high resolution training was un- OpenCLIP ViT-H/14 (laion2B-s32B-b79K) and DINOv2
stable, so we apply spectral reparametrization [73] and a g/14 as teachers. Results are compiled in Table 7.
weight decay of 0.02 to prevent attention entropy collapse. We observe that many models lag behind teachers. Addi-
Student/Teacher Resolution Mismatch. When the student tionally, CNN-like models are significantly faster than ViTs,
and teacher downsample images through their processing while the latter are more accurate. The relatively low perfor-
stack at different rates, it results in the output feature vec- mance of existing efficient backbones on the dense ADE20k
tors having different resolutions. For example, if the teach- segmentation task is not unexpected since all of them apply
ers use a ViT-H/14 architecture and student a ViT-H/16, it a spatial dimension reduction factor of 32 for final feature
means that the student outputs a 142 feature map, and the maps of size 72 for input resolution of 2242 px, thus hardly

6
 ImageNet1k
H-14 L-14
k-NN Classification ImageNet1k zero-shot Classification
H-14 50
ADE20k
H-14
Semantic
L-14
Segmentation
L L-14 B F3 6x L
84 10x
B B B
B B-14 m36 S L3 B-14 B-16 B
82 B S L3 B-16 75 S
L
m36 F3 S 45 +20% B-16 B
L1 L1 T
80 B T
64101 B-32
XT B
78 70
XT 40 B
B-32 B F3
Accuracy

Accuracy
S

mIoU
76 101
64 65 B-32
74 m36 S
35 L1
72 S 60
70 101
30 64
B1 55 B1 S
68
400 800 1600 3200 6400 800 1600 3200 6400 400 800 1600 3200 6400
Throughput Throughput Throughput

RADIO-ViT RADIO-EfficientNetV2 RADIO-RegNetY RADIO-MaxViT RADIO-ResNetV2 RADIO-PoolFormerV2 Teacher

E-RADIO RADIO-NFNet RADIO-SwinTransformerV2 RADIO-MViTV2 RADIO-ConvNext RADIO-MiTEfficientViT

Figure 4. All models followed the same training protocol. The results from three benchmarks show that RADIO and E-RADIO models
outperform others in efficiency. This under-performance in other models might be due to overfitting architectures on supervised ImageNet-
1K training. E-RADIO notably delivers results 10 times faster and with a 20% improvement over teacher models. We study E-RADIO at
224px resolution, with a window size of 7.

Backbone Param. Through- Zero k-NN ADE20k FD loss attention with “global” windowed attention as done in [24]
Count put Shot
and ViTDet [37]. To perform “global” attention we first
Teachers
downsample the feature map by 2x, apply windowed atten-
DINOv2 G/14 1.14B 313 N/A 83.41 47.53
OpenCLIP H/14 632M 556 77.19 81.10 40.04 tion, and then upsample the feature maps back to the origi-
Existing Efficient Models nal resolution. Up-/down-sampling is performed by strided
EfficientNetV2-S 21M 9017 65.37 70.72 27.75 0.415 convolution with a kernel size 3x3 and a stride of 2. The
ResNetv2-101 44M 7283 69.58 75.32 29.61 0.405 last idea is borrowed from EdgeViT [49], which uses local-
RegNetY-064 30M 6573 69.84 74.59 28.9 0.394
EfficientViT-L1 38M 6048 71.73 79.90 33.12 0.376 global-local attention. See Appendix for details. Finally,
ConvNext-B 88M 1805 75.43 81.73 38.95 0.358 E-RADIO upsamples final feature maps by 2x via a decon-
NFNet-F3 254M 1777 76.93 80.50 38.31 0.340
SwinV2-S 49M 1497 74.70 81.12 35.57 0.364 volutional layer and adds them to feature maps from the
MaxViT-B 119M 1486 77.49 79.34 38.46 0.340 third stage, resulting in only a 16x spatial resolution re-
PoolformerV2-M36 56M 1194 74.46 80.49 35.05 0.377
MViTV2-B 51M 975 75.92 81.39 41.39 0.345 duction. Such upsampling gives an improvement in dense
Proposed architecture task while being only 10% slower. Results of E-RADIO in
E-RADIO-B 118M 6422 75.19 82.21 44.03 0.319 Table 7 demonstrate that the proposed architecture signifi-
ë w/o upsample 113M 7040 75.45 82.05 41.26 0.353 cantly outperforms the competition, and can be seen as an
E-RADIO-L 265M 3472 77.87 83.73 45.5 0.265
efficient replacement for the much slower full ViT.
Table 7. Comparison of backbones. Throughput is measured using
TensorRT 9.0.1 on A100 in mixed FP16/FP32 precision at batch 5.2. Comparison with teachers
size 128 on 2242 px resolution. Sorted by descending through-
put order. FD loss is the Feature Distillation training loss against A comprehensive set of results is presented in Table 1. We
the DINOv2 teacher, it exhibits high correlation with the ADE20k notice that MetaCLIP is better than OpenCLIP, and DFN
mIoU. Bolded models form the speed/quality Pareto front. CLIP better than MetaCLIP. DINOv2 provides important
properties for dense tasks: ADE20k and VOC. Our E-
RADIO-L model is significantly faster than all ViT mod-
capable of capturing fine-grain spatial information. els. At the same time, it strongly outperforms MetaCLIP
E-RADIO: To overcome this issue, we propose a novel on most metrics at matched throughput, while also enabling
hybrid architecture, named E-RADIO (Efficient RADIO). Zero-shot capability that is absent in DINOv2 and SAM.
This design borrows ideas from existing literature and in- Our full model, ViT-H/16, is as fast as the teachers but out-
cludes an input stem with strided convolutions to downsam- performs them on 6 out of 9 tasks, demonstrating the effi-
ple the input image by 4x. It then proceeds with 2 stages ciency of the proposed distillation framework.
of YOLOv8 C2f convolution blocks and 2 stages of trans- Drop-In SAM Replacement. Following [8], we use their
former. For the transformer variant we pick windowed at- evaluation harness to compute the mIOU for instance seg-
tention (like in SWIN [43]), and interleave local windowed mentation using pretrained SAM with vision encoder re-

7
 COCO 2017 drop-in SAM replacement at 1024x1024 Backbone Depth Surface Multi-view
Family Arch mIOU Throughput Normals corr.
Base 75.78 50.94 DFN CLIP-H/14 52.5 23.0 20.3
SAM Large 77.02 20.62 OpenAI CLIP-L/14 53.7 25.3 20.7
Huge 77.18 11.83 DINOv2-g/14-reg 83.2 59.6 59.9
E-RADIO (ours) Large 76.31 121.74 SAM-H/16 68.2 50.3 45.3
ViTDet-H/16-W8: 76.09 29.09
RADIO (ours) RADIO-ViT-H/16 (ours) 81.0 58.5 62.1
ViTDet-H/16-W16: 76.23 27.91

Table 9. Probing 3D Awareness: we use the code from [17] and

Table 8. We substitute SAM’s vision encoder with our RADIO
evaluate our RADIO model and its teachers on monocular depth,
model. RADIO aligns with SAM’s features just before the en-
surface normals and multi-view correspondance tasks, using the
coder’s neck layer. We also examine the impact of varying ViT-
NAVI[31] dataset. For each task we report the accuracy, averaged
Det window sizes. Differences in throughput owe to the fact that
over all thresholds.
RADIO doesn’t use relative positional embeddings and we re-
duced shuffling with our patch reordering algorithm (in appendix).
Throughput is computed on an NVIDIA A100 GPU using Ten-
sorRT and batch size 16. : This is the same model, just with a
5.5. 3D Awareness Probing
different window size setting. Following the work from [17], we probe our model’s ability
to extract 3D features such as depth, surface normals and
multi-view keypoint correspondance. Our results are sum-
placed by our model. Table 8 shows the results of the marized in Table 9 and show that our model’s performance
COCO Instance Segmentation task using the baseline SAM is on par with the bigger DINOv2-g-14-reg[14] and signifi-
models and RADIO. cantly better than other comparably-sized teachers.

6. Conclusion and Key Insights

5.3. Semantic Segmentation Linear Probing
Most VFMs have unique properties such as language
We train a linear head on top of the frozen features of the grounding (CLIP), dense correspondences (DINOv2), and
teachers and students alike and evaluate performance in the detailed segmentation (SAM), but also large holes in capa-
MMSeg [13] framework using the mIoU metric on ADE20k bility. Distillation allows uniting all these properties in a
and PascalVOC2012 datasets. We use a training and eval- single model that often outperforms any of the teachers. We
uation crop size of 512 for RADIO, 518 for DINOv2, and have also observed that better teachers yield better students,
the native resolution for the others. We use the “slide” eval- which allows RADIO to absorb and challenge the current
uation mode with a stride of 23 the crop size. We train the SOTA foundation models at a given point in time.
linear head for 160k steps using a total batch size of 16, a Feature distillation loss. We observe the crucial impor-
base learning rate of 10´3 and the AdamW optimizer. tance of full feature distillation to boost the performance of
the teacher in dense image understanding tasks, such as an
5.4. Visual Question Answering 18% relative improvement on ADE20K.
SAM vs DINOv2. We find that, out of the box, SAM is not
We replace the vision encoder in a LLaVA 1.5[40] setup well-suited for downstream tasks, whereas DINOv2 signif-
with our own encoder. A 2-layer MLP is used to project icantly outperforms in zero- and few-shot tasks. For exam-
frozen visual features into the language token space. Under ple, ADE20K segmentation via linear probing is 1.7x bet-
the default LLaVA 1.5 settings, we pretrain a multimodal ter with the latter, and the ImageNet1k k-NN metric is 4x
projection MLP and then run instruction tuning to finetune better. SAM excels in detecting edges and segmenting ob-
a Vicuna 7B-1.5 model[76]. We evaluate models using the jects but performs poorly in high-level object description
validation sets of GQA [29], TextVQA [55], POPE [39] and combining the semantics of multiple objects (Figure 4).
(popular), and we score the model on the Test-Dev set of Dense features. As seen in figure 1, RADIO is capable of
VQAv2 [23] using EvalAI[65]. We use the vision encoder’s producing high resolution and low-noise features. An issue
native input resolution, resizing the long edge and padding we identified, however, shown in figure 5 is that RADIO
the short edge. Experimental results are compiled in Ta- appears to have a latent ‘low resolution’ and ‘high resolu-
ble 1. Owing to the increased input resolution flexibility of tion’ mode, likely due to the partitioned training between
RADIO, we resize the long edge of the image to 432px as- CLIP+DINO and SAM objectives, which we intend to fix
pect preserving, only padding to the nearest multiple of the in future work.
patch size. This results in 462 tokens on average, versus Efficient backbone. Based on our analysis of distilling ef-
the 576 tokens required by the 336px patch-14 encoders, a ficient backbones, we conclude that most model designs are
20% reduction. overly tailored towards supervised training on ImageNet1K,

8
 models defining a new era in vision: A survey and outlook,
2023. 2, 3
[4] Jimmy Ba and Rich Caruana. Do deep nets really need to
be deep? In Advances in Neural Information Processing
Systems, pages 2654–2662, 2014. 3
[5] L. Beyer, X. Zhai, A. Royer, L. Markeeva, R. Anil, and A.
Kolesnikov. Knowledge distillation: A good teacher is pa-
tient and consistent. In 2022 IEEE/CVF Conference on Com-
puter Vision and Pattern Recognition (CVPR), pages 10915–
10924, Los Alamitos, CA, USA, 2022. IEEE Computer So-
ciety. 3
[6] Daniel Bolya, Chaitanya Ryali, Judy Hoffman, and
Christoph Feichtenhofer. Window attention is bugged: How
not to interpolate position embeddings, 2023. 5, 6
[7] Andrew Brock, Soham De, Samuel L. Smith, and Karen Si-
monyan. High-performance large-scale image recognition
without normalization, 2021. 6
[8] Han Cai, Junyan Li, Muyan Hu, Chuang Gan, and Song Han.
Efficientvit: Multi-scale linear attention for high-resolution
dense prediction, 2023. 2, 4, 6, 7
[9] Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou,
Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerg-
ing properties in self-supervised vision transformers, 2021. 4
[10] Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen,
Sen Xing, Muyan Zhong, Qinglong Zhang, Xizhou Zhu,
Lewei Lu, Bin Li, Ping Luo, Tong Lu, Yu Qiao, and Jifeng
Dai. Internvl: Scaling up vision foundation models and
aligning for generic visual-linguistic tasks. arXiv preprint
arXiv:2312.14238, 2023. 2
[11] Mehdi Cherti, Romain Beaumont, Ross Wightman, Mitchell
Figure 5. RADIO “mode switches” when resolution is increased. Wortsman, Gabriel Ilharco, Cade Gordon, Christoph Schuh-
In the plot, we show the MSE error between the RADIO features mann, Ludwig Schmidt, and Jenia Jitsev. Reproducible scal-
coming from its DINOv2 head at different resolutions, versus the ing laws for contrastive language-image learning, 2022. 2
features actually produced by DINOv2 at 518px. We bilinearly [12] R. Cipolla, Y. Gal, and A. Kendall. Multi-task learning using
interpolate the RADIO features to match the DINOv2 feature res- uncertainty to weigh losses for scene geometry and seman-
olution. At 720px, there is a sudden jump in the error, which cor- tics. In 2018 IEEE/CVF Conference on Computer Vision and
responds with a complete change in color space in the image. Pattern Recognition (CVPR), pages 7482–7491, Los Alami-
tos, CA, USA, 2018. IEEE Computer Society. 5, 7
and as a result, do not scale well to VFM settings. We
[13] MMSegmentation Contributors. MMSegmentation:
designed a new vision backbone, E-RADIO, with a hy- Openmmlab semantic segmentation toolbox and
brid CNN-Transformer architecture that improves upon the benchmark. https : / / github . com / open -
Pareto frontier. mmlab/mmsegmentation, 2020. 8
[14] Timothée Darcet, Maxime Oquab, Julien Mairal, and Piotr
References Bojanowski. Vision transformers need registers, 2023. 2, 8
[1] S. Ahn, S. Hu, A. Damianou, N. D. Lawrence, and Z. Dai. [15] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li,
Variational information distillation for knowledge transfer. and Li Fei-Fei. Imagenet: A large-scale hierarchical image
In 2019 IEEE/CVF Conference on Computer Vision and Pat- database. In 2009 IEEE Conference on Computer Vision and
tern Recognition (CVPR), pages 9155–9163, Los Alamitos, Pattern Recognition, pages 248–255, 2009. 3
CA, USA, 2019. IEEE Computer Society. 2, 3 [16] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov,
[2] Umar Asif, Jianbin Tang, and Stefan Harrer. Ensemble Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner,
knowledge distillation for learning improved and efficient Mostafa Dehghani, Matthias Minderer, Georg Heigold, Syl-
networks. In European Conference on Artificial Intelligence, vain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is
2019. 3 worth 16x16 words: Transformers for image recognition at
[3] Muhammad Awais, Muzammal Naseer, Salman Khan, scale, 2021. 4, 5
Rao Muhammad Anwer, Hisham Cholakkal, Mubarak Shah, [17] Mohamed El Banani, Amit Raj, Kevis-Kokitsi Maninis, Ab-
Ming-Hsuan Yang, and Fahad Shahbaz Khan. Foundational hishek Kar, Yuanzhen Li, Michael Rubinstein, Deqing Sun,

9
 Leonidas Guibas, Justin Johnson, and Varun Jampani. Prob- world images. CoRR, abs/1902.09506, 2019. 4, 8, 9, 10,
ing the 3D Awareness of Visual Foundation Models. In 11
CVPR, 2024. 8 [30] Gabriel Ilharco, Mitchell Wortsman, Ross Wightman, Cade
[18] M. Everingham, S. M. A. Eslami, L. Van Gool, C. K. I. Gordon, Nicholas Carlini, Rohan Taori, Achal Dave,
Williams, J. Winn, and A. Zisserman. The pascal visual ob- Vaishaal Shankar, Hongseok Namkoong, John Miller, Han-
ject classes challenge: A retrospective. International Journal naneh Hajishirzi, Ali Farhadi, and Ludwig Schmidt. Open-
of Computer Vision, 111(1):98–136, 2015. 2 clip, 2021. 3
[19] Alex Fang, Albin Madappally Jose, Amit Jain, Ludwig [31] Varun Jampani, Kevis-Kokitsi Maninis, Andreas Engel-
Schmidt, Alexander Toshev, and Vaishaal Shankar. Data fil- hardt, Arjun Karpur, Karen Truong, Kyle Sargent, Stefan
tering networks, 2023. 2 Popov, André Araujo, Ricardo Martin-Brualla, Kaushal Pa-
[20] Takashi Fukuda, Masayuki Suzuki, Gakuto Kurata, Samuel tel, Daniel Vlasic, Vittorio Ferrari, Ameesh Makadia, Ce
Thomas, Jia Cui, and Bhuvana Ramabhadran. Efficient Liu, Yuanzhen Li, and Howard Zhou. Navi: Category-
knowledge distillation from an ensemble of teachers. In In- agnostic image collections with high-quality 3d shape and
terspeech, 2017. 3 pose annotations, 2023. 8
[21] Samir Yitzhak Gadre, Gabriel Ilharco, Alex Fang, Jonathan [32] Glenn Jocher, Ayush Chaurasia, and Jing Qiu. Ultralytics
Hayase, Georgios Smyrnis, Thao Nguyen, Ryan Marten, yolov8, 2023. 2
Mitchell Wortsman, Dhruba Ghosh, Jieyu Zhang, Eyal Or- [33] Dahun Kim, Anelia Angelova, and Weicheng Kuo. Region-
gad, Rahim Entezari, Giannis Daras, Sarah Pratt, Vivek aware pretraining for open-vocabulary object detection with
Ramanujan, Yonatan Bitton, Kalyani Marathe, Stephen vision transformers, 2023. 4, 5, 6
Mussmann, Richard Vencu, Mehdi Cherti, Ranjay Krishna, [34] Jangho Kim, SeongUk Park, and Nojun Kwak. Paraphrasing
Pang Wei Koh, Olga Saukh, Alexander Ratner, Shuran complex network: Network compression via factor transfer.
Song, Hannaneh Hajishirzi, Ali Farhadi, Romain Beaumont, In Proceedings of the 32nd International Conference on Neu-
Sewoong Oh, Alex Dimakis, Jenia Jitsev, Yair Carmon, ral Information Processing Systems, page 2765–2774, Red
Vaishaal Shankar, and Ludwig Schmidt. Datacomp: In Hook, NY, USA, 2018. Curran Associates Inc. 3
search of the next generation of multimodal datasets, 2023.
[35] Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao,
3
Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer White-
[22] Ross Girshick. Fast r-cnn. In Proceedings of the IEEE inter- head, Alexander C. Berg, Wan-Yen Lo, Piotr Dollár, and
national conference on computer vision, pages 1440–1448, Ross Girshick. Segment anything, 2023. 1, 2, 3
2015. 4
[36] Xu Lan, Xiatian Zhu, and Shaogang Gong. Knowledge dis-
[23] Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Ba-
tillation by on-the-fly native ensemble, 2018. 3
tra, and Devi Parikh. Making the V in VQA matter: Ele-
[37] Yanghao Li, Hanzi Mao, Ross Girshick, and Kaiming He.
vating the role of image understanding in Visual Question
Exploring plain vision transformer backbones for object de-
Answering. In Conference on Computer Vision and Pattern
tection, 2022. 5, 7, 6
Recognition (CVPR), 2017. 4, 8
[24] Ali Hatamizadeh, Greg Heinrich, Hongxu Yin, Andrew Tao, [38] Yanghao Li, Chao-Yuan Wu, Haoqi Fan, Karttikeya Man-
Jose M. Alvarez, Jan Kautz, and Pavlo Molchanov. Fastervit: galam, Bo Xiong, Jitendra Malik, and Christoph Feichten-
Fast vision transformers with hierarchical attention, 2023. 6, hofer. Mvitv2: Improved multiscale vision transformers for
7 classification and detection, 2022. 6
[25] B. Heo, J. Kim, S. Yun, H. Park, N. Kwak, and J. Choi. [39] Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Wayne Xin
A comprehensive overhaul of feature distillation. In 2019 Zhao, and Ji-Rong Wen. Evaluating object hallucination in
IEEE/CVF International Conference on Computer Vision large vision-language models. In The 2023 Conference on
(ICCV), pages 1921–1930, Los Alamitos, CA, USA, 2019. Empirical Methods in Natural Language Processing, 2023.
IEEE Computer Society. 2, 3 8
[26] Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distill- [40] Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee.
ing the knowledge in a neural network. arXiv preprint Improved baselines with visual instruction tuning, 2023. 2,
arXiv:1503.02531, 2015. 1, 2, 3 3, 4, 8
[27] Hanzhang Hu, Debadeepta Dey, Martial Hebert, and J. An- [41] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee.
drew Bagnell. Learning anytime predictions in neural net- Visual instruction tuning, 2023. 3
works via adaptive loss balancing. In Proceedings of the [42] Yuang Liu, Wei Zhang, and Jun Wang. Adaptive multi-
Thirty-Third AAAI Conference on Artificial Intelligence and teacher multi-level knowledge distillation. Neurocomputing,
Thirty-First Innovative Applications of Artificial Intelligence 415:106–113, 2020. 3, 5
Conference and Ninth AAAI Symposium on Educational Ad- [43] Ze Liu, Han Hu, Yutong Lin, Zhuliang Yao, Zhenda Xie,
vances in Artificial Intelligence. AAAI Press, 2019. 5, 8 Yixuan Wei, Jia Ning, Yue Cao, Zheng Zhang, Li Dong, Furu
[28] Zehao Huang and Naiyan Wang. Like what you like: Wei, and Baining Guo. Swin transformer v2: Scaling up
Knowledge distill via neuron selectivity transfer. CoRR, capacity and resolution, 2022. 6, 7, 2
abs/1707.01219, 2017. 2, 3 [44] Zhuang Liu, Hanzi Mao, Chao-Yuan Wu, Christoph Feicht-
[29] Drew A. Hudson and Christopher D. Manning. GQA: a enhofer, Trevor Darrell, and Saining Xie. A convnet for the
new dataset for compositional question answering over real- 2020s, 2022. 6

10
[45] Ilya Loshchilov and Frank Hutter. Decoupled weight de- [57] Christian Szegedy, Sergey Ioffe, and Vincent Vanhoucke.
cay regularization. In International Conference on Learning Inception-v4, inception-resnet and the impact of residual
Representations, 2019. 5 connections on learning. CoRR, abs/1602.07261, 2016. 6
[46] Pan Lu, Swaroop Mishra, Tony Xia, Liang Qiu, Kai-Wei [58] Mingxing Tan and Quoc V. Le. Efficientnetv2: Smaller mod-
Chang, Song-Chun Zhu, Oyvind Tafjord, Peter Clark, and els and faster training. CoRR, abs/2104.00298, 2021. 6
Ashwin Kalyan. Learn to explain: Multimodal reasoning [59] Zhengzhong Tu, Hossein Talebi, Han Zhang, Feng Yang,
via thought chains for science question answering. In The Peyman Milanfar, Alan Bovik, and Yinxiao Li. Maxvit:
36th Conference on Neural Information Processing Systems Multi-axis vision transformer, 2022. 6
(NeurIPS), 2022. 4
[60] Haoxiang Wang, Pavan Kumar Anasosalu Vasu, Fartash
[47] Seyed Iman Mirzadeh, Mehrdad Farajtabar, Ang Li, Nir
Faghri, Raviteja Vemulapalli, Mehrdad Farajtabar, Sachin
Levine, Akihiro Matsukawa, and Hassan Ghasemzadeh. Im-
Mehta, Mohammad Rastegari, Oncel Tuzel, and Hadi
proved knowledge distillation via teacher assistant. In AAAI
Pouransari. Sam-clip: Merging vision foundation models
Conference on Artificial Intelligence, 2019. 3
towards semantic and spatial understanding, 2023. 3, 7, 8
[48] Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy
Vo, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, [61] Yixuan Wei, Han Hu, Zhenda Xie, Zheng Zhang, Yue Cao,
Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, Mah- Jianmin Bao, Dong Chen, and Baining Guo. Contrastive
moud Assran, Nicolas Ballas, Wojciech Galuba, Russell learning rivals masked image modeling in fine-tuning via
Howes, Po-Yao Huang, Shang-Wen Li, Ishan Misra, Michael feature distillation, 2022. 2, 3
Rabbat, Vasu Sharma, Gabriel Synnaeve, Hu Xu, Hervé Je- [62] Zhirong Wu, Yuanjun Xiong, X Yu Stella, and Dahua Lin.
gou, Julien Mairal, Patrick Labatut, Armand Joulin, and Piotr Unsupervised feature learning via non-parametric instance
Bojanowski. Dinov2: Learning robust visual features with- discrimination. In Proceedings of the IEEE Conference on
out supervision, 2023. 1, 2, 3, 4 Computer Vision and Pattern Recognition, 2018. 4
[49] Junting Pan, Adrian Bulat, Fuwen Tan, Xiatian Zhu, Lukasz [63] Qizhe Xie, Minh-Thang Luong, Eduard Hovy, and Quoc V.
Dudziak, Hongsheng Li, Georgios Tzimiropoulos, and Brais Le. Self-training with noisy student improves imagenet clas-
Martinez. Edgevits: Competing light-weight cnns on mobile sification. In 2020 IEEE/CVF Conference on Computer Vi-
devices with vision transformers. In ECCV, 2022. 7, 2 sion and Pattern Recognition (CVPR), pages 10684–10695,
[50] Seonguk Park and Nojun Kwak. Feature-level ensemble 2020. 2
knowledge distillation for aggregating knowledge from mul- [64] Hu Xu, Saining Xie, Xiaoqing Ellen Tan, Po-Yao Huang,
tiple networks. In European Conference on Artificial Intelli- Russell Howes, Vasu Sharma, Shang-Wen Li, Gargi Ghosh,
gence, 2020. 3 Luke Zettlemoyer, and Christoph Feichtenhofer. Demystify-
[51] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya ing clip data. 2023. 2
Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, [65] Deshraj Yadav, Rishabh Jain, Harsh Agrawal, Prithvijit Chat-
Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen topadhyay, Taranjeet Singh, Akash Jain, Shiv Baran Singh,
Krueger, and Ilya Sutskever. Learning transferable visual Stefan Lee, and Dhruv Batra. Evalai: Towards better evalua-
models from natural language supervision. In Proceedings tion systems for ai agents, 2019. 8
of the 38th International Conference on Machine Learning,
[66] Jiawei Yang, Boris Ivanovic, Or Litany, Xinshuo Weng, Se-
pages 8748–8763. PMLR, 2021. 1, 2, 3, 4
ung Wook Kim, Boyi Li, Tong Che, Danfei Xu, Sanja Fidler,
[52] Ilija Radosavovic, Raj Prateek Kosaraju, Ross Girshick, Marco Pavone, and Yue Wang. Emernerf: Emergent spatial-
Kaiming He, and Piotr Dollár. Designing network design temporal scene decomposition via self-supervision, 2023. 5,
spaces, 2020. 6 6
[53] Adriana Romero, Nicolas Ballas, Samira Ebrahimi Kahou,
[67] Jiawei Yang, Katie Z Luo, Jiefeng Li, Kilian Q Weinberger,
Antoine Chassang, Carlo Gatta, and Yoshua Bengio. Fitnets:
Yonglong Tian, and Yue Wang. Denoising vision transform-
Hints for thin deep nets. CoRR, abs/1412.6550, 2014. 2, 3
ers, 2024. 5, 6
[54] Christoph Schuhmann, Richard Vencu, Romain Beaumont,
Robert Kaczmarczyk, Clayton Mullis, Aarush Katta, Theo [68] Ze Yang, Linjun Shou, Ming Gong, Wutao Lin, and Daxin
Coombes, Jenia Jitsev, and Aran Komatsuzaki. Laion-400m: Jiang. Model compression with two-stage multi-teacher
Open dataset of clip-filtered 400 million image-text pairs, knowledge distillation for web question answering system.
2021. 3 In Proceedings of the 13th International Conference on Web
[55] Amanpreet Singh, Vivek Natarajan, Meet Shah, Yu Jiang, Search and Data Mining, page 690–698, New York, NY,
Xinlei Chen, Dhruv Batra, Devi Parikh, and Marcus USA, 2020. Association for Computing Machinery. 3
Rohrbach. Towards vqa models that can read. In Proceed- [69] Shan You, Chang Xu, Chao Xu, and Dacheng Tao. Learning
ings of the IEEE/CVF Conference on Computer Vision and from multiple teacher networks. In Proceedings of the 23rd
Pattern Recognition (CVPR), 2019. 4, 8, 12, 13, 14 ACM SIGKDD International Conference on Knowledge Dis-
[56] X. Sun, R. Panda, C. Chen, A. Oliva, R. Feris, and K. covery and Data Mining, page 1285–1294, New York, NY,
Saenko. Dynamic network quantization for efficient video USA, 2017. Association for Computing Machinery. 3
inference. In 2021 IEEE/CVF International Conference on [70] Weihao Yu, Mi Luo, Pan Zhou, Chenyang Si, Yichen Zhou,
Computer Vision (ICCV), pages 7355–7365, Los Alamitos, Xinchao Wang, Jiashi Feng, and Shuicheng Yan. Metaformer
CA, USA, 2021. IEEE Computer Society. 2, 3 is actually what you need for vision, 2022. 6

11
[71] Fei Yuan, Linjun Shou, Jian Pei, Wutao Lin, Ming Gong,
Yan Fu, and Daxin Jiang. Reinforced multi-teacher selection
for knowledge distillation, 2020. 3
[72] Sergey Zagoruyko and Nikos Komodakis. Paying more at-
tention to attention: Improving the performance of convolu-
tional neural networks via attention transfer. In 5th Interna-
tional Conference on Learning Representations, ICLR 2017,
Toulon, France, April 24-26, 2017, Conference Track Pro-
ceedings. [Link], 2017. 2, 3
[73] Shuangfei Zhai, Tatiana Likhomanenko, Etai Littwin, Dan
Busbridge, Jason Ramapuram, Yizhe Zhang, Jiatao Gu, and
Joshua M Susskind. Stabilizing transformer training by pre-
venting attention entropy collapse. In International Con-
ference on Machine Learning, pages 40770–40803. PMLR,
2023. 6
[74] Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and
Lucas Beyer. Sigmoid loss for language image pre-training.
arXiv preprint arXiv:2303.15343, 2023. 2
[75] Haoran Zhao, Xin Sun, Junyu Dong, Changrui Chen, and
Zihe Dong. Highlight every step: Knowledge distillation via
collaborative teaching. IEEE Transactions on Cybernetics,
52(4):2070–2081, 2022. 3
[76] Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan
Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan
Li, Dacheng Li, Eric. P Xing, Hao Zhang, Joseph E. Gonza-
lez, and Ion Stoica. Judging llm-as-a-judge with mt-bench
and chatbot arena, 2023. 8
[77] Bolei Zhou, Hang Zhao, Xavier Puig, Sanja Fidler, Adela
Barriuso, and Antonio Torralba. Scene parsing through
ade20k dataset. In 2017 IEEE Conference on Computer
Vision and Pattern Recognition (CVPR), pages 5122–5130,
2017. 2
[78] Konrad Zuchniak. Multi-teacher knowledge distillation as an
effective method for compressing ensembles of neural net-
works, 2023. 2, 3

12
AM-RADIO: Agglomerative Vision Foundation Model
Reduce All Domains Into One
Supplementary Material

1
 Stage 1 Stage 2 Stage 3 Stage 4

Downsample

Downsample

Downsample
Conv Conv Multi-Resolution Multi-Resolution
Input Conv (s2) Conv (s2) Head
Block Block Attention Attention

Figure 6. High level architecture of the ERADIO network architecture. Overall architecture is composed of multiple stages: 1) the stem,
2) 2 convolutional blocks from YOLOv8, 3) 2 transformer blocks with multi-resolution windowed self attention.

Figure 7. Multi-resolution attention for E-RADIO

A. E-RADIO architecture details

The architecture of E-RADIO is illustrated in Figure 6. It is a hybrid CNN-Transformer architecture. First 2 stages follow
convolution paradigm and have the C2f architecture from YOLOv8 model [32]. The last 2 stages have the Transformer
architecture with windowed attention and multi-resolution attention (MRA) structure. Every stage, except the last one, are
followed by downsample block. We implement it as a strided convolution with 3x3 kernel and stride 2, followed by batch
normalization layer.
A.1. Multi-Resolution Attention
Standard transformers struggle to scale with high input image resolution because of quadratic complexity of the attention.
SWIN [43] proposed to use windowed attention to reduce the complexity of attention. We reuse windowed attention in the
E-RADIO. To address for missing communication between windows, SWIN introduced window shifting, unfortunately, it
has non-negligible compute cost. Instead, we propose multi-resolution attention inspired by EdgeViT’s Local-Global-Local
attention [49]. The idea is illustrated in Figure 7. Every layer in the transformer will have a local windowed attention
with optional subsampling via convolutional operator. For example, if susbampling is dissabled, then it is just a standard
windowed attention. If the subsampling ratio is 2, then the feature map is downsampled by a factor of 2, windowed attention
is performed, and then the feature map is upsampled to the original resolution with deconvolution. For FasterVIT2 models,
we interleave subsampled attention with ratio 2 and the normal attention with no subsampling.
A.2. Configurations
All models in the family follow the same configuration except the embedding dimension (hide dimension). We simply scale
it up with bigger models. Other parameters:
• Input resolution is 224
• In-stem contains 2 3x3 convolutions with stride 2
• Total stages: 2 convolutional and 2 transformer
• First stage takes input feature size of 56x56, has 3 layers with C2f structure from YOLO8 [32].
• Second stage takes input feature size of 28x28, has 3 layers of C2f.

2
• Third stage takes features of size 14x14, has 5x multi-resolution attention, window size 7.
• Forth stage takes features of size 7x7, has 5x windowed attention of window size 7.
• Embedding dimension for different model variants: XT - 64, T - 80, S - 96, B - 128, L - 192. The smallest XT and T
models have [1, 3, 4, 5] layers for each of 4 stages.
• Output features have resolution of 14x14 and are obtained by upsampling the features of stage 4 by 2x with deconvolution
and adding to stage 3 features of size 14x14.

B. PCA Visualizations

We visualize various models using PCA to reduce the model’s spatial feature dimensionality down to 3 dimensions, and
directly map those to RGB. Most models are only able to handle square inputs at fixed resolutions, however DINOv2 and
RADIO can handle arbitrary resolutions and aspect ratios, so we visualize them in both settings.

3
4
B.1. Square Models
Model Resolution Images

OpenCLIP-H/14 224

MetaCLIP-H/14 224

SigLIP-M/14 384

InternViT-6B 224

448

DFN CLIP 378

OpenAI CLIP 336

DINOv2-g 518

SAM-H 1024

RADIO 512

1024

5
B.2. Flexible Models
Model Resolution Images

DINOv2-g 518

1022

2044

RADIO 512

1024

2048

C. ViTDet Augmentation
The following python code shows how the alternating window/global architecture of ViTDet [37] can be applied to a trans-
former. We take advantage of the fact that transformers are permutation invariant after position encodings have been applied,
and thus it’s easy to organize the patch order such that contiguous chunks of patches belong to the same window. Once
reordered in this way, alternating between windowed and global attention is achieved simply by absorbing the windows into
the batch dimension or returning to the original shape respectively. We also enforce that the final transformer layer always
applies global attention.
from e i n o p s import r e a r r a n g e
def r e o r d e r p a t c h e s ( p a t c h e s : t o r c h . Tensor ,

6
 p a t c h e d s i z e : Tuple [ int , i n t ] ,
window size : int ) :
p idxs = t o r c h . arange ( patches . shape [ 1 ] )
p i d x s = r e a r r a n g e ( p i d x s , ’ ( wy y wx x ) −> ( wy wx y x ) ’ ,
wy= p a t c h e d s i z e [ 0 ] / / w i n d o w s i z e , y= w i n d o w s i z e ,
wx= p a t c h e d s i z e [ 1 ] / / w i n d o w s i z e , x= w i n d o w s i z e )
p i d x s = p i d x s . r e s h a p e ( 1 , −1 , 1 ) . e x p a n d a s ( p a t c h e s )

return t o r c h . g a t h e r ( patches , p i d x s ) , p i d x s

d e f v i t d e t a u g ( b l o c k s : nn . S e q u e n t i a l ,
p a t c h e s : t o r c h . Tensor ,
p a t c h e d s i z e : Tuple [ int , i n t ] ,
window sizes : L i s t [ int ] ,
num windowed : i n t ) :
B, T , C = patches . shape
window size = sample ( window sizes )
s q w i n d o w s i z e = w i n d o w s i z e ** 2
patches , p idxs = r e o r d e r p a t c h e s ( patches , patched size , window size )
p e r i o d = num windowed + 1
f o r i , b l o c k i n enumerate ( b l o c k s [ : − 1 ] ) :
i f i % p e r i o d == 0 :
p a t c h e s = p a t c h e s . r e s h a p e ( B * s q w i n d o w s i z e , −1 , C )
e l i f i % p e r i o d == num windowed :
p a t c h e s = p a t c h e s . r e s h a p e (B , T , C)
patches = block ( patches )

# Always use g l o b a l a t t e n t i o n w i t h t h e l a s t b l o c k
p a t c h e s = p a t c h e s . r e s h a p e (B , T , C)
patches = blocks [ −1]( patches )

# F i n a l l y , put t h e p a t c h e s back i n i n p u t order

ret = torch . empty like ( patches )
r e t = r e t . s c a t t e r ( dim =1 , i n d e x = p i d x s , s r c = p a t c h e s )
return r e t

D. Comparison with SAM-CLIP [60]

Concurrently with our work, SAM-CLIP was introduced as a method of fusing SAM and CLIP into a single model. Due to
the concurrency of effort, we don’t compare our model with the full suite of metrics demonstrated in their method, however,
we do have some overlap in key metrics such as Zero-Shot ImageNet-1k, and ADE20k semantic segmentation via linear
probing. We present the comparison in table 10, however we note that there are enough differences between these two
models that we can’t conclude one way or another what is the superior approach. Instead we’ll argue that DINOv2 does a
better job of ADE20k linear probing than SAM, and thus our significantly higher quality on this metric is likely due to the
inclusion of DINOv2, which is a key introduction with our approach.

E. Automatic Loss Balancing

E.1. Uncertainty
Following [12], we have:

L(x) = \sum _k \frac {1}{2\sigma _k^2} L_k(x) + \log \sigma _k \label {eq:uncertainty_definition} (4)

7
 Family Model Zero-Shot ADE20k
SAM ViTDet-H/16 28.2
DFN CLIP ViT-H/14 83.9 31.7
SAM-CLIP ViTDet-B/16 71.7 38.4
RADIO ViT-H/14 82.7 51.3

Table 10. We compare our common key metrics with those demonstrated in SAM-CLIP [60]. We note that there are numerous differences
between the two approaches, including model capacity and architecture. SAM-CLIP uses the ViT-B variant of SAM as a starting point,
which implies it’s a ViTDet-B/16 architecture. As a result of this choice, their metrics are computed at a resolution of 1024. RADIO
trains a vanilla ViT-H/14 from scratch, and as a result of the flexibility gained via the CPE method, we evaluate Zero-Shot ImageNet1k at a
resolution of 432, and we run ADE20k linear probing at a resolution of 512 using the exact same weights. We note that Zero-Shot quality
is largely determined by the quality of the CLIP teacher and the capacity of the student. We attribute our superior quality on ADE20k
semantic segmentation largely to our inclusion of DINOv2 as a teacher.

where the σk values are predicted by the student. In practice, the student predicts b :“ log σk2 for numerical stability, to
avoid division by zero, and to regress unconstrained scalar values.
We make some minor modifications to (4) to make training a bit more stable in our setting. We replace the manual λ
scalars with the learned uncertainty weights, and add the loss term for large uncertainties. Altogether, this yields:

\begin {aligned} \lambda _k &= \frac {e^{-b_k}}{2} \\ L(x) &= \sum _k \lambda _k L_k(x) + \frac {b_k}{2} \end {aligned} \label {eq:uncertainty_definition_2}
(5)

ps|vq psq
Let bi px1 |Θi q be a learned function predicting balance parameters for teacher i and summary weight psq or feature
vector weight pvq, we transform equation (5) slighty to:

\begin {aligned} \psi (x) &= \log (1 + e^x) \\ \lambda _i^{(m)} &= e^{-b_i^{(m)}(x')} \\ L(x) &= \sum _i \sum _{m \in \{s,v\}} \lambda _i^{(m)} L_i^{(m)}(x) + \psi \left (b_i^{(m)}(x')\right ) \end {aligned} \label {eq:uncertainty_ours}

(6)

The function ψpxq is the familiar “softplus” nonlinear activation function. We drop the division by 2 on the left because,
assuming outputs are initially b „ N p0, σ 2 q, then the loss weights will initially have an expected value of 1, matching the
naive weighting. On the right, we replace b2k with ψpxq for a few reasons:
• When x Ç 4, then ψpxq « x, yielding the same expression as before.
• When x « 0, then ψ 1 pxq « 12 , yielding the same expression as before.
• When x ă 0, which translates to a loss weight ą 1, ψ 1 pxq Ñ 0, improving stability as the weight gets larger.
• It has range p0, 8q which aesthetically enforces the loss to be greater than zero.

E.2. AdaLoss
In addition to uncertainty auto-balancing, we also explored AdaLoss [27]. In this formulation, we have:

\begin {aligned} \lambda _i^{(m)} &= \frac {1}{\mathbb {E}(L_i^{(m)})} \\ L(x) &= \sum _i \sum _{m \in \{s,v\}} \lambda _i^{(m)} L_i^{(m)}(x) \end {aligned} \label {eq:adaloss}
(7)

F. Visual Question Answering Samples

Figures 9 to 13 show sample questions from our Visual Question Answering datasets, together with sample answers when
using our vision encoders in a LLaVA setup.

8
Figure 8. Visualization of the LLaVA attention maps over the visual features produced by a RADIO encoder. We use one sample image
from the GQA[29] validation set and one associated question: ”What color is the helmet in the middle of the image?”. For each layer in
the language model, we retrieve attention scores for all positions of the visual tokens, average them over all attention heads, and overlay
corresponding heat maps with the input image. We can see that as we progress through the layers, the model’s attention focuses on the
relevant part of the image. The model’s answer is ”Blue”.

9
 MetaCLIP DINO-g14 RADIO-H14 E-RADIO MetaCLIP DINO-g14 RADIO-H14 E-RADIO MetaCLIP DINO-g14 RADIO-H14 E-RADIO MetaCLIP DINO-g14 RADIO-H14 E-RADIO
Q:Does the boat to the left of the flag look small or large? A: small Q:Which kind of furniture is right of the drawers? A: shelves Q:Who is in front of the gray building? A: passengers Q:What is the color of the fence made of metal? A: green
Small Small Small Small Shelves Shelf Table Shelf People People People People Brown Green Green Green

Q:Is the boat to the left of the other boat small and white? A: yes Q:Is the lamp to the right or to the left of the pot? A: right Q:What are the passengers in front of? A: building Q:Which kind of animal is to the right of the sheep? A: cow
No Yes Yes Yes Right Right Right Right Building Building Building Airport Goat Goat Goat Goat
Q:Is the pan to the right of a bowl? A: no Q:Is there a window that is not large? A: yes Q:What is the small animal? A: cow

No No No No No No No No Goat Sheep Sheep Goat

Q:How are the items of furniture to the right of the vegetables that are not rotten called? A: shelves Q:Are there any doors or cars in the picture? A: no Q:Is the sheep white or black? A: black
Shelves Shelves Shelves Shelves No Yes No Yes Black Black White White
Q:Which kind of appliance is to the right of the shelf? A: stove Q:Who is before the building? A: passengers Q:Are there both fences and goats in this photograph? A: yes

Oven Blender Oven Oven People People People People No Yes Yes Yes
Q:Are there stoves to the left of the shelf? A: no Q:Who is boarding the airplane? A: passengers Q:Are there either fences or toilets? A: yes
No No No No People People People People No Yes Yes Yes

Q:Which kind of furniture is to the left of the pot? A: shelf Q:Who is boarding the plane? A: passengers Q:What type of animal is to the right of the black animal? A: cow
Shelf Shelf Shelf Shelf People People People People Sheep Sheep Sheep Sheep
Q:Which kind of material makes up the stove? A: wood Q:Are there any trucks? A: no Q:Do the animals in this photo have different species? A: yes
Wood Wood Wood Brick No No No No Yes Yes No No

Q:Is the floor gray or red? A: gray Q:Is the gray building behind the passengers that are boarding the airplane? A: yes Q:What is the animal to the right of the goat on the left? A: cow
Gray Gray Gray Gray Yes Yes Yes Yes Sheep Sheep Sheep Sheep
Q:Is the wood stove to the right or to the left of the shelf? A: right Q:Does the logo look white and large? A: no Q:Are there either horses or sheep that are not black? A: no

Right Right Right Right Yes Yes Yes Yes No No No No

Q:Which kind of furniture is to the left of the shelves? A: drawers Q:Are there both a fence and a goat in this photo? A: yes
Shelf Shelf Shelf Shelf No No Yes Yes

Q:What is the color of the container that is filled with water? A: brown Q:Does the goat's tail look small and gray? A: yes
Brown Brown Black Blue Yes Yes Yes Yes
Q:Are the shelves and the drawers made of the same material? A: yes Q:Is the sheep to the right or to the left of the animal that is not big? A: left

No Yes Yes No Right Right Right Right

Q:Is the wood table to the left of the appliance the pot is on? A: no Q:What animal is to the left of the small animal? A: goat

No No No No Goat Sheep Sheep Goat

Q:Is there any kettle on the appliance that the vegetable is to the left of? A: no Q:Are there any horses or cows that are not small? A: no
Yes No Yes No No No No No

Q:Is the container that looks brown filled with water? A: yes
Yes Yes Yes No
Q:What are the shelves made of? A: wood

Wood Wood Wood Wood

Q:On which side of the image is the pot? A: right
Left Right Right Right

Q:In which part of the picture is the container, the top or the bottom? A: bottom
Top Top Top Top
Q:Are there any lamps or beds in the photograph? A: yes

No No No No
Q:What is on the stove? A: pot

Pot Pot Pot Pot

Q:Are the drawers in the bottom part or in the top of the picture? A: top
Bottom Top Top Bottom

Q:What is the floor made of? A: brick

Brick Stone Stone Stone
Q:Is the floor made of hardwood? A: no

No No No No
Q:Is the wood stove to the right of the wood table? A: no
Yes No No Yes

Q:What is the item of furniture to the left of the shelves made of wood? A: shelf
Chair Shelf Shelf Shelf
Q:What is the stove made of? A: wood

Wood Wood Wood Wood

Figure 9. Sample questions from the GQA[29] and their answers from our LLaVA models, using various image encoders. Answers are
painted green when they match the ground truth, pink otherwise.

10
 MetaCLIP DINO-g14 RADIO-H14 E-RADIO MetaCLIP DINO-g14 RADIO-H14 E-RADIO MetaCLIP DINO-g14 RADIO-H14 E-RADIO MetaCLIP DINO-g14 RADIO-H14 E-RADIO
Q:Is the woman to the left or to the right of the man that is wearing trousers? A: right Q:Does the remote control to the left of the other remote control look black? A: yes Q:Is it indoors or outdoors? A: outdoors Q:Does the belt look tan? A: no
Left Left Right Right Yes Yes Yes Yes Outdoors Outdoors Outdoors Outdoors No No No No

Q:Who is wearing a shirt? A: man Q:What is the device above the box? A: controller Q:Which side of the photo is the orange vegetable on? A: right Q:Is the baseball the same color as the field? A: no
Man Man Man Woman Television Television Television Television Right Right Right Right Yes Yes Yes Yes
Q:Is the bag in the top part or in the bottom of the photo? A: bottom Q:Which kind of device is above the box? A: controller Q:Does the shirt look green? A: no Q:Which kind of clothing is gray? A: trunks

Bottom Bottom Bottom Bottom Television Television Television Television No No No No Pants Pants Pants Pants
Q:Is the bag to the right of the other bag tan or black? A: black Q:Is the controller above the container to the left of the speaker? A: yes Q:On which side of the picture is the person? A: right Q:Which place is it? A: field
Black Tan Tan Tan Yes Yes Yes Yes Right Right Right Right Field Field Field Field
Q:Who is wearing the glasses? A: man Q:Are there any tables or couches that are not tan? A: no Q:Does the bracelet look blue? A: yes Q:Is the shirt the same color as the belt? A: yes

Man Man Man Woman No No No No Yes Yes Yes Yes Yes Yes Yes Yes
Q:What is the color of the logo? A: black Q:What is the color of the table? A: tan Q:What animal is to the left of the house? A: giraffe Q:Is the cap black? A: yes
White White White Red Brown Brown Brown Brown Giraffe Giraffe Giraffe Giraffe Yes No Yes Yes

Q:Do you see women to the left of the man that is wearing pants? A: no Q:What is the device on the carpet? A: speaker Q:Is the giraffe to the left or to the right of the person that is on the right? A: left Q:What color are the trunks, gray or yellow? A: gray
No Yes No Yes Remote control Remote control Speaker Television Left Left Left Left Gray Gray Gray Gray
Q:What does the man to the left of the traffic light wear? A: suit Q:Is there any speaker on the carpet? A: yes Q:Which place is it? A: field Q:What are the gray clothing items called? A: trunks
Jacket Jacket Jacket Jacket Yes Yes Yes No Field Zoo Field Field Pants Pants Pants Pants

Q:What vehicle is the black logo painted on? A: van Q:What device is on the carpet? A: speaker Q:Are there both fences and giraffes in the image? A: yes Q:What color does the shirt have? A: black
Van Van Van Sign Remote control Remote control Speaker Television Yes Yes Yes Yes Black Blue Black Black
Q:What's painted on the door? A: logo Q:Which color is the speaker that is to the right of the remote? A: black Q:Which kind of animal is this, a giraffe or a bear? A: giraffe Q:Do the baseball and the number have a different colors? A: yes

Nothing Nothing Nothing Logo Black Black Black Black Giraffe Giraffe Giraffe Giraffe No No Yes No
Q:What is the color of the car? A: yellow Q:Is the remote in the top part or in the bottom of the photo? A: bottom Q:Is the person to the right or to the left of the animal near the fence? A: right
White Yellow Yellow White Bottom Bottom Bottom Bottom Right Right Right Right

Q:Is the jacket different in color than the logo? A: no Q:What size is the device that is in the top of the image? A: small Q:Is the person to the right of the animal near the fence? A: yes
Yes Yes No No Large Large Large Large Yes Yes Yes Yes
Q:Which color is the sweater the woman wears? A: white Q:Is it outdoors? A: no Q:Is the house on the right side? A: yes

Black Black Brown Black No No No No Yes Yes Yes Yes

Q:What does the woman wear? A: sweater Q:In which part of the photo is the box, the bottom or the top? A: bottom Q:What kind of vegetable is to the right of the giraffe? A: carrot

Jacket Jacket Skirt Jacket Bottom Bottom Bottom Bottom Carrot Carrot Carrot Carrot
Q:Is the blue bag in the bottom or in the top? A: bottom Q:In which part is the small device, the bottom or the top? A: top
Bottom Bottom Bottom Bottom Bottom Bottom Bottom Bottom

Q:What is the color of the jacket the man wears? A: black Q:What is on the tan table? A: remote control
Black Black Black Black Remote control Remote Remote Television
Q:What is painted on the van to the right of the man? A: logo Q:Which kind of device is on the table? A: remote control

Logo Logo Logo Logo Remote control Remote control Remote control Television
Q:Is the car in the top part of the photo? A: no Q:What type of device is on the table? A: remote control
No No No No Remote control Remote control Remote Television

Q:Is the yellow vehicle to the right or to the left of the man that wears jeans? A: left Q:What color is the carpet? A: gray
Left Left Left Left Gray Brown Gray White
Q:Who is wearing a jacket? A: man

People Man Man People

Q:Is the woman to the right of the man carrying a bag? A: yes

Yes Yes Yes No

Q:What is the woman that is to the right of the man carrying? A: bag
Bag Bag Bag Bag

Q:Does the door look white? A: yes

Yes No No Yes
Q:Who is wearing jeans? A: man

People Man People People

Q:Are there either chairs or bags? A: yes
Yes Yes Yes No

Q:Are there blue bags or cars? A: yes

Yes Yes Yes Yes
Q:Which side of the picture is the van on? A: right

Right Right Right Right

Q:On which side of the picture is the woman? A: left

Left Left Right Left

Q:Is the blue bag to the right or to the left of the woman that wears a sweater? A: left
Left Left Left Left

Q:Is the man to the left of the bun wearing shorts? A: no

No No No No
Q:Who is wearing the pants? A: man

Man Man Man Woman

Q:Is the car different in color than the jacket? A: yes

Yes Yes Yes Yes

Q:Is the color of the door different than the van? A: no

No No No No

Figure 10. Sample questions from the GQA[29] and their answers from our LLaVA models, using various image encoders. Answers are
painted green when they match the ground truth, pink otherwise.

11
 MetaCLIP DINO-g14 RADIO-H14 E-RADIO MetaCLIP DINO-g14 RADIO-H14 E-RADIO MetaCLIP DINO-g14 RADIO-H14 E-RADIO MetaCLIP DINO-g14 RADIO-H14 E-RADIO
Q:What is the brand of this camera? A: dakota, clos culombu, Q:What does the small white text spell? A: copenhagen, Q:What kind of beer is this? A: self righteous, sublimely Q:What brand liquor is on the right? A: bowmore , bowmore,

nous les gosses, dakota digital thursday self-righteous ale, ale, stone bowmore islay, dowmore islay

Dakota Dakota digital Dakota Dakota Drupalcon copenhagen

Rupertcon Drupalcon copenhagen
Palcon copenhagen
Self-righteous Stone self-rich Ale Stone self-rich Ownmor Morangie Bowmore Morangie

Q:How long has the drink on the right been aged? A: 10 year,

10 years , 10, 10 years, martial arts

10 years 10 years 10 years 10 years

MetaCLIP DINO-g14 RADIO-H14 E-RADIO MetaCLIP DINO-g14 RADIO-H14 E-RADIO MetaCLIP DINO-g14 RADIO-H14 E-RADIO MetaCLIP DINO-g14 RADIO-H14 E-RADIO

Q:What number is on the player's jersey? A: 22 Q:What is the time? A: 5:42, 5:41, 8:00, 5:40 Q:Who is at the center of all of this? A: agile experience Q:Who was the photographer? A: philippe molitor, philippe
22 22 22 22 11:00 11:00 11:55 11:00 design makeup, bryan owens, alexa curtis, mahou molitar, no, phillipe meltow, l, clardajne, phillipe molida

Q:What brand of watch is that? A: unanswerable, audemars, Aithell Man Chris O'Leary Owens Philippe molitor Philippe molitor Philippe molitor Philippe molitor
ap, af

Tissot Tissot Tudor Rolex

Figure 11. Sample questions from the TextVQA [55] dataset and their answers from our LLaVA models, using various image encoders.
Answers are painted green when they match the ground truth, pink otherwise.

MetaCLIP DINO-g14 RADIO-H14 E-RADIO MetaCLIP DINO-g14 RADIO-H14 E-RADIO MetaCLIP DINO-g14 RADIO-H14 E-RADIO MetaCLIP DINO-g14 RADIO-H14 E-RADIO
Q:Are these switches on or off? A: off Q:What candy bar is down there on the bottom? A: hershey's, Q:What does the light sign read on the farthest right window? Q:Is this denny's? A: pet center, yes, one man show
Off Off Off Off hersheys A: all 2 liters, bud light Yes Yes Yes No

Honey maid Honey maid Honey maid Honey maid

12 Bud light Light Light
Q:How much for a can of skoal? A: $3.32, $3.82, 3.82
452 Q:What color are the letters on this sign? A: red, answering
does not require reading text in the image, first, reds

4.52 1.99 1.99 1.99 Red Red Red Yellow

 MetaCLIP DINO-g14 RADIO-H14 E-RADIO MetaCLIP DINO-g14 RADIO-H14 E-RADIO MetaCLIP DINO-g14 RADIO-H14 E-RADIO MetaCLIP DINO-g14 RADIO-H14 E-RADIO
Q:What is the 3 letter word to the left of casa in the text? A: Q:What year was this made? A: 2012 Q:Is this a reference book? A: foreign words, yes Q:What is the license plate number? A: jiba, no numbers but

fica, tua 2012 2012 2012 2012 Yes Yes Yes No the letters jiba, items handes into london undergrounf lost

Libano Casa Jes Dos property

JIBA BURLINGAME Jiba Burl

MetaCLIP DINO-g14 RADIO-H14 E-RADIO MetaCLIP DINO-g14 RADIO-H14 E-RADIO MetaCLIP DINO-g14 RADIO-H14 E-RADIO MetaCLIP DINO-g14 RADIO-H14 E-RADIO

Q:What is the alcohol content? A: 9%, 2009, 9.0, 9.0% alc/vol., Q:What is the beer brand front center? A: coors light, coors, Q:Who is usa today's bestselling author? A: cathy williams Q:What is this food place selling? A: bratwurst, wurst,
9, smashed pumpkin, 9.0%, lego coors light , secret Cathy williams Cathy williams Cathy williams Cathy williams krainerwurst, burenwurst, hotdogs, krainerwurst and

9.0% 9.0% 9.0% 9.0% Coors light Coors light Coors light Coors light Q:What is the name of this bestselling books? A: secrets of a burenwurst, krainerwurst burenwurst, krainerwurst,
Q:What is the name of this ale? A: smashed pumpkin, Q:What is the company name to the left of the coors logo? A: ruthless tycoon, cathy williams, secret of ruthless tycoon burenwurst

shipyard, shipyard smashed pumpkin safeway, calculator, safeway Cathy williams Harlequin Presents
Cathy williams Harlequin presents
Hot dog Hot dog Fran Debreziner Hot dog
Smashed pumpkin
Shipyard
ale smashed
Shipyard
pumpkin ale Shipyard smashed
Coors
pumpkin ale Coors Safeway Pg&e Q:What is the top word on the sign on the left? A: krainerwurst
Krainerwurst Hot dog Krainerwurst Hot dog

Figure 12. Sample questions from the TextVQA [55] dataset and their answers from our LLaVA models, using various image encoders.
Answers are painted green when they match the ground truth, pink otherwise.

MetaCLIP DINO-g14 RADIO-H14 E-RADIO MetaCLIP DINO-g14 RADIO-H14 E-RADIO MetaCLIP DINO-g14 RADIO-H14 E-RADIO MetaCLIP DINO-g14 RADIO-H14 E-RADIO
Q:What does his jacket say? A: fox Q:What brand of diaper area is this? A: koala bear, koala kare Q:What time is it? A: 7:28 Q:What's the name of the book on the top of the pile? A: ariel
Fox Coca cola Fox Coca cola Koala kare Koala kare Koala kare Koala kare 7:28 7:28 7:28 7:28 Saint joan The painted veil The painted veil The painted veil

Q:What brand is in white letters with a red background? A:

cocacola, fox, coca-cola, coca cola
Q:What animal is shown on the changing table? A: koala bear,
bear, koala
13 Q:What is the title of the green book that is second from the
bottom? A: the speaking eye

Coca cola Coca cola Coca cola Coca cola Koala Koala Koala Koala Saint joan The painted veil The painted veil The painted veil
 MetaCLIP DINO-g14 RADIO-H14 E-RADIO MetaCLIP DINO-g14 RADIO-H14 E-RADIO MetaCLIP DINO-g14 RADIO-H14 E-RADIO MetaCLIP DINO-g14 RADIO-H14 E-RADIO
Q:What brand of cellphone is this? A: like you, verizon, Q:What brand is the remote control? A: kicker Q:What channel is this helicopter from? A: joma, fox hd, fox Q:What's the name of the store? A: tanamira, tanamera,

verizon nokia Kicker Kickstick Kicker Kick Fox Fux nit Fux Fux tanamela

Verizon Verizon Verizon Verizon Q:What is the text to the right of fox? A: hd Ana Ana mer Anamelia Ana mer

Q:Was this picture sent? A: le web, yes Fox NIT Fox 1 Nitro

Yes Yes Yes Yes

MetaCLIP DINO-g14 RADIO-H14 E-RADIO MetaCLIP DINO-g14 RADIO-H14 E-RADIO MetaCLIP DINO-g14 RADIO-H14 E-RADIO MetaCLIP DINO-g14 RADIO-H14 E-RADIO

Q:Who is the author of this book? A: kimberly kay hoant, Q:What is the company on the box? A: silicongraphics, silicon Q:How much does the coin weight? A: 1oz, 1 oz., 1 ounce, 1 oz Q:What is the flavor of the beer on the left? A: amber, ambree
kimberely kay hoang, kimberly kay hoang graphics, silicon graphics , silicon grphics 104 104 1 oz 104 Tourmente Blonde Blonde Blanche

Kimberly Kay Hoang

Kimberly Kay Hoang
Kimberly Kay Hoang
Kimberly Kay Hoang
Silicon graphics Silicon graphics Silicon graphics Silicon graphics Q:Now coin using or not? A: unanswerable, no, answering
Q:What is the book title? A: dealing in desire Q:How wide is the diagonal screen? A: 17.3, 17.3 inch, 17.3 does not require reading text in the image

Dealing in Desire:Dealing
Asian Ascendancy,
in Desire:Dealing
Asian
Western
Ascendancy,
in Desire:
Decline,
Dealing
Asian
Western
and
Ascendancy,
inthe
desire:
Decline,
Hidden
inches
Asian
Currencies
Western
and
ascendancy,
the Decline,
Hidden
of Global
Western
Currencies
andSex
thedecline,
Work
Hidden
of Global
and
Currencies
Sex
the hidden
Work
of Global
currencies
Sex Work
of global
Not sex work Not Not Not
1600 1600 1600 1600

Figure 13. Sample questions from the TextVQA [55] dataset and their answers from our LLaVA models, using various image encoders.
Answers are painted green when they match the ground truth, pink otherwise.

MetaCLIP DINO-g14 RADIO-H14 E-RADIO MetaCLIP DINO-g14 RADIO-H14 E-RADIO MetaCLIP DINO-g14 RADIO-H14 E-RADIO MetaCLIP DINO-g14 RADIO-H14 E-RADIO
Q:What word is on the box? A: reynolds, frict, reynolds Q:What does the sign at the top right say is not allowed? A: Q:Which program is seen on the screen? A: office, microsoft Q:What brand is this laptop? A: motorola
Reynolds Reynolds Reynolds Reynolds minors, no minors, smoking, minor office Motorola Motorola Motorola Motorola

Smoking Smoking Smoking Smoking

14 Office Office Office Office
Q:What program is being opened on the computer? A: office
Q:What kind of coins are printed in gray area of the screen? A:
unanswerable, bitcoin

Office Office Microsoft office Office Motorola Bitcoin None Dollars