Enhancing CLIP with GPT-4: Harnessing Visual Descriptions as Prompts

Mayug Maniparambil, Chris Vorster, Derek Molloy, Noel Murphy,

Kevin McGuinness, Noel E. O’Connor
ML Labs, Dublin City University,
Dublin, Ireland
arXiv:2307.11661v2 [[Link]] 8 Aug 2023

Abstract constrained to a fixed set of classes. In CLIP, the classifier

is constructed by plugging the class name into a predeter-
Contrastive pretrained large Vision-Language Models mined prompt template like ‘a photo of {class name}’. A
(VLMs) like CLIP have revolutionized visual representation straightforward way to adapt CLIP to different domains is
learning by providing good performance on downstream by prompt engineering, which usually involves modifying
datasets. VLMs are 0-shot adapted to a downstream dataset the prompt template to include semantic information about
by designing prompts that are relevant to the dataset. Such the target task. For example, to classify bird images, one
prompt engineering makes use of domain expertise and a could construct a prompt ‘a photo of {classname}, a type
validation dataset. Meanwhile, recent developments in gen- of bird’. This prompt engineering process, however, is not
erative pretrained models like GPT-4 mean they can be used optimal because it: 1.) requires domain expertise in the
as advanced internet search tools. They can also be ma- target domain; 2.) has high variance – small changes to
nipulated to provide visual information in any structure. the prompt result in large variation in performance; 3.) has
In this work, we show that GPT-4 can be used to gener- a fixed prompt template for all the classes, therefore only
ate text that is visually descriptive and how this can be the class name in the prompt provides the classification an-
used to adapt CLIP to downstream tasks. We show con- chor, which might not contain enough information to dis-
siderable improvements in 0-shot transfer accuracy on spe- tinguish different classes. For example, in Fig 1 we see an
cialized fine-grained datasets like EuroSAT (∼ 7%), DTD image of a Green Heron, which from the name would sug-
(∼ 7%), SUN397 (∼ 4.6%), and CUB (∼ 3.3%) when gest that it is predominantly a green-colored bird and we
compared to CLIP’s default prompt. We also design a sim- would assume that it is similar to Green Woodpecker if we
ple few-shot adapter that learns to choose the best possi- have never seen either bird. However, we can see that it is
ble sentences to construct generalizable classifiers that out- in fact a blackish-brown bird with a chestnut-colored neck
perform the recently proposed CoCoOP by ∼ 2% on aver- and visually more similar to a bird like the Black Bittern.
age and by over 4% on 4 specialized fine-grained datasets. For 0-shot transfer to fine-grained datasets like this to work
The code, prompts, and auxiliary text dataset is available at well, CLIP has to either have seen and associated images
[Link]/mayug/VDT-Adapter. of a Green Heron to the text ‘Green Heron’ from its large
pretraining dataset or additional information in the form of
visually descriptive textual (VDT) information is required.
1. Introduction Here we define VDT as a set of sentences that describe the
visual features of the class under consideration including
Contrastive pre-training of large-scale VLMs has shape, size, color, environment, patterns, composition, etc.
demonstrated remarkable image classification performance While most humans can identify many different common
on open-set classes. Models like CLIP [25] and ALIGN bird species just from their names, they would need access
[13] are pretrained on web-scale datasets consisting of to an ornithology taxonomy of bird descriptions to identify
image-text pairs (over 400 million and 1.8 billion respec- more rare bird species. Similarly, we argue that CLIP’s 0-
tively), resulting in a highly generalizable model with com- shot accuracy can be improved by incorporating VDT in-
petent 0-shot domain adaptation capabilities. While vanilla formation into the prompts. As shown, in Fig 1, including
supervised training is performed on a closed set of concepts VDT information like black crown and black rump moves
or classes, CLIP pretraining uses natural language. This the classification prototype of Green Heron away from the
results in a joint text-vision embedding space that is not classification prototype of Green Woodpecker and towards
1 Corresponding
that of Black Bittern in the text-encoder’s embedding space.
author: mayugmaniparambil@[Link]
Figure 1: An example showing three birds, Green Heron, Green Woodpecker, and Black Bittern. Green Heron and Green
Woodpecker have close-by classification prototypes by virtue of not having enough details in the prompt template. Only the
text-encoder’s embedding space is visualized. Here we see that adding visual descriptions to the prompt resolves this issue
and moves the classification prototypes in the word-encoder’s space such that classification prototypes for visually similar
birds (Green Woodpecker and Black Bittern) lie together.

In this work, we first show that we can use VDT in- formance improvements over vanilla CLIP with the default
formation for each class in the target domain to construct prompt on 12 datasets with an average improvement of 2%
class conditional prompts that achieve performance im- and even better improvements in fine-grained datasets like
provements over CLIP’s default prompt. We show this on EuroSAT (∼ 7%), DTD (∼ 7%), SUN397 (∼ 4.6%), and
the CUB dataset [1] by constructing sentences from domain CUB (∼ 3.3%). The prompts and all the auxiliary class
experts about the bird species in Section 3.2.1 as they are information will be made publicly available to promote re-
readily available as part of the dataset. search in prompt ensembling and multi-modal adapter de-
sign.
However, we acknowledge that domain expert annota- Finally, we design a simple adapter that learns to adap-
tions are costly and time-consuming to obtain, hampering tively select and aggregate the best sentences for any given
the scalability of our method to other datasets. To address dataset and show that making use of this additional VDT
this, we focus on the recent advances in generative pre- information improves the few-shot domain transfer perfor-
trained Large Language Models (LLMs) like GPT-4 to con- mance of CLIP as well. We demonstrate the few-shot adap-
struct these class conditional prompts in a manner easily tation performance for the recently proposed Base-to-New
scalable to other datasets. These models are a good fit for setting on a benchmark of 12 datasets and outperform recent
the task of constructing sophisticated prompts, because: 1) methods like CoOp [35] and CoCoOp [34] despite having
they are a condensed form of human knowledge (trained fewer model parameters, shorter training time, and a sim-
on web-scale text data) [32]; 2) they can be manipulated to pler model architecture.
produce information in any form or structure which makes In short, our contributions are as follows:
compatibility with CLIP’s prompt style relatively simple.
Therefore we use GPT-4 to construct visually descriptive 1. We show that including visually descriptive textual
textual information about the classes with special emphasis (VDT) information in prompts results in better 0-shot
in the GPT-4 prompts about visual cues like shape, color, domain transfer performance of CLIP.
structure, and compositionality. We use the generated VDT
information to construct prompt ensembles that are passed 2. We use GPT-4 to generate VDT sentences in a scalable
through CLIP’s text encoder and aggregated to generate manner and show consistent performance improve-
classifiers that are then used for 0-shot classification. Us- ments over CLIP in 0-shot domain transfer.
ing GPT-4 circumvents the need for domain knowledge and
conveniently provides class conditional prompts. Prompt 3. We design a simple adapter network to make use of
ensembling the VDT sentences reduce CLIP’s performance this extra information for few-shot transfer and show
sensitivity to small changes in the prompt. We show per- performance improvements over methods like CLIP-
 Adapter and CoCoOp [34] for few-shot domain trans- as an active area of research. [20] employs Palm540B LLM
fer in the Base-to-New setting. [5] to generate semantic data for unsupervised class embed-
ding vectors in 0-shot classification, but only tests on three
4. We release all the VDT information for all 12 datasets legacy datasets. Our research presents results on a mod-
to promote further research in multi-modal prompt and ern benchmark of 12 datasets. Recently, [24, 19] leverage
adapter design for low-shot domain transfer of large GPT-3 for class conditional prompts to enhance CLIP’s 0-
VLMs. shot domain transfer on 6 datasets. While [19] focuses on
using GPT-3 to construct visual descriptors that aid in the
2. Related Works interpretability of CLIP’s predictions during 0-shot domain
2.1. Vision Language Models transfer, we argue that 0-shot domain transfer performance
improves with the inclusion of high-quality VDT informa-
Recent VLMs [13, 25, 9] jointly learn the vision and tion. Hence, we make use of GPT-4 for richer, more diverse,
language encoders from scratch and have demonstrated im- and more accurate VDT information.
pressive 0-shot domain transfer performance. As mentioned
While [19] utilize GPT-3, probability space ensemble,
in [35], this can be attributed to transformer networks [28],
and highlight VDT’s role in 0-shot transfer, our method dif-
contrastive losses [4, 11], and web-scale training datasets
fers. We use GPT-4 for auxiliary data collection, perform
[25, 14].
ensemble in word-encoder space, and introduce a few-shot
While our GPT-generated prompt ensembles are simi- adapter for optimal VDT selection in few-shot transfer. [27]
lar to CLIP’s prompt ensembles, CLIP’s prompt ensembles uses GPT-3 for prompt construction in diffusion models to
were constructed and tuned manually, and are class agnos- generate images for support sets while our work only uses
tic, while ours were generated by GPT models that were GPT4 to acquire auxiliary text data. To our knowledge, we
prompted to provide VDT information for each class. are the first to prompt GPT-4 for visually descriptive sen-
2.2. Prompt Learning tences to improve CLIP’s 0-shot and few-shot domain trans-
fer.
CoOp [35] successfully used prompt learning in VLMs
but had generalizability limitations due to overfitting on
the few-shot dataset [34]. In response, CoCoOp was 3. Methodology
proposed, enhancing performance with image-conditioned
prompt learning using a meta-network, albeit at a higher re- 3.1. Review of CLIP and CLIP-Adapter
source cost. We address generalizability differently by us-
ing class conditional VDT information. Our simpler and Through contrastive pretraining on large image-text
more efficient model, CLIP-A-self, outperforms CoCoOp datasets, CLIP performs image classification on various
in the Base-to-New few-shot setting. concepts, aligning related images and texts in a shared em-
bedding space, while separating dissimilar ones. After pre-
2.3. Few-shot adapters for Vision Language models training, CLIP directly performs image classification on the
CLIP-Adapter [10] (CLIP-A) offers a simpler few-shot target dataset without any finetuning. First, we review how
transfer method for VLMs, utilizing an MLP trained on the CLIP model performs 0-shot classification on an open
fixed image/text encoders. Our CLIP-A-self is different set.
from CLIP-A in that we apply a self-attention mechanism The CLIP model, comprising a vision and language
on the set of all sentences for any class, learning to se- model, encodes an image and its corresponding caption into
lect and aggregate the best subset of VDT information for visual and textual embeddings, respectively. During infer-
the dataset from the few-shot training set. Although Tip- ence, these embeddings are compared using cosine similar-
adapter [33] showed superior performance on base classes ity. Given an image I ∈ RH×W ×C , where H, W , C de-
with a cache model, it’s inapplicable in the Base-to-New notes the height, width, and number of channels of the im-
setting due to its reliance on few-shot test class examples, age, the vision encoder transforms the image into the joint
making it irrelevant for our comparison. embedding space to get the image features f ∈ RD where
D represents the dimension of the features.
2.4. Semantic information from Large Language
During inference, a prompt template such as ‘A photo of
Models
{classname}’ is used to generate sentences for K different
Recent advancements in transformer-based language classes and passed through the text-encoder to yield classi-
models, particularly the GPT family [3, 22], have demon- fier weight matrix W ∈ RD×K . Prediction probabilities are
strated exceptional abilities in semantic extraction from in- then calculated by multiplying image feature f with W and
tricate texts. Their application to vision tasks has emerged applying a softmax function:
 Table 1: Comparing visual and non-visual prompt ensem-
bles for 0-shot domain transfer to the CUB dataset.
exp(WiT f )/τ
f = Backbone(I), pi = PK , (1)
T
j=1 exp(Wj f )/τ Non-Visual- Visual- Visual-
Prompting Default
GT GT GPT
In CLIP [25], 0-shot domain transfer utilizes domain-
specific information in the prompt template, such as ‘A Accuracy 54.7 53.0 57.7 57.4
photo of a {class-name}, a type of bird’ for bird images.
[25] reports that careful prompt design and prompt ensem- Table 2: Results of including LLM generated VDT on 6
bling are important to improve 0-shot classification accu- datasets for comparison with other works. We see that
racy. Prompt ensembling is achieved by constructing sev- higher quality VDT from GPT-4 outperforms GPT-3 gen-
eral prompts for each class and then averaging the classifi- erated VDT on specialized datasets like DTD OxfordPets
cation vectors. In our work, we show that prompt ensembles and EuroSAT.
of VDT information improve CLIP’s 0-shot domain trans-
fer. Oxford
Methods EuroSAT Food101 DTD CUB ImageNet Average
Pets
CLIP-A [10] is a learnable MLP adapter applied to im-
CLIP 47.69 85.97 43.09 89.07 54.70 64.51 64.17
age and/or word encoder features for few-shot transfer to
DCLIP[19] 48.82 88.50 45.59 86.92 57.75 68.03 65.93
target datasets. During few-shot transfer, given N images CLIP-GPT 54.86 86.43 50.15 91.54 57.43 68.92 68.21
i=N,j=K
per class with labels, denoted as (xi,k , yi,k )i=1,k=1 , K
classifier weights are constructed using the prompt template default prompt and prompts containing non-visual informa-
H and text encoder g as W = g(H(classname({yi,k }))). tion.
The image features f and text features W pass through the Using the CUB dataset with expert annotations, we
learnable adapters Av , At to get adapted features as follows. contrast the 0-shot performance of visual and non-visual
prompt ensembles. For the visual prompts, we take class at-
tribute vectors detailing attributes like color, pattern, shape,
f ⋆ = αAv (f )T + (1 − α)f, (2) etc. for 28 bird body parts, leading to 312 scores per bird.
⋆ T We use the most pronounced attribute-value pairs to form
W = βAt (W) + (1 − β)W. (3)
28 visual prompts (denoted Visual-GT) such as ‘A photo
The hyperparameters α and β blend CLIP’s knowledge of Green Heron. Green Heron has a greenish-black head
with fine-tuned knowledge to avoid CLIP-Adapter overfit- cap.’ Conversely, for non-visual prompts (denoted Non-
ting. Logits are calculated as per Eqn 1, and cross entropy Visual-GT), we collect information on bird calls, migration,
i=N,j=K
loss over the entire training set (xi,k , yi,k )i=1,k=1 is used behavior, and habitat, yielding 12 different prompts like ‘A
to optimize Av , At . photo of Green Heron. The green heron’s bird call is a loud,
In the All setting, few-shot transfer is tested on a hold-out harsh ‘skeow” per class.
dataset with images from the K classes used in training. In We derive classification vectors for Visual-GT and Non-
the Base-to-New setting, proposed by [34], the evaluation Visual-GT by averaging class-level sentence embeddings
occurs on U non-overlapping classes. Our model is evalu- within CLIP’s joint embedding space, considering its 77-
ated in the more practical Base-to-New setting. token limit. Table 1 shows no improvement using Non-
Visual-GT prompts over the default, yet a 4% improvement
3.2. Language Model Prompt Design with Visual-GT.
In this section, we show that using VDT information in
the prompt template improves CLIP’s 0-shot transfer capa- 3.2.2 Prompting LLMs for visually descriptive infor-
bilities and describe our approach to generate class-specific mation
prompts using an LLM.
In the prior section, we highlighted the use of expert VDT
information in creating class-specific prompts to enhance
3.2.1 Visual Descriptive Sentences
CLIP’s 0-shot performance. However, acquiring expert an-
[25] demonstrates that careful prompt design and prompt notations is both expensive and time-consuming. To over-
ensembling improve the 0-shot classification performance come this, we utilize GPT language models, known for their
of CLIP. Here we ask the question: What type of informa- large-scale knowledge and flexibility [32]. Our approach in-
tion can be appended to the prompt template to improve the volves using GPT-4 to generate visual descriptions for any
0-shot domain transfer performance? We show that append- given dataset thereby aiding in the construction of prompt
ing visually descriptive information to the prompt template ensembles for CLIP in a scalable manner.
and ensembling improves the 0-shot performance over the Our prompting strategy takes inspiration from chain-of-
Figure 2: CLIP-A-self, our simple self-attention based adapter learns to select and aggregate the most relevant subset of
Visually Descriptive Text (VDT) to generate more generalizable classifiers. First, we prompt GPT-4 to generate VDT, N
sentences for K classes that are then passed through the text encoder to get embeddings for each of the N*K sentences.
Self-attention is applied over the N sentences of each class and averaged to get K adapted classifier embeddings.

Table 3: Results of 12 datasets with ViT-B/16.

Oxford FGVC Oxford Stanford

Methods EuroSAT Caltech101 Food101 DTD Sun397 UCF101 CUB ImageNet Average
Flowers Aircraft Pets Cars
CLIP 47.69 93.75 70.69 85.97 24.81 43.09 89.07 65.55 62.61 67.54 54.70 64.51 64.16
CLIP-GPT 54.86 94.51 73.40 86.43 23.42 50.15 91.54 65.01 67.24 65.51 57.43 68.9 66.53

thought prompting [29] and is as follows: First, we ask Base New H

GPT-4 to list all the attributes that may be necessary to dis-
criminate between images of the K classes under consider- CLIP 68.45 73.89 71.05
ation. Second, we ask GPT-4 to provide the values for all CoOp 82.39 62.39 70.99
these attributes for all the K classes as sentences. An ex- CoCoOp 79.35 71.89 75.37
ample for the CUB dataset is shown in the left side of Fig CLIP-A 78.90 72.14 75.07
1. CLIP-A-self 82.12 74.20 77.78

Table 4: Comparing our CLIP-A-self against other methods

on average accuracy over 12 datasets.
The last row in Table 1 shows that the GPT-4 generated
visual sentences’ performance is similar to that of sentences
generated from the class attribute vectors annotated by do- 3.3. Simple few-shot adapters for visual sentences
main experts. We follow the same simple strategy for all
the datasets in the benchmark suite to generate visually de- We design a simple adapter that can use VDT informa-
scriptive sentences in a scalable and flexible manner and use tion to improve the few-shot transfer of CLIP to the target
them to construct prompt ensembles. datasets. Similar to the CLIP-A text, we append a small set
of learnable parameters to the output of the word encoder tings, extending our suite to 12. These include generic ob-
and train the adapter using cross-entropy loss. Our CLIP- ject datasets ImageNet [7] and Caltech101 [8]; fine-grained
A-self uses a self-attention layer that applies attention over classification datasets OxfordPets [23], StanfordCars [16],
the embeddings of the different sentences for each class and Flowers102 [21], Food101 [2] and FGVCAircraft [17];
averages the output to get the final classification vector. SUN397 [31] for scene recognition; UCF101 [26] for ac-
Given we have M GPT generated sentences for tion recognition; DTD [6] for texture classification; Eu-
each of the K classes tm,k , we construct M prompts roSAT [12] for satellite imagery; and CUB for bird iden-
by appending each sentence to the prompt template tification.
like H(classname(yi,k ), {tm,k }) and pass them through For 0-shot transfer with visual sentences, we test on All
CLIP’s word encoder to get W sent ∈ RD×M ×K . classes across these datasets while for the Base-to-New set-
For the self-attention adapter, we apply vanilla self- ting, following [34], we equally sample classes for base and
attention [28] over all the visual descriptive sentences such new sets without overlap. We use the 150-base and 50-
that during training it learns to select and aggregate the most new class split from ZSL and few-shot literature [30, 18]
relevant visual sentences for identifying each class. Just like for CUB. Like [34], our CLIP-A-self is evaluated on the
before, we first obtain the classification vector for all sen- 16-shot setting for easier comparison with other methods.
tences W s ∈ RK×M ×D and pass them as the key, query,
and value to the self-attention module Bself and average out 4.2. Baselines
the output tokens to get the final classification vector W ⋆ . We compare the performance of visual sentences ensem-
Here the attention is applied over the M different visually ble on 0-shot transfer against the CLIP model [25] whose
descriptive sentences. default prompts for each dataset have been extensively fine-
M
tuned using a test set. We also compare against DCLIP [19]
Wavg = 1/M
X
s
Wm,k (4) a recent work that uses GPT-3 to generate VDT information
m=1
for 0-shot transfer. We compare our CLIP-A-self against
 a M  s M  s M  s M two prompt learning methods CoOp [35] which learns static
Wm,k 1
= Bself ( Wm,k 1
, Wm,k 1
, Wm,k 1
) prompts and CoCoOp [34] which learns a dynamic prompt
(5) that is specifically designed to improve Base-to-New trans-
M
X fer. We also compare our CLIP-A-self against CLIP-A [10]
a
Wa−mean = 1/M Wm,k (6) due to the similarity in architecture and to show that the per-
m=1 formance improvements are from making use of the visual
W⋆ = βWa−mean T + (1 − β)Wavg (7) sentences and not from the just adapting the text features.

We finally obtain the new adapter classifier weights 4.3. Training settings
W ∈ RD×K that have been adapted to focus on the most
⋆
Our implementation is based on CoOp’s and CLIP-A’s
visually discriminative information among the M visually code. 1 We make all our comparisons on VIT CLIP back-
descriptive sentences for any given dataset. We make use of bone i.e., VIT-B/16. We take the results for CoOp and Co-
1 to calculate the probabilities and predict the image cate- CoOp for all datasets (except CUB) from their respective
gory by selecting the class with the highest probability. papers, while we make use of practices from the respective
During the few-shot training only the weights of the papers like context length set to 4 and context initializa-
adapter network Bself are trained using cross-entropy loss. tion to “a photo of” to ensure the best results on the CUB
dataset. For CLIP-A, we re-run all experiments on VIT-
4. Experiments B/16 backbone as they were not reported in the paper. For
We assess the significance of visual sentence ensembles all adapter models including ours, we only tune the residual
in two scenarios: (i) we gauge visual sentence quality by ratio β hyper-parameter. For CLIP-A, we use the version
comparing an ensemble of these prompts with CLIP’s de- where the MLP is applied on top of the visual encoder as
fault prompts across 12 benchmark datasets; (ii) we con- it performed the best [10]. We make use of May version of
trast the performance of adapters using these visual prompts GPT-4 for obtaining the auxiliary dataset.
against other few-shot transfer techniques in Base-to-New 4.4. GPT generated visual sentences improve 0-shot
class generalization within a dataset. Prior to discussing the transfer.
results, we detail the datasets and experimental setup.
We compare the performance of CLIP-GPT prompt en-
4.1. Datasets semble with the default prompts of CLIP in Table 3. GPT-
We use 11 diverse image recognition datasets from [35] 1 [Link] https:
and the bird species CUB dataset [1] for both study set- //[Link]/gaopengcuhk/CLIP-Adapter
Table 5: Comparison of GPT-Adapters with CLIP, CoOp and CoCoOp in the Base-to-New generalization setting.
For prompt learning-based methods (CoOp and CoCoOp), their prompts are learned from the base classes (16 shots). The
results strongly justify the importance of including extra visual information. H denotes Harmonic mean (to highlight the
generalization trade-off [30]).

(a) CUB. (b) Caltech101. (c) OxfordPets.

Base New H Base New H Base New H

CLIP 58.7 70.3 63.90 CLIP 96.84 94.00 95.40 CLIP 91.17 97.26 94.12
CoOp 79.2 53.3 63.71 CoOp 98.00 89.81 93.73 CoOp 93.67 95.29 94.47
CoCoOp 67.1 74.1 70.40 CoCoOp 97.96 93.81 95.84 CoCoOp 95.20 97.69 96.43
CLIP-A 68.3 70.8 69.53 CLIP-A 97.7 93.6 95.61 CLIP-A 94.8 97.0 95.89
CLIP-A-self 78.6 71.3 74.77 CLIP-A-self 98.3 95.9 97.09 CLIP-A-self 94.4 97.0 95.68
(d) StanfordCars. (e) Flowers102. (f) Food101.

Base New H Base New H Base New H

CLIP 63.37 74.89 68.65 CLIP 72.08 77.80 74.83 CLIP 90.10 91.22 90.66
CoOp 78.12 60.40 68.13 CoOp 97.60 59.67 74.06 CoOp 88.33 82.26 85.19
CoCoOp 70.49 73.59 72.01 CoCoOp 94.87 71.75 81.71 CoCoOp 90.70 91.29 90.99
CLIP-A 70.5 73.3 71.87 CLIP-A 94.6 71.5 81.44 CLIP-A 90.3 91.2 90.75
CLIP-A-self 76.8 72.9 74.80 CLIP-A-self 97.4 75.3 84.94 CLIP-A-self 90.4 91.2 90.80

(g) FGVCAircraft. (h) SUN397. (i) DTD.

Base New H Base New H Base New H

CLIP 27.19 36.29 31.09 CLIP 69.36 75.35 72.23 CLIP 53.24 59.90 56.37
CoOp 40.44 22.30 28.75 CoOp 80.60 65.89 72.51 CoOp 79.44 41.18 54.24
CoCoOp 33.41 23.71 27.74 CoCoOp 79.74 76.86 78.27 CoCoOp 77.01 56.00 64.85
CLIP-A 34.9 33.5 34.19 CLIP-A 80.1 75.9 77.94 CLIP-A 74.9 53.0 62.08
CLIP-A-self 37.8 33.0 35.24 CLIP-A-self 81.4 76.8 79.03 CLIP-A-self 81.8 62.3 70.73

(j) EuroSAT. (k) UCF101. (l) ImageNet.

Base New H Base New H Base New H

CLIP 56.48 64.05 60.03 CLIP 70.53 77.50 73.85 CLIP 72.43 68.14 70.22
CoOp 92.19 54.74 68.69 CoOp 84.69 56.05 67.46 CoOp 76.47 67.88 71.92
CoCoOp 87.49 60.04 71.21 CoCoOp 82.33 73.45 77.64 CoCoOp 75.98 70.43 73.10
CLIP-A 82.5 62.4 71.06 CLIP-A 82.9 74.9 78.70 CLIP-A 75.4 68.6 71.84
CLIP-A-self 88.5 70.5 78.48 CLIP-A-self 84.1 76.4 80.07 CLIP-A-self 76.4 68.3 72.12

generated prompt ensemble improves upon the performance transfer performance matches that of CLIP, with the excep-
of CLIP 0-shot by 2% on average over 12 datasets. The tion being the action recognition dataset UCF-101. We in-
improvement over CLIP-ZS is significant; over 5% for spe- spected the sentences generated for UCF-101 and notice
cialized fine-grained datasets like CUB, SUN397, EuroSAT, that several of the sentences generated by GPT involves
and DTD and over 2% for oxford-flowers and oxford-pets. temporal information instead of visual descriptions and we
This shows that CLIP does not recognize several of the believe this could be the reason for the drop in accuracy.
classnames in these datasets and describing the class in the However, we notice in Section 4.5.1 that the self-attention
form of visually descriptive sentences results in better clas- module of the few-shot adapter learns to emphasize the vi-
sifiers from the text-encoder and better classification accu- sual sentences out of the generated sentences which might
racy. It is also worth noting that only including the visually explain the improvement in the performance of few-shot
descriptive sentences in the prompts can help improve the adapters in the new setting in Section 4.5. We also com-
performance of general datasets like Imagenet (over 4%) pare against recent work [19] on their subset of 6 datasets
and Caltech-101 (over 1%) too. For all other datasets, the for VIT-B/16 encoder in 2. We see that using the larger
 Prompting ZS Base New H 4.5.1 Attention weights Analysis
Default 54.7 NA NA NA We note that even though CLIP-gpt ensembles were outper-
OpenAssistant 56.0 78.3 69.8 73.80 formed by CLIP default prompt on FGVC Aircraft, UCF-
GPT-3.5 55.7 78.1 70.6 74.16 101, and Stanford Cars dataset, we see that CLIP-A-self
GPT-4 57.4 78.6 71.3 74.77 outperforms CLIP-A and CoCoOp [34] on these datasets
in the few-shot transfer setting. We believe that this is be-
Table 6: Comparing different GPT models for obtaining the cause, during few-shot training, the self-attention mecha-
VDT information. We see that the larger models provide nism learns to select the most relevant visual sentences out
higher quality VDT information but CLIP-A-self is capa- of the set of visually descriptive text and helps produce gen-
ble of producing generalizable classifiers even with smaller eralizable classifiers. In Table 1 in supplementary, we show
models like OpenAssistant. the top 3 and bottom 3 attributes picked by attention scores
for each of these datasets and show that the sentences with
the highest attention scores correspond to visually descrip-
GPT-4 model over the GPT-3 model results in much higher tive attributes in the set and vice versa for the lowest scored
improvements for specialized datasets like DTD (∼ 5%) attributes. For example, for both Stanford Cars and FGVC it
and EuroSAT (∼ 6%). We compare the text used by [19] is interesting to see that the color scheme is one of the least
against our GPT4-generated VDT in the supplementary. used attributes as it’s difficult to identify a car or a plane
from its color or livery. For UCF-101, information like the
4.5. GPT-Adapters improve few-shot transfer per-
force involved or temporal information like speed and range
formance.
of motion of the action is unlikely to be encoded in the im-
We compare the performance of our CLIP-A-self against age and hence is not selected by the attention mechanism.
CLIP, CoOp, and CoCoOp on the benchmark suite of 12 Information regarding the subject and the object of the ac-
datasets in the Base-to-New setting in Table 5. Here we tion, like the posture of the person, description of the object,
see that GPT-Adapters that make use of the VDT informa- and interaction between objects are visible in the images
tion outperform CoCoOp by 3% in the new setting while and hence weighted highly by the attention mechanism.
maintaining similar performance to that of CoOp in the base
setting on the average accuracy over 12 datasets. This is 4.6. Ablation over different GPT models
impressive considering that CoCoOp makes use of a meta- In this section, we see if other GPT models like GPT-3.5
network and forward pass through the text encoder making and open-source model, OpenAssistant [15], are as capa-
it computationally intensive to train. CoCoOp takes up to 5 ble as GPT-4 in generating visually descriptive information.
hours to train on 16-shot ImageNet for VIT-B/16 encoder, We explore this on the CUB dataset as it is fine-grained and
in comparison, our CLIP-A-self takes only 10 mins (on an specialized. The results are presented in Table 6. We find
RTX 3090 GPU). The Base-to-New generalization ability of that the performance improves with larger models which
our adapters is even more impressive for fine-grained, spe- are more capable of memorizing accurate class information
cialized datasets as evidenced by the gains over CoCoOp with less hallucination [32]. Even though we obtain de-
in Harmonic mean of base and new accuracy. For exam- cent performance with the open-source model OpenAssis-
ple, CLIP-A-self demonstrates gains in datasets like FGV- tant, the outputs were always inconsistent and noisy, result-
CAircraft ( 7.5%), EuroSat ( 7.4%), DTD ( 5.8%), CUB ing in a lot of clean-up effort in comparison to GPT-3.5 and
( 4.3%), Flowers102 ( 4%), Stanford Cars ( 2.4%) and UCF- GPT-4 where the outputs were in the form of concise sen-
101 ( 2.4% ). This demonstrates that our adapters make use tences following a dictionary format. It is worth noting that
of semantic information in the form of visually descriptive our few-shot adapter is capable of picking out the the best
sentences and fuse this with CLIP’s 0-shot knowledge to VDT information even from a noisy set, pushing the Base-
build more generalizable classifiers that transfer well to un- to-New generalization performance of OpenAssistant, and
seen classes within the same dataset. It is also worth not- GPT-3.5 close to that of GPT-4.
ing that even though the same set of VDT did not provide
any improvements in 0-shot domain transfer for datasets 5. Conclusion
like FGVC-Aircraft, Stanford-Cars, and UCF-101, our self-
attention adapter was able to choose the most informative In this work, we show that using visually descriptive
subset of VDT and produce few-shot classifiers that provide textual (VDT) information can improve the 0-shot domain
substantial few-shot transfer performance gains in compar- transfer performance of CLIP over non-visual information
ison to CoCoOp. We show in Section 4.5.1 the sentences and the default prompts. We demonstrate GPT-4 to be an ac-
picked by the attention mechanism for these datasets to curate and flexible source of VDT information by improv-
qualitatively verify this. ing the 0-shot domain transfer performances on a suite of
12 benchmark datasets. Our few-shot adapter CLIP-A-self [8] Li Fei-Fei, Rob Fergus, and Pietro Perona. Learning gener-
learns to pick the best VDT information from the GPT gen- ative visual models from few training examples: An incre-
erated set and improve the few-shot domain transfer in the mental bayesian approach tested on 101 object categories. In
Base-to-New setting even when the quality of the generated CVPR-W, 2004. 6
text deteriorates. We release all prompts and VDT infor- [9] Andreas Fürst, Elisabeth Rumetshofer, Johannes Lehner,
Viet T Tran, Fei Tang, Hubert Ramsauer, David Kreil,
mation for all 12 datasets to promote further research in the
Michael Kopp, Günter Klambauer, Angela Bitto, et al.
fertile research direction of using LLMs for learning multi-
Cloob: Modern hopfield networks with infoloob outperform
modal adapters for foundation models. clip. Advances in neural information processing systems,
35:20450–20468, 2022. 3
References [10] Peng Gao, Shijie Geng, Renrui Zhang, Teli Ma, Rongyao
[1] Caltech-UCSD Birds 200. 2, 6 Fang, Yongfeng Zhang, Hongsheng Li, and Yu Qiao.
[2] Lukas Bossard, Matthieu Guillaumin, and Luc Van Gool. Clip-adapter: Better vision-language models with feature
Food-101–mining discriminative components with random adapters. arXiv preprint arXiv:2110.04544, 2021. 3, 4, 6
forests. In ECCV, 2014. 6 [11] Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross
[3] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Sub- Girshick. Momentum contrast for unsupervised visual rep-
biah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakan- resentation learning. In CVPR, 2020. 3
tan, Pranav Shyam, Girish Sastry, Amanda Askell, Sand- [12] Patrick Helber, Benjamin Bischke, Andreas Dengel, and
hini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Damian Borth. Eurosat: A novel dataset and deep learning
Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, benchmark for land use and land cover classification. IEEE
Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Journal of Selected Topics in Applied Earth Observations
Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack and Remote Sensing, 2019. 6
Clark, Christopher Berner, Sam McCandlish, Alec Radford, [13] Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh,
Ilya Sutskever, and Dario Amodei. Language models are Hieu Pham, Quoc V Le, Yunhsuan Sung, Zhen Li, and Tom
few-shot learners. In H. Larochelle, M. Ranzato, R. Hadsell, Duerig. Scaling Up Visual and Vision-Language Representa-
M.F. Balcan, and H. Lin, editors, Advances in Neural Infor- tion Learning With Noisy Text Supervision. page 139, 2021.
mation Processing Systems, volume 33, pages 1877–1901. 1, 3
Curran Associates, Inc., 2020. 3 [14] Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh,
[4] Ting Chen, Simon Kornblith, Mohammad Norouzi, and Ge- Hieu Pham, Quoc V Le, Yunhsuan Sung, Zhen Li, and Tom
offrey Hinton. A simple framework for contrastive learning Duerig. Scaling up visual and vision-language representation
of visual representations. In ICML, 2020. 3 learning with noisy text supervision. In ICML, 2021. 3
[5] Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, [15] Andreas Köpf, Yannic Kilcher, Dimitri von Rütte,
Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Sotiris Anagnostidis, Zhi-Rui Tam, Keith Stevens, Ab-
Barham, Hyung Won Chung, Charles Sutton, Sebas- dullah Barhoum, Nguyen Minh Duc, Oliver Stanley,
tian Gehrmann, Parker Schuh, Kensen Shi, Sasha Richárd Nagyfi, et al. Openassistant conversations–
Tsvyashchenko, Joshua Maynez, Abhishek Rao, Parker democratizing large language model alignment. arXiv
Barnes, Yi Tay, Noam Shazeer, Vinodkumar Prabhakaran, preprint arXiv:2304.07327, 2023. 8, 3
Emily Reif, Nan Du, Ben Hutchinson, Reiner Pope, James [16] Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei.
Bradbury, Jacob Austin, Michael Isard, Guy Gur-Ari, 3d object representations for fine-grained categorization. In
Pengcheng Yin, Toju Duke, Anselm Levskaya, Sanjay ICCV-W, 2013. 6
Ghemawat, Sunipa Dev, Henryk Michalewski, Xavier [17] Subhransu Maji, Esa Rahtu, Juho Kannala, Matthew
Garcia, Vedant Misra, Kevin Robinson, Liam Fedus, Denny Blaschko, and Andrea Vedaldi. Fine-grained visual classi-
Zhou, Daphne Ippolito, David Luan, Hyeontaek Lim, fication of aircraft. arXiv preprint arXiv:1306.5151, 2013.
Barret Zoph, Alexander Spiridonov, Ryan Sepassi, David 6
Dohan, Shivani Agrawal, Mark Omernick, Andrew M. [18] Mayug Maniparambil, Kevin Mcguinness, and Noel
Dai, Thanumalayan Sankaranarayana Pillai, Marie Pellat, O’connor. BaseTransformers: Attention over base data-
Aitor Lewkowycz, Erica Moreira, Rewon Child, Oleksandr points for One Shot Learning. In British Machine Vision
Polozov, Katherine Lee, Zongwei Zhou, Xuezhi Wang, Conference 2022, BMVC 2022, 2022. 6
Brennan Saeta, Mark Diaz, Orhan Firat, Michele Catasta, [19] Sachit Menon and Carl Vondrick. Visual classification via
Jason Wei, Kathy Meier-Hellstern, Douglas Eck, Jeff Dean, description from large language models. ICLR, 2023. 3, 4,
Slav Petrov, and Noah Fiedel. Palm: Scaling language 6, 7, 8
modeling with pathways, 2022. 3 [20] Muhammad Ferjad Naeem, Muhammad Gul Zain Ali Khan,
[6] Mircea Cimpoi, Subhransu Maji, Iasonas Kokkinos, Sammy Yongqin Xian, Muhammad Zeshan Afzal, Didier Stricker,
Mohamed, and Andrea Vedaldi. Describing textures in the Luc Van Gool, and Federico Tombari. I2mvformer: Large
wild. In CVPR, 2014. 6 language model generated multi-view document supervi-
[7] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, sion for zero-shot image classification. In Proceedings of
and Li Fei-Fei. Imagenet: A large-scale hierarchical image the IEEE/CVF Conference on Computer Vision and Pattern
database. In CVPR, 2009. 6 Recognition, pages 15169–15179, 2023. 3
[21] Maria-Elena Nilsback and Andrew Zisserman. Automated [35] Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei
flower classification over a large number of classes. In Liu. Learning to prompt for vision-language models. In-
ICVGIP, 2008. 6 ternational Journal of Computer Vision, 130(9):2337–2348,
[22] OpenAI. Gpt-4 technical report, 2023. 3 2022. 2, 3, 6
[23] Omkar M Parkhi, Andrea Vedaldi, Andrew Zisserman, and
CV Jawahar. Cats and dogs. In CVPR, 2012. 6
[24] Sarah Pratt, Rosanne Liu, and Ali Farhadi. What does a
platypus look like? Generating customized prompts for zero-
shot image classification. 2023. 3
[25] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya
Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry,
Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning
transferable visual models from natural language supervi-
sion. In International conference on machine learning, pages
8748–8763. PMLR, 2021. 1, 3, 4, 6
[26] Khurram Soomro, Amir Roshan Zamir, and Mubarak Shah.
Ucf101: A dataset of 101 human actions classes from videos
in the wild. arXiv preprint arXiv:1212.0402, 2012. 6
[27] Vishaal Udandarao, Ankush Gupta DeepMind, and Samuel
Albanie. SuS-X: Training-Free Name-Only Transfer of
Vision-Language Models. 3
[28] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszko-
reit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia
Polosukhin. Attention is all you need. Advances in neural
information processing systems, 30, 2017. 3, 6
[29] Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten
Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al.
Chain-of-thought prompting elicits reasoning in large lan-
guage models. Advances in Neural Information Processing
Systems, 35:24824–24837, 2022. 5
[30] Yongqin Xian, Christoph H Lampert, Bernt Schiele, and
Zeynep Akata. Zero-shot learning—a comprehensive eval-
uation of the good, the bad and the ugly. IEEE transactions
on pattern analysis and machine intelligence, 41(9):2251–
2265, 2018. 6, 7
[31] Jianxiong Xiao, James Hays, Krista A Ehinger, Aude Oliva,
and Antonio Torralba. Sun database: Large-scale scene
recognition from abbey to zoo. In CVPR, 2010. 6
[32] Jifan Yu, Xiaozhi Wang, Shangqing Tu, Shulin Cao, Daniel
Zhang-Li, Xin Lv, Hao Peng, Zijun Yao, Xiaohan Zhang,
Hanming Li, Chunyang Li, Zheyuan Zhang, Yushi Bai, Yan-
tao Liu, Amy Xin, Nianyi Lin, Kaifeng Yun, Linlu Gong,
Jianhui Chen, Zhili Wu, Yunjia Qi, Weikai Li, Yong Guan,
Kaisheng Zeng, Ji Qi, Hailong Jin, Jinxin Liu, Yu Gu,
Yuan Yao, Ning Ding, Lei Hou, Zhiyuan Liu, Bin Xu, Jie
Tang, and Juanzi Li. KoLA: Carefully Benchmarking World
Knowledge of Large Language Models. 2, 4, 8
[33] Renrui Zhang, Wei Zhang, Rongyao Fang, Peng Gao, Kun-
chang Li, Jifeng Dai, Yu Qiao, and Hongsheng Li. Tip-
adapter: Training-free adaption of clip for few-shot classifi-
cation. In European Conference on Computer Vision, pages
493–510. Springer, 2022. 3
[34] Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei
Liu. Conditional prompt learning for vision-language mod-
els. In Proceedings of the IEEE/CVF Conference on Com-
puter Vision and Pattern Recognition, pages 16816–16825,
2022. 2, 3, 4, 6, 8
 Supplementary Materials: Enhancing CLIP classes in dataset e.g. ’Boeing 707-320’, ’Boeing
with GPT-4: Harnessing Visual Descriptions 727-200’,..., ’Yakovlev Yak-42’]
as Prompts
GPT-4 response:
1. Attention weights visualized Manufacturer: The company that produces the
We visualize the attention weights learned by the CLIP- aircraft (e.g., Boeing, Airbus). Model Number:
A-self for datasets Stanford Cars, UCF101, FGVC Air- Differentiates different aircraft within the same
craft, Oxford Flowers and CUB in Table 1. We notice that manufacturer. Aircraft Function: The primary
the self-attention mechanism in CLIP-A-self assigns more role or use of the aircraft, such as commercial
weight to visually descriptive sentences that are most rel- (passenger), cargo (freight transport), or military
evant for discriminating between the classes of the dataset (fighter, bomber, etc). Engine Count: The number
under consideration. For instance, we see that for discrim- of engines on the aircraft. Engine Position: The
inating images of birds species (CUB dataset) and flower location of the engines, either under the wings or
species (Oxford Flowers) sentences describing the color of at the rear of the fuselage. Wing Position: The
the head and wings of birds and petals of the flowers are position of the wings: high-wing, mid-wing, or
important but for identifying different car or aircraft models low-wing design. Tail Type: The shape and type
sentences describing the color or livery is one of the least of the tail, such as traditional, T-tail, V-tail, etc.
important. We also see that if the information being de- Body Length: The length of the aircraft’s fuse-
scribed by the VDT sentence is not clearly visible in the lage. Body Shape: The overall shape of the fuse-
image, the attention weight assigned to it by CLIP-A-self lage, like narrow, wide, double-decker, etc. Wing
is low. For instance, in CUB dataset, the the undersides Shape: The shape of the wings, either swept or
of birds or the sepals in Oxford Flowers dataset are often straight. Nose Shape: The shape of the nose or
not visible in the images, hence the VDT sentence corre- front of the aircraft. Landing Gear Type: The
ponding to this is is in the bottom 3 attributes picked by the type of landing gear, like tricycle or taildragger.
learnt attention weights. It’s also worth noting that, some Cockpit Window Shape: The shape and number
of the VDT sentences do not have much variation between of cockpit windows. Color Scheme: Typical color
different classes and hence are not useful in dsicrimination schemes or liveries associated with the aircraft
between the classes of the dataset. For instance, in Oxford- type. Number of Decks: Either single-deck or
flowers, the color of the leaves, the color of the stem are double-deck (like the Airbus A380). Existence of
often green for most flowers in the dataset, which maybe Winglets: Whether or not the aircraft has winglets
why low attention score was learnt for this attribute. at the end of the wings. Existence of Canards:
Whether or not the aircraft has canards, small
2. Prompts for GPT-4 wing-like surfaces located at the front. Propeller
or Jet: Indicates if the aircraft is propeller-driven
Throughout our experiments, we use a two-step prompt-
or jet-powered. Number of Seats: An estimate of
ing strategy in which we first ask the LLM to generate a
the seating capacity of the aircraft. Range: The
list of attributes which will aid in visually distinguishing
maximum distance the aircraft can travel with-
between the different classes in a particular dataset. The
out refueling. Unique Visual Identifier: Specific
second prompt asks the LLM to create a description using
structures or features that are unique to a model
the attributes provided by the first prompt and specifies the
or series of aircraft. Aircraft Type: Differentiates
expected output format. We request a python dictionary
between commercial, cargo, or military aircraft.
as output with a list of sentences, each corresponding to
one attribute. The output structure is simple to use (down-
stream), preserves attribute-level detail and encourages at-
tribute richness. Example of the second prompt for the FGVC Aircraft
Example of first prompt for the FGVC Aircraft dataset: dataset which uses GPT-4’s response for the first prompt as
I am creating class attributes for a zero-shot im- part of the prompt:
age recognition algorithm to classify different im-
ages of a diverse set of aircrafts. The attributes Describe the following aircraft by adding one
are part of side information about the classes. sentence about each attribute for the following
List 20 attributes that can form part of a descrip- aircraft: Airbus A340-200. Return the answer
tion of the class that will aid in distinguishing be- as a python dictionary with the aircraft variant
tween the following list of classes visually: [list of as the key (i.e. remove the manufacturer) and
 Dataset Top 3 attributes selected Bottom 3 attributes selected
FGVC Unique visual identifier, Color scheme,
presence of canards, model number,
tail type. commercial or cargo.
Cars Body shape, Interior description,
fender description, brand logo description,
spoiler description, color scheme
UCF-101 Equipment used, Body muscles used,
Posture of person, force involved,
Interaction info. speed of motion
Oxford-Flowers Shape of the flower, Stem color,
Color, shape and number of petals, Color of leaves,
Texture and description of veins in leaves Description of sepals
CUB Wings color and shape, Color and description of legs,
Head color and shape, Underparts color,
Beak color and shape Tail shape and color
Table 1: The top 3 and bottom 3 attributes selected by the attention mechanism in GPT-A-self for 3 different datasets. For
UCF101, We see that attention learns to pick visually descriptive sentences like posture and description of objects over
temporal information like speed of motion and force applied.

the value is a list of sentences. Rewrite the at- Scheme: Typical color schemes or liveries asso-
tribute as a full sentence. Do not include the at- ciated with the aircraft type. Number of Decks:
tributes as keys. Attributes: Manufacturer: The Either single-deck or double-deck (like the Airbus
company that produces the aircraft (e.g., Boeing, A380). Existence of Winglets: Whether or not the
Airbus). Model Number: Differentiates differ- aircraft has winglets at the end of the wings. Exis-
ent aircraft within the same manufacturer. Air- tence of Canards: Whether or not the aircraft has
craft Function: The primary role or use of the canards, small wing-like surfaces located at the
aircraft, such as commercial (passenger), cargo front. Propeller or Jet: Indicates if the aircraft
(freight transport), or military (fighter, bomber, is propeller-driven or jet-powered. Number of
etc). Engine Count: The number of engines on Seats: An estimate of the seating capacity of the
the aircraft. Engine Position: The location of the aircraft. Range: The maximum distance the air-
engines, either under the wings or at the rear of craft can travel without refueling. Unique Visual
the fuselage. Wing Position: The position of the Identifier: Specific structures or features that are
wings: high-wing, mid-wing, or low-wing design. unique to a model or series of aircraft. Aircraft
Tail Type: The shape and type of the tail, such as Type: Differentiates between commercial, cargo,
traditional, T-tail, V-tail, etc. Body Length: The or military aircraft.
length of the aircraft’s fuselage. Body Shape: The
overall shape of the fuselage, like narrow, wide, The response of the second prompt constitutes the VDT
double-decker, etc. Wing Shape: The shape of the information we utilise as side-information for Airbus A340-
wings, either swept or straight. Nose Shape: The 200 as an example:
shape of the nose or front of the aircraft. Landing
Gear Type: The type of landing gear, like tricy- ”A340-200”: [ ”The Airbus A340-200 is pro-
cle or taildragger. Cockpit Window Shape: The duced by Airbus, a renowned aircraft manufac-
shape and number of cockpit windows. Color turer.”, ”It differentiates itself from other air-
craft within the Airbus family through its unique
 model number: A340-200.”, ”This aircraft pri- ual cleaning and reformatting to get into Python dictionary
marily serves a commercial role, typically used format. GPT-3.5 performed slightly worse than GPT-4 in
for passenger transport.”, ”The Airbus A340-200 terms of adherence to the prompt, as it did not consistently
is equipped with four engines.”, ”These engines return only a dictionary. In such cases, we simply called the
are situated under the aircraft’s wings.”, ”The API again. After repeated incorrect format responses, we
aircraft features a low-wing design, with wings manually cleaned those cases.
positioned at the bottom of the fuselage.”, ”It We primarily utilized GPT-4 via the ChatGPT Plus sub-
has a traditional tail configuration, common to scription plan at a cost of $20 since the GPT-4 API was
many large commercial aircraft.”, ”The A340- not generally available during most of our experimentation
200 has a lengthy fuselage, extending to about phase. The GPT-4 API cost to create the VDT information
59.4 meters.”, ”The body of the Airbus A340- for the SUN397 dataset was $14.90, as opposed to $1.94
200 is wide-bodied, facilitating a larger passen- using the GPT-3.5 API.
ger capacity.”, ”Its wings are swept back, a de-
sign aspect that improves fuel efficiency and per- 3. Comparing our VDT with GPT3
formance at high speeds.”, ”The aircraft features
a rounded nose shape, contributing to its aero- In Table 2, we compare the VDT generated by GPT-4
dynamic design.”, ”The Airbus A340-200 uses using our prompting technique with that of [19] who used
a tricycle type landing gear, supporting stability GPT-3 to obtain visual descriptors for different classes of
during takeoffs and landings.”, ”Its cockpit win- the dataset. Here we notice that, including a prompt step
dows are angular and include six panels, giving asking the GPT-4 for visual attributes necessary for classi-
pilots a comprehensive view of their surround- fying between images of the classes result in a fixed number
ings.”, ”Color schemes vary by airline, but the of sentences per class, a fixed order guaranteeing that every
Airbus corporate livery features a predominantly class is accompanied by as much visual information as pos-
white body with blue and teal accents.”, ”This sible. By using GPT-4 we also get much richer and more ac-
model is a single-deck aircraft, focusing on width curate visual descriptions. For example, for the class indus-
rather than height for passenger capacity.”, ”The trial, our descriptions provide inforrmation about density of
A340-200 does not have winglets, differing from buildings, shadow in the image, road accessibility and lay-
some newer Airbus models.”, ”There are no ca- out while the description used by [19] is only ‘evidence of
human activity’. A similar phenomenon can be observed
nards present on the Airbus A340-200, instead, it
employs a more traditional aircraft design.”, ”As for DTD dataset. This explains the jump in performance for
a jet-powered aircraft, the A340-200 uses high- specialized datasets like DTD and Eurosat over DCLIP.
speed jet engines for propulsion.”, ”The A340-
200 typically accommodates around 260 passen- 4. Generalizability at lower shots
gers, though the exact number can vary with In Figure 1, we compare the harmonic mean of Base and
the configuration.”, ”With a range of approxi- New accuracies of CLIP-A-self with that of CLIP-A over
mately 7,800 nautical miles, the Airbus A340-200 number of shots = 1, 5, 10, 16. Our CLIP-A-self demon-
can cover considerable distances without refuel- strates performance improvements at lower shots, outper-
ing.”, ”The aircraft’s four-engine configuration forming CLIP-A on average by over 1.5%/ for the 1-shot
and lengthy, wide-bodied design are unique vi- case and over 2.5%/ for the 5-shot case. Our adapter shows
sual identifiers of the A340-200 model.”, ”Clas- higher improvements over CLIP-A in the higher shot sce-
sified as a commercial aircraft, the Airbus A340- nario because of the number of parameters and the inherent
200 is primarily used for passenger transporta- difficulty in identifying the VDT sentences that are discrim-
tion.” ] inative for the current classes in the low shot scenario. For
instance, identifying the class from a single image is often
GPT-4 generally adheres to the python dictionary out-
difficult because of co-occuring objects, environment, back-
put requirement in the User prompt, but tends to return ad-
ground etc which can be resolved if we have more exmaple
ditional explanations, motivations or clarifications. To en-
images from the same class. The largest improvements
courage the LLM to only return a Python dictionary as re-
are for specialized and fine-grained datasets like Stanford-
quested, we add the following System prompt:
Cars, EuroSat Oxford Flowers, DTD and CUB. Oxford-pets
You are ChatGPT, a large language model and Food-101 results do not improve much because these
trained by OpenAI. Return only the python dic- datasets are relatively easy and already show good perfor-
tionary, with no explanation. mance with default CLIP.
Conversely, OpenAssistant’s [15] output requires man-
 Table 2: Comparing our VDT with that of descriptors from [19] for 2 random classes of datasets DTD and Eurosat

Ours DCLIP[19]
’The surface feels moderately smooth, with slight roughness due to the layered structure.’ ’a series of layers’
’There is no distinct pattern, but the layers create a natural, linear visual effect.’ ’each layer is of a different material’
’The structure is characterized by multiple layers stacked upon each other.’ ’the layers are parallel to each other’
’The texture has a two-dimensional feel, with the layers adding a sense of depth.’ ’the layers may be of different thicknesses’
Stratified (DTD)
’The density varies, with some layers appearing closely packed while others are more sparse.’ ’the layers may be of different colors’
’The regularity of the texture is defined by the consistent layering.’ ’the layers may have different textures’
’The texture is opaque, with no transparency between the layers.’
’There are no significant surface defects, but minor irregularities may occur between layers.’
’The texture feels moderately smooth to the touch, not too rough nor too sleek.’ ’a series of parallel lines’
’It exhibits a lined pattern, reminiscent of ruled notebook paper.’ ’can be straight or curved’
’The structure of the texture is stratified, with lines arranged one after the other.’ ’may be of different colors’
’The texture has a two-dimensional quality, with no noticeable depth or relief.’ ’may be of different widths’
Lined (DTD)
’The lines are densely packed, leaving little space between them.’ ’may be of different thicknesses’
’The texture displays a high degree of regularity, with the lines evenly spaced and parallel.’
’The texture is opaque, with no transparency or translucency.’
’There are no noticeable surface defects, the lines are clean and uninterrupted.’
’Industrial buildings have texture that is smooth, regular.’ ’evidence of human activity’
’Industrial buildings have shape that is rectangular, irregular.’
’Industrial buildings have size (relative) that is large.’
’Industrial buildings have pattern that is regular, dense.’
’Industrial buildings have spectral reflectance that is high in visible spectrum.’
Industrial (Eurosat) ’Industrial buildings have a shadow that is present (due to high-rise buildings).’
’Industrial buildings have adjacent land features that is commercial, residential, roads.’
’Industrial buildings have change over time that is stable.’
’Industrial buildings have density that is high.’
’Industrial buildings have proximity to water bodies that is variable.’
’Industrial buildings have road accessibility that is high.’
’Forest has texture that is rough.’ ’a large area of trees’
’Forest has shape that is irregular.’ ’green leaves’
’Forest has size (relative) that is large.’
’Forest has pattern that is no pattern.’
’Forest has spectral reflectance that is high in near-infrared.’
Forest (Eurosat) ’Forest has shadow that is present (due to trees).’
’Forest has adjacent land features that is land, mountains, rivers.’
’Forest has change over time that is mostly stable.’
’Forest has density that is high.’
’Forest has proximity to water bodies that is variable.’
’Forest has road accessibility that is low.’
Figure 1: Main results of Base-to-New few shot learning on 12 datasets. CLIP-A-self consistently shows better performance
over CLIP-A over different training shots, demonstrating the importance of Visually descriptive text in improving the gener-
alizability of few-shot classifiers for CLIP.