InstaGen: Enhancing Object Detection by Training on Synthetic Dataset

Chengjian Feng1 Yujie Zhong1 Zequn Jie1,† Weidi Xie2,† Lin Ma1
1
Meituan Inc. 2 CMIC, Shanghai Jiao Tong University
fcjian@[Link] jaszhong@[Link] [Link]@[Link]
weidi@[Link] [Link]@[Link]
[Link]
arXiv:2402.05937v3 [[Link]] 8 Apr 2024

Figure 1. (a) The synthetic images generated from Stable Diffusion and our proposed InstaGen, which can serve as a dataset synthesizer
for sourcing photo-realistic images and instance bounding boxes at scale. (b) On open-vocabulary detection, training on synthetic images
demonstrates significant improvement over CLIP-based methods on novel categories. (c) Training on the synthetic images generated from
InstaGen also enhances the detection performance in close-set scenario, particularly in data-sparse circumstances.

Abstract gorization of objects within images [3, 5, 12, 26, 27]. The
common practise is to train the detectors on large-scale im-
In this paper, we present a novel paradigm to enhance age datasets, such as MS-COCO [20] and Object365 [30],
the ability of object detector, e.g., expanding categories where objects are exhaustively annotated with bounding
or improving detection performance, by training on syn- boxes and corresponding category labels. However, the pro-
thetic dataset generated from diffusion models. Specifically, cedure for collecting images and annotations is often labo-
we integrate an instance-level grounding head into a pre- rious and time-consuming, limiting the datasets’ scalability.
trained, generative diffusion model, to augment it with the In the recent literature, text-to-image diffusion models
ability of localising instances in the generated images. The have demonstrated remarkable success in generating high-
grounding head is trained to align the text embedding of quality images [28, 29], that unlocks the possibility of train-
category names with the regional visual feature of the dif- ing vision systems with synthetic images. In general, ex-
fusion model, using supervision from an off-the-shelf object isting text-to-image diffusion models are capable of syn-
detector, and a novel self-training scheme on (novel) cat- thesizing images based on some free-form text prompt, as
egories not covered by the detector. We conduct thorough shown in the first row of Figure 1a. Despite being photo-
experiments to show that, this enhanced version of diffusion realistic, such synthesized images can not support training
model, termed as InstaGen, can serve as a data synthe- sophisticated systems, that normally requires the inclusion
sizer, to enhance object detectors by training on its gen- of instance-level annotations, e.g., bounding boxes for ob-
erated samples, demonstrating superior performance over ject detection in our case. In this paper, we investigate a
existing state-of-the-art methods in open-vocabulary (+4.5 novel paradigm of dataset synthesis for training object de-
AP) and data-sparse (+1.2 ∼ 5.2 AP) scenarios. tector, i.e., augmenting the text-to-image diffusion model to
generate instance-level bounding boxes along with images.
1. Introduction To begin with, we build an image synthesizer by fine-
tuning the diffusion model on existing detection dataset.
Object detection has been extensively studied in the field
This is driven by the observation that off-the-shelf diffu-
of computer vision, focusing on the localization and cate-
sion models often generate images with only one or two ob-
†: corresponding author. jects on simplistic background, training detectors on such

1
images may thus lead to reduced robustness in complex training data with bounding box annotations and can only
real-world scenarios. Specifically, we exploit the existing recognize a predetermined set of categories present in the
detection dataset, and subsequently fine-tune the diffusion training data. In the recent literature, to further expand
model with the image-caption pairs, constructed by taking the ability of object detector, open-vocabulary object detec-
random image crops, and composing the category name of tion (OVD) has been widely researched, for example, OVR-
the objects in the crop. As illustrated in the second row of CNN [37] introduces the concept of OVD and pre-trains a
the Figure 1a, once finetuned, the image synthesizer now vision-language model with image-caption pairs. The sub-
enables to produce images with multiple objects and intri- sequent works make use of the robust multi-modal repre-
cate contexts, thereby providing a more accurate simulation sentation of CLIP [24], and transfer its knowledge to object
of real-world detection scenarios. detectors through knowledge distillation [9, 36], exploiting
To generate bounding boxes for objects within synthetic extra data [5, 41] and text prompt tuning [2, 5]. In this paper,
images, we propose an instance grounding module that es- we propose to expand the ability of object detectors, e.g.,
tablishes the correlation between the regional visual fea- expanding categories or improving detection performance,
tures from diffusion model and the text embedding of cat- by training on synthetic dataset.
egory names, and infers the coordinates for the objects’ Generative Models. Image generation has been consid-
bounding boxes. Specifically, we adopt a two-step training ered as a task of interest in computer vision for decades. In
strategies, firstly, we train the grounding module on syn- the recent literature, significant progress has been made, for
thetic images, with the supervision from an off-the-shelf example, the generative adversarial networks (GANs) [8],
object detector, which has been trained on a set of base cat- variational autoencoders (VAEs) [15], flow-based mod-
egories; secondly, we utilize the trained grounding head to els [14], and autoregressive models (ARMs) [32]. More
generate pseudo labels for a larger set of categories, includ- recently, there has been a growing research interest in diffu-
ing those not seen in existing detection dataset, and self- sion probabilistic models (DPMs), which have shown great
train the grounding module. Once finished training, the promise in generating high-quality images across diverse
grounding module will be able to identify the objects of ar- datasets. For examples, GLIDE [23] utilizes a pre-trained
bitrary category and their bounding boxes in the synthetic language model and a cascaded diffusion structure for text-
image, by simply providing the name in free-form language. to-image generation. DALL-E 2 [25] is trained to gener-
To summarize, we explore a novel approach to enhance ate images by inverting the CLIP image space, while Ima-
object detection capabilities, such as expanding detectable gen [29] explores the advantages of using pre-trained lan-
categories and improving overall detection performance, guage models. Stable Diffusion [28] proposes the diffusion
by training on synthetic dataset generated from diffusion process in VAE latent spaces rather than pixel spaces, effec-
model. We make the following contribution: (i) We develop tively reducing resource consumption. In general, the rapid
an image synthesizer by fine-tuning the diffusion model, development of generative models opens the possibility for
with image-caption pairs derived from existing object de- training large models with synthetic dataset.
tection datasets, our synthesizer can generate images with
multiple objects and complex contexts, offering a more re- 3. Methodology
alistic simulation for real-world detection scenarios. (ii) We
introduce a data synthesis framework for detection, termed In this section, we present details for constructing a dataset
as InstaGen. This is achieved through a novel ground- synthesizer, that enables to generate photo-realistic images
ing module that enables to generate labels and bounding with bounding boxes for each object instance, and train an
boxes for objects in synthetic images. (iii) We train standard object detector on the combined real and synthetic datasets.
object detectors on the combination of real and synthetic 3.1. Problem Formulation
dataset, and demonstrate superior performance over exist-
ing state-of-the-art detectors across various benchmarks, in- Given a detection dataset of real images with manual an-
cluding open-vocabulary detection (increasing Average Pre- notations, i.e., Dreal = {(x1 , B1 , Y1 ), . . . , (xN , BN , YN )},
cision [AP] by +4.5), data-sparse detection (enhancing AP where Bi = {b1 , . . . , bm |bj ∈ R2×2 } denotes the set of
by +1.2 to +5.2), and cross-dataset transfer (boosting AP by box coordinates for the annotated instances in one image,
+0.5 to +1.1). and Yi = {y1 , . . . , ym |yj ∈ RCbase } refers to the categories
of the instances. Our goal is thus to exploit the given real
2. Related Work dataset (Dreal ), to steer a generative diffusion model into
dataset synthesizer, that enables to augment the existing de-
Object Detection. Object detection aims to simultane- tection dataset, i.e., Dfinal = Dreal + Dsyn . As a result, detec-
ously predict the category and corresponding bounding box tors trained on the combined dataset demonstrate enhanced
for the objects in the images. Generally, object detec- ability, i.e., extending the detection categories or improving
tors [3, 4, 6, 26, 27] are trained on a substantial amount of the detection performance.

2
 (a) Fine-tuning diffusion model on detection dataset. (b) Supervised training and self-training for grounding head (i.e. student).
Figure 2. Illustration of the process for finetuning diffusion model and training the grounding head: (a) stable diffusion model is fine-tuned
on the detection dataset on base categories. (b) The grounding head is trained on synthetic images, with supervised learning on base
categories and self-training on novel categories.

In the following sections, we first describe the procedure We adopt a mild strategy by taking random crops from the
for constructing an image synthesizer, that can generate im- images, and construct the text prompt with categories in the
ages suitable for training object detector (Section 3.2). To image crops, as shown in Figure 2a. If an image crop con-
simultaneously generate the images and object bounding tains multiple objects of the same category, we only use this
boxes, we propose a novel instance-level grounding mod- category name once in the text prompt.
ule, which aligns the text embedding of category name with Fine-tuning loss. We use the sampled image crop and con-
the regional visual features from image synthesizer, and in- structed text prompt to fine-tune SDM with a squared error
fers the coordinates for the objects in synthetic images. To loss on the predicted noise term as follows:
further improve the alignment towards objects of arbitrary \mathcal {L}_{\text {fine-tune}} = \mathbb {E}_{z, \epsilon \sim \mathcal {N}(0,1),t,y} \Big [||\epsilon - \epsilon _{\theta }(z^{t},t,y)||_{2}^{2} \Big ], (1)
category, we adopt self-training to tune the grounding mod-
ule on object categories not existing in Dreal (Section 3.3). where z denotes a latent vector mapped from the input im-
As a result, the proposed model, termed as InstaGen, can age with VAE, t denotes the denoising step, uniformly sam-
automatically generate images along with bounding boxes pled from {1, . . . , T }, T refers to the length of the diffusion
for object instances, and construct synthetic dataset (Dsyn ) Markov chain, and ϵθ refers to the estimated noise from
at scale, leading to improved ability when training detectors SDM with parameters θ being updated. We have experi-
on it (Section 3.4). mentally verified the necessity of this fine-tuning step, as
shown in Table 4.
3.2. Image Synthesizer for Object Detection
3.3. Dataset Synthesizer for Object Detection
Here, we build our image synthesizer based on an off-the-
shelf stable diffusion model (SDM [28]). Despite of its im- In this section, we present details for steering the image
pressive ability in generating photo-realistic images, it of- synthesizer into dataset synthesizer for object detection,
ten outputs images with only one or two objects on sim- which enables to simultaneously generate images and ob-
plistic background with the text prompts, for example, ‘a ject bounding boxes. Specifically, we propose an instance-
photograph of a [category1 name] and a [category2 name]’, level grounding module that aligns the text embedding of
as demonstrated in Figure 4b. As a result, object detec- object category, with the regional visual feature of the
tors trained on such images may exhibit reduced robustness diffusion model, and infers the coordinates for bounding
when dealing with complex real-world scenarios. To bridge boxes, effectively augmenting the image synthesizer with
such domain gap, we propose to construct the image syn- instance grounding, as shown in Figure 3. To further im-
thesizer by fine-tuning the SDM with an existing real-world prove the alignment in large visual diversity, we propose a
detection dataset (Dreal ). self-training scheme that enables the grounding module to
generalise towards arbitrary categories, including those not
Fine-tuning procedure. To fine-tune the stable diffusion exist in real detection dataset (Dreal ). As a result, our data
model (SDM), one approach is to naı̈vely use the sample synthesizer, termed as InstaGen, can be used to construct
from detection dataset, for example, randomly pick an im- synthetic dataset for training object detectors.
age and construct the text prompt with all categories in the
image. However, as the image often contains multiple ob- 3.3.1 Instance Grounding on Base Categories
jects, such approach renders significant difficulty for fine- To localise the object instances in synthetic images, we in-
tuning the SDM, especially for small or occluded objects. troduce an open-vocabulary grounding module, that aims

3
Figure 3. Illustration of the dataset generation process in InstaGen. The data generation process consists of two steps: (i) Image collection:
given a text prompt, SDM generates images with the objects described in the text prompt; (ii) Annotation generation: the instance-level
grounding head aligns the category embedding with the visual feature region of SDM, generating the corresponding object bounding-boxes.

to simultaneously generate image (x) and the correspond- an image cross-attention layer for combining image fea-
ing instance-level bounding boxes (B) based on a set of tures, and a text cross-attention layer for combining text
categories (Y), i.e., {x, B, Y} = ΦInstaGen (ϵ, Y), where features. Finally, we apply the dot product between each
ϵ ∼ N (0, I) denotes the sampled noise. query and the text features, followed by a Sigmoid function
to predict the classification score ŝ for each category. Addi-
To this end, we propose an instance grounding head, as
tionally, the object queries are passed through a Multi-Layer
shown in Figure 3, it takes the intermediate representation
Perceptron (MLP) to predict the object bounding boxes b̂, as
from image synthesizer and the text embedding of category
shown in Figure 3. We train the grounding head by align-
as inputs, then predicts the corresponding object bounding
ing the category embedding with the regional visual fea-
boxes, i.e., {Bi , Yi } = Φg-head (Fi , Φt-enc (g(Yi ))), where
tures from diffusion model, as detailed below. Once trained,
Fi = {fi1 , . . . , fin } refers to the multi-scale dense features
the grounding head is open-vocabulary, i.e., given any cat-
from the image synthesizer at time step t = 1, g(·) denotes
egories (even beyond the training categories), the ground-
a template that decorates each of the visual categories in the
ing head can generate the corresponding bounding-boxes
text prompt, e.g., ‘a photograph of [category1 name] and
for the object instances.
[category2 name]’, Φt-enc (·) denotes the text encoder.
Inspired by GroundingDINO [22], our grounding head Training triplets of base categories. Following [18], we
Φg-head (·) mainly contains four components: (i) a channel- apply an automatic pipeline to construct the {visual feature,
compression layer, implemented with a 3×3 convolution, bounding-box, text prompt} triplets, with an object detec-
for reducing the dimensionality of the visual features; (ii) a tor trained on base categories from a given dataset (Dreal ).
feature enhancer, consisting of six feature enhancer layers, In specific, assuming there exists a set of base categories
to fuse the visual and text features. Each layer employs a de- {c1base , . . . , cN
base }, e.g., the classes in MS-COCO [20]. We
formable self-attention to enhance image features, a vanilla first select a random number of base categories to construct
self-attention for text feature enhancers, an image-to-text a text prompt, e.g., ‘a photograph of [base category1] and
cross-attention and a text-to-image cross-attention for fea- [base category2]’, and generate both the visual features and
ture fusion; (iii) a language-guided query selection module images with our image synthesizer. Then we take an off-
for query initialization. This module predicts top-N anchor the-shelf object detector, for example, pre-trained Mask R-
boxes based on the similarity between text features and im- CNN [12], to run the inference procedure on the synthetic
age features. Following DINO [38], it adopts a mixed query images, and infer the bounding boxes of the selected cate-
selection where the positional queries are initialized with gories. To acquire the confident bounding-boxes for train-
the anchor boxes and the content queries remain learnable; ing, we use a score threshold α to filter out the bounding-
(iv) a cross-modality decoder for classification and box re- boxes with low confidence (an ablation study on the se-
finement. It comprises six decoder layers, with each layer lection of the score threshold has been conducted in Sec-
utilizing a self-attention mechanism for query interaction, tion 4.5). As a result, an infinite number of training triplets

4
(a) Stable Diffusion + Grounding head w/ Super- (b) Stable Diffusion + Grounding head w/ (c) Stable Diffusion w/ Fine-tuning + Grounding
vised training. Supervised- and Self-training. head w/ Supervised- and Self-training.
Figure 4. Visualization of the synthetic images and bounding-boxes generated from different models. The bounding-boxes with green
denote the objects from base categories, while the ones with red denote the objects from novel categories.

for the given base categories can be constructed by repeat- module) with teacher model on the visual features to pro-
ing the above operation. duce bounding boxes, and then use a score threshold β to
Training loss. We use the constructed training triplets to filter out those with low confidence, and use the remain-
train the grounding head: ing training triplets (Fi , b̂i , yinovel ) to train the student, i.e.,
grounding head.
\label {eq:base-grounding} \mathcal {L}_{\text {base}} = \sum \limits _{i=1}^N[\mathcal {L}_{\text {cls}}(\hat {s}_{i}, c_{i}) + \mathds {1}_{\{c_i\neq \varnothing \}} \mathcal {L}_{\text {box}}(\hat {b}_i, b_{i})], \vspace {-3pt} (2) Training loss. Now, we can also train the grounding head
on the mined triplets of novel categories (that are unseen in
the existing real dataset) with the training loss Lnovel defined
where the ith prediction (ŝi , b̂i ) from the N object queries
similar to Eq. 2. Thus, the total training loss for training the
is assigned to a ground-truth (ci , bi ) or ∅ (no object) with
grounding head can be: Lgrounding = Lbase + Lnovel .
bipartite matching. Lcls and Lbox denote the classification
loss (e.g. Focal loss) and box regression loss (e.g. L1 loss
and GIoU loss), respectively. 3.4. Training Detector with Synthetic Dataset
In this section, we augment the real dataset (Dreal ), with syn-
3.3.2 Instance Grounding on Novel Categories thetic dataset (Dsyn ), and train popular object detectors, for
example, Faster R-CNN [27] with the standard training loss:
Till here, we have obtained a diffusion model with open-
vocabulary grounding, which has been only trained with \mathcal {L}_{\text {det}} = \mathcal {L}_{\text {rpn\_cls}} + \mathcal {L}_{\text {rpn\_box}} + \mathcal {L}_{\text {det\_cls}} + \mathcal {L}_{\text {det\_box}}, (3)
base categories. In this section, we propose to further lever-
age the synthetic training triplets from a wider range of where Lrpn cls , Lrpn box are the classification and box regres-
categories to enhance the alignment for novel/unseen cat- sion losses of region proposal network, and Ldet cls , Ldet box
egories. Specifically, as shown in Figure 2b, we describe a are the classification and box regression losses of the detec-
framework that generates the training triplets for novel cat- tion head. Generally speaking, the synthetic dataset enables
egories using the grounded diffusion model, and then self- to improve the detector’s ability from two aspects: (i) ex-
train the grounding head. panding the original data with more categories, (ii) improve
the detection performance by increasing data diversity.
Training triplets of novel categories. We design the text
prompts of novel categories, e.g., ‘a photograph of [novel Expanding detection categories. The grounding head is
category1] and [novel category2]’, and pass them through designed to be open-vocabulary, that enables to generate ob-
our proposed image synthesizer, to generate the visual fea- ject bounding boxes for novel categories, even though it is
tures. To acquire the corresponding bounding-boxes for trained with a specific set of base categories. This feature
novel categories, we propose a self-training scheme that enables InstaGen to construct a detection dataset for any
takes the above grounding head as the student, and apply a category. Figure 4 demonstrates several synthetic images
mean teacher (an exponential moving average (EMA) of the and object bounding boxes for novel categories, i.e., the ob-
student model) to create pseudo labels for update. In con- ject with red bounding box. We evaluate the effectiveness of
trast to the widely adopted self-training scheme that takes training on synthetic dataset through experiments on open-
the image as input, the student and teacher in our case only vocabulary detection benchmark. For more details, please
take the visual features as input, thus cannot apply data aug- refer to Figure 1b and Section 4.2.
mentation as for images. Instead, we insert dropout module Increasing data diversity. The base diffusion model is
within each feature enhancer layer and decoder layer in the trained on a large corpus of image-caption pairs, that en-
student. During training, we run inference (without dropout ables to generate diverse images. Taking advantage of such

5
 Method Supervision Detector Backbone AP50box
all AP50box
base AP50box
novel

Detic [41] CLIP Faster R-CNN R50 45.0 47.1 27.8

PromptDet [5] CLIP Faster R-CNN R50 - 50.6 26.6
BARON [34] CLIP Faster R-CNN R50 53.5 60.4 34.0
OADP [33] CLIP Faster R-CNN R50 47.2 53.3 30.0
ViLD [9] CLIP Mask R-CNN R50 51.3 59.5 27.6
F-VLM [16] CLIP Mask R-CNN R50 39.6 - 28.0
RO-ViT [13] CLIP Mask R-CNN ViT-B [1] 41.5 - 30.2
VLDet [19] CLIP CenterNet2 [40] R50 45.8 50.6 32.0
CxORA [35] CLIP DAB-DETR [21] R50 35.4 35.5 35.1
DK-DETR [17] CLIP Deformable DETR [42] R50 - 61.1 32.3
EdaDet [31] CLIP Deformable DETR [42] R50 52.5 57.7 37.8
InstaGen Stable Diffusion Faster R-CNN R50 52.3 55.8 42.3

Table 1. Results on open-vocabulary COCO benchmark. AP50box

novel is the main metric for evaluation. Our detector, trained on synthetic
dataset from InstaGen, significantly outperforms state-of-the-art CLIP-based approaches on novel categories.

capabilities, InstaGen is capable of generating dataset with the text encoder of CLIP is kept frozen, while the remaining
diverse images and box annotations, which can expand the components are trained for 6 epochs with a batch size of 16
original dataset, i.e., increase the data diversity and improve and a learning rate of 1e-4.
detection performance, particularly in data-sparse scenar-
ios. We conducted experiments with varying proportions Instance grounding module. We start by constructing the
of COCO [20] images as available real data, and show the training triplets using base categories i.e., the categories
effectiveness of training on synthetic dataset when the num- present in the existing dataset. The text prompt for each
ber of real-world images is limited. We refer the readers for triplet is constructed by randomly selecting one or two cat-
more details in Section 4.3, and results in Figure 1c. egories. The regional visual features are taken from the im-
age synthesizer time step t = 1, and the oracle ground-truth
4. Experiment bounding boxes are obtained using a Mask R-CNN model
trained on base categories, as explained in Section 3.3.1.
In this section, we use the proposed InstaGen to construct
synthetic dataset for training object detectors, i.e., generat- Subsequently, we train the instance grounding module
ing images with the corresponding bounding boxes. Specif- with these training triplets for 6 epochs, with a batch size
ically, we present the implementation details in Section 4.1. of 32. In the 6th epoch, we transfer the weights from the
To evaluate the effectiveness of the synthetic dataset for student model to the teacher model, and proceed to train the
training object detector, we consider three protocols: open- student for an additional 6 epochs. During this training, the
vocabulary object detection (Section 4.2), data-sparse ob- student receives supervised training on the base categories
ject detection (Section 4.3) and cross-dataset object detec- and engages in self-training on novel categories, and the
tion (Section 4.4). Lastly, we conduct ablation studies on teacher model is updated using exponential moving average
the effectiveness of the proposed components and the selec- (EMA) with a momentum of 0.999. The initial learning rate
tion of hyper-parameters (Section 4.5). is set to 1e-4 and is subsequently reduced by a factor of 10
at the 11th epoch, and the score thresholds α and β are set
4.1. Implementation details to 0.8 and 0.4, respectively.
Network architecture. We build image synthesizer from Training object detector on combined dataset. In our ex-
the pre-trained Stable Diffusion v1.4 [28], and use the CLIP periment, we train an object detector (Faster R-CNN [27])
text encoder [24] to get text embedding for the category with ResNet-50 [11] as backbone, on a combination of the
name. The channel compression layer maps the dimension existing real dataset and the synthetic dataset. Specifically,
of visual features to 256, which is implemented with a 3×3 for synthetic dataset, we randomly select one or two cat-
convolution. For simplicity, the feature enhancer, language- egories at each iteration, construct the text prompts, and
guided query selection module and cross-modality decoder feed them as input to generates images along with the cor-
are designed to the same structure as the ones in [22]. The responding bounding boxes with β of 0.4. Following the
number of the object queries is set to 900. standard implementation [27], the detector is trained for 12
Constructing image synthesizer. In our experiments, we epochs (1× learning schedule) unless specified. The initial
first fine-tune the stable diffusion model on a real detection learning rate is set to 0.01 and then reduced by a factor of
dataset, e.g., the images of base categories. During training, 10 at the 8th and the 11th epochs.

6
InstaGen 10% 25% 50% 75% 100% Method Supervision Detector Extra Data Object365 LVIS
✗ 23.3 29.5 34.1 36.1 37.5 Gao et al. [7] CLIP CenterNet2 ✓ 6.9 8.0
✓ 28.5 32.6 35.8 37.3 38.5 VL-PLM [39] CLIP Mask R-CNN ✓ 10.9 22.2
InstaGen Stable Diffusion Faster R-CNN ✗ 11.4 23.3
Table 2. Results on data-sparse object de-
tection. We employ Faster R-CNN with the Table 3. Results on generalizing COCO-base to Object365 and LVIS. All detectors utilize
ResNet-50 backbone as the default object the ResNet-50 backbone. The evaluation protocol follows [7] and reports AP50. Extra
detector and evaluate its performance using data refers to an additional dataset that encompasses objects from the categories within the
the AP metric on MS COCO benchmark. target dataset. In both experiments, the extra data consists of all the images from COCO,
Please refer to the text for more details. which has covered the majority of categories in Object365 and LVIS.

G-head ST FT AP50box
all AP50box
base AP50box
novel
synthesizer, and train a Mask R-CNN for generating oracle
ground-truth bounding boxes in synthetic images. We em-
✓ 50.6 55.3 37.1
✓ ✓ 51.1 55.0 40.3 ploy 1250 synthetic images per category to train a Faster R-
✓ ✓ ✓ 52.3 55.8 42.3 CNN in conjunction with the corresponding COCO subset.
The performance is measured by Average Precision [20].
Table 4. The effectiveness of the proposed components. G-head,
ST and FT refer to the grounding head, self-training the grounding
Comparison to baseline. As shown in Table 2, the Faster
head and fine-tuning SDM, respectively. R-CNN trained with synthetic images achieves consistent
improvement across various real training data budgets. No-
4.2. Open-vocabulary object detection tably, as the availability of real data becomes sparse, syn-
Experimental setup. Following the previous works [5, 39], thetic dataset plays even more important role for perfor-
we conduct experiments on the open-vocabulary COCO mance improvement, for instance, it improves the detector
benchmark, where 48 classes are treated as base categories, by +5.2 AP (23.3→28.5 AP) when only 10% real COCO
and 17 classes as the novel categories. More results for training subset is available.
LVIS can be found in the supplementary material. To
4.4. Cross-dataset object detection
train the grounding head, we employ 1250 synthetic images
per category per training epoch. While for training the ob- Experimental setup. In this section, we assess the ef-
ject detector, we use 3000 synthetic images per category, fectiveness of synthetic data on a more challenging task,
along with the original real dataset for base categories. The namely cross-dataset object detection. Following [39], we
object detector is trained with input size of 800 × 800 and evaluate the COCO-trained model on two unseen datasets:
scale jitter. The performance is measured by COCO Aver- Object365 [30] and LVIS [10]. Specifically, we consider
age Precision at an Intersection over Union of 0.5 (AP50). the 48 classes in the open-vocabulary COCO benchmark
Comparison to SOTA. As shown in Table 1, we eval- as the source dataset, while Object365 (with 365 classes)
uate the performance by comparing with existing CLIP- and LVIS (with 1203 classes) serve as the target dataset.
based open-vocabulary object detectors. It is clear that our When training the instance grounding module, we acquire
detector trained on synthetic dataset from InstaGen out- 1250 synthetic images for base categories from the source
performs existing state-of-the-art approaches significantly, dataset, and 100 synthetic images for the category from the
i.e., around +5AP improvement over the second best. In target dataset at each training iteration. In the case of train-
essence, through the utilization of our proposed open- ing the object detector, we employ 500 synthetic images per
vocabulary grounding head, InstaGen is able to generate category from the target dataset for each training iteration.
detection data for novel categories, enabling the detector to The detector is trained with input size of 1024 × 1024 and
attain exceptional performance. To the best of our knowl- scale jitter [39].
edge, this is the first work that applies generative diffusion Comparison to SOTA. The results presented in Table 3
model for dataset synthesis, to tackle open-vocabulary ob- demonstrate that the proposed InstaGen achieves supe-
ject detection, and showcase its superiority in this task. rior performance in generalization from COCO-base to Ob-
ject365 and LVIS, when compared to CLIP-based methods
4.3. Data-sparse object detection
such as [7, 39]. It is worth noting that CLIP-based methods
Experimental setup. Here, we evaluate the effectiveness require the generation of pseudo-labels for the categories
of synthetic dataset in data-spare scenario, by varying the from the target dataset on COCO images, and subsequently
amount of real data. We randomly select subsets compris- train the detector using these images. These methods neces-
ing 10%, 25%, 50%, 75% and 100% of the COCO training sitate a dataset that includes objects belonging to the cate-
set, this covers all COCO categories. These subsets are used gories of the target dataset. In contrast, InstaGen possesses
to fine-tune stable diffusion model for constructing image the ability to generate images featuring objects of any cat-

7
 #Images AP50box
all AP50box box
base AP50novel α AP50box
all AP50box box
base AP50novel β AP50box
all AP50box box
base AP50novel

1000 51.6 55.9 39.7 0.7 51.3 55.1 40.6 0.3 46.4 53.3 26.9
2000 51.7 55.4 41.1 0.8 52.3 55.8 42.3 0.4 52.3 55.8 42.3
3000 52.3 55.8 42.3 0.9 51.8 55.6 41.1 0.5 51.2 55.4 39.2
Table 5. Number of generated images. Table 6. α for bounding-box filtration. Table 7. β for bounding-box filtration.

egory without the need for additional datasets, thereby en- categories, showing the scalability of the proposed training
hancing its versatility across various scenarios. mechanism.
Score thresholds for bounding box filtration. We com-
4.5. Ablation study
pare the performance with different score thresholds α and
To understand the effectiveness of the proposed compo- β for filtering bounding boxes on base categories and novel
nents, we perform thorough ablation studies on the open- categories, respectively. From the experiment results in Ta-
vocabulary COCO benchmark [20], investigating the ef- ble 6, we observe that the performance is not sensitive to the
fect of fine-tuning stable diffusion model, training instance value of α, and α = 0.8 yields the best performance. The
grounding module, self-training on novel categories. Addi- experimental results using different β are presented in Ta-
tionally, we investigate other hyper-parameters by compar- ble 7. With a low score threshold (α = 0.3), there are still
ing the effectiveness of synthetic images and different score numerous inaccurate bounding boxes remaining, resulting
thresholds for base and novel categories. in an AP of 26.9 for novel categories. by increasing β to
Fine-tuning diffusion model. We assess the effective- 0.4, numerous inaccurate bounding boxes are filtered out,
ness of fine-tuning stable diffusion model, and its impact resulting in optimal performance. Hence, we set α = 0.8
for synthesizing images for training object detector. Fig- and β = 0.4 in our experiments.
ure 4c illustrates that InstaGen is capable of generating
images with more intricate contexts, featuring multiple ob- 5. Limitation
jects, small objects, and occluded objects. Subsequently, we Using synthetic or artificially generated data in training AI
employed these generated images to train Faster R-CNN for algorithms is a burgeoning practice with significant poten-
object detection. The results are presented in Table 4, show- tial. It can address data scarcity, privacy, and bias issues.
ing that image synthesizer from fine-tuning stable diffusion However, there remains two limitations for training object
model delivers improvement detection performance by 2.0 detectors with synthetic data, (i) synthetic datasets com-
AP (from 40.3 to 42.3 AP). monly focus on clean, isolated object instances, which lim-
Instance grounding module. To demonstrate the effective- its the exposure of the detector to the complexities and con-
ness of the grounding head in open-vocabulary scenario, we textual diversity of real-world scenes, such as occlusions,
exclusively train it on base categories. Visualization exam- clutter, varied environmental factors, deformation, there-
ples of the generated images are presented in Figure 4a. fore, models trained on synthetic data struggle to adapt to
These examples demonstrate that the trained grounding real-world conditions, affecting their overall robustness and
head is also capable of predicting bounding boxes for in- accuracy, (ii) existing diffusion-based generative model also
stances from novel categories. Leveraging these generated suffers from long-tail issue, that means the generative model
images to train the object detector leads to a 37.1 AP on struggles to generate images for objects of rare categories,
novel categories, surpassing or rivaling all existing state-of- resulting in imbalanced class representation during training
the-art methods, as shown in Table 1 and Table 4. and reduced detector performance for less common objects.
Self-training scheme. We evaluate the performance after
self-training the grounding head with novel categories. As 6. Conclusion
shown in Table 4, training Faster R-CNN with the generated This paper proposes a dataset synthesis pipeline, termed
images of novel categories, leads to a noticeable enhance- as InstaGen, that enables to generate images with ob-
ment in detection performance, increasing from 37.1 to 40.3 ject bounding boxes for arbitrary categories, acting as a
AP. Qualitatively, it also demonstrates enhanced recall for annotation-free approach for constructing large-scale syn-
novel objects after self-training, as shown in Figure 4b. thetic dataset to train object detector. We have conducted
Number of synthetic images. We investigate the perfor- thorough experiments to show the effectiveness of train-
mance variation while increasing the number of the gener- ing on synthetic data, on improving detection performance,
ated images per category for detector training. As shown in or expanding the number of detection categories. Signif-
Table 5, when increasing the number of generated images icant improvements have been shown in various detection
from 1000 to 3000, the detector’s performance tends to be scenarios, including open-vocabulary (+4.5 AP) and data-
increasing monotonically, from 39.7 to 42.3 AP on novel sparse (+1.2 ∼ 5.2 AP) detection.

8
References [17] Liangqi Li, Jiaxu Miao, Dahu Shi, Wenming Tan, Ye Ren, Yi
Yang, and Shiliang Pu. Distilling detr with visual-linguistic
[1] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, knowledge for open-vocabulary object detection. In ICCV,
Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, pages 6501–6510, 2023. 6
Mostafa Dehghani, Matthias Minderer, Georg Heigold, Syl-
[18] Ziyi Li, Qinye Zhou, Xiaoyun Zhang, Ya Zhang, Yanfeng
vain Gelly, et al. An image is worth 16x16 words: Trans-
Wang, and Weidi Xie. Open-vocabulary object segmentation
formers for image recognition at scale. arXiv preprint
with diffusion models. In CVPR, pages 7667–7676, 2023. 4
arXiv:2010.11929, 2020. 6
[19] Chuang Lin, Peize Sun, Yi Jiang, Ping Luo, Lizhen Qu, Gho-
[2] Yu Du, Fangyun Wei, Zihe Zhang, Miaojing Shi, Yue Gao,
lamreza Haffari, Zehuan Yuan, and Jianfei Cai. Learning
and Guoqi Li. Learning to prompt for open-vocabulary ob-
object-language alignments for open-vocabulary object de-
ject detection with vision-language model. In CVPR, pages
tection. 2022. 6
14084–14093, 2022. 2
[20] Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays,
[3] Chengjian Feng, Yujie Zhong, Yu Gao, Matthew R Scott,
Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence
and Weilin Huang. Tood: Task-aligned one-stage object de-
Zitnick. Microsoft coco: Common objects in context. In
tection. In ICCV, pages 3490–3499. IEEE Computer Society,
ECCV, pages 740–755. Springer, 2014. 1, 4, 6, 7, 8
2021. 1, 2
[21] Shilong Liu, Feng Li, Hao Zhang, Xiao Yang, Xianbiao Qi,
[4] Chengjian Feng, Yujie Zhong, and Weilin Huang. Exploring
Hang Su, Jun Zhu, and Lei Zhang. Dab-detr: Dynamic
classification equilibrium in long-tailed object detection. In
anchor boxes are better queries for detr. arXiv preprint
ICCV, pages 3417–3426, 2021. 2
arXiv:2201.12329, 2022. 6
[5] Chengjian Feng, Yujie Zhong, Zequn Jie, Xiangxiang Chu,
[22] Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao
Haibing Ren, Xiaolin Wei, Weidi Xie, and Lin Ma. Prompt-
Zhang, Jie Yang, Chunyuan Li, Jianwei Yang, Hang Su, Jun
det: Towards open-vocabulary detection using uncurated im-
Zhu, et al. Grounding dino: Marrying dino with grounded
ages. In ECCV, pages 701–717. Springer, 2022. 1, 2, 6, 7
pre-training for open-set object detection. arXiv preprint
[6] Chengjian Feng, Zequn Jie, Yujie Zhong, Xiangxiang Chu, arXiv:2303.05499, 2023. 4, 6
and Lin Ma. Aedet: Azimuth-invariant multi-view 3d object
[23] Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav
detection. In CVPR, pages 21580–21588, 2023. 2
Shyam, Pamela Mishkin, Bob McGrew, Ilya Sutskever, and
[7] Mingfei Gao, Chen Xing, Juan Carlos Niebles, Junnan Li, Mark Chen. Glide: Towards photorealistic image generation
Ran Xu, Wenhao Liu, and Caiming Xiong. Open vocabulary and editing with text-guided diffusion models. arXiv preprint
object detection with pseudo bounding-box labels. In ECCV, arXiv:2112.10741, 2021. 2
pages 266–282. Springer, 2022. 7
[24] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya
[8] Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry,
Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learn-
Yoshua Bengio. Generative adversarial networks. Commu- ing transferable visual models from natural language super-
nications of the ACM, 63(11):139–144, 2020. 2 vision. pages 8748–8763. PMLR, 2021. 2, 6
[9] Xiuye Gu, Tsung-Yi Lin, Weicheng Kuo, and Yin Cui. [25] Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu,
Open-vocabulary object detection via vision and language and Mark Chen. Hierarchical text-conditional image gener-
knowledge distillation. arXiv preprint arXiv:2104.13921, ation with clip latents. arXiv preprint arXiv:2204.06125, 1
2021. 2, 6, 1 (2):3, 2022. 2
[10] Agrim Gupta, Piotr Dollar, and Ross Girshick. Lvis: A [26] Joseph Redmon, Santosh Divvala, Ross Girshick, and Ali
dataset for large vocabulary instance segmentation. In CVPR, Farhadi. You only look once: Unified, real-time object de-
pages 5356–5364, 2019. 7, 1 tection. In CVPR, pages 779–788, 2016. 1, 2
[11] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. [27] Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun.
Deep residual learning for image recognition. In CVPR, Faster r-cnn: Towards real-time object detection with region
pages 770–778, 2016. 6 proposal networks. Advances in neural information process-
[12] Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Gir- ing systems, 28, 2015. 1, 2, 5, 6
shick. Mask r-cnn. In ICCV, pages 2961–2969, 2017. 1, [28] Robin Rombach, Andreas Blattmann, Dominik Lorenz,
4 Patrick Esser, and Björn Ommer. High-resolution image syn-
[13] Dahun Kim, Anelia Angelova, and Weicheng Kuo. Region- thesis with latent diffusion models. In CVPR, pages 10684–
aware pretraining for open-vocabulary object detection with 10695, 2022. 1, 2, 3, 6
vision transformers. In CVPR, pages 11144–11154, 2023. 6 [29] Chitwan Saharia, William Chan, Saurabh Saxena, Lala
[14] Durk P Kingma and Prafulla Dhariwal. Glow: Generative Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour,
flow with invertible 1x1 convolutions. NeurIPS, 31, 2018. 2 Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans,
[15] Diederik P Kingma and Max Welling. Auto-encoding varia- et al. Photorealistic text-to-image diffusion models with deep
tional bayes. arXiv preprint arXiv:1312.6114, 2013. 2 language understanding. NeurIPS, 35:36479–36494, 2022.
[16] Weicheng Kuo, Yin Cui, Xiuye Gu, AJ Piergiovanni, and 1, 2
Anelia Angelova. F-vlm: Open-vocabulary object detection [30] Shuai Shao, Zeming Li, Tianyuan Zhang, Chao Peng, Gang
upon frozen vision and language models. 2022. 6, 2 Yu, Xiangyu Zhang, Jing Li, and Jian Sun. Objects365:

9
 A large-scale, high-quality dataset for object detection. In
ICCV, pages 8430–8439, 2019. 1, 7
[31] Cheng Shi and Sibei Yang. Edadet: Open-vocabulary ob-
ject detection using early dense alignment. In ICCV, pages
15724–15734, 2023. 6
[32] Aaron Van den Oord, Nal Kalchbrenner, Lasse Espeholt,
Oriol Vinyals, Alex Graves, et al. Conditional image gen-
eration with pixelcnn decoders. NeurIPS, 29, 2016. 2
[33] Luting Wang, Yi Liu, Penghui Du, Zihan Ding, Yue Liao,
Qiaosong Qi, Biaolong Chen, and Si Liu. Object-aware dis-
tillation pyramid for open-vocabulary object detection. In
CVPR, pages 11186–11196, 2023. 6
[34] Size Wu, Wenwei Zhang, Sheng Jin, Wentao Liu, and
Chen Change Loy. Aligning bag of regions for open-
vocabulary object detection. In CVPR, pages 15254–15264,
2023. 6, 2
[35] Xiaoshi Wu, Feng Zhu, Rui Zhao, and Hongsheng Li. Cora:
Adapting clip for open-vocabulary detection with region
prompting and anchor pre-matching. In CVPR, pages 7031–
7040, 2023. 6
[36] Johnathan Xie and Shuai Zheng. Zsd-yolo: Zero-shot
yolo detection using vision-language knowledge distillation.
arXiv preprint arXiv:2109.12066, 2(3):4, 2021. 2
[37] Alireza Zareian, Kevin Dela Rosa, Derek Hao Hu, and Shih-
Fu Chang. Open-vocabulary object detection using captions.
In CVPR, pages 14393–14402, 2021. 2
[38] Hao Zhang, Feng Li, Shilong Liu, Lei Zhang, Hang Su, Jun
Zhu, Lionel M Ni, and Heung-Yeung Shum. Dino: Detr
with improved denoising anchor boxes for end-to-end object
detection. arXiv preprint arXiv:2203.03605, 2022. 4
[39] Shiyu Zhao, Zhixing Zhang, Samuel Schulter, Long Zhao,
BG Vijay Kumar, Anastasis Stathopoulos, Manmohan Chan-
draker, and Dimitris N Metaxas. Exploiting unlabeled data
with vision and language models for object detection. In
ECCV, pages 159–175. Springer, 2022. 7
[40] Xingyi Zhou, Vladlen Koltun, and Philipp Krähenbühl.
Probabilistic two-stage detection. arXiv preprint
arXiv:2103.07461, 2021. 6
[41] Xingyi Zhou, Rohit Girdhar, Armand Joulin, Philipp
Krähenbühl, and Ishan Misra. Detecting twenty-thousand
classes using image-level supervision. In ECCV, pages 350–
368. Springer, 2022. 2, 6, 1
[42] Xizhou Zhu, Weijie Su, Lewei Lu, Bin Li, Xiaogang
Wang, and Jifeng Dai. Deformable detr: Deformable trans-
formers for end-to-end object detection. arXiv preprint
arXiv:2010.04159, 2020. 6

10
 InstaGen: Enhancing Object Detection by Training on Synthetic Dataset
Supplementary Material
In this supplementary document, we present the experi- Lbase Lnovel Detector AP Precision Recall
mental results of the LVIS-OVD benchmark in Section S1.
✓ 70.2 87.9 68.3
Additionally, we perform an ablation study to evaluate the
✓ ✓ 79.7 89.1 90.0
coupling between the diffusion model and the grounding
head in Section S2. Furthermore, we evaluate the quality
Table S1. The quality of the pseudo-labels.
of the pseudo-labels generated by the grounding head in
Section S3. Lastly, we provide more qualitative results in
Section S4. diffusion model and the instance-level grounding head,
namely, the grounding head predicts the bounding-boxes
S1. Open-vocabulary setting on LVIS based on the SDM’s internal representation. To demonstrate
the effectiveness of the tight coupling design, we compare
Experimental setup. We conduct experiments on the
it with a loose coupling design. For the latter, we train
LVIS-OVD benchmark. The latest LVIS v1.0 [10] con-
an open-vocabulary detector (i.e. ResNet-101 + instance
sists of 1203 categories, each with bounding box and in-
level grounding head) on the synthetic images with base
stance mask annotations. The categories are divided into
categories, and generate pseudo-labels for novel categories.
three groups based on the number of images in which each
When training detectors on such synthetic dataset, it gives
category appears in the training set: rare (1-10 images),
31.9 AP on novel categories on the COCO-OVD bench-
common (11-100 images), and frequent (more than 100 im-
mark, 10.4 AP lower than tight coupling, showing the ben-
ages). In line with the problem setting in ViLD [9] and
efits of rich semantic and positional information encoded in
Detic [41], we treat the frequent and common classes as
SDM’s visual features.
base categories, while considering the rare classes as novel
categories. For evaluation on LVIS v1.0 minival set, we
mainly consider the mask Average Precision for novel cat-
S3. Quality of Pseudo-labels
egories, i.e. APnovel . However, to complete the AP metric, Here we evaluate the quality of the pseudo-labels generated
we also report APc (for common classes), APf (for frequent by the proposed grounding head. We adopt two metrics to
classes) and AP (for all classes). assess their quality: (i) Detector AP and (ii) Precision and
Similar to PromptDet [5], we enhance the prompt tem- Recall. For Detector AP, we leverage the pre-trained Mask-
plate by incorporating a more detailed description to mit- RCNN model on the COCO dataset to generate ground
igate lexical ambiguity, particularly for the rare classes in truths (GTs) for the synthetic images, and then compute the
LVIS. It should be noted that the description can be easily AP of the pseudo labels derived from the teacher model. In
extracted from the metadata of the dataset. Consequently, the case of Precision and Recall, we randomly select and
the text prompt for the selected categories is generated as annotate 200 synthetic images, then calculate the precision
follows: ‘a photograph of [category1 name] ([category1 de- and recall of their pseudo-labels. As shown in Table S1,
scription]) and [category2 name] ([category2 description])’. after self-training on novel categories, the quality of the
During the training of the grounding head, we utilize 500 pseudo-labels can be significantly improved in terms of De-
synthetic images per category per training epoch. In addi- tector AP (70.2%→79.7%), Precision (87.9%→89.1%) and
tion, for the training of the object detector, we employ 250 Recall (68.3%→90.0%).
synthetic images per category per training epoch and con-
duct 24 epochs of training. S4. Qualitative Results
Comparison to SOTA. We conduct a comparison with the We show more qualitative results generated by our InstaGen
existing CLIP-based open-vocabulary object detectors us- in Figure S1. Without any manual annotations, InstaGen
ing the Mask-RCNN model with ResNet-50, as shown in can generate high-quality images with object bounding-
Table S2. The results indicate that our detector, trained on boxes of novel categories. In Figure S2, we further show the
synthetic dataset from InstaGen, achieves comparable or qualitative results predicted by the Faster R-CNN trained
improved performance over existing CLIP-based methods. with the synthetic images form InstaGen on COCO vali-
dation set. The detector can now accurately localize and
S2. Tight coupling vs. Loose coupling recognize the objects from novel categories.
To generate high-quality bounding-boxes for the synthetic
images, we have designed a tight coupling between the

1
 Method Supervision Detector Backbone Input Size AP APc APf APnovel
ViLD-ens. [9] CLIP Mask R-CNN R50 1024×1024 25.5 24.6 30.3 16.6
Detic [41] CLIP Mask R-CNN R50 1024×1024 26.8 26.3 31.6 17.8
F-VLM [16] CLIP Mask R-CNN R50 1024×1024 24.2 - - 18.6
PromptDet [5] CLIP Mask R-CNN R50 800×800 21.4 18.5 25.8 19.0
DetPro [2] CLIP Mask R-CNN R50 800×800 25.9 25.6 28.9 19.8
BARON [34] CLIP Mask R-CNN R50 800×800 25.1 24.4 28.9 18.0
BARON [34]† CLIP Mask R-CNN R50 800×800 27.6 27.6 29.8 22.6
InstaGen Stable Diffusion Mask R-CNN R50 800×800 23.0 20.6 27.1 20.3

Table S2. Results on open-vocabulary LVIS benchmark. † indicates using ensembling strategy for classification scores and learned prompts
for the category’s names.

Figure S1. Qualitative results generated by our InstaGen. The bounding-boxes with green denote the objects from base categories, while
the ones with red denote the objects from novel categories.

2
Figure S2. Qualitative results from our Faster R-CNN trained with the synthetic images from InstaGen on COCO validation set. The
bounding-boxes with green denote the objects from base categories, while the ones with red denote the objects from novel categories.