Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction

without Convolutions

Wenhai Wang1 , Enze Xie2 , Xiang Li3 , Deng-Ping Fan4B ,

Kaitao Song3 , Ding Liang5 , Tong Lu1B , Ping Luo2 , Ling Shao4
1
Nanjing University 2 The University of Hong Kong
3
Nanjing University of Science and Technology 4 IIAI 5 SenseTime Research
arXiv:2102.12122v2 [[Link]] 11 Aug 2021

[Link]

TASK TASK TASK

Conv 4 TF-E 4
1. CLS 1. CLS 1. CLS
Conv 3 2. DET TF-E 3 2. DET
3. SEG 3. SEG
... ...
Conv 2 TF-E 2

!× !! ×
Conv 1 TF-E Transformer TF-E 1 Transformer
Block Block

Shrink

(a) CNNs: VGG [54], ResNet [22], etc. (b) Vision Transformer [13] (c) Pyramid Vision Transformer (ours)
Figure 1: Comparisons of different architectures, where “Conv” and “TF-E” stand for “convolution” and “Transformer
encoder”, respectively. (a) Many CNN backbones use a pyramid structure for dense prediction tasks such as object detection
(DET), instance and semantic segmentation (SEG). (b) The recently proposed Vision Transformer (ViT) [13] is a “columnar”
structure specifically designed for image classification (CLS). (c) By incorporating the pyramid structure from CNNs, we
present the Pyramid Vision Transformer (PVT), which can be used as a versatile backbone for many computer vision tasks,
broadening the scope and impact of ViT. Moreover, our experiments also show that PVT can easily be combined with
DETR [6] to build an end-to-end object detection system without convolutions.

Abstract ous vision tasks without convolutions, where it can be used

as a direct replacement for CNN backbones. (3) We val-
Although convolutional neural networks (CNNs) have idate PVT through extensive experiments, showing that it
achieved great success in computer vision, this work inves- boosts the performance of many downstream tasks, includ-
tigates a simpler, convolution-free backbone network use- ing object detection, instance and semantic segmentation.
ful for many dense prediction tasks. Unlike the recently- For example, with a comparable number of parameters,
proposed Vision Transformer (ViT) that was designed for PVT+RetinaNet achieves 40.4 AP on the COCO dataset,
image classification specifically, we introduce the Pyra- surpassing ResNet50+RetinNet (36.3 AP) by 4.1 absolute
mid Vision Transformer (PVT), which overcomes the diffi- AP (see Figure 2). We hope that PVT could serve as an
culties of porting Transformer to various dense prediction alternative and useful backbone for pixel-level predictions
tasks. PVT has several merits compared to current state and facilitate future research.
of the arts. (1) Different from ViT that typically yields low-
resolution outputs and incurs high computational and mem-
ory costs, PVT not only can be trained on dense partitions
1. Introduction
of an image to achieve high output resolution, which is im-
portant for dense prediction, but also uses a progressive Convolutional neural network (CNNs) have achieved re-
shrinking pyramid to reduce the computations of large fea- markable success in computer vision, making them a ver-
ture maps. (2) PVT inherits the advantages of both CNN satile and dominant approach for almost all tasks [54, 22,
and Transformer, making it a unified backbone for vari- 73, 49, 21, 39, 9, 32]. Nevertheless, this work aims to ex-
B Corresponding authors: Deng-Ping Fan (dengpfan@[Link]); plore an alternative backbone network beyond CNN, which
Tong Lu (lutong@[Link]). can be used for dense prediction tasks such as object detec-

1
 44 shorter edge of 800 pixels in the COCO benchmark [40]).
PVT-L
PVT-M
42 To address the above limitations, this work proposes a
PVT-S pure Transformer backbone, termed Pyramid Vision Trans-
40 X101-64x4d
COCO BBox AP (%)

X101-32x4d former (PVT), which can serve as an alternative to the CNN

Backbone #Param (M) AP
38 R101
backbone in many downstream tasks, including image-level
PVT-T R18 [22] 21.3 31.8
PVT-T (ours) 23.0 36.7 prediction as well as pixel-level dense predictions. Specifi-
36 R50 R50 [22] 37.7 36.3 cally, as illustrated in Figure 1 (c), our PVT overcomes the
PVT-S (ours) 34.2 40.4
34 difficulties of the conventional Transformer by (1) taking
R101 [22] 56.7 38.5
X101-32x4d [73] 56.4 39.9 fine-grained image patches (i.e., 4×4 pixels per patch) as in-
32 ViT-S/32 [13] 60.8 31.7
R18 ViT-S/32 put to learn high-resolution representation, which is essen-
PVT-M (ours) 53.9 41.9
30 X101-64x4d [73] 95.5 41.0 tial for dense prediction tasks; (2) introducing a progressive
28
PVT-L (ours) 71.1 42.6 shrinking pyramid to reduce the sequence length of Trans-
20 30 40 50 60 70 80 90 100 110 120 130 former as the network deepens, significantly reducing the
#Parameter (M)
computational cost, and (3) adopting a spatial-reduction at-
tention (SRA) layer to further reduce the resource consump-
Figure 2: Performance comparison on COCO val2017 tion when learning high-resolution features.
of different backbones using RetinaNet for object detec- Overall, the proposed PVT possesses the following mer-
tion, where “T”, “S”, “M” and “L” denote our PVT models its. Firstly, compared to the traditional CNN backbones
with tiny, small, medium and large size. We see that when (see Figure 1 (a)), which have local receptive fields that in-
the number of parameters among different models are com- crease with the network depth, our PVT always produces a
parable, PVT variants significantly outperform their corre- global receptive field, which is more suitable for detection
sponding counterparts such as ResNets (R) [22], ResNeXts and segmentation. Secondly, compared to ViT (see Fig-
(X) [73], and ViT [13]. ure 1 (b)), thanks to its advanced pyramid structure, our
method can more easily be plugged into many represen-
tative dense prediction pipelines, e.g., RetinaNet [39] and
tion [40, 14], semantic [83] and instance segmentation [40], Mask R-CNN [21]. Thirdly, we can build a convolution-
in addition to image classification [12]. free pipeline by combining our PVT with other task-specific
Inspired by the success of Transformer [64] in natu- Transformer decoders, such as PVT+DETR [6] for ob-
ral language processing, many researchers have explored ject detection. To our knowledge, this is the first entirely
its application in computer vision. For example, some convolution-free object detection pipeline.
works [6, 85, 72, 56, 24, 42] model the vision task as a dic- Our main contributions are as follows:
tionary lookup problem with learnable queries, and use the (1) We propose Pyramid Vision Transformer (PVT),
Transformer decoder as a task-specific head on top of the which is the first pure Transformer backbone designed for
CNN backbone. Although some prior arts have also incor- various pixel-level dense prediction tasks. Combining our
porated attention modules [70, 48, 80] into CNNs, as far PVT and DETR, we can construct an end-to-end object de-
as we know, exploring a clean and convolution-free Trans- tection system without convolutions and handcrafted com-
former backbone to address dense prediction tasks in com- ponents such as dense anchors and non-maximum suppres-
puter vision is rarely studied. sion (NMS).
Recently, Dosovitskiy et al. [13] introduced the Vision (2) We overcome many difficulties when porting Trans-
Transformer (ViT) for image classification. This is an in- former to dense predictions, by designing a progressive
teresting and meaningful attempt to replace the CNN back- shrinking pyramid and a spatial-reduction attention (SRA).
bone with a convolution-free model. As shown in Figure 1 These are able to reduce the resource consumption of Trans-
(b), ViT has a columnar structure with coarse image patches former, making PVT flexible to learning multi-scale and
as input.1 Although ViT is applicable to image classifi- high-resolution features.
cation, it is challenging to directly adapt it to pixel-level (3) We evaluate the proposed PVT on several differ-
dense predictions such as object detection and segmenta- ent tasks, including image classification, object detection,
tion, because (1) its output feature map is single-scale and instance and semantic segmentation, and compare it with
low-resolution, and (2) its computational and memory costs popular ResNets [22] and ResNeXts [73]. As presented
are relatively high even for common input image sizes (e.g., in Figure 2, our PVT with different parameter scales can
1 Due
consistently archived improved performance compared to
to resource constraints, ViT cannot use fine-grained image
patches (e.g., 4×4 pixels per patch) as input, instead only receive coarse
the prior arts. For example, under a comparable number
patches (e.g., 32×32 pixels per patch) as input, which leads to its low out- of parameters, using RetinaNet [39] for object detection,
put resolution (e.g., 32-stride). PVT-Small achieves 40.4 AP on COCO val2017, outper-
forming ResNet50 by 4.1 points (40.4 vs. 36.3). Moreover, resolution or multi-scale feature maps for accurate object
PVT-Large achieves 42.6 AP, which is 1.6 points better than detection.
ResNeXt101-64x4d, with 30% less parameters. Semantic Segmentation. CNNs also play an important
role in semantic segmentation. In the early stages, FCN
2. Related Work [44] introduced a fully convolutional architecture to gen-
erate a spatial segmentation map for a given image of any
2.1. CNN Backbones size. After that, the deconvolution operation was introduced
CNNs are the work-horses of deep neural networks in vi- by Noh et al. [47] and achieved impressive performance on
sual recognition. The standard CNN was first introduced in the PASCAL VOC 2012 dataset [52]. Inspired by FCN, U-
[34] to distinguish handwritten numbers. The model con- Net [50] was proposed for the medical image segmentation
tains convolutional kernels with a certain receptive field domain specifically, bridging the information flow between
that captures favorable visual context. To provide trans- corresponding low-level and high-level feature maps of the
lation equivariance, the weights of convolutional kernels same spatial sizes. To explore richer global context rep-
are shared over the entire image space. More recently, resentation, Zhao et al. [81] designed a pyramid pooling
with the rapid development of the computational resources module over various pooling scales, and Kirillov et al. [32]
(e.g., GPU), the successful training of stacked convolutional developed a lightweight segmentation head termed Seman-
blocks [33, 54] on large-scale image classification datasets tic FPN, based on FPN [38]. Finally, the DeepLab family
(e.g., ImageNet [51]) has become possible. For instance, [8, 41] applies dilated convolutions to enlarge the receptive
GoogLeNet [59] demonstrated that a convolutional opera- field while maintaining the feature map resolution. Similar
tor containing multiple kernel paths can achieve very com- to object detection methods, semantic segmentation models
petitive performance. The effectiveness of a multi-path also rely on high-resolution or multi-scale feature maps.
convolutional block was further validated in Inception se-
ries [60, 58], ResNeXt [73], DPN [10], MixNet [65] and 2.3. Self-Attention and Transformer in Vision
SKNet [36]. Further, ResNet [22] introduced skip connec- As convolutional filter weights are usually fixed after
tions into the convolutional block, making it possible to cre- training, they cannot be dynamically adapted to different
ate/train very deep networks and obtaining impressive re- inputs. Many methods have been proposed to alleviate this
sults in the field of computer vision. DenseNet [25] intro- problem using dynamic filters [30] or self-attention oper-
duced a densely connected topology, which connects each ations [64]. The non-local block [70] attempts to model
convolutional block to all previous blocks. More recent ad- long-range dependencies in both space and time, which
vances can be found in recent survey/review papers [31, 53]. has been shown beneficial for accurate video classifica-
Unlike the full-blown CNNs, the vision Transformer tion. However, despite its success, the non-local opera-
backbone is still in its early stage of development. In this tor suffers from the high computational and memory costs.
work, we try to extend the scope of Vision Transformer by Criss-cross [26] further reduces the complexity by gen-
designing a new versatile Transformer backbone suitable erating sparse attention maps through a criss-cross path.
for most vision tasks. Ramachandran et al. [48] proposed the stand-alone self-
attention to replace convolutional layers with local self-
2.2. Dense Prediction Tasks
attention units. AANet [3] achieves competitive results
Preliminary. The dense prediction task aims to perform when combining the self-attention and convolutional oper-
pixel-level classification or regression on a feature map. ations. LambdaNetworks [2] uses the lambda layer, an ef-
Object detection and semantic segmentation are two rep- ficient self-attention to replace the convolution in the CNN.
resentative dense prediction tasks. DETR [6] utilizes the Transformer decoder to model ob-
Object Detection. In the era of deep learning, ject detection as an end-to-end dictionary lookup problem
CNNs [34] have become the dominant framework for ob- with learnable queries, successfully removing the need for
ject detection, which includes single-stage detectors (e.g., handcrafted processes such as NMS. Based on DETR, de-
SSD [43], RetinaNet [39], FCOS [62], GFL [37, 35], Po- formable DETR [85] further adopts a deformable atten-
larMask [71] and OneNet [55]) and multi-stage detectors tion layer to focus on a sparse set of contextual elements,
(Faster R-CNN [49], Mask R-CNN [21], Cascade R-CNN obtaining faster convergence and better performance. Re-
[4] and Sparse R-CNN [57]). Most of these popular ob- cently, Vision Transformer (ViT) [13] employs a pure
ject detectors are built on high-resolution or multi-scale fea- Transformer [64] model for image classification by treat-
ture maps to obtain good detection performance. Recently, ing an image as a sequence of patches. DeiT [63] further
DETR [6] and deformable DETR [85] combined the CNN extends ViT using a novel distillation approach. Different
backbone and the Transformer decoder to build an end- from previous models, this work introduces the pyramid
to-end object detector. Likewise, they also require high- structure into Transformer to present a pure Transformer
 𝐻×𝑊×3 𝐻 𝑊 𝐻 𝑊 𝐻 𝑊 𝐻 𝑊
𝐹!: × ×𝐶! 𝐹" : × ×𝐶" 𝐹# : × ×𝐶# 𝐹$ : × ×𝐶
4 4 8 8 16 16 32 32 $

Stage 1 Stage 2 Stage 3 Stage 4

Patch Emb

Patch Emb

Patch Emb

Patch Emb
Encoder

Encoder

Encoder

Encoder
Stage i
SRA

Multi-Head
Attention

Forward
Linear

Norm

Norm

Norm
Reshape Reshape

Feed
Reduction
Spacial
Position Embedding
𝐻!"#𝑊!"#
×𝐶!
𝑃!$
Element-wise Add Patch 𝐻!"# 𝑊!"#
×
𝐻!"# 𝑊!"# 𝑃! 𝑃!
× Embedding ×𝐶!
𝑃! 𝑃!
Feature Map ×(𝑃!$𝐶!"#) Transformer Encoder (𝐿% ×)

Figure 3: Overall architecture of Pyramid Vision Transformer (PVT). The entire model is divided into four stages, each
of which is comprised of a patch embedding layer and a Li -layer Transformer encoder. Following a pyramid structure, the
output resolution of the four stages progressively shrinks from high (4-stride) to low (32-stride).

backbone for dense prediction tasks, rather than a task- downstream tasks, including image classification, object de-
specific head or an image classification model. tection, and semantic segmentation.

3. Pyramid Vision Transformer (PVT) 3.2. Feature Pyramid for Transformer

Unlike CNN backbone networks [54, 22], which use
3.1. Overall Architecture
different convolutional strides to obtain multi-scale feature
Our goal is to introduce the pyramid structure into the maps, our PVT uses a progressive shrinking strategy to con-
Transformer framework, so that it can generate multi-scale trol the scale of feature maps by patch embedding layers.
feature maps for dense prediction tasks (e.g., object detec- Here, we denote the patch size of the i-th stage as Pi . At
tion and semantic segmentation). An overview of PVT is the beginning of stage i, we first evenly divide the input fea-
depicted in Figure 3. Similar to CNN backbones [22], our ture map Fi−1 ∈ RHi−1×Wi−1×Ci−1 into Hi−P1 W 2
i−1
patches, and
i
method has four stages that generate feature maps of dif- then each patch is flatten and projected to a Ci -dimensional
ferent scales. All stages share a similar architecture, which embedding. After the linear projection, the shape of the em-
consists of a patch embedding layer and Li Transformer en- bedded patches can be viewed as HPi−i 1 × WPi−i 1 × Ci , where
coder layers. the height and width are Pi times smaller than the input.
In the first stage, given an input image of size H×W ×3, In this way, we can flexibly adjust the scale of the feature
we first divide it into HW 2
42 patches, each of size 4×4×3. map in each stage, making it possible to construct a feature
Then, we feed the flattened patches to a linear projection pyramid for Transformer.
and obtain embedded patches of size HW 42 ×C1 . After that,
the embedded patches along with a position embedding are 3.3. Transformer Encoder
passed through a Transformer encoder with L1 layers, and The Transformer encoder in the stage i has Li encoder
the output is reshaped to a feature map F1 of size H W
4 × 4 ×C1 . layers, each of which is composed of an attention layer
In the same way, using the feature map from the previ- and a feed-forward layer [64]. Since PVT needs to process
ous stage as input, we obtain the following feature maps: high-resolution (e.g., 4-stride) feature maps, we propose a
F2 , F3 , and F4 , whose strides are 8, 16, and 32 pixels spatial-reduction attention (SRA) layer to replace the tradi-
with respect to the input image. With the feature pyramid tional multi-head attention (MHA) layer [64] in the encoder.
{F1 , F2 , F3 , F4 }, our method can be easily applied to most Similar to MHA, our SRA receives a query Q, a key K,
2 As done for ResNet, we keep the highest resolution of our output fea- and a value V as input, and outputs a refined feature. The
ture map at 4-stride. difference is that our SRA reduces the spatial scale of K
 • Ci : the channel number of the output of Stage i;
Multi-Head Multi-Head • Li : the number of encoder layers in Stage i;
Attention Attention 𝐻! 𝑊! • Ri : the reduction ratio of the SRA in Stage i;
×𝐶!
𝑅"!
Spatial
• Ni : the head number of the SRA in Stage i;
Reduction • Ei : the expansion ratio of the feed-forward layer [64]
in Stage i;
Q K V Q K V (𝐻! 𝑊! )×𝐶!
Multi-Head Attention Spatial-Reduction Attention (ours) Following the design rules of ResNet [22], we (1) use small
output channel numbers in shallow stages; and (2) concen-
Figure 4: Multi-head attention (MHA) vs. spatial- trate the major computation resource in intermediate stages.
reduction attention (SRA). With the spatial-reduction op- To provide instances for discussion, we describe a series
eration, the computational/memory cost of our SRA is of PVT models with different scales, namely PVT-Tiny, -
much lower than that of MHA. Small, -Medium, and -Large, in Table 1, whose parameter
numbers are comparable to ResNet18, 50, 101, and 152 re-
spectively. More details of employing these models in spe-
and V before the attention operation (see Figure 4), which cific downstream tasks will be introduced in Section 4.
largely reduces the computational/memory overhead. De-
tails of the SRA in the stage i can be formulated as follows: 3.5. Discussion
The most related work to our model is ViT [13]. Here,
SRA(Q, K, V ) = Concat(head0 , ..., headNi )W O , (1)
we discuss the relationship and differences between them.
First, both PVT and ViT are pure Transformer models with-
headj = Attention(QWjQ , SR(K)WjK , SR(V)WjV ), (2)
out convolutions. The primary difference between them
where Concat(·) is the concatenation operation as in [64]. is the pyramid structure. Similar to the traditional Trans-
WjQ ∈ RCi×dhead , WjK ∈ RCi×dhead , WjV ∈ RCi×dhead , and former [64], the length of ViT’s output sequence is the same
W O ∈ RCi×Ci are linear projection parameters. Ni is the as the input, which means that the output of ViT is single-
head number of the attention layer in Stage i. Therefore, the scale (see Figure 1 (b)). Moreover, due to the limited re-
Ci
dimension of each head (i.e., dhead ) is equal to N . SR(·) is source, the input of ViT is coarse-grained (e.g., the patch
i
the operation for reducing the spatial dimension of the input size is 16 or 32 pixels), and thus its output resolution is rel-
sequence (i.e., K or V ), which is written as: atively low (e.g., 16-stride or 32-stride). As a result, it is
difficult to directly apply ViT to dense prediction tasks that
SR(x) = Norm(Reshape(x, Ri )W S ). (3) require high-resolution or multi-scale feature maps.
Our PVT breaks the routine of Transformer by intro-
Here, x ∈ R(Hi Wi )×Ci represents a input sequence, and ducing a progressive shrinking pyramid. It can gener-
Ri denotes the reduction ratio of the attention layers in ate multi-scale feature maps like a traditional CNN back-
Stage i. Reshape(x, Ri ) is an operation of reshaping the bone. In addition, we also designed a simple but effec-
input sequence x to a sequence of size HRi W2
i
× (Ri2 Ci ). tive attention layer—SRA, to process high-resolution fea-
i
2
WS ∈ R(Ri Ci )×Ci is a linear projection that reduces the di- ture maps and reduce computational/memory costs. Ben-
mension of the input sequence to Ci . Norm(·) refers to efiting from the above designs, our method has the fol-
layer normalization [1]. As in the original Transformer [64], lowing advantages over ViT: 1) more flexible—can gen-
our attention operation Attention(·) is calculated as: erate feature maps of different scales/channels in differ-
ent stages; 2) more versatile—can be easily plugged and
qkT played in most downstream task models; 3) more friendly
Attention(q, k, v) = Softmax( √ )v. (4)
dhead to computation/memory—can handle higher resolution fea-
ture maps or longer sequences.
Through these formulas, we can find that the computa-
tional/memory costs of our attention operation are Ri2 times 4. Application to Downstream Tasks
lower than those of MHA, so our SRA can handle larger
input feature maps/sequences with limited resources. 4.1. Image-Level Prediction
3.4. Model Details Image classification is the most classical task of image-
level prediction. To provide instances for discussion, we
In summary, the hyper parameters of our method are
design a series of PVT models with different scales, namely
listed as follows:
PVT-Tiny, -Small, -Medium, and -Large, whose parameter
• Pi : the patch size of Stage i; numbers are similar to ResNet18, 50, 101, and 152, respec-
 Output Size Layer Name PVT-Tiny PVT-Small PVT-Medium PVT-Large
Patch Embedding    P
1 = 4;  1 = 64 
C  
H W R1 = 8 R1 = 8 R1 = 8 R1 = 8
Stage 1
4 × 4
Transformer  N1 = 1  × 2  N1 = 1  × 3  N1 = 1  × 3  N1 = 1  × 3
Encoder
E1 = 8 E1 = 8 E1 = 8 E1 = 8
Patch Embedding    P2= 2; C
 = 128 
2  
H W R2 = 4 R2 = 4 R2 = 4 R2 = 4
Stage 2
8 × 8
Transformer  N2 = 2  × 2  N2 = 2  × 3  N2 = 2  × 3  N2 = 2  × 8
Encoder
E2 = 8 E2 = 8 E2 = 8 E2 = 8
Patch Embedding    P3= 2;  = 320 
C 3  
H W R3 = 2 R3 = 2 R3 = 2 R3 = 2
Stage 3
16 × 16
Transformer  N3 = 5  × 2  N3 = 5  × 6  N3 = 5  × 18  N3 = 5  × 27
Encoder
E3 = 4 E3 = 4 E3 = 4 E3 = 4
Patch Embedding    P4 = 2;  = 512 
C4  
H W R4 = 1 R4 = 1 R4 = 1 R4 = 1
Stage 4
32 × 32
Transformer  N4 = 8  × 2  N4 = 8  × 3  N4 = 8  × 3  N4 = 8  × 3
Encoder
E4 = 4 E4 = 4 E4 = 4 E4 = 4

Table 1: Detailed settings of PVT series. The design follows the two rules of ResNet [22]: (1) with the growth of network
depth, the hidden dimension gradually increases, and the output resolution progressively shrinks; (2) the major computation
resource is concentrated in Stage 3.

tively. Detailed hyper-parameter settings of the PVT series be an arbitrary shape, the position embeddings pre-trained
are provided in the supplementary material (SM). on ImageNet may no longer be meaningful. Therefore, we
For image classification, we follow ViT [13] and perform bilinear interpolation on the pre-trained position
DeiT [63] to append a learnable classification token to the embeddings according to the input resolution.
input of the last stage, and then employ a fully connected
(FC) layer to conduct classification on top of the token. 5. Experiments
We compare PVT with the two most representative CNN
4.2. Pixel-Level Dense Prediction
backbones, i.e., ResNet [22] and ResNeXt [73], which are
In addition to image-level prediction, dense prediction widely used in the benchmarks of many downstream tasks.
that requires pixel-level classification or regression to be
performed on the feature map, is also often seen in down- 5.1. Image Classification
stream tasks. Here, we discuss two typical tasks, namely Settings. Image classification experiments are performed
object detection, and semantic segmentation. on the ImageNet 2012 dataset [51], which comprises 1.28
We apply our PVT models to three representative dense million training images and 50K validation images from
prediction methods, namely RetinaNet [39], Mask R- 1,000 categories. For fair comparison, all models are
CNN [21], and Semantic FPN [32]. RetinaNet is a widely trained on the training set, and report the top-1 error on the
used single-stage detector, Mask R-CNN is the most pop- validation set. We follow DeiT [63] and apply random crop-
ular two-stage instance segmentation framework, and Se- ping, random horizontal flipping [59], label-smoothing reg-
mantic FPN is a vanilla semantic segmentation method ularization [60], mixup [78], CutMix [76], and random eras-
without special operations (e.g., dilated convolution). Us- ing [82] as data augmentations. During training, we employ
ing these methods as baselines enables us to adequately ex- AdamW [46] with a momentum of 0.9, a mini-batch size of
amine the effectiveness of different backbones. 128, and a weight decay of 5 × 10−2 to optimize models.
The implementation details are as follows: (1) Like The initial learning rate is set to 1 × 10−3 and decreases fol-
ResNet, we initialize the PVT backbone with the weights lowing the cosine schedule [45]. All models are trained for
pre-trained on ImageNet; (2) We use the output feature 300 epochs from scratch on 8 V100 GPUs. To benchmark,
pyramid {F1 , F2 , F3 , F4 } as the input of FPN [38], and we apply a center crop on the validation set, where a 224×
then the refined feature maps are fed to the follow-up de- 224 patch is cropped to evaluate the classification accuracy.
tection/segmentation head; (3) When training the detec- Results. In Table 2, we see that our PVT models are supe-
tion/segmentation model, none of the layers in PVT are rior to conventional CNN backbones under similar parame-
frozen; (4) Since the input for detection/segmentation can ter numbers and computational budgets. For example, when
 Method #Param (M) GFLOPs Top-1 Err (%) ers. Our models are trained with a batch size of 16 on 8
ResNet18* [22] 11.7 1.8 30.2
ResNet18 [22] 11.7 1.8 31.5
V100 GPUs and optimized by AdamW [46] with an ini-
DeiT-Tiny/16 [63] 5.7 1.3 27.8 tial learning rate of 1 × 10−4 . Following common prac-
PVT-Tiny (ours) 13.2 1.9 24.9 tices [39, 21, 7], we adopt 1× or 3× training schedule (i.e.,
ResNet50* [22] 25.6 4.1 23.9 12 or 36 epochs) to train all detection models. The training
ResNet50 [22] 25.6 4.1 21.5
ResNeXt50-32x4d* [73] 25.0 4.3 22.4
image is resized to have a shorter side of 800 pixels, while
ResNeXt50-32x4d [73] 25.0 4.3 20.5 the longer side does not exceed 1,333 pixels. When using
T2T-ViTt -14 [75] 22.0 6.1 19.3 the 3× training schedule, we randomly resize the shorter
TNT-S [19] 23.8 5.2 18.7 side of the input image within the range of [640, 800]. In
DeiT-Small/16 [63] 22.1 4.6 20.1
PVT-Small (ours) 24.5 3.8 20.2 the testing phase, the shorter side of the input image is fixed
ResNet101* [22] 44.7 7.9 22.6 to 800 pixels.
ResNet101 [22] 44.7 7.9 20.2 Results. As shown in Table 3, when using RetinaNet for
ResNeXt101-32x4d* [73] 44.2 8.0 21.2
object detection, we find that under comparable number of
ResNeXt101-32x4d [73] 44.2 8.0 19.4
T2T-ViTt -19 [75] 39.0 9.8 18.6 parameters, the PVT-based models significantly surpasses
ViT-Small/16 [13] 48.8 9.9 19.2 their counterparts. For example, with the 1× training sched-
PVT-Medium (ours) 44.2 6.7 18.8 ule, the AP of PVT-Tiny is 4.9 points better than that of
ResNeXt101-64x4d* [73] 83.5 15.6 20.4
ResNet18 (36.7 vs. 31.8). Moreover, with the 3× training
ResNeXt101-64x4d [73] 83.5 15.6 18.5
ViT-Base/16 [13] 86.6 17.6 18.2 schedule and multi-scale training, PVT-Large archive the
T2T-ViTt -24 [75] 64.0 15.0 17.8 best AP of 43.4, surpassing ResNeXt101-64x4d (43.4 vs.
TNT-B [19] 66.0 14.1 17.2 41.8), while our parameter number is 30% fewer. These re-
DeiT-Base/16 [63] 86.6 17.6 18.2
PVT-Large (ours) 61.4 9.8 18.3
sults indicate that our PVT can be a good alternative to the
CNN backbone for object detection.
Table 2: Image classification performance on the Ima- Similar results are found in instance segmentation exper-
geNet validation set. “#Param” refers to the number of iments based on Mask R-CNN, as shown in Table 4. With
parameters. “GFLOPs” is calculated under the input scale the 1× training schedule, PVT-Tiny achieves 35.1 mask AP
of 224 × 224. “*” indicates the performance of the method (APm ), which is 3.9 points better than ResNet18 (35.1 vs.
trained under the strategy of its original paper. 31.2) and even 0.7 points higher than ResNet50 (35.1 vs.
34.4). The best APm obtained by PVT-Large is 40.7, which
is 1.0 points higher than ResNeXt101-64x4d (40.7 vs. 39.7),
the GFLOPs are roughly similar, the top-1 error of PVT- with 20% fewer parameters.
Small reaches 20.2, which is 1.3 points higher than that of
ResNet50 [22] (20.2 vs. 21.5). Meanwhile, under similar or 5.3. Semantic Segmentation
lower complexity, PVT models archive performances com- Settings. We choose ADE20K [83], a challenging scene
parable to the recently proposed Transformer-based mod- parsing dataset, to benchmark the performance of semantic
els, such as ViT [13] and DeiT [63] (PVT-Large: 18.3 vs. segmentation. ADE20K contains 150 fine-grained semantic
ViT(DeiT)-Base/16: 18.3). Here, we clarify that these re- categories, with 20,210, 2,000, and 3,352 images for train-
sults are within our expectations, because the pyramid struc- ing, validation, and testing, respectively. We evaluate our
ture is beneficial to dense prediction tasks, but brings little PVT backbones on the basis of Semantic FPN [32], a sim-
improvements to image classification. ple segmentation method without dilated convolutions [74].
Note that ViT and DeiT have limitations as they are In the training phase, the backbone is initialized with the
specifically designed for classification tasks, and thus are weights pre-trained on ImageNet [12], and other newly
not suitable for dense prediction tasks, which usually re- added layers are initialized with Xavier [18]. We optimize
quire effective feature pyramids. our models using AdamW [46] with an initial learning rate
of 1e-4. Following common practices [32, 8], we train our
5.2. Object Detection
models for 80k iterations with a batch size of 16 on 4 V100
Settings. Object detection experiments are conducted on GPUs. The learning rate is decayed following the polyno-
the challenging COCO benchmark [40]. All models are mial decay schedule with a power of 0.9. We randomly
trained on COCO train2017 (118k images) and evalu- resize and crop the image to 512 × 512 for training, and
ated on val2017 (5k images). We verify the effectiveness rescale to have a shorter side of 512 pixels during testing.
of PVT backbones on top of two standard detectors, namely Results. As shown in Table 5, when using Seman-
RetinaNet [39] and Mask R-CNN [21]. Before training, we tic FPN [32] for semantic segmentation, PVT-based
use the weights pre-trained on ImageNet to initialize the models consistently outperforms the models based on
backbone and Xavier [18] to initialize the newly added lay- ResNet [22] or ResNeXt [73]. For example, with al-
 #Param RetinaNet 1x RetinaNet 3x + MS
Backbone
(M) AP AP50 AP75 APS APM APL AP AP50 AP75 APS APM APL
ResNet18 [22] 21.3 31.8 49.6 33.6 16.3 34.3 43.2 35.4 53.9 37.6 19.5 38.2 46.8
PVT-Tiny (ours) 23.0 36.7(+4.9) 56.9 38.9 22.6 38.8 50.0 39.4(+4.0) 59.8 42.0 25.5 42.0 52.1
ResNet50 [22] 37.7 36.3 55.3 38.6 19.3 40.0 48.8 39.0 58.4 41.8 22.4 42.8 51.6
PVT-Small (ours) 34.2 40.4(+4.1) 61.3 43.0 25.0 42.9 55.7 42.2(+3.2) 62.7 45.0 26.2 45.2 57.2
ResNet101 [22] 56.7 38.5 57.8 41.2 21.4 42.6 51.1 40.9 60.1 44.0 23.7 45.0 53.8
ResNeXt101-32x4d [73] 56.4 39.9(+1.4) 59.6 42.7 22.3 44.2 52.5 41.4(+0.5) 61.0 44.3 23.9 45.5 53.7
PVT-Medium (ours) 53.9 41.9(+3.4) 63.1 44.3 25.0 44.9 57.6 43.2(+2.3) 63.8 46.1 27.3 46.3 58.9
ResNeXt101-64x4d [73] 95.5 41.0 60.9 44.0 23.9 45.2 54.0 41.8 61.5 44.4 25.2 45.4 54.6
PVT-Large (ours) 71.1 42.6(+1.6) 63.7 45.4 25.8 46.0 58.4 43.4(+1.6) 63.6 46.1 26.1 46.0 59.5

Table 3: Object detection performance on COCO val2017. “MS” means that multi-scale training [39, 21] is used.

#Param Mask R-CNN 1x Mask R-CNN 3x + MS

Backbone
(M) APb APb
50 APb
75 AP
m APm
50 APm
75 APb APb b
50 AP75 AP
m APm
50 APm
75
ResNet18 [22] 31.2 34.0 54.0 36.7 31.2 51.0 32.7 36.9 57.1 40.0 33.6 53.9 35.7
PVT-Tiny (ours) 32.9 36.7(+2.7) 59.2 39.3 35.1(+3.9) 56.7 37.3 39.8(+2.9) 62.2 43.0 37.4(+3.8) 59.3 39.9
ResNet50 [22] 44.2 38.0 58.6 41.4 34.4 55.1 36.7 41.0 61.7 44.9 37.1 58.4 40.1
PVT-Small (ours) 44.1 40.4(+2.4) 62.9 43.8 37.8(+3.4) 60.1 40.3 43.0(+2.0) 65.3 46.9 39.9(+2.8) 62.5 42.8
ResNet101 [22] 63.2 40.4 61.1 44.2 36.4 57.7 38.8 42.8 63.2 47.1 38.5 60.1 41.3
ResNeXt101-32x4d [73] 62.8 41.9(+1.5) 62.5 45.9 37.5(+1.1) 59.4 40.2 44.0(+1.2) 64.4 48.0 39.2(+0.7) 61.4 41.9
PVT-Medium (ours) 63.9 42.0(+1.6) 64.4 45.6 39.0(+2.6) 61.6 42.1 44.2(+1.4) 66.0 48.2 40.5(+2.0) 63.1 43.5
ResNeXt101-64x4d [73] 101.9 42.8 63.8 47.3 38.4 60.6 41.3 44.4 64.9 48.8 39.7 61.9 42.6
PVT-Large (ours) 81.0 42.9(+0.1) 65.0 46.6 39.5(+1.1) 61.9 42.5 44.5(+0.1) 66.0 48.3 40.7(+1.0) 63.4 43.7

Table 4: Object detection and instance segmentation performance on COCO val2017. APb and APm denote bounding
box AP and mask AP, respectively.
.

Semantic FPN DETR (50 Epochs)

Backbone Method
#Param (M) GFLOPs mIoU (%) AP AP50 AP75 APS APM APL
ResNet18 [22] 15.5 32.2 32.9 ResNet50 [22] 32.3 53.9 32.3 10.7 33.8 53.0
PVT-Tiny (ours) 17.0 33.2 35.7(+2.8) PVT-Small (ours) 34.7(+2.4) 55.7 35.4 12.0 36.4 56.7
ResNet50 [22] 28.5 45.6 36.7
PVT-Small (ours) 28.2 44.5 39.8(+3.1) Table 6: Performance of the pure Transformer object
ResNet101 [22] 47.5 65.1 38.8 detection pipeline. We build a pure Transformer detector
ResNeXt101-32x4d [73] 47.1 64.7 39.7(+0.9)
PVT-Medium (ours) 48.0 61.0 41.6(+2.8) by combining PVT and DETR [6], whose AP is 2.4 points
ResNeXt101-64x4d [73] 86.4 103.9 40.2 higher than the original DETR based on ResNet50 [22].
PVT-Large (ours) 65.1 79.6 42.1(+1.9)
PVT-Large* (ours) 65.1 79.6 44.8
benefiting from the global attention mechanism.
Table 5: Semantic segmentation performance of differ-
ent backbones on the ADE20K validation set. “GFLOPs”
5.4. Pure Transformer Detection & Segmentation
is calculated under the input scale of 512 × 512. “*” indi-
cates 320K iterations training and multi-scale flip testing. PVT+DETR. To reach the limit of no convolution, we build
a pure Transformer pipeline for object detection by sim-
ply combining our PVT with a Transformer-based detection
most the same number of parameters and GFLOPs, our head—DETR [6]. We train models on COCO train2017
PVT-Tiny/Small/Medium are at least 2.8 points higher than for 50 epochs with an initial learning rate of 1 × 10−4 .
ResNet-18/50/101. In addition, although the parameter The learning rate is divided by 10 at the 33rd epoch. We
number and GFLOPs of our PVT-Large are 20% lower than use random flipping and multi-scale training as data aug-
those of ResNeXt101-64x4d, the mIoU is still 1.9 points mentation. All other experimental settings is the same as
higher (42.1 vs. 40.2). With a longer training schedule and those in Sec. 5.2. As reported in Table 6, PVT-based DETR
multi-scale testing, PVT-Large+Semantic FPN archives the archieves 34.7 AP on COCO val2017, outperforming
best mIoU of 44.8, which is very close to the state-of-the-art the original ResNet50-based DETR by 2.4 points (34.7 vs.
performance of the ADE20K benchmark. Note that Seman- 32.3). These results prove that a pure Transformer detector
tic FPN is just a simple segmentation head. These results can also works well in the object detection task.
demonstrate that our PVT backbones can extract better fea- PVT+[Link] build a pure Transformer model
tures for semantic segmentation than the CNN backbone, for semantic segmentation by combining our PVT with
Method #Param (M) GFLOPs mIoU (%)
45 42.2
ResNet50-d8+DeeplabV3+ [9] 26.8 120.5 41.5 40.4

COCO BBox AP (%)

ResNet50-d16+DeeplabV3+ [9] 26.8 45.5 40.6
ResNet50-d16+Trans2Seg [72] 56.1 79.3 39.7 35 33.8
PVT-Small+Trans2Seg 32.1 31.6 42.6(+2.9)
26.6
25
Table 7: Performance of the pure Transformer semantic PVT-Small w pre-train 1x
segmentation pipeline. We build a pure Transformer de- PVT-Small w/ pre-train 3x
15
tector by combining PVT and Trans2Seg [72]. It is 2.9% PVT-Small w/o pre-train 1x
PVT-Small w/o pre-train 3x
higher than ResNet50-d16+Trans2Seg and 1.1% higher 5
than ResNet50-d8+DeeplabV3+ with lower GFlops. “d8” 0 5 10 15 20 25 30 35
and “d16” means dilation 8 and 16, respectively. 45
42.2

COCO BBox AP (%)

40 40.4
#Param RetinaNet 1x 39.0
Method
(M) AP AP50 AP75 APS APM APL
ViT-Small/4 [13] 60.9 Out of Memory 35 36.3
ViT-Small/32 [13] 60.8 31.7 51.3 32.3 14.8 33.7 47.9 PVT-Small 1x
PVT-Small (ours) 34.2 40.4 61.3 43.0 25.0 42.9 55.7 PVT-Small 3x
30
ResNet50 1x
Table 8: Performance comparison between ViT and our ResNet50 3x
PVT using RetinaNet for object detection. ViT-Small/4 25
runs out of GPU memory due to small patch size (i.e., 0 5 10 15 20 25 30 35
4×4 per patch). ViT-Small/32 obtains 31.7 AP on COCO Epoch
val2017, which is 8.7 points lower than our PVT-Small.
Figure 5: AP curves of RetinaNet on COCO val2017
under different backbone settings. Top: using weights
Trans2Seg [72], a Transformer-based segmentation head. pre-trained on ImageNet vs. random initialization. Bottom:
According to the experimental settings in Sec. 5.3, we PVT-S vs. R50 [22].
perform experiments on ADE20K [83] with 40k iter-
ations training, single scale testing, and compare it #Param RetinaNet 1x
Method Top-1
with ResNet50+Trans2Seg [72] and DeeplabV3+ [9] with (M) AP AP50 AP75
ResNet50-d8 (dilation 8) and -d16(dilation 8) in Table Wider PVT-Small 46.8 19.3 40.8 61.8 43.3
Deeper PVT-Small 44.2 18.8 41.9 63.1 44.3
7. We find that our PVT-Small+Trans2Seg achieves 42.6
mIoU, outperforming ResNet50-d8+DeeplabV3+ (41.5). Table 9: Deeper vs. Wider. “Top-1” denotes the top-1 error
Note that, ResNet50-d8+DeeplabV3+ has 120.5 GFLOPs on the ImageNet validation set. “AP” denotes the bounding
due to the high computation cost of dilated convolution, and box AP on COCO val2017. The deep model (i.e., PVT-
our method has only 31.6 GFLOPs, which is 4 times fewer. Medium) obtains better performance than the wide model
In addition, our PVT-Small+Trans2Seg performs better than (i.e., PVT-Small-Wide ) under comparable parameter num-
ResNet50-d16+Trans2Seg (mIoU: 42.6 vs. 39.7, GFlops: ber.
31.6 vs. 79.3). These results prove that a pure Transformer
segmentation network is workable.
mance (31.7 AP on COCO val2017),3 as shown in Table
5.5. Ablation Study 8. When using fine-grained image patches (e.g., 4×4 pixels
per patch) as input like our PVT, ViT will exhaust the GPU
Settings. We conduct ablation studies on ImageNet [12]
memory (32G). Our method avoids this problem through a
and COCO [40] datasets. The experimental settings on Im-
progressive shrinking pyramid. Specifically, our model can
ageNet are the same as the settings in Sec. 5.1. For COCO,
process high-resolution feature maps in shallow stages and
all models are trained with a 1× training schedule (i.e., 12
low-resolution feature maps in deep stages. Thus, it obtains
epochs) and without multi-scale training, and other settings
a promising AP of 40.4 on COCO val2017, 8.7 points
follow those in Sec. 5.2.
higher than ViT-Small/32 (40.4 vs. 31.7).
Pyramid Structure. A Pyramid structure is crucial when
Deeper vs. Wider. The problem of whether the CNN back-
applying Transformer to dense prediction tasks. ViT (see
bone should go deeper or wider has been extensively dis-
Figure 1 (b)) is a columnar framework, whose output is
cussed in previous work [22, 77]. Here, we explore this
single-scale. This results in a low-resolution output fea-
ture map when using coarse image patches (e.g., 32×32 3 For adapting ViT to RetinaNet, we extract the features from the layer

pixels per patch) as input, leading to poor detection perfor- 2, 4, 6, and 8 of ViT-Small/32, and interpolate them to different scales.
 #Param Mask R-CNN 1x Time RetinaNet 1x
Method GFLOPs Method Scale GFLOPs
(M) APm APm 50 APm75 (ms) AP AP50 AP75
ResNet50+GC r4 [5] 54.2 279.6 36.2 58.7 38.3 ResNet50 [22] 800 239.3 55.9 36.3 55.3 38.6
PVT-Small (ours) 44.1 304.4 37.8 60.1 40.3 640 157.2 51.7 38.7 59.3 40.8
PVT-Small (ours)
800 285.8 76.9 40.4 61.3 43.0
m
Table 10: PVT vs. CNN w/ non-local. AP denotes
mask AP. Under similar parameter nubmer and GFLOPs, Table 11: Latency and AP under different input scales.
our PVT outperform the CNN backbone w/ Non-Local “Scale” and “Time” denote the input scale and time cost
(ResNet50+GC r4) by 1.6 APm (37.8 vs. 36.2). per image. When the shorter side is 640 pixels, the PVT-
Small+RetinaNet has a lower GFLOPs and time cost (on a
V100 GPU) than ResNet50+RetinaNet, while obtaining 2.4
problem in our PVT. For fair comparisons, we multiply points better AP (38.7 vs. 36.3).
the hidden dimensions {C1 , C2 , C3 , C4 } of PVT-Small by
a scale factor 1.4 to make it have an equivalent parameter
equipped with non-local blocks (e.g., GCNet).
number to the deep model (i.e., PVT-Medium). As shown
(2) Regular convolutions can be deemed as special in-
in Table 9, the deep model (i.e., PVT-Medium) consistently
stantiations of spatial attention mechanisms [84]. In other
works better than the wide model (i.e., PVT-Small-Wide) on
words, the format of MHA is more flexible than the regular
both ImageNet and COCO. Therefore, going deeper is more
convolution. For example, for different inputs, the weights
effective than going wider in the design of PVT. Based on
of the convolution are fixed, but the attention weights of
this observation, in Table 1, we develop PVT models with
MHA change dynamically with the input. Thus, the features
different scales by increasing the model depth.
learned by the pure Transformer backbone full of MHA lay-
Pre-trained Weights. Most dense prediction models (e.g., ers, could be more flexible and expressive.
RetinaNet [39]) rely on the backbone whose weights are
Computation Overhead. With increasing input scale, the
pre-trained on ImageNet. We also discuss this problem in
growth rate of the GFLOPs of our PVT is greater than
our PVT. In the top of Figure 5, we plot the validation AP
ResNet [22], but lower than ViT [13], as shown in Figure
curves of RetinaNet-PVT-Small w/ (red curves) and w/o
6. However, when the input scale does not exceed 640×640
(blue curves) pre-trained weights. We find that the model
pixels, the GFLOPs of PVT-Small and ResNet50 are simi-
w/ pre-trained weights converges better than the one w/o
lar. This means that our PVT is more suitable for tasks with
pre-trained weights, and the gap between their final AP
medium-resolution input.
reaches 13.8 under the 1× training schedule and 8.4 under
On COCO, the shorter side of the input image is 800
the 3× training schedule and multi-scale training. There-
pixels. Under this condition, the inference speed of Reti-
fore, like CNN-based models, pre-training weights can also
naNet based on PVT-Small is slower than the ResNet50-
help PVT-based models converge faster and better. More-
based model, as reported in Table 11. (1) A direct solution
over, in the bottom of Figure 5, we also see that the con-
for this problem is to reduce the input scale. When reduc-
vergence speed of PVT-based models (red curves) is faster
ing the shorter side of the input image to 640 pixels, the
than that of ResNet-based models (green curves).
model based on PVT-Small runs faster than the ResNet50-
PVT vs. “CNN w/ Non-Local” To obtain a global recep- based model (51.7ms vs., 55.9ms), with 2.4 higher AP
tive field, some well-engineered CNN backbones, such as (38.7 vs. 36.3). 2) Another solution is to develop a self-
GCNet [5], integrate the non-local block in the CNN frame- attention layer with lower computational complexity. This
work. Here, we compare the performance of our PVT (pure is a worth exploring direction, we recently propose a solu-
Transformer) and GCNet (CNN w/ non-local), using Mask tion PVTv2 [67].
R-CNN for instance segmentation. As reported in Table Detection & Segmentation Results. In Figure 7, we also
10, we find that our PVT-Small outperforms ResNet50+GC present some qualitative object detection and instance seg-
r4 [5] by 1.6 points in APm (37.8 vs. 36.2), and 2.0 points in mentation results on COCO val2017 [40], and semantic
APm 75 (38.3 vs. 40.3), under comparable parameter number segmentation results on ADE20K [83]. These results indi-
and GFLOPs. There are two possible reasons for this result: cate that a pure Transformer backbone (i.e., PVT) without
(1) Although a single global attention layer (e.g., non- convolutions can also be easily plugged in dense prediction
local [70] or multi-head attention (MHA) [64]) can ac- models (e.g., RetinaNet [39], Mask R-CNN [21], and Se-
quire global-receptive-field features, the model perfor- mantic FPN [32]), and obtain high-quality results.
mance keeps improving as the model deepens. This indi-
cates that stacking multiple MHAs can further enhance the 6. Conclusions and Future Work
representation capabilities of features. Therefore, as a pure
Transformer backbone with more global attention layers, We introduce PVT, a pure Transformer backbone for
our PVT tends to perform better than the CNN backbone dense prediction tasks, such as object detection and seman-
 300 References
ViT-Small/16
250 ViT-Small/32 [1] Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hin-
PVT-Small (ours) ton. Layer normalization. arXiv preprint arXiv:1607.06450,
200 2016. 5
ResNet50
GFLOPs

[2] Irwan Bello. Lambdanetworks: Modeling long-range inter-

150
actions without attention. In Proc. Int. Conf. Learn. Repre-
100
sentations, 2021. 3
[3] Irwan Bello, Barret Zoph, Ashish Vaswani, Jonathon Shlens,
50 and Quoc V Le. Attention augmented convolutional net-
works. In Proc. IEEE Int. Conf. Comp. Vis., 2019. 3
0
[4] Zhaowei Cai and Nuno Vasconcelos. Cascade r-cnn: Delv-
0 160 320 480 640 800 960 1120 1280
ing into high quality object detection. In Proc. IEEE Conf.
Input Scale Comp. Vis. Patt. Recogn., 2018. 3
[5] Yue Cao, Jiarui Xu, Stephen Lin, Fangyun Wei, and Han
Figure 6: Models’ GFLOPs under different input scales. Hu. Gcnet: Non-local networks meet squeeze-excitation net-
The growth rate of GFLOPs: ViT-Small/16 [13]>ViT- works and beyond. In Proc. IEEE Int. Conf. Comp. Vis.,
Small/32 [13]>PVT-Small (ours)>ResNet50 [22]. When 2019. 10
the input scale is less than 640 × 640, the GFLOPs of PVT- [6] Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas
Small and ResNet50 [22] are similar. Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-
end object detection with transformers. In Proc. Eur. Conf.
Comp. Vis., 2020. 1, 2, 3, 8
[7] Kai Chen, Jiaqi Wang, Jiangmiao Pang, Yuhang Cao, Yu
tic segmentation. We develop a progressive shrinking pyra- Xiong, Xiaoxiao Li, Shuyang Sun, Wansen Feng, Ziwei Liu,
mid and a spatial-reduction attention layer to obtain high- Jiarui Xu, et al. Mmdetection: Open mmlab detection tool-
resolution and multi-scale feature maps under limited com- box and benchmark. arXiv preprint arXiv:1906.07155, 2019.
putation/memory resources. Extensive experiments on ob- 7
ject detection and semantic segmentation benchmarks ver- [8] Liang-Chieh Chen, George Papandreou, Iasonas Kokkinos,
ify that our PVT is stronger than well-designed CNN back- Kevin Murphy, and Alan L Yuille. Deeplab: Semantic image
bones under comparable numbers of parameters. segmentation with deep convolutional nets, atrous convolu-
tion, and fully connected crfs. IEEE Trans. Pattern Anal.
Although PVT can serve as an alternative to CNN back-
Mach. Intell., 2017. 3, 7
bones (e.g., ResNet, ResNeXt), there are still some specific
modules and operations designed for CNNs and not consid- [9] Liang-Chieh Chen, Yukun Zhu, George Papandreou, Florian
Schroff, and Hartwig Adam. Encoder-decoder with atrous
ered in this work, such as SE [23], SK [36], dilated convo-
separable convolution for semantic image segmentation. In
lution [74], model pruning [20], and NAS [61]. Moreover, Proc. Eur. Conf. Comp. Vis., 2018. 1, 9
with years of rapid developments, there have been many
[10] Yunpeng Chen, Jianan Li, Huaxin Xiao, Xiaojie Jin,
well-engineered CNN backbones such as Res2Net [17], Shuicheng Yan, and Jiashi Feng. Dual path networks. Proc.
EfficientNet [61], and ResNeSt [79]. In contrast, the Advances in Neural Inf. Process. Syst., 2017. 3
Transformer-based model in computer vision is still in [11] Mingmei Cheng, Le Hui, Jin Xie, and Jian Yang. SSPC-
its early stage of development. Therefore, we believe Net: Semi-supervised semantic 3D point cloud segmentation
there are many potential technologies and applications (e.g., network. In Proc. AAAI Conf. Artificial Intell., 2021. 11
OCR [68, 66, 69], 3D [28, 11, 27] and medical [15, 16, 29] [12] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li,
image analysis) to be explored in the future, and hope that and Li Fei-Fei. Imagenet: A large-scale hierarchical im-
PVT could serve as a good starting point. age database. In Proc. IEEE Conf. Comp. Vis. Patt. Recogn.,
2009. 2, 7, 9
[13] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov,
Acknowledgments Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner,
Mostafa Dehghani, Matthias Minderer, Georg Heigold, Syl-
This work was supported by the Natural Science Founda- vain Gelly, et al. An image is worth 16x16 words: Trans-
tion of China under Grant 61672273 and Grant 61832008, formers for image recognition at scale. Proc. Int. Conf.
the Science Foundation for Distinguished Young Schol- Learn. Representations, 2021. 1, 2, 3, 5, 6, 7, 9, 10, 11
ars of Jiangsu under Grant BK20160021, Postdoctoral In- [14] Mark Everingham, Luc Van Gool, Christopher KI Williams,
novative Talent Support Program of China under Grant John Winn, and Andrew Zisserman. The pascal visual object
BX20200168, 2020M681608, the General Research Fund classes (voc) challenge. Int. J. Comput. Vision, 88(2):303–
of Hong Kong No. 27208720. 338, 2010. 2
 pe rson :0 .88

pe rson :0 .97

ze bra:1 .00
ze bra:1 .00
ze bra:1 .00

pe rson :0 .99 pe rson :1 .00

pe rson :1 .00

pe rson :0 .85

sk ateb oar d:0 .83

ca ke:0 .99
pe rson :0 .99

pe rson :0 .94
pe rson :1 .00
tr uck:0 .84
ca r:1 .00
pe rson :0 .94 pe rson :0 .99
be nch:1 .00

ca r:0 .89 pe rson :0 .99

Object Detection on COCO Instance Segmentation on COCO Semantic Segmentation on ADE20K

Figure 7: Qualitative results of object detection and instance segmentation on COCO val2017 [40], and semantic
segmentation on ADE20K [83]. The results (from left to right) are generated by PVT-Small-based RetinaNet [39], Mask
R-CNN [21], and Semantic FPN [32], respectively.

[15] Deng-Ping Fan, Ge-Peng Ji, Ming-Ming Cheng, and Ling arXiv:2103.00112, 2021. 7
Shao. Concealed object detection. IEEE Trans. Pattern Anal. [20] Song Han, Huizi Mao, and William J Dally. Deep com-
Mach. Intell., 2021. 11 pression: Compressing deep neural networks with pruning,
[16] Deng-Ping Fan, Ge-Peng Ji, Tao Zhou, Geng Chen, Huazhu trained quantization and huffman coding. arXiv preprint
Fu, Jianbing Shen, and Ling Shao. Pranet: Parallel reverse arXiv:1510.00149, 2015. 11
attention network for polyp segmentation. In International [21] Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Gir-
Conference on Medical Image Computing and Computer- shick. Mask r-cnn. In Proc. IEEE Int. Conf. Comp. Vis.,
Assisted Intervention, 2020. 11 2017. 1, 2, 3, 6, 7, 8, 10, 12
[17] Shanghua Gao, Ming-Ming Cheng, Kai Zhao, Xin-Yu [22] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.
Zhang, Ming-Hsuan Yang, and Philip HS Torr. Res2net: A Deep residual learning for image recognition. In Proc. IEEE
new multi-scale backbone architecture. IEEE Trans. Pattern Conf. Comp. Vis. Patt. Recogn., 2016. 1, 2, 3, 4, 5, 6, 7, 8, 9,
Anal. Mach. Intell., 2019. 11 10, 11
[18] Xavier Glorot and Yoshua Bengio. Understanding the diffi- [23] Jie Hu, Li Shen, and Gang Sun. Squeeze-and-excitation net-
culty of training deep feedforward neural networks. In Proc. works. In Proc. IEEE Conf. Comp. Vis. Patt. Recogn., 2018.
Int. Conf. Artificial Intell. & Stat., 2010. 7 11
[19] Kai Han, An Xiao, Enhua Wu, Jianyuan Guo, Chunjing Xu, [24] Ronghang Hu and Amanpreet Singh. Transformer is all you
and Yunhe Wang. Transformer in transformer. arXiv preprint need: Multimodal multitask learning with a unified trans-
 former. arXiv preprint arXiv:2102.10772, 2211. 2 [40] Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays,
[25] Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kil- Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence
ian Q Weinberger. Densely connected convolutional net- Zitnick. Microsoft coco: Common objects in context. In
works. In Proc. IEEE Conf. Comp. Vis. Patt. Recogn., 2017. Proc. Eur. Conf. Comp. Vis., 2014. 2, 7, 9, 10, 12
3 [41] Chenxi Liu, Liang-Chieh Chen, Florian Schroff, Hartwig
[26] Zilong Huang, Xinggang Wang, Lichao Huang, Chang Adam, Wei Hua, Alan L Yuille, and Li Fei-Fei. Auto-
Huang, Yunchao Wei, and Wenyu Liu. Ccnet: Criss-cross at- deeplab: Hierarchical neural architecture search for semantic
tention for semantic segmentation. In Proc. IEEE Int. Conf. image segmentation. In Proc. IEEE Conf. Comp. Vis. Patt.
Comp. Vis., 2019. 3 Recogn., 2019. 3
[27] Le Hui, Mingmei Cheng, Jin Xie, and Jian Yang. Efficient [42] Nian Liu, Ni Zhang, Kaiyuan Wan, Ling Shao, and Junwei
3D point cloud feature learning for large-scale place recog- Han. Visual saliency transformer. In Proc. IEEE Int. Conf.
nition. arXiv preprint arXiv:2101.02374, 2021. 11 Comp. Vis., 2021. 2
[28] Le Hui, Rui Xu, Jin Xie, Jianjun Qian, and Jian Yang. Pro- [43] Wei Liu, Dragomir Anguelov, Dumitru Erhan, Christian
gressive point cloud deconvolution generation network. In Szegedy, Scott Reed, Cheng-Yang Fu, and Alexander C
Proc. Eur. Conf. Comp. Vis., 2020. 11 Berg. Ssd: Single shot multibox detector. In Proc. Eur. Conf.
Comp. Vis., 2016. 3
[29] Ge-Peng Ji, Yu-Cheng Chou, Deng-Ping Fan, Geng Chen,
Huazhu Fu, Debesh Jha, and Ling Shao. Progressively nor- [44] Jonathan Long, Evan Shelhamer, and Trevor Darrell. Fully
malized self-attention network for video polyp segmentation. convolutional networks for semantic segmentation. In Proc.
In International Conference on Medical Image Computing IEEE Conf. Comp. Vis. Patt. Recogn., 2015. 3
and Computer-Assisted Intervention, 2021. 11 [45] Ilya Loshchilov and Frank Hutter. SGDR: stochastic gradi-
ent descent with warm restarts. In Proc. Int. Conf. Learn.
[30] Xu Jia, Bert De Brabandere, Tinne Tuytelaars, and Luc V
Representations, 2017. 6
Gool. Dynamic filter networks. In Proc. Advances in Neural
Inf. Process. Syst., 2016. 3 [46] Ilya Loshchilov and Frank Hutter. Decoupled weight decay
regularization. In Proc. Int. Conf. Learn. Representations,
[31] Asifullah Khan, Anabia Sohail, Umme Zahoora, and
2019. 6, 7
Aqsa Saeed Qureshi. A survey of the recent architectures of
[47] Hyeonwoo Noh, Seunghoon Hong, and Bohyung Han.
deep convolutional neural networks. Artificial Intelligence
Learning deconvolution network for semantic segmentation.
Review, 53(8):5455–5516, 2020. 3
In Proc. IEEE Int. Conf. Comp. Vis., 2015. 3
[32] Alexander Kirillov, Ross Girshick, Kaiming He, and Piotr
[48] Niki Parmar, Prajit Ramachandran, Ashish Vaswani, Irwan
Dollár. Panoptic feature pyramid networks. In Proc. IEEE
Bello, Anselm Levskaya, and Jon Shlens. Stand-alone
Conf. Comp. Vis. Patt. Recogn., 2019. 1, 3, 6, 7, 10, 12
self-attention in vision models. In Hanna M. Wallach,
[33] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Hugo Larochelle, Alina Beygelzimer, Florence d’Alché-
Imagenet classification with deep convolutional neural net- Buc, Emily B. Fox, and Roman Garnett, editors, Proc. Ad-
works. Proc. Advances in Neural Inf. Process. Syst., 2012. vances in Neural Inf. Process. Syst., 2019. 2, 3
3
[49] Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun.
[34] Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Faster r-cnn: Towards real-time object detection with region
Haffner. Gradient-based learning applied to document recog- proposal networks. In Proc. Advances in Neural Inf. Process.
nition. 1998. 3 Syst., 2015. 1, 3
[35] Xiang Li, Wenhai Wang, Xiaolin Hu, Jun Li, Jinhui Tang, [50] Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-
and Jian Yang. Generalized focal loss v2: Learning reliable net: Convolutional networks for biomedical image segmen-
localization quality estimation for dense object detection. In tation. In International Conference on Medical image com-
Proc. IEEE Conf. Comp. Vis. Patt. Recogn., 2021. 3 puting and computer-assisted intervention, 2015. 3
[36] Xiang Li, Wenhai Wang, Xiaolin Hu, and Jian Yang. Selec- [51] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, San-
tive kernel networks. In Proc. IEEE Conf. Comp. Vis. Patt. jeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy,
Recogn., 2019. 3, 11 Aditya Khosla, Michael Bernstein, et al. Imagenet large
[37] Xiang Li, Wenhai Wang, Lijun Wu, Shuo Chen, Xiaolin Hu, scale visual recognition challenge. Int. J. Comput. Vision,
Jun Li, Jinhui Tang, and Jian Yang. Generalized focal loss: 2015. 3, 6
Learning qualified and distributed bounding boxes for dense [52] Suyash Shetty. Application of convolutional neural net-
object detection. In Proc. Advances in Neural Inf. Process. work for image classification on pascal voc challenge 2012
Syst., 2020. 3 dataset. arXiv preprint arXiv:1607.03785, 2016. 3
[38] Tsung-Yi Lin, Piotr Dollár, Ross Girshick, Kaiming He, [53] Connor Shorten and Taghi M Khoshgoftaar. A survey on
Bharath Hariharan, and Serge Belongie. Feature pyramid image data augmentation for deep learning. Journal of Big
networks for object detection. In Proc. IEEE Conf. Comp. Data, 6(1):1–48, 2019. 3
Vis. Patt. Recogn., 2017. 3, 6 [54] Karen Simonyan and Andrew Zisserman. Very deep con-
[39] Tsung-Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and volutional networks for large-scale image recognition. In
Piotr Dollár. Focal loss for dense object detection. In Proc. Yoshua Bengio and Yann LeCun, editors, Proc. Int. Conf.
IEEE Int. Conf. Comp. Vis., 2017. 1, 2, 3, 6, 7, 8, 10, 12 Learn. Representations, 2015. 1, 3, 4
[55] Peize Sun, Yi Jiang, Enze Xie, Zehuan Yuan, Changhu efficient and accurate end-to-end spotting of arbitrarily-
Wang, and Ping Luo. Onenet: Towards end-to-end one-stage shaped text. IEEE Trans. Pattern Anal. Mach. Intell., 2021.
object detection. arXiv preprint arXiv:2012.05780, 2020. 3 11
[56] Peize Sun, Yi Jiang, Rufeng Zhang, Enze Xie, Jinkun Cao, [70] Xiaolong Wang, Ross Girshick, Abhinav Gupta, and Kaim-
Xinting Hu, Tao Kong, Zehuan Yuan, Changhu Wang, and ing He. Non-local neural networks. In Proc. IEEE Conf.
Ping Luo. Transtrack: Multiple-object tracking with trans- Comp. Vis. Patt. Recogn., 2018. 2, 3, 10
former. arXiv preprint arXiv:2012.15460, 2020. 2 [71] Enze Xie, Peize Sun, Xiaoge Song, Wenhai Wang, Xuebo
[57] Peize Sun, Rufeng Zhang, Yi Jiang, Tao Kong, Chenfeng Liu, Ding Liang, Chunhua Shen, and Ping Luo. Polarmask:
Xu, Wei Zhan, Masayoshi Tomizuka, Lei Li, Zehuan Yuan, Single shot instance segmentation with polar representation.
Changhu Wang, et al. Sparse r-cnn: End-to-end object de- In Proc. IEEE Conf. Comp. Vis. Patt. Recogn., 2020. 3
tection with learnable proposals. In Proc. IEEE Conf. Comp. [72] Enze Xie, Wenjia Wang, Wenhai Wang, Peize Sun, Hang Xu,
Vis. Patt. Recogn., 2021. 3 Ding Liang, and Ping Luo. Segmenting transparent object in
[58] Christian Szegedy, Sergey Ioffe, Vincent Vanhoucke, and the wild with transformer. In Proc. Int. Joint Conf. Artificial
Alexander Alemi. Inception-v4, inception-resnet and the im- Intell., 2021. 2, 9
pact of residual connections on learning. In Proc. AAAI Conf. [73] Saining Xie, Ross Girshick, Piotr Dollár, Zhuowen Tu, and
Artificial Intell., 2017. 3 Kaiming He. Aggregated residual transformations for deep
[59] Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, neural networks. In Proc. IEEE Conf. Comp. Vis. Patt.
Scott Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Recogn., 2017. 1, 2, 3, 6, 7, 8
Vanhoucke, and Andrew Rabinovich. Going deeper with
[74] Fisher Yu and Vladlen Koltun. Multi-scale context aggrega-
convolutions. In Proc. IEEE Conf. Comp. Vis. Patt. Recogn.,
tion by dilated convolutions. In Yoshua Bengio and Yann Le-
2015. 3, 6
Cun, editors, Proc. Int. Conf. Learn. Representations, 2016.
[60] Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jon 7, 11
Shlens, and Zbigniew Wojna. Rethinking the inception ar-
[75] Li Yuan, Yunpeng Chen, Tao Wang, Weihao Yu, Yujun Shi,
chitecture for computer vision. In Proc. IEEE Conf. Comp.
Zihang Jiang, Francis EH Tay, Jiashi Feng, and Shuicheng
Vis. Patt. Recogn., 2016. 3, 6
Yan. Tokens-to-token vit: Training vision transformers
[61] Mingxing Tan and Quoc Le. Efficientnet: Rethinking model
from scratch on imagenet. arXiv preprint arXiv:2101.11986,
scaling for convolutional neural networks. In Proc. Int. Conf.
2021. 7
Mach. Learn., 2019. 11
[76] Sangdoo Yun, Dongyoon Han, Seong Joon Oh, Sanghyuk
[62] Zhi Tian, Chunhua Shen, Hao Chen, and Tong He. Fcos:
Chun, Junsuk Choe, and Youngjoon Yoo. Cutmix: Regular-
Fully convolutional one-stage object detection. In Proc.
ization strategy to train strong classifiers with localizable fea-
IEEE Int. Conf. Comp. Vis., 2019. 3
tures. In Proc. IEEE Conf. Comp. Vis. Patt. Recogn., pages
[63] Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco 6023–6032, 2019. 6
Massa, Alexandre Sablayrolles, and Hervé Jégou. Training
[77] Erwan Zerhouni, Dávid Lányi, Matheus Viana, and Maria
data-efficient image transformers & distillation through at-
Gabrani. Wide residual networks for mitosis detection.
tention. In Proc. Int. Conf. Mach. Learn., 2021. 3, 6, 7
In IEEE International Symposium on Biomedical Imaging,
[64] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszko-
2017. 9
reit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia
Polosukhin. Attention is all you need. In Proc. Advances in [78] Hongyi Zhang, Moustapha Cissé, Yann N. Dauphin, and
Neural Inf. Process. Syst., 2017. 2, 3, 4, 5, 10 David Lopez-Paz. mixup: Beyond empirical risk minimiza-
tion. In Proc. Int. Conf. Learn. Representations, 2018. 6
[65] Wenhai Wang, Xiang Li, Jian Yang, and Tong Lu. Mixed
link networks. Proc. Int. Joint Conf. Artificial Intell., 2018. [79] Hang Zhang, Chongruo Wu, Zhongyue Zhang, Yi Zhu, Zhi
3 Zhang, Haibin Lin, Yue Sun, Tong He, Jonas Mueller, R
[66] Wenhai Wang, Xuebo Liu, Xiaozhong Ji, Enze Xie, Ding Manmatha, et al. Resnest: Split-attention networks. arXiv
Liang, ZhiBo Yang, Tong Lu, Chunhua Shen, and Ping Luo. preprint arXiv:2004.08955, 2020. 11
Ae textspotter: Learning visual and linguistic representation [80] Hengshuang Zhao, Jiaya Jia, and Vladlen Koltun. Exploring
for ambiguous text spotting. In Proc. Eur. Conf. Comp. Vis., self-attention for image recognition. In Proc. IEEE Conf.
2020. 11 Comp. Vis. Patt. Recogn., 2020. 2
[67] Wenhai Wang, Enze Xie, Xiang Li, Deng-Ping Fan, Kaitao [81] Hengshuang Zhao, Jianping Shi, Xiaojuan Qi, Xiaogang
Song, Ding Liang, Tong Lu, Ping Luo, and Ling Shao. Wang, and Jiaya Jia. Pyramid scene parsing network. In
Pvtv2: Improved baselines with pyramid vision transformer. Proc. IEEE Conf. Comp. Vis. Patt. Recogn., 2017. 3
arXiv preprint arXiv:2106.13797, 2021. 10 [82] Zhun Zhong, Liang Zheng, Guoliang Kang, Shaozi Li, and
[68] Wenhai Wang, Enze Xie, Xiang Li, Wenbo Hou, Tong Lu, Yi Yang. Random erasing data augmentation. In Proc. AAAI
Gang Yu, and Shuai Shao. Shape robust text detection with Conf. Artificial Intell., 2020. 6
progressive scale expansion network. In Proc. IEEE Conf. [83] Bolei Zhou, Hang Zhao, Xavier Puig, Sanja Fidler, Adela
Comp. Vis. Patt. Recogn., 2019. 11 Barriuso, and Antonio Torralba. Scene parsing through
[69] Wenhai Wang, Enze Xie, Xiang Li, Xuebo Liu, Ding Liang, ade20k dataset. In Proc. IEEE Conf. Comp. Vis. Patt.
Yang Zhibo, Tong Lu, and Chunhua Shen. Pan++: Towards Recogn., 2017. 2, 7, 9, 10, 12
[84] Xizhou Zhu, Dazhi Cheng, Zheng Zhang, Stephen Lin, and
Jifeng Dai. An empirical study of spatial attention mech-
anisms in deep networks. In Proc. IEEE Int. Conf. Comp.
Vis., 2019. 10
[85] Xizhou Zhu, Weijie Su, Lewei Lu, Bin Li, Xiaogang Wang,
and Jifeng Dai. Deformable DETR: deformable transformers
for end-to-end object detection. In Proc. Int. Conf. Learn.
Representations, 2021. 2, 3