A Simple Pooling-Based Design for Real-Time Salient Object Detection

Jiang-Jiang Liu1∗ Qibin Hou1∗ Ming-Ming Cheng1 † Jiashi Feng2 Jianmin Jiang3
1 2 3
TKLNDST, College of CS, Nankai University NUS Shenzhen University
{[Link], andrewhoux}@[Link]
arXiv:1904.09569v1 [[Link]] 21 Apr 2019

Abstract detail features in multiple scale space.

As pointed out in many previous approaches [9, 28, 44],
We solve the problem of salient object detection by in- because of the pyramid-like structural characteristics of
vestigating how to expand the role of pooling in convolu- CNNs, shallower stages usually have larger spatial sizes
tional neural networks. Based on the U-shape architecture, and keep rich, detailed low-level information while deeper
we first build a global guidance module (GGM) upon the stages contain more high-level semantic knowledge and are
bottom-up pathway, aiming at providing layers at different better at locating the exact places of salient objects. Based
feature levels the location information of potential salient on the aforementioned knowledge, a variety of new archi-
objects. We further design a feature aggregation module tectures [9, 17, 38, 10] for salient object detection have been
(FAM) to make the coarse-level semantic information well designed. Among these approaches, U-shape based struc-
fused with the fine-level features from the top-down path- tures [32, 22] receive the most attentions due to their abil-
way. By adding FAMs after the fusion operations in the top- ity to construct enriched feature maps by building top-down
down pathway, coarse-level features from the GGM can be pathways upon classification networks.
seamlessly merged with features at various scales. These
Despite the good performance achieved by this type of
two pooling-based modules allow the high-level semantic
approaches, there is still a large room for improving it. First,
features to be progressively refined, yielding detail enriched
in the U-shape structure, high-level semantic information
saliency maps. Experiment results show that our proposed
is progressively transmitted to shallower layers, and hence
approach can more accurately locate the salient objects
the location information captured by deeper layers may be
with sharpened details and hence substantially improve
gradually diluted at the same time. Second, as pointed out
the performance compared to the previous state-of-the-arts.
in [47], the receptive field size of a CNN is not propor-
Our approach is fast as well and can run at a speed of more
tional to its layer depth. Existing methods solve the above-
than 30 FPS when processing a 300 × 400 image. Code can
mentioned problems by introducing attention mechanisms
be found at [Link]
[46, 24] into U-shape structures, refining feature maps in
a recurrent way [23, 46, 36], combining multi-scale fea-
ture information [9, 28, 44, 10], or add extra constraints to
1. Introduction
saliency maps like the boundary loss term in [28].
Benefiting from the capability of detecting the most vi- In this paper, different from the methods mentioned
sually distinctive objects from a given image, salient ob- above, we investigate how to solve these problems by ex-
ject detection plays an important role in many computer vi- panding the role of the pooling techniques in U-shape based
sion tasks, such as visual tracking [8], content-aware image architectures. In general, our model consists of two pri-
editing [4], and robot navigation [5]. Traditional methods mary modules on the base of the feature pyramid networks
[11, 25, 14, 31, 2, 12, 41, 3] mostly rely on hand-crafted fea- (FPNs) [22]: a global guidance module (GGM) and a fea-
tures to capture local details and global context separately ture aggregation module (FAM). As shown in Fig. 1, our
or simultaneously, but the lack of high-level semantic infor- GGM composes of a modified version of pyramid pooling
mation restricts their ability to detect the integral salient ob- module (PPM) and a series of global guiding flows (GGFs).
jects in complex scenes. Luckily, convolutional neural net- Unlike [37] which directly plugs PPM into the U-shape net-
works (CNNs) greatly promote the development of salient works, our GGM is an individual module. More specifi-
object detection models because of their capability of ex- cally, the PPM is placed on the top of the backbone to cap-
tracting both high-level semantic information and low-level ture global guidance information (where the salient objects
∗ Indicatesequal contributions. are). By introducing GGFs, high-level semantic informa-
† M.M. Cheng (cmm@[Link]) is the corresponding author. tion collected by PPM can be delivered to feature maps at

1
 F F F R residual block
A Feature Aggregation Module (FAM)

P Pyramid Pooling Module (PPM)

R R R
Illustration of FPN structure
Global Guiding Flows (GGFs)
8×up 4×up 2×up
Global Guidance Module (GGM)

F F F F
optional edge-related paths
A A A A

score map
P feature visualization positions in Fig. 4

F upsample

3×3 conv ＋ upsample

backbone
Figure 1. The overall pipeline of our proposed approach. For clarity, we also place a standard U-shape FPN structure [22] at the top-left
corner. The top part for edge detection is optional.

all pyramid levels, remedying the drawback of U-shape net- to the effective utilization of pooling techniques. PoolNet,
works that top-down signals are gradually diluted. Taking therefore, can be viewed as a baseline to help ease future
into account the fusion problem of the coarse-level feature research in salient object detection.
maps from GGFs with the feature maps at different scales
of the pyramid, we further propose a feature aggregation 2. Related Work
module (FAM), which takes the feature maps after fusion
as input. This module first converts the fused feature maps Recently, benefiting from the powerful feature extraction
into multiple feature spaces to capture local context infor- capability of CNNs, most of the traditional saliency detec-
mation at different scales and then combines the informa- tion methods based on hand-crafted features [3, 12, 20, 31]
tion to weigh the compositions of the fused input feature have been gradually surpassed. Li et al. [18] used the
maps better. multi-scale features extracted from a CNN to compute
the saliency value for each super-pixel. Wang et al. [34]
As both the above modules are based on the pooling adopted two CNNs, aiming at combining local super-pixel
techniques, we call our method PoolNet. To the best of estimation and global proposal searching together, to pro-
our knowledge, this is the first paper that aims at study- duce saliency maps. Zhao et al. [48] presented a multi-
ing how to design various pooling-based modules to assist context deep learning framework which extracts both local
in improving the performance for salient object detection. and global context information by employing two indepen-
As an extension of this work, we also equip our architecture dent CNNs. Lee et al. [6] combined low-level heuristic fea-
with an edge detection branch to further sharpen the details tures, such as color histogram and Gabor responses, with
of salient objects by joint training our model with edge de- high-level features extracted from CNNs. All these meth-
tection. To evaluate the performance of our proposed ap- ods take image patches as the inputs of CNNs and hence
proach, we report results on multiple popular salient ob- are time-consuming. Moreover, they ignore the essential
ject detection benchmarks. Without bells and whistles, our spatial information of the whole input image.
PoolNet surpasses all previous state-of-the-art methods in a To overcome the above problems, more research atten-
large margin. In addition, we conduct a series of ablation tions are put on predicting pixel-wise saliency maps, in-
experiments to let readers better understand the impact of spired by the fully convolutional networks [27]. Wang et
each component in our architecture on the performance and al. [36] generated saliency prior maps using low-level cues
show how joint training with edge detection helps enhance and further exploited it to guide the prediction of saliency
the details of the predicted results. recurrently. Liu et al. [23] proposed a two-stage network
Our network can run at a speed of more than 30 FPS on which produces coarse saliency maps first and then inte-
a single NVIDIA Titan Xp GPU for an input image with grates local context information to refine them recurrently
size 300 × 400. When the edge branch is not incorporated, and hierarchically. Hou et al. [9] introduced short connec-
training only takes less than 6 hours on a training set of tions into multi-scale side outputs to capture fine details.
5,000 images, which is quite faster than most of the pre- Luo et al. [28] and Zhang et al. [44] both advanced the U-
vious methods [24, 43, 28, 44, 45, 9]. This is mainly due shape structures and utilized multiple levels of context in-
formation for accurate detection of salient objects. Zhang
et al. [46] and Liu et al. [24] combined attention mecha-
nisms with U-shape models to guide the feature integration
process. Wang et al. [38] proposed a network to recurrently
locate the salient object and then refine them with local con-
text information. Zhang et al. [43] used a bi-directional
structure to pass messages between multi-level features ex- (a) (b) (c) (d) (e) (f) (g)
tracted by CNNs for better predicting saliency maps. Xiao Figure 2. Visual comparisons for salient object detection with dif-
et al. [39] adopted one network to tailor the distracting re- ferent combinations of our proposed GGM and FAMs. (a) Source
gions first and then used another network for saliency de- image; (b) Ground truth; (c) Results of FPN baseline; (d) Results
tection. of FPN + FAMs; (e) Results of FPN + PPM; (f) Results of FPN +
Our method is quiet different from the above approaches. GGM; (g) Results of FPN + GGM + FAMs.
Instead of exploring new network architectures, we investi-
gate how to apply the simple pooling techniques to CNNs backbone, one of the problems to this type of U-shape archi-
to simultaneously improve the performance and accelerate tectures is that the high-level features will be gradually di-
the running speed. luted when they are transmitted to lower layers. It is shown
in [49, 47] that the empirical receptive fields of CNNs are
3. PoolNet much smaller than the ones in theory especially for deeper
It has been pointed out in [23, 9, 37, 38] that high-level layers, so the receptive fields of the whole networks are not
semantic features are helpful for discovering the specific lo- large enough to capture the global information of the input
cations of salient objects. At the meantime, low- and mid- images. The immediate effect on this is that only parts of
level features are also essential for improving the features the salient objects can be discovered as shown in Fig. 2c.
extracted from deep layers from coarse level to fine level. Regarding the lack of high-level semantic information for
Based on the above knowledge, in this section, we propose fine-level feature maps in the top-down pathway, we intro-
two complementary modules that are capable of accurately duce a global guidance module which contains a modified
capturing the exact positions of salient objects and mean- version of pyramid pooling module (PPM) [47, 37] and a
while sharpening their details. series of global guiding flows (GGFs) to explicitly make
feature maps at each level be aware of the locations of the
3.1. Overall Pipeline salient objects.
We build our architecture based on the feature pyramid To be more specific, the PPM in our GGM consists of
networks (FPNs) [22] which are a type of classic U-shape four sub-branches to capture the context information of the
architectures designed in a bottom-up and top-down man- input images. The first and last sub-branches are respec-
ner as shown at the top-left corner of Fig. 1. Because of the tively an identity mapping layer and a global average pool-
strong ability to combine multi-level features from classifi- ing layer. For the two middle sub-branches, we adopt the
cation networks [7, 33], this type of architectures has been adaptive average pooling layer1 to ensure the output feature
widely adopted in many vision tasks, including salient ob- maps of them are with spatial sizes 3 × 3 and 5 × 5, respec-
ject detection. As shown in Fig. 1, we introduce a global tively. Given the PPM, what we need to do now is how to
guidance module (GGM) which is built upon the top of the guarantee that the guidance information produced by PPM
bottom-up pathway. By aggregating the high-level infor- can be reasonably fused with the feature maps at different
mation extracted by GGM with into feature maps at each levels in the top-down pathway.
feature level, our goal is to explicitly notice the layers at Quite different from the previous work [37] which sim-
different feature levels where salient objects are. After the ply views the PPM as a part of the U-shape structure, our
guidance information from GGM is merged with the fea- GGM is independent of the U-shape structure. By intro-
tures at different levels, we further introduce a feature ag- ducing a series of global guiding flows (identity mappings),
gregation module (FAM) to ensure that feature maps at dif- the high-level semantic information can be easily delivered
ferent scales can be merged seamlessly. In what follows, we to feature maps at various levels (see the green arrows in
describe the structures of the above mentioned two modules Fig. 1). In this way, we explicitly increase the weight of the
and explain their functions in detail. global guidance information in each part of the top-down
pathway to make sure that the location information will not
3.2. Global Guidance Module be diluted when building FPNs.
FPNs provide a classic architecture for combining multi- To better demonstrate the effectiveness of our GGM, we
level features from the classification backbone. However, 1 [Link]

because the top-down pathway is built upon the bottom-up adaptiveavgpool2d

 GGFs GGFs
8× up FAM conv conv
FAM

8× down
a b c d
4× up A F A F
4× down

2× down 2× up
sum 3×3 conv
avg pool 3×3 conv

Figure 3. Detailed illustration of our feature aggregation module

(FAM). It comprises four sub-branches, each of which works in
an individual scale space. After upsampling, all sub-branches are
combined and then fed into a convolutional layer. (a) (b) (c) (d)

show some visual comparisons. As depicted in Fig. 2c, we

show some saliency maps produced by a VGGNet version
of FPNs2 . It can be easily found that with only the FPN
backbone, it is difficult to locate salient objects for some Figure 4. Visualizing feature maps around FAMs. Feature maps
complex scenes. There are also some results in which only shown on the left are from models with FAMs, while feature maps
parts of the salient object are detected. However, when our displayed on the right are from models replacing FAMs with two
convolution layers. The last row are source images and the cor-
GGM is incorporated, the quality of the resulting saliency
responding ground-truth annotations. (a-d) are visualizations of
maps are greatly improved. As shown in Fig. 2f, salient feature maps at different places. As can be seen, when our FAMs
objects can be precisely discovered, which demonstrates the are used, feature maps after FAMs can more precisely capture the
importance of GGM. location and detail information of salient objects (Column a), com-
pared to those after two convolution layers (Column c).
3.3. Feature Aggregation Module
The utilization of our GGM allows the global guid-
ance information to be delivered to feature maps at dif- average pooling layers with varying downsampling rates.
ferent pyramid levels. However, a new question that de- The upsampled feature maps from different sub-branches
serves asking is how to make the coarse-level feature maps are then merged together, followed by a 3 × 3 convolutional
from GGM seamlessly merged with the feature maps at layer.
different scales of the pyramid. Taking the VGGNet ver- Generally speaking, our FAM has two advantages. First,
sion of FPNs as an example, feature maps corresponding it assists our model in reducing the aliasing effect of upsam-
to C = {C2 , C3 , C4 , C5 } in the pyramid have downsam- pling, especially when the upsampling rate is large (e.g. , 8).
pling rates of {2, 4, 8, 16} compared to the size of the input In addition, it allows each spatial location to view the local
image, respectively. In the original top-down pathway of context at different scale spaces, further enlarging the recep-
FPNs, feature maps with coarser resolutions are upsampled tive field of the whole network. To the best of our knowl-
by a factor of 2. Therefore, adding a convolutional layer edge, this is the first work revealing that FAMs are helpful
with kernel size 3 × 3 after the merging operation can effec- for reducing the aliasing effect of upsampling.
tively reduce the aliasing effect of upsampling. However, To verify the effectiveness of our proposed FAMs, we
our GGFs need larger upsampling rates (e.g. , 8). It is es- visualize the feature maps near the FAMs in Fig. 4. By
sential to bridge the big gaps between GGFs and the feature comparing the left part (w/ FAMs) with the right part (w/o
maps of different scales effectively and efficiently. FAMs), feature maps after FAMs (Column a) can better cap-
To this end, we propose a series of feature aggregation ture the salient objects than those without FAMs (Column
modules, each of which contains four sub-branches as illus- c). In addition to visualizing the intermediate feature maps,
trated in Fig. 3. In the forward pass, the input feature map we also show some saliency maps produced by models with
is first converted to different scale spaces by feeding it into different settings in Fig. 2. By comparing the results in Col-
2
umn f (w/o FAMs) and Column g (w/ FAMs), it can be eas-
Similarly to [22], we use the feature maps outputted by conv2, conv3,
conv4, conv5 which are denoted by {C2 , C3 , C4 , C5 } to build the feature
ily found that introducing FAM multiple times allows our
pyramid upon the VGGNet [33]. The channel numbers corresponding to network to better sharpen the details of the salient objects.
{C2 , C3 , C4 , C5 } are set to {128, 256, 512, 512}, respectively. This phenomenon is especially clear by observing the sec-
 tail information of the object boundaries. For example, as
demoed in Column c of Fig. 5, the resulting saliency maps
and boundary maps are still ambiguous for scenes with low
contrast between the foreground and background. The rea-
son for this might be that the ground-truth edge maps de-
rived from salient objects still lack most of the detailed in-
formation of salient objects. They just tell us where the
outermost boundaries of salient objects are, especially for
cases where there are overlaps between salient objects.
(a) (b) (c) (d) (e) (f) Taking the aforementioned argument into account, we at-
Figure 5. Visual results by joint training with edge detection. (a) tempt to perform joint training with the edge detection task
Source image; (b) Ground truth; (c-d) Edge maps and saliency using the same edge detection dataset [1, 29] as in [26].
maps using the boundaries of salient objects as ground truths of the During training, images from the salient object detection
edge branch; (e-f) Edge maps and saliency maps by joint training dataset and the edge detection dataset are inputted alterna-
with the edge dataset [1, 29]. By comparing the results in Column tively. As can be seen in Fig. 5, joint training with the edge
d and Column f, we can easily observe that joint training with
detection task greatly improves the details of the detected
high-quality edge datasets substantially improves the details of the
detected salient objects.
salient objects. We will provide more quantitative analysis
in our experiment section.
ond row of Fig. 2. All the aforementioned discussions verify
5. Experimental Results
the significant effect of our FAMs on better fusing feature
maps at different scales. In our experiment section, we will In this section, we first describe the experiment setups,
give more numerical results. including the implementation details, the used datasets and
the evaluation metrics. We then conduct a series of abla-
4. Joint Training with Edge Detection tion studies to demonstrate the impact of each component
of our proposed approach on the performance. At last, we
The architecture described in Sec. 3 has already sur- report the performance of our approach and compare it with
passed all previous state-of-the-art single-model results on previous state-of-the-art methods.
multiple popular salient object detection benchmarks. De-
spite so, by observing the resulting saliency maps produced 5.1. Experiment Setup
by our model, we find out that many inaccurate (incomplete
Implementation Details. The proposed framework is im-
or over-predicted) predictions are caused by unclear object
plemented based on the PyTorch repository3 . All the exper-
boundaries.
iments are performed using the Adam [13] optimizer with
At first, we attempt to solve this problem by adding
a weight decay of 5e-4 and an initial learning rate of 5e-
an extra prediction branch built upon the architecture pre-
5 which is divided by 10 after 15 epochs. Our network
sented in Sec. 3 to estimate the boundaries of the salient
is trained for 24 epochs in total. The backbone parame-
objects. The detailed structure can be found on the top side
ters of our network (e.g. , VGG-16 [33] and ResNet-50 [7])
of Fig. 1. We add three residual blocks [7] after the FAMs
are initialized with the corresponding models pretrained on
at three feature levels in the top-down pathway, which are
the ImageNet dataset [16] and the rest ones are randomly
used for information transformation. These residual blocks
initialized. By default, our ablation experiments are per-
are similar to the design in [7] and have channel numbers of
formed based on the VGG-16 backbone and the union set
{128, 256, 512} from the fine level to the coarse level. As
of MSRA-B [25] and HKU-IS [18] datasets as done in [17]
done in [26], each residual block is then followed by a 16-
unless special explanations. We only use the simple random
channel 3 × 3 convolutional layer for feature compression
horizontal flipping for data augmentation. In both training
plus a one-channel 1×1 convolutional layer for edge predic-
and testing, the sizes of the input images are kept unchanged
tion. We also concatenate these three 16-channel 3 × 3 con-
as done in [9].
volutional layers and feed them to three consecutive 3 × 3
convolutional layers with 48 channels to transmit the cap- Datasets & Loss Functions. To evaluate the performance
tured edge information to the salient object detection branch of our proposed framework, we conduct experiments on 6
for detail enhancement. commonly used datasets, including ECSSD [41], PASCAL-
Similar to [17], during the training phase, we use the S [21], DUT-OMRON [42], HKU-IS [18], SOD [30] and
boundaries of the salient objects as our ground truths for DUTS [35]. Sometimes, for convenience, we use the ini-
joint training. However, this procedure does not bring us tials of the datasets as their abbreviations if there is no ex-
any performance gain, and some results are still short of de- 3 [Link]
 Image GT Ours PiCANet[24] DGRL[38] PAGR[46] SRM[37] Amulet[44] DSS[9] MSR[17] DCL[19]
Figure 6. Qualitative comparisons to previous state-of-the-art methods. Obviously, compared to other methods, our approach is capable of
not only locating the integral salient objects but also refining the details of the detected salient objects. This makes our resulting saliency
map very close to the ground-truth annotations.

plicit conflict. We use standard binary cross entropy loss for GGM + FAMs DUT-O [42] SOD [30]
No.
salient object detection and balanced binary cross entropy
PPM GGFs FAMs MaxF ↑ MAE ↓ MaxF ↑ MAE ↓
loss [40] for edge detection.
1 0.770 0.076 0.838 0.124
Evaluation Criteria. We evaluate the performance of our 2 3 0.783 0.071 0.847 0.125
approach and other methods using three widely-used met- 3 3 0.772 0.076 0.843 0.121
rics: precision-recall (PR) curves, F-measure score, and 4 3 3 0.790 0.069 0.855 0.120
mean absolute error (MAE). F-measure, denoted as Fβ , is
5 3 0.798 0.065 0.852 0.118
an overall performance measurement and is computed by
6 3 3 3 0.806 0.063 0.861 0.117
the weighted harmonic mean of the precision and recall:
Table 1. Ablation analysis for the proposed architecture on two
(1 + β 2 ) × P recision × Recall popular datasets. All experiments are based on the VGG-16 back-
Fβ = (1)
β 2 × P recision + Recall bone and trained on the union set of MSRA-B [25] and HKU-IS
[18]. By default, our baseline is the VGG-16 version of FPN [22].
where β 2 is set to 0.3 as done in previous work to weight As can be observed, each component in our architecture plays an
precision more than recall. The MAE score indicates how important role and contributes to the performance. Best result in
similar a saliency map S is compared to the ground truth G: each column are highlighted in red.

1 XW X H GGM and FAMs, all other configurations are the same. Ta-
M AE = |S(x, y) − G(x, y)| (2) ble 1 shows the performance on two challenging datasets:
W × H x=1 y=1
DUT-O and SOD. The corresponding visual comparisons
can be found in Fig. 2.
where W and H denote the width and height of S, respec-
tively. • GGM Only. The addition of GGM (the 4th row in
Table 1) gives performance gains in terms of both F-
5.2. Ablation Studies measure and MAE on the two datasets over the FPN
In this subsection, we investigate the effectiveness of our baseline. The global guidance information produced
proposed GGM and FAMs first. Then, we conduct more by GGM allows our network to focus more on the in-
experiments on the configurations of our GGM and FAMs. tegrity of salient objects, greatly improving the quality
Finally, we show the effect of joint training with edge de- of the resulting saliency maps. Therefore, the details
tection on the performance. of the salient objects can be sharpened, which might
be wrongly estimated as background for models with
Effectiveness of GGM and FAMs. To demonstrate the ef- limited receptive fields (e.g. , the last row in Fig. 2).
fectiveness of our proposed GGM and FAMs, we conduct
ablation experiments based on the FPN baseline with the • FAMs Only. Simply embedding FAMs (the 5th row
VGG-16 backbone. Except for different combinations of of Table 1) into the FPN baseline as shown in Fig. 1
 DCL MSR DSS Amulet SRM PAGR DGRL PiCANet Ours
1 1 1

0.9 0.9 0.9

0.8 0.8 0.8

0.7 0.7 0.7

0.6 0.6 0.6

0 0.2 0.4 0.6 0.8 1 0 0.2 0.4 0.6 0.8 1 0 0.2 0.4 0.6 0.8 1
(a) PASCAL-S [21] (b) HKU-IS [18] (c) DUTS-TE [35]
Figure 7. Precision (vertical axis) recall (horizontal axis) curves on three popular salient object datasets.

also helps improve the performance on both F-measure PASCAL-S [21] DUT-O [42] SOD [30]
and MAE scores on the same two datasets. This might
Settings MaxF MAE MaxF MAE MaxF MAE
be because the pooling operations inside FAMs also
enlarge the receptive field of the whole network com- Baseline (B) 0.838 0.093 0.806 0.063 0.861 0.117
pared to the baseline, and the FPN baseline still needs B + SalEdge 0.835 0.096 0.805 0.063 0.863 0.120
to merge feature maps from different levels, which in- B + StdEdge 0.849 0.077 0.808 0.059 0.872 0.105
dicates the effectiveness of our FAMs for solving the Table 2. Ablation analysis of our approach when different kinds
aliasing effect of upsampling. of boundaries are used. The baseline here refers to the VGG-16
version of FPN plus GGM + FAMs. We also use the combination
• GGM & FAMs. By introducing both GGM and FAMs of MSRA-B [25] and HKU-IS [18] as the training set. ‘SalEdge’
into the baseline (the last row of Table 1), the perfor- refers to the boundaries of salient objects and ‘StdEdge’ refers to
mance compared to the above two cases can be further the standard datasets for edge detection, which include BSDS500
enhanced on both F-measure and MAE scores. This [1] and PASCAL VOC Context [29] as done in [26, 15].
phenomenon demonstrates that our GGM and FAM are
two complementary modules. The utilization of them datasets especially on the MAE metric. This indicates that
allows our approach to possess the strong capability of involving detailed edge information is helpful for salient ob-
accurately discovering the salient objects and refining ject detection.
the details as illustrated in Fig. 2. More qualitative re- 5.3. Comparisons to the State-of-the-Arts
sults can be found in Fig. 6 as well.
In this section, we compare our proposed PoolNet with
Configuration of GGM. To have a better understanding of 13 previous state-of-the-art methods, including DCL [19],
the constitution of our proposed GGM, we perform two ab- RFCN [36], DHS [23], MSR [17], DSS [9], NLDF [28],
lation experiments, which correspond to the 2nd and 3rd UCF [45], Amulet [44], GearNet[10], PAGR [46], Pi-
rows of Table 1, respectively. We alternatively remove one CANet [24], SRM [37], and DGRL [38]. For fair com-
of the PPM and GGFs while keeping the other one un- parisons, the saliency maps of these methods are gener-
changed. As can be seen, both operations make the per- ated by the original code released by the authors or directly
formance decline compared to the results with both of them provided by them. Moreover, all results are directly from
considered (the 4th row). These numerical results indicate single-model test without relying on any post-processing
that both PPM and GGFs play an important role in our tools and all the predicted saliency maps are evaluated with
GGM. The absence of any one of them is harmful for the the same evaluation code.
performance of our approach. Quantitative Comparisons. Quantitative results are listed
The Impact of Joint Training. To further improve the in Table 3. We consider both VGG-16 [33] and ResNet-50
quality of saliency maps produced by our approach, we at- [7] as our backbones and show results on both of them. Ad-
tempt to combine edge detection with salient object detec- ditionally, we also conduct experiments on different train-
tion in a joint training manner. In Table 2, we list the results ing sets to eliminate the potential performance fluctuation.
when two kinds of boundary information are considered. From Table 3, we can observe that our PoolNet surpasses al-
As can be seen, using the boundaries of salient objects as most all previous state-of-the-art results on all datasets with
supervision results in no improvement while using standard the same backbone and training set. Average speed (FPS)
boundaries can greatly boost the performance on all three comparisons among different methods (tested in the same
 Training ECSSD [41] PASCAL-S [21] DUT-O [42] HKU-IS [18] SOD [30] DUTS-TE [35]
Model #Images Dataset MaxF ↑ MAE ↓ MaxF ↑ MAE ↓ MaxF ↑ MAE ↓ MaxF ↑ MAE ↓ MaxF ↑ MAE ↓ MaxF ↑ MAE ↓
VGG-16 backbone
DCL [19] 2,500 MB 0.896 0.080 0.805 0.115 0.733 0.094 0.893 0.063 0.831 0.131 0.786 0.081
RFCN [36] 10,000 MK 0.898 0.097 0.827 0.118 0.747 0.094 0.895 0.079 0.805 0.161 0.786 0.090
DHS [23] 9,500 MK+DTO 0.905 0.062 0.825 0.092 - - 0.892 0.052 0.823 0.128 0.815 0.065
MSR [17] 5,000 MB + H 0.903 0.059 0.839 0.083 0.790 0.073 0.907 0.043 0.841 0.111 0.824 0.062
DSS [9] 2,500 MB 0.906 0.064 0.821 0.101 0.760 0.074 0.900 0.050 0.834 0.125 0.813 0.065
NLDF [28] 3,000 MB 0.903 0.065 0.822 0.098 0.753 0.079 0.902 0.048 0.837 0.123 0.816 0.065
UCF [45] 10,000 MK 0.908 0.080 0.820 0.127 0.735 0.131 0.888 0.073 0.798 0.164 0.771 0.116
Amulet [44] 10,000 MK 0.911 0.062 0.826 0.092 0.737 0.083 0.889 0.052 0.799 0.146 0.773 0.075
GearNet[10] 5,000 MB + H 0.923 0.055 - - 0.790 0.068 0.934 0.034 0.853 0.117 - -
PAGR [46] 10,553 DTS 0.924 0.064 0.847 0.089 0.771 0.071 0.919 0.047 - - 0.854 0.055
PiCANet [24] 10,553 DTS 0.930 0.049 0.858 0.078 0.815 0.067 0.921 0.042 0.863 0.102 0.855 0.053
PoolNet (Ours) 2,500 MB 0.918 0.057 0.828 0.098 0.783 0.065 0.908 0.044 0.846 0.124 0.819 0.062
PoolNet (Ours) 5,000 MB + H 0.930 0.053 0.838 0.093 0.806 0.063 0.936 0.032 0.861 0.118 0.855 0.053
PoolNet (Ours) 10,553 DTS 0.936 0.047 0.857 0.078 0.817 0.058 0.928 0.035 0.859 0.115 0.876 0.043
PoolNet† (Ours) 10,553 DTS 0.937 0.044 0.865 0.072 0.821 0.056 0.931 0.033 0.866 0.105 0.880 0.041
ResNet-50 backbone
SRM [37] 10,553 DTS 0.916 0.056 0.838 0.084 0.769 0.069 0.906 0.046 0.840 0.126 0.826 0.058
DGRL [38] 10,553 DTS 0.921 0.043 0.844 0.072 0.774 0.062 0.910 0.036 0.843 0.103 0.828 0.049
PiCANet [24] 10,553 DTS 0.932 0.048 0.864 0.075 0.820 0.064 0.920 0.044 0.861 0.103 0.863 0.050
PoolNet (Ours) 10,553 DTS 0.940 0.042 0.863 0.075 0.830 0.055 0.934 0.032 0.867 0.100 0.886 0.040
PoolNet† (Ours) 10,553 DTS 0.945 0.038 0.880 0.065 0.833 0.053 0.935 0.030 0.882 0.102 0.892 0.036
MB: MSRA-B [25], MK: MSRA10K [3], DTO: DUT-OMRON [42], H: HKU-IS [18], DTS: DUTS-TR [35].
Table 3. Quantitative salient object detection results on 6 widely used datasets. The best results with different backbones are highlighted
in blue and red, respectively. † : joint training with edge detection. As can be seen, our approach achieves the best results on nearly all
datasets in terms of F-measure and MAE.

Ours PiCANet [24] DGRL [38] SRM [37] Amulet [44] tain their sharp boundaries in almost all circumstances.
Size 400 × 300 224 × 224 384 × 384 353 × 353 256 × 256
FPS 32 7 8 14 16
6. Conclusion
UCF [45] NLDF [28] DSS [9] MSR [17] DHS [23]
Size 224 × 224 400 × 300 400 × 300 400 × 300 224 × 224 In this paper, we explore the potentials of pooling on
FPS 23 12 12 2 23 salient object detection by designing two simple pooling-
Table 4. Average speed (FPS) comparisons between our approach based modules: global guidance module (GGM) and feature
(ResNet-50, w/ edge) and the previous state-of-the-art methods. aggregation module (FAM). By plugging them into the FPN
architecture, we show that our proposed PoolNet can sur-
environment) are also reported in Table 4. Obviously, our pass all previous state-of-the-art approaches on six widely-
approach runs in real time and faster than other methods. used salient object detection benchmarks. Furthermore, we
PR Curves. Other than numerical results, we also show also reveal that joint training our network with the standard
the PR curves on three datasets as shown in Fig. 7. As edge detection task in an end-to-end learning manner can
can be seen, the PR curves by our approach (red ones) are greatly enhance the details of the detected salient objects.
especially outstanding compared to all other previous ap- Our modules are independent of network architectures and
proaches. As the recall score approaches 1, our precision hence can be flexibly applied to any pyramid-based models.
score is much higher than other methods. This phenomenon These directions also provide promising ways to improve
reveals that the false positives in our saliency map are low. the quality of saliency maps.
Visual Comparisons. To further explain the advantages of
our approach, we show some qualitative results in Fig. 6. Acknowledgements. This research was supported by
From top to bottom, the images correspond to scenes with NSFC (61620106008, 61572264), the national youth tal-
transparent objects, small objects, large objects, complex ent support program, Tianjin Natural Science Foundation
texture, and low contrast between foreground and back- (17JCJQJC43700, 18ZXZNGX00110) and the Fundamen-
ground, respectively. It can be easily seen that our approach tal Research Funds for the Central Universities (Nankai
can not only highlight the right salient objects but also main- University, NO. 63191501).
References [18] Guanbin Li and Yizhou Yu. Visual saliency based on multi-
scale deep features. In CVPR, pages 5455–5463, 2015. 2, 5,
[1] Pablo Arbelaez, Michael Maire, Charless Fowlkes, and Ji- 6, 7, 8
tendra Malik. Contour detection and hierarchical image seg- [19] Guanbin Li and Yizhou Yu. Deep contrast learning for salient
mentation. IEEE TPAMI, 33(5):898–916, 2011. 5, 7 object detection. In CVPR, 2016. 6, 7, 8
[2] Ali Borji and Laurent Itti. Exploiting local and global patch [20] Xiaohui Li, Huchuan Lu, Lihe Zhang, Xiang Ruan, and
rarities for saliency detection. In CVPR, pages 478–485, Ming-Hsuan Yang. Saliency detection via dense and sparse
2012. 1 reconstruction. In ICCV, pages 2976–2983, 2013. 2
[3] Ming Cheng, Niloy J Mitra, Xumin Huang, Philip HS Torr, [21] Yin Li, Xiaodi Hou, Christof Koch, James M Rehg, and
and Song Hu. Global contrast based salient region detection. Alan L Yuille. The secrets of salient object segmentation.
IEEE TPAMI, 2015. 1, 2, 8 In CVPR, pages 280–287, 2014. 5, 7, 8
[4] Ming-Ming Cheng, Fang-Lue Zhang, Niloy J Mitra, Xi- [22] Tsung-Yi Lin, Piotr Dollár, Ross B Girshick, Kaiming He,
aolei Huang, and Shi-Min Hu. Repfinder: finding approx- Bharath Hariharan, and Serge J Belongie. Feature pyramid
imately repeated scene elements for image editing. ACM networks for object detection. In CVPR, 2017. 1, 2, 3, 4, 6
TOG, 29(4):83, 2010. 1 [23] Nian Liu and Junwei Han. Dhsnet: Deep hierarchical
[5] Celine Craye, David Filliat, and Jean-François Goudou. En- saliency network for salient object detection. In CVPR, 2016.
vironment exploration for object-based visual saliency learn- 1, 2, 3, 7, 8
ing. In ICRA, pages 2303–2309, 2016. 1 [24] Nian Liu, Junwei Han, and Ming-Hsuan Yang. Picanet:
[6] Lee Gayoung, Tai Yu-Wing, and Kim Junmo. Deep saliency Learning pixel-wise contextual attention for saliency detec-
with encoded low level distance map and high level features. tion. In CVPR, pages 3089–3098, 2018. 1, 2, 3, 6, 7, 8
In CVPR, 2016. 2 [25] Tie Liu, Zejian Yuan, Jian Sun, Jingdong Wang, Nanning
[7] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Zheng, Xiaoou Tang, and Heung-Yeung Shum. Learning to
Deep residual learning for image recognition. In CVPR, detect a salient object. IEEE TPAMI, 33(2):353–367, 2011.
2016. 3, 5, 7 1, 5, 6, 7, 8
[8] Seunghoon Hong, Tackgeun You, Suha Kwak, and Bohyung [26] Yun Liu, Ming-Ming Cheng, Xiaowei Hu, Kai Wang, and
Han. Online tracking by learning discriminative saliency Xiang Bai. Richer convolutional features for edge detection.
map with convolutional neural network. In ICML, pages In CVPR, 2017. 5, 7
597–606, 2015. 1 [27] Jonathan Long, Evan Shelhamer, and Trevor Darrell. Fully
[9] Qibin Hou, Ming-Ming Cheng, Xiaowei Hu, Ali Borji, convolutional networks for semantic segmentation. In
Zhuowen Tu, and Philip Torr. Deeply supervised salient CVPR, pages 3431–3440, 2015. 2
object detection with short connections. IEEE TPAMI, [28] Zhiming Luo, Akshaya Kumar Mishra, Andrew Achkar,
41(4):815–828, 2019. 1, 2, 3, 5, 6, 7, 8 Justin A Eichel, Shaozi Li, and Pierre-Marc Jodoin. Non-
[10] Qibin Hou, Jiang-Jiang Liu, Ming-Ming Cheng, Ali Borji, local deep features for salient object detection. In CVPR,
and Philip HS Torr. Three birds one stone: A unified frame- 2017. 1, 2, 7, 8
work for salient object segmentation, edge detection and [29] Roozbeh Mottaghi, Xianjie Chen, Xiaobai Liu, Nam-Gyu
skeleton extraction. arXiv preprint arXiv:1803.09860, 2018. Cho, Seong-Whan Lee, Sanja Fidler, Raquel Urtasun, and
1, 7, 8 Alan Yuille. The role of context for object detection and se-
[11] Laurent Itti, Christof Koch, and Ernst Niebur. A model mantic segmentation in the wild. In CVPR, pages 891–898,
of saliency-based visual attention for rapid scene analysis. 2014. 5, 7
IEEE TPAMI, 20(11):1254–1259, 1998. 1 [30] Vida Movahedi and James H Elder. Design and perceptual
[12] Huaizu Jiang, Jingdong Wang, Zejian Yuan, Yang Wu, Nan- validation of performance measures for salient object seg-
ning Zheng, and Shipeng Li. Salient object detection: A dis- mentation. In CVPR, pages 49–56, 2010. 5, 6, 7, 8
criminative regional feature integration approach. In CVPR, [31] Federico Perazzi, Philipp Krähenbühl, Yael Pritch, and
pages 2083–2090, 2013. 1, 2 Alexander Hornung. Saliency filters: Contrast based filtering
[13] Diederik P Kingma and Jimmy Ba. Adam: A method for for salient region detection. In CVPR, pages 733–740, 2012.
stochastic optimization. In ICLR, 2015. 5 1, 2
[14] Dominik A Klein and Simone Frintrop. Center-surround di- [32] Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-
vergence of feature statistics for salient object detection. In net: Convolutional networks for biomedical image segmen-
ICCV, 2011. 1 tation. In International Conference on Medical image com-
[15] Iasonas Kokkinos. Pushing the boundaries of bound- puting and computer-assisted intervention, pages 234–241,
ary detection using deep learning. arXiv preprint 2015. 1
arXiv:1511.07386, 2015. 7 [33] Karen Simonyan and Andrew Zisserman. Very deep convo-
[16] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. lutional networks for large-scale image recognition. In ICLR,
Imagenet classification with deep convolutional neural net- 2015. 3, 4, 5, 7
works. In NIPS, 2012. 5 [34] Lijun Wang, Huchuan Lu, Xiang Ruan, and Ming-Hsuan
[17] Guanbin Li, Yuan Xie, Liang Lin, and Yizhou Yu. Instance- Yang. Deep networks for saliency detection via local estima-
level salient object segmentation. In CVPR, 2017. 1, 5, 6, 7, tion and global search. In CVPR, pages 3183–3192, 2015.
8 2
[35] Lijun Wang, Huchuan Lu, Yifan Wang, Mengyang Feng,
Dong Wang, Baocai Yin, and Xiang Ruan. Learning to de-
tect salient objects with image-level supervision. In CVPR,
pages 136–145, 2017. 5, 7, 8
[36] Linzhao Wang, Lijun Wang, Huchuan Lu, Pingping Zhang,
and Xiang Ruan. Saliency detection with recurrent fully con-
volutional networks. In ECCV, 2016. 1, 2, 7, 8
[37] Tiantian Wang, Ali Borji, Lihe Zhang, Pingping Zhang, and
Huchuan Lu. A stagewise refinement model for detecting
salient objects in images. In ICCV, pages 4019–4028, 2017.
1, 3, 6, 7, 8
[38] Tiantian Wang, Lihe Zhang, Shuo Wang, Huchuan Lu, Gang
Yang, Xiang Ruan, and Ali Borji. Detect globally, refine
locally: A novel approach to saliency detection. In CVPR,
pages 3127–3135, 2018. 1, 3, 6, 7, 8
[39] Huaxin Xiao, Jiashi Feng, Yunchao Wei, Maojun Zhang, and
Shuicheng Yan. Deep salient object detection with dense
connections and distraction diagnosis. IEEE Transactions
on Multimedia, 2018. 3
[40] Saining Xie and Zhuowen Tu. Holistically-nested edge de-
tection. In ICCV, pages 1395–1403, 2015. 6
[41] Qiong Yan, Li Xu, Jianping Shi, and Jiaya Jia. Hierarchical
saliency detection. In CVPR, pages 1155–1162, 2013. 1, 5,
8
[42] Chuan Yang, Lihe Zhang, Huchuan Lu, Xiang Ruan, and
Ming-Hsuan Yang. Saliency detection via graph-based man-
ifold ranking. In CVPR, pages 3166–3173, 2013. 5, 6, 7,
8
[43] Lu Zhang, Ju Dai, Huchuan Lu, You He, and Gang Wang. A
bi-directional message passing model for salient object de-
tection. In CVPR, pages 1741–1750, 2018. 2, 3
[44] Pingping Zhang, Dong Wang, Huchuan Lu, Hongyu Wang,
and Xiang Ruan. Amulet: Aggregating multi-level convolu-
tional features for salient object detection. In ICCV, 2017. 1,
2, 6, 7, 8
[45] Pingping Zhang, Dong Wang, Huchuan Lu, Hongyu Wang,
and Baocai Yin. Learning uncertain convolutional features
for accurate saliency detection. In ICCV, 2017. 2, 7, 8
[46] Xiaoning Zhang, Tiantian Wang, Jinqing Qi, Huchuan Lu,
and Gang Wang. Progressive attention guided recurrent net-
work for salient object detection. In CVPR, pages 714–722,
2018. 1, 3, 6, 7, 8
[47] Hengshuang Zhao, Jianping Shi, Xiaojuan Qi, Xiaogang
Wang, and Jiaya Jia. Pyramid scene parsing network. In
CVPR, 2017. 1, 3
[48] Rui Zhao, Wanli Ouyang, Hongsheng Li, and Xiaogang
Wang. Saliency detection by multi-context deep learning.
In CVPR, pages 1265–1274, 2015. 2
[49] Bolei Zhou, Aditya Khosla, Agata Lapedriza, Aude Oliva,
and Antonio Torralba. Object detectors emerge in deep scene
cnns. In ICLR, 2015. 3