Sparse Cross-scale Attention Network for Efficient LiDAR Panoptic Segmentation

Shuangjie Xu1‡ , Rui Wan2 , Maosheng Ye1 , Xiaoyi Zou2 , Tongyi Cao2
1 2
The Hong Kong University of Science and Technology [Link]
{sxubj, myeag}@[Link], {ruiwan, xiaoyizou, tongyicao}@[Link]

65
arXiv:2201.05972v1 [[Link]] 16 Jan 2022

SCAN
60
Abstract
Panoptic-PolarNet
55

PQ Performance
Two major challenges of 3D LiDAR Panoptic Segmentation
(PS) are that point clouds of an object are surface-aggregated Panoptic-
50
and thus hard to model the long-range dependency especially PolarNet-mini
KPConv/
for large instances, and that objects are too close to sepa- PointPillars
rate each other. Recent literature addresses these problems by 45
time-consuming grouping processes such as dual-clustering,
mean-shift offsets, etc., or by bird-eye-view (BEV) dense 40 RangeNet++/ LPSAD
centroid representation that downplays geometry. However, PointPillars
the long-range geometry relationship has not been sufficiently 35
0 5 10 15 20
modeled by local feature learning from the above methods. Runtime (Hz)
To this end, we present SCAN, a novel sparse cross-scale at-
tention network to first align multi-scale sparse features with Figure 1: PQ performance vs. inference speed (Hz) in Se-
global voxel-encoded attention to capture the long-range re- manticKITTI panoptic task. The blue vertical line represents
lationship of instance context, which can boost the regres- the boundary of real-time runtime. The proposed SCAN
sion accuracy of the over-segmented large objects. For the
surface-aggregated points, SCAN adopts a novel sparse class-
achieves the state-of-the-art performance.
agnostic representation of instance centroids, which can not
only maintain the sparsity of aligned features to solve the
under-segmentation on small objects, but also reduce the due to similarity in both physical metric space and feature
computation amount of the network through sparse convo- space (Fig. 2 c). Therefore, how to correctly group points
lution. Our method outperforms previous methods by a large that belong to individual objects through efficient and effec-
margin in the SemanticKITTI dataset for the challenging 3D tive feature learning becomes a crucial problem.
PS task, achieving 1st place with a real-time inference speed.
To group the surface-aggregated point clouds, many re-
cent literature (Yang et al. 2019; Liu et al. 2020; Engelmann
Introduction et al. 2020) adopts the two-stage framework to first pro-
pose bounding boxes and then segment instances. To kick
3D scene understanding using point clouds has been an over the traces of bounding boxes, most approaches (Wang
essential and challenging task for many robotics applica- et al. 2019; Lahoud et al. 2019) adopt clustering algorithms.
tions, including autonomous driving systems. One of the key VoteNet (Qi et al. 2019) employs voting to offer broader
tasks in 3D scene understanding is 3D Panoptic segmenta- coverage of “good” seed points. PointGroup (Jiang et al.
tion (PS), with two sub-tasks, semantic and instance seg- 2020) proposes a dual-clustering method to refine inaccurate
mentation. The semantic segmentation task aims to attach offset predictions around object boundaries. DS-Net (Hong
semantic information at the point level, while the instance et al. 2020) utilizes Mean Shift with kernel functions of
segmentation task intends to identify individual countable learned bandwidths to cluster shifted points. Such a clus-
objects. Point clouds have sparse, unordered, and irregular- tering process requires high time consumption. More impor-
sampled natures and aggregate only on the surface of ob- tantly, these methods use the center offset as the only guide
jects. Such natures pose two main challenges for 3D PS: (1) to instance clustering, making it hard to capture fine-grained
The surface-aggregated points are far from their object cen- long-range geometry relationships and thus resulting in po-
troids, leading to false segmentation of big objects (Fig. 2 a, tential over-segmentation of large objects.
b); (2) Closely-distributed small objects are falsely merged
Due to the increasing demand for real-time deployment in
Copyright © 2022, Association for the Advancement of Artificial the industry and academia, plenty of research works focus
Intelligence ([Link]). All rights reserved. on efficient LiDAR 3D PS with range or BEV representa-
‡
The work was done during an internship at [Link]. tions. LPSAD (Milioto et al. 2020) and EfficientLPS (Sirohi
 mance among the published papers with a real-time speed.
Our main contributions are summarized as follows:
• We present SCAN, a novel sparse cross-scale attention
network to first address the surface-aggregated problem
by our cross-scale global attention module that directly
models long-range dependencies of sparse voxels.
• We propose multi-scale sparse supervision to obtain fine-
(a) (b) (c) grained features for the cross-scale attention.
• We propose BEV sparse distribution for centroid predic-
Figure 2: Challenging cases. (a) Two close cars are under-
tion for the first time, which boosts performance for in-
segmented. (b) Surface-aggregated points of a truck are
stance occlusion and ensures time efficiency.
over-segmented. (c) Three pedestrians are merged to one.
• Our method achieves the best performance among the
published papers in the SemanticKITTI dataset for the
et al. 2021) transfer 3D point clouds into range images and challenging 3D PS task with a real-time inference speed.
achieve high inference speed through 2D convolutional net-
works. Panoptic-PolarNet (Zhou, Zhang, and Foroosh 2021)
uses polar BEV grids to circumvent the issue of occlusion Related Work
among instances. The above two methods compress the data This section briefly summarizes recent research related to
from 3D to dense 2D representations, leading to the loss our work, including 3D semantic segmentation, 3D panoptic
of original 3D correlation and the waste of most compu- segmentation, an attention mechanism.
tation in the empty grids. Besides, convolution layers ap- 3D Semantic Segmentation. Point-based methods (Qi et al.
plied on dense representations spread information to the in- 2017a,b; Li et al. 2018; Hu et al. 2020; Thomas et al.
valid grids, making false centroid predictions on occlusion 2019) take raw point clouds as input. They usually sam-
instances and causing under-segmented issues. ple key points and rely on set abstraction to aggregate lo-
Recent works have made great achievements with the cal features. However, sampling leads to information loss,
sparse voxel architecure (Cheng et al. 2021; Tang et al. and set abstraction is computationally costly. To save com-
2020), showing the significance of sparsity to point clouds. putational cost, regular representations, including 3d vox-
The widely-used sparse convolutions (Graham, Engelcke, els and 2d grids, polar and cylinder grids, and range im-
and Van Der Maaten 2018) focus on the valid voxels to re- ages (Zhou and Tuzel 2018; Zhang et al. 2020; Zhu et al.
duce computation and to avoid dilating the sparse features 2020b; Milioto et al. 2019; Xu et al. 2020) are used to or-
to invalid voxels. However, the internal association among ganize sparse points. Recently, hybrid methods (Tang et al.
voxels is difficult to capture with sparse convolutions, espe- 2020; Xu et al. 2021; Ye et al. 2021) that combine multi-
cially when the kernel size fails to cover voxels with long- ple representations are proposed to integrate the advantages
range intervals. Such a “long-range” effect is not obvious of both fine-grained point-wise features and effective fea-
in the semantic task while critical to the instance task. This ture aggregation of regular representations. Sparse convolu-
can be mitigated by projecting sparse features to the 2D tion (Graham 2015; Graham, Engelcke, and Van Der Maaten
dense representation with 2D convolutions diffusing infor- 2018) is also widely used to restrict convolution output only
mation, which is, however, neither efficient nor effective. in the active regions, accelerating the volumetric convolu-
To this end, we propose our efficient sparse cross-scale at- tion and enabling larger model size.
tention network (SCAN) that directly models the long-range 3D Panoptic Segmentation. Current 3D panoptic segmen-
relationship by a cross-scale global attention module. This tation methods usually consist of a semantic branch and an
bottom-up attention mechanism aggregates low-scale, ge- instance branch. LPSAD (Milioto et al. 2020) obtains in-
ometrically strong features with high-scale, geometrically stances by clustering shifted points in range images. DS-
weak features, tackling the surface-aggregated problem with Net (Hong et al. 2021) adaptively and iteratively clusters
multi-scale internal voxel dependency. Besides, we propose the learned point-wise centers. Based on KPConv (Thomas
multi-scale sparse supervision to provide fine-grained fea- et al. 2019), PanosterK (Gasperini et al. 2021) introduces
tures for the attention module. the impurity loss and fragmentation loss to train seman-
We also explore the under-segmented issue for occlusion tic and instance branches jointly, and outputs instance ids
among instances. Instead of timing-consuming clustering- straightway from the network. 4D Panoptic (Aygun et al.
based approach, we follow recent literature (Zhou, Zhang, 2021) proposes a density-based clustering as the initializa-
and Foroosh 2021; Cheng et al. 2020) to use the BEV cen- tion and refines it based on the temporal-spatial consistency.
troid representation. Compared with the existing 3D sparse Panoptic-PolarNet (Zhou, Zhang, and Foroosh 2021) pro-
and 2D dense centroid representations, we propose the BEV poses the 2d dense center heatmap and instance offsets heads
sparse distribution as our instance centroid prediction for for proposal-free instance regression. Major voting is widely
the first time, which keeps the sparsity and the aligned long- adopted as the post-processing to unify the final predictions.
range geometry relationships while ensures efficiency at the Attention Mechanism. Attention is defined as the weighted
same time. Experimental results validate that our SCAN is sum of features at multiple positions. SENet (Hu, Shen,
effective and efficient: our method achieves the best perfor- and Sun 2018), CBAM (Woo et al. 2018), and non-local
 … …
Block 4 Point Features Semantic Prediction

BEV Sparse Distribution Supervision

Sparse Features
Block 4
M2×C s4 b4
Sparse
Block 3 Cross-scale
M2×C s4 b3 Distribution
Attention
Block 2 Head
M1×C s2 b2 A
Block 1 Sparse Features Rbn M×C s1 b1
Point Features Pbn N×C b1
Voxelization Sparse Convs 3×3×3 Multi-scale Sparse Supervision
LiDAR Raw Inputs N×4
Project to Points MLPs
Sparse Voxel Head b4
Backbone Sparse Voxel Head b3
Sparse Voxel Head b2
Cross-scale Attention
Sparse ALignment Operation
M1×C s2 b2 Project to Points Point Offset Supervision
Scatter Index
M2×3 s4
Max Alignment
Voxel Index
M2×C s4 b2 N×CE b2 N×CE b3 N×CE b4 N×CE PA Concat
MLPs
Position
K,V Point Features
Encoding
M2×C s4 b3 Q
Multi-head Attention
Semantic Supervision
Shared M2×C s4 Sparse Distribution Head
Weights K,V
M2×C s4 b4 Cross-scale Scatter
Q MLPs
Sparse Convs
Multi-head Attention Attention Outputs Max on z

Cross-scale Attention Layer M2×C s4

Sparse Distribution G×C

Figure 3: The overall network. Each block of network backbone encodes a sparse voxel feature and a point-wise feature. The
point-wise feature is propagated into the next block. Voxel features from the last three blocks are aggregated by the proposed
cross-scale attention module to acquire the BEV sparse centroid distribution. Besides, we apply multi-scale sparse supervision
on voxel features directly for superior feature learning. Point features from the last three blocks and attention features are
concatenated for point-wise offsets from centroids and semantic predictions.

operation (Wang et al. 2018) have been proposed to ex- mantic predictions.
ploit channel-wise and spatial attention to adaptively re- Backbone Architecture. The raw LiDAR inputs P ∈
fine features and capture long-range dependencies, which RN ×4 (xyz and intensity) are first fed into the network
are effective plug-ins for various computer vision tasks, in- backbone that is composed of four blocks. Each block
cluding classification, detection, and segmentation. Trans- bn , n = 1, 2, 3, 4 first voxelizes the input point features
former (Vaswani et al. 2017) and DETR (Carion et al. 2020) P bn under voxel
 size s and encodes them to sparse fea-
are the pioneers that rely entirely on attention mechanisms tures Rbn = I bn ∈ RM ×3 , F bn ∈ RM ×C by Submani-
to draw global dependencies between inputs and outputs by fold Sparse Convolutional (SSC) layers (Graham, Engelcke,
stacking self-attention and cross-attention modules. Trans- and Van Der Maaten 2018), where I bn is the coordinate
former architectures have also been applied to some indoor indexes of valid voxels in the form of x0 = bx/sc , y 0 =
point cloud tasks, including classification and segmenta- by/sc , z 0 = bz/sc with floor operation bc, and F bn denotes
tion (Engel, Belagiannis, and Dietmayer 2020; Zhao et al. feature tensors corresponding to I bn . M denotes valid voxel
2020; Pan et al. 2021; Guo et al. 2021). However, their mem- numbers under current voxel size and C is the channel num-
ory and computation complexity boost at vast key element ber of the tensor. Then sparse features are projected back
numbers and thus hinder the model scalability. Therefore, to point features that flow to the next block by Multi-layer
the variants including deformable attention modules (Zhu Perceptions (MLPs). The voxel size of block b1→4 is set to
et al. 2020a) and linear attention modules (Katharopoulos s, 2s, 4s and 4s respectively, where s denotes the size of
et al. 2020; Choromanski et al. 2020) have been proposed voxels in x, y, z-axis measured in meters. Inspired by great
to reduce the computation by utilizing deformable convolu- progress achieved by (Ye et al. 2021; Tang et al. 2020), we
tions and matrix properties, respectively. recognize the importance of multiple representation learn-
ing that helps extract better context information. Based on
Approach DRINet (Ye et al. 2021), we utilize both point-wise features
The overall network is illustrated in Fig. 3. The raw point and sparse voxel features. The point-wise features from mul-
clouds are fed to our network backbone that is composed tiple blocks are fused to generate point-wise offset with its
of four blocks. Each block takes a point-wise feature as corresponding semantic prediction, and voxel-wise features
the input, and outputs a point-wise feature and sparse voxel can provide instance-level prediction and semantic predic-
feature. The point-wise feature is propagated into the next tion in sparse format with our proposed cross scale attention.
block. Voxel features from the last three blocks are aggre- Task Abstract. The 3D panoptic segmentation task is ab-
gated by the proposed cross-scale attention module (Sec. 1) stracted into three sub-tasks inspired by (Zhou, Zhang,
to acquire the BEV sparse centroid distribution (Sec. 2). and Foroosh 2021): the point-wise semantic predictions,
Moreover, we apply multi-scale sparse supervision (Sec. 2) BEV centroid distribution and centroid-related point offsets,
on voxel features directly for superior feature learning. Ad- which is a preferred pipeline to take advantage of the voxel-
dition network details are described in Sec. 3. Besides, point wise features for the long-range relationship acquisition. In
features from the last three blocks and attention features are this work, we abstract the pipeline with four heads: 1) BEV
concatenated for point-wise offsets from centroids and se- Sparse Distribution head for instance heatmap prediction; 2)
Multi-scale Sparse head for auxiliary supervision; 3) Point representation (Ye et al. 2021; Xu et al. 2021). Global atten-
Offset head and 4) Point-wise Semantic head. For the first tion has been applied for 2D computer vision and indoor
two heads, we use sparse voxel features from multi-block point cloud tasks (Vaswani et al. 2017; Pan et al. 2021).
Rbn with an attention feature A, and for the last two heads, However, the core issue of applying global attention for
we use point-wise features P bn , P A projected from corre- long-range relationships on large-scale point clouds is that
sponding sparse features, as shown in Fig. 3. it would look over all valid voxels. When the voxel scale is
The cross-scale attention module takes Rbn under multi- low, the number of sparse valid voxels is usually on the order
ple scales 2s, 4s and multiple levels b2 , b3 , b4 as inputs, out- of ten or hundred thousands, making the computation and
putting aligned and fused sparse features A with internal as- memory unbearable. Hence, we apply the attention layer on
sociation among voxels. Furthermore, the additional sparse the high-level sparse voxel features by aligning them to the
voxel semantic predictions from b2→4 are supervised only same scale 4s. To further reduce the computation, we adopt
in training to instruct superior features for feeding into the the Generalized Kernelizable Attention (GKA) (Choroman-
cross-scale attention. To keep the internal association, the ski et al. 2020) as our implementation.
BEV sparse centroid distribution is adopted instead of the Given an input sparse feature Rbn from current block bn ,
dense version in previous literature. let the aligned sparse context feature Ra from Rbn−1 be en-
coded as key and value, the cross-scale attention feature A
Cross-scale Global Attention is calculated by:
In this module, we first align the sparse features from dif-
R̄bn , R̄a = PosEnc Rbn , PosEnc (Ra )


ferent scales by the proposed sparse alignment operation.
Then the multi-scale features with aligned coordinate encod- (3)
A = GKA R̄bn , MLP R̄a , MLP0 (Ra )



ing are fed into the cross-scale global attention layer that
scores the relevance between each pair of voxels and cap- , where the three inputs of GKA are Query, Key and Value
tures the global geometry relationships between the source features respectively. To model the voxel position informa-
and the target sparse features. Furthermore, we propose the tion into Query and Key features, we employ a 3D posi-
multi-scale sparse supervision to enhance the sparse features tion embedding fuction PosEnc. For dimension x in I, we
as the inputs to the cross-scale attention. use commonly used embedding functions
 (Vaswani et al.
Sparse Alignment Operation. Shown as Fig. 3, sparse fea-
 .
2017): P E2i x
= sin x 100002i/dmodel and P E2i+1 x
=
tures from different voxel scales have different voxel coordi-  . 
nates. Therefore, we propose the sparse alignment operation cos x 100002i/dmodel , where dmodel = CA /3 and CA de-
(SA) to align Rb2 at scale 2s with Rb3 , Rb4 at scale 4s:
notes the attention embedding length, i denotes the i-th po-
I¯b2 = I b2 (4s/2s) , I a = Unique I¯b2
  

sition along the feature channel. The same embedding func-
tion is applied on y and z. The final position embedding
F a = maxj F b2 , ∀j ∈ I a


(1)
P E ∈ RMsi ×CA from input voxel coordinates I ∈ RMsi ×3
Ra = SA (Rb2 ) = {I a , F a } is obtained by setting the first dmodel channels to P E x , the
, where the function Unique returns the unique coordinates middle dmodel channels to P E y and the left channels to P E z .
of input I and maxj is over F b2 whose corresponding co- Finally, P E is applied to Query and Key.
ordinates in I¯b2 are j. The coordinate I a of aligned sparse Shown as Fig. 3, the bottom-up cross-scale attention starts
voxel feature Ra is obtained by first downscaled from 2s to from aggregating Rb2 and Rb3 to sparse attention feature A1 ,
4s and then Unique is applied to merge duplicated coordi- and then A1 under scale 4s is aggregated with Rb4 for the
nates. The aligned tensor F a is aggregated over same voxel attention output A. After our attention module, A now con-
indexes under scale 4s. However, the aggregated feature Ra tains the internal voxel relationship cross multi-scale, and is
still doesn’t align with Rb3 voxel by voxel, because the or- used for the following BEV sparse centroid distribution.
der of valid voxels may differ between I a and I b3 . There- Multi-scale Sparse Supervision. Existing works make hard
fore, the additional operation Rearrange named Voxel-wise semantic labels to supervise dense voxels, which not only
Rearrangement is proposed to reorder I a and F a with the costs a large memory footprint but also ignores the possible
queried mask on hashed coordinates: diversity of point labels within the voxels. Therefore, we su-
pervise multi-scale sparse features from block b2→4 directly
H b3 = Hash I b3 , H a = Hash (I a )


with soft voxelized semantic labels, shown as “Multi-scale
E = HashQuery H b3 , H a


(2) Sparse Supervision” in Fig. 3. By making statistics of se-
mantic labels of points in each voxel, the proportion of each
R̄a = {Index (I a , E) , Index (F a , E)} category is taken as the semantic label of the voxel, which
, where the jth element Ej = i of index mask E indicates constructs the sparse voxel labels S bn . After a few SSC lay-
the index i of I a whose hashed value Hia = Hjb3 . Then we ers on Rbn , we use the above-mentioned Rearrange to align
the coordinates from sparse voxel prediction to the sparse
use E to index I a and F a by Index function to obtain the
label S bn . We employ the L1 loss to obtain the loss of the
rearranged feature R̄a whose order is the same as Rb3 .
semantic voxel:
Cross-scale Global Attention Layer. We propose cross-
scale global attention layer to exploit the inherent multi-
X
LL1 Rearrange SSC Rbn , S bn . (4)




Lv =
scale property, which is proved to be crucial to sparse voxel n=2→4
A “hard” method takes the major vote category within the Panoptic Inference. During inference, to further obtain the
voxel as the voxel’s category. On the contrary, our “soft” centroid prediction, we first apply sparse max pooling on D
method calculates statistics of the point number for each cat- and then keep the voxel coordinates with unchanged features
egory, where the target to regress is N class ratio values for before and after this pooling. We keep K centroids with top
each voxel, making it a regression task. Therefore, we chose confidence scores as the final centroid predictions.
the L1 loss instead of a classification loss. By the point-wise semantic predictions S, we get the thing
points. By the predicted K centroids and point-wise offsets
BEV Sparse Centroid Distribution O, we shift each thing point and then assign each shifted
Many previous methods model instance segmentation based point to its closest centroid to get clustering results. Since K
on points or centroids. Since instances are spatially sepa- is set to cover the max number of instances, some predicted
rable, the discretized centroid representation (Zhou, Zhang, centroids are assigned with no points, which are removed
and Foroosh 2021) is highly suitable for LiDAR instances. during inference. To further refine panoptic results, we ob-
Therefore, we choose to use the efficient BEV sparse cen- tain the semantic label of each centroid by majority voting
troid distribution. In this section, we rethink several possible within the semantic predictions of its associated points, then
representations and their pros and cons: we relabel the outlier points in each voxel. Besides, the in-
BEV Dense Distribution. As early adopted in the 2D panop- stance IDs of stuff points are set to 0.
tic task, some works apply the dense distribution to the 3D
tasks under BEV (Zhou, Zhang, and Foroosh 2021; Ge et al. Experiments
2020). The discretized BEV centroid distribution removes In this section, we investigate our method’s performance
the z-axis degree of freedom (DOF) and can naturally ap- on the standard benchmark dataset SemanticKITTI (Behley,
ply 2D convolutions. However, the dense distribution wastes Milioto, and Stachniss 2020) and Nuscenes (Caesar et al.
computation on invalid positions that occupy the majority of 2019). We compare our model with state-of-the-art methods
the BEV map, especially for heavy network heads. Besides, and perform an ablation study to demonstrate the advantage
the 2D convolutions diffuse the captured sparse relationship, of each module in SCAN.
which is harmful to our network.
SemanticKITTI. SemanticKITTI (Behley et al. 2019;
3D Sparse Distribution. The core issues of 3D sparse dis-
Behley, Milioto, and Stachniss 2020) is a challenging
tribution are twofold: 1) the load of computation/memory
dataset, proposed to provide full 360-degree point-wise la-
is heavy; 2) the z-axis DOF makes the task more difficult
bels for the large-scale LiDAR data of the KITTI Odometry
compared with the BEV distribution. The benefit is that it
Benchmark (Geiger, Lenz, and Urtasun 2012). It contains
can make better use of the geometry information.
23201 scans with 3D semantic and instance annotations for
BEV Sparse Distribution. According to the above rethink- training and 20351 for testing. The test evaluation is on the
ing, we propose to use the BEV sparse distribution to model official server with 11 stuff classes and 8 thing classes.
the instance centroids in point clouds, which can main-
Nuscenes. The large-scale Nuscenes dataset (Caesar et al.
tain the sparsity and internal relationship in voxel features
2019) has newly released the panoptic segmentation chal-
while keep efficiency by only computing valid BEV posi-
lenge. The annotations include 10 thing classes and 6 stuff
tions through SSC. With the attention output A, we first set
classes out of total 16 semantic classes. The dataset contains
all z → 0 of I A to flatten the z-axis. Then we apply Unique
1000 scenes, including 850 scenes for training and valida-
on the new coordinates and obtain the max features over
tion and 150 scenes for testing. Since the leaderboard has not
BEV unique voxels by maxj operator in Equ. 1, which gen-
opened until this paper is submitted, we only use the training
erates the BEV sparse feature Ā. With several SSC layers, and validation set in the experiment, which have 28130 and
the final BEV sparse distribution is obtained as D ∈ Rw×h 6019 frames, respectively.
where w and h denote grid sizes in x-axis and y-axis. Evaluation. To assess the semantic segmentation, we
rely on the commonly-used mean intersection-over-union
Network Details
(mIoU) metric (Behley et al. 2019) over all classes. To mea-
Supervision. According to our task abstract, we divide 3D sure the quality of point cloud panoptic segmentation, we
panoptic task into three sub-tasks: 1) BEV sparse centroid adopt the standard convention (Behley, Milioto, and Stach-
distribution prediction D supervised with Focal loss (Lin niss 2020) with PQ, SQ and RQ.
et al. 2017) Ld ; 2) point-wise offset prediction O ∈ RN ×2
that denotes the distances on x, y axes respectively between Implementation Details
points and the corresponding centroids; 3) point-wise se-
mantic prediction S ∈ RN ×1 . We supervise point-wise Training. We fix the voxelization space to be limited in
O with L1 loss as Lo and S with the sum of the Lovász [[±48], [±48], [−3, 1.5]]. We do global rotation along z axis
loss (Berman, Rannen Triki, and Blaschko 2018) and Fo- in range of [−π, π] degrees and flip the points along x,
cal loss as Ls . Besides, we mask out the background points y, and x + y axes. Each augmentation is applied indepen-
during calculating Lo . Furthermore, we supervise the multi- dently with a probability of 50%. In addition, we set the de-
scale sparse semantic prediction with L1 loss as an auxiliary fault scale s = [0.2, 0.2, 0.1] measured in metres, thus the
loss Lv . The total loss is the sum of the above losses: w = 120, h = 120 for the BEV sparse centroid distribu-
tion. The feature channels are set to C = 64 in the network,
L = Ld + Lo + Ls + Lv (5) and we configure the GKA attention by setting the number of
100 RangeNet++/PointP.
KPConv/PointP.
90 PanosterK
80 EfficientLPS
Panoptic-PolarNet
70
DS-Net
60 SCAN [ours]
50

40

30

20
10
0

Figure 4: Per-class results of metric PQ on the SemanticKITTI test set.

Method PQ PQ† SQ RQ PQth SQth RQth PQst SQst RQst mIoU FPS
RangeNet++/PointPillars 37.1 45.9 75.9 47.0 20.2 75.2 25.2 49.3 76.5 62.8 52.4 2.4
KPConv/PointPillars 44.5 52.5 80.0 54.4 32.7 81.5 38.7 53.1 79.0 65.9 58.8 1.9
LPSAD 38.0 47.0 76.5 48.2 25.6 76.8 31.8 47.1 76.2 60.1 50.9 11.8
Panoster 52.7 59.9 80.7 64.1 49.4 83.3 58.5 55.1 78.8 68.2 59.9 -
DS-Net 55.9 62.5 82.3 66.7 55.1 87.2 62.8 56.5 78.7 69.5 61.6 -
Panoptic-PolarNet 54.1 60.7 81.4 65.0 53.3 87.2 60.6 54.8 77.2 68.1 59.5 11.6
EfficientLPS 57.4 63.2 83.0 68.7 53.1 87.8 60.5 60.5 79.5 74.6 61.4 -
GP-S3Net 60.0 69.0 82.0 72.1 65.0 86.6 74.5 56.4 78.7 70.4 70.8 -
SCAN 61.5 67.5 84.5 72.1 61.4 88.1 69.3 61.5 81.8 74.1 67.7 12.8

Table 1: Comparison of test set results on SemanticKITTI using stuff (st) and thing(th) classes. All results in [%].

formance for the LiDAR panoptic segmentation. We present

our Nuscenes validation results in Table 2, and compare re-
sults with models that use the same settings of the dataset.
Inference Speed. Tab. 1 also shows the frames per second
(FPS) for each approach. The reported speed for our ap-
Figure 5: Results of SCAN. The first two samples demon- proach is the full time including all pre- and post- process-
strate its effectiveness for challenging under-segmentation, ing. We investigate that our model can achieve a real-time in-
and the last shows the robustness on over-segmentation. ference speed for autonomous driving. Compared with other
models that reported speed (containing preprocessing), our
approach maintains a faster speed while improves accuracy.
heads 8, attention depth 2, channels of each head 16 and dis- Improvement. Many previous works are troubled by over-
abling the causality inference. We implement the network in or under- segmentation problems. However, as shown in
PyTorch (Paszke et al. 2017) and train the SCAN model on 8 Fig. 4, our method handles these challenges well. In the case
NVIDIA 3090 GPUs for 40 epochs with Adam (Kingma and of large objects like trucks which are often over-segmented,
Ba 2014) and 1-Cycle Schedule (Smith 2017). We set the our method outperforms other approaches by a large mar-
batch size per GPU as 4 and the initial learning rate 0.003. gin. Also, we investigate that our method is robust enough
The learning rate first raises tenfold before the 16-th epoch to deal with the cases of under-segmentation and small ob-
and then decays. jects, especially for person, other-vehicle and motorcyclist
Inference. During inference, the auxiliary voxel semantic classes. Such improvements demonstrate the delicate local-
prediction is cut off to save the computation. The max cen- ization and effective object grouping of SCAN.
troid number K is set to 100 and the centroid score threshold Qualitative Results. The qualitative performance on the
is set to 0.1. For points out of the fixed voxelization space, challenging scans can be illustrated in Fig. 4. Besides, shown
we set both the semantic and instance predictions to 0. as Fig. 5, our approach can handle the challenging samples
with over- and under- segmentation.
Experiment Results
Comparison to the State-of-the-arts. Tab. 1 shows the Ablation Study.
quantitative comparison on the SemanticKITTI test set sub- To analyze the effectiveness of different modules in
mitted to the official test server. We find that SCAN outper- SCAN, adequate ablation studies are conducted on the Se-
forms GP-S3Net (Razani et al. 2021) by 1.5% while ensures manticKITTI valid set. Starting from a baseline whose net-
real-time inference speed, setting a new state-of-the-art per- work backbone is the same with SCAN, we switch on pro-
 Method PQ PQ† SQ RQ PQth SQth RQth PQst SQst RQst mIOU
DS-Net 42.5 51.0 50.3 83.6 32.5 83.1 38.3 59.2 84.4 70.3 70.7
EfficientLPS 59.2 62.8 82.9 70.7 51.8 80.6 62.7 71.5 84.3 84.1 69.4
SCAN(ours) 65.1 68.9 85.7 75.3 60.6 85.7 70.2 72.5 85.7 83.8 77.4

Table 2: Comparison of validation set results on Nuscenes using stuff (st) and thing(th) classes. All results in [%].

Semantic GT Semantic Pred Panoptic GT Panoptic Pred

Figure 6: Qualitative results of SCAN from the SemanticKITTI validation sets.

MSS CGA DD 3SD SD PQ mIoU Runtime b1 b2 b3 b4 PE SW PQ mIoU

X 53.3 65.8 66ms X X 54.5 67.4
X X 53.7 66.1 66ms X X X 55.9 68.1
X X 56.0 67.9 81ms X X X X 56.8 68.7
X X X 56.7 68.5 81ms X X X X X 56.6 68.6
X X X 56.5 68.4 85ms X X X 55.6 68.1
X X X 57.2 68.9 78ms X X X X X 57.2 68.9

Table 3: Ablation study of SCAN on the validation split Table 4: Ablation study of the cross-scale global attention
of SemanticKITTI. CGA, MSS, SD, DD, 3SD represent for module. b1→4 stands features from which block are as inputs
cross-scale global attention (CGA), multi-scale sparse su- to the module. PE represents the 3D position encoding and
pervision (MSS), the BEV sparse centroid distribution (SD) SW denotes sharing weight for attention layers.
with its dense version (DD) and 3d sparse version (3SD).

posed modules respectively, as shown in Tab. 3, which indi- may be the insufficient learning of sparse features in the first
cates the metrics and inference Runtime (batch size is 1) of block. Moreover, we try to remove the 3D position encoding
models with NVIDIA 3090 GPU. Instead of directly sum- PE and the performance has a sharp drop by 1.2%, demon-
ming multi-scale sparse features, the proposed cross-scale strating the importance of 3D voxel coordinates towards
global attention CGA improves by the largest margin of global attention learning. Forcing the same attention layer
2.7% compared to the baseline in PQ with small computa- SW to learn the long-range relationship gives another 0.6%
tion increment, demonstrating the validity. The multi-scale improvement. By sharing weights among attention layers,
sparse supervision MSS achieves a 0.4% promotion sepa- our cross-scale global attention module concentrates on the
rately but gives 0.7% gain together with CGA, which indi- learning of cross-scale attention patterns.
cates the proposed CGA can be stimulated by fine-grained
features. Notice that MSS brings no influence in speed be-
cause that MSS will be cut off during inference. We also Conclusion
conduct experiments on different representations of centroid
distribution. The performance of the default used dense ver- We present efficient SCAN, a novel sparse cross-scale atten-
sion DD is 0.2% higher than the 3D sparse version 3SD. Our tion network to first address the surface-aggregated problem
BEV sparse centroid distribution SD achieves the best per- in the 3D panoptic segmentation task by modeling the long-
formance with an even smaller computation. range dependency among sparse voxel representation. The
We also conduct experiments on the cross-scale global at- proposed cross-scale attention module introduces the atten-
tention module, as shown in Tab. 4 to investigate the module tion mechanism to align and fuse multi-level and multi-scale
that brings the most gain. We first adopt various input com- sparse features in global instead of only stacking sparse con-
binations of sparse features from different blocks b1→4 . Re- volution layers for local context information. Moreover, the
sults indicate that attention on b2→4 outperforms other com- multi-scale sparse voxel supervision is proposed to obtain
binations. Only taking b4 as input makes the module tune fine-grained features for the cross-scale attention. In addi-
into the self-attention. The biggest improvement occurs be- tion, we rethink centroid distributions and finally choose the
tween b3 , b4 cross-attention and b4 self-attention, where the BEV sparse distribution for better performance with lower
former provides a foundation for the acquisition of the in- computation and memory footprint. Our method achieves
ternal voxel relationship across sparse features. Including the state-of-the-art among published work and 1st in the Se-
the b1 feature degrades performance, for which the reason manticKITTI challenge with a real-time runtime speed.
 References Hong, F.; Zhou, H.; Zhu, X.; Li, H.; and Liu, Z. 2021. Lidar-based
Aygun, M.; Osep, A.; Weber, M.; Maximov, M.; Stachniss, C.; panoptic segmentation via dynamic shifting network. In IEEE
Behley, J.; and Leal-Taixé, L. 2021. 4D Panoptic LiDAR Segmen- Conf. Comput. Vis. Pattern Recog., 13090–13099.
tation. In IEEE Conf. Comput. Vis. Pattern Recog., 5527–5537. Hu, J.; Shen, L.; and Sun, G. 2018. Squeeze-and-excitation net-
Behley, J.; Garbade, M.; Milioto, A.; Quenzel, J.; Behnke, S.; works. In IEEE Conf. Comput. Vis. Pattern Recog., 7132–7141.
Stachniss, C.; and Gall, J. 2019. SemanticKITTI: A dataset for se- Hu, Q.; Yang, B.; Xie, L.; Rosa, S.; Guo, Y.; Wang, Z.; Trigoni,
mantic scene understanding of lidar sequences. In Int. Conf. Com- N.; and Markham, A. 2020. RandLA-Net: Efficient semantic seg-
put. Vis., 9297–9307. mentation of large-scale point clouds. In IEEE Conf. Comput. Vis.
Behley, J.; Milioto, A.; and Stachniss, C. 2020. A Benchmark Pattern Recog., 11108–11117.
for LiDAR-based Panoptic Segmentation based on KITTI. arXiv Jiang, L.; Zhao, H.; Shi, S.; Liu, S.; Fu, C.-W.; and Jia, J. 2020.
preprint arXiv:2003.02371. PointGroup: Dual-Set Point Grouping for 3D Instance Segmenta-
Berman, M.; Rannen Triki, A.; and Blaschko, M. B. 2018. The tion. In IEEE Conf. Comput. Vis. Pattern Recog., 4867–4876.
lovász-softmax loss: A tractable surrogate for the optimization of Katharopoulos, A.; Vyas, A.; Pappas, N.; and Fleuret, F. 2020.
the intersection-over-union measure in neural networks. In IEEE Transformers are rnns: Fast autoregressive transformers with lin-
Conf. Comput. Vis. Pattern Recog., 4413–4421. ear attention. In ICML, 5156–5165. PMLR.
Caesar, H.; Bankiti, V.; Lang, A. H.; Vora, S.; Liong, V. E.; Xu, Kingma, D. P.; and Ba, J. 2014. Adam: A method for stochastic
Q.; Krishnan, A.; Pan, Y.; Baldan, G.; and Beijbom, O. 2019. optimization. arXiv preprint arXiv:1412.6980.
nuScenes: A multimodal dataset for autonomous driving. arXiv
Lahoud, J.; Ghanem, B.; Pollefeys, M.; and Oswald, M. R. 2019.
preprint arXiv:1903.11027.
3d instance segmentation via multi-task metric learning. In Int.
Carion, N.; Massa, F.; Synnaeve, G.; Usunier, N.; Kirillov, A.; and Conf. Comput. Vis., 9256–9266.
Zagoruyko, S. 2020. End-to-end object detection with transform-
Li, Y.; Bu, R.; Sun, M.; Wu, W.; Di, X.; and Chen, B. 2018.
ers. In Eur. Conf. Comput. Vis., 213–229. Springer.
Pointcnn: Convolution on x-transformed points. In Adv. Neural
Cheng, B.; Collins, M. D.; Zhu, Y.; Liu, T.; Huang, T. S.; Adam, Inform. Process. Syst., 820–830.
H.; and Chen, L.-C. 2020. Panoptic-deeplab: A simple, strong, and
fast baseline for bottom-up panoptic segmentation. In IEEE Conf. Lin, T.-Y.; Goyal, P.; Girshick, R.; He, K.; and Dollár, P. 2017.
Comput. Vis. Pattern Recog., 12475–12485. Focal loss for dense object detection. In Int. Conf. Comput. Vis.,
2980–2988.
Cheng, R.; Razani, R.; Taghavi, E.; Li, E.; and Liu, B. 2021. 2-
S3Net: Attentive feature fusion with adaptive feature selection for Liu, S.-H.; Yu, S.-Y.; Wu, S.-C.; Chen, H.-T.; and Liu, T.-L. 2020.
sparse semantic segmentation network. In IEEE Conf. Comput. Vis. Learning Gaussian Instance Segmentation in Point Clouds. arXiv
Pattern Recog., 12547–12556. preprint arXiv:2007.09860.
Choromanski, K.; Likhosherstov, V.; Dohan, D.; Song, X.; Gane, Milioto, A.; Behley, J.; McCool, C.; and Stachniss, C. 2020. Li-
A.; Sarlos, T.; Hawkins, P.; Davis, J.; Mohiuddin, A.; Kaiser, L.; DAR Panoptic Segmentation for Autonomous Driving. In IROS.
et al. 2020. Rethinking attention with performers. arXiv preprint Milioto, A.; Vizzo, I.; Behley, J.; and Stachniss, C. 2019.
arXiv:2009.14794. RangeNet++: Fast and accurate LiDAR semantic segmentation. In
Engel, N.; Belagiannis, V.; and Dietmayer, K. 2020. Point trans- 2019 IEEE/RSJ International Conference on Intelligent Robots and
former. arXiv preprint arXiv:2011.00931. Systems (IROS), 4213–4220. IEEE.
Engelmann, F.; Bokeloh, M.; Fathi, A.; Leibe, B.; and Nießner, M. Pan, X.; Xia, Z.; Song, S.; Li, L. E.; and Huang, G. 2021. 3d object
2020. 3D-MPA: Multi-Proposal Aggregation for 3D Semantic In- detection with pointformer. In IEEE Conf. Comput. Vis. Pattern
stance Segmentation. In IEEE Conf. Comput. Vis. Pattern Recog., Recog., 7463–7472.
9031–9040. Paszke, A.; Gross, S.; Chintala, S.; Chanan, G.; Yang, E.; DeVito,
Gasperini, S.; Mahani, M.-A. N.; Marcos-Ramiro, A.; Navab, N.; Z.; Lin, Z.; Desmaison, A.; Antiga, L.; and Lerer, A. 2017. Auto-
and Tombari, F. 2021. Panoster: End-to-end Panoptic Segmentation matic differentiation in PyTorch. In NIPS-W.
of LiDAR Point Clouds. IEEE Robotics and Automation Letters. Qi, C. R.; Litany, O.; He, K.; and Guibas, L. J. 2019. Deep hough
Ge, R.; Ding, Z.; Hu, Y.; Wang, Y.; Chen, S.; Huang, L.; and Li, voting for 3d object detection in point clouds. In Int. Conf. Comput.
Y. 2020. Afdet: Anchor free one stage 3d object detection. arXiv Vis., 9277–9286.
preprint arXiv:2006.12671. Qi, C. R.; Su, H.; Mo, K.; and Guibas, L. J. 2017a. Pointnet: Deep
Geiger, A.; Lenz, P.; and Urtasun, R. 2012. Are we ready for au- learning on point sets for 3d classification and segmentation. In
tonomous driving? the kitti vision benchmark suite. In IEEE Conf. IEEE Conf. Comput. Vis. Pattern Recog., 652–660.
Comput. Vis. Pattern Recog., 3354–3361. IEEE. Qi, C. R.; Yi, L.; Su, H.; and Guibas, L. J. 2017b. Pointnet++: Deep
Graham, B. 2015. Sparse 3D convolutional neural networks. Brit. hierarchical feature learning on point sets in a metric space. In Adv.
Mach. Vis. Conf. Neural Inform. Process. Syst., 5099–5108.
Graham, B.; Engelcke, M.; and Van Der Maaten, L. 2018. 3d se- Razani, R.; Cheng, R.; Li, E.; Taghavi, E.; Ren, Y.; and Bingbing,
mantic segmentation with submanifold sparse convolutional net- L. 2021. GP-S3Net: Graph-based Panoptic Sparse Semantic Seg-
works. In IEEE Conf. Comput. Vis. Pattern Recog., 9224–9232. mentation Network. arXiv preprint arXiv:2108.08401.
Guo, M.-H.; Cai, J.-X.; Liu, Z.-N.; Mu, T.-J.; Martin, R. R.; and Sirohi, K.; Mohan, R.; Büscher, D.; Burgard, W.; and Valada,
Hu, S.-M. 2021. PCT: Point cloud transformer. Computational A. 2021. EfficientLPS: Efficient LiDAR Panoptic Segmentation.
Visual Media, 7(2): 187–199. arXiv preprint arXiv:2102.08009.
Hong, F.; Zhou, H.; Zhu, X.; Li, H.; and Liu, Z. 2020. LiDAR- Smith, L. N. 2017. Cyclical learning rates for training neural net-
based Panoptic Segmentation via Dynamic Shifting Network. works. In 2017 IEEE winter conference on applications of com-
arXiv preprint arXiv:2011.11964. puter vision (WACV), 464–472. IEEE.
Tang, H.; Liu, Z.; Zhao, S.; Lin, Y.; Lin, J.; Wang, H.; and Han,
S. 2020. Searching Efficient 3D Architectures with Sparse Point-
Voxel Convolution. In Eur. Conf. Comput. Vis.
Thomas, H.; Qi, C. R.; Deschaud, J.-E.; Marcotegui, B.; Goulette,
F.; and Guibas, L. J. 2019. Kpconv: Flexible and deformable con-
volution for point clouds. In Int. Conf. Comput. Vis., 6411–6420.
Vaswani, A.; Shazeer, N.; Parmar, N.; Uszkoreit, J.; Jones, L.;
Gomez, A. N.; Kaiser, Ł.; and Polosukhin, I. 2017. Attention is
all you need. In Adv. Neural Inform. Process. Syst., 5998–6008.
Wang, X.; Girshick, R.; Gupta, A.; and He, K. 2018. Non-local
neural networks. In IEEE Conf. Comput. Vis. Pattern Recog., 7794–
7803.
Wang, X.; Liu, S.; Shen, X.; Shen, C.; and Jia, J. 2019. Associa-
tively segmenting instances and semantics in point clouds. In IEEE
Conf. Comput. Vis. Pattern Recog., 4096–4105.
Woo, S.; Park, J.; Lee, J.-Y.; and Kweon, I. S. 2018. Cbam: Convo-
lutional block attention module. In Eur. Conf. Comput. Vis., 3–19.
Xu, C.; Wu, B.; Wang, Z.; Zhan, W.; Vajda, P.; Keutzer, K.; and
Tomizuka, M. 2020. Squeezesegv3: Spatially-adaptive convolution
for efficient point-cloud segmentation. In Eur. Conf. Comput. Vis.,
1–19. Springer.
Xu, J.; Zhang, R.; Dou, J.; Zhu, Y.; Sun, J.; and Pu, S. 2021.
RPVNet: A Deep and Efficient Range-Point-Voxel Fusion Net-
work for LiDAR Point Cloud Segmentation. arXiv preprint
arXiv:2103.12978.
Yang, B.; Wang, J.; Clark, R.; Hu, Q.; Wang, S.; Markham, A.;
and Trigoni, N. 2019. Learning object bounding boxes for 3d in-
stance segmentation on point clouds. In Adv. Neural Inform. Pro-
cess. Syst., 6740–6749.
Ye, M.; Xu, S.; Cao, T.; and Chen, Q. 2021. DRINet: A Dual-
Representation Iterative Learning Network for Point Cloud Seg-
mentation. arXiv:2108.04023.
Zhang, Y.; Zhou, Z.; David, P.; Yue, X.; Xi, Z.; Gong, B.; and
Foroosh, H. 2020. PolarNet: An Improved Grid Representation
for Online LiDAR Point Clouds Semantic Segmentation. In IEEE
Conf. Comput. Vis. Pattern Recog., 9601–9610.
Zhao, H.; Jiang, L.; Jia, J.; Torr, P.; and Koltun, V. 2020. Point
transformer. arXiv preprint arXiv:2012.09164.
Zhou, Y.; and Tuzel, O. 2018. Voxelnet: End-to-end learning for
point cloud based 3d object detection. In IEEE Conf. Comput. Vis.
Pattern Recog., 4490–4499.
Zhou, Z.; Zhang, Y.; and Foroosh, H. 2021. Panoptic-PolarNet:
Proposal-free LiDAR Point Cloud Panoptic Segmentation. In IEEE
Conf. Comput. Vis. Pattern Recog., 13194–13203.
Zhu, X.; Su, W.; Lu, L.; Li, B.; Wang, X.; and Dai, J. 2020a. De-
formable detr: Deformable transformers for end-to-end object de-
tection. arXiv preprint arXiv:2010.04159.
Zhu, X.; Zhou, H.; Wang, T.; Hong, F.; Ma, Y.; Li, W.; Li, H.; and
Lin, D. 2020b. Cylindrical and Asymmetrical 3D Convolution Net-
works for LiDAR Segmentation. arXiv preprint arXiv:2011.10033.