IEEE TRANSACTIONS ON MEDICAL IMAGING 1

UN-SAM: Universal Prompt-Free Segmentation

for Generalized Nuclei Images
Zhen Chen, Qing Xu, Xinyu Liu, and Yixuan Yuan, Member, IEEE

Abstract— In digital pathology, precise nuclei segmen-

arXiv:2402.16663v1 [[Link]] 26 Feb 2024

tation is pivotal yet challenged by the diversity of tis-

sue types, staining protocols, and imaging conditions. Re-
cently, the segment anything model (SAM) revealed over-
whelming performance in natural scenarios and impressive
adaptation to medical imaging. Despite these advantages,
the reliance of labor-intensive manual annotation as seg-
mentation prompts severely hinders their clinical applica-
bility, especially for nuclei image analysis containing mas-
sive cells where dense manual prompts are impractical.
To overcome the limitations of current SAM methods while
retaining the advantages, we propose the Universal prompt-
free SAM framework for Nuclei segmentation (UN-SAM),
by providing a fully automated solution with remarkable
generalization capabilities. Specifically, to eliminate the
labor-intensive requirement of per-nuclei annotations for
prompt, we devise a multi-scale Self-Prompt Generation
(SPGen) module to revolutionize clinical workflow by au-
Fig. 1. Performance comparison on nuclei image segmentation. The
tomatically generating high-quality mask hints to guide the semantic segmentation is measured by Dice, and instance segmenta-
segmentation tasks. Moreover, to unleash the generaliza- tion (marked as inst.) is measured by Aggregated Jaccard Index (AJI).
tion capability of SAM across a variety of nuclei images,
we devise a Domain-adaptive Tuning Encoder (DT-Encoder)
to seamlessly harmonize visual features with domain-
common and domain-specific knowledge, and further de- within tissue samples. In particular, instance segmentation
vise a Domain Query-enhanced Decoder (DQ-Decoder) methods designed for natural images [4], [5] rely on region
by leveraging learnable domain queries for segmentation proposals and struggle to discriminate adjacent nuclei, thus
decoding in different nuclei domains. Extensive experi- compromising the precise segmentation of individual nucleus.
ments prove that UN-SAM with exceptional performance
surpasses state-of-the-arts in nuclei instance and seman- In this challenging context, classical nuclei image segmen-
tic segmentation, especially the generalization capability tation algorithms [6]–[10] are investigated through tailored
in zero-shot scenarios. The source code is available at network structures and supervision strategies targeting the
[Link] characteristics of cell nuclei. These studies perform with high
Index Terms— Nuclei image, semantic segmentation, in- accuracy under fully supervised conditions in homogeneous
stance segmentation, domain generalization. datasets, and play an integral role in advancing the field. How-
ever, these methods require carefully crafted post-processing
I. I NTRODUCTION to identify nuclei instances, which are vulnerable to inten-
sive hyper-parameter search. More seriously, their limitations

I N the field of digital pathology, the task of nuclei im-

age segmentation on histopathological images stands as
a cornerstone for morphological quantification and tumor
become apparent when faced with the inherent heterogeneity
of tissue types, staining protocols, and imaging conditions
encountered in routine clinical practice. This variability poses
grade assessment [1]. Despite its significance, this task poses significant challenges to developing segmentation models with
substantial difficulties to nuclei image segmentation tasks [2], the necessary generality and accuracy across different datasets.
[3], arising from the highly dense packing of cell nuclei The recent advent of the segment anything model (SAM)
This work was supported by the Hong Kong Research Grants Council [11] has ushered in a new era of segmentation, and holds
(RGC) General Research Fund 14204321, 14220622 and the InnoHK promise in addressing these issues with versatile capabilities
program. (Equal contribution: Z. Chen and Q. Xu, Corresponding author: that extend beyond traditional constraints. The remarkable
Yixuan Yuan)
Z. Chen is with the Centre for Artificial Intelligence and Robotics efficacy of SAM in natural image scenarios has been well
(CAIR), Hong Kong Institute of Science & Innovation, Chinese Academy validated to showcase its robustness and adaptability across a
of Sciences, Hong Kong SAR. (e-mail: [Link]@[Link]). variety of scenarios. On this basis, SAM has begun to penetrate
Q. Xu, X. Liu and Y. Yuan are with Department of Electronic Engineer-
ing, Chinese University of Hong Kong, Hong Kong SAR, China (e-mail: the medical imaging community and revealed its capability
yxyuan@[Link]). in the segmentation of medical images with the potential for
2 IEEE TRANSACTIONS ON MEDICAL IMAGING

diverse scenarios [12]–[15]. Additionally, preliminary inves- of the labor-intensive nuclei annotations for prompt, we de-
tigations have indicated that SAM can adapt to the unique vise a multi-scale Self-Prompt Generation (SPGen) module
challenges posed by medical data, though further refinement to automatically generate high-quality mask hints, guiding
is necessary for clinical deployment [16]–[18]. These studies segmentation decoding of the UN-SAM. As such, the SPGen
collectively underscore the potential of SAM to revolutionize module can streamline the clinical workflow and make high-
medical image segmentation tasks across diverse scenarios. throughput analysis feasible without the labor-intensive pro-
Despite the progress made by existing medical SAM studies cess of manually crafted prompts. Moreover, to unleash the
[16]–[18], the adaptation of SAM to nuclei segmentation is generalization capability of SAM across a variety of nuclei
hampered by two significant obstacles, including the reliance images, we devise a Domain-adaptive Tuning Encoder (DT-
on manual annotations for segmentation prompt and the chal- Encoder) for UN-SAM to seamlessly harmonize visual fea-
lenge of generalizing across diverse nuclei images. Firstly, tures with domain-common and domain-specific knowledge,
one limitation of medical SAM algorithms is the reliance on and further devise a tailored Domain Query-enhanced Decoder
manual annotations, e.g., the point-based prompt and bounding (DQ-Decoder) for UN-SAM, by leveraging learnable domain
box-based prompt, to guide the segmentation decoding of the queries to distinguish nuclei types and regions in different
target objects [14], [15]. Although SAM provides an automatic domains of complex semantic and instance segmentation. In
mode to generate the bounding box-based prompt by filtering Fig. 1, the comparison on diverse nuclei image datasets proves
a sliding window of the input image [11], current studies that the proposed UN-SAM achieves superior generalization
[12], [15] have demonstrated that such simple automatic on different datasets, with a remarkable performance advan-
prompts perform poorly in medical image segmentation, and tage over classical medical segmentation method [20] and
even the simple manual annotations (e.g., using only a pair medical SAMs [15]–[18].
of positive and negative points) are difficult to ensure that The contributions of this work are summarized as follows:
medical SAM achieves satisfactory segmentation predictions. • We propose a prompt-free UN-SAM framework to pro-
In contrast, most existing medical SAM studies [14], [16] vide an automatic solution for nuclei instance and seman-
rely on more manual annotations, by labeling extra points or tic segmentation with remarkable generalization capabil-
bounding boxes as the prompt. The extensive use of man- ities across diverse datasets.
ual annotations interrupts clinical workflows, and makes the • We devise a multi-scale SPGen module to autonomously
process resource-intensive and typically impractical in clinical generate high-quality mask hints and effectively guide the
settings [19], especially for nuclei images with massive cells UN-SAM decoder, thereby eliminating the requirement of
that are extremely costly. Therefore, the ideal SAM algorithm the labor-intensive nuclei annotations for prompt.
in nuclei segmentation tasks should eliminate the need for • We devise a DT-Encoder to harmonize visual features
manual annotation, and automatically achieve accurate nuclei with domain-common and domain-specific knowledge,
segmentation based on the knowledge of foundation models. and a DQ-Decoder to leverage domain queries for seg-
Furthermore, the heterogeneity of nuclei segmentation tasks, mentation decoding. These two designs enable our UN-
which encompass various domains such as different tissue SAM to generalize across different nuclei domains.
types, staining protocols, and imaging equipment [2], as • We conduct extensive experiments on diverse nuclei
well as distinct task requirements like semantic and instance image datasets, and our UN-SAM outperforms state-of-
segmentation, necessitates the segmentation algorithm with the-art nuclei segmentation methods and medical SAMs,
robust generalization capabilities. However, existing medical with remarkable zero-shot generalization performance.
SAM algorithms [14], [15] primarily rely on their inherent
capacities to transfer general knowledge to medical imaging,
e.g., directly utilizing the image encoder of SAM [14] and II. R ELATED W ORK
fine-tuning its encoder on downstream medical data [16]–[18].
A. Nuclei Image Segmentation
As such, these strategies, by simply utilizing the knowledge
SAM, may be relatively difficult to deliver high performance The nuclei image segmentation is crucial for histopathol-
across different medical datasets, especially for histopathology ogy image analysis and can benefit pathologists in rendering
nuclei analysis where the diversity and complexity of images precise diagnoses [3]. Existing studies can be categorized into
are particularly pronounced [2]. As illustrated in Fig. 1, these nuclei semantic segmentation and nuclei instance segmenta-
medical SAMs [15]–[18] cannot perform satisfactorily on the tion. For the semantic segmentation that aims to identify the
nuclei segmentation with diverse scenes, and is even inferior type and spatial region of nuclei, the early studies utilized the
to classical medical segmentation networks [20] with the U-Net structure [21] to achieve automatic nuclei segmentation.
same training data. Therefore, to enhance the generalization To improve the network capability in segmentation, [22]–
capability in such diverse settings, the SAM model should be [26] further enhanced U-Net derivatives through recurrent and
improved with tailored designs for nuclei segmentation tasks. attention mechanisms. Particularly, ConTrans [26] adopted
To address these two aforementioned challenges in nuclei a dual attention encoder to capture both global and local
segmentation, we propose the Universal Prompt-Free SAM information of nuclei details, which efficiently recognized
(UN-SAM) to achieve accurate and automatic semantic seg- nuclei with different shapes in histopathology images.
mentation and instance segmentation with remarkable general- In the realm of instance segmentation that further identifies
ization capabilities. Specifically, to eliminate the requirement each nucleus [27], [28], existing studies [6]–[10] has made
UN-SAM: UNIVERSAL PROMPT-FREE SEGMENTATION FOR GENERALIZED NUCLEI IMAGES 3

notable strides through tailored network structures and super- endeavors to resolve this bottleneck with tailored efficient
vision strategies targeting the characteristics of cell nuclei. training and automatic prompt generation mechanisms, thereby
For example, HoVer-Net [6] utilized horizontal and vertical streamlining the segmentation process for nuclei images.
distance maps to discern the boundaries of individual nuclei In the field of nuclei segmentation, All-in-SAM [35] utilized
within histopathology images. To refine segmentation outputs, high-frequency image information extracted by Fourier trans-
CDNet [8] utilized directional feature maps and PROnet [9] form to tune the prompt encoder and mask decoder of SAM.
leveraged offset maps, to enhance the delineation of nuclei With the centroid of each nucleus as prompt, the SPPNet [19]
boundaries. The CPP-Net [7] forged a different path by calculated the surrounding neighborhood points of centroids
generating complementary boundary and distance masks for as auxiliary prompts to facilitate the segmentation. Different
each nucleus, facilitating the separation of nuclei via the from existing methods that targeted a single nuclei domain
integration of these masks. Despite the progress, these nu- [19], [30], [35], our UN-SAM fully leverages the knowledge
clei instance segmentation methods [6]–[10] demand complex of foundation models, and achieves generalization on varying
post-processing, and necessitate training on each nuclei image nuclei domains without any manual prompts.
dataset, which are difficult to generalize to unseen nuclei
image domains. In contrast, as a universal and prompt-free III. M ETHODOLOGY
segmentation framework, our UN-SAM transcends dataset- A. Overview of UN-SAM
specific limitations, and reveals superior generalization across
As illustrated in Fig. 2, we present the UN-SAM to provide
a variety of nuclei image domains.
automatic segmentation with remarkable generalization across
a variety of nuclei images. Given nuclei images from the k-th
B. The SAM in Medical Imaging domain, we first utilize the Domain-adaptive Tuning Encoder
(DT-Encoder) to generate domain knowledge-enhanced image
The segment anything model (SAM) [11] has well revealed
embeddings. Then, these embeddings are delivered to the
the advantage in image segmentation across a variety of
multi-scale Self-Prompt Generation (SPGen) module, which
scenarios. By leveraging both sparse (e.g., point, box, and
autonomously produces a set of self-generated prompt tokens
text) and dense (e.g., mask) prompts, SAM benefits from a
to guide nuclei segmentation. Following this, the Domain
robust feature extraction capability [29], enabling it to perform
Query-enhanced Decoder (DQ-Decoder) leverages the query
zero-shot generalization for diverse image segmentation tasks.
embedding of the corresponding domain to accurately predict
To migrate the powerful segmentation capabilities of SAM
the segmentation mask.
to downstream scenarios, existing works adopted different
In general, this novel UN-SAM, comprising the DT-Encoder
fine-tuning strategies, including directly fine-tuning the image
and DQ-Decoder, is specifically designed to tackle the chal-
encoder [30] or mask decoder [14]. Considering the huge
lenges of model generalization across varying domains in
amount of SAM parameters, the parameter efficient fine-tuning
nuclei image segmentation. Meanwhile, the SPGen module
(PEFT) has become a hot research topic, such as the low-
eliminates the demand of labor-intensive manual annotations
rank adaptation (LoRA) [31] and adapter [32] techniques. The
for prompts that are typically associated with the SAM frame-
recent Conv-LoRA [33] introduced a PEFT strategy for tuning
works [11], [14]. Therefore, these key modules form our UN-
SAM, and facilitated feature representation with inductive
SAM that enhances the generalization and applicability to a
biases by dynamically selecting the proper feature scale.
wide array of nuclei image segmentation.
On this basis, many medical SAM works [14]–[18] have
been investigated to customize segmentation capability to
medical imaging. Huang et al. [15] explored the capability of B. Domain-adaptive Tuning Encoder
vanilla SAM [11] with different types of prompts on medical Recent studies have attributed the remarkable segmentation
image segmentation, and further fine-tuned the SAM with capabilities of SAM to its large-capacity image encoder [13],
point and bounding box prompts on large-scale medical image [29]. When adapting SAM to medical images, existing studies
datasets, which surpasses the state-of-the-art performance on mainly utilized the pre-trained SAM encoder directly [14] or
45 public datasets. The MedSAM [14] collected a diverse fine-tuned it on downstream datasets [16]–[18]. However, these
medical image segmentation dataset dominated by CT and studies that rely entirely on a set of image encoder parameters
MRI scans for tuning SAM with bounding box prompts. are suboptimal for the complex task of nuclei segmentation,
Furthermore, the SAMUS [16] adopted the PEFT strategy and especially considering the heterogeneity of the nuclei image
integrated additional convolutional network embeddings with domains. To extract discriminative visual features for nuclei
adapter modules to refine feature representations of the image segmentation across different domains, we propose a Domain-
encoder for downstream datasets. For the surgical images, adaptive Tuning Encoder (DT-Encoder) that leverages a learn-
the SurgicalSAM [34] introduced the class prototypes and able set of domain-common and domain-specific embeddings
designated target class to guide the segmentation with the for downstream fine-tuning with a tailored strategy, facilitating
category information. In general, most medical SAMs either the UN-SAM with superior segmentation performance.
demand huge computational resources in fine-tuning [30] Specifically, the DT-Encoder follows the ViT [36] architec-
or rely on manual annotations for prompt during inference ture of the SAM encoder [11], consisting of L transformer
[14]–[18], which is impractical for nuclei images containing layers. To enhance the transformer layer with the domain-
hundreds or thousands of nuclei in each image. Our UN-SAM adaptive capability, we devise the domain bypass to process the
4 IEEE TRANSACTIONS ON MEDICAL IMAGING

Trainable In-loop ⊙ Dot product

DQ-Decod
Trainable In-loop ⊙ Dot product Domain query
DQ-Decoder Frozen
{𝒒% }&
%"#
Blocked ⨁ Element-wise addition
Frozen Blocked ⨁ Element-wise addition ℎ$ 𝑔
ℎ$ 𝑔$ DT-Encoder 𝑳𝐒𝐞𝐠
DT-Encoder

Cross Att.
Self Att.

MLP
⨁
⨁ Cross Att.

Transformer
Transformer
MLP

Patch embedding
Trans

Layer
Trans Conv
Patch embedding

Layer
Reverse
⊙

Conv
Cross Att.

Self-prompt

SPGen
𝑔$
{ℎ! }$!"# Multi-scale
Thresh
Fusion {ℎ! }$!"#
𝑔
𝑳𝐒𝐏𝐆𝐞𝐧

Fig. 2. The overview of the proposed UN-SAM for nuclei image segmentation, consisting of DT-Encoder, SPGen and DQ-Decoder. For ease of
understanding, we elaborate on the case of UN-SAM with four nuclei image domains. Our UN-SAM can achieve superior generalization performance
on these domains without the need for manual annotations.
𝒘! (𝒘%&' ℎ )
{𝒘! }$
!"#
MLP Domain 𝒘! (𝒘%&' ℎ )
image embedding h after the multi-head𝒘self-attention (MSA)

Trans Conv
!
LN Bypass {𝒘! }$
!"#
operation, as illustrated
ℎ
in Fig. 3. For each transformer layer, MLP Domain
we introduce a learnable set of domain-common embedding LN Bypass 𝒘!
MSA
wcom andLN
domain-specific
ℎ embeddings {wk }K k=1 for the do- ℎ
𝒘%&'
main bypass, where K is the number of nuclei domains. Note MSA
that we only update the learnable wcom and {wk }K
the UN-SAM fine-tuning, and keep the original parameters
k=1 in
Img+ LN ℎ
𝒘%&'

in SAM frozen.
prom Im

Attention
Cross
Fig. 3. The illustration of the transformer layer with the domain bypass

MLP
In the domain bypass, we first leverage the domain-common
pt indomain-specific
the DT-Encoder. In the domain bypass, the domain-common and

pro
embeddings wcom to process the image embedding h using the embeddings process image embeddings in sequence.
common knowledge of K nuclei domains, and then enhance

pt
the image embedding h with the corresponding domain-
specific knowledge, e.g., using the domain-specific embedding Encoder can unleash the generalization capability of UN-SAM
wk for k-th nuclei domain. After that, the output of the with both domain-common and domain-specific knowledge.
domain bypass is added to enhance the image embedding in
the residual, as follows:
C. Multi-scale Self-prompt Generation
h ← wk (wcom (h)) + MLP(h), (1)
The standard SAM [11] and medical SAM [14], [15]
where MLP is the multi-layer perceptron (MLP) after the layer
demand manual annotations as prompts during image segmen-
normalization (LN) operation. In this way, the domain bypass
tation, e.g., the point prompt and bounding box prompt. In
in each transformer layer enables the generalization of the
practice, manually labeling points and even bounding boxes of
DT-Encoder across a variety of nuclei images by introducing
medical images are time-consuming and expensive in clinical
a slight amount of learnable parameters.
scenarios, especially for histopathology images with numerous
Furthermore, to provide the DT-Encoder with adequate
nuclei. To address this issue, we propose a multi-scale Self-
nuclei image knowledge while freezing SAM parameters, we
Prompt Generation (SPGen) module that can automatically
devise a domain inheritance strategy for domain-common em-
provide a set of high-quality self-prompt with multi-scale
bedding to fully exploit discriminative features for nuclei im-
knowledge to facilitate nuclei segmentation tasks.
age segmentation. When our UN-SAM is trained on different
As depicted in Fig. 2, the proposed SPGen module first
domains sequentially, we let the domain-common embedding
k+1 takes the image embeddings {hl }L l=1 from different layers of
of the next domain wcom inherit from the previous domain
k DT-Encoder as the input, and performs the multi-scale fusion
wcom to maintain continuous learning on nuclei segmentation,
to generate the multi-scale image embeddings hms . To achieve
as follows:
k+1 k this, we adopt convolutional layer heads with different strides
wcom ← wcom . (2)
on {hl }Ll=1 to produce the image embeddings at different
Therefore, the DT-Encoder can exploit nuclei image knowl- scales, and then apply the Feature Pyramid Network (FPN)
edge adequately and make the fine-tuning efficient. In contrast, [37] to integrate multi-level features from top to down. As
the domain-specific embedding is used for the corresponding such, the generated multi-scale image embeddings hms are
domain of nuclei images, which makes the DT-Encoder adap- capable to perceive nuclei with different sizes and exploit
tive to different nuclei domains. In this way, the proposed DT- discriminative representations for the segmentation hint.
UN-SAM: UNIVERSAL PROMPT-FREE SEGMENTATION FOR GENERALIZED NUCLEI IMAGES 5

After that, to facilitate the decoding of nuclei segmentation, Algorithm 1: UN-SAM pipeline across varying domains.
the SPGen module generates the self-prompt by filtering out Input : The network structure: E, S, D;
high-quality foreground regions. First, we apply a 1 × 1 Learnable wcom , {wk }K k K
k=1 , {q }k=1 ;
convolutional head on the multi-scale image embedding hms to Domains: K, Epochs: M ;
predict the foreground regions. The foreground segmentation Output: The well-trained UN-SAM.
is optimized by the binary cross-entropy loss, as follows: 1: Initialize the model parameters E, S, D for all domains;
N 2: for domain k = 1 to K do
1 X
LSPGen = − yn log(σ(gn )) + (1 − yn ) log(1 − σ(gn )), 3: for epoch m = 1 to M do
N n=1
4: Perform DT-Encoder: {hl }L k
l=1 ← E(x; wcom , w );
(3) 5: L
Perform SPGen: ĝ ← S({hl }l=1 );
where yn is the n-th token of ground truth, and N is the 6: Perform DQ-Decoder: ŷ ← D(hL , ĝ; q k );
token numbers of image embeddings. The sigmoid function 7: Compute LSPGen in Eq. (3);
σ processes gn into the predicted foreground probability, and 8: Compute LSeg in Eq. (7);
a higher score means the corresponding token plays a more 9: Optimize S and D with learnable wcom , wk , q k .
important role in nuclei prediction. To avoid the generation of 10: end for
noisy hints and reduce the false positive error in mask predic- 11: Update wcom in E: wcomk+1
← wcomk
in Eq. (2);
tion, we dynamically retain the tokens with high probability 12: end for
and filter out low-confidence tokens prone to misclassification,
as follows: (
gn , if σ (gn ) ⩾ τ
ĝn = (4) embedding f with the updated domain query embeddings q k ,
0, otherwise
as follows:
where τ is a threshold to determine the high-quality fore- (f + ψ) · (q k )T
ground tokens. In this way, the proposed SPGen module can f ← softmax( √ ) · q k + f, (6)
d
produce a set of high-confidence self-prompt tokens ĝ =
{ĝn }N Following the standard SAM [11], we repeat the decoder block
n=1 to promote the nuclei segmentation, and eliminate
the need for UN-SAM on manual annotations. of the above processes twice to obtain updated domain queries
and image embedding.
After that, we upscale the domain query-enhanced image
D. Domain Query-enhanced Decoder embedding f with two transposed convolutional layers into
The decoder in the standard SAM [11] and medical SAM the targeted resolution. Meanwhile, we update the domain
[14], [15] adopted the learnable output token as the query to query with a cross-attention layer with the image embedding,
generate the corresponding segmentation mask. However, it is followed by a MLP to adjust the channel dimension the same
challenging to leverage the same set of query embeddings to as the upscaled image embedding. Finally, the DQ-Decoder
process nuclei images in different domains. To further promote generates the segmentation prediction by performing the dot
the generalization of UN-SAM across a variety of domains, we product between the upscaled image embedding and the k-
propose a Domain Query-enhanced Decoder (DQ-Decoder) to th domain query. In this way, the domain information in the
generate accurate segmentation masks with domain-specific DQ-Decoder does not interfere with each other, which further
query embeddings. improves the generalization of the proposed UN-SAM.
To enhance the image embeddings with foreground hints,
the DQ-Decoder takes the self-prompt ĝ from the SPGen E. Optimization Pipeline
module and the image embedding hL from the DT-Encoder
as input, and performs element-wise addition, as f = hL + ĝ. We summarize the training pipeline of UN-SAM across
Meanwhile, to facilitate UM-SAM to segment a variety of varying nuclei domains in Algorithm III-D. We first initialize
nuclei images, we set a group of domain query embeddings the UN-SAM including the DT-Encoder (E), SPGen module
{q k }K k N ×d
is for k-th domain, and q k has (S) and DQ-Decoder (D), and particularly we utilize the pre-
k=1 , where q ∈ R
the same shape as f . As such, the DQ-Decoder first performs trained SAM [11] to initialize corresponding parameters in the
the self-attention layer based on domain query embeddings q k , DT-Encoder and keep them frozen. The UN-SAM is optimized
followed by the cross-attention layer with the hint-enhanced on varying nuclei domains in sequence, and at the end of each
image embedding f to update the domain query embeddings domain, the wcom inherits the parameters from the previous
q k , as follows: domain ones. For each batch of k-th domain nuclei images, the
DT-Encoder generates the domain-adaptive features {hl }L l=1 ,
T
q k · (f + ψ) then the SPGen module provides the self-prompt ĝ as the hint
q k ← softmax( √ ) · f + qk , (5) for segmentation, and finally the DQ-Decoder generates the
d
segmentation prediction ŷ. The predicted segmentation mask
where · is the matrix multiplication, and the positional encod- ŷ on each domain is supervised by the weighted combination
ing ψ is inserted to improve the dependence between geomet- of focal loss Lfocal and dice loss Ldice [11], as follows:
ric location and type. After the MLP, the DQ-Decoder further
adopts the reverse cross-attention layer to update the image LSeg = λLfocal + (1 − λ)Ldice , (7)
6 IEEE TRANSACTIONS ON MEDICAL IMAGING

nnU-Net* Med-SA+ UN-SAM Ground Truth

where λ is the coefficient to balance these two loss terms. By
optimizing LSeg and LSPGen , our UN-SAM achieves accu-

DSB
rate segmentation with superior generalization across different
domains, without the need for manual annotations.

MoNuSeg
IV. E XPERIMENT
A. Datasets and Implementations
1) Datasets: To validate the effectiveness of the proposed
UN-SAM, we adopt four nuclei segmentation datasets, in-

TNBC
cluding the DSB [2], MoNuSeg [27], TNBC [28], SegPC
[3] datasets, to perform comprehensive cross-domain compar-
isons, and one additional CryoNuSeg [38] dataset to evaluate
zero-shot generalization capability.
SPGen
TheUN-SAM
details are as follows:
Ground Truth Fig. 4. Visualization of generalized nuclei SAMed
instance segmentation.
U-Net* nnU-Net* + Med-SA+ SPGen
DSB [2] dataset, derived from the 2018 Data Science Bowl + indicates medical SAMs using point prompts, and ∗ indicates that
challenge, contains 670 microscopic slides of different image classical methods are trained and tested separately for each dataset.
Our UN-SAM segments more nuclei with accurate boundaries while

DSB
types across diverse cell lines, imaging conditions, and staining having fewer false positives.
protocols. The image size varies from 256 × 256 to 696 × 520.
MoNuSeg [27] is a nuclei segmentation dataset collected from
multiple organs, including the breast, liver, kidney, prostate,

MoNuSeg
distance (HD). For nuclei instance segmentation, we compare
bladder, colon and stomach. It contains 44 H&E stained the performance with four metrics, including the aggregated
images with the resolution of 1, 000 × 1, 000. Jaccard index (AJI), detection quality (DQ), segmentation
TNBC [28] is a nuclei segmentation dataset from triple- quality (SQ), and panoptic quality (PQ). Except for HD, higher
negative breast cancer patients, and contains 50 histopathology scores for these metrics indicate better segmentation quality.
TNBC

images of 512×512 resolution captured at 40× magnification.

SegPC [3] is a multiple myeloma plasma cell segmentation
dataset, and contains 498 nuclei images annotated with two B. Comparison on Nuclei Instance Segmentation
categories of the cytoplasm and nucleus. Each image is either To evaluate our UN-SAM in nuclei instance segmenta-
SegPC

2, 040 × 1, 536 or 2, 560 × 1, 920. tion, we perform the comparison with state-of-the-art instance
CryoNuSeg [38] is a cryosectioned nuclei segmentation segmentation methods [4], [5], nuclei instance segmentation
dataset of H&E stained tissues from 10 different organs, and methods [6]–[8] and medical SAM [15]–[18] on the DSB,
contains 30 slides of 512×512 captured at 40× magnification. MoNuSeg and TNBC datasets. Except for the vanilla SAM
We keep this dataset invisible from model training to evaluate [11] as the baseline for foundation models, we guarantee
zero-shot generalization capability. fair comparisons and fine-tune all models with the same
2) Implementation Details: We perform all experiments on training set to adapt to nuclei images. First, we evaluate the
a single NVIDIA A800 GPU using PyTorch. For a fair com- performance of all models fine-tuned on each single nuclei
parison, we implement all nuclei segmentation methods with domain, as illustrated in Table I. We observe that the fine-
the same training settings and configurations, where all SAM tuned SAM methods [15]–[18] are inferior to nuclei instance
models use ViT-H [36] structure as the image encoder. We segmentation methods [6]–[8] when manual prompt is unavail-
perform the optimization using Adam with the batch size of 4 able. In contrast, given each nucleus centroid as the point
for 30 epochs. The learning rate is initialized as 1 × 10−4 and prompt, these medical SAMs reveal the advantage of founda-
adjusted using the exponential decay strategy with the factor tion models, e.g., Med-SA [18] surpasses CPP-Net [7] with a
as 0.98. For our UN-SAM, we set the foreground threshold τ 1.37% AJI increase on the MoNuSeg dataset. Remarkably, our
as 0.95 in the SPGen, and the loss coefficient λ as 0.8 during UN-SAM, without relying on manual prompts, achieves the
the training. To generate instance segmentation predictions, we best performance among different domains of instance nuclei
use the connectedComponents function in OpenCV to identify segmentation, including AJI of 78.75%, 50.59% and 59.48%
each nucleus from semantic segmentation masks, without bells on the respective datasets. These comparisons prove the ef-
and whistles. In the comparison, the medical SAM methods ficacy of our UN-SAM, showcasing its superior performance
[15]–[18] is fine-tuned and evaluated under the same dataset on diverse nuclei segmentation datasets without the need for
protocol as our UN-SAM. We implement two modes of these manual annotation.
medical SAM methods, including the automatic mode without Furthermore, to evaluate the generalization capability across
manual annotations [11] and the prompt mode using the different nuclei domains, we compare the performance of the
centroid of each nucleus instance as point prompts. UN-SAM and SAM-based methods on the DSB, MoNuSeg
3) Evaluation Metrics: To perform the comprehensive evalu- and TNBC datasets after fine-tuning them on these three
ation of nuclei segmentation, we adopt diverse metrics in terms datasets sequentially, as shown in Table II. Our UN-SAM
of semantic segmentation and instance segmentation. For reveals the overwhelming generalization performance of four
nuclei semantic segmentation, we select the Dice coefficient, metrics on these datasets. When manual prompts are unavail-
mean intersection over union (mIoU), F1 score, and Hausdorff able for medical SAM, our UN-SAM achieves a significant
UN-SAM: UNIVERSAL PROMPT-FREE SEGMENTATION FOR GENERALIZED NUCLEI IMAGES 7

TABLE I
C OMPARISON WITH S TATE - OF - THE - ARTS ON N UCLEI I NSTANCE S EGMENTATION .

Manual DSB MoNuSeg TNBC

Methods
Prompt AJI DQ SQ PQ AJI DQ SQ PQ AJI DQ SQ PQ
U-Net [21] 71.53 81.37 83.42 68.89 34.98 50.94 68.59 34.96 50.52 68.02 74.24 50.52
nnU-Net [20] 74.57 84.49 84.07 72.11 45.22 65.12 71.78 48.81 56.48 75.77 77.97 59.30
Mask-RCNN [4] 71.80 82.07 83.11 69.23 36.15 52.17 70.03 37.46 51.04 67.33 75.01 50.83
SparseInst [5] ✘ 72.50 83.67 83.93 71.35 40.19 55.89 70.01 40.35 53.48 68.26 76.04 53.94
HoVer-Net [6] 74.91 85.23 84.13 73.45 46.76 66.37 70.92 47.58 56.45 76.19 77.65 59.11
CDNet [8] 74.87 84.89 84.30 73.12 43.49 59.77 70.16 43.98 54.25 74.28 76.13 56.85
CPP-Net [7] 75.54 86.95 84.58 74.35 44.63 60.57 70.53 42.44 52.20 69.17 74.75 51.67
Vanilla SAM [11] 22.30 16.17 49.32 12.96 7.36 8.55 50.14 6.08 8.42 8.95 38.92 5.90
SAMMI [15] 63.19 73.93 72.54 59.90 27.02 45.45 57.39 28.88 45.65 57.58 68.88 41.46
SAMUS [16] ✘ 62.11 74.14 72.52 59.97 29.58 47.14 58.17 31.88 45.35 58.85 68.06 41.39
SAMed [17] 65.75 75.19 74.48 62.58 29.72 46.93 60.52 30.21 46.52 59.69 68.36 40.21
Med-SA [18] 63.50 75.78 73.09 61.73 31.49 47.71 61.31 32.32 47.50 59.67 70.39 43.32
Vanilla SAM [11] 36.67 45.10 62.47 36.16 14.11 14.64 53.46 9.32 20.54 24.24 68.07 16.73
SAMMI [15] 75.29 85.82 84.21 72.82 44.83 61.53 71.41 44.01 57.33 70.61 76.97 54.40
SAMUS [16] point 74.93 86.63 83.86 73.09 43.80 61.71 71.59 44.24 56.53 73.95 77.71 57.47
SAMed [17] 75.58 86.82 84.10 73.55 44.46 61.64 71.03 43.81 53.77 67.85 76.23 51.73
Med-SA [18] 74.82 86.49 84.48 73.59 46.00 65.90 71.87 47.39 57.00 76.79 77.72 59.72
UN-SAM (Ours) ✘ 78.75 92.25 86.79 78.23 50.59 70.37 72.88 50.86 59.48 78.13 78.64 61.47

TABLE II
C OMPARISON OF G ENERALIZATION C APABILITY ON N UCLEI I NSTANCE S EGMENTATION .

Manual DSB MoNuSeg TNBC

Methods
Prompt AJI DQ SQ PQ AJI DQ SQ PQ AJI DQ SQ PQ
Vanilla SAM [11] 22.30 16.17 49.32 12.96 7.36 8.55 50.14 6.08 8.42 8.95 38.92 5.90
SAMMI [15] 62.37 73.04 71.96 58.56 26.13 44.68 55.97 27.98 43.92 56.17 66.42 40.59
SAMUS [16] ✘ 61.03 72.98 71.26 58.74 27.32 46.91 56.13 30.27 44.16 57.54 67.52 40.78
SAMed [17] 63.98 74.12 73.33 60.93 28.56 45.81 59.04 29.97 45.37 58.14 67.96 39.15
Med-SA [18] 63.26 75.16 72.29 60.68 30.33 46.62 60.04 31.15 46.43 58.59 68.74 42.08
Vanilla SAM [11] 36.67 45.10 62.47 36.16 14.11 14.64 53.46 9.32 20.54 24.24 68.07 16.73
SAMMI [15] 72.37 83.36 83.58 69.75 41.04 58.42 70.49 41.21 52.04 63.91 74.84 53.12
SAMUS [16] point 72.46 84.52 83.47 70.12 43.15 61.04 71.39 43.56 55.72 71.24 76.84 56.07
SAMed [17] 72.81 85.28 83.35 71.36 42.57 60.85 70.76 41.99 51.95 68.30 75.23 50.16
Med-SA [18] 73.14 85.53 83.38 71.80 44.24 63.44 71.04 45.11 56.19 73.07 77.24 56.91
UN-SAM (Ours) ✘ 75.56 89.18 84.67 76.12 49.37 69.16 71.74 49.52 59.22 77.91 78.60 61.28

advantage over the second-best Med-SA [18], e.g., a PQ in- methods [15]–[18] benefit from manual prompts and reveal
crease of 15.44%, 18.37% and 19.20% on the DSB, MoNuSeg remarkable performance over classical medical segmentation
and TNBC datasets, respectively. Even compared with the methods, e.g., Med-SA [18] has a 2.05% Dice increase over
medical SAM using nucleus centroids as prompt (denote as nnU-Net [20] on the TNBC dataset. Our UN-SAM further
point), our UN-SAM also achieves superior performance, by promotes SAM capability on nuclei segmentation tasks, and
outperforming these medical SAM models [17], [18], [20] achieves overwhelming performance on these four datasets,
with the P-value < 0.005 in both AJI and PQ. We further with the best Dice of 93.12%, 84.17%, 85.72% and 89.01%,
compare the quantitative results of our UN-SAM with the best respectively.
baseline methods [18], [20] in Fig. 4, and our UN-SAM can
segment nuclei more accurately with better boundaries. In this Moreover, we compare the generalization capability of
way, these results demonstrate the significant generalization SAM-based methods across different nuclei domains after fine-
advantage of our UN-SAM over medical foundation models tuning them on these four datasets sequentially, as shown in
on different nuclei segmentation datasets. Table IV. Our UN-SAM achieves the best performance on
these four domains of nuclei images simultaneously, which
confirms the advantage of generalization capability in semantic
C. Comparison on Nuclei Semantic Segmentation segmentation tasks. In particular, our UN-SAM surpasses
To perform the universal evaluation of our UN-SAM in the second-best Med-SA [18] without manual prompts, by
nuclei segmentation, we further conduct comparisons on nuclei a remarkable Dice increase of 11.30%, 19.29%, 9.62% and
semantic segmentation with advanced medical segmentation 13.47% on these four datasets, respectively. Compared with
algorithms. First, we compare the performance of models point-prompted medical SAMs, our UN-SAM reveals more
fine-tuned on each single nuclei domain, as shown in Table than 2% advantage of Dice in most domains. We further
III. Among medical segmentation methods, nnU-Net [20] elaborate on the semantic segmentation results in Fig 6,
achieves leading results in nuclei semantic segmentation tasks, and these four datasets reveal significant data heterogeneity
and also outperforms fine-tuned SAM methods when manual due to different tissue types, staining protocols, and imaging
prompts are unavailable. In contrast, these fine-tuned SAM conditions. Our UN-SAM generates the best segmentation
8 IEEE TRANSACTIONS ON MEDICAL IMAGING

TABLE III
C OMPARISON WITH S TATE - OF - THE - ARTS ON N UCLEI S EMANTIC S EGMENTATION .
Manual DSB MoNuSeg TNBC SegPC
Methods
Prompt Dice mIoU F1 HD Dice mIoU F1 HD Dice mIoU F1 HD Dice mIoU F1 HD
U-Net [21] 88.16 81.42 89.49 32.57 74.06 60.25 75.57 18.94 80.64 67.62 81.00 38.39 84.89 76.20 86.75 58.17
Unet++ [25] 90.48 83.53 90.96 28.03 76.78 62.97 78.31 18.53 81.19 68.44 81.60 37.70 85.61 77.10 87.04 56.61
Attention-UNet [24] 91.38 84.76 91.79 24.51 76.89 62.92 77.45 21.63 81.25 68.59 81.84 40.68 84.25 75.91 85.75 54.07
ResUNet++ [39] 89.76 82.31 90.19 33.13 77.96 64.05 78.85 19.45 76.81 62.55 77.53 33.20 83.06 73.52 84.52 62.47
R2U-Net [22] ✘ 88.56 81.82 89.02 32.64 78.87 65.32 79.35 21.34 76.19 61.93 78.33 40.12 83.79 74.67 85.49 58.11
DoubleU-Net [40] 91.09 84.43 91.62 27.47 78.20 64.43 78.81 19.01 81.59 68.98 81.83 40.66 86.02 77.68 87.49 53.41
UNet3+ [41] 91.39 84.78 91.77 25.76 78.03 64.25 78.46 19.25 80.90 68.05 81.13 35.41 84.49 75.76 86.11 55.72
TransUNet [23] 90.76 83.73 91.08 31.38 75.08 61.41 76.55 20.47 78.30 64.37 77.65 40.94 82.91 73.85 84.53 58.37
nnU-Net [20] 91.61 85.00 91.96 30.29 81.09 68.28 81.24 18.15 82.11 69.74 82.35 31.71 87.35 79.32 88.48 52.09
Vanilla SAM [11] 36.49 25.83 49.04 95.03 20.28 12.62 25.25 33.72 20.44 12.53 35.80 95.09 39.88 19.17 40.40 201.56
SAMMI [15] 82.31 74.61 82.72 37.23 58.55 42.13 59.85 27.13 71.44 58.55 71.72 43.86 75.74 64.37 77.52 87.71
SAMUS [16] ✘ 82.25 74.42 82.57 35.52 64.96 50.70 65.11 25.25 76.60 62.56 76.84 41.42 77.45 66.65 79.22 90.47
SAMed [17] 82.62 75.07 82.94 35.36 65.76 51.82 65.96 26.02 75.74 61.42 75.87 40.37 77.96 67.22 79.74 86.72
Med-SA [18] 82.84 75.24 83.14 34.72 64.91 51.02 65.06 25.25 77.59 62.95 78.63 40.65 77.59 66.92 78.87 86.31
Vanilla SAM [11] 70.72 59.10 74.56 44.03 37.54 22.47 46.33 23.01 83.46 46.78 64.83 39.19 62.67 44.78 72.74 154.98
SAMMI [15] 91.42 84.79 91.83 21.49 78.64 64.92 78.87 19.23 80.45 67.43 80.58 30.91 86.11 77.77 87.33 49.20
SAMUS [16] point 92.10 85.74 92.38 29.95 81.82 69.33 82.00 18.34 83.94 71.94 84.07 29.36 87.52 79.85 88.70 53.69
SAMed [17] 92.39 86.27 92.66 27.20 82.26 69.94 82.43 18.64 83.19 71.23 83.33 33.82 86.87 78.68 87.90 55.90
Med-SA [18] 92.52 86.45 92.80 24.05 82.47 70.24 82.57 18.21 84.16 72.68 84.32 29.11 87.39 79.54 88.54 55.87
UN-SAM (Ours) ✘ 93.12 87.41 93.30 20.90 84.17 72.93 84.27 16.55 85.72 75.02 85.81 26.83 89.01 82.14 89.88 46.15

TABLE IV
C OMPARISON OF G ENERALIZATION C APABILITY ON N UCLEI S EMANTIC S EGMENTATION .
Manual DSB MoNuSeg TNBC SegPC
Methods
Prompt Dice mIoU F1 HD Dice mIoU F1 HD Dice mIoU F1 HD Dice mIoU F1 HD
Vanilla SAM [11] 36.49 25.83 49.04 95.03 20.28 12.62 25.25 33.72 20.44 12.53 35.80 95.09 39.88 19.17 40.40 201.56
SAMMI [15] 77.63 71.02 78.50 39.11 57.71 41.47 58.41 30.41 68.71 56.47 69.41 43.41 72.61 60.56 75.67 102.68
SAMUS [16] ✘ 80.85 73.82 81.29 37.18 62.77 48.41 75.10 26.07 74.18 60.56 74.79 40.24 73.47 61.91 75.38 86.21
SAMed [17] 80.87 73.85 81.23 37.73 64.42 50.61 65.54 25.14 73.51 59.67 73.81 40.73 75.47 64.46 77.46 80.21
Med-SA [18] 81.54 74.78 81.84 36.07 63.57 49.15 64.83 25.96 74.27 60.49 75.42 39.42 74.70 62.24 75.92 81.07
Vanilla SAM [11] 70.72 59.10 74.56 44.03 37.54 22.47 46.33 23.01 83.46 46.78 64.83 39.19 62.67 44.78 72.74 154.98
SAMMI [15] 90.37 83.15 90.77 33.67 77.70 63.80 78.07 19.61 77.77 63.83 78.19 38.93 85.19 76.36 85.19 67.43
SAMUS [16] point 91.39 84.69 91.76 29.45 78.11 64.22 78.41 18.75 81.28 68.48 81.42 33.18 86.49 78.32 87.75 55.25
SAMed [17] 91.49 84.89 91.83 30.92 80.14 66.96 80.37 18.78 81.69 66.74 80.22 32.43 86.55 78.37 87.67 53.98
Med-SA [18] 92.16 85.99 92.46 25.35 80.79 67.87 81.01 17.68 81.99 69.58 82.29 34.65 86.13 77.79 87.56 49.26
UN-SAM (Ours) ✘ 92.84 86.93 93.03 20.93 82.86 70.82 82.96 16.69 83.89 72.27 84.05 27.71 88.17 81.06 89.05 48.93

87.4
93.0 28.0 of all datasets gain of 7.89%, 1.93%, 7.56%, respectively. We
87.0 additionally investigate the effect of combined DT-Encoder
HD (voxel)

26.0
mIoU (%)
Dice (%)

92.8
86.6 24.0 and DQ-Decoder (5th row), resulting in superior performance,
92.6
22.0 with the Dice of 92.63%, 82.73%, 83.33%, 88.00% and mIoU
92.4 86.2 of 86.65%, 70.53%, 71.48%, 81.02% on four datasets. The
0.5 0.6 0.7 0.8 0.9 0.95 0.96 1 0.5 0.6 0.7 0.8 0.9 0.95 0.96 1
Threshold Threshold result proves that these two improvements can promote the
generalization across different nuclei domains. By comparing
Fig. 5. Hyper-parameter analysis of confidence threshold on nuclei
semantic segmentation of the DSB dataset. 6th and 7th rows with 2nd and 4th rows, configurations with
SPGen show competitive performance compared to the manual
prompt encoder of SAM, which overcomes the laborious
results, especially reducing false positive predictions of nuclei process of manual prompt generation and facilitates clinical
significantly. These comparisons validate the superiority of our workflows for high-throughput analysis. On this basis, our
UN-SAM on nuclei semantic segmentation tasks and better UN-SAM (8th row) simultaneously adopts DT-Encoder and
generalization performance across different nuclei domains. DQ-Decoder to further address both issues. In this way, these
ablation experiments prove that the tailor-made DT-Encoder,
SPGen module, and DQ-Decoder play a significant role to
D. Ablation Study
facilitate our UN-SAM on nuclei segmentation by eliminating
To investigate the effectiveness of the DT-Encoder E, the annotation need and enhancing the generalization across
SPGen module S and DQ-Decoder D, we further conduct the varying nuclei domains.
comprehensive ablation study on the semantic segmentation
of four nuclei datasets, as illustrated in Table V. By removing It is worth noting that the confidence threshold τ in the
the tailored modules from UN-SAM, the fine-tuned SAM (1st SPGen module is crucial for UN-SAM, by determining the
row) serves as the ablation baseline. By separately introducing high-quality self-prompt to guide the segmentation decoding.
the DT-Encoder (2nd row), SPGen (3rd row) and DQ-Decoder For the possible value range as 0.5 ≤ τ ≤ 1, we perform a
(4th row), the performance is improved with the average Dice grid search with the Dice, mIoU and HD metrics on the DSB
 TNBC
UN-SAM: UNIVERSAL PROMPT-FREE SEGMENTATION FOR GENERALIZED NUCLEI IMAGES 9

Ground Truth
U-Net* nnU-Net* SAMed+ Med-SA+ SPGen UN-SAM Ground Truth

DSB
MoNuSeg
TNBC
SegPC

Fig. 6. Visualization of generalized nuclei semantic segmentation. + indicates medical SAMs using point prompts, and ∗ indicates that classical
methods are trained and tested separately for each dataset. Our UN-SAM exhibits the best results, segmenting more nuclei with accurate boundaries
while having fewer false positives. The SPGen produces the self-prompt to provide high-quality segmentation hints for our UN-SAM.

TABLE V
A BLATION S TUDY OF UN-SAM ON N UCLEI S EMANTIC S EGMENTATION .
DSB MoNuSeg TNBC SegPC-2021
E S D
Dice mIoU F1 HD Dice mIoU F1 HD Dice mIoU F1 HD Dice mIoU F1 HD
1 90.08 82.62 90.59 36.82 72.43 57.15 73.13 18.56 76.83 62.65 77.54 36.15 73.47 61.91 75.38 86.21
2 ✓ 92.14 85.96 92.46 30.98 82.27 70.02 82.41 17.41 82.75 70.61 82.84 31.25 87.20 79.27 88.34 54.79
3 ✓ 90.18 82.77 90.72 31.74 76.47 62.04 76.69 17.62 78.92 65.32 79.32 33.13 74.95 63.35 76.86 76.66
4 ✓ 91.90 85.38 92.21 28.23 82.19 69.85 82.38 17.37 82.57 70.34 82.69 31.59 86.37 78.12 87.73 55.62
5 ✓ ✓ 92.63 86.65 92.87 25.51 82.73 70.53 82.81 16.94 83.33 71.48 83.41 30.12 88.00 81.02 89.03 51.37
6 ✓ ✓ 92.44 86.32 92.69 21.73 82.42 70.12 82.50 16.78 83.02 70.98 83.24 28.16 87.89 79.91 88.82 49.25
7 ✓ ✓ 92.19 85.92 92.53 24.44 82.22 69.88 82.38 17.06 82.71 70.51 82.80 29.11 87.20 79.10 88.28 49.61
8 ✓ ✓ ✓ 92.84 86.93 93.03 20.93 82.86 70.82 82.96 16.69 83.89 72.27 84.05 27.71 88.17 81.06 89.05 48.93

dataset. As shown in Fig. 5, our UN-SAM achieves the best TABLE VI

performance when τ is 0.95, where a lower threshold would Z ERO - SHOT G ENERALIZATION C OMPARISON FOR C RYO N U S EG .
cause interference to the decoder and a larger threshold cannot Manual
Methods Mode Dice mIoU F1 HD
provide enough prompt information for nuclei segmentation. Prompt
In general, the proposed SPGen module with the appropriate U-Net [21]
fine-tune ✘
80.15 67.22 80.19 25.24
nnU-Net [20] 82.15 69.56 82.22 23.15
confidence threshold can significantly benefit the UN-SAM
and eliminate the dependence on manual annotation for seg- Vanilla SAM [11] 40.86 26.19 43.87 86.13
SAMMI [15] 64.14 50.73 65.19 37.88
mentation prompts. SAMUS [16] zero-shot ✘ 65.52 51.01 66.63 35.43
SAMed [17] 67.01 53.53 68.61 36.00
Med-SA [18] 67.16 53.60 68.82 34.36
E. Comparison on Zero-shot Generalization Vanilla SAM [11] 61.05 44.58 66.29 30.51
To further validate the generalization capability of our SAMMI [15] 75.49 61.39 76.25 29.66
SAMUS [16] zero-shot point 77.78 64.36 78.50 25.32
UN-SAM, we compare zero-shot generalization of medical SAMed [17] 78.20 64.85 78.93 25.29
foundation models (i.e., the model weights in Table IV) on Med-SA [18] 78.46 65.18 79.10 25.80
the CryoNuSeg [38] test set. As demonstrated in Table VI, UN-SAM (Ours) zero-shot ✘ 80.42 67.67 80.67 25.18
our UN-SAM achieves the best performance among medical
SAMs, with the overwhelming Dice of 80.42% and mIoU
of 67.67%. In particular, our UN-SAM reveals significant our UN-SAM has superior zero-shot generalization capability,
performance advantages with the P-value < 0.005 compared to with more potential to be applied in clinical scenarios.
all medical SAMs [15]–[18] with or without manual prompts.
Moreover, we present medical segmentation methods, i.e., U-
V. C ONCLUSION
Net [21] and nnU-Net [20] fine-tuned on the CryoNuSeg
training set. Remarkably, our UN-SAM outperforms the fine- In this work, we propose the UN-SAM framework to
tuned U-Net [21], and is approaching the fined-tuned nnU- achieve a universal solution for nuclei segmentation within
Net [20] as the upper bound. These comparisons confirm that the realm of digital pathology. In particular, the DT-Encoder is
10 IEEE TRANSACTIONS ON MEDICAL IMAGING

proposed to harmonize domain-common and domain-specific [19] Q. Xu, W. Kuang, Z. Zhang, X. Bao, H. Chen, and W. Duan, “Sppnet: A
features within the image encoder, which propels the gen- single-point prompt network for nuclei image segmentation,” in MICCAI
Workshop on MLMI. Springer, 2023, pp. 227–236.
eralization potential of SAM across diverse nuclei datasets. [20] F. Isensee, P. F. Jaeger, S. A. Kohl, J. Petersen, and K. H. Maier-Hein,
Then, the SPGen module is devised to autonomously pro- “nnu-net: a self-configuring method for deep learning-based biomedical
duce high-quality mask hints instead of manual annotated image segmentation,” Nature Methods, vol. 18, no. 2, pp. 203–211, 2021.
[21] O. Ronneberger, P. Fischer, and T. Brox, “U-net: Convolutional networks
prompt, to facilitate the clinical workflow towards enhanced for biomedical image segmentation,” in MICCAI. Springer, 2015, pp.
efficiency. Moreover, we further propose the DQ-Decoder to 234–241.
leverage learnable domain queries for segmentation decoding [22] M. Z. Alom, C. Yakopcic, M. Hasan, T. M. Taha, and V. K. Asari,
“Recurrent residual u-net for medical image segmentation,” J. Med.
in different nuclei domains. Extensive experiments confirm Imaging, vol. 6, no. 1, pp. 014 006–014 006, 2019.
the advantage of UN-SAM over existing medical SAMs by [23] J. Chen, Y. Lu, Q. Yu, X. Luo, E. Adeli, Y. Wang, L. Lu, A. L. Yuille, and
negating the need for labor-intensive manual annotations and Y. Zhou, “Transunet: Transformers make strong encoders for medical
image segmentation,” arXiv preprint arXiv:2102.04306, 2021.
offering an unparalleled zero-shot learning capability that [24] J. Schlemper, O. Oktay, M. Schaap, M. Heinrich, B. Kainz, B. Glocker,
adapts to varying domains with ease. and D. Rueckert, “Attention gated networks: Learning to leverage salient
regions in medical images,” Med. Image Anal., vol. 53, pp. 197–207,
2019.
R EFERENCES [25] Z. Zhou, M. M. R. Siddiquee, N. Tajbakhsh, and J. Liang, “Unet++:
[1] N. F. Greenwald, G. Miller, E. Moen, A. Kong, A. Kagel, T. Dougherty, Redesigning skip connections to exploit multiscale features in image
C. C. Fullaway, B. J. McIntosh, K. X. Leow, M. S. Schwartz et al., segmentation,” IEEE Trans. Med. Imaging, vol. 39, no. 6, pp. 1856–
“Whole-cell segmentation of tissue images with human-level perfor- 1867, 2019.
mance using large-scale data annotation and deep learning,” Nature [26] A. Lin, J. Xu, J. Li, and G. Lu, “Contrans: Improving transformer with
Biotechnology, vol. 40, no. 4, pp. 555–565, 2022. convolutional attention for medical image segmentation,” in MICCAI.
[2] J. C. Caicedo, A. Goodman, K. W. Karhohs, B. A. Cimini, J. Ackerman, Springer, 2022, pp. 297–307.
M. Haghighi, C. Heng, T. Becker, M. Doan, C. McQuin et al., “Nucleus [27] N. Kumar, R. Verma, S. Sharma, S. Bhargava, A. Vahadane, and
segmentation across imaging experiments: the 2018 data science bowl,” A. Sethi, “A dataset and a technique for generalized nuclear segmenta-
Nature Methods, vol. 16, no. 12, pp. 1247–1253, 2019. tion for computational pathology,” IEEE Trans. Med. Imaging, vol. 36,
[3] A. Gupta, S. Gehlot, S. Goswami, S. Motwani, R. Gupta, Á. G. Faura, no. 7, pp. 1550–1560, 2017.
D. Štepec, T. Martinčič, R. Azad, D. Merhof et al., “Segpc-2021: A [28] P. Naylor, M. Laé, F. Reyal, and T. Walter, “Segmentation of nuclei in
challenge & dataset on segmentation of multiple myeloma plasma cells histopathology images by deep regression of the distance map,” IEEE
from microscopic images,” Med. Image Anal., vol. 83, p. 102677, 2023. Trans. Med. Imaging, vol. 38, no. 2, pp. 448–459, 2018.
[4] K. He, G. Gkioxari, P. Dollár, and R. Girshick, “Mask r-cnn,” in ICCV, [29] Y. Xiong, B. Varadarajan, L. Wu, X. Xiang, F. Xiao, C. Zhu,
2017, pp. 2961–2969. X. Dai, D. Wang, F. Sun, F. Iandola et al., “Efficientsam: Leveraged
[5] T. Cheng, X. Wang, S. Chen, W. Zhang, Q. Zhang, C. Huang, Z. Zhang, masked image pretraining for efficient segment anything,” arXiv preprint
and W. Liu, “Sparse instance activation for real-time instance segmen- arXiv:2312.00863, 2023.
tation,” in CVPR, 2022, pp. 4433–4442. [30] Z. Shui, Y. Zhang, K. Yao, C. Zhu, Y. Sun, and L. Yang, “Unleashing the
[6] S. Graham, Q. D. Vu, S. E. A. Raza, A. Azam, Y. W. Tsang, J. T. power of prompt-driven nucleus instance segmentation,” arXiv preprint
Kwak, and N. Rajpoot, “Hover-net: Simultaneous segmentation and arXiv:2311.15939, 2023.
classification of nuclei in multi-tissue histology images,” Med. Image [31] E. J. Hu, Y. Shen, P. Wallis, Z. Allen-Zhu, Y. Li, S. Wang, L. Wang,
Anal., vol. 58, p. 101563, 2019. and W. Chen, “LoRA: Low-rank adaptation of large language models,”
[7] S. Chen, C. Ding, M. Liu, J. Cheng, and D. Tao, “Cpp-net: Context- in ICLR, 2022.
aware polygon proposal network for nucleus segmentation,” IEEE Trans. [32] N. Houlsby, A. Giurgiu, S. Jastrzebski, B. Morrone, Q. De Laroussilhe,
Image Process., vol. 32, pp. 980–994, 2023. A. Gesmundo, M. Attariyan, and S. Gelly, “Parameter-efficient transfer
[8] H. He, Z. Huang, Y. Ding, G. Song, L. Wang, Q. Ren, P. Wei, Z. Gao, learning for nlp,” in ICML. PMLR, 2019, pp. 2790–2799.
and J. Chen, “Cdnet: Centripetal direction network for nuclear instance [33] Z. Zhong, Z. Tang, T. He, H. Fang, and C. Yuan, “Convolution meets
segmentation,” in ICCV, 2021, pp. 4026–4035. lora: Parameter efficient finetuning for segment anything model,” in
[9] S. Nam, J. Jeong, M. Luna, P. Chikontwe, and S. H. Park, “Pronet: ICLR, 2024.
Point refinement using shape-guided offset map for nuclei instance [34] W. Yue, J. Zhang, K. Hu, Y. Xia, J. Luo, and Z. Wang, “Surgicalsam:
segmentation,” in MICCAI. Springer, 2023, pp. 528–538. Efficient class promptable surgical instrument segmentation,” in AAAI,
[10] H. He, J. Wang, P. Wei, F. Xu, X. Ji, C. Liu, and J. Chen, “Toposeg: 2024.
Topology-aware nuclear instance segmentation,” in ICCV, 2023, pp. [35] C. Cui, R. Deng, Q. Liu, T. Yao, S. Bao, L. W. Remedios, Y. Tang, and
21 307–21 316. Y. Huo, “All-in-sam: from weak annotation to pixel-wise nuclei segmen-
[11] A. Kirillov, E. Mintun, N. Ravi, H. Mao, C. Rolland, L. Gustafson, tation with prompt-based finetuning,” arXiv preprint arXiv:2307.00290,
T. Xiao, S. Whitehead, A. C. Berg, W.-Y. Lo, P. Dollar, and R. Girshick, 2023.
“Segment anything,” in ICCV, October 2023, pp. 4015–4026. [36] A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai,
[12] J. Cheng, J. Ye, Z. Deng, J. Chen, T. Li, H. Wang, Y. Su, Z. Huang, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly et al.,
J. Chen, L. Jiang et al., “Sam-med2d,” arXiv preprint arXiv:2308.16184, “An image is worth 16x16 words: Transformers for image recognition
2023. at scale,” in ICLR, 2020.
[13] Y. Zhang, Z. Shen, and R. Jiao, “Segment anything model for medical [37] A. Ali, H. Touvron, M. Caron, P. Bojanowski, M. Douze, A. Joulin,
image segmentation: Current applications and future directions,” arXiv I. Laptev, N. Neverova, G. Synnaeve, J. Verbeek et al., “Xcit: Cross-
preprint arXiv:2401.03495, 2024. covariance image transformers,” NeurIPS, vol. 34, pp. 20 014–20 027,
[14] J. Ma, Y. He, F. Li, L. Han, C. You, and B. Wang, “Segment anything in 2021.
medical images,” Nature Communications, vol. 15, no. 1, p. 654, 2024. [38] A. Mahbod, G. Schaefer, B. Bancher, C. Löw, G. Dorffner, R. Ecker,
[15] Y. Huang, X. Yang, L. Liu, H. Zhou, A. Chang, X. Zhou, R. Chen, and I. Ellinger, “Cryonuseg: A dataset for nuclei instance segmentation
J. Yu, J. Chen, C. Chen et al., “Segment anything model for medical of cryosectioned h&e-stained histological images,” Comput. Biol. Med.,
images?” Med. Image Anal., vol. 92, p. 103061, 2024. vol. 132, p. 104349, 2021.
[16] X. Lin, Y. Xiang, L. Zhang, X. Yang, Z. Yan, and L. Yu, “Samus: [39] D. Jha, P. H. Smedsrud, M. A. Riegler, D. Johansen, T. De Lange,
Adapting segment anything model for clinically-friendly and generaliz- P. Halvorsen, and H. D. Johansen, “Resunet++: An advanced architecture
able ultrasound image segmentation,” arXiv preprint arXiv:2309.06824, for medical image segmentation,” in ISM. IEEE, 2019, pp. 225–2255.
2023. [40] D. Jha, M. A. Riegler, D. Johansen, P. Halvorsen, and H. D. Johansen,
[17] K. Zhang and D. Liu, “Customized segment anything model for medical “Doubleu-net: A deep convolutional neural network for medical image
image segmentation,” arXiv preprint arXiv:2304.13785, 2023. segmentation,” in CBMS. IEEE, 2020, pp. 558–564.
[18] J. Wu, W. Ji, Y. Liu, H. Fu, M. Xu, Y. Xu, and Y. Jin, “Medical [41] H. Huang, L. Lin, R. Tong, H. Hu, Q. Zhang, Y. Iwamoto, X. Han, Y.-
sam adapter: Adapting segment anything model for medical image W. Chen, and J. Wu, “Unet 3+: A full-scale connected unet for medical
segmentation,” arXiv preprint arXiv:2304.12620, 2023. image segmentation,” in ICASSP. IEEE, 2020, pp. 1055–1059.