Using Diffusion Priors for Video Amodal Segmentation

Kaihua Chen Deva Ramanan Tarasha Khurana

Carnegie Mellon University
arXiv:2412.04623v1 [[Link]] 5 Dec 2024

Figure 1. In this work, we tackle the problem of video amodal segmentation and content completion: given a modal (visible) object
sequence in a video, we develop a two-stage method that generates its amodal (visible + invisible) masks and RGB content. We capitalize
on the shape and temporal consistency priors baked into video foundation models because of their large-scale pretraining. Finetuning these
models enables us to infer complete shapes and RGB details of objects that undergo occlusion. Our method is effectively able to handle
severe occlusions and generalizes across diverse object categories, achieving state-of-the-art results on synthetic and real-world datasets.
We show one such example of an unseen deformable object category ‘laptop’ that undergoes a complete occlusion in the highlighted frame.

Abstract extent of an object. This is followed by a content completion

stage which is able to inpaint the occluded regions of an ob-
Object permanence in humans is a fundamental cue that ject. We benchmark our approach alongside a wide array
helps in understanding persistence of objects, even when of state-of-the-art methods on four datasets and show a dra-
they are fully occluded in the scene. Present day methods matic improvement of upto 13% for amodal segmentation in
in object segmentation do not account for this amodal na- an object’s occluded region.
ture of the world, and only work for segmentation of visible
or modal objects. Few amodal methods exist; single-image
segmentation methods cannot handle high-levels of occlu- 1. Introduction
sions which are better inferred using temporal information,
and multi-frame methods have focused solely on segment- Gestalt psychology [27] suggests that human perception in-
ing rigid objects. To this end, we propose to tackle video herently organizes visual elements into cohesive wholes.
amodal segmentation by formulating it as a conditional When an object is occluded, humans can often infer the
generation task, capitalizing on the foundational knowledge complete outline of the object – an ability that is devel-
in video generative models. Our method is simple; we re- oped in humans in their early years [21, 35]. Addition-
purpose these models to condition on a sequence of modal ally, object permanence [2] suggests that with some tem-
mask frames of an object along with contextual pseudo- poral context, humans can perceive objects to persist even
depth maps, to learn which object boundary may be oc- when they undergo complete occlusions. Replicating these
cluded and therefore, extended to hallucinate the complete phenomena of gestalt psychology and object permanence in

1
object segmentation has traditionally been ignored, as the multiple plausible interpretations for the completion of oc-
community has focused largely on segmenting the visible cluded objects. We show that the outputs of our approach
or modal regions of objects (as exemplified by models like can be used for downstream applications like 4D recon-
SAM [26, 41]). Recent focus has shifted to include amodal struction, scene manipulation, and pseudo-groundtruth gen-
segmentation [28, 62], which involves segmenting an ob- eration.
ject’s full shape, including both visible and occluded parts.
This task has broad real-world applications, including safe 2. Related Work
navigation in robotic manipulation and autonomous driv-
ing [38, 42], understanding occluder-occludee relationships Image amodal segmentation. Most previous amodal seg-
in complex scenes [60], and enhancing advanced image and mentation research has concentrated on image-based ap-
video editing tools [36]. proaches. Some methods [10, 23, 28, 38, 51–53] adopt
a similar strategy to modal segmentation, where models
Why is this hard? In a monocular setup, amodal per-
are trained to take RGB images as input and directly out-
ception is an ill-posed problem because there are multiple
put amodal masks for all objects in the scene. Another
plausible explanations for how an object boundary should
line of methods [8, 29, 36, 55, 59, 60] leverages exist-
be extended in an occluded region. Recent innovations for
ing modal masks, generated by modal segmentation mod-
amodal segmentation [36, 59] and inpainting [32, 37] use
els, to predict amodal masks based on these and additional
diffusion frameworks for learning this multi-modal distri-
inputs like image frames. Besides inferring the complete
bution, but they are not able to handle scenarios where an
object shape, some approaches also hallucinate the RGB
object could be fully-occluded. This issue is exacerbated
content in the occluded regions. Generic inpainting meth-
by the lack of real-world datasets that have groundtruths for
ods [32, 37] often fail at this task, as they rely on sur-
both amodal masks of objects, and their RGB content.
rounding context, which often includes occluders. In con-
Status quo. Despite this, current image-based amodal trast, content completion methods explicitly condition on
segmentation algorithms [11, 23, 36, 51, 52, 59, 60] have the modal content, either by directly generating the amodal
shown impressive performance. However, these approaches content based on modal information [36, 55] or by inpaint-
are set in the single-frame setting, where they struggle with ing within the predicted amodal segmentation area [29, 60].
cases where objects are heavily or completely occluded. A Due to the availability of high-quality real-world amodal
potential solution is to approach amodal segmentation in a image datasets [33, 38, 62], image amodal segmentation
multi-frame setting [24] so as to infer complete occlusions and content completion methods have shown strong per-
with temporal context. However, existing video amodal seg- formance by learning robust shape priors. However, these
mentation algorithms [9, 57] are typically limited to rigid methods frequently struggle with cases of significant occlu-
objects, and are dependent on additional inputs (like cam- sion and fail entirely for fully occluded objects because the
era poses or optical flow) which hinders their scalability and amodal cues cannot be inferred in a single-frame setting.
therefore, generalization to unseen data. Video amodal segmentation. Recently, video amodal
Key insight. To address these challenges, we propose re- segmentation methods have emerged [9, 11, 57]. These
purposing a video diffusion model, Stable Video Diffusion approaches integrate information from preceding and suc-
(SVD) [3], to achieve highly accurate and generalizable ceeding frames in a video sequence, enabling temporally
video amodal segmentation. One key insight is that founda- consistent predictions. However, the training and evaluation
tional diffusion models trained to generate pixels also bake- of most of these algorithms are limited to synthetic datasets
in strong priors on object shape. Such priors have been ex- with rigid objects of similar scale [12, 14, 38, 49]. Although
poited by conditional image generation [40, 45, 61] meth- these algorithms outperform image-based amodal segmen-
ods that condition on semantic maps and object boundaries. tation methods within synthetic datasets, their practical ap-
We similarly exploit these priors for our task. But crucially, plications remain limited. In contrast, we utilize both syn-
our multi-frame video setup allows us to propagate object thetic [14, 19] and real-world datasets [1, 7, 18], which in-
shape and content across time; e.g., one can infer the shape clude deformable objects with diverse motions and scales,
of a fully occluded by object by looking at other frames often mixed with complex camera movements. Moreover,
where it is visible (Fig. 1). to our knowledge, this work is the first to explore video-
Our proposed model achieves state-of-the-art perfor- level amodal content completion.
mance across four synthetic and real-world video datasets, Real-world priors from diffusion models. Diffusion
compared to a wide-variety of single-frame and multi-frame models have achieved significant success in generative tasks
amodal segmentation baselines. We train on only syn- within computer vision. Initially developed for uncondi-
thetic data, but demonstrate strong zero-shot generalization tional image generation [16], the scope of diffusion models
to real-world data. Thanks to the multi-modal generation has expanded in multiple directions. These advancements
capability of diffusion models, our approach can provide include, but are not limited to, implementing conditional

2
 Stage 1: Amodal Segmentation Stage 2: Content Completion
Input RGB video Predicted amodal masks
Ours - Completion

,
Modal masks Pseudo-depth Ours - Segmentation Predicted amodal content

Figure 2. Model pipeline for amodal segmentation and content completion. The first stage of our pipeline generates amodal masks {Ât }
for an object, given its modal masks {Mt } and pseudo-depth of the scene {Dt } (which is obtained by running a monocular depth estimator
on RGB video sequence {It } ). The predicted amodal masks from the first stage are then sent as input to the second stage, along with the
modal RGB content of the occluded object in consideration. The second stage then inpaints the occluded region and outputs the amodal
RGB content {Cˆt } for the occluded object. Both stages employ a conditional latent diffusion framework with a 3D UNet backbone [3].
Conditionings are encoded via a VAE encoder into latent space, concatenated, and processed by a 3D UNet with interleaved spatial and
temporal blocks. CLIP embeddings of {Mt } and the modal RGB content provide cross-attention cues for the first and second stage
respectively. Finally, the VAE decoder translates outputs back to pixel space.

techniques for tasks like style transfer [6, 61] and text-to- the RGB content of the occluded areas {C1 , C2 , · · · , CT }.
image synthesis [40], transitioning from pixel-space noise
to latent-space noise [4, 43], developing various training 3.1. Preliminary: diffusion framework
and sampling strategies [16, 20, 48], and extending their
application from realistic image generation to video gen- We make use of an open-source video latent diffusion
eration [3, 17, 34]. In addition, the successful adaptation model [4, 43] (Stable Video Diffusion (SVD) [3]) and use
of diffusion models for multiple downstream tasks, includ- the EDM framework [20] for both training and inference.
ing depth estimation [22], multi-view synthesis [31, 47], Compared to pixel-space diffusion, latent diffusion models
and scene reconstruction [30], underscores their ability to reduce computational and memory demands by encoding
capture object shape priors and understand potential 3D in- frames into compact latent representations while preserv-
formation [58]. While recent image amodal segmentation ing both perceptual and region-based alignment. The EDM
methods have demonstrated initial success in incorporating framework further accelerates training convergence and re-
diffusion models [36, 52, 55, 59], our approach advances duces the required number of denoising steps during infer-
this progress by applying video diffusion techniques to the ence without compromising generation quality.
domain of video amodal segmentation. Our diffusion model takes as input the latent represen-
tation z0 , additional conditioning c, a noise scale σ fol-
3. Method lowing log σ ∼ N (Pmean , Pstd ), and Gaussian noise ϵ ∼
N (0, σ 2 I). The training objective is defined as:
Consider a video sequence {I1 , I2 , . . . , IT } with modal
(or visible) segmentation masks {M1 , M2 , . . . , MT } for
min Eσ,z0 ,c,ϵ λ||Dθ (z0 + ϵ; σ, c) − z0 ||22
 
(1)
a target object. Such masks can be readily obtained by θ
conventional modal segmentors, such as Segment Anything
v2 [41]. We first describe a (diffusion-based) model to gen- Here, λ is a scalar related to σ, and Dθ = c1 (z0 + ϵ) +
erating amodal masks {A1 , A2 , . . . , AT } that capture the c2 Fθ (z0 + ϵ; σ, c) represents the predicted latent represen-
full extent of the target object, including occluded portions. tation, which combines the noisy latent input with the v-
We then train a second stage (diffusion-based) model that prediction [46] output of the diffusion backbone Fθ , using
uses the input video and amodal masks to fill in (or inpaint) additional scalars c1 and c2 that also depends on σ.

3
3.2. Modal masks in, amodal masks out
To train a high-quality amodal segmentor with limited data,
one strategy is to leverage the shape and content priors of
video foundation models pretrained on large-scale datasets.
For this, we lean on the foundational knowledge in SVD,
learnt by pretraining on the extensive LDM-F dataset [3]
with 152 million examples. However, as the vanilla SVD Figure 3. Modal-amodal RGB training pair for content comple-
was designed for image-to-video tasks, we adapt its struc- tion. The left frame displays the partially occluded modal RGB
ture and conditioning to suit our modal-to-amodal sequence content, generated by overlaying amodal masks (black regions)
generation task. We describe this below. onto the amodal object to disrupt its visual integrity. The right
First, we replace the input conditioning c, originally frame shows the original, unoccluded amodal RGB object.
an RGB image, with binary modal masks of shape
RT ×1×H×W . By default, the variational autoencoder
(VAE) [25] in SVD requires a 3-channel input. To ad- put latents for our 3D U-Net backbone have the shape
H W
dress this mismatch in the number of channels, we repli- RT ×3C1 × F × F , requiring a new first convolutional layer
cate the binary mask three times, following the approach in the 3D U-Net to accommodate the increased channels.
for single-channel VAE inputs in a recent work [22]. Af- Rather than finetuning our model with both modal masks
ter encoding each (replicated) mask seperately, we obtain a and pseudo-depth conditionings directly, we find that it is
H W
latent tensor of shape RT ×C1 × F × F . This latent represen- more efficient to do a two-stage finetuning, where we fine-
tation, concatenated with a noise image of the same shape, tune our mask conditioned model first and then use it to
forms the input to our backbone which is a spatio-temporal initialize the finetuning of the mask-and-depth conditioned
3D U-Net [4, 44]. The final shape of this input becomes model. We call this approach two-stage finetuning, allowing
RT ×2C1 ×H×W . In contrast to the vanilla SVD, where the the model to adapt gradually to the new conditions.
latent space of a single image is duplicated T times to align Inspired by ControlNet [61], we retain the parameters of
with the 3D U-Net’s input requirements, our 3D U-Net gets the first channels 2C1 in the input layer from the previously
as input T unique frames of the modal mask sequence being trained model and initialize the newly added channels C1
used as conditioning. to zero. This zero convolution approach ensures the model
Additionally, we use CLIP embeddings [39] for the retains its initial predictive capability during the first few
modal masks, and inject them into the transformer layers for fine-tuning steps with the added pseudo-depth conditioning.
cross-attention. This provides temporal information about We demonstrate the importance of these training strategies
the visibility of objects in surrounding frames. After the 3D in the ablation study.
U-Net, the VAE decoder converts the latent amodal mask
predictions back into the pixel space. 3.4. Amodal content completion
Till now, we discussed the first stage of our pipeline which
3.3. Conditioning on pseudo-depth outputs amodal masks for occluded objects. However, the
Till now, we described how SVD is modified to enable pre- RGB content in the occluded region is unknown. To inpaint
dicting amodal masks from modal masks. We find that one these occluded areas, we use a second SVD model with the
can add more contextual cues about the object and scene in same architecture but with different conditionings; the first
consideration through different data modalities. A natural conditioning is the RGB content from an object’s modal re-
choice for conditioning is RGB frames, as used in previ- gion, and the second conditioning is the predicted amodal
ous work [36, 60]. However, since occlusions of the target mask from the first stage. We train this model to generate
object are typically caused by objects closer to the cam- RGB content across the entire amodal region.
era, we empirically find that pseudo-depth maps provide Synthetic data curation A key challenge with this ap-
more implicit clues about potential occluders than RGB proach is the lack of ground-truth RGB content in occluded
frames, making them a more effective indicator for deter- regions, even in synthetic datasets like SAIL-VOS [19]. In-
mining regions to complete. We demonstrate the advan- spired by self-supervised training-pair construction used ex-
tages of pseudo-depth over RGB conditioning in our abla- tensively in image amodal tasks [36, 60], we extend this ap-
tion study. To integrate this, we utilize the Depth Anything proach to video sequences. Figure 3 illustrates an example
V2 monocular depth estimator [56] to convert RGB images of a modal-amodal RGB content training pair. To construct
into pseudo-depth maps, which are then incorporated into such a pair, we first select an object from the dataset with
our video diffusion model as additional channels concate- near-complete visibility (above 95%). We then sequentially
nated to the aforementioned input. overlay random amodal mask sequences onto this fully vis-
With the addition of pseudo-depth conditioning, the in- ible object until its visibility falls below a set threshold,

4
thereby simulating occlusion. This effectively generates Table 1. Quantitative comparison on SAIL-VOS and TAO-
ground-truth RGB data for the occluded regions. Amodal. We compare our method with image-based methods
(top) and video-based methods (bottom). Our method outperforms
all methods on the synthetic SAIL-VOS dataset, achieving nearly a
4. Experiments 13% improvement in Top-1 mIoUocc . Additionally, when trained
on SAIL-VOS, our method demonstrates strong generalization,
4.1. Setup outperforming others in zero-shot evaluations on the real-world
Implementation details. For training, we load the offi- TAO-Amodal dataset. Bold values indicate the best method, and
cial SVD-xt 1.1 pretrained checkpoint and use the AdamW underlined values indicate the second best.
optimizer with β1 =0.9, β2 =0.999. The learning rates for
SAIL-VOS TAO-Amodal
the two-stage fine-tuning are set to 3 · 10−5 and 3 · 10−6 , Method
mIoU mIoUocc AP25 AP50 AP75
for training without and with additional pseudo-depth con-
Modal 67.89 - 93.73 82.22 63.12
ditioning, respectively. In the case of SAIL-VOS, due to
Convex [60] 63.18 27.54 93.73 82.22 63.12
computational limitations, we set the batch size to 8 and the ConvexR [60] 71.21 34.27 93.73 82.22 63.12
frame size to 128 × 256. Training takes approximately 30 PCNet-M [60] 74.2 42.52 94.89 85.11 65.97
hours on 8 Nvidia RTX 3090 GPUs. During inference, we AISFormer [51] 73.51 39.16 95.45 81.93 59.84
pix2gestalt (Top-1) [36] 54.83 26.59 80.73 57.50 28.95
set the EDM denoising step to 25, the guidance scale to 1.5,
pix2gestalt (Top-3) [36] 60.79 33.76 91.80 71.19 38.80
and use a higher frame size of 256 × 512 to ensure more
VideoMAE [50] 69.67 29.39 69.14 56.71 41.19
accurate pixel-level predictions. We cover more implemen-
3D-UNet 72.79 39.54 94.59 83.83 64.33
tation details in the appendix. Ours (Top-1) 77.07 55.12 97.28 89.25 71.99
Datasets. Since amodal mask can be reliably annotated Ours (Top-3) 79.23 59.69 98.31 92.46 77.48
only in synthetic datasets or game engines, our model is
primarily trained and evaluated on synthetic datasets. We mIoU vs Visibility mIoUocc vs Visibility
100 100
include a zero-shot evaluation on a real-world dataset to as- Modal Modal
PCNet-M 80 PCNet-M
sess its generalization ability. Among synthetic datasets, 80 3D UNet 3D UNet
SAIL-VOS [19] includes 210 long video sequences with Ours Ours
60

mIoUocc (%)
60
mIoU (%)

162 common object classes generated from the photo- 40

40
realistic game GTA-V, featuring frequent and significant
20 20
occlusions. We use PySceneDetect [5] to identify shot
0
transitions within these long videos, selecting only con- 0
0 10 20 30 40 50 60 70 80 90 100 0 10 20 30 40 50 60 70 80 90 100
Visibility (%) Visibility (%)
tinuous scenes and segmenting them into 21,237 25-frame
object sequences. MOVi-B and MOVi-D, generated by
Figure 4. Comparison across visibility levels on SAIL-VOS. Our
Kubrics [14], feature rich annotations of simulated en- method outperforms the second-best image and video amodal seg-
vironments, rigid objects, and camera motions. These mentation methods across all visibility ranges (we use Top-1 met-
datasets have been adapted as video amodal segmentation rics). This highlights the ability of our method to handle heavy
benchmarks by previous studies [9, 11] and contain 13,997 occlusions, and understand when an object is not occluded.
and 12,010 sequences, each with an approximate length
of 25 frames. For real-world evaluation, we use TAO-
Amodal [18], a high-quality amodal tracking dataset com- MAE [50] and SVD’s backbone 3D U-Net. We also evalu-
prising 993 video sequences in its validation set. Un- ate the ground-truth modal masks.
like synthetic datasets, TAO-Amodal provides only amodal Metrics. Following common practice in amodal seg-
bounding box annotations, as annotating amodal masks by mentation [9, 11, 57], we use mIoU and mIoUocc as eval-
humans is challenging. Similar to SAIL-VOS, we segment uation metrics. Given a modal-amodal sequence pair in
these videos into 1,392 object sequences. each frame, where the ground-truth modal mask is Mi , and
Baselines. We compare our method against recent the predicted and ground-truth amodal masks are Âi and
baselines for both image and video amodal segmentation. Ai , respectively, we define IoU as  i ∩Ai
and mIoUocc as
 ∪A
For image-based amodal segmentation, our baselines in- (Âi −Mi )∩(Ai −Mi )
i i

clude creating a convex hull around a given modal mask (Âi −Mi )∪(Ai −Mi )
.
We report the mean values across all
[60], AISFormer [51], PCNet-M [60], and pix2gestalt [36]. frames in the dataset as mIoU and mIoUocc . For TAO-
For video-based amodal segmentation, we evaluate against Amodal, which uses bounding box evaluation instead of
SaVos [57], Bi-LSTM [9, 13], EoRaS [9], and C2F- masks, we adopt average precision metrics used in a re-
Seg [11]. We discuss more details about these baselines cent amodal tracking work [18] – AP25 , AP50 , and AP75 ,
in the appendix. Additionally, to benchmark against re- based on varying IoU thresholds calculated over bounding
gression approaches, we include transformer-based Video- box areas. Additionally, to account for the multimodal gen-

5
Table 2. Quantitative Comparison on MOVi-B/D. Due to strong
camera motion and higher occlusions in these datasets, multi-
frame methods generally outperform single-frame methods. Our
method surpasses all prior state-of-the-art, achieving over a 4%
improvement in Top-1 mIoUocc across both datasets.

MOVi-B MOVi-D
Method
mIoU mIoUocc mIoU mIoUocc
Modal 59.19 - 56.92 -
Convex [60] 64.21 18.42 60.18 16.48
PCNet-M [60] 65.79 24.02 64.35 27.31
AISFormer [51] 77.34 43.53 67.72 33.65
SaVos [57] 70.72 33.61 60.61 22.64 Figure 5. Temporal consistency comparison with an image
Bi-LSTM [9, 13] 77.93 46.21 68.43 36.00 amodal segmentation method. We highlight the lack of temporal
EoRaS [9] 81.76 49.39 74.1 38.33 coherence in a single-frame diffusion based method, pix2gestalt,
C2F-Seg [11] - - 71.67 36.13 for both the predicted amodal segmentation mask and the RGB
VideoMAE [50] 78.74 42.86 70.93 32.78 content for the occluded person in the example shown. By lever-
3D-UNet 82.16 49.81 75.65 40.86 aging temporal priors, our approach achieves significantly higher
Ours (Top-1) 83.51 53.75 77.03 44.23 temporal consistency across occlusions.
Ours (Top-3) 83.93 54.56 77.76 45.6

the-art. Despite strong camera motion in MOVi-B/D, our

eration capability of diffusion-based methods, we adopt a model adapts well without access to camera extrinsics or op-
probabilistic evaluation with Top-K metrics [24], selecting tical flow (unlike some baselines [9, 57]). We posit that our
the best IoU or AP score in each frame from K predictions. method is able to use the 3D priors from Stable Video Dif-
fusion and is therefore, successfully able to maintain con-
4.2. Comparison to state-of-the-art sistent object shapes from different view-points. Notably,
prior works on MOVi-B/D are evaluated using a cropped
Table 1 shows the quantitative comparisons on SAIL-
modal bounding box enlarged by 2 times as input; we adopt
VOS and TAO-Amodal, where our method surpasses all
the same setting here for a fair comparison. However, we
baselines. Notably, it achieves nearly 13% improvement
observed that using the full, uncropped image as input can
over the second-best method, PCNet-M [60], in terms of
significantly enhance model performance, and we include
mIoUocc , highlighting effective completion of occluded ob-
these results in the appendix.
ject regions. Despite being trained exclusively on synthetic
SAIL-VOS, a zero-shot evaluation on TAO-Amodal high- For content completion, due to the lack of ground truth
lights the strong generalization of our model. We posit that, and standardized metrics, we conducted a user study on 20
in addition to leveraging foundational knowledge and rich randomly selected sequences from SAIL-VOS and TAO-
priors from the large-scale pretraining of SVD, our model Amodal. In this user study, we did A/B testing and forced
is able to learn temporal cues that help it amodally com- participants to choose between our method and pix2gestalt.
plete any unseen object classes from neighboring frames. We found that users showed a preference of 85.6% for our
Figure 4 further illustrates our method’s consistent perfor- method over pix2gestalt.
mance across all visibility ranges on SAIL-VOS [19], indi-
cating that our method can realistically hallucinate masks in
4.3. Ablation studies
occluded regions, across the entire range of visibility levels. Conditioning. Here we ablate our choice of modal mask
We also compare with a single-frame diffusion-based and pseudo-depth conditioning. We also examine the ef-
method, pix2gestalt [36], in Table 1. Perhaps unsurpris- fect of additionally using RGB video frames as condition-
ingly, pix2gestalt performs poorly on these video bench- ing. As shown in Table 3, adding RGB or pseudo-depth in-
marks, likely because many objects undergo high degrees of formation improves model performance, with pseudo-depth
occlusion. Since, pix2gestalt is a single-frame method, we providing a more substantial enhancement. Although com-
also find that its predictions vary significantly across frames bining both RGB and pseudo-depth yields a higher mIoU
and lack temporal coherence (c.f. Figure 5). In contrast, our on SAIL-VOS, conditioning on pseudo-depth alone outper-
method does drastically better because it can handle both, forms across other metrics. This supports our claim that
high occlusions and temporal coherence across frames. pseudo-depth is a more generalizable modality, and the de-
Table 2 provides quantitative comparisons on the MOVi- pendence on texture and appearance cues in fact hinders
B/D datasets, where our method beats the prior state-of- generalization of our model to TAO-Amodal.

6
Figure 6. Qualitative comparison of amodal segmentation methods across diverse datasets. Our method leverages strong shape priors,
such as for humans, chairs, and teapots, to generate clean and realistic object shapes. It also excels in handling heavy occlusions; even
when objects are nearly fully occluded (e.g., “chair” in the second row of SAIL-VOS), our method achieves high-fidelity shape completion
by utilizing temporal priors. Note that TAO-Amodal contains out-of-frame occlusions which none of the methods are trained for, but our
method is able to handle such cases.

Figure 7. Qualitative results for content completion. Although our content completion module, initialized from pretrained SVD weights,
is finetuned solely on synthetic SAIL-VOS, it achieves photorealistic, high-fidelity object inpainting even in real-world scenarios. Further-
more, our method can complete unseen categories, such as giraffes and plastic bottle, likely due to its ability to transfer styles and patterns
from the visible parts of objects to occluded areas in the current or neighboring frames. We show examples from TAO-Amodal (top) and
in-the-wild YouTube videos (bottom).

7
Table 3. Ablation study for input conditioning. We study the
effect of conditioning our model on different input modalities.
Results show that pseudo-depth conditioning yields greater per-
formance improvements than RGB conditioning across almost all
metrics. We therefore drop RGB conditioning in the final method.
Figure 8. We show an example of multi-modal generation from
Conditions SAIL-VOS TAO-Amodal our diffusion model. Since there are multiple plausible explana-
mask RGB depth mIoU mIoUocc AP25 AP50 AP75 tions for the shape of the person in his occluded region, our model
✓ ✗ ✗ 75.17 51.28 94.89 85.03 66.87 predicts two such plausible amodal masks (with the person’s oc-
✓ ✓ ✗ 76.59 53.3 95.86 86.59 70.12 cluded legs in two different orientations).
✓ ✗ ✓ 77.07 55.12 97.28 89.25 69.65
✓ ✓ ✓ 77.19 54.59 96.6 87.16 69.64
83 mIoU and mIoUocc vs Top-k
63
Table 4. Ablation study for training strategies. We study the 81 61

mIoUocc (%)
effect of two-stage finetuning for segmentation. We find that zero

mIoU (%)
convolution helps significantly, while two-stage fine-tuning gives 79 59
us an additional, moderate improvement. 57
77

Training strategies SAIL-VOS TAO-Amodal 75 55

1 2 3 4 5 6 7 8 9 10
2-stage ft. zero-conv mIoU mIoUocc AP25 AP50 AP75 Top-k
✗ ✗ 73.73 41.35 96.27 85.93 66.45
✓ ✗ 72.72 32.23 95.38 86.1 68.74 Figure 9. Ablation of Top-K on SAIL-VOS. We find that increas-
✗ ✓ 76.92 54.25 96.58 87.64 69.34 ing the number of output samples from our method, K, leads to
✓ ✓ 77.07 55.12 97.28 89.25 71.99 improvements in both mIoU and mIoUocc ; however, these im-
provements gradually plateau as we increase K.

Training strategies. Table 4 shows the impact of our

two-stage fine-tuning strategy and use of zero convolutions. multi-view generation methods like SV4D [54] to recon-
Compared to randomly initializing the new input convo- struct dynamic objects across space and time, even when
lution layer in the 3D U-Net, we find that zero convo- they get fully occluded. Second, we show that once all ob-
lution significantly improves model performance. Addi- jects in a scene are de-occluded, they can be rearranged in
tionally, compared to training with both modal mask and the scene to simulate different object interactions and real-
pseudo-depth conditionings from scratch, two-stage fine- ities. Third, we can generate amodal segmentation pseudo-
tuning (where we train with modal masks first and then add groundtruth on real-world datasets to fill in the gap for lack
pseudo-depth) leads to further quantitative improvements. of real-world training data for video amodal segmentation.
We also include an ablation study on the weights initializa- Please see appendix for more details and visual examples.
tion in the appendix.
Top-k evaluation. Like other diffusion models, ours 5. Discussion
also supports multimodal generation. For instance, when In this work, we focus on the problem of video amodal
parts of an object remain consistently occluded in a video segmentation – segmenting objects to their full extent even
(e.g., a person’s legs), multiple plausible interpretations of when they may be partially or fully occluded in videos.
the occluded area (e.g., standing, sitting) may exist, as il- We lean on the large-scale pretraining of video foundation
lustrated in Figure 8. By setting different random seeds, models and adapt Stable Video Diffusion [3] for the task of
our model generates varying predictions for the same in- video amodal segmentation. Given an object’s modal mask
put, leading to different IoU values against the ground truth. sequence, and pseudo-depth maps of the scene, we aim to
Figure 9 reports the Top-10 mIoU and mIoUocc results for predict amodal masks of the occluded object. This amodal
our model. As expected, performance improves with the mask is used by another model to inpaint the RGB content
number of outputs, although the gains gradually diminish. in the object’s occluded region. One of the key insights of
our work is that one can use the shape and temporal pri-
4.4. Applications
ors learnt by video foundation models. More crucially, our
Our amodal segmentation masks and content completions multi-frame setup allows us to propagate mask and RGB
can enable multiple downstream applications. We touch on content from the frames where an object maybe fully visible
three such applications in the appendix – 4D reconstruc- to the frames of high occlusion. We find that, our models
tion, scene manipulation, and pseudo-groundtruth genera- can perform exceedingly well even for unseen categories,
tion. First, we find that one can use monocular video to likely because of their pretraining on foundational data.

8
Acknowledgments We would like to thank Carl amodal segmentation with shape prior. In Proceedings of
Vondrick, Ege Ozguroglu and Achal Dave for in- the IEEE/CVF International Conference on Computer Vi-
sightful discussions and clarifications on pix2gestalt’s sion, pages 1262–1271, 2023. 2, 5, 6, 12
evaluation protocol. Mosam Dabhi helped with demon- [12] Andreas Geiger, Philip Lenz, Christoph Stiller, and Raquel
strating the application of our work to 4D reconstruction. Urtasun. Vision meets robotics: The kitti dataset. The Inter-
national Journal of Robotics Research, 32(11):1231–1237,
2013. 2
References [13] Alex Graves. Long short-term memory, pages 37–45. 2012.
5, 6
[1] Ali Athar, Jonathon Luiten, Paul Voigtlaender, Tarasha Khu- [14] Klaus Greff, Francois Belletti, Lucas Beyer, Carl Doersch,
rana, Achal Dave, Bastian Leibe, and Deva Ramanan. Burst: Yilun Du, Daniel Duckworth, David J. Fleet, et al. Kubric: A
A benchmark for unifying object recognition, segmentation scalable dataset generator. In Proceedings of the IEEE/CVF
and tracking in video. In Proceedings of the IEEE/CVF Win- Conference on Computer Vision and Pattern Recognition,
ter Conference on Applications of Computer Vision, pages pages 3749–3761, 2022. 2, 5
1674–1683, 2023. 2 [15] Jonathan Ho and Tim Salimans. Classifier-free diffusion
[2] Renée Baillargeon and Julie DeVos. Object permanence in guidance. arXiv preprint arXiv:2207.12598, 2022. 12
young infants: Further evidence. Child Development, 62(6): [16] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising dif-
1227–1246, 1991. 1 fusion probabilistic models. In NeurIPS, pages 6840–6851,
[3] Andreas Blattmann, Tim Dockhorn, Sumith Kulal, Daniel 2020. 2, 3
Mendelevitch, Maciej Kilian, Dominik Lorenz, Yam Levi, [17] Jonathan Ho, William Chan, Chitwan Saharia, Jay Whang,
et al. Stable video diffusion: Scaling latent video diffusion Ruiqi Gao, Alexey Gritsenko, Diederik P. Kingma, et al. Im-
models to large datasets. arXiv preprint arXiv:2311.15127, agen video: High definition video generation with diffusion
2023. 2, 3, 4, 8, 12, 13 models. arXiv preprint arXiv:2210.02303, 2022. 3
[4] Andreas Blattmann, Robin Rombach, Huan Ling, Tim Dock- [18] Cheng-Yen Hsieh, Tarasha Khurana, Achal Dave, and Deva
horn, Seung Wook Kim, Sanja Fidler, and Karsten Kreis. Ramanan. Tao-amodal: A benchmark for tracking any object
Align your latents: High-resolution video synthesis with la- amodally. arXiv preprint arXiv:2312.12433, 2023. 2, 5, 14
tent diffusion models. In CVPR, pages 22563–22575, 2023. [19] Yuan-Ting Hu, Hong-Shuo Chen, Kexin Hui, Jia-Bin Huang,
3, 4, 12 and Alexander G. Schwing. Sail-vos: Semantic amodal in-
[5] Breakthrough. PySceneDetect: Video scene cut detection stance level video object segmentation-a synthetic dataset
tool, 2024. Accessed: 2024-11-12. 5 and baselines. In Proceedings of the IEEE/CVF Conference
[6] Tim Brooks, Aleksander Holynski, and Alexei A. Efros. In- on Computer Vision and Pattern Recognition, pages 3105–
structpix2pix: Learning to follow image editing instructions. 3115, 2019. 2, 4, 5, 6
In Proceedings of the IEEE/CVF Conference on Computer [20] Tero Karras, Miika Aittala, Timo Aila, and Samuli Laine.
Vision and Pattern Recognition, pages 18392–18402, 2023. Elucidating the design space of diffusion-based generative
3 models. In NeurIPS, pages 26565–26577, 2022. 3, 12
[7] Achal Dave, Tarasha Khurana, Pavel Tokmakov, Cordelia [21] Michael Kavsek. The influence of context on amodal com-
Schmid, and Deva Ramanan. Tao: A large-scale bench- pletion in 5-and 7-month-old infants. Journal of Cognition
mark for tracking any object. In Computer Vision–ECCV and Development, 5(2):159–184, 2004. 1
2020: 16th European Conference, Glasgow, UK, August 23– [22] Bingxin Ke, Anton Obukhov, Shengyu Huang, Nando Met-
28, 2020, Proceedings, Part V, pages 436–454. Springer In- zger, Rodrigo Caye Daudt, and Konrad Schindler. Repurpos-
ternational Publishing, 2020. 2 ing diffusion-based image generators for monocular depth
[8] Kiana Ehsani, Roozbeh Mottaghi, and Ali Farhadi. Segan: estimation. In Proceedings of the IEEE/CVF Conference
Segmenting and generating the invisible. In Proceedings on Computer Vision and Pattern Recognition, pages 9492–
of the IEEE Conference on Computer Vision and Pattern 9502, 2024. 3, 4
Recognition, pages 6144–6153, 2018. 2 [23] Lei Ke, Yu-Wing Tai, and Chi-Keung Tang. Deep occlusion-
[9] Ke Fan, Jingshi Lei, Xuelin Qian, Miaopeng Yu, Tianjun aware instance segmentation with overlapping bilayers. In
Xiao, Tong He, Zheng Zhang, and Yanwei Fu. Rethink- Proceedings of the IEEE/CVF Conference on Computer Vi-
ing amodal video segmentation from learning supervised sig- sion and Pattern Recognition, pages 4019–4028, 2021. 2
nals with object-centric representation. In Proceedings of [24] Tarasha Khurana, Achal Dave, and Deva Ramanan. Detect-
the IEEE/CVF International Conference on Computer Vi- ing invisible people. In Proceedings of the IEEE/CVF Inter-
sion, pages 1272–1281, 2023. 2, 5, 6, 12 national Conference on Computer Vision, pages 3174–3184,
[10] Patrick Follmann, Rebecca König, Philipp Härtinger, 2021. 2, 6
Michael Klostermann, and Tobias Böttger. Learning to see [25] Diederik P. Kingma. Auto-encoding variational bayes. arXiv
the invisible: End-to-end trainable amodal instance segmen- preprint arXiv:1312.6114, 2013. 4
tation. In 2019 IEEE Winter Conference on Applications of [26] Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao,
Computer Vision (WACV), pages 1328–1336. IEEE, 2019. 2 Chloe Rolland, Laura Gustafson, Tete Xiao, et al. Segment
[11] Jianxiong Gao, Xuelin Qian, Yikai Wang, Tianjun Xiao, anything. In Proceedings of the IEEE/CVF International
Tong He, Zheng Zhang, and Yanwei Fu. Coarse-to-fine Conference on Computer Vision, pages 4015–4026, 2023. 2

9
[27] Kurt Koffka. Principles of Gestalt Psychology. Routledge, supervision. In International Conference on Machine Learn-
2013. 1 ing, pages 8748–8763. PMLR, 2021. 4
[28] Ke Li and Jitendra Malik. Amodal instance segmentation. [40] Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray,
In Computer Vision–ECCV 2016: 14th European Confer- Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever.
ence, Amsterdam, The Netherlands, October 11-14, 2016, Zero-shot text-to-image generation. In International Confer-
Proceedings, Part II, pages 677–693. Springer International ence on Machine Learning, pages 8821–8831. PMLR, 2021.
Publishing, 2016. 2 2, 3
[29] Huan Ling, David Acuna, Karsten Kreis, Seung Wook Kim, [41] Nikhila Ravi, Valentin Gabeur, Yuan-Ting Hu, Ronghang
and Sanja Fidler. Variational amodal object completion. Hu, Chaitanya Ryali, Tengyu Ma, Haitham Khedr, et al. Sam
Advances in Neural Information Processing Systems, 33: 2: Segment anything in images and videos. arXiv preprint
16246–16257, 2020. 2 arXiv:2408.00714, 2024. 2, 3
[30] Fangfu Liu, Wenqiang Sun, Hanyang Wang, Yikai Wang, [42] N. Dinesh Reddy, Robert Tamburo, and Srinivasa G.
Haowen Sun, Junliang Ye, Jun Zhang, and Yueqi Duan. Re- Narasimhan. Walt: Watch and learn 2d amodal repre-
conx: Reconstruct any scene from sparse views with video sentation from time-lapse imagery. In Proceedings of the
diffusion model. arXiv preprint arXiv:2408.16767, 2024. 3 IEEE/CVF Conference on Computer Vision and Pattern
[31] Ruoshi Liu, Rundi Wu, Basile Van Hoorick, Pavel Tok- Recognition, pages 9356–9366, 2022. 2
makov, Sergey Zakharov, and Carl Vondrick. Zero-1-to-3: [43] Robin Rombach, Andreas Blattmann, Dominik Lorenz,
Zero-shot one image to 3d object. In Proceedings of the Patrick Esser, and Björn Ommer. High-resolution image syn-
IEEE/CVF International Conference on Computer Vision, thesis with latent diffusion models. In CVPR, pages 10684–
pages 9298–9309, 2023. 3 10695, 2022. 3
[44] Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-
[32] Andreas Lugmayr, Martin Danelljan, Andres Romero, Fisher
net: Convolutional networks for biomedical image segmen-
Yu, Radu Timofte, and Luc Van Gool. Repaint: Inpainting
tation. In Medical Image Computing and Computer-Assisted
using denoising diffusion probabilistic models. In Proceed-
Intervention–MICCAI 2015: 18th International Conference,
ings of the IEEE/CVF Conference on Computer Vision and
Munich, Germany, October 5-9, 2015, Proceedings, Part III,
Pattern Recognition, pages 11461–11471, 2022. 2
pages 234–241. Springer International Publishing, 2015. 4
[33] David Martin, Charless Fowlkes, Doron Tal, and Jitendra
[45] Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li,
Malik. A database of human segmented natural images
Jay Whang, Emily L. Denton, Kamyar Ghasemipour, et al.
and its application to evaluating segmentation algorithms and
Photorealistic text-to-image diffusion models with deep lan-
measuring ecological statistics. In Proceedings of the Eighth
guage understanding. Advances in Neural Information Pro-
IEEE International Conference on Computer Vision (ICCV
cessing Systems, 35:36479–36494, 2022. 2
2001), pages 416–423. IEEE, 2001. 2
[46] Tim Salimans and Jonathan Ho. Progressive distillation
[34] Willi Menapace, Aliaksandr Siarohin, Ivan Skorokhodov, for fast sampling of diffusion models. arXiv preprint
Ekaterina Deyneka, Tsai-Shien Chen, Anil Kag, Yuwei arXiv:2202.00512, 2022. 3
Fang, et al. Snap video: Scaled spatiotemporal transform-
[47] Kyle Sargent, Zizhang Li, Tanmay Shah, Charles Herrmann,
ers for text-to-video synthesis. In CVPR, pages 7038–7048,
Hong-Xing Yu, Yunzhi Zhang, Eric Ryan Chan, et al. Ze-
2024. 3
ronvs: Zero-shot 360-degree view synthesis from a single
[35] Yumiko Otsuka, So Kanazawa, and Masami K. Yamaguchi. real image. arXiv preprint arXiv:2310.17994, 2023. 3
Development of modal and amodal completion in infants. [48] Jiaming Song, Chenlin Meng, and Stefano Ermon.
Perception, 35(9):1251–1264, 2006. 1 Denoising diffusion implicit models. arXiv preprint
[36] Ege Ozguroglu, Ruoshi Liu, Dı́dac Surı́s, Dian Chen, Achal arXiv:2010.02502, 2020. 3
Dave, Pavel Tokmakov, and Carl Vondrick. pix2gestalt: [49] Matthias Tangemann, Steffen Schneider, Julius Von
Amodal segmentation by synthesizing wholes. In Proceed- Kügelgen, Francesco Locatello, Peter Gehler, Thomas
ings of the IEEE/CVF Conference on Computer Vision and Brox, Matthias Kümmerer, Matthias Bethge, and Bernhard
Pattern Recognition, pages 3931–3940, 2024. 2, 3, 4, 5, 6, Schölkopf. Unsupervised object learning via common fate.
12, 13 arXiv preprint arXiv:2110.06562, 2021. 2
[37] Dustin Podell, Zion English, Kyle Lacey, Andreas [50] Zhan Tong, Yibing Song, Jue Wang, and Limin Wang.
Blattmann, Tim Dockhorn, Jonas Müller, Joe Penna, and Videomae: Masked autoencoders are data-efficient learners
Robin Rombach. Sdxl: Improving latent diffusion mod- for self-supervised video pre-training. Advances in Neural
els for high-resolution image synthesis. arXiv preprint Information Processing Systems, 35:10078–10093, 2022. 5,
arXiv:2307.01952, 2023. 2 6, 12, 13
[38] Lu Qi, Li Jiang, Shu Liu, Xiaoyong Shen, and Jiaya Jia. [51] Minh Tran, Khoa Vo, Kashu Yamazaki, Arthur Fernan-
Amodal instance segmentation with kins dataset. In Pro- des, Michael Kidd, and Ngan Le. Aisformer: Amodal
ceedings of the IEEE/CVF Conference on Computer Vision instance segmentation with transformer. arXiv preprint
and Pattern Recognition, pages 3014–3023, 2019. 2 arXiv:2210.06323, 2022. 2, 5, 6, 12
[39] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya [52] Minh Tran, Khoa Vo, Vuong Ho, Tri Nguyen, and
Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, et al. Ngan Hoang Le. Amodal instance segmentation with dif-
Learning transferable visual models from natural language fusion shape prior estimation. In The First Workshop on

10
 Populating Empty Cities–Virtual Humans for Robotics and
Autonomous Driving at CVPR 2024, 2024. 2, 3
[53] Yuting Xiao, Yanyu Xu, Ziming Zhong, Weixin Luo, Jiawei
Li, and Shenghua Gao. Amodal segmentation based on vis-
ible region segmentation and shape prior. In Proceedings of
the AAAI Conference on Artificial Intelligence, pages 2995–
3003, 2021. 2
[54] Yiming Xie, Chun-Han Yao, Vikram Voleti, Huaizu Jiang,
and Varun Jampani. Sv4d: Dynamic 3d content generation
with multi-frame and multi-view consistency. arXiv preprint
arXiv:2407.17470, 2024. 8, 13
[55] Katherine Xu, Lingzhi Zhang, and Jianbo Shi. Amodal com-
pletion via progressive mixed context diffusion. In Proceed-
ings of the IEEE/CVF Conference on Computer Vision and
Pattern Recognition, pages 9099–9109, 2024. 2, 3
[56] Lihe Yang, Bingyi Kang, Zilong Huang, Zhen Zhao, Xiao-
gang Xu, Jiashi Feng, and Hengshuang Zhao. Depth any-
thing v2. arXiv preprint arXiv:2406.09414, 2024. 4
[57] Jian Yao, Yuxin Hong, Chiyu Wang, Tianjun Xiao, Tong
He, Francesco Locatello, David P. Wipf, Yanwei Fu, and
Zheng Zhang. Self-supervised amodal video object segmen-
tation. Advances in Neural Information Processing Systems,
35:6278–6291, 2022. 2, 5, 6, 12
[58] Guanqi Zhan, Chuanxia Zheng, Weidi Xie, and Andrew
Zisserman. A general protocol to probe large vision
models for 3d physical understanding. arXiv preprint
arXiv:2310.06836, 2023. 3
[59] Guanqi Zhan, Chuanxia Zheng, Weidi Xie, and Andrew Zis-
serman. Amodal ground truth and completion in the wild.
In Proceedings of the IEEE/CVF Conference on Computer
Vision and Pattern Recognition, pages 28003–28013, 2024.
2, 3
[60] Xiaohang Zhan, Xingang Pan, Bo Dai, Ziwei Liu, Dahua
Lin, and Chen Change Loy. Self-supervised scene de-
occlusion. In Proceedings of the IEEE/CVF Conference
on Computer Vision and Pattern Recognition, pages 3784–
3792, 2020. 2, 4, 5, 6, 12
[61] Lvmin Zhang, Anyi Rao, and Maneesh Agrawala. Adding
conditional control to text-to-image diffusion models. In
Proceedings of the IEEE/CVF International Conference on
Computer Vision, pages 3836–3847, 2023. 2, 3, 4
[62] Yan Zhu, Yuandong Tian, Dimitris Metaxas, and Piotr
Dollár. Semantic amodal segmentation. In Proceedings
of the IEEE Conference on Computer Vision and Pattern
Recognition, pages 1464–1472, 2017. 2

11
 Appendix that recovers amodal masks within occluder areas based
on frame-level object ordering recovery. ’AISFormer’ [51]
In this appendix, we extend the discussion of our ap- employs a transformer-based head appended to a modal seg-
proach on video amodal segmentation. We first discuss ad- mentation backbone to directly predict all amodal bounding
ditional setup details for our method (Sec. A), and then boxes and masks within an image. ‘pix2gestalt,’ [36] is an
cover more experimental analysis (Sec. B), followed by ex- image diffusion-based method that generates amodal con-
amples of our method’s potential applications (Sec. C). We tent conditioned on the RGB image and modal masks of the
also show more qualitative results from our method (Sec. objects.
D). Please see the project page for a video version of all
figures. For video-level amodal segmentation, ‘SaVos’ [57] em-
ploys a CNN-LSTM architecture that processes RGB and
A. Additional setup details modal mask patches, along with optical flow, to pre-
dict amodal masks and motions. ‘EoRaS’ [9] proposes
A.1. Inference details an object-attention encoder that incorporates Bird’s-Eye
During inference with our video diffusion model, we follow View (BEV) 3D information, relying on having access to
common practices [3] by employing the stochastic sampler groundtruth camera parameters. ‘C2F-Seg’ [11] leverages
from EDM [20]. We simplify this process by omitting the a vector-quantized latent space for coarse feature learning,
second-order correction and keeping the explicit Langevin- refined with a convolutional module; though designed for
like “churn” factors constant. The denoising process is per- image-level tasks, it extends to video segmentation using a
formed over 25 steps. Specifically, when denoising the la- spatial-temporal transformer block.
tents from zt to z0 for i ∈ {t, · · · , 1}, each denoising step For generic video regression approaches, ‘Video-
can be expressed as: MAE’ [50] is a transformer-based autoencoder that we
adapt for our task by setting the masking ratio to zero, ap-
(ẑi − Dθ (ẑi ; σi ))
ẑi−1 ← ẑi + (σi−1 − σi ) (2) plying supervised training, and using the decoder during in-
σi ference. ‘3D-UNet’ [4], the backbone of our video diffusion
Furthermore, we employ classifier-free guidance model, contains interleaved residual and transformer blocks
(CFG) [15] to balance the quality and diversity of the with spatial and temporal modules but is trained to perform
generated samples. During training, we randomly set the one-step generation without any iterative denoising.
conditioning to zero with a probability of ρ = 0.1 to
simulate the unconditional case. During inference, we
combine the conditional and unconditional predictions B. Additional experiments
using a guidance scale of s = 1.5, as defined as:
Note that the video versions of all qualitative results in this
F̃θ (z, c) = Fθ (z, ∅) + s(Fθ (z, c) − Fθ (z, ∅)) (3) and the following sections can be found directly on the
project page.
After denoising, the latent predictions are projected back
into pixel space using the VAE decoder, which yields
three-channel representations. To convert these into single-
channel binary masks in the amodal segmentation stage, we Improved results on MOVi-B/D. All results we report
sum the channel values (from 0 to 255) and binarize the pre- till now on MOVi-B/D follow prior work in segmenting ob-
dictions by thresholding. The threshold is chosen as a per jects in a region which is defined as a 100% extension of
channel pixel-value of 200. Finally, we take the union of the region enclosed by the input modal mask. Therefore,
the prediction with the input modal masks, ensuring modal all images are cropped to this region before being sent as
masks remain a subset of amodal masks and are properly input to any of the methods. This is different from the stan-
reflected in the output. dard protocol used in other datasets, where the entire im-
age is sent as input to the methods (without any cropping).
A.2. Baselines Here, we include results from training our model with the
In this section, we provide additional details of the image- entire image as input on the MOVi-B/D datasets. As shown
and video-level amodal segmentation methods used for in Table 5, this fix significantly improves metrics, with our
comparison. method achieving 4% and 6% gains in mIoU on MOVi-B
For image-level amodal segmentation, ‘Convex’ [60] and MOVi-D, respectively. Regression methods also benefit
generates the geometric convex hull of modal masks, while notably from this setting. We conclude that this is because
‘ConvexR ’ [60] refines this by including only the con- MOVi-B/D include many instances of complete occlusions
vex hull within occluded regions predicted by ‘PCNet- of objects, for which segmentation in a cropped region is
M’. ‘PCNet-M’ [60] is a self-supervised regression method not enough for predicting the amodal mask.

12
Table 5. Quantitative results on MOVi-B/D with uncropped Table 6. Ablation of SVD priors. We study the effect of us-
input. Enlarged modal region-cropped input limits the model’s ing pretrained SVD weights as initialization for our training. We
ability to predict an amodal mask when an object is fully occluded. find that leveraging priors from large-scale pretraining of SVD en-
Using the entire image as input restores the model’s ability to com- hances both our method and the 3D UNet baseline, with particu-
plete amodal masks fully, especially when the modal area is small. larly substantial improvements observed for our method.
This results in substantial metric improvements compared to Table
2 in the main paper. We copy over the results here for reference. pretrained SAIL-VOS TAO-Amodal
Method
ckpt? mIoU mIoUocc AP25 AP50 AP75
MOVi-B MOVi-D Ours ✗ 68.89 26.96 93.73 79.45 57.87
Input Method
mIoU mIoUocc mIoU mIoUocc
Ours ✓ 75.17 51.28 94.89 85.03 66.87
VideoMAE [50] 78.74 42.86 70.93 32.78
modal 3D-UNet 82.16 49.81 75.65 40.86 3D UNet ✗ 70.85 32.66 94.88 83.81 59.75
cropped Ours (Top-1) 83.51 53.75 77.03 44.23 3D UNet ✓ 72.79 39.54 94.59 83.83 64.33
Ours (Top-3) 83.93 54.56 77.76 45.6
VideoMAE [50] 85.35 49.53 79.13 42.41
3D-UNet 84.24 46.17 76.90 36.69 Table 7. Ablation study on end-to-end amodal content com-
uncropped
Ours (Top-1) 87.8 53.69 82.97 47.86 pletion. We train an end-to-end version of our two-stage pipeline
Ours (Top-3) 88.43 54.64 84.04 49.43 with a dataset of curated modal-amodal RGB training pairs from
SAIL-VOS, in a similar fashion to pix2gestalt [36]. Compared
to the two-stage results in Table 1 of the main paper, this ap-
proach shows a significant performance drop in both in-domain
and zero-shot evaluations, highlighting the superiority of the two-
stage method.

SAIL-VOS TAO-Amodal
Method
mIoU mIoUocc AP25 AP50 AP75
Two-stage 77.07 55.12 97.28 89.25 71.99
Figure 10. We show how pseudo-depth aids amodal segmenta- One-stage 66.15 40.31 70.65 57.51 37.22
tion. Object’s surrounding regions with lower depth values, i.e.,
closer to the camera, act as potential occluders. In the top row, the
occluders are the person and chair to the left of the object; in the Building an end-to-end segmentation and completion
bottom row, the occluder is the car door below the person. Depth
model Unlike our two-stage method, which first performs
information implicitly guides our method to complete these oc-
cluded regions.
amodal segmentation and then inpaints content, the image
diffusion-based method pix2gestalt [36] adopts a one-stage
approach to directly generate amodal content and derive
masks. A similar one-stage approach can be extended for
Qualitative evidence for pseudo-depth conditioning.
our video setting. However, as shown in Table 7, our two-
The quantitative advantage of pseudo-depth conditioning
stage method demonstrates clear advantages over the one-
was demonstrated in Table 3 of the main paper. Here, we
stage approach. We attribute this low performance of the
provide qualitative evidence to illustrate the source of this
end-to-end method to the lack of data available for train-
improvement. As shown in Figure 10, pseudo-depth condi-
ing such a single-stage method. In contrast, the two-stage
tioning encourages our method to segment areas closer to
method benefits from breaking down the pipeline into video
the camera, suggesting that depth serves as an implicit indi-
amodal segmentation and content completion. For the for-
cator of potential occluders and therefore, gives information
mer, it is easy to find large-scale training data of modal-
about which occluded boundary to extend in order to predict
amodal mask pairs from synthetic datasets. For the latter,
the amodal mask.
since the content completion task reduces to video inpaint-
ing, less amount of training data is sufficient for finetuning.
Ablation on weights initialization. We leverage the real- C. Examples of applications
world priors learnt by large-scale diffusion models by utiliz-
ing pretrained SVD checkpoints [3]. Here, we evaluate the 4D reconstruction. Our method enables 4D reconstruc-
importance of this initialization. In Table 6, we compare tion for occluded objects when used in conjunction with off-
the performance of our model and the 3D U-Net with and the-shelf SV4D [54]. In Figure 11, we compare reconstruc-
without pretrained weights. Results show that excluding the tions with and without completion. Without completion,
checkpoint leads to a performance drop for both models, blank regions appear in occluded areas, making it more dif-
with a more pronounced decline for ours. These results un- ficult to hallucinate reasonable re-projections across differ-
derscore the importance of the SVD priors. ent views. In contrast, our method allows SV4D to produce

13
 Figure 12. Scene manipulation examples. Using de-occluded
objects from our method, we can reposition and reorder them to
create new scenes. In the top rows, the relationship between the
person and the soccer ball is altered, changing the scene from “the
person is juggling” to “the person places the soccer ball aside and
practices a juggling posture.” In the bottom rows, the middle gi-
raffe is moved to the front and its position is adjusted.
Figure 11. 4D reconstruction results. Without amodal comple-
tion by our method, the 4D reconstruction exhibits blank regions Table 8. Pseudo-groundtruths on TAO-Amodal. We show
and unrealistic artifacts in occluded areas, such as the person’s that using the amodal bounding box prior from the TAO-Amodal
back and leg. The varying occluded portions over time confuse dataset to specify the extent of the output amodal segmentation
SV4D, disrupting its understanding of the object’s 3D structure. mask, can help improve the quality of video amodal segmenta-
In contrast, using completed objects from our method significantly tion. We use this version of our method to produce ‘pseudo-
improves the reconstruction quality, producing more consistent groundtruths’ for TAO-Amodal. We find that these pseudo-
and clear novel-views. annotations can help improve the quantitative performance of
baselines like VideoMAE. See text for more details

consistent and clearer 4D reconstructions. SAIL-VOS TAO-Amodal

Input setting
mIoU mIoUocc AP25 AP50 AP75
Scene manipulation. With amodally completed objects Uncropped 77.07 55.12 97.28 89.25 71.99
in the scene, we can change their orderings and positions Amodal cropped 87.44 69.81 99.59 99.59 99.48
without exposing previously occluded regions. Figure 12
shows examples of scene manipulation, where our method
facilitates manual re-composition of scenes by inpainting tuning baselines like VideoMAE (which have already been
the occluded content of objects. pre-trained on SAIL-VOS), improves their performance on
the TAO-Amodal dataset by around 25%, 25%, and 20%
Pseudo-groudtruth for TAO-Amodal masks. TAO- on AP25 , AP50 , and AP75 respectively. Apart from this,
Amodal [18] provides ground truth for amodal bounding the generated pseudo-groundtruths can be used to semi-
boxes but lacks annotations for amodal masks due to the automate the amodal mask annotation process as this is a
challenges of manual labeling of occluded objects in videos. challenging and inherently ill-posed problem.
We show that our method can be used to generate high- Note that we do not include this data point in the main
quality pseudo-ground truth masks for this dataset by us- paper as at inference we cannot expect to have access to
ing the information about ground-truth amodal bounding amodal bounding boxes but in order to produce pseudo-
boxes, which define the extent of the amodal shape. We groundtruth, one can adopt this approach.
find that using the amodal bounding boxes to crop the input
modal mask sequences, one can train a more accurate video D. Qualitative results
amodal segmentation method exclusively on SAIL-VOS.
This way, our approach significantly improves evaluation A video version of all figures in this section are available on
metrics and aligns precisely with the amodal bounding box the project page.
extent, as shown in Table 8. Figure 13 further illustrates the Here, we present qualitative results from all datasets and
qualitative results of the pseudo-ground truth masks which additional, in-the-wild scenarios. Figures 15, 16, 17, 18 and
are high-fidelity across diverse object categories. Quantita- 19 compare our amodal segmentation method with more
tively, we find that using the pseudo-groundtruths for fine- baselines on SAIL-VOS, TAO-Amodal, and MOVi-B/D.

14
Figure 13. Qualitative results for pseudo-ground truth of TAO-Amodal masks. Leveraging the amodal bounding box as a strong
prior, our method demonstrates versatility across diverse categories, such as person, tractor, and bottles, and generalizes well to unseen
categories like snowboards and horses. This high-quality pseudo-ground truth can semi-automate the manual annotation of amodal masks
in real-world videos.

Figure 14. Qualitative analysis of failure cases of our method. See text for more details.

Our method demonstrates superior performance in gener- is occluded throughout the extent of the video; our model is
ating high-fidelity shapes in the occluded regions of ob- not able to understand if this is a completely visible object
jects. Figure 20 showcases additional in-the-wild content or a consistently occluded object. In the second case, the
completion results, highlighting the photo-realistic quality occluded object is a bow, which has never been seen before
and strong generalization capability of our method. and is completely out-of-distribution from the set of objects
in SAIL-VOS. Our method fails in this case. In the third
and fourth case, our method incorrectly assumes the height
Failure cases. In Figure 14, we show four different kinds of a completely visible man to be greater than what it is,
of failure cases. In the first case with a person swimming, and predicts a sitting person to be standing. Therefore, our
our method does not successfully complete the person’s method lacks contextual cues about what the scene is and
amodal region. This happens often if the object of interest how the modal region looks like in the first-stage.

15
Figure 15. Qualitative results on SAIL-VOS. (1/2)

16
 Figure 16. Qualitative results on SAIL-VOS. (2/2)

Figure 17. Qualitative results on TAO-Amodal. (1/2)

17
Figure 18. Qualitative results on TAO-Amodal. (2/2)
18
 Figure 19. Qualitative results on MOVi-B/D.

Figure 20. Qualitative results for amodal content completion for in-the-wild scenarios.

19