In Omnimatte, one aims to decompose a given video into semantically meaningful layers, including the background and individual objects along with their associated effects, such as shadows and reflections. Existing methods often require extensive training or costly self-supervised optimization.
In this paper, we present OmnimatteZero, a training-free approach that leverages off-the-shelf pre-trained video diffusion models for omnimatte. It can remove objects from videos, extract individual object layers along with their effects, and composite those objects onto new videos.
These are accomplished by adapting zero-shot image inpainting techniques for video object removal, a task they fail to handle effectively out-of-the-box. To overcome this, we introduce temporal and spatial attention guidance modules that steer the diffusion process for accurate object removal and temporally consistent background reconstruction.
We further show that self-attention maps capture information about the object and its footprints and use them to inpaint the object's effects, leaving a clean background. Additionally, through simple latent arithmetic, object layers can be isolated and recombined seamlessly with new video layers to produce new videos.
Evaluations show that OmnimatteZero not only achieves superior performance in terms of background reconstruction but also sets a new record for the fastest Omnimatte approach, achieving real-time performance with minimal frame runtime.
Overview of our Object Removal strategy in OmnimatteZero. (a) We first identify potential background correspondences across frames. (b) Temporal Attention Guidance (TAG): Temporal attention scores between a foreground point and its background correspondences are replaced with the average attention between all background pairs, promoting consistent inpainting across time. (c) Spatial Attention Guidance (SAG): Within a frame, the attention from a foreground point to nearby background points is adjusted to reflect the mean attention among background points themselves, improving inpainting quality when temporal context is unavailable.
(a) Pretrained video diffusion models can associate objects with their effects by analyzing self-attention maps between query and key tokens related to the object of interest [Lee et al, 2025].
We propose to directly derive the masks from the attention maps, allowing a training-free object approach for removing objects with their associated effects.
(b) Unlike video diffusion models, image models do not capture object effects from still images. This aligns with the principle of common fate in Gestalt psychology.
@inproceedings{samuel2025omnimattezero,
author = {Dvir Samuel and Matan Levy and Nir Darshan and Gal Chechik and Rami Ben-Ari},
title = {OmnimatteZero: Fast Training-free Omnimatte with Pre-trained Video Diffusion Models},
booktitle = {SIGGRAPH Asia 2025 Conference Papers},
year = {2025}
}