AI
Xiaomi targets flawed scores for AI object removal
Xiaomi’s MiLM Plus introduces PROVE, reference-free metrics that better align video object-removal scores with human judgments.
Object-removal systems can produce convincing shadows, reflections, and reconstructed structures while earning worse scores from standard image metrics. Xiaomi’s MiLM Plus team is addressing that mismatch with PROVE, a reference-free evaluation framework accepted at ACM MM 2026, according to Marktechpost.
PROVE evaluates the edited region rather than the entire frame through two metrics:
- RC-S measures spatial coherence in a single frame.
- RC-T measures temporal consistency across adjacent frames.
Both use sliding-window Maximum Mean Discrepancy (MMD) over DINOv2 features and do not require a reference video. RC-S separates connected objects in the removal mask, expands each object’s bounding box by one-third, and compares masked features with nearby background features using a Gaussian RBF kernel. RC-T crops adjacent frames jointly, uses the union of their masks for alignment, and computes MMD only in the intersection—the area restored in both frames.
That local approach is designed to avoid weaknesses in metrics such as PSNR, SSIM, LPIPS, ReMOVE, and CFD. Full-reference metrics can reward copy-paste results because object removal is a one-to-many problem with no single correct reconstruction. Existing no-reference methods can also reward blurred regions or overlook residual shadows. Full-frame temporal metrics are similarly insensitive when the edited area occupies only a small part of the image.
PROVE’s benchmark results
Against rankings from 20 human participants, RC-S achieved an average Kendall’s τ of 0.59 and Spearman’s ρ of 0.66. ReMOVE scored 0.26/0.29, while CFD scored 0.16/0.18. RC-S ranked first on five of six tested benchmarks.
On RORD-Val, RC-S selected clean images over blurred and region-swapped versions in 100% of cases. ReMOVE reached 60.06% under blur, and CFD reached 49.27%. RC-T also responded monotonically as temporal corruption increased, unlike the compared Temporal Consistency and Temporal Flickering metrics.
Ablation tests found that DINOv2 outperformed DINOv3 and SAM, with average Kendall’s τ values of 0.59, 0.51, and 0.44, respectively. Removing the sliding window reduced performance by 0.11, while replacing MMD with cosine similarity reduced it by 0.07. RC-S was also 13.7× faster than CFD.
PROVE-Bench contains two tiers. PROVE-M includes 80 paired videos, each 81 frames at 1080p, captured from real-world footage and processed with frame-refined SAM3 masks, quality control, and synchronized Ken Burns-style motion augmentation. PROVE-H adds 100 difficult videos without ground truth, including crowds, flowing water, flames, textured terrain, reflections, and fast motion.
On the public leaderboard, SVOR, a 1.3-billion-parameter model, leads combined RC-S with 0.5197, while EffectErase leads RC-T with 0.2525.
PROVE is available as an Apache 2.0 PyTorch repository with a single CLI entry point, run_prove_metrics.py. It requires Python 3.10 or later, PyTorch 2.6 or later, Transformers 4.51 or later, DINOv2-giant weights, and removal masks in which white pixels identify the object.
The authors report RC-S running at 134.6 milliseconds per frame on one RTX 4090, making it suitable for offline evaluation and continuous-integration quality gates. The framework is not intended for real-time on-device scoring, and its cropped evaluation can miss side effects such as large shadows or reflections extending beyond the region being assessed.