Sparse-LaViDa: Sparse Multimodal Discrete Diffusion Language Models

Source

• Raw Markdown: paper_sparse-lavida-2025.md
• PDF: paper_sparse-lavida-2025.pdf
• Preprint: arXiv 2512.14008v2
• Official project page: Sparse-LaViDa

Status And Credibility

The paper was first submitted on 2025-12-16 and revised to v2 on 2026-07-15. Shufan Li’s official publication page lists it as CVPR 2026. The Adobe/UCLA authors and tier-1 venue make it credible current evidence, but artifact status is weaker than LaViDa-O: the official project page labels its GitHub link as a placeholder pointing to the LaViDa-O repository, and no separate Sparse-LaViDa checkpoint release was verified at ingest time.

Core Claim

Sparse-LaViDa accelerates LaViDa-O by avoiding repeated computation over masked tokens that do not need to be decoded at the current diffusion step. It introduces:

• a sparse parameterization that materializes the prompt, already revealed tokens, and only the masked subset selected for the current step;
• learned register tokens that replace model capacity lost when semantically empty mask positions are truncated, rather than summarizing hidden future content;
• a step-causal attention mask that makes revealed tokens cacheable while preserving bidirectional interaction within each current masked block.

Unlike left-to-right block diffusion, the selected masked positions do not need to form a contiguous suffix, which preserves image inpainting, arbitrary spatial reveal order, and constrained infilling.

Evidence

Task	LaViDa-O latency	Sparse-LaViDa latency	Reported result
1024-pixel text-to-image	21.27 s	10.86 s	1.96x speedup with comparable GenEval.
Image editing	63.98 s	22.55 s	2.83x speedup and slightly higher ImgEdit score.
MathVista with 1024-token reasoning budget	10.41 s	3.72 s	2.80x speedup with nearly unchanged accuracy.

The paper’s ablations show that prompt caching, revealed-token caching, and mask truncation each contribute to speed. Register tokens matter more for fine visual detail than high-level prompt alignment. Applying sparse inference without fine-tuning collapses quality, and removing the step-causal training mask also degrades results.

The paper is internally inconsistent about post-training hardware: the main setup names 64 H100 GPUs, while the appendix names 64 A100 GPUs for five days and reports roughly 15% of LaViDa-O’s from-scratch training budget. The budget comparison is therefore useful but the exact hardware attribution is unresolved. This is an efficient continuation, not a training-free sampler.

Official Narrative Versus Paper Evidence

The project page advertises broad ~2x acceleration. The paper supports large gains for long output sequences, especially images and long reasoning traces, but explicitly finds little or no speedup for short QA and object grounding where outputs fit in one 32-token block. The durable claim is therefore length-dependent sparse decoding, not universal inference acceleration.

Foundation TSFM Relevance

Agenda slot	Verdict	Evidence	Missing pieces
Dynamic compute allocation	adjacent	Selects a subset of masked output positions per step and caches previously revealed tokens.	Selection follows a fixed block schedule; no learned allocation over temporal regimes, channels, or uncertainty.
Extra-long context and memory	adjacent	Separates cacheable prompt/revealed tokens from truncated unknown tokens and uses learned registers to restore capacity after removing semantically empty mask positions.	Registers do not summarize hidden future content; evidence concerns output decoding, not streaming history retention or persistent system state.
Multi-modal future distributions and generation	adjacent	Preserves arbitrary-order image/text infilling while reducing sequence length.	No numeric trajectory generation, calibrated multi-future coverage, or event timing.
Benchmarks	warning	Large speedups appear on long sequences but disappear on short-output tasks.	TSFM evaluation must stratify by horizon length, sequence sparsity, batch size, hardware, and quality tolerance.

Limitations

• The method requires additional training; it is not a drop-in training-free cache.
• Speedup benefits long generation and does not improve short QA or one-step grounding.
• Experiments use post-training from LaViDa-O rather than pretraining the sparse parameterization from scratch.
• The model inherits LaViDa-O hallucination and image-editing pixel-shift failure modes.
• No separate official code or checkpoint release was verified at ingest time.
• The main setup and appendix disagree on whether the 64-GPU post-training run used H100 or A100 hardware.

Links Into The Wiki

• Sparse-LaViDa
• LaViDa-O
• LaViDa-R1
• Diffusion Language Models
• Inference Dynamics
• Time-Series Scaling And Efficiency
• Extra-Long Context Time Series
• GPU Inference Optimization

Open Questions

• Can learned uncertainty or regime structure choose which future time-series positions to materialize rather than using fixed blocks?
• Can register tokens restore sparse-decoding capacity without hiding losses in rare visual details or low-frequency output structures?
• How does sparse masked decoding compare with autoregressive KV caching under matched batch size, output length, quality, and end-to-end latency?