--- abstract: | Modern Vision-Language Models (VLMs) can solve a wide range of tasks requiring visual reasoning. In real-world scenarios, desirable properties for VLMs include fast inference and controllable generation (e.g., constraining outputs to adhere to a desired format). However, existing autoregressive (AR) VLMs like LLaVA struggle in these aspects. Discrete diffusion models (DMs) offer a promising alternative, enabling parallel decoding for faster inference and bidirectional context for controllable generation through text-infilling. While effective in language-only settings, DMs' potential for multimodal tasks is underexplored. We introduce `\ours`{=latex}, a family of VLMs built on DMs. We build `\ours`{=latex} by equipping DMs with a vision encoder and jointly fine-tune the combined parts for multimodal instruction following. To address challenges encountered, `\ours`{=latex} incorporates novel techniques such as complementary masking for effective training, prefix KV cache for efficient inference, and timestep shifting for high-quality sampling. Experiments show that `\ours`{=latex} achieves competitive or superior performance to AR VLMs on multi-modal benchmarks such as MMMU, while offering unique advantages of DMs, including flexible speed-quality tradeoff, controllability, and bidirectional reasoning. On COCO captioning, `\ours`{=latex} surpasses `\openlnxt`{=latex} by +4.1 CIDEr with 1.92$\times$ speedup. On bidirectional tasks, it achieves +59% improvement on Constrained Poem Completion. These results demonstrate `\ours`{=latex} as a strong alternative to AR VLMs. Code and models is available at author: - | Shufan Li$^{1*}$, Konstantinos Kallidromitis$^{2*}$, Hritik Bansal$^{1*}$, Akash Gokul$^{4*}$, Yusuke Kato$^2$\ **Kazuki Kozuka$^2$, Jason Kuen$^3$, Zhe Lin$^3$, Kai-Wei Chang$^1$, Aditya Grover$^1$**\ $^1$UCLA $^2$Panasonic AI Research $^3$Adobe Research $^4$Salesforce Research\ \* Equal Contribution\ bibliography: - ref.bib title: "LaViDa: A Large Diffusion Language Model for Multimodal Understanding" --- \newcommand{\ag}[1]{\textcolor{red}{[AG:#1]}} \newcommand{\hb}[1]{\textcolor{blue}{HB:#1}} \newcommand{\jk}[1]{\textcolor{brown}{#1}} \newcommand{\kk}[1]{\textcolor{purple}{[KK: #1]}} \newcommand{\textgreen}[1]{\textcolor{green!50!black}{#1}} \newcommand{\TODO}[1]{\textcolor{red}{[TODO:#1]}} \newcommand{\ours}[0]{LaViDa} \newcommand{\llava}[0]{LLaVa} \newcommand{\lnxt}[0]{LLaVa-1.6-7B} \newcommand{\openlnxt}[0]{Open-LLaVa-Next-Llama3-8B} \maketitle \sidecaptionvpos{figure}{c} \begin{SCfigure}[50][h]\includegraphics[scale=0.6]{images/teaser-8.pdf} \caption{\textbf{We propose \ours,} the first family of diffusion-based discrete VLM models. \ours~ models achieve competitive performance against AR baselines (LLaVa-1.6, Open-LLaVa-Next) across multiple visual understanding tasks including MMMU (world knowledge), MathVista (reasoning), ChartQA (OCR), and ScienceQA (science).} \label{fig:teaser_graph} \end{SCfigure} # Introduction Vision-Language Models (VLMs) have shown remarkable utility across diverse domains, from end-user applications like virtual assistants [@li2024multimodal], to research tasks such as scientific image captioning and document understanding [@liu2023visual; @chen2024far; @bai2025qwen25-vl; @alayrac2022flamingo; @liu2024nvila]. In industrial settings, VLMs support automated product tagging, content moderation, and quality control in manufacturing [@team2024gemini; @anthropic2024claude]. Their ability to jointly process visual and textual information makes them indispensable for practical applications and cutting-edge research. Currently, nearly all popular VLMs---such as Qwen-VL [@bai2025qwen25-vl; @wang2024qwen2-vl], Intern-VL [@chen2024internvl; @zhu2025internvl3], and GPT-4 [@openai2024gpt4o] are built on top of large language models (LLMs) that generate text in an autoregressive (AR) manner; that is, they produce tokens one by one in a left-to-right sequence. While these models have demonstrated strong performance on many tasks, they suffer from several key limitations. First, their sequential generation process is inherently hard to parallelize, resulting in slow inference speed [@bengio2015scheduled]. More critically, their left-to-right generation makes it difficult to handle tasks that benefit from bidirectional context or structural constraints---such as text infilling [@fried2022incoder]. For example, generating a poem where each line starts with a specific syllable, or extracting structured information from an image in a predefined JSON format, often requires the model to fill in or coordinate content across the sequence. Autoregressive models still struggle to consistently satisfy such constraints, even with carefully crafted prompts and examples. Recently, discrete diffusion models (DMs) have emerged as a promising alternative to AR LLMs. Most notably, LLaDA [@nie2025large] and Dream [@dream2025] achieved comparable results to AR LLMs across diverse language tasks. Unlike AR LLMs, DMs treat text generation as a diffusion process over discrete tokens. A forward process gradually corrupts a sequence of discrete text tokens into a sequence of mask tokens. At inference, we start with a sequence of mask tokens and gradually transform them into a sequence of meaningful text tokens through a learned reverse process. Compared to AR LLMs, diffusion models offer several theoretical advantages that directly address the limitations of autoregressive generation. While AR LLMs have a fixed throughput---generating one token at a time---DMs allow flexible control over the speed-quality trade-off by adjusting the number of diffusion steps [@dream2025; @sahoo2024simple; @lou2023discrete-sedd; @nie2025large]. Moreover, their ability to model bidirectional context makes them well-suited for tasks like text infilling, enabling more effective constrained generation and structured output formatting---capabilities especially valuable in vision-language settings where outputs may need to follow specific schemas. In this work, we propose `\ours`{=latex} (**L**arge **Vi**sion-Language **D**iffusion Model with M**a**sking), the first family of VLMs based on diffusion. `\ours`{=latex} enables pretrained DMs to perceive visual inputs by integrating vision features into the diffusion backbone via a vision encoder---analogous to how LLaVA [@liu2023visual; @liu2024improved] augments large language models (LLMs) with visual inputs. Specifically, we adopt a two-stage training pipeline with a diffusion objective: pretraining followed by supervised fine-tuning. Adapting DMs for vision-language tasks presents several practical challenges. First, standard DM training is data-inefficient: the model learns only from the subset of corrupted tokens at each timestep. For example, in a question-answering task where the target answer is `“The answer is dog”`, the corruption process may mask `“The [mask] is dog”`, leaving the key token `“dog”` unmasked and thus excluded from the loss. This is especially problematic for multimodal tasks, where answer tokens often carry crucial semantic content grounded in the image [@raffel2020exploring]. To address this, we introduce a complementary masking scheme that ensures every token in the output sequence contributes to learning, improving data efficiency. Second, inference algorithms used by existing DMs are slow in practice due to the lack of KV cache support---an inherent limitation of bidirectional context modeling [@arriola2025block]. This leads to repeated recomputation over the full prompt at every decoding step. While tolerable for short-text settings, it becomes a significant bottleneck in vision-language tasks, where multimodal prompts may include hundreds of visual tokens. To address this, we propose Prefix-DLM decoding, a simple yet effective technique that enables caching of multimodal prompts (i.e., image and question tokens), significantly accelerating inference. Lastly, DMs offer a unique advantage over autoregressive models: the ability to trade off speed and quality by varying the number of diffusion steps. However, the widely used linear masking schedule---which unmasks a fixed number of tokens per step---performs poorly at low step counts. Motivated by text-to-image diffusion models like SD3 [@esser2024scaling-sd3], we adopt a timestep shifting strategy that adaptively adjusts how many tokens are unmasked per step. This leads to better sample quality under aggressive step reduction, allowing faster generation without large degradation in quality. We conducted extensive evaluations of `\ours`{=latex} across a wide range of vision-language tasks. Results show that `\ours`{=latex} achieves competitive performance on most benchmarks, including MMMU [@yue2023mmmu], MathVista [@lu2023mathvista], ChartQA [@masry-etal-2022-chartqa] and ScienceQA [@lu2022learn], when compared with AR VLMs like `\lnxt `{=latex} [@liu2024llavanext; @liuhaotian_llava_v1_6_vicuna_7b] and `\openlnxt `{=latex} [@chen2024open]. We highlight these results in Figure `\ref{fig:teaser_graph}`{=latex}. On constrained generation tasks, `\ours`{=latex} greatly outperforms AR baselines (+59% on Poem Completion). It also supports flexible speed-quality tradeoff, achieving higher quality on COCO image captioning (+ 4.1 CIDEr) and a 1.92$\times$ speedup [@lin2014microsoft]. In summary, our contributions are: - We introduce `\ours`{=latex}, the first family of VLMs based on diffusion models. Our models achieve competitive performance across a wide range of tasks compared to AR VLMs, while offering the unique benefits of DMs. - We introduce several novel training and inference techniques for DMs, including complementary masking, Prefix-DLM, and timestep shifting that improve the training efficiency, inference speed, and sample quality of `\ours`{=latex}. - We conduct a systematic study of various design choices for adapting DMs to vision-language tasks (e.g. image resolution), offering insights for future work in this direction. # Background and Related Works ## Vision-Language Models Vision-Language Models (VLM) extend the capability of Large Language Models to visual understanding tasks [@li2024llava; @liu2023visual; @wang2024qwen2-vl; @bai2025qwen25-vl; @cambrin2024level; @zhu2025internvl3; @openai2024gpt4o]. The common recipe to build a VLM is to start with a strong large language model and combine it with a vision encoder [@liu2023visual; @wang2024qwen2-vl]. These VLMs typically undergo multiple stages of training, which can be generally categorized into pretraining and finetuning stages. The pretraining data usually consists of text-image pairs for vision-language alignment, while the finetuning data consists of a wide range of instruction-following tasks. There are several dedicated lines of work focusing on different components of this overarching framework, such as the design of vision encoders [@guo2024llava; @yang2024pvc] and training techniques [@lin2024vila; @wang2024q]. To this date, most vision-language models such as LLaVa [@liu2023visual; @li2024llava] and Qwen-VL [@bai2025qwen25-vl; @wang2024qwen2-vl] series employ an autoregressive training objective. ## Diffusion Language Models Diffusion models first emerged as a powerful alternative to GANs for generating continuous data such as images [@rombach2022high; @podell2023sdxl; @esser2024scaling-sd3]. Early explorations in diffusion language models directly built continuous diffusion models for latent text embeddings [@li2022diffusion; @lovelace2023latent], with limited success. More recently, discrete diffusion models [@austin2021structured-d3pm; @sahoo2024simple; @lou2023discrete-sedd; @shi2024simplified] have proven to be better candidates for language modeling, achieving performance comparable to AR models while offering unique advantages, such as speed-quality tradeoffs and controllability. Most notably, LLaDa-8B and Dream-8B [@nie2025large; @dream2025] demonstrated that DLMs can achieve competitive performance against AR LLMs at scale. Formally, given a text sequence of $L$ tokens $X_0=[X_0^1,X_0^2...X_0^L]$, the forward process $q(X_t|X_s)$ gradually converts it to a sequence full of mask tokens \"\[M\]\", denoted by $X_1=[X_1^1,X_1^2...X_1^L]$, through the continuous time-interval $[0,1]$, with $1\ge t\ge s\ge0$. A neural network $p_\theta$ is used to model the reverse process $p(X_s|X_t)$. The diffusion language modeling objective can be defined as: $$\begin{equation} \mathcal{L_{\text{DLM}}}=-\mathbb{E}_{t,X_0,X_t}\left[\frac{1}{t}\log p_\theta(X_0|X_t)\right] \label{eq:dlm-obj-ref} \end{equation}$$ where $\log p_\theta(X_0|X_t)$ is assumed to be factorized into $\prod_{i=1}^L p_\theta(X_0^i|X_t)$. At each training step, we sample $t$ uniformly from the interval $[0,1]$ and sample $X_0$ from some data distribution $\mathcal{D}$. $X_t$ is then sampled from the forward process $q(X_t|X_0)$. The loss is only computed over the masked tokens in $X_t$, since $p_\theta(X_0^i|X_t)$ has a closed form representation that does not depend on $\theta$ when $X_t^i\ne[M]$. We offer additional background on the details of these formulations in Appendix `\ref{sec:appendix-dm-background}`{=latex}. We also incorporate addition backgrounds on other relevant literature, such as masked generative models [@chang2023muse; @chang2022maskgit] and multimodal diffusion models [@tang2023any-codi; @li2024omniflow], in Appendix `\ref{sec:appendix-background-extra}`{=latex}. `\label{sec:background-dlm}`{=latex} # Method ## Model Architecture {#sec:model-arch}
Overall design of . ’s architecture consists of a vision encoder, a diffusion language model, and an MLP vision projector. The bottom half of the figure illustrates the image encoding process, while the top half depicts the diffusion language modeling process. These two pipelines are described in detail in Sec. .
`\ours`{=latex}'s model architecture follows a similar design to common AR VLMs like LLaVa [@li2024llava]. It consists of a vision encoder and a diffusion language model. These two parts are connected by a MLP projection network. The overall design is illustrated in Figure `\ref{fig:explainer}`{=latex}. **Vision Encoder**. Given an input image $I$ and text prompt $P$, we first resize the image to $768^2$ and divide it into four non-overlapping views of $384^2$, denoted $I_{1:4}$. We also resize the original image to $384^2$ to obtain a fifth view, $I_5$, following the design of prior works [@li2024llava; @liu2024llavanext]. These five views are independently encoded by the vision encoder (SigLIP-400M [@zhai2023sigmoid]), each producing $27^2$ embeddings, denoted $V_{1:5}$. In total, this yields 3645 embeddings per image. To reduce sequence length for efficient training, we apply $2\times2$ average pooling on each view, reducing embeddings to $14^2$ per view, or 980 total. The embeddings of five views are then flattened and concatenated into a 1D sequence before being processed by the projection network to obtain the final visual context of the diffusion language model. This process mirrors the vision encoding process in AR LLMs [@guo2024llava] and is illustrated in the bottom part of Figure `\ref{fig:explainer}`{=latex}. **Diffusion Language Model**. The diffusion language model is a multi-layer Transformer [@vaswani2017attention] whose architecture resembles that of LLMs. The only major difference is that its attention mask is non-causal, and it uses the diffusion language modeling objective described in Section `\ref{sec:background-dlm}`{=latex} instead of the next-token prediction used in AR models. The input to the diffusion language model consists of the projected vision embeddings, the prompt $P$, and partially masked response $X_t$. The outputs of the last transformer block are passed through a final linear layer to obtain token-wise logits $p_\theta(X^i_0|I,P,X_t)$ for the unmasked response $X_0$. In our experiments, we explored LLaDA-8B (default) and Dream-7B as our diffusion language model. This process is illustrated in the upper half of Figure `\ref{fig:explainer}`{=latex}. ## Training Algorithms {#sec:training-algo} Our training objective is based on the diffusion language modeling objective described in Section `\ref{sec:background-dlm}`{=latex}. Each training sample consists of an image $I$, text prompt $P$ and clean text answer $X_0$ from the training data. For multi-round conversation, we sample one round as the \"answer\" and treat the history as \"prompt\". We first sample a timestep $t\in[0,1]$ and a partially masked answer $X_t$ using the forward process described in Section `\ref{sec:background-dlm}`{=latex}. `\ours`{=latex} then implements the conditioned reverse process $p_\theta(X_0|I,P,X_t)$. The canonical diffusion vision-language modeling objective is formulated as:
Technical Details of . (a) We propose Complementary Masking to ensure loss is calculated over all tokens in the data for training efficiency. (b) We propose Prefix-DLM attention mask that enables KV caching. We visualize the attention mask of image tokens (Img1-2), prompt tokens (Pmt1-2), and text and mask tokens in the noise answer Xt (Ans1, [M], Ans3). Rows represent queries, while columns are keys. Colored squares indicate that queries and keys can interact.
$$\begin{equation} \mathcal{L_{\text{D-VLM}}}=-\mathbb{E}_{t,I,P,X_0,X_t}\left[\frac{1}{t}\log p_\theta(X_0|I,P,X_t)\right] \label{eq:dlm-obj} \end{equation}$$ where $p_\theta(X_0|I,P,X_t)$ factorizes into $\prod_{i=1}^L p_\theta(X_0^i|I,P,X_t)$ following the formulation in Section `\ref{sec:background-dlm}`{=latex}. Notably, the loss is only computed over mask tokens where $X_t^i=[M]$, because $p_\theta(X_0^i|I,P,X_t)$ is not dependent on $\theta$ when $X_t^i\neq [M]$. **Complementary Masking.** Prior diffusion language models (e.g., LLaDa, Dream) apply a stochastic estimator to Equation `\ref{eq:dlm-obj}`{=latex}, masking tokens independently across samples in a batch. However, for vision-language tasks, this leads to inefficiencies: (1) only $\sim$``{=html}50% of tokens contribute to the loss on average, and (2) critical answer tokens---often short and sparse in vision tasks like VQA---may not be masked, resulting in misaligned gradients for the vision encoder. For example, in `"The answer is dog."` the key token `"dog"` might be unmasked in $X_t$ and thus ignored in loss computation. To address this, we introduce complementary masking: for each sample, we generate two masked versions $X_t$ and $X_t^C$ with disjoint corrupted spans (e.g., one masks `"The [M] [M] dog ."`, the other `"[M] answer is [M] [M]"`, ensuring all tokens are eventually used in training and improving sample efficiency and gradient flow. When computing the loss over $X_t$ and $X_t^C$, we copy the encoded vision embeddings to further boost training efficiency. This process is illustrated in Figure `\ref{fig:complementary-mask}`{=latex}. ## Inference Algorithms At inference time, we first create a sequence of $L$ mask tokens as $X_1$, where $L$ is the response generation length. Then we gradually unmask them through $K$ discrete timestamps $t_1..t_K$, where $t_1=1$ and $t_K=0$, until we reach a clean, mask-free sequence $X_0$. At each timestamp $t_i$, we sample a fully unmasked sequence through $p_\theta(X_0|X_{t_i})$ and re-mask $L\times {t_{i+1}}$ tokens to obtain $X_{t_{i+1}}$. Both $L$ and $K$ are hyperparameters for inference. Additionally, we define $\frac{K}{L}$ as \"fraction of the number of functional evaluations (NFE)\" to measure sample efficiency. For example, when NFE = 100%, the diffusion model generates one token per forward pass; at NFE = 50%, it generates an average of two tokens per forward pass. Overall, the inference process of `\ours`{=latex} is similar to prior DMs such as LLaDa, with two key exceptions: **Prefix-DLM.** While DLMs theoretically offer superior speed--quality tradeoffs at inference, they are often slower than AR models in practice because they cannot leverage KV caching [@nie2025large]. This issue is particularly evident for multimodal prompts containing many visual tokens. To avoid recomputing keys and values for the visual embeddings and text prompts, we propose a novel Prefix-DLM scheme inspired by the autoregressive prefix-LM. Prefix-DLM adopts a specialized attention mask in which visual and prompt tokens can only attend to other visual and prompt tokens, while answer tokens can attend to all tokens. Figure `\ref{fig:prefix-dlm}`{=latex} illustrates this setup. With this design, we can cache the keys and values of the visual and prompt tokens. Empirically, this leads to a speedup of up to 3.9$\times$ on COCO captioning tasks. Further details are provided in Section `\ref{sec:speed-quality}`{=latex}. **Schedule Shift.** Diffusion language models (DLMs) allow trading speed for quality via the number of discretization steps $K$. Prior models like LLaDa and Dream use a linear schedule, unmasking $\frac{L}{K}$ tokens uniformly over $t \in [0,1]$. However, we find this leads to performance degradation at low sampling steps. Inspired by SD3 [@esser2024scaling-sd3], we adopt a shifting schedule: $$\begin{equation} t_i' = s_\alpha(t_i) = \frac{\alpha t_i}{1 + (\alpha - 1)t_i} \end{equation}$$ Here, $s_\alpha(t)$ is a monotonic map with $t_0 = t_0' = 0$, $t_K = t_K' = 1$. When $\alpha < 1$ (we use $\alpha = \frac{1}{3}$), the schedule is convex---leading to more tokens being unmasked earlier. We found that this setup outperforms alternatives. Notably, this conclusion differs from that of continuous diffusion models like SD3 and previous masked diffusion models for image generation [@chang2022maskgit], which showed concave schedules ($\alpha > 1$) are more preferable. We ensure at least one token is unmasked per step. Further details are provided in Section `\ref{fig:speed-quality-cooc}`{=latex} and Appendix `\ref{sec:appendix-timestep}`{=latex}. # Experiments ## Setup {#sec:exp-setup} At a high level, `\ours`{=latex} employs a two-stage training process. In the pretraining phase (stage-1), only the projector is updated to align the visual embeddings with the latent space of the DLM. In the finetuning phase (stage-2), we jointly train all components end-to-end for instruction-following. Additionally, we further finetune the stage-2 model for additional steps and obtain two specialized models for reasoning and text-infilling tasks (`\ours`{=latex}-Reason and `\ours`{=latex}-FIM). We provide more details of these specialized models in Section `\ref{sec:reasoning_distillation}`{=latex} and `\ref{sec:text-infilling}`{=latex}. We use 558K image-text pairs as our stage-1 data, and 1M visual instruction-following examples as our stage-2 data. Further details on the dataset and training setup are provided in Appendix `\ref{sec:appendix-setup}`{=latex}. We evaluate `\ours`{=latex} on a wide range of vision-language tasks. Unless otherwise stated, we report results obtained using the stage-2 model with LLaDa-8B as the language backbone. We use `lmms-eval` package [@zhang2024lmmsevalrealitycheckevaluation] for evaluation and set the sequence length $L$ to be the maximum generation length used for evaluating AR models. We set $K=L$, or NFE=$100\%$ by default. Results under differet NFE are explored in Section `\ref{sec:speed-quality}`{=latex} and Appendix `\ref{sec:appendix-math}`{=latex} and `\ref{sec:appendix-speed}`{=latex}. ## Main Results Table `\ref{tab:main-results}`{=latex} reports the results of `\ours`{=latex} using LLaDA-8B (`\ours`{=latex}-L) and Dream-7B (`\ours`{=latex}-D) as the language backbones on vision-understanding tasks. We compare with several open-source, open-data models with similar data sizes and parameter counts: `\lnxt `{=latex} [@liu2024llavanext; @liuhaotian_llava_v1_6_vicuna_7b] and `\openlnxt `{=latex} [@chen2024open]. We also include comparisons with frontier open-sourced models of similar size that are trained on larger datasets, namely LLaVa-OneVision-7B [@li2024llava], Qwen2.5-VL-7B [@bai2025qwen25-vl], and InternVL-3­8B [@zhu2025internvl3]. `\ours`{=latex} demonstrates competitive performance across a wide range of tasks spanning General, Reasoning, OCR, and Science categories. In general vision-language understanding, `\ours`{=latex}-L achieves the highest score on MMMU [@yue2023mmmu] ($43.3$), outperforming all comparable models. `\ours`{=latex}-D also ranks second on several benchmarks in this category. For reasoning tasks, both models surpass similarly scaled baselines on math-heavy and spatially grounded benchmarks. In Science, `\ours`{=latex} achieves the best and second-best scores on ScienceQA [@lu2022learn] ($81.4$ and $80.2$, respectively) while performing on par with Open-Llava-Next on AI2D [@kembhavi2016diagram], a complex diagram-based benchmark. Finally, in OCR `\ours`{=latex} shows competitive performance but lags behind some of the latest AR models. This gap is primarily due to our use of average pooling for vision token compression, which leads to the loss of fine-grained spatial information. While this was a necessary trade-off given our limited compute budget, it poses challenges for tasks requiring precise text recognition and layout understanding. These results highlight the strength of `\ours`{=latex}, demonstrating that diffusion-based approaches can scale competitively with AR models while achieving robust performance across a wide range of vision-language tasks. \renewcommand{\arraystretch}{1.2} \setlength{\tabcolsep}{4pt} \begin{tabular}{cHHcccc|ccccc} \toprule \textbf{} & \textbf{Split} & \textbf{Data} & \textbf{\ours-L} & \textbf{\ours-D} & \textbf{LLaVa-1.6} & \textbf{Open-Lnxt} & \textbf{L-OV} & \textbf{Qwen2.5} & \textbf{Intern3} \\ \hline \#Params & & & 8B & 7B & 7B & 8B & 7B & 7B & 8B \\ \makecell[c]{\#Images \\\textcolor{gray}{(Pretrain)}} & & & 0.6M & 0.6M & 0.6M & 0.6M & 0.6M & >7M & - \\ \makecell[c]{\#Images\\ \textcolor{gray}{(SFT)}} & & & 1.0M & 1.0M & 0.7M & 1.0M & 7.2M & $\sim$2M & 21.7M \\ \hline \textit{General} \\ MME-P \cite{fu2023mme} & test & General & 1365.6 & 1463.5 & \underline{1519.3} & \textbf{1610.9} & 1580.0 & - & - \\ VQAv2 \cite{balanced_vqa_v2} & val & General & 72.2 & \underline{75.2} & \textbf{80.1} & 71.9 & - & - & - \\ MMBench \cite{MMBench} & dev & General & 70.5 & \underline{73.8} & 54.6 & \textbf{74.4 }& 80.8 & 83.5 & 83.4 \\ MMMU \cite{yue2023mmmu} & val & General &\textbf{43.3} & \underline{42.6} & 35.1 & 37.4 & 48.8 & 58.6 & 65.6 \\ \hline \textit{Reasoning} \\ MME-C \cite{fu2023mme} & test & Reasoning & \underline{341.1} & \textbf{378.9} & 322.5 & 336.8 & 418.0 & - & - \\ MathVista \cite{lu2023mathvista} & testmini\_format & Reasoning &\textbf{44.8} & \underline{42.1} & 34.4 & 41.8 & 63.2 & 68.2 & 75.2 \\ MathVerse \cite{zhang2024mathverse} & testmini\_vision\_dominant & Reasoning & \textbf{27.2}& \underline{24.1} & 14.3 & 14.6 & - & - & - \\ MathVision \cite{wang2024measuring} & mathvision\_testmini & Reasoning & \textbf{20.4} & \underline{19.4} & 12.8 & 14.1 & - & -& - \\ \hline \textit{Science} \\ ScienceQA \cite{lu2022learn} & scienceqa-full & Science knowledge & \underline{80.2} & \textbf{81.4} & 73.2 & 78.3 & 96.0 & - & - \\ AI2D \cite{kembhavi2016diagram} & test & Science Diagrams & \underline{70.0} & 69.0 & 66.6 & \textbf{70.2 }& 81.4 & 83.9 & 85.2 \\ \hline \textit{OCR} \\ TextVQA \cite{singh2019towards} & val & OCR & 56.3 & 57.1 & \textbf{64.9} & \underline{61.7} & - & 84.9 & 80.2 \\ DocVQA \cite{mathew2021docvqa} & test & OCR & 59.0 & 56.1 & \textbf{74.4} & \underline{69.9} & 87.5 & 95.7 & 92.7 \\ ChartQA \cite{masry-etal-2022-chartqa} & test & OCR & \underline{64.6} & 61.0 & 54.8 & \textbf{69.7} & 80.0 & 87.3 & 86.6 \\ InfoVQA \cite{mathew2022infographicvqa} & test & OCR & 34.2 & 36.2 & \textbf{37.1} & \underline{36.7} & 68.8 & 82.6 & 76.8 \\ \bottomrule \end{tabular} ## Reasoning Distillation {#sec:reasoning_distillation} Prior work has distilled LLMs [@openthoughts] and VLMs [@deng2025openvlthinker; @wang2025vl] using long chain-of-thought (CoT) data to elicit strong reasoning capabilities [@guo2025deepseek]. In the same spirit, we study the reasoning abilities of `\ours`{=latex} by conducting additional stage-3 training using 19.2K CoT examples distilled from VL-Rethinker-7B, a strong reasoning model. We refer to the finetuned model as `\ours`{=latex}-Reason. We evaluate it on MathVista [@lu2023mathvista], MathVerse [@zhang2024mathverse], and MathVision [@wang2024measuring] with CoT generation, comparing against the stage-2 results without CoT. We set the maximum generation length $L=1024$ for these tasks. We report these results in Table `\ref{tab:cot-reason}`{=latex}. We find that `\ours`{=latex}-Reason outperforms `\ours`{=latex} across all benchmarks, with the most significant gains observed on the most challenging MathVision reasoning dataset ($+18\%$ relative improvement). Further details are provided in Appendix `\ref{sec:appendix-math}`{=latex}. \begin{subtable}[t]{0.50\textwidth} \caption{Math benchmark results after long CoT distillation. \label{tab:cot-reason}} \begin{tabular}{lccc} \toprule & M.Vista$\uparrow$ & M.Verse$\uparrow$ & M.Vision$\uparrow$ \\ \midrule \ours & 44.8 & 27.2 & 20.4 \\ \hline \makecell[c]{\ours\\-Reason} & \textbf{45.2} & \textbf{29.3} & \textbf{24.0} \\ \hline \textgreen{Rel. $\Delta$} & \textgreen{+1\%} & \textgreen{+8\%} & \textgreen{+18\%} \\ \bottomrule \end{tabular} \end{subtable} \hfill \begin{subtable}[t]{0.45\textwidth} \caption{Sentence and sample-level constraint satisfaction for poem completion tasks.} \begin{tabular}{lHcc} \toprule & & Sentence$\uparrow$ & Sample$\uparrow$ \\ \midrule \ours & DLM & \textbf{1.00} & \textbf{1.00} \\ \ours-FIM & DLM & \textbf{1.00} & \textbf{1.00} \\ \hline LLaVa-1.6-7B & AR & 0.41 & 0.37 \\ Qwen2.5-VL-7B & AR & 0.45 & 0.16 \\ \bottomrule \end{tabular} \label{tab:poem} \end{subtable} ## Text Infilling {#sec:text-infilling} `\ours`{=latex} offers strong controllability for text generation, particularly in text infilling. Given a draft of $L$ tokens containing $L_M$ masks, we jump to timestep $t = \frac{L_M}{L}$ and run standard inference to reach $t = 0$. This directly replaces $L_M$ masks with $L_M$ tokens. However, in practice, the intended completion may require fewer tokens---e.g., `"There is a ``[M][M][M][M]`` in the image"` might become either `"dog"` or `"traffic light"`. To allow variable-length completions, we conduct an additional stage-3 training using a 20% subset of stage-2 data and refer to this model as `\ours`{=latex}-FIM. During training, we insert random-length `[S]...[S][FIM]` sequences mid-text. At inference, we append `[FIM]` to masked segments (e.g., `[M][M][M][M][FIM]`) to signal flexible termination. The model can then generate completions like `[dog][S][S][S][FIM]` or `[traffic][light][S][S][FIM]`. While FIM objectives are often discussed in the context of language tasks (e.g., code completion) [@roziere2023code; @bavarian2022efficient], they are equally relevant to multimodal applications. Figure `\ref{fig:text-inpainting}`{=latex} shows qualitative results on constrained poem generation, where the models generate a poem describing an image, with each line starting with specific syllables. Both `\ours`{=latex} and `\ours`{=latex}-FIM complete the task successfully, unlike AR models. Notably, `\ours`{=latex}-FIM adapts token counts per line. Table `\ref{tab:poem}`{=latex} shows quantitative results over 100 samples: both `\ours`{=latex} variants achieve 100% constraint satisfaction, while AR baselines remain below 50%. Additional results on other text-infilling use cases are provided in Appendix `\ref{sec:appendix-infill}`{=latex}. ## Speed vs. Quality Trade Off {#sec:speed-quality} `\ours`{=latex} offers a convenient way to achieve speed-quality tradeoffs by controlling the number of discretization steps $K$. We compare the performance on image captioning with 500 images from the COCO 2017 val dataset [@lin2014microsoft] with varying $K$. We set the maximum generation length to 32, and experimented with $K\in\{32,24,16,8\}$, or equivalently, NFE$\in\{100\%,75\%,50\%,25\%\}$. We report the average latency per image measured on a single A5000 GPU and the CIDEr score in Figure `\ref{fig:speed-quality}`{=latex}. At NFE=$100\%$, `\ours`{=latex} achieves a higher CIDEr score than AR baselines but is slightly slower. At NFE=$75\%$ and NFE=$50\%$, `\ours`{=latex} is faster than the AR baselines and achieves better quality. At NFE=$25\%$, it is significantly faster but trails in performance. This indicates that `\ours`{=latex} can flexibly adjust its inference speed based on application needs---allowing users to trade off generation latency and output quality depending on their specific requirements.
We showcase the advantages of  over AR VLMS in terms of controllability and speed. (a) Qualitative comparison on constrained poem generation between  /-FIM and AR models.  variants successfully satisfy line-level constraints and adapt token length per line, unlike AR baselines. (b) Speed–quality tradeoff for image captioning on COCO 2017. By adjusting the number of discretization steps (K),  offers a tunable balance between latency and output quality (CIDEr score).
**Effect of KV Cache.** The speed of `\ours`{=latex} relies on our proposed Prefix-DLM setup, which allows us to cache the keys and values of visual and prompt tokens [@arriola2025block]. In Table `\ref{tab:kv_cache}`{=latex}, we compare the speed and sample quality between the proposed Prefix-DLM step and the uncached full-attention-mask step used in prior works like LLaDa. We find that Prefix-DLM significantly reduces latency and achieves a maximum speedup of $3.9\times$, with marginal performance cost. These experiments are performed using our stage-2 model, which is trained on a full-attention mask. We discuss training with customized masks and customized kernels in Appendix `\ref{sec:appendix-speed}`{=latex}. In short, we found that these alternatives lead to considerable training overhead while offering little benefit. \begin{subtable}[t]{0.47\textwidth} \caption{Effect of KV Cache} \begin{tabular}{ccHcc} \toprule Method & NFE & MME & CIDEr $\uparrow$ & Latency$\downarrow$ \\ \midrule Full-DLM & 100\% & &\textbf{121.0} & 7.65 \\ Prefix-DLM & 100\% & & 117.3 & 1.93 \\ \hline Full-DLM & 50\% & &118.6 & 4.13 \\ Prefix-DLM & 50\% & &114.8 & \textbf{1.23} \\ \hline Open-Lnxt-8B & -- & & 111.8 & 1.71 \\ \bottomrule \end{tabular} \label{tab:kv_cache} \end{subtable} \hfill \begin{subtable}[t]{0.50\textwidth} \caption{Effect of timestep shifting} \begin{tabular}{lccccHHHH} \toprule & COCO Caption (CIDEr)$\uparrow$ & \\ NFE & 25\% & 50\% & 75\% & 100\% & 25\% & 50\% & 75\% & 100\%\\ \midrule cosine & 87.7 & 102.2 & 110.8 & 117.3 \\ linear & 84.9 & 105.2 & 108.6 & 117.3 \\ $\alpha$=3 & 48.7 & 74.7 & 92.4 & 117.3 \\ $\alpha$=$3^{-1}$ & \textbf{101.1} & \textbf{114.8} &\textbf{117.3} & \textbf{117.3} \\ \bottomrule \end{tabular} \label{tab:cider_schedules} \end{subtable} `\label{tab:combined}`{=latex} **Noise Schedule.** To study the effect of the proposed time-step shifting schedule, we compare the performance of different schedules with NFE$\in\{100\%,75\%,50\%,25\%\}$. We compare the proposed time-step shifting with $\alpha=3$, and $\alpha=3^{-1}$, as well as linear and cosine schedule baselines. We report results on COCO image captioning [@lin2014microsoft] in Table `\ref{tab:cider_schedules}`{=latex}. The convex schedule with $\alpha=3^{-1}$ works the best. We also observe similar behaviors when conducting CoT inference using `\ours`{=latex}-Reason on MathVision [@wang2024measuring] dataset. At NFE=$50\%$, $\alpha=3^{-1}$ achieves an accuracy of 21.05, which is 30% higher than 16.12 achieved by a linear schedule. We provide results on MathVision in Appendix `\ref{sec:appendix-math}`{=latex}. ## Ablation Studies We conducted a thorough ablation of various design choices. In the main paper, we discuss the effect of complementary masking and input image resolution. We provide further discussion of other design choices in the Appendix `\ref{sec:appendix-ablation}`{=latex}. We conducted these experiments using a 200k subset of our (stage-2) training data. We report results in Tables `\ref{tab:comp-masking}`{=latex} and `\ref{tab:img-resolution-transposed}`{=latex} respectively. Table `\ref{tab:comp-masking}`{=latex} shows that our proposed **complementary masking** scheme leads to considerable improvements across all benchmarks, most notably, complementary masking leads to a relative improvement of 67% on ScienceQA [@lu2022learn] with affordable compute overhead during the training (8% slowdown). Table `\ref{tab:img-resolution-transposed}`{=latex} shows that **high-resolution input** improves overall performance, with the gain on OCR tasks being more pronounced than generic vision tasks (e.g., VQAv2). We did not use average pooling for the low-resolution setup. We provide additional details in the Appendix `\ref{sec:appendix-setup}`{=latex}. \begin{subtable}[t]{0.48\textwidth} % \begin{adjustbox}{max width=\textwidth} \caption{Effect of complementary masking.} \begin{tabular}{lcc} \toprule & w/o Comp.M. & w/ Comp.M. \\ \midrule MME$\uparrow$ & 260.00 & \textbf{297.00} \\ MathVista$\uparrow$ & 28.40 & \textbf{33.40} \\ ScienceQA$\uparrow$ & 48.74 & \textbf{81.49} \\ MMMU$\uparrow$ & 38.56 & \textbf{41.78} \\ \hline Runtime$\downarrow$ & \textbf{8.2 hr} & 8.9 hr \\ \bottomrule \end{tabular} % \end{adjustbox} \label{tab:comp-masking} \end{subtable} \hfill \begin{subtable}[t]{0.48\textwidth} % \begin{adjustbox}{max width=\textwidth} \caption{Effect of image resolution.} \begin{tabular}{lcc} \toprule & 384$^2$ & 768$^2$ \\ \midrule TextVQA$\uparrow$ & 48.40 & \textbf{55.65} \\ DocVQA$\uparrow$ & 43.22 & \textbf{58.72} \\ ChartQA$\uparrow$ & 42.20 & \textbf{57.70} \\ InfoVQA$\uparrow$ & 26.48 & \textbf{36.23} \\ \hline VQAv2$\uparrow$ & 65.92 & \textbf{66.78} \\ \bottomrule \end{tabular} % \end{adjustbox} \label{tab:img-resolution-transposed} \end{subtable} # Conclusion In conclusion, we propose `\ours`{=latex}, the first family of vision-language models based on DMs. To address various challenges, we introduce several novel training and inference techniques, including complementary masking, Prefix-DLM cache, and timestep shifting. Through extensive experiments, we show that these techniques significantly outperform a naive adaptation of DMs for visual tasks. Using a comprehensive evaluation suite, we demonstrate that `\ours`{=latex} achieves competitive performance compared to AR models trained under similar settings, while offering unique advantages such as speed--quality tradeoffs and controllability via text infilling. Our work proves that `\ours`{=latex} can be a powerful alternative to exitsing AR VLMs, extending the prior success of DMs in the language domain to the vision space. # Acknowledgement AG would like to acknowledge support from NSF CAREER Grant #2341040, Schmidt Sciences Early Career Fellowship, and Amazon Research Award. \bibliographystyle{plain} \clearpage \appendix \newcommand{\cat}[0]{\text{Cat}} \newcommand{\alphats}[0]{\frac{1-t}{1-s}} \newcommand{\oneminusalphats}[0]{\frac{t-s}{1-s}} # Additional Technical Details ## Formulation of Discrete Diffusion Models (DMs) In this section, we provide a detailed review of the formulation of Discrete Diffusion Models (DMs) briefly described in `\ref{sec:background-dlm}`{=latex}. Given a text sequence of $L$ tokens $X_0=[X_0^1,X_0^2,...X_0^L]$, the forward process $q(X_t|X_s)$ gradually coverts it to a sequence of mask tokens $[M]$, denoted by $X_1=[X_1^1,X_1^2,...X_1^L]$ over the continuous time-interval $[0,1]$, with $1\ge t \ge s \ge 0$. Formally, this process is defined as $$\begin{equation} q(X_t^i|X_s^i) = \begin{cases} \cat(X_t^i;\textbf{M}), & \text{if}\ X_s^i=M \\ \cat(X_t^i;\alphats \mathbf{X_s^i}+\oneminusalphats \textbf{M}), & \text{if}\ X_s^i\ne M \end{cases} \end{equation}$$ which has the marginal $$\begin{equation} q(X_t^i|X_0^i) = \cat(X_t^i;(1-t) \mathbf{X_0^i}+t \textbf{M}) \end{equation}$$ where $\cat (.)$ denotes a categorical distribution and $\textbf{M},\mathbf{X_0^i},\mathbf{X_s^i}$ are probability vectors. MDLM [@sahoo2024simple] showed that the posterior of the reversal process $p(X_s|X_t,X_0)$ can be simplified into the following $$\begin{equation} p(X_s^i|X_t^i,X_0^i) = \begin{cases} \cat(X_s^i;\mathbf{X_t^i}), & \text{if}\ X_s^i \ne M \\ \cat(X_t^i;\frac{t-s}{t} \mathbf{X_0^i}+\frac{s}{t} \textbf{M}), & \text{if}\ X_s^i = M \end{cases} \label{eq:appendix-eq-p} \end{equation}$$ In practice, we use the categorical distribution induced by the neural network's prediction $p_\theta(X_0^i|X_t)$ in place of $\mathbf{X_0^i}$ the sample from the reverse process, which gives the following parametrization $$\begin{equation} p_\theta(X_s^i|X_t) = \begin{cases} \cat(X_s^i;\mathbf{X_t^i}), & \text{if}\ X_s^i \ne M \\ \cat(X_t^i;\frac{t-s}{t} p_\theta(X_0^i|X_t)+\frac{s}{t} \textbf{M}), & \text{if}\ X_s^i = M \end{cases} \label{eq:appendix-inference} \end{equation}$$ **Inference Algorithm.** Given a target length $L$ and discretization steps $t_0,t_1...t_K$ where $t_0=0$ and $t_K=1$, we first initialize $X_{t_K}^{1:L}=X_{1}^{1:L}=M$, then use Equation `\ref{eq:appendix-inference}`{=latex} to repetitively sample $p_\theta(X_{t_{k-1}}|X_{t_k})$ until we reach $t_0=0$. In this process, we also assume $p_\theta(X_{t_{k-1}}|X_{t_k})$ factorize into $\prod _{i=1}^L p_\theta(X_{t_{k-1}}^i|X_{t_k})$ following previous works [@nie2025large; @sahoo2024simple; @lou2023discrete-sedd]. **Training Objective.** Recall that the training objective of DMs introduced in Section `\ref{sec:background-dlm}`{=latex} is formulated as $$\begin{equation} \mathcal{L_{\text{DLM}}}=-\mathbb{E}_{t,X_0,X_t}\left[\frac{1}{t}\log p_\theta(X_0|X_t)\right] \label{eq:dlm-obj-ref2} \end{equation}$$ where $p_\theta(X_0|X_t)$ factorizes into $\prod_{i=1}^L p_\theta(X_0^i|X_t)$. However, Equation `\ref{eq:appendix-eq-p}`{=latex} shows that $p_\theta(X_0^i|X_t)$ has a closed form solution that depends only on $X_t$ when $X_t^i \ne M$. Intuitively, this comes from the fact that in the reverse process, once a token $X_i$ changes from a mask token to a clean text token, it stays the same thereafter. Based on this observation, we can remove the terms that does not depend on the neural network $\theta$ from the learning objective, giving us the following loss $$\begin{equation} \mathcal{L_{\text{DLM}}}=-\mathbb{E}_{t,X_0,X_t}\left[\frac{1}{t}\sum_{X_t^i \ne M}\log p_\theta(X_0^i|X_t)\right] \label{eq:dlm-obj-ref3} \end{equation}$$ Hence, the loss is only computed over the masked indices in $X_t$. `\label{sec:appendix-dm-background}`{=latex} ## Timestep shifting Schedule {#sec:appendix-timestep}
Discretized Schedules at Different K.
Different Choice of Continuous Schedules f.
Visualization of Schedules (a) We visualize discretization results of the same continuous schedule (dashed line) under different numbers of sampling steps K ∈ {16, 24, 32} at L = 32. (b) We visualize various choices of continuous schedules: Shift(α = 3−1), Shift(α = 3), Cosine, and Linear.
Given a sequence of $L$ tokens, and a timestep $t\in[0,1]$, in expectation $X_t$ contains $t L$ masked tokens. In practice, we implement Equation `\ref{eq:appendix-inference}`{=latex} such that there are exactly $\lfloor t L \rfloor$ masked tokens and $L-\lfloor t L \rfloor$ clean text tokens at timestep $t$ during the sampling process. Given a fixed number of sampling step $K$, we define the canonical discretization as $t_i=\frac{i}{K}$ for $i=0,1..K$, with $t_0=0$ and $t_1=1$. This forms a uniformed sampling schedule, where roughly a fixed mount of $\frac{L}{K}$ tokens is unmakes at each sampling step. Any other schedule $t_i'$ can be defined as $t_i'=f(t_i)$ where $f(.)$ is a monotonic function such that $f(0)=0$ and $f(1)=1$. When $f(.)$ is convex, the slope will be steeper when $t$ get closer to $1$ , indicating that more tokens are decoded in earlier sampling steps. By contrast, when $f(.)$ is concave, the slope will be steeper when $t$ get closer to $0$, indicating that more tokens are decoded in later sampling steps. **Choice of Schedule.** We explored a wide range of choices for the continuous schedule $f(.)$. The *timestep shifting schedule* is a family of schedule defined as $$\begin{equation} f(t)=s_\alpha(t) = \frac{\alpha t}{1 + (\alpha - 1)t} \end{equation}$$ where $\alpha$ is a hyperparameter. When $\alpha<1$, the schedule is convex. When $\alpha>1$, the schedule is concave. The *cosine schedule* is defined as $$\begin{equation} f(t)= 1- \frac{1}{2} (1 + \text{cos}(\pi t)) \end{equation}$$ The *linear schedule* is just the identity function $f(t)=t$. We visualize these choices in Figure `\ref{fig:appendix-schedules}`{=latex}. **Rounding in Discretization.** In principle, we can pick any $f(.)$. However, given a particular choice of $L$ and $K$, if $\lfloor f(t_k)L \rfloor$ and $\lfloor f(t_{k-1}) L \rfloor$ yields the same integer, then no tokens are unmasked when we compute $p_\theta(X_{t_{k-1}}|X_{t_k})$. Hence, the actual number of function calls to the model $\theta$ may be less than, $K$ depending on the choice of $f(.)$. To make the sampling compute cost more predictable and allow for a fair comparison across different schedulers, we augment all choices of $f(.)$ to $f^K(.)$ such that $\lfloor f^K(t_{k-1}) L \rfloor < \lfloor f^K(t_{k}) L \rfloor$ (i.e. at least one token is decoded at each step). Note that in the sampling process, the exact real value of $f^K(t_{k})$ does not matter as long as it does not change $\lfloor f^K(t_{k}) L \rfloor$. Hence, we can parameterize the sampling process in an alternative manner using a sequence of integers $F^K_k=\lfloor f^K(t_{k}) L \rfloor$, with $F^K_0=0$ and $F^K_1=L$. Formally, we set $F^K$ by solving the following optimization objective $$\begin{align*} \min_{\{F_{0:K}^K\} \subset \{0,1..L\}} \quad & \sum_{k=0}^K \left\| F_k^K - f(t_k)L \right\|^2 \\ \text{subject to} \quad & F_{k-1} < F_k, \quad \text{for } k = 1, 2, \dots, K \\ & F_0^K= 0 \\ & F_1^K= L \\ \end{align*}$$ We visualize such examples in Figure `\ref{fig:appendix-discretization}`{=latex}. We set $L=32$ and $K\in\{16,24,32\}$. When $K=32$, $F^K$ is effectively a linear schedule, since the only schedule with 32 steps that satisfy the constraint $F_{k-1} < F_k$ is a uniform schedule where we unmask exactly one token per step. As $K$ reduces to 24 and 16, we see the discretized schedule becomes closer to the continuous scheduler (visualized in dashed line). ## Padding {#sec:appendix-padding} We follow the design of LLaDa [@nie2025large] and apply the loss function to both standard text tokens and padding tokens. AR models typically do not compute the loss over padding tokens. However, when sampling from DMs, we have a specified generation length $L$. In the generation process, we unmask $L$ mask tokens to $L$ non-mask tokens. Since the length of the desired answer may not be exactly $L$ tokens, the model will generate padding tokens. To achieve this capability, we pad the sequences during the training, and apply the loss on the padding tokens following LLaDa. # Additional Experiment Details and Results {#sec:appendix-setup} ## Setup {#setup} In this section, we document the detailed training setup, including data, hyperparameters and compute used for the main experiments. Additional details about Prefix-DLM Cache can be found in `\ref{sec:appendix-speed}`{=latex}. Additional details about stage-3 math reasoning experiments can be found in `\ref{sec:appendix-math}`{=latex}. **Training Data Composition.** For the pretraining phase, we use LCS-558K [@liu2023visual] consists of 558K image-text pairs. For the finetuning phase, we mostly use the dataset released by Open-LLaVa-Next [@chen2024open]. We made some small adjustments to the weight of each data source and increased the weight of some QA dataset. This is used to compensate the fact that our model only learns from a randomly chosen round at each training step for multi-round QA data. We document the precise dataset composition of our stage-2 training in Table `\ref{tab:stage-2-data}`{=latex}. Data Source Size Data Source Size Data Source Size ------------------------------------- --------- ------------------------------------ -------- ----------------------------- -------- COCO[@lin2014microsoft] 349,860 GQA[@hudson2019gqa] 72,140 DocVQA[@mathew2021docvqa] 10,211 ALLaVA-VFLAN[@chen2024allava] 202,966 Synthdog-En[@kim2022donut-synth] 29,765 DVQA[@kafle2018dvqa] 10,000 Visual Genome[@krishna2017visual] 86,417 TextVQA[@ocrVQA] 21,953 SA-1B[@kirillov2023segment] 8,997 OCR VQA[@ocrVQA] 80,000 ChartQA [@masry-etal-2022-chartqa] 18,317 LLaVA-150K [@liu2023visual] 2,988 GeoQA+ [@chen2021geoqa] 72,318 AI2D[@kembhavi2016diagram] 12,413 WikiArt[@artgan2018] 500 Share-TextVQA [@chen2023sharegpt4v] 500 Web-Celebrity[@chen2024open] 500 Web-Landmark[@chen2024open] 500 : **Compositio of Stage-2 Training Data**. We report the data sources and sample sizes used to compose the Stage-2 finetuning data. `\label{tab:stage-2-data}`{=latex} **Training Hyperparameters.** We use AdamW optimizer with a learning rate of 5e-3 with a cosine decay schedule for all experiments. For pretraining (Stage 1), we adopted a global batch size of 256 and trained for 1 epoch. For finetuning (Stage 2), we adopted a global batch size of 512 and trained for two epochs. **Compute Used.** We used a mixture of A100s and A6000s for training experiments and A5000 for evaluations and inference speed benchmarks. Because of the memory constraint, we set the per GPU batch size to 8 on A100s and 4 on A6000s. We adjust the gradient accumulation steps accordingly so that the global batch size is always 256 for the pretraining and 128 for the finetuning stage. **Evaluation Setup.** We implement our evlaution using `LMMS-Eval` [@zhang2024lmmsevalrealitycheckevaluation] library. We use the default prompt provided by the library for all benchmarks. We report the split used and the generation length $L$ in Table `\ref{tab:eval-split}`{=latex}. \renewcommand{\arraystretch}{1.2} Dataset Split $L$ Dataset Split $L$ --------- ------- ----- -------------- -------------------------- ----- MME-P test 100 MathVista\* testmini_format 100 VQAv2 val 16 MathVerse\* testmini_vision_dominant 100 MMBench dev 100 MathVision\* mathvision_testmini 100 MMMU val 16 ScienceQA scienceqa-full 16 MME-C test 16 AI2D test 16 TextVQA val 16 ChartQA test 16 DocVQA test 32 InfoVQA test 32 : **Evaluation Setup.** We report evaluation split and generation length $L$ used to produce results of Table `\ref{tab:main-results}`{=latex} in the main paper. \*We use a generation length of 100 for `\ours`{=latex} and a generation length of 1024 for `\ours`{=latex}-Reason. `\label{tab:eval-split}`{=latex} ## Text-Infilling {#sec:appendix-infill} ![**Additional Qualitative Results for Text Infilling.** We showcase several useful applications of text-infilling capabilities. ](images/appendix-infill.png){#fig:additional-inpainting width="90%"} In this section, we provide additional qualitative results of the text-infilling capability of `\ours`{=latex}. We visualize the results in Figure `\ref{fig:additional-inpainting}`{=latex}. In the first example, we ask the model to extract multiple attributes from the image in JSON format. In the second example, we ask the model to edit a sentence based on the image. This is achieved by deleting the original sentence and inserting mask tokens. In the final example, we ask the model to complete a movie script based on the image prompt. `\ours`{=latex} was able to successfully complete these tasks. While it may be possible to achieve similar results using an AR model, they require careful prompting. By contrast, as shown in Figure `\ref{fig:additional-inpainting}`{=latex}, using a diffusion model for text-infilling is more straightfoward. ## Math Reasoning {#sec:appendix-math} In this section, we provide additional training setup for `\ours`{=latex}-Reason and provide additional experiment results on the speed-quality tradeoff on math reasoning. **Data and Training Setup.** In §`\ref{sec:reasoning_distillation}`{=latex}, we train `\ours `{=latex}on long chain-of-thought (CoT) data to get `\ours`{=latex}-Reason. Specifically, we choose a strong 7B reasoning model, VLRethinker-7B [@wang2025vl] as a teacher model to generate the long reasoning traces. Further, we choose their own ViRL-39K data that contains (image, question, final answer). Subsequently, we generate the CoT and predicted final answer from the teacher model and filter the ones that lead to the correct final answer [@zelikman2022star; @bansal2024smaller]. This led to the creation of the final dataset of size $19.2K$.[^1] In particular, we finetune `\ours `{=latex}on this data for 5 epochs using the identical training setup as stage-2 (e.g., batch size, learning rates) and chose the checkpoint that achieves the best performance on MathVision (testmini). We observe that the same checkpoint achieved a good performance on the MathVerse and MathVista dataset too. During inference, we set the generation length to $1024$ since `\ours`{=latex}-Reason synthesizes long chain-of-thoughts for problem-solving. **Speed-Quality Tradeoff.** In the main paper, we reported the speed-quality tradeoff results on COCO image captioning and discovered that the convex schedule works the best. We conducted similar study on `\ours`{=latex}-Reason for CoT inference on MathVision dataset. We report these results in Table `\ref{tab:appendix-math-speed}`{=latex}. Overall, the conclusion on CoT math reasoning task is similar to that on image captioning task, with the convex schedule performing the best across different choices of sampling steps. To further examine the speed-quality tradeoff, we visualize the inference throughput (tokens/s) and accuracy on MathVista dataset under different inference setup in Figure `\ref{fig:frontier}`{=latex} and compare with Open-LLaVA-Next-8B baseline. `\ours`{=latex}'s frontier strictly dominates the performance of autoregressive Open-LLaVA-Next-8B both in terms of speed and quality. \begin{tabular}{lccHcHHHHH} \toprule & MathVision Acc$\uparrow$ & \\ NFE & 25\% (256 steps)& 50\% (512 steps) & 75\% & 100\% (1024 steps) & 25\% (256 steps) & 50\% (512 steps) & 75\% & 100\% (1024 steps)\\ \midrule cosine & 8.55 & 13.49& & 24.02 \\ linear & 10.86 & 16.12 & & 24.02\\ $\alpha$=3 & 5.59 & 8.88 & & 24.02 \\ $\alpha$=$3^{-1}$ & \textbf{12.5}& \textbf{21.05}&& \textbf{24.02} \\ \bottomrule \end{tabular} `\label{tab:appendix-math-speed}`{=latex} ![**Speed-Quality Tradeoff on MathVista Dataset.** We report the throughput and accuracy on MathVista dataset of `\ours`{=latex} under a variety of inference setup.](images/frontier.png){#fig:frontier width="50%"} ## Prefix-DLM {#sec:appendix-speed} In this section, we discuss several alternatives to our prefix-DLM setup that we explored and document the experiment results. **Inference Algorithm.** Autoregressive models employ a causal attention mask. Because of this, they can leverage KV cache for effective inference. By contrast, discrete diffusion models (DMs) used a full attention mask. While DMs can decode multiple tokens in parallel, it cannot leverage attention mask for fast inference. Prefix-DLM combine the best of both worlds by introducing a prefix attention mask such that the queries of image embeddings and text prompts can only interact with keys and values of image embeddings and text prompts, but not the keys and values of answer tokens. Through this mechanism, we can leverage the KV cache for the image embeddings and text prompts. In vision-language applications with a long context (900+ vision tokens per image), this saves a lot of compute at inference time, while preserving the full bidirectional context. A alternative to our prefix-DLM was the recently proposed semi-autoregressive Block-Diffusion [@arriola2025block], which uses a block-wise causal attention mask. In this setup, the input sequece are chunked into a sequence of fixed length blocks, each containing $L_B$ tokens. A token in Block $i$ can see all tokens in the past and current Block $j$ with $j\le i$, but cannot see all future blocks Block $j$ with $j > i$. While this design allows it to leverage block-wise KV cache, it limits the bi-directional context to at most $L_B$ tokens in the future, which is undesirable for tasks like text-infilling. Additionally, because of its semi-autoregressive nature, when we are generating Block $i$, we must see all mask-free past Blocks $j$ with $j
Visualization of Different Choices of Attention Mask at Inference Time. I represents the image embeddings, P represents the prompt tokens, and Xt represents the partially masked answer tokens. Each row represents a query and each column represents a key. Colored region indicts tokens queries and keys can interact with each other.
\begin{tabular}{c|c|cccH} Method & Attn. Mask & Bi-Direction Context & KV Cache & Training Overhead \\ \hline AR & Causal & None & \textgreen{Yes} & \textgreen{No}\\ DM & Full & \textgreen{Full Seq.}& No & \textgreen{No}\\ Block-Diffusion & Block-Causal & Within Block & \textgreen{Yes} & Yes\\ \ours & Prefix-DLM & \textgreen{Full Seq.} & \textgreen{Yes} & \textgreen{No}\\ \hline \end{tabular} `\label{tab:appendix-attn-mask}`{=latex} **Training Algorithm.** We adopt Prefix-DLM as a pure inference algorithm. Our training process is identical to that of a standard DM with full attention mask. We made this choice mostly because of efficiency reasons. We also explored adopting the prefix-DLM attention mask during the training (Prefix-DLM-FT) with four different but equivalent implementations. Generally, these four implementations are categorized into two classes: Prefix-DLM-FT1 and Prefix-DLM-FT2. We visualize these setups in Figure `\ref{fig:appendix-attn-mask-train}`{=latex}. Recall from Section `\ref{sec:training-algo}`{=latex} that we adopted a complementary masking scheme, wherein given each triplet of image $I$, prompt $P$ and answer $X$, we create two versions of partially masked answer $X_t$ and $X_t^C$ with complementary masking in order to utilize all tokens in the clean answer $X$. Empirically, this is achieved by copying the prompt embedding and concatenate $X_t$ and $X_t^C$ to different copies respectively (Left Column of Figure `\ref{fig:appendix-attn-mask-train}`{=latex}). By default, we use the full attention for both copies. We can adopt the Prefix-DLM attention mask during the training for each copy individually. We call this setup Prefix-DLM-FT1 (Middle Column of Figure `\ref{fig:appendix-attn-mask-train}`{=latex}). Alternatively, we can concatenate $I,P,X_t,X_t^C$ into a single long sequence and manipulate the attention mask such that queries of $I,P$ can see keys of $I,P$, queries of $X_t$ can see keys of $I,P,X_t$ and queries of $X_t^C$ can see keys of $I,P,X_t^C$. We call this setup Prefix-DLM-FT2 (Right Column of Figure `\ref{fig:appendix-attn-mask-train}`{=latex}). For each of the two variant Prefix-DLM-FT1 and Prefix-DLM-FT2, we designed two concrete implementations. The first set of implementations (Prefix-DLM-FT1-Mask,Prefix-DLM-FT2-Mask) simply construct a 2D attention mask of size $L \times L$ for each sample, where $L$ is the sequence length, and pass it to the torch SDPA kernel. The second variant (Prefix-DLM-FT1-Flex,Prefix-DLM-FT2-Flex) merely constructs an integer tensor of size 3 per sample, containing the length of $I,P,X$. We then use a customized `flex_attention` module [@dong2024flex] to implement the attention mask implicitly based on the length of $I,P,X$. Notably, these four variants of Prefix-DLM-FT generates identical loss value and will produce exactly the same training dynamics (disregarding small numerical differences between implementations). The only difference is the efficiency. Hence, we only benchmarked the performance and performed the full training using the most efficient version. We report the speed benchmark in Table `\ref{tab:appendix-math-speed}`{=latex}. Overall, even the fastest finetuning version is 62% slower than the full attention mask baseline, suggesting a high overhead caused by batch-dependent masking strategies during training. We also report the model performance after 1,500 training steps (roughly 200k samples from the training data) in Table `\ref{tab:appendix--prefix-dlm-ft}`{=latex}. Overall, Prefix-DLM and Prefix-DLM-FT has mostly identical performance, with Prefix-DLM having a small lead over many tasks. Because of this, we consider the 62% overhead as unacceptable and opt for a training procedure using the full attention mask. This also gives user an additional dimension for speed-quality tradeoff: they can disable Prefix-DLM cache and use the full attention mask at inference to achieve a slightly better performance (Results shown in Table `\ref{tab:kv_cache}`{=latex} in main paper).
Visualization of Training Strategies for Prefix-DLM. Given each triplet of image I, prompt P and answer X, we create two versions of partially masked answer Xt and XtC with complementary masking.(Left) By default, we construct two sequence and apply full attention mask. (Middle) Prefix-DLM-FT1 applies prefix attention mask to each copy independently. (Right) Prefix-DLM-FT2 combines I, P, Xt, XtC into a single sequence and manipulate the attention mask to achieve an equivalent effect to Prefix-DLM-FT1.
Method Speed(s/training step) ----------------------- ------------------------ Prefix-DLM **37.2** Prefix-DLM-FT1-Mask 67.3 Prefix-DLM-FT2-Mask 82.6 Prefix-DLM-FT1-Kernel 60.2 Prefix-DLM-FT2-Kernel 74.4 : **Training Speed of Different Variants of Prefix-DLM.** We report the average training speed of different setup with a batch size of 128 on 8 A6000 GPUs. `\label{tab:appendix-training-speed-mask}`{=latex} MMMU VQAv2 MME M.Vista M.Verse ScienceQA AI2D --------------- ----------- ----------- ------------ ----------- ----------- ----------- ----------- Prefix-DLM **40.56** **63.26** 286.79 33.60 **19.67** **80.36** **64.31** Prefix-DLM-FT 40.10 61.02 **290.71** **35.00** 18.15 80.15 64.15 : **Performance of Prefix-DLM and Prefix-DLM-FT across benchmarks.** We compare the performance of two variants after 1,500 steps of training. `\label{tab:appendix--prefix-dlm-ft}`{=latex} MMMU VQAv2 MME M.Vista M.Verse ScienceQA AI2D ---------- ----------- ----------- ------------ ----------- ----------- ----------- ----------- SigLip **40.56** **63.26** 286.79 33.60 19.67 **80.36** **64.31** CLIP 40.44 59.58 **288.21** 30.90 **20.05** 76.56 59.78 MetaCLIP 40.33 55.96 280.36 **33.80** 17.26 78.66 62.79 : **Ablation Studies on the Choice of Vision Encoders.** We compare the performance of models with different vision encoders after 1,500 steps of training. On Average, SigLip improves over CLIP by **`\textgreen{+1.86}`{=latex}** and improves over MetaCLIP by **`\textgreen{+2.77}`{=latex}**. `\label{tab:ablation-vision}`{=latex} ## Ablation Studies {#sec:appendix-ablation} We conduct additional ablation studies over the choice of vision encoders. We experimented with SigLip[@zhai2023sigmoid], CLIP [@radford2021learning], and MetaCLIP[@xu2023demystifying]. We report the model performance after 1,500 training steps (roughly 200k samples from the training data) in Table `\ref{tab:ablation-vision}`{=latex}. Overall, SigLip achieves the strongest overall performance, with notably gains in VQAv2, ScienceQA, and AI2D. ## Additional Scaling While `\ours`{=latex} outperforms AR VLMs under comparable data scale, there is still a considerable gap between `\ours`{=latex} and state-of-the-art AR VLMs. To validate the scalability of `\ours`{=latex}, we scale the training schedule by 2x, resulting in considerable performance on general understanding and OCR tasks. We report these results in Table `\ref{tab:scaling-training-data}`{=latex}. Due to compute constraints, we left further scaling to future works. \small \setlength{\tabcolsep}{6pt} **Model** **MME** **MMBench** **ChartQA** **DocVQA** **InfoVQA** ---------------------- ----------- ------------- ------------- ------------ ------------- `\ours `{=latex} 341.1 70.5 64.6 59.0 34.2 +Additional Training **444.3** **75.6** **74.9** **68.6** **43.4** Open-LLaVA-Next-8B 336.8 74.4 69.7 69.9 36.7 : **Additional Scaling Experiments on OCR and General Understanding Tasks.** `\label{tab:scaling-training-data}`{=latex} # Additional Backgrounds {#sec:appendix-background-extra} In this section, we provide additional discussions with relevant works not covered in the main paper. **Masked Generative Models.** Masked generative modeling has a long history before the recent advancements of DMs. Earliest works such as BERT[@devlin2019bert] and MAE[@he2022masked] adopt the masked generative modeling objective as a pretraining objective to learn rich text and vision features. They mainly concern the utility of learnt features to downstream perception and understanding tasks, instead of the generation capability of the model. A series of subsequent works use mask modeling to build generative models for images [@chang2022maskgit; @chang2023muse], texts [@liang2023open] and audio [@ziv2024masked]. Compared with these early works relying on ad-hoc sampler designs, recent works on DMs [@lou2023discrete-sedd; @sahoo2024simple] provided a principled way for training and sampling from masked generative models. **Multi-Modal Diffusion Models.** Several works have explored to build a multi-modal diffsion models with vision-language capabilities. CoDi [@tang2023any-codi] and OmniFlow [@li2024omniflow] build continuous diffusion model over a latent text embedding space and use autoregressive decoder to decode the generated latent mebeddings to actual texts. UniD3 [@hu2022unified] and UniDisc [@swerdlow2025unidisc] build discrete diffusion models for simultaneous text and image generation. Overall, these models have limited language generation capabilities. Their experiments on text generations are limited in scale and mostly focusing on captioning. They cannot perform more complex instruction-following and understanding tasks (e.g. reasoning) like many modern autoregressive VLMs. # Limitations {#sec:appendix-limitation} In this section, we discuss the limitations of `\ours`{=latex}. There are mainly two limitations. **Scale.** While `\ours `{=latex}achieves competitive results when compared against similar-sized AR VLMs trained on a similar scale of data, there remain a considerable gap between `\ours`{=latex}'s performance and that of state-of-the-art open sourced VLMs such as LLaVa-OneVision and Qwen2.5-VL. These models either comes with more training data or larger model sizes. Future work should study if DMs scale well with a larger model and more training data. **OCR.** `\ours`{=latex}'s performance on OCR tasks are slightly worse than the baselines, this can be mainly attributed to the average pooling operation which we introduced to reduce the sequence length by compressing the visual information. Concretely, `\lnxt`{=latex} and `\openlnxt`{=latex} baseline do not adopt average pooling, and uses 2880 tokens ($24\times24\times5 \text{~Views}$) to represent each image. In our setup, removing the average pooling would lead to a total of 3645 tokens ($24\times24\times5 \text{~Views}$) per image. Average pooling is necessary because the base model LLaDa and Dream has a context length of 4096 and 2048 respectively. Without average pooling, LLaDa will not have enough context length to fit longer training samples, while Dream will not have enough context length to even fit one image. We also tried to extend the context length of these models via techniques such as rope rescaling, but achieved limited success. We experimented with extending Dream to 4096 context length and evaluate the model using the needle-in-a-haystack task [@hsieh2024ruler] at 4096 context length. We found that Dream-7B (72.4 Acc) performs worse than LLaDa-8B (91.6 Acc) and Llama-3-8B (95.4 Acc). We hope future works on DMs with longer context will provide stronger base models to finetune on vision-language applications. # Boarder Impacts and Safeguards {#sec:appendix-boarder-impacts} Our model may inherit the biases embedded in the base model LLaDa-8B and Dream-7B, as well as biases incorporated in the training data. Our model may also suffer from the same hallucination problem as the base model. Just as any LLMs and VLMs, there is also the risk that our model is used to generate harmful or offensive content. Overall, our model is intended to be used by researchers to build a strong diffusion model for vision-language applications. We do not recommend it be used for any other purposes. # License Information for Data and Models {#sec:appendix-licenses} We report the licenses of the used artifacts in Table `\ref{tab:license-info}`{=latex}. We followed the intended use of all respective artifacts. \renewcommand{\arraystretch}{1.3} \setlength{\tabcolsep}{6pt} **Category** **Name** **License** **Platform** -------------------- ---------------------------------------- ------------------------- ------------------ Vision Encoder SigLIP Apache 2.0 Hugging Face Language Model LLaDA-8B MIT Hugging Face Language Model Dream-7B Apache 2.0 Hugging Face VLM LLaVA-NeXT (1.6) Apache 2.0 GitHub VLM LLaVA-NeXT (1.6) LLaMA 2 License Hugging Face VLM Open-LLaVA-NeXT Apache 2.0 Hugging Face Train Dataset LLaVA-Pretrain CC3M^†^ + BSD-3-Clause Hugging Face Train Dataset Open-LLaVA-NeXT-mix1M^`\ddag`{=latex}^ CC BY-NC 4.0 Hugging Face Evaluation Dataset MME (MME-P) arXiv Non-exclusive^\*^ arXiv Evaluation Dataset VQAv2 CC BY 4.0 Official Website Evaluation Dataset MMBench Apache 2.0 GitHub Evaluation Dataset MMMU Apache 2.0 Hugging Face Evaluation Dataset MME (MME-C) arXiv Non-exclusive^\*^ arXiv Evaluation Dataset MathVista CC BY-SA 4.0 Hugging Face Evaluation Dataset MathVerse MIT Hugging Face Evaluation Dataset MathVision MIT Hugging Face Evaluation Dataset ScienceQA CC BY-NC-SA Official Website Evaluation Dataset AI2D arXiv Non-exclusive^\*^ arXiv Evaluation Dataset TextVQA CC BY 4.0 Official Website Evaluation Dataset DocVQA arXiv Non-exclusive^\*^ arXiv Evaluation Dataset ChartVQA GPL-3.0 GitHub Evaluation Dataset InfoVQA arXiv Non-exclusive^\*^ arXiv : **Licenses and sources for datasets and models used.** Datasets marked with ^\*^ have no published license; the accompanying paper's arXiv license is used as fallback. Training entries marked with ^†^ or ^`\ddag`{=latex}^ reflect compound or inferred licensing conditions. `\label{tab:license-info}`{=latex} `\noindent`{=latex}**Note ^†^:** Subject to the CC-3M dataset may be freely used for any purpose, although acknowledgement of Google LLC (\"Google\") as the data source would be appreciated. The dataset is provided \"AS IS\" without any warranty, express or implied. Google disclaims all liability for any damages, direct or indirect, resulting from the use of the dataset `\noindent`{=latex}**Note ^`\ddag`{=latex}^:** Our training data is based on Open-LLaVA-NeXT-mix1M, which combines ShareGPT4V data (CC BY-NC 4.0, research-only, with restrictions from LLaMA, Vicuna, and GPT-4 licenses) and AVG-LLaVA (Apache 2.0). All data was used strictly for academic research. # LLM Usage {#sec:appendix-llm} LLM is used to generate the math reasoning date for stage-3 training. See details in `\ref{sec:appendix-math}`{=latex}. # Concurrent Work Concurrent to this work, MMaDa[@yang2025mmada] (released after submission deadline) proposed a unified understanding and generation modeling using DM formulation. However, they do not leverage many of the novel techniques that we propose, leading to inferior performance and slow inference speed. We list a brief comparison in Table `\ref{tab:mmada-comp-latency}`{=latex}. \renewcommand{\arraystretch}{1.2} \setlength{\tabcolsep}{6pt} **Model** **MME** **MMMU** **MMB** **Latency (s/image)** -------------- ------------ ---------- ---------- ----------------------- LaViDa-Dream **1463.5** 42.6 **73.8** **1.13** LaViDa-LLaDa 1365.6 **43.3** 70.5 1.32 MMaDa 1410.7 30.2 68.5 7.68 : **Comparison with MMaDa.** We report scores on MME, MMMU, and MMB benchmarks, along with average latency for image captioning. `\label{tab:mmada-comp-latency}`{=latex} [^1]: We will release this data in the camera-ready version.