Diffusion Language Models

Summary

Diffusion language models are the wiki’s tracker for language-generation systems that do not commit to ordinary left-to-right autoregressive next-token decoding as the only generative path. The current local cluster separates three evidence types:

1. Training objective evidence: masked diffusion language models such as iLLaDA keep a denoising objective through pre-training and SFT.
2. Continuous-trajectory evidence: ELF moves text generation through continuous contextual embedding flows and discretizes only at final decoding.
3. Inference and self-correction evidence: DMax improves aggressive parallel decoding by training on model-sampled noisy states and keeping intermediate predictions soft/revisable.

For this knowledge base, diffusion language models matter less as a replacement slogan for autoregression and more as a progress signal for future multimodal and time-series generators: can a model generate, edit, or roll out long horizons without strict serial commitment, while still preserving dense detail and calibrated uncertainty?

Current Tracker

Local source	What changed	Why it matters	Boundary
ELF	Continuous embedding-space flow matching for language, final token decoding only at the end.	Makes text more compatible with continuous diffusion/flow substrates used by images and time-series generation.	Language-only, depends on contextual embeddings, no numeric time-series or action rollouts.
DMax	On-policy self-correction plus soft parallel decoding for masked diffusion LMs.	Shows tentative decoded positions should remain revisable under aggressive parallel generation.	Fine-tunes a pretrained dLLM for math/code decoding; not a foundation pre-training result.
iLLaDA	8B masked diffusion language model trained from scratch on 12T tokens, with 25B-token SFT, GQA, 8192 context, variable-length generation, and confidence scoring.	Shows that the full masked-diffusion recipe can close much of the base-model gap to Qwen2.5 7B on reported benchmarks.	Current preprint; instruct model still lags Qwen2.5 Instruct; comparisons are not matched-compute scaling laws.

What The Wiki Currently Believes

• Diffusion language modeling is now credible enough to track as a live branch. iLLaDA is the current local milestone because it improves the full training and inference stack rather than only a sampler.
• Autoregressive dominance is not yet overturned. iLLaDA-Base is reported slightly stronger than Qwen2.5 7B on the paper’s base-model average, but iLLaDA-Instruct remains far behind Qwen2.5 7B Instruct on the reported average.
• Evaluation protocol is part of the model claim. iLLaDA’s confidence-based multiple-choice scoring and DMax’s tokens-per-forward tradeoff show that diffusion-LM progress cannot be reduced to one perplexity-like number.
• The continuous/discrete boundary is movable. ELF keeps the generation trajectory continuous until final decoding, DMax keeps intermediate decoded positions as soft token-mask embeddings, and iLLaDA keeps discrete masked tokens but changes reveal/commitment order.
• For time-series and world-model work, the immediate transfer is still an analogy: parallel denoising, confidence-based commitment, continuous latent trajectories, and variable-length block generation need direct tests on numeric histories, event streams, exogenous variables, and action-conditioned rollouts before they count as TSFM progress.

flowchart LR
  Objective[Masked / flow training objective]
  Intermediate[Revisable intermediate state]
  Commit[Final token or output commitment]
  Score[Scoring and convergence protocol]
  Transfer[Time-series horizon / scenario rollout hypothesis]

  Objective --> Intermediate
  Intermediate --> Commit
  Intermediate --> Score
  Score --> Commit
  Commit -. language evidence only .-> Transfer

Progress Signals To Track

• Scale: Does masked diffusion pre-training scale beyond 8B and 12T tokens under matched compute and data quality?
• Alignment: Can RL, preference optimization, or diffusion-specific post-training close the instruct gap without destroying self-revision advantages?
• Likelihood and calibration: Which metrics compare diffusion and autoregressive LMs fairly when diffusion scoring uses confidence-based reveal order or likelihood bounds?
• Serving cost: Does parallel denoising improve wall-clock latency and throughput after batching, KV/cache-style memory, scheduler overhead, and custom kernels are counted?
• Long context: Do diffusion LMs exploit bidirectional context better for editing, fill-in, or constrained generation than autoregressive baselines?
• Multimodal transfer: Can the same denoising/flow substrate handle text, images, time-series latents, and event streams without erasing modality-specific detail?

Relation To Foundation TSFM Agenda

Diffusion language models are adjacent to the Foundation Time-Series Model Research Agenda through generation, editing, dynamic compute, and uncertainty-aware commitment. They do not close a TSFM slot by themselves. A foundation TSFM needs direct evidence that diffusion or flow generation preserves calibrated numeric values, rare regimes, channel identity, event timing, exogenous variables, action histories, and intervention effects.

The strongest current TSFM hypothesis is:

history + context + candidate action/control inputs
  -> noisy or partially masked future state blocks
  -> iterative denoising with confidence/calibration checks
  -> multiple plausible future trajectories or edits

That hypothesis remains unvalidated until tested on multivariate time series, event streams, and action-conditioned world-model benchmarks.

Open Questions

• Should future time-series diffusion models use discrete tokens, continuous latents, raw numeric diffusion, or hybrid final-readout schemes?
• What is the time-series equivalent of iLLaDA’s confidence-based scoring: calibrated event confidence, horizon-cell confidence, trajectory likelihood surrogate, or downstream utility score?
• Can DMax-style soft intermediate states prevent early parallel-rollout errors from becoming irreversible in numeric horizons?
• Does ELF-style continuous denoising help preserve numeric scale and units, or does it make dense-value calibration harder?
• Which diffusion-LM progress should trigger local source ingestion: original LLaDA lineage papers, Dream-style diffusion fine-tuning, long-context diffusion LMs, RL-aligned diffusion LMs, or production serving releases?

Related Pages

• Time-Series Generation
• Time-Series Scaling And Efficiency
• Time-Series Benchmark Hygiene
• LLM Post-Training
• Latent Tokenization
• Unified Multimodal Models
• Foundation Time-Series Model Research Agenda