Improved Large Language Diffusion Models

Source

• Raw Markdown: paper_illada-2026.md
• PDF: paper_illada-2026.pdf
• Preprint: arXiv 2606.25331v1
• DOI: 10.48550/arXiv.2606.25331
• Official code and inference/evaluation repository: ML-GSAI/LLaDA
• Official weights: GSAI-ML/iLLaDA-8B-Base, GSAI-ML/iLLaDA-8B-Instruct
• Alex-provided X pointer: Tanishq Mathew Abraham post. Local API snapshot and text extract are stored as papers/illada-2026/x_post_iscienceluvr_2070059785832317212.json and papers/illada-2026/x_post_iscienceluvr_2070059785832317212.md.
• Local official-artifact metadata: papers/illada-2026/official_artifacts_metadata.json
• Local research context: papers/illada-2026/research-context-links.md

Status And Credibility

arXiv lists Improved Large Language Diffusion Models as cs.CL, with cs.AI and cs.LG cross-listing, submitted on 2026-06-24 as v1. The paper is a current preprint, not a peer-reviewed venue result at ingest time.

Credibility is strong enough for an important ingest because the author list combines Renmin University of China and ByteDance Seed affiliations, the paper reports large-scale 8B training rather than only small ablations, the official ML-GSAI/LLaDA repository is public, and separate Apache-2.0 Hugging Face checkpoints are available for iLLaDA-8B-Base and iLLaDA-8B-Instruct. The caveat is that the paper is still author-reported preprint evidence: there is no matched-compute autoregressive replication, no independent evaluation, no released training corpus, and no reinforcement-learning alignment run for iLLaDA at ingest time.

The user-provided X post is a useful discovery and curation signal, not the source of truth. The source of truth for paper claims is the arXiv paper plus official project artifacts.

Core Claim

The paper introduces iLLaDA, an 8B fully bidirectional masked diffusion language model trained from scratch. The claim is not only that diffusion can generate text, but that a masked diffusion objective can remain the training objective through pre-training and supervised fine-tuning while reaching competitive base-model benchmark performance against a strong 7B autoregressive baseline.

Relative to the earlier LLaDA recipe, iLLaDA scales the corpus to 12T pre-training tokens, increases context length to 8192, adds grouped-query attention, ties input/output embeddings, changes the learning-rate schedule after loss plateau, uses a variable-length SFT format over a 25B-token instruction corpus for 12 epochs, and evaluates with variable-length generation plus confidence-based multiple-choice scoring.

Mechanism

The pre-training objective samples a masking ratio $t\sim U[0,1]$, corrupts a clean sequence $x_0$ into a masked sequence $x_t$, and trains the model to predict masked tokens under full bidirectional attention:

$$\mathcal{L}(\theta )=-{\mathbb{E}}_{t,x_0,x_t}\left[\frac{1}{t}\sum _{i=1}^{L}1[x_t^i=\mathrm{M}]\log p_{\theta }(x_0^i\mid x_t)\right].$$

The notable system-level changes are practical rather than a new diffusion objective:

• Architecture: dense Transformer with RMSNorm, SwiGLU, RoPE, no attention/MLP bias, GQA, tied input/output embeddings, 32 layers, 4096 model dimension, 8 KV heads, 155,136 vocabulary size, and 8192 maximum sequence length.
• Pre-training: 12T tokens, variable-length packed batches, random splitting of 8192-token sequences with 30% probability, FlashAttention-style variable-length attention, linear warmup to $2\times 10^{-4}$, then constant learning rate, then cosine decay after loss stopped decreasing.
• SFT: a 25B-token instruction corpus is concatenated and masked with the same pre-training-style random masking over prompt, response, and EOS tokens; the paper reports 12 SFT epochs.
• Inference: open-ended generation appends blocks of mask tokens and iteratively reveals high-confidence positions; multiple-choice evaluation uses a confidence-scoring surrogate rather than direct likelihood.

flowchart LR
  Clean[Clean text sequence] --> Mask[Random mask ratio t]
  Mask --> Bidirectional[Bidirectional masked diffusion Transformer]
  Bidirectional --> Predict[Predict all masked tokens]
  Predict --> SFT[Same objective through SFT]
  SFT --> BlockGen[Variable-length block generation]
  BlockGen --> Scores[Confidence-based evaluation]

Evidence

The paper evaluates iLLaDA against LLaDA 8B, Dream 7B, and Qwen2.5 7B on general, math, and code benchmarks.

Setting	Key reported result	Local interpretation
Base average	iLLaDA 63.9 vs LLaDA 51.1, Dream 61.4, Qwen2.5 63.3	Strong evidence that the improved diffusion recipe closes much of the base-model gap at 8B scale.
Base reasoning	BBH 71.3 vs LLaDA 49.7 and Qwen2.5 63.9	The largest gains support tracking diffusion-LM reasoning progress, but the comparison is not a matched-compute scaling law.
Base math/code	GSM8K 81.9; HumanEval 50.0; MBPP 57.8	Competitive but not uniformly best; Dream and Qwen2.5 still win some code/math cells.
Instruct average	iLLaDA 67.1 vs Qwen2.5 Instruct 77.1	The diffusion recipe improves SFT strongly but still lags a strong autoregressive instruct model.
Instruct gap diagnosis	Authors leave reinforcement-learning alignment to future work	The remaining gap may be post-training/alignment rather than only pre-training objective, but the paper does not prove that.
Ablations	Confidence scoring improves multiple-choice results; more SFT epochs continue helping up to 12	Useful recipe evidence; also warns that scoring protocol and data reuse materially affect reported progress.

Diffusion Progress Interpretation

iLLaDA is a meaningful milestone for Diffusion Language Models because it improves the full-stack recipe, not just the sampler. Earlier local sources in this cluster mostly covered continuous language flows or decoding/self-correction after a pretrained diffusion language model. iLLaDA shows that scale, architecture, SFT format, and evaluation protocol still have substantial headroom inside the masked-diffusion language-model family.

The durable interpretation should be narrow:

• Fully bidirectional masked diffusion is now credible enough at 8B scale to track as a live language-model branch.
• Base-model results are close to a Qwen2.5 7B baseline on the reported suite, but this does not establish compute-optimal superiority over autoregressive training.
• Instruct-model results still lag Qwen2.5 Instruct, so diffusion-language progress needs alignment/post-training evidence, not only pre-training and sampler evidence.
• The confidence-based multiple-choice scorer is a benchmark-protocol variable; it should be tracked separately from model-likelihood quality.

Relevance To This Wiki

This is upstream language-model evidence, not a time-series model. Its value for the public wiki is that it strengthens the case for diffusion or flow-style sequence generation as a serious substrate, while preserving the caveat that text success does not prove numeric time-series fidelity, event-stream modeling, or action-conditioned world-model rollouts.

For time-series and world-model work, the useful transfer hypotheses are:

1. Bidirectional denoising for future generation: masked blocks could generate forecast horizons or scenario windows without strict left-to-right dependence.
2. Variable-length generation: block termination and new-block appending are relevant to adaptive horizon generation.
3. Confidence-based commitment: confidence can decide which positions or spans are stable enough to reveal, but it must become calibrated uncertainty before operational use.
4. Data reuse under SFT: repeated instruction-corpus training appears helpful for iLLaDA; a TSFM analogue must check whether repeated domain data improves useful state or overfits frequent regimes.

Foundation TSFM Relevance

Agenda slot	Verdict	Evidence	Missing pieces
Multi-modal future distributions / generation	adjacent	iLLaDA strengthens diffusion-style sequence generation as a credible language-model branch and uses variable-length block generation.	Needs numeric time-series sample paths, calibrated multi-modal futures, dense-value fidelity, and event-stream generation.
Dynamic compute under serving budget	adjacent	Block generation, confidence-based reveal, and GQA make decoding efficiency part of the recipe.	Needs matched wall-clock, memory, and latency measurements for time-series horizons and action-conditioned rollouts.
Context interface	adjacent	Fully bidirectional attention can condition on both sides of a masked span and may be useful for fill-in or editing-style tasks.	Needs explicit natural-language context plus numeric history interfaces, not only text benchmarks.
Benchmark hygiene	warning	Multiple-choice results depend on confidence-based scoring rather than direct likelihood, and Qwen2.5 comparisons are not matched-compute scaling experiments.	Need standardized likelihood, generation, wall-clock, and downstream utility protocols before calling diffusion superior or compute-optimal.
Native multivariate encoding and control	insufficient evidence	The model operates on text tokens and has no multivariate numeric channels, action history, control inputs, interventions, or counterfactual rollouts.	Needs action-conditioned multivariate time-series experiments.

Limitations

• The paper is a current arXiv preprint, not an accepted venue paper at ingest time.
• The strongest comparison is author-reported and not matched for training compute, data mixture, alignment budget, or implementation maturity.
• iLLaDA-Instruct still lags Qwen2.5 7B Instruct on the reported average, especially on math and code, and the paper leaves RL alignment to future work.
• Confidence-based multiple-choice scoring is a task-specific surrogate, not a likelihood estimate. Benchmark gains may depend on protocol choices.
• The model and paper are text-only; they do not evaluate numeric time series, graph time series, telemetry, event streams, actions, control inputs, or interventions.
• Public artifacts include inference/evaluation code and weights, but not the full 12T-token pre-training corpus or 25B-token instruction corpus.
• Hugging Face model cards use custom_code; downstream users need the usual remote-code security review before deployment.

Links Into The Wiki

• iLLaDA
• Diffusion Language Models
• DMax
• Embedded Language Flows
• Time-Series Generation
• Time-Series Scaling And Efficiency
• Time-Series Benchmark Hygiene
• LLM Post-Training
• Foundation Time-Series Model Research Agenda

Open Questions

• Does diffusion-language pre-training continue to scale beyond 8B and 12T tokens under matched compute against autoregressive baselines?
• How much of iLLaDA’s improvement comes from more tokens, longer context, GQA/tied embeddings, learning-rate schedule, SFT format, confidence scoring, or variable-length generation?
• Can RL alignment for masked diffusion LMs close the instruct gap without breaking bidirectional or self-revision advantages?
• What is the right likelihood or calibration metric for diffusion language models when benchmark scoring uses confidence-based reveal order?
• Can diffusion-style block generation transfer to numeric time-series horizons while preserving dense numeric values, rare regimes, exogenous variables, and action histories?
• Should future TSFM diffusion experiments use discrete tokens, continuous latents, or a hybrid where only final readout is discrete?