A Random Matrix Theory Perspective on the Consistency of Diffusion Models

Source

• Raw Markdown: paper_diffusion-consistency-rmt-2026.md
• PDF: paper_diffusion-consistency-rmt-2026.pdf
• Preprint: arXiv 2602.02908v2
• ICML: ICML 2026 oral
• OpenReview: ICML 2026 forum
• Award: ICML 2026 Outstanding Paper Honorable Mention
• Official project page: Diffusion Consistency RMT
• Official code: Animadversio/diffusion-consistency-rmt
• Gonzo ML discussion: Telegram post 5727 (local extract at papers/diffusion-consistency-rmt-2026/telegram-post-gonzo-ml-5727.md)
• Gonzo-linked review: ArXivIQ ICML 2026 review
• Local artifact audit: papers/diffusion-consistency-rmt-2026/official_artifacts_metadata.json

The arXiv title includes Theory; the official ICML event and award pages use the shorter title A Random Matrix Perspective on the Consistency of Diffusion Models. This page uses the arXiv title. No official author/lab X thread was found through authenticated exact-title, arXiv-ID, author-name, and mechanism searches at ingest time.

Status And Credibility

The paper was first submitted to arXiv on 2026-02-02 and revised as v2 on 2026-07-04. The official ICML site lists it as an ICML 2026 oral, and the ICML program chairs selected it for an Outstanding Paper Honorable Mention. The authors are affiliated with the Kempner Institute, Harvard Society of Fellows, Harvard SEAS, and Harvard Center for Brain Science.

This is current, tier-1, award-recognized theory and controlled experimental evidence. The strongest formal results are asymptotic deterministic equivalents for linear Gaussian denoisers and sampling maps. The UNet/DiT results validate qualitative structure and partial predictive correlations; they do not turn the linear analysis into an exact theory of every deep diffusion model.

Core Claim

Under deterministic diffusion sampling, models trained on disjoint samples from the same distribution can map the same initial noise to similar outputs because their learned score fields share a stable low-order component determined by the data mean and covariance. A linear Gaussian denoiser already reproduces much of the observed cross-split agreement.

For population covariance $\mathbf{\Sigma }$, empirical covariance $\hat{\mathbf{\Sigma }}$, raw ridge/noise variance $\lambda$, and aspect ratio $\gamma =d/n$, the central deterministic equivalence is

$$\hat{\mathbf{\Sigma }}(\hat{\mathbf{\Sigma }}+\lambda I{)}^{-1}\asymp \mathbf{\Sigma }(\mathbf{\Sigma }+\kappa (\lambda )I{)}^{-1},$$

where the renormalized noise scale $\kappa (\lambda )$ satisfies

$$\kappa (\lambda )-\lambda =\gamma \kappa (\lambda )\mathrm{tr}\left[\mathbf{\Sigma }(\mathbf{\Sigma }+\kappa (\lambda )I{)}^{-1}\right].$$

Here $\mathrm{tr}$ follows the paper’s normalized-trace convention, $\mathrm{tr}(A)=\mathrm{Tr}(A)/d$.

Because $\kappa (\lambda )\ge \lambda$, finite data make the expected linear denoiser behave as though the noise were larger. Lower-variance spectral directions are overshrunk toward the mean and require more samples before they are reproduced consistently.

The paper further factorizes denoiser variance across independently sampled datasets. With

$$A(\mathbf{a};\kappa ,\mathbf{\Sigma })={\mathbf{a}}^{\top }(\mathbf{\Sigma }+\kappa I{)}^{-2}\mathbf{\Sigma }\mathbf{a},$$

its generic contribution has the form

$$\mathrm{Var}\left[{\mathbf{v}}^{\top }{D}_{\hat{\mathbf{\Sigma }}}^{\ast }(\mathbf{x};\sigma )\right]\asymp \frac{\kappa ({\sigma }^2{)}^2}{n-{\mathrm{df}}_2(\kappa ({\sigma }^2))}A(\mathbf{v};\kappa ,\mathbf{\Sigma })A(\mathbf{x}-\mathbf{\mu };\kappa ,\mathbf{\Sigma }).$$

This separates:

1. anisotropy — which output/eigenvector direction is probed;
2. inhomogeneity — where the noised input or initial seed lies relative to the covariance eigenframe;
3. global scaling — how disagreement changes with sample count and effective degrees of freedom.

The fractional-matrix-power extension carries the same structure from one denoising step to the full deterministic sampling map.

Evidence

Question	Paper evidence	Calibrated reading
Do disjoint training splits yield aligned samples?	On FFHQ32, UNet and DiT models trained on disjoint 30k-image splits produce similar outputs from matched noise seeds; paired pixel MSE over 512 seeds is lower across models than to nearest training images.	Strong controlled evidence for same-seed alignment under the tested deterministic sampler, not universal sample identity.
Is the effect specific to one architecture or dataset?	The study covers FFHQ, AFHQ, CIFAR-10/100, and LSUN bedroom/church at 32 or 64 pixels; UNet is tested broadly and DiT on a subset.	Broad within low-resolution image diffusion, but not text-to-image, video, audio, or time series.
Does the linear Gaussian model explain the effect?	Empirical-mean/covariance Wiener-filter samples are highly consistent across splits and visually predict part of the deep-model output.	Explains a substantial shared component; nonlinear models show larger absolute deviations than linear theory predicts.
Does finite data create two regimes?	At 50k training steps, the tested models usually memorize for $n\le 1000$ and enter a non-memorizing renormalization regime for $n\ge 3000$.	The transition depends on model capacity, image resolution, architecture, data, and training recipe; 1000/3000 are not universal thresholds.
Can RMT predict which seeds disagree?	For FFHQ64 UNets with $n=30,000$, the seed-wise RMT score correlates with cross-split deviation at Spearman $r=0.33$ with $p=2.5\times 10^{-26}$ over 1000 seeds.	Statistically strong but moderate predictive power; useful structure, not precise per-seed error certification.
Can RMT predict linear-denoiser inhomogeneity?	At $n=1000$ and ${\sigma }^2=1$, the paper reports Pearson $r=0.94$ across noised images.	Strong evidence in the tractable linear setting; this number should not be transferred to deep networks.
Are shared first two moments causally relevant to consistency?	PC-stratified splits deliberately mismatch mean/covariance and substantially increase same-seed cross-model MSE relative to ordinary random splits.	Supports first-two-moment alignment as a necessary baseline in these experiments, not as a sufficient description of semantic quality.
What happens over longer training?	In 250k-step FFHQ64 runs, models first approach the linear predictor, later drift away, and often become less cross-split consistent; small datasets drift earlier and can show FID double-descent-like behavior.	Dataset size and checkpoint time are separate axes of the generalization/memorization story.

Gonzo Claim Audit

Gonzo statement	Audit against the paper
Models trained on different subsets of one distribution generate the same pictures from the same seed.	Supported with scope. The outputs are strikingly similar, not identical, under deterministic matched-seed sampling in controlled low-resolution image experiments.
The work has two phases: memorization and renormalization.	Supported, but the axis matters. The headline phase split varies dataset size at a fixed 50k-step recipe. The longer-training appendix adds a checkpoint-time drift. This is not automatically the same two-phase mechanism as Information Bottleneck compression or language-model capacity saturation.
Consistency is proved to come from shared low-rank Gaussian statistics rather than memorization.	Too strong as written. The paper rigorously derives the effect for linear Gaussian denoisers and shows that shared first and second moments explain much of deep-model consistency in the non-memorizing regime. It does not prove that all deep-model consistency is linear or that higher-order statistics and architecture are irrelevant.
Finite dataset size merely shifts the effective noise scale.	Exact for the expectation-level linear deterministic equivalent. Deep networks preserve the qualitative overshrinkage pattern but have additional nonlinear and architecture-dependent variability.
The framework can calculate the minimum sample density needed for a chosen high-frequency feature.	A plausible use, not an end-to-end demonstrated guarantee. The spectral formulas show which eigenmodes need more samples and the code evaluates $\kappa$, but the paper does not validate a production feature-level sample-size calculator with target error guarantees.
The work opens semantic editing directly in noise space without fine-tuning.	Exploratory evidence only. The appendix perturbs top covariance eigenspaces in FFHQ64 noise and shows interpretable background/face changes. The authors present this as inspiration for future training-free adaptation, not a general semantic editor with disentanglement or user-control benchmarks.

Relation To Nearby Sources

Why Diffusion Models Don’t Memorize and this paper describe complementary, not contradictory, axes:

• Why Diffusion Models Don’t Memorize varies checkpoint time and finds early generalization before later memorization, with memorization time increasing with dataset size.
• Diffusion Consistency RMT mainly varies dataset size at a fixed 50k-step recipe and finds small-data memorization versus larger-data renormalization; its 250k-step appendix then recovers the time axis by showing late drift from the linear predictor and reduced consistency.

How much can language models memorize? uses a third axis: dataset information relative to empirically measured parameter storage. Its 3.6 bits-per-parameter estimate and membership-inference fit are not evidence for the diffusion paper’s covariance-renormalization law.

Together, these sources argue against using two phases without naming the independent variable: checkpoint time, dataset size, data information relative to model capacity, or representation compression can produce different boundaries.

Foundation TSFM Relevance

Agenda slot	Verdict	Evidence	Missing pieces
Generative future distributions	adjacent	Supplies a theory and paired-seed protocol for reproducibility of deterministic noise-to-sample maps across training splits.	No numeric time series, probabilistic forecast paths, event streams, or trajectory generators are tested.
Scaling and data adequacy	adjacent	Links finite sample count and data spectrum to eigenmode-specific overshrinkage and cross-run disagreement.	Time-series windows are temporally dependent and non-stationary, so raw window count is not effective sample size.
Benchmark hygiene	warning	Shows that average sample quality, memorization, same-seed consistency, and spectral agreement are different measurements.	Needs time-series split-unit rules, stochastic-sampler decomposition, calibration, rare-regime slices, and task-utility tests.
Latent-state preservation	warning	Lower-variance directions can be consistently erased by finite-data shrinkage even when large-scale structure is reproducible.	A time-series study must show whether those directions encode noise, weak channels, rare regimes, exogenous variables, or decision-relevant latent state.
Control and counterfactuals	insufficient evidence	Moment-mismatched split construction is a useful diagnostic intervention on the dataset.	No actions, control inputs, interventions on system dynamics, counterfactual trajectories, rewards, planning, or closed-loop evaluation.

Because the direct agenda contribution is adjacent, warning, or insufficient evidence, this source should not be added to the central Foundation TSFM coverage matrix.

Time-Series Transfer

The strongest transfer is a paired reproducibility audit for generative multivariate time-series models:

1. split data by independent system, episode, device, tenant, or subject rather than overlapping windows;
2. train multiple generators on disjoint splits and, where possible, different architectures;
3. run each deterministic generator from matched base-noise trajectories;
4. report same-noise cross-run error separately from sample quality, diversity, calibration, memorization, and downstream utility;
5. decompose disagreement by frequency band, channel subspace, learned latent-state direction, regime, and horizon;
6. compare ordinary random splits with deliberately moment- or regime-shifted splits;
7. vary both effective sample size and checkpoint time.

A useful time-series analogue of the paper’s central warning is:

A generator can be reproducible because all runs retain the same dominant covariance structure while all runs consistently erase a rare but decision-critical regime.

Consistency is therefore a diagnostic, not a sufficient objective.

Artifact Audit

The official repository contains the reusable RMT solver/toolkit, numerical scripts, and interactive project site. Its README says curated example notebooks reproducing key figures are planned, so the release should not be described as a complete end-to-end reproduction package. GitHub exposed no repository license at ingest time. No named model checkpoint is released, which is consistent with the paper’s focus on many separately trained experimental models rather than one deployment model.

Limitations And Gotchas

• The formal analysis assumes a linear denoiser and a general-covariance sample model in a Wishart-like universality class.
• Expectation formulas isolate empirical-covariance error by setting empirical and population means equal.
• The variance result keeps a generic factorized contribution; strongly aligned probes can require an additional exchange term.
• The deep-model validation is qualitative or correlational; absolute deep-network variability is larger than linear RMT predicts.
• Same-seed alignment is tied to deterministic probability-flow-style sampling. Stochastic samplers add another source of variance.
• The experiments use low-resolution image datasets, fixed architectures/optimizers, 50k-step main runs, and specific sample-size grids.
• Same-seed consistency does not establish sample quality, diversity, calibration, privacy, semantic fidelity, or downstream utility.
• Nearest-neighbor evidence reduces a simple copying explanation but is not a complete privacy or training-data extraction audit.
• Matching mean and covariance can preserve broad structure while missing higher-order, rare, causal, or action-conditioned dynamics.
• In time series, overlapping windows, autocorrelation, seasonality, and shared entities can make nominal $n$ much larger than the effective number of independent trajectories.

Links Into The Wiki

• Training Dynamics
• Time-Series Generation
• Time-Series Benchmark Hygiene
• Foundation Time-Series Model Research Agenda
• Why Diffusion Models Don’t Memorize
• How much can language models memorize?

Open Questions

• Do diffusion or flow time-series generators exhibit stable same-noise mappings across disjoint trajectory splits?
• Which split unit gives approximately independent training realizations: window, episode, entity, device, tenant, or site?
• Can effective sample size replace raw window count in the renormalized-noise equation under autocorrelation and non-stationarity?
• Do lower-variance time-series directions correspond to disposable noise or to rare decision-critical state?
• Can covariance-eigenframe diagnostics be replaced by learned latent-state modes when channel covariance is non-stationary or nonlinear?
• How much same-seed consistency survives stochastic samplers, conditioning changes, classifier-free guidance, and variable horizons?
• Can training-free spectral edits manipulate a time-series scenario without corrupting constraints, cross-channel relations, or action effects?
• What matched protocol separates useful reproducibility from mode collapse or systematic rare-regime erasure?