Dust: Pretraining Transformers Without Backpropagation
Dust probes activations rather than all weights against one global reward. Token-level signals make many local probes parallelizable, but the implementation still estimates gradients and uses structural credit-assignment rules.
Source
- Archived primary article.
- Official technical article · appendix, October 2026.
- Samip Dahal, Bishwas Mandal, Serdar Gülbahar and Akshay Vegesna, Q Labs.
- Official code, captured at
b20f7c03eac630b6441ba6f254128c1761dc45ad. - Author Noise announcement, 2026-10-22; preserved with Noise, not treated as a separate Dust experiment.
Status And Credibility
Author-hosted research article with methods, appendix, comparative experiments and an MIT-licensed public implementation. No arXiv record, accepted venue or official paper PDF was verified from the linked primary artifacts; the preserved source is HTML/Markdown and visual assets, not a fabricated PDF publication. Treat this as an independently unreplicated Q Labs research release. Inclusion is Alex-directed, not a claim of tier-1 review.
Mechanism
- Add independent Gaussian noise to linear-layer outputs at each token.
- Score perturbations by centered token-level loss reductions, with future-token credit where the architecture warrants it.
- Average reward-weighted noise to estimate local output error.
- Form a weight-gradient estimate from output error and the cached layer input, then use an ordinary optimizer.
For , the conceptual distinction is
“Virtual population” refers to independent token-level perturbation signals evaluated in parallel; it is not a collection of completely independent sequence rollouts. Attention couples positions, and finite-population interference remains important. The implementation separates layer types, caches clean activations and reruns downstream blocks. Attention internals are scored against an estimated attention-output error. Residual mixing scalars use weight-space ES.
Thus forward-only does not mean no gradient estimates, no cached activations, no architecture-specific structure, or constant memory in sequence length. The recurrent streaming-memory argument for CD-RGE is not automatically a Dust property.
Evidence And Calibration
- Main experiments use GPT-style models on FineWeb, BPE vocabulary 4,096, sequences of 2,048 tokens and one epoch. The base model is the 37.7M-parameter system reused by Noise. Validation and test are distinct held-out sets; the article reports test loss at a validation-selected checkpoint.
- The main SGD-with-momentum grid varies token budgets from 100k to 20M and populations from 64 to 16k, with three seeds per cell. Dust beats its SGD-backprop comparison in some low-data regimes; gaps at larger budgets depend on population.
- The 1,000–10,000× claim compares against an EGGROLL-Transformer reproduction and depends on extrapolating population curves. It is not a measured speedup over backprop, nor a universal ranking against EGGROLL’s original recurrent/integer system.
- Predicted infinite-population loss limits are fits, not attained losses. In particular the article labels its 20M-token limit as loosely constrained.
- The Adam section separately measures and extrapolates population scaling. Noise’s Dust+Adam baseline trails its BP+Adam baseline; do not combine results across token budgets, optimizers or the later community recipes into a blanket “Dust beats backprop” claim.
- Larger-model/population and gradient-alignment probes extend beyond the main training grid. Gradient-alignment diagnostics use backprop-trained checkpoints at 10M, 100M and 1B tokens; they do not demonstrate Dust-trained trajectories reaching those scales or end-to-end training superiority.
- Tuning may use cosine alignment with a backprop gradient on one batch to propose hyperparameters. The training loop can be forward-only while the research/tuning process still uses derivatives.
BPTT Boundary
The authors explicitly defer external-program-in-the-loop networks and Transformers looped over many steps. Dust therefore supplies an activation-space estimator to investigate, not an experimental replacement for long-horizon BPTT. Dense per-token supervision also differs from a single delayed trajectory outcome.
The SOMA author’s criticism correctly asks about late-stage convergence and scale, but a generic count of weight-space probe directions does not directly evaluate Dust’s token-wise estimator.
Foundation TSFM Relevance
| Agenda slot | Verdict | Evidence | Missing pieces |
|---|---|---|---|
| Dynamic compute allocation | adjacent | Parallel local perturbation evaluations and estimator/optimizer design. | Numeric-model cost-quality frontiers and full cache/probe accounting. |
| Streaming state, long context, and constant updates | insufficient evidence | Recurrent/looped applications are proposed. | Actual recurrent-state and long-memory experiments. |
| Benchmarks and evaluation | warning | Population extrapolations and SGD/Adam distinctions change the headline. | Measured comparable costs to the same validation-selected target. |
Open Questions
- Can token-local credit retain an early event whose only supervision arrives far later?
- Which attention-specific paths translate to a nonlinear recurrent latent-state updater?
- Are the advantages retained after counting clean caches, all local probe families and tuning?