Noise: Speedrun for Pretraining without Backpropagation
Noise measures forward-only training under versioned benchmark rules. Its 1.6-second record is time to a modest validation threshold on eight H100s, not time to a general-purpose LLM or a 4,000× win over backpropagation.
Source
- Archived announcement · official article, October 2026.
- Official repository, created 2026-10-12, snapshot 2026-10-25, commit
3343973b6f1af119effcdea2622f1fe672c51b1e. - Pinned rules/current leaderboard.
- Pinned 10M-token archive.
- Samip Dahal’s announcement thread, 2026-10-22. Raw full conversation and same-account narrative are preserved under
papers/noise-2026/. - Francois Chaubard’s criticism, archived with SOMA.
Status And Credibility
Open-source Q Labs benchmark and engineering competition, not a separate peer-reviewed paper. Records link to scripts and logs; this ingest verifies documentation and pinned artifacts, not GPU reproduction. Authors in the repository citation: Serdar Gülbahar, Bishwas Mandal, Akshay Vegesna and Samip Dahal. Core contributors listed there are Gülbahar and Mandal.
Versioned Benchmark Contract
| Field | Historical 10M track discussed on X | Snapshot after the 2026-10-22 revision |
|---|---|---|
| Model | 37.7M-parameter Dust Transformer | Same fixed model for tracks 1/2; new track 3 frees architecture |
| Corpus | 10M FineWeb training tokens | 30M FineWeb training tokens |
| Training exposure | At most one epoch | At most one epoch; each training sequence feeds at most one update |
| Speed target | Validation cross-entropy at most 4.2 | At most 3.5 |
| Hardware | One node, 8×H100 | One node, 8×H100 |
| Speed record | 1.6 s, final loss 4.186 | 8 min 20 s, final loss 3.499 |
| Timing | Training updates only | First update to last; validation excluded; random-token compilation warmup allowed, then fresh state |
| Selection | Final-weight evaluation | Seed 0; no checkpoint selection using validation loss |
The old and new timings are different contracts, not a regression on the same task. The speedrun can stop before exhausting its data allowance. Headline training-token budget is not proof that a winning run consumed every token. Model/data preparation and compilation are outside the update timer.
The no-backprop rule forbids both autodiff modes and hand-written full backward passes. It permits locally measured forward-loss credit with at most one nonlinear propagation step; linear maps do not count toward that step. Hence entries are not necessarily pure black-box weight-space search.
Where The Record Came From
The archived progression includes estimator and optimizer changes, not merely kernel acceleration of unchanged Dust:
- orthogonal/structured probes and an associative credit memory;
- Boost → Cut stagewise and selected-path credit;
- direct identity/residual-path credit and approximate attention paths;
- structured initialization, averaging and early skipping;
- GPU sharding, fused kernels, fewer launches and overlapped communication.
The final archived recipe says 16 rows per update, 250 updates, with skipping from the first update. Its pinned implementation and result log are retained. The approximately 4,000× headline compares reaching 4.2 with the initial Dust+Adam forward-only baseline, not with backprop.
The quality track is separate: the old best is 3.676 versus the listed 3.769 BP+Adam baseline; the announcement says backprop with the same stronger optimizer remains ahead. In the current 30M README, forward-only reaches 3.409, versus BP+Adam 3.419 and BP with ANVIL II on block matrices plus Adam elsewhere 3.340. These are reported quality scores, not proof of equal-compute superiority.
Dispute Audit
Chaubard worries that quick early threshold attainment hides late-stage noise and dimensionality problems, and recommends independent small experts. Noise’s organizers explicitly acknowledge that some tricks will not scale and have increased the data/quality target. Their new open-architecture track also relaxes the fixed-Transformer restriction.
The unresolved empirical question is how methods behave along model/data/quality/compute frontiers, not whether the record is fabricated. Specialization to one public benchmark is a generalization risk, not by itself proof of test leakage. Dust uses activation-space probes with token-level credit, so generic weight-space dimensionality arithmetic is not a direct refutation. SOMA supplies a different architecture/objective intervention and does not establish that all alternatives fail.
Foundation TSFM Relevance
| Agenda slot | Verdict | Evidence | Missing pieces |
|---|---|---|---|
| Benchmarks and evaluation | warning | Rules, quality thresholds, hardware and timing boundaries alter the speed claim. | Private final evaluation, multiple seeds and harder unseen regimes. |
| Dynamic compute allocation | adjacent | Fast iteration on estimator/optimizer/kernel design. | Fair budget frontiers for numeric models. |
| Streaming state, long context, and constant updates | insufficient evidence | This benchmark trains a feed-forward Transformer. | Recurrent-memory, delayed-outcome and long-horizon BPTT tasks. |
Open Questions
- Which estimator improvements survive larger corpora and stricter target quality without retuning to a single public score?
- Can a forward-only speedrun include both long-memory capability gates and total GPU-hour/peak-memory frontiers?