Scaling Recurrent Neural Networks to a Billion Parameters with Zero-Order Optimization

Direct evidence that recurrent models can train without BPTT, but not evidence that billion-parameter long-context language modeling is solved. The headline convergence improvement is mostly an optimizer-step comparison, not a matched-compute speedup.

Source

• Paper Markdown · PDF
• arXiv:2505.17852v1, 2025-05-23.
• Francois Chaubard and Mykel Kochenderfer, Stanford University.
• Official code, captured at 920138927e3bd4083a9989bbc243e70fc6077a6f.
• Author homepage. The later SOMA discussion supplies the user-provided social provenance, not a newly verified original launch thread.

Status And Credibility

Preprint by a credible Stanford team with public implementation; no venue acceptance verified. It is older than one year at ingest and is included as the specifically requested precursor to the 2026 SOMA/1.5-SPSA line, not as a current SOTA recommendation or citation-established golden standard.

Method

Central-difference random-vector gradient estimation (CD-RGE) evaluates complete recurrent forwards at perturbed weights:

$$\hat{g}=\frac{1}{K}\sum _{i=1}^K\frac{L(\theta +ϵu_i)-L(\theta -ϵu_i)}{2ϵ}{u}_i.$$

The paper uses Rademacher directions, reconstructible random seeds, chunked in-place perturbations, distributed loss evaluations and FlashRNN where supported. The update does not differentiate through time. At fixed model, batch and recurrent-state size, a streaming implementation need not retain the temporal activation graph. It still processes the sequence for each probe: constant state memory is not constant total compute or unlimited memory capacity.

Evidence And Limits

• Fixed-batch overfitting: DNC experiments use a random length-100 batch, batch size one, five runs and up to 512 perturbations per update. Figure 1 reports up to 19× fewer iterations for a 300k model and about 2× for 270M. The large-model feasibility experiment extends to 4.3B parameters; it is not a demonstrated 4.3B general-purpose language model.
• Synthetic transduction: COPY, REVERSE and ROLLING SUM train on short lengths (the introduction says 2–10; the methods specify 1–10) and evaluate 11–60. This is bounded length-generalization evidence, not ten-thousand-step retention.
• Penn Treebank: character-level language modeling with batch size 1,024 and training sequence length 10; the paper describes models through 1.1B and explicitly does not target transformer SOTA.
• Runtime table: Table 1 uses batch 1,024, sequence length 10 and A40 GPUs; distributed CD-RGE uses eight devices while the BPTT path is not equivalently distributed. Microbatching and kernel limits matter. Its linearized costs per 100k steps are not measured total costs to a common target quality.
• Probe accounting: 512 central-difference directions require paired evaluations, not one forward per optimizer step. Step-count gains alone do not establish compute efficiency.

Technical Caveats

The raw source’s statement that symmetric central differences are simply unbiased must be qualified: at finite perturbation radius, bias generally remains; its own expansion gives an $O({ϵ}^2)$ bias under smoothness assumptions. Likewise, an exact Gaussian smoothing identity does not automatically transfer unchanged to discrete Rademacher probes. Treat smoothing/generalization explanations as scoped arguments, not a universal theorem for every probe distribution.

The introduction calls FlashAttention a linear-attention mechanism. That is incorrect: FlashAttention is IO-aware exact attention, not a linear-attention architecture. The raw text is preserved, but the wiki does not inherit its attention-complexity or universal cost claims.

Foundation TSFM Relevance

Agenda slot	Verdict	Evidence	Missing pieces
Streaming state, long context, and constant updates	adjacent	Nonlinear recurrence and external-memory models trained without BPTT storage.	Long-lag numeric memory tests and continuous state carry.
Dynamic compute allocation	warning	Many probes can be parallelized.	Matched total FLOPs/GPU-hours, optimized BPTT and memory-cost frontiers.

Open Questions

• Does the advantage persist under checkpointed BPTT, tuned truncation horizons, equal device counts and target-quality comparisons?
• Can endpoint losses identify useful early-memory changes rather than merely support short-sequence fitting?

Links Into The Wiki

• SOMA · 1.5-SPSA · EGGROLL
• Forward-Only Sequence Training
• Efficient Recurrent Sequence Models