Forward-Only Sequence Training
Removing a temporal backward graph can make a long rollout fit in memory; it does not guarantee that the model learns which early event matters. Forward-only methods change the estimator, objective, architecture, or hardware tradeoff. They do not all solve the same problem.
Mechanism Map
| Route | What changes | What the source actually supports |
|---|---|---|
| Recurrent CD-RGE | Paired weight probes replace BPTT | Large-model feasibility, fixed-batch fitting, short transduction and PTB; not arbitrary long-memory learning |
| EGGROLL | Low-rank weight perturbations improve population throughput | Recurrent/integer pretraining and post-training; systems gains are not universal BP wins |
| SOMA | Independent objectives for domain-specialized recurrent experts | Reduced cross-expert estimator noise and improvements over tested monolithic ZO; BPTT remains better in reported budget controls |
| 1.5-SPSA | Directional-curvature damping and batch/probe allocation | Primarily post-training; step count hides many forwards; public test-best checkpoint path is a risk |
| Dust | Activation probes and token-level local credit | Transformer pretraining research; long recurrent/looped training is future work |
| Noise | Versioned forward-only training benchmark | Fast threshold attainment and improved quality under specified rules, not long-memory evidence |
Related but not equivalent: ParaRNN parallelizes the actual recurrent trajectory/adjoint solve rather than removing derivatives; DMT uses predictive-memory supervision with teacher and rollout-distribution constraints; DiffusionBlocks replaces recurrent-depth training with local denoising supervision in tested settings. Recurrent depth is not temporal memory.
What BPTT Replacement Means
For a fixed-state recurrent model, parameter perturbations can be evaluated over an entire trajectory without retaining all activations for backward. A final loss can therefore influence updates to parameters used at early times without a truncated derivative window.
However:
- full sequential forward work remains, repeated across probes;
- weak influence of an early event can become an indistinguishable loss difference rather than a vanishing analytic gradient;
- chaotic/sensitive trajectories, stochastic losses and rare delayed outcomes can make estimates noisy;
- finite state capacity and forgetting remain architecture problems;
- cached activations, optimizer moments, batching and parallel probe populations can dominate real memory even without an autodiff graph;
- token-local loss credit is not equivalent to one scalar outcome at the end of a long trajectory.
The practical question is which cost or learning bottleneck moved, not simply whether backward() disappeared.
Independent Experts Are An Objective Change
SOMA’s independent losses remove noise generated by perturbing other experts on a separable objective. Dividing an arbitrary coupled model into blocks while keeping one shared final loss does not automatically reproduce that benefit. Its fixed router and separately learned recurrent states trade away joint representation learning. For multivariate time series, verify cross-channel relationships, rare regimes and action-conditioned state transitions before accepting that decomposition.
The original small-expert recipe, wide comparison recipes, seed learning and updated browser demo have different head/combiner contracts. The SOMA source records these rather than treating the name as one invariant architecture.
Evidence Ladder And Accounting
- Can run: fits memory and produces parameter updates.
- Can fit: reduces loss on a fixed batch.
- Can generalize: succeeds on a held-out distribution under validation-only selection.
- Can remember: retains early task-relevant information over long state-carry rollouts.
- Is efficient: improves a target-quality frontier at matched aggregate compute, memory and hardware.
- Scales: benefits persist across model sizes, data budgets, optimization stages and unseen regimes.
Neither a billion-parameter fitting experiment nor a small-model speedrun skips these steps. Compare full-trajectory evaluations, batch/token exposure, probe count, clean passes, optimizer updates, tuning, seed pretraining, communication and compilation separately. Equal optimizer steps, equal wall-clock with different GPU counts, and equal aggregate GPU-hours on different devices are distinct controls.
Proposed Time-Series / World-Model Test
This is a research hypothesis, not a measured result: begin with a small recurrent memory updater or adapter trained by ZO while holding the rest of a predictor fixed. Compare it against the same trainable module using full BPTT, checkpointed BPTT and several TBPTT horizons; retain an end-to-end BP model as a capability reference. Test multiple seeds and lag lengths.
The task SHOULD require remembering an early event, a rare regime change or a control input whose consequence appears later. Include both dense intermediate losses and a delayed endpoint-only loss, plus distractors and held-out longer horizons. Measure retained-information probes, long-rollout prediction, cross-channel coupling, peak memory, aggregate GPU-hours and wall-clock time to the same held-out target. Tune/checkpoint on validation only. Account for repeated full rollouts per update and reset/carry rules.
A useful falsifier is that ZO fits the sequence in memory but cannot recover early-event information at the same total budget as an optimized BP baseline. A memory saving without a preservation advantage is still an engineering result, not solved temporal credit assignment.
Relation To Foundation TSFM Agenda
The cluster is adjacent to streaming latent-state maintenance and dynamic compute, and a warning for benchmark hygiene. It does not close numeric multivariate modeling, causal intervention identification or reliable long-horizon action-conditioned rollout. SOMA sharpens the modularity-versus-coupling tradeoff; Noise sharpen the estimator-versus-speedrun distinction.
Open Questions
- Is useful update quality governed by total perturbed dimension, effective task dimension, local activation dimension, or an architecture-dependent combination?
- Can local objectives preserve cross-module temporal dependencies without restoring most shared-loss variance?
- Where is the crossover between additional forward probes and checkpointed backward recomputation on actual hardware?