1.5-SPSA: Probe-Space Preconditioning for Fast and Stable Zero-Order Training

Reweight random update directions using directional curvature, while spending more compute per update and taking fewer updates. The strongest evidence is LLM post-training, not large-scale pretraining or long-horizon recurrent memory.

Source

• Paper Markdown · PDF
• arXiv:2609.38095v1, 2026-09-29.
• Francois Chaubard, Mykel J. Kochenderfer, Chris Ré; Stanford University.
• Official code, pinned to 68036ff9d10b7fb7ec3625450db1190cecc66cc3; README, trainer and task-loader snapshots retained under the raw directory.
• Author homepage and author reply naming this method.

Status And Credibility

Stanford preprint with official implementation. No conference acceptance verified; the PDF’s ICML template/keyword and a repository filename containing ICML are not acceptance evidence. The homepage’s earlier work date and repository’s 2025 citation text do not supersede the canonical 2026 arXiv submission date. This is an explicitly requested discussion source, with substantial evaluation caveats below.

Mechanism

The method shares an unperturbed loss evaluation across probe directions, estimates directional curvature from three loss values, and dampens updates along high-curvature probes. Conceptually:

$${\hat{c}}_i=\frac{L(\theta +ϵu_i)-2L(\theta )+L(\theta -ϵu_i)}{{ϵ}^2}.$$

This is scalar preconditioning in probe space, not construction of a full parameter Hessian. The pinned repository uses a curvature-magnitude power with a floor, then divides the central-difference update by that damping factor. Bit-packed probes, fused kernels and distributed evaluations address implementation overhead. Plain in-place updates can avoid Adam moment storage; this does not imply every ZO implementation avoids all gradient buffers.

Evidence And Compute Accounting

• The paper evaluates OPT and Qwen3 post-training on classification/reasoning datasets and adds controlled ill-conditioning and DNC overfitting tests.
• Its introduction reports 94.5% SST-2 accuracy in 70 updates using approximately 179k forward passes, with +3.1 percentage points versus MeZO and +2.5 points versus its BP baseline. The abstract instead says +3.1 over both; preserve the discrepancy rather than repeat the stronger wording.
• The grid reallocates a nominal 500,000-forward-pass budget among effective batch size, probe count and update count. “70 versus 100,000 steps” is not the same compute ratio.
• For $s$ updates, $a$ accumulation batches and $K$ paired probes, the paper accounts for $sa(2K+1)$ forwards for 1.5-SPSA. Comparisons MUST retain microbatch sizes, tokens evaluated, clean passes, validation and search costs; a generic forward-pass count is not hardware-independent FLOPs.
• DNC results concern length-100 overfitting, not established long-range generalization.

Paper–Code And Evaluation Audit

The pinned public release is not an unqualified validation-selected reproduction:

1. The README says “checkpointing on best test accuracy.” In repo_1.5-SPSA-LLM_scripts_train.py, the branch around lines 1913–1937 updates best_test_acc and saves a checkpoint when test accuracy improves. That branch is gated on an active W&B run (if wandb_run is not None, line 1894). The paper instead defines convergence using a validation-loss plateau. Run provenance is needed to reconcile these protocols; this is a concrete selection-risk observation, not proof that all paper numbers used that exact code path.
2. The README’s best row says 72 iterations, while the paper says 70; its illustrative best log shows iteration 38. The 160-probe/128-batch result also differs. Treat README, illustrative logs and paper grid as separate artifacts until run provenance reconciles them.
3. The task loader can split a labeled validation set into validation/test when official test labels are hidden. Consequently the term “test” does not by itself establish identity with the comparator’s split. Matching dataset names is insufficient.
4. Equation 2 in the paper prints an extra factor of two in the probe average compared with the usual central-difference estimator; this ingest describes the mechanism rather than silently correcting the archived equation or claiming an exact paper–code match.
5. The implementation has explicit gradient-buffer paths as well as memory-efficient options. “No backward graph” is the defensible general property; universal zero auxiliary parameter-sized storage is not.

A credible superiority comparison needs frozen split IDs, validation-only checkpoint/hyperparameter selection, multiple seeds, matched total evaluated-token/FLOP budgets, and an untouched final test set. No training reproduction was run during this ingest.

Foundation TSFM Relevance

Agenda slot	Verdict	Evidence	Missing pieces
Dynamic compute allocation	adjacent	Trades many small updates for fewer, more parallel probe evaluations.	Hardware-calibrated cost-quality frontiers for recurrent predictors.
Streaming state, long context, and constant updates	insufficient evidence	Derivative-free updates can apply to recurrent bodies in principle.	Long-memory benchmarks, numeric trajectories and state-retention evidence.
Benchmarks and evaluation	warning	Public test-best checkpoint selection path and artifact discrepancies.	Clean selection and matched protocol replication.

Open Questions

• Does curvature weighting help on long recurrent rollouts after holding batch/probe allocation fixed?
• Do reported accuracy gains survive validation-only selection and identical held-out splits?

Links Into The Wiki

• SOMA · Recurrent CD-RGE
• Forward-Only Sequence Training
• Training Dynamics
• LLM Post-Training
• Time-Series Benchmark Hygiene