Small-Scale Experiments: Are We There Yet?

Source

• Raw Markdown: paper_small-scale-experiments-2026.md
• PDF: paper_small-scale-experiments-2026.pdf
• Preprint: arXiv 2608.11859v1
• arXiv PDF: https://arxiv.org/pdf/2608.11859v1
• arXiv source archive: https://arxiv.org/src/2608.11859v1
• Upstream training library: facebookresearch/lingua at 437d680
• Analysis library: nicholaslourie/opda v0.8.0

Status And Credibility

Nicholas Lourie, Kyunghyun Cho, Karen Ullrich, and Sanae Lotfi submitted v1 to arXiv on 2026-08-12. The paper lists FAIR at MSL Meta and New York University affiliations. It is a very recent cs.LG preprint, not peer-reviewed or independently reproduced evidence. The team is credible for this topic: the paper builds directly on the authors’ prior peer-reviewed work on confidence bands, hyperparameter loss surfaces, and downstream scaling-law reliability, and it documents thousands of random-search configurations across held-out model scales.

The artifact boundary is material. The paper identifies the public Lingua commit used as its base and the released OPDA analysis package, but its training section also describes local Lingua modifications for data handling, learning-rate decay, SDPA, and post-normalization. Exact-title searches of the web, GitHub repositories/code, and the authenticated X recent-search API on 2026-08-17 did not find a paper-specific repository, project page, or author announcement. Treat the result as unusually detailed preprint evidence with partial, not complete, reproducibility.

Core Intuition

Small models do not lack scaling laws; they hide the fully tuned frontier. At small parameter and data budgets, validation loss is highly sensitive to hyperparameters, so a small sweep samples mostly suboptimal runs and makes the scale trend look noisy or absent. As parameters and data grow, near-optimal configurations occupy more of the search space and the effective dimension of the hyperparameter loss surface falls. Large models therefore look more regular partly because they are easier to tune.

This reverses a common workflow assumption. Small-scale work is not a cheap miniature of ordinary large-scale training. It needs more exploration per scale, followed by conservative scaling-law comparisons near observed data rather than blind extrapolation to the asymptote.

Method

The paper studies decoder-only Llama variants implemented in Meta Lingua and pretrained on a fixed FineWeb-Edu 100B-token subset. Effective parameter counts range from about 4M to 268M, and the token budget is 32 times effective parameters. The main ladder search reports 3,814 successful configurations across seven scales; a separate random-architecture search reports 766 successful configurations across 4M—134M scales. The pre-norm/post-norm case study uses 128 pre-norm configurations at each of 4M, 34M, and 134M, versus 511, 512, and 128 post-norm configurations at those scales.

The fitted pretraining law is

$$\mathcal{L}(p,d)=ϵ+\frac{\zeta }{p^{\iota }}+\frac{\eta }{d^{\kappa }},$$

where $p$ is effective parameter count, $d$ is training data, and the law describes the best loss found on the tuned frontier. Scales 4M—34M are used for fitting, 67M—134M for research-time validation, and 268M for the final held-out test.

The second tool is the noisy quadratic limit. Near an optimum, a hyperparameter configuration $\mathbf{X}$ is modeled as

$$\mathcal{L}(\mathbf{X})\approx y_{\ast }+(\mathbf{X}-{\mathbf{x}}_{\ast }{)}^{\top }{H}_{{\mathbf{x}}_{\ast }}(\mathbf{X}-{\mathbf{x}}_{\ast })+E,E\sim \mathcal{N}(0,\sigma ).$$

Its fitted tail distribution supplies two diagnostics: whether random search reached the asymptotic regime near the optimum, and the effective hyperparameter count $\gamma$, interpreted as local intrinsic dimension.

flowchart LR
    S[Small model scales] --> R[Extensive random search]
    R --> N[Noisy-quadratic diagnostic]
    N -->|tail fit fails| T[Search or implementation is not yet trustworthy]
    N -->|tail fit emerges| F[Fit tuned-frontier scaling law]
    F --> H[Validate on held-out larger scales]
    H --> P[Check loss-to-capability relation on fixed data]
    P --> C[Compare model families near observed data]
    C --> U[Carry the better family upward with light retuning]

Key Findings

1. Hyperparameter search depth is the enabling variable. With 4 or 16 configurations per scale, the scaling law is absent or unreliable; it becomes visible at 64 and accurate in the reported experiment at 256 configurations per scale.
2. Method details refine an existing law. Tuning per parameter—data budget cuts held-out test MSE by 50% in the reported ablation; decaying the learning rate cuts it by 98%. Parameter-count conventions and tying scaling exponents matter less than reaching the frontier.
3. Sensitivity falls with scale. Random-search loss distributions concentrate near the optimum as parameters and data grow. Increasing either axis helps most when the other is already large.
4. The local loss surface becomes lower-dimensional. Fitted effective hyperparameter count $\gamma$ trends downward with model size and reaches about one in the largest tested regimes. Parameters appear to drive this reduction more strongly than additional training data.
5. Extrapolation is still statistically fragile. Independent small-scale fits agree near the observed range but can diverge far away because the irreducible-error estimate is poorly identified before saturation.
6. The pre-norm case study recovers the accepted large-scale answer. Both architectures show scaling structure, but post-norm is consistently harder to tune. Pre-norm scales better over the observed region and remains better under the assumption that both families share an irreducible error.

Practical Diagnostic Contract

A small-scale architecture experiment SHOULD answer four questions rather than report one extrapolated number:

Diagnostic	Question	Failure signal
Search-tail fit	Did tuning reach the local optimum?	No noisy-quadratic tail or a tiny unstable asymptotic regime
Sensitivity trend	Will transfer get easier with scale?	Near-optimal configurations do not become more common
Loss-to-capability check	Is pretraining loss a valid comparison proxy?	Fixed-loss models separate on downstream tasks despite fixed data
Held-out scaling law	Does the tuned frontier emerge across scales?	Poor near-data fit or failure on a held-out larger scale

A model family should be compared where the law is supported by data. Far-range ranking that is driven by the fitted irreducible error should be presented as an assumption-sensitive scenario, not as a forecast.

Foundation TSFM Relevance

This is upstream language-model evidence, but its protocol transfers directly to time-series and world-model scaling studies.

Agenda slot	Verdict	Transfer	Missing pieces
Scaling and efficiency	strong adjacent evidence	Treat every architecture/data budget as a tuned frontier, not one run.	Direct TSFM IsoFLOP sweeps with irregular sampling, channel count, context, and horizon axes.
Training dynamics	strong adjacent evidence	Hyperparameter sensitivity and local loss-surface dimension are scale-dependent observables.	Numeric time-series tests across optimizers, patching schemes, and native multivariate models.
Evaluation	protocol contribution	Reserve larger scales as held-out validation/test points and report near-data versus far-extrapolation conclusions separately.	Capability probes for rare regimes, event streams, channel coupling, and action-conditioned rollout.
Model-centric research	conditional support	Fixed-corpus loss can rank model families by cost to reach a capability level.	Evidence that the loss-to-capability correspondence survives TSFM tasks and operational metrics.
Data-centric research	not supported	None: the paper explicitly holds data fixed.	Changing corpus composition can break the proxy and requires separate methodology.

For TSFMs, “data held fixed” MUST include more than the dataset name. Sampling rate, patching/tokenization, channel selection, missingness policy, event alignment, context fields, exogenous variables, and train/validation regime composition can all change what a unit of loss means.

Limitations

• The paper is a first-version preprint dated 2026-08-12.
• The evidence is decoder-only language-model pretraining on one fixed corpus and one broad recipe; it is not direct time-series, multimodal, or action-conditioned world-model evidence.
• The paper-specific training code, configurations, logs, and checkpoints were not found during ingest. Lingua and OPDA are public, but the described local training modifications are not packaged as an exact reproduction artifact.
• Most comparisons vary hyperparameters while fixing seeds for data order and model initialization. The noisy term is modeled, but full across-seed replication of every configuration would be much more expensive.
• The effective-dimension claim is local to the noisy-quadratic asymptotic regime and the chosen search parameterization. Maximum-update or other scale-invariant parameterizations may change the trend.
• The central loss-to-capability argument requires fixed pretraining data. It does not justify data-mixture, tokenization, curriculum, augmentation, or data-filtering comparisons.
• Scaling laws can extrapolate over the tested order-of-magnitude gap without supporting arbitrary frontier-scale forecasts. The irreducible-error term dominates far-range behavior and is weakly identified before saturation.

Links Into The Wiki

• Small-Scale Experiments
• Time-Series Scaling And Efficiency
• Training Dynamics
• Time-Series Benchmark Hygiene
• Foundation Time-Series Model Research Agenda
• Scaling Laws Versus Tiny Efficient Models
• Scaling Laws, Carefully
• Compute Optimal Tokenization

Open Questions

• How many configurations per scale are needed for TSFMs once patch size, context length, channel mixing, horizon, optimizer, and learning-rate schedule all enter the search space?
• Does effective hyperparameter dimension decrease with scale for native multivariate time-series models, or do channel count, irregularity, and regime diversity create new sensitive directions?
• Which TSFM capability probes stay aligned at equal validation loss when data composition is fixed: rare-regime detection, event-conditioned transitions, context use, cross-channel state, or action-conditioned rollout?
• Can μP/maximum-update-style parameterizations reduce the small-scale search burden without hiding architecture-specific sensitivity?
• How should an IsoFLOP study divide compute between more model scales, more configurations per scale, and more independent random seeds?
• Which conclusions remain stable if the irreducible-error term is shared across model families, fit separately, or omitted from the comparison range?