Training Dynamics

Summary

Training dynamics tracks the optimizer, loss, batch, learning-rate, curvature, compression, forgetting, and noise effects that shape what a model learns before any architecture-level claim is evaluated.

For the wiki’s agenda, this is supporting evidence rather than a model family. It matters because TSFM papers often report gains from architecture, data, or scale while treating optimizer dynamics as incidental. The training recipe can itself change representation geometry, stability, compression, retention, and generalization.

Current Evidence

Deep Learning is Not So Mysterious or Different adds a hypothesis-selection lens for generalization. Raw parameter count and the ability to fit randomized labels describe the capacity of the hypothesis space, not the complexity of the solution selected by architecture, prior, regularization, loss, and optimizer. For TSFM work, overparametrization claims should therefore report train fit, held-out performance, effective rank or dimensionality, compressibility, structured-versus-randomized controls, and rare-state preservation separately. This is upstream ICML position/theory synthesis rather than direct numeric time-series evidence.

Small-Scale Experiments adds a scale-dependent hyperparameter-sensitivity result. In its language-model sweeps, near-optimal configurations occupy more of the search space as parameters and data grow, while the noisy-quadratic effective hyperparameter count trends downward with model size. The training-dynamics lesson is that small and large models should not receive the same sweep budget by default, and a missing small-scale law can be an optimization/search failure rather than evidence against scaling. The result remains conditional on one corpus, recipe, search parameterization, and very recent preprint.

CF-Reg adds a model-relative boundary diagnostic and regularization mechanism. In its binary-classification experiments, the probability of finding a prediction-flipping sample inside a fixed radius rises with overtraining, and maximizing average counterfactual distance often improves tabular held-out accuracy. For TSFM work this is an ablation proposal, not a ready recipe: the distance metric, perturbation semantics, generator, rare-regime behavior, sequential grouping, and compute overhead must be reported, and model-boundary counterfactuals must not be confused with action-conditioned futures.

CFLight adds a targeted replay-data construction pattern rather than general optimizer evidence: rare unsafe events trigger alternative-action transitions from a shared pre-event state, and those generated transitions train a safety/efficiency controller. Its tabular Q-learning lemma does not establish convergence of the deep implementation; for TSFM training studies, the useful ablations are direct-simulator versus learned-model replay, factual/generated mixing, rare-event oversampling bias, uncertainty gating, and group-safe episode splits.

Learning is Forgetting adds a whole-model representation-dynamics frame. It treats LLM pretraining as lossy compression: representations first expand to capture target-relevant information, then compress input information as they approach an Information Bottleneck bound. The useful wiki lesson is that learning can be a controlled forgetting process, not only accumulation of raw detail.

How much can language models memorize? adds a distinct capacity-accounting frame. It estimates how much sample-specific information a saturated GPT-style training procedure stores, reports an empirical lower bound near 3.6 bits per parameter, and finds a double-descent boundary when dataset information exceeds measured model capacity. This is not the same measurement as checkpoint-time Information Bottleneck compression: it varies data relative to capacity and depends on a compression estimator plus reference model.

LLMs as Noisy Channels adds a scaling-law version of the same boundary. It models ordinary monotonic pretraining as a high-SNR regime and shows that Gaussian noise, low-bit quantization, and aggressive SFT can create U-shaped loss basins. For training-dynamics synthesis, this means the training recipe should name the perturbation regime, not only the model size or token count.

Implicit Curriculum Hypothesis adds the capability-clock view. It argues that language-model pretraining can learn skills in a stable compositional order, that aggregate validation loss hides this sequence, and that residual-stream function-vector geometry can predict held-out composite-task trajectories. For TSFM synthesis, the direct transfer is to log checkpoint-time emergence of latent-state, context-use, rare-regime, channel-coupling, and action-conditioned probes instead of treating one validation loss as the whole training state.

The Illusion of Superposition adds a latent-reasoning training-dynamics warning. Next-token pretraining biases models toward late-layer token commitment, fine-tuned latent thinking can learn answer shortcuts, and even from-scratch latent reasoning can lose superposition when capacity makes shortcuts easier. For TSFMs, latent-step or dynamic-compute claims should therefore report training regime, capacity, no-latent ablations, and state probes.

Treat these compression curves as comparative diagnostics over a fixed sample and estimator, not exact measurements of all latent information. The OLMo2 scale result also suggests compression depends on model capacity relative to data complexity.

FADE adds a parameter-level forgetting mechanism. It adapts per-parameter weight decay online so some weights can retain stable information while others forget stale mappings faster. This is not TSFM evidence yet, but it makes the retention question more precise: continual systems need selective forgetting, not blanket preservation or blanket decay.

The current neural evidence is final-layer adaptation. The naive all-layer extension underperformed head-only FADE, so this is a controlled-forgetting diagnostic and research direction rather than a general optimizer recipe.

SGD at the Edge of Stability adds a concrete optimizer-dynamics warning. In full-batch gradient descent, sharpness rises toward the classical $2/\eta$ edge-of-stability threshold. In mini-batch SGD, full-batch sharpness can stabilize below that threshold because stochastic gradient noise projected onto the top Hessian eigenvector changes the self-stabilizing oscillation.

The paper’s useful distinction is full-batch sharpness versus batch sharpness. A model can look “below the edge” under one sharpness definition while the batch-conditioned curvature is the quantity that approaches the stochastic stability edge.

DiffusionBlocks adds a local-objective training-dynamics branch. Instead of backpropagating one global loss through every layer, each block is assigned a denoising interval and learns a local score-matching-derived objective. The open training-dynamics question is whether those local objectives preserve global coordination, rare states, and pretrained representations once the method moves beyond from-scratch experiments.

DMax adds an on-policy train-inference-mismatch case: post-training includes the model’s own sampled noisy predictions, so the denoiser learns to recover from states it will actually create during aggressive parallel decoding. For TSFMs, the analogous test is whether self-generated corrupted trajectories preserve rare regimes, dense numeric values, exogenous variables, and actions rather than training the model toward its own shortcuts.

The Flexibility Trap adds rollout order as a training-dynamics variable. In the tested diffusion LMs, confidence-driven arbitrary order can postpone high-entropy reasoning forks until future context suppresses their ambiguity; left-to-right GRPO rollouts preserve broader Pass@$k$ coverage. For TSFMs, this is a hypothesis about exploration and credit assignment, not permission to condition on held-out future observations.

Why Diffusion Models Don’t Memorize adds a checkpoint-time branch. In diffusion training, sample quality and memorization can move on different clocks: ${\tau }_{\mathrm{gen}}$ arrives early, while ${\tau }_{\mathrm{mem}}$ grows roughly linearly with dataset size. This makes training duration and checkpoint selection part of the generalization mechanism, not just an implementation detail.

Diffusion Consistency RMT adds a separate dataset-size and spectral branch. At a fixed 50k-step recipe, the tested image generators move from small-data memorization toward a larger-data renormalization regime where finite samples overshrink lower-variance modes and same-seed cross-split disagreement becomes partly predictable from the data covariance. Its 250k-step appendix reconnects this axis to checkpoint time: models first approach the shared linear predictor, then can drift away and become less consistent. Training reports should therefore keep dataset size/effective sample size, checkpoint time, deterministic sampler, paired seed, and spectral slice separate rather than calling every boundary the same two-phase law.

Dragon Hatchling adds a fast-state BPTT caveat. Sparse synapse activations suggest possible cheaper gradient-routing approximations, but the paper’s preliminary no-BPTT variant loses cross-language concept matching while retaining some language modeling ability. For this wiki, that makes BDH a reminder that fast-state architecture claims should report what the training path preserves, not only what inference state can represent.

Pretraining Recurrent Networks without Recurrence adds a recurrent-credit-assignment branch. SMT turns nonlinear RNN pretraining into supervised one-step prediction of Transformer-derived predictive memory states, giving an $O(1)$ token-to-token credit path instead of BPTT through an $O(T)$ recurrent chain. The training-dynamics caveat is exactly the rollout mismatch: one-step memory labels can drift under the model’s own latent-state trajectory, so DMT/post-training and long-rollout diagnostics are part of the claim, not optional engineering.

Synthetic Data for any Differentiable Target adds a data-attribution training-dynamics result. It shows that per-example value can depend on higher-order gradients through the target training trajectory, and that Adam optimizer-state terms can make the difference between a useful synthetic-data reward and a failing SGD/influence-function-style approximation.

Motion Attribution for Video Generation adds a cheaper local-gradient counterpart. Motion-masked per-example gradients are projected to 512 dimensions and compared with target-query gradients; this turns update direction into a fine-tuning-data value score without differentiating through the whole inner training trajectory. The approximation is materially imperfect—$\rho =74.7\%$ versus full-gradient ranking at 512 dimensions and $\rho =66\%$ for one timestep versus a ten-timestep reference—so projected-gradient data value should be reported with ranking-preservation and refresh-cadence diagnostics, not treated as exact influence.

A Bitter Lesson for Data Filtering adds a data-scaling branch: the apparent value of a filtered versus unfiltered corpus can reverse with model size, training steps, epoching, and regularization, so data-quality claims should name the compute regime and stopping rule.

The Universal Weight Subspace Hypothesis adds an adjacent weight-space diagnostic: repeated adaptation runs may converge into architecture-specific update directions. Treat this as update-geometry evidence, and require mean-update, rank-1, and retention baselines before calling it task-specific structure.

Reinforcement Learning Finetunes Small Subnetworks adds a direct post-training update-sparsity diagnostic. In the paper’s LLM RL settings, the update delta is sparse, nearly full-rank, and spread across layers; SFT updates are much denser. The key training-dynamics lesson is that update geometry should be measured with tolerance, precision, rank, layer distribution, and data-policy distance labels.

Exploration: Fine-Tuning With Parameter Decomposition adds a third weight-space adaptation diagnostic: first learn a mechanistic component basis, then edit scalar masks or prefactors over interpreted subcomponents. Treat its small-model German edit as mechanism evidence, not as a total-cost win unless decomposition and autointerpretation cost are included.

RLPT adds an objective-design branch. It turns unlabeled pre-training text into segment-level RL tasks, so segmentation, reward-model semantics, prefix matching, and cold-start SFT become part of the training dynamics. Use it as current LLM evidence that training-time scaling can be changed by objective design, not as direct TSFM or action-conditioned world-model evidence.

ExpRL adds a mid-training exploration branch. It shows a staged recipe where hidden reference solutions create dense on-policy rewards before ordinary sparse-reward RL. The training-dynamics lesson is that coverage under sampling, entropy collapse, reference-conditioned judge calibration, and prefix slicing can be as important as the final sparse reward itself.

Latent Thought Flow adds a flow-balance training objective for latent reasoning. Instead of direct reward maximization, it uses continuous GFlowNet subtrajectory balance, entropy weighting, and a reference-prior schedule to allocate credit to hidden trajectories according to answer quality and compute cost. The training-dynamics lesson is that latent-step utility depends on the objective and entropy regime, not just the existence of continuous hidden states.

Reading Frame For TSFMs

Use this page when a time-series paper’s gains may depend on training protocol rather than only model architecture:

• overparametrization does not establish overfitting by itself; separate representational capacity from the complexity and capability profile of the solution actually selected;
• aggregate loss can hide ordered capability emergence, so checkpoint-time probe suites need absolute competence thresholds and representation diagnostics;
• latent reasoning can shortcut unless the training objective and capacity make hidden state causally necessary;
• latent trajectory objectives should report entropy dynamics, reward/prior ablations, no-latent baselines, and whether the hidden trajectories are causally used rather than only shorter.
• compression pressure can change what information survives in the representation;
• memorization and privacy reports should name the compression estimator, reference model, data-to-capacity ratio, attack family, average-versus-worst-case scope, and rare-sample behavior; chance average F1 for one loss-based membership attack is not a privacy guarantee;
• SNR and perturbation pressure can make scaling non-monotonic, so training duration, precision, learning rate, and data repetition need to be logged alongside loss;
• hyperparameter sensitivity can itself change with scale, so report the sampled search distribution, configurations per scale, near-optimum tail diagnostics, and whether larger models receive only inherited or freshly tuned settings;
• “forgetting” can be destructive capability loss or a useful way to discard stale mappings;
• batch size and learning rate can change the effective curvature regime;
• the relevant noise is directional, not only scalar gradient variance;
• sharpness measurements need protocol labels;
• checkpoint age can change whether a generative model is in a generalization or memorization regime;
• dataset size can change whether a diffusion generator is in a memorization or covariance-renormalization regime, while eigenband-specific overshrinkage can hide behind good aggregate quality;
• the loss function can decide whether an edge-of-stability mechanism appears at all;
• optimizer and data-mixture changes can create representation differences that look architectural.
• data value estimates should name the target model, optimizer, number of inner training steps, and differentiable metric, because those choices can change which examples appear useful.
• projected-gradient data-selection reports should name the masked objective, query distribution, parameter subset, projection dimension, timestep/noise protocol, ranking-preservation error, score-refresh policy, and upfront attribution compute.
• post-training update-sparsity reports should name the precision/tolerance, update rank, layer distribution, data-policy distance, training duration, and retention tests.
• segment-level RL objectives should report the segmenter, reward model, false-positive controls, contamination policy, cold-start SFT recipe, KL/reference-policy settings, and matched-compute next-token or SFT baselines.
• reference-guided RL priming should report reference quality, wrong-reference controls, judge size, no-reference controls, prefix/step slicing, entropy and pass@k dynamics, downstream sparse-RL transfer, and retention/update-geometry diagnostics.
• diffusion or flow RL should report rollout order, sampler-learner mismatch, proposal coverage, per-position entropy, exact-versus-surrogate likelihood cost, and whether inference returns to a different parallel schedule.

Limitations

The current sources are mostly LLM, controlled online-learning, or upstream architecture-training evidence. They should be cited as training-dynamics and representation-dynamics warnings, not as direct recipes for numeric time-series models.

Related Pages

• Time-Series Scaling And Efficiency
• Inference Dynamics
• Time-Series Benchmark Hygiene
• LLM Post-Training
• Company-Local Block-Wise Fine-Tuning
• Foundation Time-Series Model Research Agenda

Local Feature Specialization Versus Global Metrics

Revenge of Monosemanticity supplies a controlled example of feature learning that is not merely recovery of one global low-dimensional subspace: different dense-MLP neurons align with different cluster-specific predictive directions. Its small-initialization population-gradient-flow theorem explains specialization, while its polynomial sample-complexity result concerns a separate constrained-ERM estimator with noiseless routing. The two results do not establish one finite-sample gradient-training guarantee. The global-AGOP kernel obstruction also does not cover every empirical RFM or cluster-local kernel method; the latter improve when given MLP-derived local structure.

For heterogeneous multivariate time-series representations, this sharpens an untested question: do training dynamics recover regime-specific useful features, or do global diagnostics conceal rare-regime failures? Evaluate overlapping and imbalanced regimes, and keep sample efficiency separate from matched-compute efficiency.

Open Questions

• Do larger TSFMs develop stronger useful compression biases at matched data and compute, or do they mainly gain raw capacity while erasing rare regimes?
• Which sharpness and noise diagnostics should be logged for TSFM pretraining?
• Does noisy-quadratic effective hyperparameter count predict TSFM tuning difficulty across model size, context, horizon, channel count, and data volume?
• Can TSFM checkpoint selection use compression diagnostics rather than only validation loss?
• Can a multivariate time-series model’s sample-specific capacity be estimated in a unit stable across sampling rate, channel count, patching, tokenization, and missingness?
• Can TSFM pretraining show stable capability-emergence orders over latent-state probes, and can internal representations predict future probe trajectories?
• Which information should a time-series model learn to forget under non-stationarity, and which rare state must be protected?
• Do TSFM objectives enter edge-of-stability regimes under practical training recipes?
• Do diffusion or flow TSFM objectives have a measurable ${\tau }_{\mathrm{gen}}$/${\tau }_{\mathrm{mem}}$ separation?
• Do diffusion or flow TSFMs show a separate data-size transition from memorization to renormalization, and can effective trajectory count plus spectral diagnostics predict it?
• Can local denoising objectives preserve global sequence or time-series state as well as end-to-end objectives?
• Can on-policy corrupted-state training improve TSFM horizon generation without amplifying base-model blind spots or erasing tail regimes?
• Can sequential training at decision-critical timestamps preserve diverse valid time-series trajectories without leaking future targets, and can inference still refine horizon blocks in parallel?
• Can fast-state models avoid full BPTT without losing cross-concept or cross-channel state binding?
• Which diagnostics predict long-rollout latent-state drift after one-step predictive-state training, before it appears as downstream forecast or rollout failure?
• Do repeated TSFM adaptation runs share task-specific update directions beyond a mean update?
• Do TSFM or world-model post-training runs show sparse but full-rank update deltas, or does dense numeric/state adaptation require denser movement?
• Can TSFM or world-model checkpoints be decomposed into stable component bases that expose reusable edit handles for channels, regimes, event-stream motifs, latent-state transitions, or intervention responses?
• Can segment-level RL objectives be designed for time-series windows or event streams without rewarding surface continuation and losing rare state?
• Can optimizer-dynamics probes separate architecture gains from training-recipe gains?
• Which no-latent or no-loop ablations should be required before crediting latent-step training with real dynamic compute?
• Can flow-balance or GFlowNet-style latent objectives preserve diverse useful trajectories without sacrificing serving latency or interpretability?
• Can SNR-aware scaling-law fits predict when additional data, larger models, low-bit serving, or post-training will hurt rather than help?
• Can metagradient data valuation be approximated cheaply enough to guide TSFM pretraining curricula without full backpropagation through long inner loops?
• How do AdamW, Muon, momentum, weight decay, gradient clipping, and distributed data parallelism change projected-noise sharpness dynamics?