Training Dynamics

Summary

Training dynamics tracks the optimizer, loss, batch, learning-rate, curvature, compression, forgetting, and noise effects that shape what a model learns before any architecture-level claim is evaluated.

For the wiki’s agenda, this is supporting evidence rather than a model family. It matters because TSFM papers often report gains from architecture, data, or scale while treating optimizer dynamics as incidental. The training recipe can itself change representation geometry, stability, compression, retention, and generalization.

Current Evidence

Deep Learning is Not So Mysterious or Different adds a hypothesis-selection lens for generalization. Raw parameter count and the ability to fit randomized labels describe the capacity of the hypothesis space, not the complexity of the solution selected by architecture, prior, regularization, loss, and optimizer. For TSFM work, overparametrization claims should therefore report train fit, held-out performance, effective rank or dimensionality, compressibility, structured-versus-randomized controls, and rare-state preservation separately. This is upstream ICML position/theory synthesis rather than direct numeric time-series evidence.

Learning is Forgetting adds a whole-model representation-dynamics frame. It treats LLM pretraining as lossy compression: representations first expand to capture target-relevant information, then compress input information as they approach an Information Bottleneck bound. The useful wiki lesson is that learning can be a controlled forgetting process, not only accumulation of raw detail.

LLMs as Noisy Channels adds a scaling-law version of the same boundary. It models ordinary monotonic pretraining as a high-SNR regime and shows that Gaussian noise, low-bit quantization, and aggressive SFT can create U-shaped loss basins. For training-dynamics synthesis, this means the training recipe should name the perturbation regime, not only the model size or token count.

Implicit Curriculum Hypothesis adds the capability-clock view. It argues that language-model pretraining can learn skills in a stable compositional order, that aggregate validation loss hides this sequence, and that residual-stream function-vector geometry can predict held-out composite-task trajectories. For TSFM synthesis, the direct transfer is to log checkpoint-time emergence of latent-state, context-use, rare-regime, channel-coupling, and action-conditioned probes instead of treating one validation loss as the whole training state.

The Illusion of Superposition adds a latent-reasoning training-dynamics warning. Next-token pretraining biases models toward late-layer token commitment, fine-tuned latent thinking can learn answer shortcuts, and even from-scratch latent reasoning can lose superposition when capacity makes shortcuts easier. For TSFMs, latent-step or dynamic-compute claims should therefore report training regime, capacity, no-latent ablations, and state probes.

Treat these compression curves as comparative diagnostics over a fixed sample and estimator, not exact measurements of all latent information. The OLMo2 scale result also suggests compression depends on model capacity relative to data complexity.

FADE adds a parameter-level forgetting mechanism. It adapts per-parameter weight decay online so some weights can retain stable information while others forget stale mappings faster. This is not TSFM evidence yet, but it makes the retention question more precise: continual systems need selective forgetting, not blanket preservation or blanket decay.

The current neural evidence is final-layer adaptation. The naive all-layer extension underperformed head-only FADE, so this is a controlled-forgetting diagnostic and research direction rather than a general optimizer recipe.

SGD at the Edge of Stability adds a concrete optimizer-dynamics warning. In full-batch gradient descent, sharpness rises toward the classical $2/\eta$ edge-of-stability threshold. In mini-batch SGD, full-batch sharpness can stabilize below that threshold because stochastic gradient noise projected onto the top Hessian eigenvector changes the self-stabilizing oscillation.

The paper’s useful distinction is full-batch sharpness versus batch sharpness. A model can look “below the edge” under one sharpness definition while the batch-conditioned curvature is the quantity that approaches the stochastic stability edge.

DiffusionBlocks adds a local-objective training-dynamics branch. Instead of backpropagating one global loss through every layer, each block is assigned a denoising interval and learns a local score-matching-derived objective. The open training-dynamics question is whether those local objectives preserve global coordination, rare states, and pretrained representations once the method moves beyond from-scratch experiments.

DMax adds an on-policy train-inference-mismatch case: post-training includes the model’s own sampled noisy predictions, so the denoiser learns to recover from states it will actually create during aggressive parallel decoding. For TSFMs, the analogous test is whether self-generated corrupted trajectories preserve rare regimes, dense numeric values, exogenous variables, and actions rather than training the model toward its own shortcuts.

The Flexibility Trap adds rollout order as a training-dynamics variable. In the tested diffusion LMs, confidence-driven arbitrary order can postpone high-entropy reasoning forks until future context suppresses their ambiguity; left-to-right GRPO rollouts preserve broader Pass@$k$ coverage. For TSFMs, this is a hypothesis about exploration and credit assignment, not permission to condition on held-out future observations.

Why Diffusion Models Don’t Memorize adds a checkpoint-time branch. In diffusion training, sample quality and memorization can move on different clocks: ${\tau }_{\mathrm{gen}}$ arrives early, while ${\tau }_{\mathrm{mem}}$ grows roughly linearly with dataset size. This makes training duration and checkpoint selection part of the generalization mechanism, not just an implementation detail.

Dragon Hatchling adds a fast-state BPTT caveat. Sparse synapse activations suggest possible cheaper gradient-routing approximations, but the paper’s preliminary no-BPTT variant loses cross-language concept matching while retaining some language modeling ability. For this wiki, that makes BDH a reminder that fast-state architecture claims should report what the training path preserves, not only what inference state can represent.

Pretraining Recurrent Networks without Recurrence adds a recurrent-credit-assignment branch. SMT turns nonlinear RNN pretraining into supervised one-step prediction of Transformer-derived predictive memory states, giving an $O(1)$ token-to-token credit path instead of BPTT through an $O(T)$ recurrent chain. The training-dynamics caveat is exactly the rollout mismatch: one-step memory labels can drift under the model’s own latent-state trajectory, so DMT/post-training and long-rollout diagnostics are part of the claim, not optional engineering.

Synthetic Data for any Differentiable Target adds a data-attribution training-dynamics result. It shows that per-example value can depend on higher-order gradients through the target training trajectory, and that Adam optimizer-state terms can make the difference between a useful synthetic-data reward and a failing SGD/influence-function-style approximation.

Motion Attribution for Video Generation adds a cheaper local-gradient counterpart. Motion-masked per-example gradients are projected to 512 dimensions and compared with target-query gradients; this turns update direction into a fine-tuning-data value score without differentiating through the whole inner training trajectory. The approximation is materially imperfect—$\rho =74.7\%$ versus full-gradient ranking at 512 dimensions and $\rho =66\%$ for one timestep versus a ten-timestep reference—so projected-gradient data value should be reported with ranking-preservation and refresh-cadence diagnostics, not treated as exact influence.

A Bitter Lesson for Data Filtering adds a data-scaling branch: the apparent value of a filtered versus unfiltered corpus can reverse with model size, training steps, epoching, and regularization, so data-quality claims should name the compute regime and stopping rule.

The Universal Weight Subspace Hypothesis adds an adjacent weight-space diagnostic: repeated adaptation runs may converge into architecture-specific update directions. Treat this as update-geometry evidence, and require mean-update, rank-1, and retention baselines before calling it task-specific structure.

Reinforcement Learning Finetunes Small Subnetworks adds a direct post-training update-sparsity diagnostic. In the paper’s LLM RL settings, the update delta is sparse, nearly full-rank, and spread across layers; SFT updates are much denser. The key training-dynamics lesson is that update geometry should be measured with tolerance, precision, rank, layer distribution, and data-policy distance labels.

Exploration: Fine-Tuning With Parameter Decomposition adds a third weight-space adaptation diagnostic: first learn a mechanistic component basis, then edit scalar masks or prefactors over interpreted subcomponents. Treat its small-model German edit as mechanism evidence, not as a total-cost win unless decomposition and autointerpretation cost are included.

RLPT adds an objective-design branch. It turns unlabeled pre-training text into segment-level RL tasks, so segmentation, reward-model semantics, prefix matching, and cold-start SFT become part of the training dynamics. Use it as current LLM evidence that training-time scaling can be changed by objective design, not as direct TSFM or action-conditioned world-model evidence.

ExpRL adds a mid-training exploration branch. It shows a staged recipe where hidden reference solutions create dense on-policy rewards before ordinary sparse-reward RL. The training-dynamics lesson is that coverage under sampling, entropy collapse, reference-conditioned judge calibration, and prefix slicing can be as important as the final sparse reward itself.

Latent Thought Flow adds a flow-balance training objective for latent reasoning. Instead of direct reward maximization, it uses continuous GFlowNet subtrajectory balance, entropy weighting, and a reference-prior schedule to allocate credit to hidden trajectories according to answer quality and compute cost. The training-dynamics lesson is that latent-step utility depends on the objective and entropy regime, not just the existence of continuous hidden states.

Reading Frame For TSFMs

Use this page when a time-series paper’s gains may depend on training protocol rather than only model architecture:

• overparametrization does not establish overfitting by itself; separate representational capacity from the complexity and capability profile of the solution actually selected;
• aggregate loss can hide ordered capability emergence, so checkpoint-time probe suites need absolute competence thresholds and representation diagnostics;
• latent reasoning can shortcut unless the training objective and capacity make hidden state causally necessary;
• latent trajectory objectives should report entropy dynamics, reward/prior ablations, no-latent baselines, and whether the hidden trajectories are causally used rather than only shorter.
• compression pressure can change what information survives in the representation;
• SNR and perturbation pressure can make scaling non-monotonic, so training duration, precision, learning rate, and data repetition need to be logged alongside loss;
• “forgetting” can be destructive capability loss or a useful way to discard stale mappings;
• batch size and learning rate can change the effective curvature regime;
• the relevant noise is directional, not only scalar gradient variance;
• sharpness measurements need protocol labels;
• checkpoint age can change whether a generative model is in a generalization or memorization regime;
• the loss function can decide whether an edge-of-stability mechanism appears at all;
• optimizer and data-mixture changes can create representation differences that look architectural.
• data value estimates should name the target model, optimizer, number of inner training steps, and differentiable metric, because those choices can change which examples appear useful.
• projected-gradient data-selection reports should name the masked objective, query distribution, parameter subset, projection dimension, timestep/noise protocol, ranking-preservation error, score-refresh policy, and upfront attribution compute.
• post-training update-sparsity reports should name the precision/tolerance, update rank, layer distribution, data-policy distance, training duration, and retention tests.
• segment-level RL objectives should report the segmenter, reward model, false-positive controls, contamination policy, cold-start SFT recipe, KL/reference-policy settings, and matched-compute next-token or SFT baselines.
• reference-guided RL priming should report reference quality, wrong-reference controls, judge size, no-reference controls, prefix/step slicing, entropy and pass@k dynamics, downstream sparse-RL transfer, and retention/update-geometry diagnostics.
• diffusion or flow RL should report rollout order, sampler-learner mismatch, proposal coverage, per-position entropy, exact-versus-surrogate likelihood cost, and whether inference returns to a different parallel schedule.

Limitations

The current sources are mostly LLM, controlled online-learning, or upstream architecture-training evidence. They should be cited as training-dynamics and representation-dynamics warnings, not as direct recipes for numeric time-series models.

Related Pages

• Time-Series Scaling And Efficiency
• Inference Dynamics
• Time-Series Benchmark Hygiene
• LLM Post-Training
• Company-Local Block-Wise Fine-Tuning
• Foundation Time-Series Model Research Agenda

Open Questions

• Do larger TSFMs develop stronger useful compression biases at matched data and compute, or do they mainly gain raw capacity while erasing rare regimes?
• Which sharpness and noise diagnostics should be logged for TSFM pretraining?
• Can TSFM checkpoint selection use compression diagnostics rather than only validation loss?
• Can TSFM pretraining show stable capability-emergence orders over latent-state probes, and can internal representations predict future probe trajectories?
• Which information should a time-series model learn to forget under non-stationarity, and which rare state must be protected?
• Do TSFM objectives enter edge-of-stability regimes under practical training recipes?
• Do diffusion or flow TSFM objectives have a measurable ${\tau }_{\mathrm{gen}}$/${\tau }_{\mathrm{mem}}$ separation?
• Can local denoising objectives preserve global sequence or time-series state as well as end-to-end objectives?
• Can on-policy corrupted-state training improve TSFM horizon generation without amplifying base-model blind spots or erasing tail regimes?
• Can sequential training at decision-critical timestamps preserve diverse valid time-series trajectories without leaking future targets, and can inference still refine horizon blocks in parallel?
• Can fast-state models avoid full BPTT without losing cross-concept or cross-channel state binding?
• Which diagnostics predict long-rollout latent-state drift after one-step predictive-state training, before it appears as downstream forecast or rollout failure?
• Do repeated TSFM adaptation runs share task-specific update directions beyond a mean update?
• Do TSFM or world-model post-training runs show sparse but full-rank update deltas, or does dense numeric/state adaptation require denser movement?
• Can TSFM or world-model checkpoints be decomposed into stable component bases that expose reusable edit handles for channels, regimes, event-stream motifs, latent-state transitions, or intervention responses?
• Can segment-level RL objectives be designed for time-series windows or event streams without rewarding surface continuation and losing rare state?
• Can optimizer-dynamics probes separate architecture gains from training-recipe gains?
• Which no-latent or no-loop ablations should be required before crediting latent-step training with real dynamic compute?
• Can flow-balance or GFlowNet-style latent objectives preserve diverse useful trajectories without sacrificing serving latency or interpretability?
• Can SNR-aware scaling-law fits predict when additional data, larger models, low-bit serving, or post-training will hurt rather than help?
• Can metagradient data valuation be approximated cheaply enough to guide TSFM pretraining curricula without full backpropagation through long inner loops?
• How do AdamW, Muon, momentum, weight decay, gradient clipping, and distributed data parallelism change projected-noise sharpness dynamics?