Temporal Straightening for Latent Planning

Source

• Raw Markdown: paper_temporal-straightening-2026.md
• PDF: paper_temporal-straightening-2026.pdf
• arXiv: 2603.12231v2
• ICML 2026 record: official poster page
• OpenReview: forum Ik1mKtUYlZ
• Project page: agenticlearning.ai/temporal-straightening
• Code: agentic-learning-ai-lab/temporal-straightening
• User-provided X post: Haiyu Wu commentary
• Local X snapshot: papers/temporal-straightening-2026/x_post_haiyuwu1_2077177383883624703.md
• Local conversation response: papers/temporal-straightening-2026/x_conversation_2077177383883624703.json
• Local artifact and status notes: papers/temporal-straightening-2026/research-context-links.md

Status And Credibility

This is an accepted ICML 2026 paper by Ying Wang, Oumayma Bounou, Gaoyue Zhou, Randall Balestriero, Tim G. J. Rudner, Yann LeCun, and Mengye Ren. The first arXiv version appeared on 2026-03-12; the ingested v2 is the ICML camera-ready version. Acceptance is corroborated by the official ICML poster record, the official project page, and the camera-ready source, even though the OpenReview forum was Cloudflare-gated during ingest.

The paper comes from a credible NYU/Brown/University of Toronto team and provides an MIT-licensed public implementation. At capture time the public GitHub view reported 94 stars, 10 forks, and one commit on main; the README documents training and planning commands but no independently verified reproduction or checkpoint-based evaluation was found. A later paper and independent technical commentary cite the idea, but the ingest search did not identify a full independent replication of the reported planning results. Numerical results below therefore remain paper-reported ICML evidence rather than independently reproduced measurements.

Core Claim

A latent state can be predictive yet poorly shaped for optimization. Temporal Straightening adds a local curvature penalty to an action-conditioned JEPA world model so consecutive latent displacements align. In the evaluated visual goal-reaching tasks, the resulting geometry makes latent Euclidean distance a better proxy for feasible progress and improves gradient-based action-sequence optimization.

flowchart LR
  O["observations o_t"] --> E["sensory encoder"]
  A["actions / control inputs a_t"] --> EA["action encoder"]
  E --> Z["latent states z_t"]
  EA --> P["JEPA predictor"]
  Z --> P
  Z --> C["local curvature loss"]
  P --> R["candidate latent rollout"]
  R --> G["goal cost in latent space"]
  G --> GD["GD or MPC action optimization"]

Method

For three consecutive latent states, define latent velocities

$$v_t=z_{t+1}-z_t,v_{t+1}=z_{t+2}-z_{t+1}.$$

The straightness score and curvature penalty are

$${\mathcal{C}}_t=\frac{v_t^{\top }{v}_{t+1}}{\Vert v_t{\Vert }_2\Vert v_{t+1}{\Vert }_2},{\mathcal{L}}_{\mathrm{curv}}=1-{\mathcal{C}}_t.$$

The action-conditioned JEPA world model combines this with stop-gradient next-latent prediction:

$${\mathcal{L}}_{\mathrm{total}}=\Vert {\hat{z}}_{t+1}-\mathrm{sg}(z_{t+1}){\Vert }_2^2+\lambda {\mathcal{L}}_{\mathrm{curv}}.$$

The evaluated implementations use either a frozen DINOv2 visual backbone plus a trainable projector or a ResNet trained from scratch, together with an action encoder and ViT dynamics predictor. Spatial features are important: the main setup retains the patch-token grid, reduces channels from 384 to 8, and applies curvature regularization after a learned aggregation head. The appendix compares patch-wise, mean-pooled, flattened, and learned-aggregation variants and reports the learned aggregation head as best overall.

At planning time, the model rolls forward under candidate control-input sequences. Open-loop gradient descent minimizes terminal latent goal distance; MPC replans after each executed action. The paper also compares GD with CEM and reports that CEM often achieves higher absolute success but is roughly 10 times slower in its tested single-L40S setup with 200 samples and 10 iterations.

Theory Scope

The conditioning theorem is not a general nonlinear-world-model proof. It assumes linear latent dynamics

$$z_{t+1}=A{z}_t+B{a}_t$$

and defines an $\epsilon$-straight transition through $\Vert A-I{\Vert }_2\le \epsilon$. For square actions and invertible $B$, the paper bounds the effective condition number of the planning Hessian by

$${\kappa }_{\mathrm{eff}}(H)\le \kappa (B{)}^2{\left(\frac{1+\epsilon }{1-\epsilon }\right)}^{2(K-1)}.$$

For lower-dimensional actions, the statement is restricted to the controllable subspace. The connection between the practical cosine penalty and $\Vert A-I{\Vert }_2$ requires bounded velocity variation and smooth actions along visited directions. The nonlinear experiments support the mechanism empirically, but the theorem does not prove global convexity, global controllability, or improved planning for arbitrary nonlinear latent dynamics.

Evidence And Results

The main evaluation uses Wall, PointMaze-UMaze, PointMaze-Medium, and PushT, with 50 test samples and three data-sampling seeds. The tasks are visual 2D goal-reaching environments, not physical-robot deployments or numeric multivariate time series.

For the frozen-DINOv2 spatial projector, adding curvature regularization changes open-loop/MPC success as follows:

Environment	No curvature	With curvature
Wall	80.00 / 90.67	90.67 / 100.00
PointMaze-UMaze	44.00 / 81.33	94.00 / 100.00
PointMaze-Medium	72.00 / 96.67	82.67 / 98.67
PushT	70.00 / 78.67	77.33 / 85.33

These are matched rows within the paper’s own protocol, not a cross-paper leaderboard. Other architecture rows are more heterogeneous: explicit straightening can strongly help a from-scratch ResNet while some global-feature metrics decrease. The safe conclusion is that the regularizer is useful across the tested model families but does not improve every reported cell.

For the long-horizon stress test, the target is selected 50 simulator steps away. Because the protocol uses a frameskip of five, this should not be read as 50 learned-predictor transitions. Performance drops and model error compounds. With the combined spatial-plus-aggregated planning cost, the best reported PushT MPC result is 36.00% ± 5.29 for the straightened ResNet. The camera-ready DINO-WM row is 3.33% ± 2.36 open-loop and 27.33% ± 6.66 MPC, not 0%. On PointMaze-Medium, the same ResNet/combined-cost row reports 68.67% ± 4.16 open-loop and 98.67% ± 1.15 MPC. The authors exclude Wall and UMaze from this stress test because random 50-step rollouts can still produce short geodesic distances there.

The paper also supplies a Teleported-PointMaze qualitative case where a transition is visually discontinuous but temporally short. It shows a representative straightened-model plan using the teleport. This is useful mechanism evidence, but one illustrated success is not a broad benchmark for irreversible, stochastic, or discontinuous dynamics.

X Post Provenance And Claim Audit

The supplied X post is by Haiyu Wu, who is not an author of this paper. It is third-party research commentary and a pointer to the paper through the attached title image.

The captured conversation contains the root plus 11 public replies, including four follow-up replies by Haiyu Wu. Those replies mention trying VISReg for JEPA training and say that the idea “really needs a stable manifold for collapse prevention.” The phrase is not technically defined there, and neither the follow-ups nor unrelated public replies are evidence for this paper’s method or results.

X framing	Paper-grounded reading
“Locally straightening the latent trajectory significantly improves world-model planning.”	Supported in the four tested visual goal-reaching environments, with heterogeneous gains across model/configuration cells. It is not yet a general world-model or time-series result.
The cosine loss is simple.	The displayed local loss is accurate, but the full recipe also needs latent prediction, stop-gradient anti-collapse, a tuned $\lambda$, architecture choices, and planner-specific costs.
Applying it to patch tokens is better than applying it to a CLS token.	Preserving spatial patch-token structure is strongly supported. However, the main curvature loss is applied through a learned aggregation head over spatial features; the appendix reports this aggregation variant as better than independent patch-wise cosine.
PushT reaches 36% at horizon 50 versus 0% for DINO-WM.	The 36% value is the straightened ResNet MPC result with a combined spatial/global cost. In the camera-ready table, DINO-WM is 3.33% open-loop and 27.33% MPC, so the “0%” baseline is not supported by the ingested version.
The paper predates LeWorldModel, whose trajectories “could be straightened further,” and the idea should benefit time series.	The arXiv chronology supports the paper-before-LeWorldModel part, and LeWorldModel reports emergent temporal path straightening. No matched experiment adds this loss to LeWorldModel, and the time-series claim remains a transfer hypothesis: the paper does not test numeric features, irregular timestamps, event streams, exogenous variables, or operational interventions.

The post is therefore useful because it highlights a transfer hypothesis and one long-horizon number, but the paper and camera-ready tables remain the evidence source.

Limitations And Gotchas

• The empirical domains are simple visual goal-reaching environments. There is no physical-robot, real operational, healthcare, power-grid, or multivariate time-series validation.
• A local cosine penalty is not the same as globally linear dynamics. A path can be locally straight along observed trajectories while remaining globally curved, discontinuous, aliased, or uncontrollable.
• The theorem assumes linear latent dynamics and explicit controllability conditions; nonlinear guarantees are left open.
• The method uses a symmetric Euclidean goal cost. The authors explicitly note that asymmetric or irreversible dynamics may require directional costs such as quasimetrics.
• Straightening can suppress meaningful turns, regime changes, intervention boundaries, and backtracking if those are real system behavior rather than nuisance curvature.
• The point-prediction world model does not represent separated multi-modal futures or calibrated uncertainty.
• Long-horizon PushT still shows rollout drift, blurry decoded predictions, and low absolute success in several settings.
• Stop-gradient is used as the main anti-collapse mechanism. The curvature loss is orthogonal to collapse prevention and does not itself guarantee a non-degenerate state.
• Public code exists, but the local ingest did not run training, checkpoints, or planning evaluation, and no independent reproduction was verified.

Foundation TSFM Relevance

Agenda slot	Verdict	Evidence	Missing pieces
Latent-state geometry	promising transfer hypothesis	Explicitly regularizes consecutive latent velocities so local Euclidean geometry better tracks feasible temporal progress.	Needs numeric multivariate time-series evidence with irregular sampling, regime changes, rare events, and dense-value probes.
Control and counterfactuals	partially closes outside time series	Candidate action sequences are differentiated through an action-conditioned latent world model and evaluated in open-loop and MPC.	Needs typed interventions, exogenous variables, constraints, uncertainty, and non-vision action-conditioned benchmarks.
Long-horizon planning	warning plus mechanism	Straightening improves several 50-step results, but compounding error and trajectory drift remain visible.	Needs much longer horizons, stochastic futures, hierarchical planning, and matched planner/latency budgets.
Representation quality	partially closes outside time series	Separates predictive accuracy from planner-facing geometry and tests whether latent distance tracks geodesic progress.	Needs tests that retain abrupt but valid events, asymmetric transitions, rare regimes, and non-Euclidean costs.
Benchmark hygiene	useful protocol	Reports architecture, feature layout, planner, open-loop/MPC split, seeds, and long-horizon failures.	Needs independent reproduction, public checkpoints/logs, robustness shifts, and matched comparisons inside stable-worldmodel.

For time-series/world-model work, the transferable object is a time-aware latent-velocity regularizer, not the literal image recipe. With irregular timestamps, velocities should be normalized by elapsed time; with typed actions and interventions, expected changes should be conditioned on those control inputs; with real regime changes, the evaluation must ensure that straightening does not erase meaningful discontinuities.

Links Into The Wiki

• Temporal Straightening
• World Models
• Latent-Space Predictive Learning
• JEPA
• LeWorldModel
• AdaJEPA
• SkyJEPA
• stable-worldmodel
• On Training in Imagination
• LeJEPA Identifiability
• Contradictions And Open Tensions
• Foundation Time-Series Model Research Agenda

Open Questions

• Does curvature regularization improve planning after matching encoder, predictor, planner, optimization steps, and wall-clock latency inside stable-worldmodel?
• Can time-normalized latent velocities handle irregular sampling without confusing long gaps with high curvature?
• How should the regularizer condition on known actions, control inputs, interventions, and exogenous variables?
• When should symmetric Euclidean goal distance be replaced by a directional quasimetric or learned cost?
• Can local straightening preserve genuine regime changes, safety events, and rare transitions instead of smoothing them away?
• Does the method remain useful with probabilistic or multi-modal latent rollouts?
• Can explicit straightening improve LeWorldModel or SkyJEPA beyond their reported emergent temporal smoothness under a matched protocol?