Planning with Reasoning using Vision Language World Model

Source

• Raw Markdown: paper_vlwm-2025.md
• PDF: paper_vlwm-2025.pdf
• Preprint: https://arxiv.org/abs/2509.02722v2
• DOI: https://doi.org/10.48550/arXiv.2509.02722
• ICLR 2026 submission: https://openreview.net/forum?id=uvRApnkdbk
• Official author announcement: https://x.com/Delong0_0/status/1963883798841483688
• Hugging Face model page: https://huggingface.co/delong-chen/VLWM
• Hugging Face dataset placeholder: https://huggingface.co/datasets/delong-chen/VLWM
• Local release-status snapshot: VLWM training data

Status And Credibility

VLWM is an arXiv preprint submitted on 2025-09-02 and revised as v2 on 2025-09-06 by a Meta FAIR team. Its ICLR 2026 OpenReview record is marked Withdrawn Submission, so it should not be presented as an accepted ICLR paper. The work is credible as a current research artifact from an established team, but its model, training-data, and code release surface remains incomplete.

As checked on 2026-07-15, the official Hugging Face model repository is manually gated and its public manifest contains only .gitattributes and README.md; no weight file is visible publicly. The dataset repository is an empty placeholder with zero stored payload bytes. A commented URL in the arXiv source points to facebookresearch/VLWM, but that GitHub URL returned 404 and is not treated as released code.

Core Idea

VLWM makes natural language the planner-facing world-state representation. Given visual context, it predicts:

$$[\text{config},\text{context}]\stackrel{\mathrm{VLWM}}{\to }[\text{goal},\text{interpretation},⟨A_0,\Delta S_0⟩,\dots ,⟨A_N,\Delta S_N⟩].$$

Here $A_t$ is a high-level textual action and $\Delta S_t$ is a textual description of the resulting world-state change. The model therefore combines a policy-like route $S_t\to A_{t+1}$ with a language dynamics route $(S_t,A_t)\to S_{t+1}$.

flowchart LR
  Video[observed video context] --> Tree[hierarchical Tree of Captions]
  Tree --> Refine[LLM Self-Refine]
  Refine --> Target[goal + interpretation + action/state trajectory]
  Target --> VLWM[VLWM-8B]
  VLWM --> S1[System 1: direct plan decoding]
  VLWM --> Rollouts[candidate action/state rollouts]
  Rollouts --> Critic[self-supervised semantic critic]
  Critic --> S2[System 2: lowest-cost plan]

System 1 And System 2

• System 1 autoregressively emits one interleaved action/state trajectory. It is fast but commits to earlier actions and largely behaves like a policy learned from offline demonstrations.
• System 2 generates candidate plans, rolls their effects forward in language space, scores progress toward the interpreted goal with a learned critic, and selects the minimum-cost plan.
• The critic is trained with ranking constraints: coherent progress should lower cost, unrelated appended steps should raise it, and shuffled trajectories should cost more than ordered ones.

This is a useful high-level world-model interface because actions and state changes are inspectable. It is also lossy: free-form language can omit geometry, uncertainty, timing, object identity, and hidden physical state that a low-level controller needs.

Training Data And Target Generation

The data pipeline first segments videos hierarchically, captions each window, linearizes the resulting Tree of Captions, and uses Llama-4 Maverick with two Self-Refine rounds to extract goals and interleaved action/state targets.

The implementation table reports six video sources plus one text-only source:

Source	Videos (k)	Hours	Trajectories (k)	Steps (k)
NaturalReasoning	—	—	1,086.4	5,166.2
HowTo100M	167.8	18,512.3	1,093.2	5,438.1
COIN	7.6	302.9	36.2	181.7
CrossTask	2.1	163.1	10.4	55.1
YouCook2	1.2	102.3	5.8	31.9
EgoExo4D	0.6	53.5	3.1	18.8
EPIC-KITCHENS-100	0.5	68.9	2.2	14.0
Paper Overall row	179.8	19,202.9	2,179.6	10,604.3

The component rows do not add up to the Overall row: they sum to 2,237.3k trajectories and 10,905.8k steps, exceeding the reported totals by 57.7k and 301.5k. The introduction also names Ego4D as training data, while the implementation section explicitly says Ego4D was excluded and the table omits it. The empty public dataset repository cannot resolve either inconsistency.

Results

• On the VPA benchmark, the paper reports that VLWM-8B beats the prior VidAssist row on 8 of 12 cells and averages absolute gains of 3.2 points in success rate, 3.9 in mean accuracy, and 2.9 in mean IoU across the four evaluated COIN/CrossTask horizons.
• PlannerArena reports System 2 at Elo 1261 and a 27% Elo improvement over System 1. This is a five-annotator internal evaluation rather than a public benchmark service; the main text and appendix also disagree on pilot/IAA sample counts, so the exact protocol should be treated cautiously.
• On RoboVQA, the reported VLWM-8B BLEU-1 is 74.2; the paper also states BLEU-4 of 55.6.
• VLWM-critic-1B reaches 96.9% goal-achievement detection accuracy on the paper’s 4,410 VLWM test trajectories and 50.0% overall on the out-of-distribution OGP evaluation.
• Removing textual state descriptions drops VLWM goal-achievement detection from 96.9% to 88.1% overall, with the largest reported drop on EPIC-KITCHENS-100, from 90.1% to 64.0%. This supports the utility of explicit state text for this critic, not the physical sufficiency of that text.
• On the 570-sample WorldPrediction-PP benchmark, the critic reports 45.4% accuracy after visual inputs are converted to captions.

Dataset Release Audit

The paper says models and data will be open-sourced, but the exact VLWM corpus is not publicly available in the official dataset repository as of 2026-07-15.

Action100M is the closest released artifact from the overlapping team. Its accepted CVPR 2026 workshop paper explicitly says that its pipeline extends and improves the VLWM data-generation procedure. It must not be conflated with the missing VLWM training set:

Property	VLWM training corpus	Public Action100M preview
Source data	Six video datasets + NaturalReasoning	HowTo100M only
Unit	Goal-plan trajectory/window	One source video with hierarchical segment nodes
Main targets	Goal, goal interpretation, action, explicit textual state change	Brief/detailed action, actor, brief/detailed summary
Exact post-action state field	Yes, in the paper schema	No
Public status	Empty HF placeholder	120k-row 10% preview

Foundation TSFM Relevance

Agenda slot	Verdict	Evidence	Missing pieces
Context interface	adjacent	Conditions high-level planning on visual context and an explicit goal interpretation.	No numeric-feature schema, channel metadata, irregular timestamps, or operational context contract.
Latent-state prediction	adjacent	Predicts inspectable textual descriptions of state change rather than raw pixels.	Language is not a calibrated latent-state distribution and may omit dense numeric or geometric state.
Control and counterfactuals	partially closes	Exposes candidate actions and rolls out textual consequences before cost-based selection.	Actions are automatically inferred high-level descriptions, not typed control inputs or interventions; no causal identification or low-level executability test.
Multi-modal future distributions	insufficient evidence	System 2 searches multiple candidate plans.	Search over candidates is not a calibrated predictive distribution over future trajectories.
Benchmark level	adjacent	Evaluates VPA, PlannerArena, RoboVQA, critic goal detection, and WorldPrediction-PP.	Internal protocol inconsistencies, no public VLWM data/weights, and no numeric time-series or operational closed loop.

Limitations

• The ICLR 2026 submission was withdrawn.
• The official data repository is empty; the model page does not expose public weights in its public manifest; code is not verified as released.
• Training labels are machine-generated from model captions and LLM Self-Refine, so fluent state descriptions are not ground-truth physical state.
• The action descriptions are observed/inferred event labels from videos, not clean logged actions, control inputs, or interventions.
• Language state is interpretable but can hide continuous geometry, uncertainty, partial observability, irreversible effects, and execution constraints.
• System 2 is candidate generation plus critic selection; it does not establish calibrated uncertainty or exhaustive planning.
• The paper contains unresolved data-table and corpus-membership inconsistencies.

Links Into The Wiki

• VLWM
• Action100M
• World Models
• Slow Thinking For Robotics And Time Series
• Unified Multimodal Models
• Vision-Language Models
• Foundation Time-Series Model Research Agenda

Open Questions

• Can language world state be paired with a dense latent or typed state so explanations remain readable without discarding geometry and uncertainty?
• How much of the reported planning gain comes from better candidate generation versus critic ranking?
• Would a public release reproduce the reported table totals and clarify whether Ego4D contributed to training?
• Can this high-level action/state interface be grounded in executable control inputs and measured next-state dynamics?
• What is the right time-series analogue: natural-language incident state, typed event/state schema, or selective language readout from a numeric latent state?