--- author: - Delong Chen - Théo Moutakanni - Willy Chung - Yejin Bang - Ziwei Ji - Allen Bolourchi - Pascale Fung bibliography: - paper.bib date: 2025-09-06 title: Planning with Reasoning using Vision Language World Model affiliation: Meta FAIR arxiv: 2509.02722v2 --- # Abstract Effective planning requires strong world models, but high-level world models that can understand and reason about actions with semantic and temporal abstraction remain largely underdeveloped. We introduce the **Vision Language World Model (VLWM)**, a foundation model trained for language-based world modeling on natural videos. Given visual observations, the VLWM first infers the overall goal achievements then predicts a trajectory composed of interleaved actions and world state changes. Those targets are extracted by iterative LLM **Self-Refine** conditioned on compressed future observations represented by a **Tree of Captions**. The VLWM learns both an action policy and a dynamics model, which respectively facilitate **reactive system-1 plan decoding** and **reflective system-2 planning** via **cost minimization**. The cost evaluates the semantic distance between hypothetical future states from VLWM rollouts and the expected goal state, and is measured by a critic model trained with self-supervision. The VLWM achieves state-of-the-art Visual Planning for Assistance (VPA) performance on both benchmark evaluations and the proposed PlannerArena human evaluation, where system-2 improves the Elo score by 27% over system-1. The VLWM models also outperform strong VLM baselines on RoboVQA and the WorldPrediction benchmark. # Introduction {#sec:intro} ![Example of a VLWM action-state trajectory given a video observation and a goal. VLWM can either generate a plan using one roll-out (system-1), or search over multiple actions by inferring the new world states and minimizing a given cost function (system-2).](figures/vpa_example.png){#fig:vlwm_rollout width="1\\linewidth"} World models enable AI agents to optimize action plans internally instead of relying on exhaustive trial-and-error in real environments [@lecun2022path; @ha2018world; @fung2025embodied], showing strong performance in planning across low-level, continuous control tasks such as robotic control [@oquab2024dinov; @yang2024learning; @assran2025vjepa2; @pan2025video] and autonomous driving [@hu2023gaia; @wang2024driving]. However, learning world models for *high-level* task planning -- where actions involve semantic and temporal abstraction [@SuttonPS99; @chen2025worldprediction] -- remains an open challenge. Bridging this gap could unlock a wide range of practical applications, such as AI agents in wearable devices assisting humans in complex tasks and embodied agents capable of autonomously pursuing long-horizon goals. To obtain a high-level world model, existing approaches fall short. Prompting-based practices [@hao2023reasoning; @tang2024worldcoder; @wang2024can; @gu2024your] is straightforward but inadequate as LLMs are not directly grounded in sensory experience. VLMs are primarily trained for visual perception and instead of action-conditioned prediction of world-state transitions. Meanwhile, learning from simulation environments [@hafner2024masteringdiversedomainsworld; @worldmodeling2025xipeng] cannot scale to divers real-world activities. Existing world models learned from natural videos often rely on generative architectures (*e.g.,* diffusion models) to generate future observations [@yang2023unisimneuralclosedloopsensor; @videoworldsimulators2024; @nvidia2025cosmosworldfoundationmodel]. Such formulation is not only ill-posed due to partial observability and uncertainty, but also inefficient, capturing task-irrelevant details and imposing high computational costs for long-horizon roll-outs. These limitations highlight the need for world models that predict in *abstract representation spaces*, rather than raw pixels. In this work, we propose to learn a world model that leverages natural language as its abstract world state representation. We introduce the **Vision Language World Model (VLWM)**, which perceives the environment through visual observations and predicts world evolution *using language-based abstraction* (Figure `\ref{fig:vlwm_rollout}`{=latex}). Language inherently provides semantic abstraction and is significantly more computationally efficient to generate compared to raw sensory observations. In comparison with latent embeddings in Joint Embedding Predictive Architecture (JEPA)-based world models [@lecun2022path; @zhou2025dinowmworldmodelspretrained; @assran2025vjepa2], language-based abstraction is intuitive, interpretable, and enables seamless integration with prior knowledge and extensive engineering ecosystems developed for LLMs/VLMs. Compared to current LLMs/VLMs paradigms that primarily focus on perception [@cho2025PerceptionLM], behavior cloning (SFT) [@zeng2023agenttuning], or reinforcement learning with verifiable rewards [@shao2024deepseekmath], we propose to perform direct world modeling as an objective based on massive, uncurated videos, *i.e.,* reward-free offline data [@sobal2025learning]. ![**Overview of VLWM.** **(a)** VLWM is a JEPA-style world model that predict abstract representation of future world states, instead of generating noisy and high-volume raw observations. **(b)** Given video contexts, VLWM's prediction target is a structured textual representation of the unobserved future. It includes goal and interleaved action ($A$) world state changes ($\Delta S$), all extracted automatically. **(c)** VLWM can infer possible goals from the context, and interpret them with current initial state and the expected final state. It supports both fast reactive system-1 plan generation and reflective system-2 reasoning based on cost minimization. ](figures/vlwm_framework.png){#fig:vlwm_framework width="1\\linewidth"} An overview of the framework is shown in Figure `\ref{fig:vlwm_framework}`{=latex}. To construct training prediction targets, VLWM employs an efficient abstraction pipeline that first compresses raw video into a hierarchical [Tree of Captions]{.smallcaps}, then refines it into structured goal-plan descriptions using an LLM-based [Self-Refine]{.smallcaps} [@madaan2023selfrefine]. The model is trained to predict these abstractions---capturing goal description, goal interpretation, actions ($A$) and world state changes ($\Delta S$) -- conditioned on visual context from past observations. From this, both a predictive world model $(S_t, A_t \rightarrow S_{t+1})$ and an action policy $(S_t \rightarrow A_{t+1})$ are learned. It enables straightforward plan generation via text completion, using the proposed action directly as policy. We term this approach **system-1** planning. However, the autoregressive nature of token decoding limits foresight and reflection, as each action decision become irreversible once made. Additionally, when training on large-scale, real-world video datasets which typically contain imperfect demonstrations, the resulting policy will also clone those suboptimal behaviors present in the data. To unleash the full potential of VLWM, we introduce a reflective **system-2** \`\`planning with reasoning" mode. In this mode, VLWM first generates multiple roll-out based on action candidates (either proposed by itself or externally provided) and predicts resulting world states. We then search for the candidate action sequence that minimize a scalar *cost*, which is evaluated by a **critic** module that assess the desirability of candidate plans. This critic is a language model trained through a self-supervised objective: it learns to assign lower costs to valid progress toward the goal and higher costs to counterfactual or irrelevant actions, effectively measuring how closely each candidate action aligns with the desired goal state. The process of optimizing action plan by searching for a cost-minimizing candidate is a form of reasoning [@lecun2022path]. It enables the agent to perform trial-and-error internally with its learned world model to obtain the optimal action plans. The VLWM is extensively trained on a large corpus of both web instruction videos and egocentric recordings, including COIN [@tang2019coinlargescaledatasetcomprehensive], CrossTask [@zhukov2019crosstaskweaklysupervisedlearning], YouCook2 [@zhou2018towards], HowTo100M [@miech2019howto100m], Ego4D [@grauman2022ego4dworld3000hours], EgoExo4D [@grauman2024ego], EPIC-KITCHENS-100 [@Damen2018EPICKITCHENS]. Collectively, there are 180k videos spanning over 800 days of duration. We generate [Tree of Captions]{.smallcaps} for each video, resulting in a total of 21M nodes of unique detailed video captions (2.7 trillion words). With iterative LLM [Self-Refine]{.smallcaps}, we extracted 1.2 million trajectories of goal-plan pairs, consisting of 5.7 million steps of actions and states. We also reformulate text-only chain-of-thought reasoning paths in NaturalReasoning [@yuan2025naturalreasoning] to action-state trajectories, obtaining additional 1.1 million goal-plan pairs. Our evaluations cover both human ratings of plan preference, and quantitative results on the Visual Planning for Assistance (VPA) benchmarks [@patel2023pretrainedlanguagemodelsvisual; @islam2024proposeassesssearchharnessing], achieving relative gains of +20% in SR, +10% in mAcc, and +4% in mIoU. Based on human ratings with our proposed `PlannerArena`, the procedural plans generated by VLWM system-2 mode is more preferred than prompting based methods. On the RoboVQA benchmark [@sermanet2024robovqa], VLWM achieves 74.2 BLEU-1 score, outperforming strong VLM baselines. We further evaluate critic models for goal achievement detection, and our trained critic outperform baseline semantic similarity models on both in-domain and OOD scenarios. It also established a state-of-the-art on [WorldPrediction]{.smallcaps} procedural planning task with 45% accuracy. Models and data will be open-sourced. # Methodology {#sec:method} We aim to train a world model that understands and predicts how actions affect physical world states, and to develop a framework for reasoning and planning where the world model serves as the core component. Our approach builds on the agent architecture introduced by @lecun2022path, where a reward-agnostic world model perform roll-out given candidate action plans, and the agent evaluates how closely each roll-out advances the current state toward the desired goal, and select the plan that minimizes this distance (*i.e.,* the cost). In the sections below, §`\ref{sec:method-plan extraction}`{=latex} details how we extract structured language-based representation as future world state abstractions, which includes semantic compression techniques for efficiency considerations and quality optimization strategies. Then, §`\ref{sec:method-system-2-planning}`{=latex} introduces how the critic is trained to evaluate cost in a self-supervised manner and explain the system-2 plan search based on cost-minimization. ## Vision-language World Modeling {#sec:method-plan extraction} Given a video, we aim to extract a structured language representation shown in Fig. `\ref{fig:vlwm_framework}`{=latex} (b), which consists of a goal (description and interpretation) and a procedural plan (action-state sequence). For such a video-to-text extraction task, one straightforward approach would be to provide a VLM with the full video and prompt it to extract the language representations. However, an impossible triangle arises: within a practical compute and memory budget, it is not feasible to simultaneously achieve 1) high spatial resolution for fine-grained perception, 2) long temporal horizon that spans many procedural steps, and 3) the use of a large and smart VLM that can follow complex instructions. To address this challenge, we propose a two-stage strategy. First, the input video is first compressed into a dense [Tree of Captions]{.smallcaps}, which significantly reduces the data volume while preserving essential semantic information (§`\ref{sec:method-tree of captions}`{=latex}). Then, structured goal-plan representations are extracted from these captions with LLMs. Because the second stage operates purely on text, it enables efficient processing with large LLMs and allows for iterative quality refinement through [Self-Refine]{.smallcaps} (§`\ref{sec:method-Self-Refine}`{=latex}). ### Compress Video into [Tree of Captions]{.smallcaps} {#sec:method-tree of captions} Each [Tree of Captions]{.smallcaps} consists of a set of video captions generated independently from different local windows of a video, collectively forming a hierarchical tree structure. It aims to holistically capture both fine-grained local details and long-horizon global information [@chen2024poca]. A key challenge lies in adaptively determining the tree structure, *i.e.,* the arrangement of different levels of windows for caption generation. Ideally, each node or leaf should correspond to a coherent *monosemantic* unit [@chen2024subobject], avoiding span across semantic boundaries. Existing temporal action localization and segmentation models [@ding2023temporal] are limited in their openness, as they rely on human annotations with closed-vocabulary action taxonomies and are typically trained on narrow video domains. We propose to create the tree structure via hierarchical feature clustering. Specifically, let $X$ be an untrimmed video, and let its feature stream be represented as $Z = \phi(X) = [\mathbf{z}_1; \dots; \mathbf{z}_T] \in \mathbb{R}^{T \times d}$, where each $\mathbf{z}_t$ is a $d$-dimensional feature vector produced by a video encoder $\phi$. We segment the feature stream $Z$, and accordingly the underlying video $X$, using **hierarchical agglomerative clustering** [@murtagh2012hierarchical]. Starting from the finest granularity---treating each item $\mathbf{z}_t$ as an individual cluster---the algorithm iteratively merges adjacent segments with the smallest increase in within-segment feature variance (*i.e.,* a measure of *polysemanticity*). This merging procedure is continued until there is only a single root node, and the full trace gives a hierarchical structure, where each node corresponds to a segment of the video. The choice of $\phi$ determines the behavior of the segmentation. In this paper, we adopt the Perception Encoder [@bolya2025perception]--a state-of-the-art model that excels at extracting scene and action information from videos. Once the hierarchical tree structure is constructed, we generate detailed captions for each video segment, excluding short segments shorter than five seconds. We use PerceptionLM [@bolya2025perception] for detailed video captioning. The resulting [Tree of Captions]{.smallcaps} achieves substantial compression: for instance, 1.1 TB video files in Ego4D [@grauman2022ego4dworld3000hours] can be compressed to under 900 MB of caption files. ### Extract Plans with LLM [Self-Refine]{.smallcaps} {#sec:method-Self-Refine} Given the compressed [Tree of Captions]{.smallcaps} extracted from the video, our next objective is to derive a structured textual representation that serves as the prediction target for VLWM. This representation includes the following four components: 1. **Goal description** is a high-level summary of the overall achievements (*e.g.,* ``` ``cook tomato and eggs” ```). In downstream applications, goal descriptions given by users are typically concise (*e.g.,* single sentence), omitting fine-grained details that holistically defines the final state. Therefore, explicit goal interpretations are required. 2. **Goal interpretation** includes contextual explanations that outlines both the initial and expected final world states. The initial state describes the current status of tools, materials, and dependencies, etc., providing essential grounding for plan generation. The final state interprets the goal description concretely to facilitate cost evaluation in system-2 planning. For example, ``` ``To achieve the goal, the eggs need to be cooked and mixed with tomatoes, and the mixture should be seasoned appropriately. The eggs should be whisked thoroughly to achieve a uniform texture...” ``` 3. **Action description** are the final outputs of the system that will be passed to downstream embodiments for execution or presented to users (*e.g.,* ``` ``Preheat the skillet on the stove” ```). They must be clear, concise, and sufficiently informative to enable the receiver to understand and produce the intended world state transitions. 4. **World states** are internal to the system and serve as intermediate representations for reasoning and plan search. They should be a information bottleneck: sufficiently capturing all task-relevant consequences of actions while containing minimal redundancy. For example: ``` ``This action prepares the skillet for cooking the eggs by increasing its temperature. The state of the skillet changes from cold to hot, making it ready for cooking. The oil used for preheating prevents the eggs from sticking to the skillet, ensuring they cook evenly...” ```. See Appendix `\ref{app:vlwm_trajectory}`{=latex} for more examples. To ensure that the generated components meet these requirements, we adopt an iterative [Self-Refine]{.smallcaps} procedure [@madaan2023selfrefine], leveraging LLMs as optimizers [@yang2023large]. We begin by providing the LLM with detailed descriptions of the output requirements, examples of the expected format, and the formatted [Tree of Captions]{.smallcaps} as input to generate an initial draft. In each refinement iteration, the LLM first provide a feedback to the current draft and produces a revised version accordingly. This self-refinement process is repeated for a predefined number of iterations, progressively optimizing output quality. To input [Tree of Captions]{.smallcaps} to LLMs, we format it using a depth-first search (DFS) traversal order. This linearization aligns with the hierarchical structure of textual documents that LLMs are typically trained on and familiar with (*e.g.,* Section 1 $\rightarrow$ 1.1 $\rightarrow$ 1.1.1 $\rightarrow$ 1.1.2 $\rightarrow$ \...). In this paper, we use `Llama-4 Maverick` for its efficient inference and support for extended context length. Notably, the [Self-Refine]{.smallcaps} methodology is not tailored to specific LLM architecture. Below are some example feedback messages generated by `Llama-4 Maverick` during the [Self-Refine]{.smallcaps} process: ```{=latex} \begin{displayquote} \textit{``Prepare the ingredients for Zucchini Curry.''} in the draft could be \textbf{broken down into more specific actions} like \textit{``Wash, peel, and chop the zucchini''} and \textit{``Chop the onions and tomatoes.''} \end{displayquote} ``` ```{=latex} \begin{displayquote} The state change after sautéing the onions, ginger, garlic, and green chilies could include more details about \textbf{how this step affects the overall flavor and texture} of the curry. \end{displayquote} ``` ```{=latex} \begin{displayquote} \textit{The action of ``Display the Zucchini Curry in a bowl''} is more of a \textbf{presentational step rather than a meaningful action} that advances the task progress, so it should be removed from the steps. \end{displayquote} ``` ### Training of Vision Language World Model {#sec:training vlwm} The training task of VLWM is defined in Eq.`\ref{eq:vlwm}`{=latex}. Here the `config` acts as system prompts. The `context` provides environmental information and can be either visual, textual, or both. The VLWM is trained to predict the future, represented by 1) `goal` description along with its interpretation (*i.e.,* the initial and expected final states), and 2) a `trajectory` consisting of sequence action ($A$) state ($\Delta S$) pairs. VLWM optimize the cross-entropy loss for next-token prediction on the right-hand side of Eq.`\ref{eq:vlwm}`{=latex}: $$ [\texttt{config}, \texttt{context}] \xrightarrow{\texttt{VLWM}} [ \texttt{goal},\; \texttt{interpretation},\; \underbrace{ \langle A_0, \Delta S_0 \rangle,\; \dots,\; \langle A_N, \Delta S_N \rangle }_{\texttt{trajectory}}\; ]. \label{eq:vlwm} $$ This input-output formulation reflects three levels of world modeling: 1) contextual goal inference, the prediction of the possible future achievements, 2) action anticipation--proposing possible next actions, and 3) action-conditioned world state dynamics prediction. Since actions and resulting state changes are generated in an interleaved, autoregressive manner, it enables straightforward **System-1 Reactive Planning** through direct text completion. Given the `config`, `context`, and the `goal` description, VLWM interprets the goal and generates a sequence of action-state pairs until an `` token is reached. From a language modeling perspective, the world state descriptions act as internal chains of thought: they articulate the consequences of each action, allowing VLWM to track task progress and suggest appropriate next steps toward the goal. This planning mode is computationally efficient and well-suited for short-horizon, simple, and in-domain tasks. Due to the $\texttt{(video, text)} \rightarrow \texttt{text}$ formulation in Eq.`\ref{eq:vlwm}`{=latex}, pretrained VLM can be used to initialize VLWM. This provides VLWM with strong visual perception, while also enabling it to inherit language understanding and generation capabilities, and commonsense knowledge in LLMs. ## Planning with Reasoning {#sec:method-system-2-planning} While the System-1 mode allows fast plan generation, it lacks the capabilities of having foresight, evaluating of alternatives, or revising suboptimal decisions. Once an action is emitted, it is fixed, preventing the model from reconsidering or correcting errors. This reactive behavior can lead to error accumulation, particularly in long-horizon or complex tasks. To address these limitations, we introduce **System-2 Reflective Planning**, where the world model is coupled with a *critic module* that evaluates the desirability of multiple predicted futures given the goal. This would enable a *reasoning* process that involves searching for the optimal plan via cost minimization [@lecun2022path]. ### Learning the Critic from Self-supervision {#sec:method-critic} ![**System-2 planning of VLWM**. **(a)**: the critic is trained in a self-supervised manner, assigning lower cost to valid progress, while assigning higher cost for adding irrelevant distractors or shuffling the steps. **(b)**: VLWM generates candidate action sequences and simulates their future state transitions. A critic evaluates the resulting state trajectories given the goal, and the planner selects the lowest-cost plan. ](figures/system-2.png){#fig:system-2 width="\\linewidth"} In world model-based planning, the cost function typically quantifies the distance between the world state resulting from a candidate plan and the desired goal state [@zhou2025dinowmworldmodelspretrained; @assran2025vjepa2]. It gives an estimation of how well the current task progress aligns with the intended goal and expected final state. In JEPA world models, this can be directly measured by *L1* or *L2* distance between fixed-dimensional embedding representations of world states. However, with VLWM, we must measure the **semantic distance** between language-based world state representations instead calculating distance in token space. Formally, given VLWM predictions as described in Eq. `\ref{eq:vlwm}`{=latex}, we aim to establish a distance function $\mathbf{critic}$ that evaluate cost $C = \mathbf{critic} (\{\texttt{goal, interpretation}\}, \{\texttt{trajectory}\})$. Ideally, the cost should be *low* when the predicted trajectory reflects meaningful progress toward the goal, and *high* when it deviates due to irrelevant or erroneous actions. To model this behavior, we train a language model in a self-supervised manner, enabling it to assess the semantic quality of predicted plans without requiring explicit annotations. As shown in Fig. `\ref{fig:system-2}`{=latex}(a), we explore two types of self-supervised training signals for the critic: 1. We construct training samples by starting from a base partial trajectory and appending either (i) valid next step(s) resulting from a coherent continuation of the task, or (ii) distractor step(s) sampled from an unrelated task. The critic independently predicts three cost scores: $C_\text{base}$, $C_\text{good}$, and $C_\text{bad}$ and the model is trained to satisfy the ranking constraints $C_\text{good} < C_\text{base} < C_\text{bad}$, encouraging the critic to distinguish meaningful progress from irrelevant or misleading continuations. 2. We generate negative samples by randomly shuffling the steps in a base trajectory, producing a corrupted sequence with cost $C_\text{shuffled}$. The critic is then trained to enforce $C_\text{base} < C_\text{shuffled}$, ensuring sensitivity to procedural order and temporal coherence. The critic is trained to minimize the following ranking loss with a fixed `margin`, supplemented with a cost centering regularization term weighted by a small constant $\lambda$ [@naik2024reward]. To construct training pairs $\langle C_\text{positive}, C_\text{negative} \rangle$, we iterate over all three types of self-supervised signal described above: $\langle C_\text{good}, C_\text{base} \rangle$, $\langle C_\text{base}, C_\text{bad} \rangle$, and $\langle C_\text{base}, C_\text{shuffled} \rangle$. $$% \scalebox{0.8}{$ \mathcal{L}_{\text{critic}} = -\max\left(0,\ \texttt{margin} + C_\text{positive} - C_\text{negative}\right)^2 + \lambda \left( C_\text{positive}^2 + C_\text{negative}^2\right). % $} \label{eq:critic-loss}$$ In addition to VLWM progress data, the critic formulation also supports supervision from external sources to enhance generalization. For example, preference tuning datasets-comprising triplets of a query, a preferred (chosen) response, and a rejected response---can be directly leveraged. Similarly, since the critic aims to model semantic distance, it can benefit from triplet-based datasets designed for learning sentence embeddings. These sources provide additional positive/negative pairs that can be used to further augment the training data of the critic. ### System-2 Planning by Cost Minimization {#sec:method-system-2} System-2 planning involves the coordination of three components: the VLWM, the critic, and an actor. As illustrated in Fig. `\ref{fig:system-2}`{=latex}(b), the actor proposes candidate action sequences, the VLWM simulates their effects, and the critic evaluates their costs. The final plan is selected by identifying the candidate sequence with the lowest predicted cost. The actor can be instantiated either as the VLWM itself or as an external module (*e.g.,* LLMs), particularly in cases where additional constraints on the action space or output format must be respected. The actor may vary the number of proposed candidates to control the search width or generate partial plans to enable more efficient tree search. In additional to the cost evaluated by the critic, task-specific penalties or guard-rails can be incorporated into the cost function, allowing the planner to respect external constraints, safety rules, or domain-specific preferences. # Experiments {#sec:expe} ## Implementation Details ### VLWM-8B **Sources of Videos.** As summarized in Table. `\ref{tab:vlwm-data}`{=latex}, the training videos for vision-language world modeling are sourced from two primary domains: 1) Web instruction videos: COIN [@tang2019coinlargescaledatasetcomprehensive], CrossTask [@zhukov2019crosstaskweaklysupervisedlearning], YouCook2 [@zhou2018towards], and a subset of HowTo100M [@miech2019howto100m] videos. These videos cover a diverse range of tasks, and provide clean expert demonstrations. 2) Egocentric recordings: EPIC-KITCHENS-100 [@damen2022rescaling] and EgoExo4D [@grauman2024ego]. These videos feature continuous, uncut recordings in realistic wearable agent scenarios. For all datasets, we collect videos from their training split. While Ego4D [@grauman2022ego4dworld3000hours] is available as large-scale egocentric recordings dataset, we excluded it from training data to avoid potential overlap with benchmarks due to inconsistent train/val splitting. **Generation of Vision-language World Modeling Data.** We use Perception Encoder `PE-G14` [@bolya2025perception] and `PerceptionLM-3B` [@cho2025PerceptionLM] (320$\times$``{=html}320 spatial resolution, 32 frames per input -- can be fit in 32GB V100) to generate the [Tree of captions]{.smallcaps}. We sample up to 5 target window per video according to the tree structure (the first 5 nodes in BFS traversal order), and use `Llama-4 Maverick` (mixture of 128 experts, 17B activated and 400B total parameters, FP8 precision) to extract plans from the window with the sub-tree of captions and two rounds of [Self-Refine]{.smallcaps}. Additional speech transcripts for web videos and the expert commentary in EgoExo4D are provided along with video captions to improve LLM's video understanding during plan extraction. In addition to video-based extraction, we repurposed the NaturalReasoning [@yuan2025naturalreasoning] dataset to world modeling by replacing [Tree of captions]{.smallcaps} with the chain-of-thoughts. Action-state trajectories are extracted by LLM [Self-Refine]{.smallcaps} with similar prompts. ```{=latex} \resizebox{\columnwidth}{!}{% \begin{tabular}{cccrrrr} \toprule \textbf{Domain} & \textbf{Additional Info} & \textbf{Dataset} & \textbf{\# Videos (k)} & \textbf{Duration (hours)} & \textbf{\# Trajectories (k)} & \textbf{\# Steps (k)} \\ \midrule Text-only & N/A & NaturalReasoning & - & - & 1,086.4 & 5,166.2 \\ \midrule Web Instruction Videos & ASR Transcripts & HowTo100M & 167.8 & 18,512.3 & 1,093.2 & 5,438.1 \\ & & COIN & 7.6 & 302.9 & 36.2 & 181.7 \\ & & CrossTask & 2.1 & 163.1 & 10.4 & 55.1 \\ & & YouCook2 & 1.2 & 102.3 & 5.8 & 31.9 \\ \midrule Egocentric Recordings & Expert Commentary & EgoExo4D & 0.6 & 53.5 & 3.1 & 18.8 \\ & N/A & EPIC-KITHCNES-100 & 0.5 & 68.9 & 2.2 & 14.0 \\ \midrule Overall & 179.8 & 19,202.9 & 2,179.6 & 10,604.3 \\ \bottomrule \end{tabular}% } ``` **Training Details**. We use `PerceptionLM-8B` [@cho2025PerceptionLM] to initialize our VLWM. The model is trained with a batch size of 128 and a maximum of 11.5k token context length. We perform uniform sampling of 32 frames in 448$^2$ resolution for visual context inputs. With 12 nodes of 8$\times$H100 GPUs, the training takes approximately 5 days. ### VLWM-critic-1B {#sec:implementation-critic} `\looseness=-1`{=latex} **Data.** We generate paired data according to §`\ref{sec:method-critic}`{=latex} from vision-language world modeling data of HowTo100M and NaturalReasoning. We also include [Tree of Captions]{.smallcaps} data by by sampling subtrees and use root as goal and leafs as trajectories. We also incorporate off-the-shelf preference modeling data to train the critic, where the user queries are treated as goals and model responses are treated as actions. We derive $\langle C_\text{positive}, C_\text{negative} \rangle$ using \<\`\`query" + \`\`chosen" and \`\`query" + \`\`rejected"\>. We include UltraFeedback [@cui2023ultrafeedback], Orca DPO pairs [@OpenOrca], Math-Step-DPO [@lai2024step] as sources of preference data. Lastly, we incorporate training data for learning semantic similarity, where we convert triplets of `` sentences to query as goal, positive sentence as positive action and negative sentence as negative action. This type of data includes MS-MARCO [@bajaj2016ms], SQUAD [@rajpurkar2016squad], HotPotQA [@yang2018hotpotqa], NaturalQuestions [@kwiatkowski2019natural], and FEVER [@thorne2018fever]. **Training Details.** The critic model is initialized from `Llama-3.2-1B` and trained for one epoch with a batch size of 128 (2.7k steps), maximum context length of 1536 tokens using a single node of 8$\times$H100 GPUs. For hyper-parameters in Eq. `\ref{eq:critic-loss}`{=latex}, we set $\lambda$=0.01 and $\texttt{margin}$=1. ## Visual Planning for Assistance (VPA) ### VPA Benchmarks To verify that VLWM's large-scale pre-training yields practical gains in procedural planning, we adopt the Visual Planning for Assistance (VPA) benchmark [@patel2023pretrainedlanguagemodelsvisual]. VPA measures how well a model can predict the next $T$ high-level steps of an ongoing activity given the video history and an explicit textual goal. We follow the standard evaluation horizons $T=3$ and $T=4$. Experiments are conducted on two widely used instructional-video corpora for procedual planning. COIN [@tang2019coinlargescaledatasetcomprehensive] contains 11827 videos spanning 180 tasks, whereas CrossTask [@zhukov2019crosstaskweaklysupervisedlearning] comprises 2750 videos across 18 tasks. We adhere to the official train/val/test splits so results are directly comparable to prior work. We benchmark VLWM against four state-of-the-art planners: `DDN` [@chang2020procedure], `LTA` [@grauman2022ego4dworld3000hours], `VLaMP` [@patel2023pretrainedlanguagemodelsvisual], and `VidAssist` [@islam2024proposeassesssearchharnessing], plus two frequency-based heuristics: `Most-Probable` (global action frequencies) and `Most-Probable w/ Goal` (task-conditioned frequencies). VLWM is fine-tuned on the VPA training splits of COIN and CrossTask using the same hyper-parameters as in pre-training. Following prior work, we report *Success Rate* (SR), *Mean Accuracy* (mAcc), and *Mean IoU* (mIoU) over the predicted step sequence, respectively measuring plan-level accuracy, step-level accuracy, and action proposal accuracy. Table `\ref{table:vpa_benchmark}`{=latex} confirms that VLWM sets a new state-of-the-art on the VPA benchmark. Across both COIN and CrossTask, and at both horizons $T=3$ and $T=4$, our model consistently outperform existing baselines. Compared to VidAssit which adopts a 70B LLM, our VLWM is much smaller (8B) while achieving superior results on 8/12 metrics. Averaged over the four settings, VLWM delivers absolute gains of +3.2% in SR, +3.9% in mAcc, and +2.9 points in mIoU. ```{=latex} \resizebox{\textwidth}{!}{% \begin{tabular}{@{}lc@{\ }c@{\ }cc@{\ }c@{\ }cc@{\ }c@{\ }cc@{\ }c@{\ }c@{}} \toprule \multirow{2}{*}{Model} & COIN T=3 & COIN T=4 & CrossTask T=3 & CrossTask T=4 \\ \cmidrule(l){2-4} \cmidrule(l){5-7} \cmidrule(l){8-10} \cmidrule(l){11-13} & SR & mAcc & mIoU & SR & mAcc & mIoU & SR & mAcc & mIoU & SR & mAcc & mIoU \\ \midrule Most Probable & 1.6 & 4.3 & 6.8 & 1.6 & 8.2 & 15.3 & 1.7 & 6.1 & 9.9 & 1.3 & 5.5 & 13.9 \\ Most Probable w/ goal & 10.9 & 18.0 & 24.9 & 9.1 & 16.3 & 32.2 & 2.4 & 8.9 & 15.5 & 1.5 & 7.9 & 20.5 \\ DDN~\citep{chang2020procedure} & 10.1 & 22.3 & 32.2 & 7.0 & 21.0 & 37.3 & 6.8 & 25.8 & 35.2 & 3.6 & 24.1 & 37.0 \\ LTA~\citep{grauman2022ego4dworld3000hours} & - & - & - & - & - & - & 2.4 & 24.0 & 35.2 & 1.2 & 21.7 & 36.8 \\ VLaMP~\citep{patel2023pretrainedlanguagemodelsvisual} & 18.3 & 39.2 & 56.6 & 9.0 & 35.2 & 54.2 & 10.3 & 35.3 & 44.0 & 4.4 & 31.7 & 43.4 \\ VidAssist~\citep{islam2024proposeassesssearchharnessing} & \underline{21.8} & \underline{44.4} & \underline{64.4} & \underline{13.8} & \underline{38.3} & \underline{66.3} & \underline{12.0} & \textbf{36.7} & \textbf{48.9} & \textbf{7.4} & \underline{31.9} & \textbf{51.6} \\ \midrule % PLM - Finetuned & & & & \underline{18.7} & \underline{45.4} & \underline{72.2} & & & & 7.0 & \textbf{34.9} & 50.5 \\ VLWM-8B (ours) & \textbf{27.9} & \textbf{50.1} & \textbf{69.3} & \textbf{19.4} & \textbf{46.7} & \textbf{74.0} & \textbf{13.5} & \underline{36.4} & \underline{48.3} & \underline{7.2} & \textbf{33.6} & \underline{51.1} \\ \bottomrule \end{tabular} } ``` ### Human Evaluation with [PlannerArena]{.smallcaps} {#sec:plannerarena}

image

Traditional benchmarks for embedded AI assistants generating human-oriented plans are inadequate as they rely on biased or low-quality ground truth data, failing to capture real-world performance and human assistance. To overcome this, we created [PlannerArena]{.smallcaps}, a human evaluation framework inspired by ChatbotArena [@chiang2024chatbotarenaopenplatform]. This Arena/Elo-based system involves human evaluators choosing the better plan from those generated by different anonymous models, pairwise outcomes are converted to Elo scores and model win rates. This approach aligns closely with the actual use case of AI assistants, ensuring the models we develop are not only theoretically sound but also practically valuable in the real world. Our experimental setup includes three dataset (COIN, CrossTask and EgoExo4D), in which we compare VLWM with a search over 20 plans guided by a 8B critic that is minimizing the cost of generated plan (VLWM System-2) and a 8B critic that is maximizing cost, against leading multimodal LLMs and ground truth plans. The pairs are sampled uniformly across every possible battle configuration to have a balanced number of battles across models. The models start with an initial rating of 1000 and we use an Elo K-factor of 32 for the score updates after each battle. Five different annotators participated in the [PlannerArena]{.smallcaps} evaluation evaluating a total of 550 battle pairs, with three annotators running a fixed pilot run of 90 samples to calculate inter-annotator agreement score. Additional details about [PlannerArena]{.smallcaps} can be found in Appendix. We show the final Elo scores of different models in Fig `\ref{fig:planner_arena}`{=latex} as well as the win rate of each model per dataset. VLWM System-2 has the highest Elo by a large margin at 1261, with Llama-4-Maverick being the second most preferred model at an Elo of 1099. Despite using a critic which maximizes cost, the plans generated by VLWM Cost-maximizing (992 elo score) are still generally preferred over the ground truth and plans generated by Qwen2.5 and PerceptionLM, which struggle more to generate meaningful plans given a video context. Importantly, we see that the quality of ground truth is bad overall and has strong variation across datasets. EgoExo4D have higher quality annotations, where the ground truth plans yield the second highest win rate with 69.5% behind VLWM System-2 with 87.9%. However, in COIN and CrossTask, the ground truth plans are barely better than the worst performing models, respectively 43.6% and 42.2%, highlighting an major issue with current procedural planning datasets. ## RoboVQA ```{=latex} \begin{wraptable}{r}{0.35\textwidth} \small \caption{\textbf{RoboVQA} BLEU-1 comparison against VLWM. *: results from \citet{ji2025robobrainunifiedbrainmodel}. $\dagger$: results from \citet{zhou2025physvlmenablingvisuallanguage}.} \label{table:robovqa_benchmark} \resizebox{\linewidth}{!}{% \begin{tabular}{@{}lc@{}} \toprule \textbf{Model} & \textbf{BLEU-1} \\ \midrule PerceptionLM-8B~\citep{cho2025PerceptionLM} & 14.2 \\ Qwen2-VL-7B*~\citep{Qwen2VL} & 33.2 \\ GPT-4V* & 32.2 \\ LLaVA-OV-7B*~\citep{li2024llavaonevisioneasyvisualtask} & 38.1 \\ 3D-VLA-4B$\dagger$~\citep{zhen20243dvla3dvisionlanguageactiongenerative} & 48.3 \\ RoboMamba-3B$\dagger$~\citep{liurobomamba} & 54.9 \\ PhysVLM-3B$\dagger$~\citep{zhou2025physvlmenablingvisuallanguage} & 65.3 \\ ThinkVLA-3B~\citep{huang2025thinkact} & 62.4 \\ ThinkAct~\citep{huang2025thinkact} & 69.1 \\ RoboBrain-7B*~\citep{ji2025robobrainunifiedbrainmodel} & \underline{72.1} \\ \midrule VLWM-8B (ours) & \textbf{74.2} \\ \bottomrule \end{tabular} } \end{wraptable} ``` To further assess VLWM's capabilities in grounded high-level reasoning and planning, we evaluate it on the RoboVQA benchmark (@sermanet2024robovqa). RoboVQA challenges models to perform robotics-focused visual question answering in realistic, multi-embodiment settings, requiring understanding of complex visual scenes and executing coherent action sequences. This benchmark complements the procedural planning evaluations by testing VLWM's ability to guide robotic agents effectively. We follow the standard evaluation protocols of RoboVQA and compare VLWM's performance using BLEU scores. We compare our model against state-of-the-art robotic LLMs: `3D-VLA-4B` [@zhen20243dvla3dvisionlanguageactiongenerative], `RoboMamba-3B` [@liurobomamba], `PhysVLM-3B` [@zhou2025physvlmenablingvisuallanguage], `RoboBrain-7B` [@ji2025robobrainunifiedbrainmodel], ThinkVLA-3B and ThinkAct [@huang2025thinkact]. Table `\ref{table:robovqa_benchmark}`{=latex} demonstrates that VLWM achieves highly competitive performance on the RoboVQA benchmark. Despite not being specialized on robotic data like some of the top-performing models such as RoboBrain, VLWM attains strong BLEU scores across all n-gram levels, ranking within the top two models. Notably, VLWM achieves the highest BLEU-4 score of 55.6, surpassing RoboBrain's 55.1, and closely follows it on BLEU-1 to BLEU-3. These results highlight VLWM's robust generalization and its ability to effectively integrate visual and language information for grounded reasoning and planning in embodied settings. ## Critic Evaluations In this section, we conduct intrinsic evaluations of the critic model independently of VLWM-8B roll-outs to assess whether it exhibits the intended behavior. ### Goal Achievement Detection **Task Definition**. Given a `goal` and a `trajectory` composed of a concatenation of $N_\text{gold}$ steps of reference plan that achieves the goal, and $N_\text{distractor}$ irrelevant steps appended after, the task asks the critic model to independently evaluate costs for every partial progress from the beginning, *i.e.,* $C_1=\texttt{critic(goal, trajectory}[0:1])$, $C_1=\texttt{critic(goal, trajectory}[0:2])$, $\dots$, until $C_{N_\text{gold}+N_\text{distractor}} =\texttt{critic(goal, trajectory}[0:N_\text{base}+N_\text{distractor}])$. Since the distance to the goal should be the lowest after $N_\text{gold}$ steps of reference plan, we calculate the goal achievement detection accuracy according to whether $N_\text{gold}=\arg \min [C_1, ..., C_{N_\text{gold}+N_\text{distractor}}]$. **Datasets**. We construct testing sample from two sources. **1) Vision-language World Modeling (VLWM)**: 4,410 action-state trajectories extracted with [Tree of Captions]{.smallcaps} and [Self-Refine]{.smallcaps}. The `goal` field combines both goal description and goal interpretation. Since VLWM-critic-1B is trained on HowTo100M trajectories, we exclude it and only sample data from other sources of instruction videos (COIN, CrossTask, YouCook2), and egocentric recordings (EgoExo4D, EPIC-KITCHENS-100). **2) Open Grounded Planning (OGP)**: @guo2024open released a collection of planning dataset containing goal-plan pairs sourced from different domains. We only use their \`\`robot" subsets sourced from VirtualHoom and SayCan and WikiHow subset, since plans in the tool usage subset often contain too few number of steps. Different from VLWM data, trajectories in OGP only contain actions, and are OOD for both VLWM-critic-1B and baseline models. There are only 9,983 trajectories in OGP data. **Main Results**. We compare VLWM-critic-1B with Qwen3-Embedding models and Qwen3-Reranker models [@zhang2025qwen3] as baselines, which are state-of-the-art models for measuring semantic similarity. the cost is computed as $C = - \texttt{sim} \langle \texttt{goal, trajectory}\rangle$. Results are shown in Table `\ref{tab:gad-benchmark}`{=latex}. Our VLWM-critic-1B outperform baselines on most subsets by a large margin. VLWM-critic-1B gives 98.4% on VLWM-Instruct while lower 92.7% on VLWM-Ego. This is probably caused by domain gap: our critic is only trained on HowTo100M instruction videos without seeing any egocentric recording data. On OGP, our critic shows clear advantage over the best performing baseline Qwen3-Reranker-8B (72.9% vs 65.6%), but performs comparably with it on OGP-WikiHow (despite having 8$\times$ fewer parameters). Possible reasons of this smaller gap includes data noise or potential overlap between Qwen3-Reranker's training data. In Figure `\ref{fig:cost_curves}`{=latex}, we visualize the normalized cost curves predicted by different critic models. The visualization can be viewed as \`\`energy landscape", and the desired shape is to have the minimum cost at the 100% goal achievement point. On VLWM data, VLWM-ciritc-1B gives a much cleaner landscape compared to baselines. However, when comes to OGP datasets, the distribution becomes nosier. Despite domain gap and dataset noise problem mentioned above, one potential reasoning of performance degradation is the OGP gives action-only trajectory without any explicit world state descriptions, which makes cost evaluation harder. ```{=latex} \resizebox{\linewidth}{!}{% \begin{tabular}{lcccccc} \toprule \multirow{2}{*}{\textbf{Model}} & \textbf{VLWM} & \textbf{OGP} \\ \cmidrule(lr){2-4}\cmidrule(lr){5-7} & \textbf{Instruct} & \textbf{Ego} & \textbf{Overall} & \textbf{Robot} & \textbf{WikiHow} & \textbf{Overall} \\ \midrule Chance Performance & 8.9 & 8.6 & 8.8 & 12.3 & 12.2 & 12.1 \\\midrule Qwen3-Embedding-0.6B & 65.0 & 55.2 & 62.5 & 30.9 & 22.6 & 22.7 \\ Qwen3-Embedding-4B & 61.1 & 54.6 & 59.4 & 32.1 & 25.0 & 24.7 \\ Qwen3-Embedding-8B & 67.8 & 62.3 & 66.2 & 29.4 & 36.4 & 35.4 \\\midrule Qwen3-Reranker-0.6B & 59.0 & 55.2 & 57.9 & 43.4 & 34.6 & 35.1 \\ Qwen3-Reranker-4B & 55.8 & 46.1 & 53.8 & 55.7 & 33.7 & 34.5 \\ Qwen3-Reranker-8B & 68.0 & 65.4 & 67.3 & 65.6 & 48.3 & 49.3 \\ \midrule \textbf{VLWM-critic-1B} & \textbf{98.4} & \textbf{92.7} & \textbf{96.9} & \textbf{72.9} & \textbf{48.3} & \textbf{50.0} \\ \bottomrule \end{tabular}% } ``` ```{=latex} \hfill ``` ```{=latex} \resizebox{\linewidth}{!}{% \begin{tabular}{ccccc} \toprule \textbf{Dataset} & \textbf{Default} & \textbf{w/o Interp.} & \textbf{w/o states} \\ \midrule Instruct & COIN & 97.1 & 96.4 (-0.7) & 91.4 (-5.7) \\ & CrossTask & 98.8 & 98.5 (-0.3) & 92.9 (-5.9) \\ & YouCook2 & 99.2 & 99.1 (-0.1) & 94.5 (-4.7) \\ \midrule Ego & EgoExo4D & 95.2 & 94.0 (-1.2) & 82.2 (-13.0) \\ & EK-100 & 90.1 & 88.6 (-1.5) & 64.0 (-26.1) \\ \midrule \textbf{Overall} & \textbf{96.9} & \textbf{96.3 (-0.6)} & \textbf{88.1 (-8.8)} \\ \bottomrule \end{tabular}% } ``` ![**Cost curves estimated by different critic models**. Each plot visualizes 3k cost curves on goal achievement detection trajectories, where each trajectory is composed of a reference gold plan (0%-100%) and distractor steps (100%-200%). Red dots ([$\boldsymbol{\cdot}$]{style="color: red"}) mark cost-minimizing steps (detected goal achievement points). VLWM-Critic accurately detects goal completion around 100% plan length, while baselines show suboptimal or noisy behavior.](figures/cost_curves.png){#fig:cost_curves width="1\\linewidth"} **Ablation Studies**. Table `\ref{tab:critic-ablation}`{=latex} provides an ablation of critic input representation using VLWM-critic-1B and the VLWM data. We tried to remove the goal interpretations which contains descriptions of current and expected final goal state, and state descriptions from the trajectory representation and leave actions only. We see both ablation leads to performance reduction on goal achievement detection, and the reduction on unseen OOD data (the Ego subset) is more severe, showing the importance of interpretation and world state description for effective generalization. ### Procedural Planning on [WorldPrediction-PP]{.smallcaps} ```{=latex} \begin{wrapfigure}{r}{0.35\textwidth} \includegraphics[width=\linewidth]{figures/critic-worldprediction.png} \caption{\textbf{\textsc{WorldPrediction-PP} results}. Our VLWM-critic-1B established a new SoTA of 45.4\% accuracy.} \label{fig:ritic-worldprediction} \end{wrapfigure} ``` The [WorldPrediction]{.smallcaps} benchmark [@chen2025worldprediction] is designed to evaluate high-level world modeling and procedural planning capabilities. Its procedural planning subset, [WorldPrediction-PP]{.smallcaps}, comprises 570 human verified samples. Each test case provides initial and final visual states alongside four candidate action plans, represented by video sequences. The task is to identify the correctly ordered sequence among shuffled counterfactual distractors, emphasizing the capability of goal-conditioned planning as well as models' understanding of semantic and temporal action order. `\looseness=-1`{=latex} To evaluate our critic modules on [WorldPrediction-PP]{.smallcaps}, we followed the evaluation protocol for Socratic LLMs in [@chen2025worldprediction]. Visual inputs were first converted into textual descriptions using captions generated by Qwen2.5-VL. Specifically, two images depicting initial and final states produced a goal description outlining the changes of world states, and video clips of candidate actions were similarly captioned. These textual inputs were provided directly to our VLWM-critic models to compute costs for each candidate plan, selecting the option with the lowest predicted cost. In Figure `\ref{fig:ritic-worldprediction}`{=latex} (b), we compare our VLWM-critic models against baseline Socratic LLMs. Our models achieve a Pareto-optimal balance of model size and accuracy. Importantly, this evaluation constitutes a zero-shot scenario for VLWM-critic models, as neither the change captioning-based goal descriptions nor the detailed video captions as action steps were part of the training corpus. # Related Work {#sec:related_work} ## Action Planning Planning is the task to generate a sequence of actions that can transit the world from initial state to a desired goal state. Our VLWM focuses on planning *high-level* actions, which is characterized by semantic and temporal abstraction [@SuttonPS99; @chen2025worldprediction], as opposed to the low-level, high-frequency continuous actions in autonomous driving [@teng2023motion], robotics [@yu2020meta], and games [@mnih2015human; @brockman2016openai], etc. Below, we compare existing methodologies for action planning. **Imitation learning** (also known as behavior cloning) is effective when extensive expert demonstrations are available [@torabi2019recent; @baker2022video]. However, it becomes considerably more challenging when demonstrations are scarce or imperfect [@wu2019imitation; @sagheb2025counterfactual]. For procedural planning and VPA tasks based on instructional videos [@chang2020procedure; @patel2023pretrainedlanguagemodelsvisual], most approaches rely fundamentally on behavior cloning. Since the action annotations [@tang2019coinlargescaledatasetcomprehensive; @zhukov2019crosstaskweaklysupervisedlearning] are confined to limited vocabularies, the ground truth plans are frequently incomplete, making them not only suboptimal reference for benchmarking (which motivates our PlannerArena in §`\ref{sec:plannerarena}`{=latex}), but also inadequate for imitation learning. **Reinforcement learning** typically requires environments where agents can perform trial-and-error and receive explicit rewards. When environments support such interactions, reinforcement learning verifiable rewards (RLVR) is highly effective [@deepseekai2025deepseekr1incentivizingreasoningcapability]. Although RL is well-suited for domains where constructing simulation environments is viable, scaling RL to more diverse and complex domains is less feasible. **Planning with reward-agnostic world model**. This approach exhibits superior generalization by learning from extensive, reward-free offline data [@sobal2025learning; @zhou2025dinowmworldmodelspretrained; @assran2025vjepa2]. World models enable planning by simulating action outcomes internally and optimizing plans based on cost minimization. Unlike methods that predict task-specific rewards (*i.e.,* model-based RL [@hafner2024masteringdiversedomainsworld]), here world models only predict future world states [@ha2018world], and action plans are optimized by minimizing the distance between the predicted resulting state and the desired goal state [@lecun2022path]. It allows inference-time scaling by conducting internal trial-and-error within the learned world model. Our VLWM's system 2 \`\`planning with reasoning" leverages this paradigm, and we proved that it outperforms reactive system-1 behavior cloning. ## World Modeling World models aim to simulate environmental dynamics, enabling agents to optimize the plan without direct online interaction with the real environment. They have demonstrated success primarily in low-level control domains, such as autonomous driving [@wang2024driving; @gao2024vista; @li2024think2drive] and robotics [@zhou2025dinowmworldmodelspretrained], where models predict fine-grained, continuous sensory data over short horizons. Below, we compare existing world modeling approaches. **Generative world models** typically utilize powerful diffusion-based architectures to reconstruct future observations directly (*e.g.,* in pixel space). Examples include Sora [@videoworldsimulators2024], Cosmos [@agarwal2025cosmos], Genie [@bruce2024genie; @parkerholder2024genie2] and UniSim [@yang2024learning], and recent multimodal chain-of-thought reasoning *i.e.,* \`\`thinking with images" models [@su2025thinking]. While intuitive, generative models inherently suffer from computational inefficiency and task-irrelevant details entangled in pixel-based representations, severely limiting their scalability for long-horizon planning. While these models generate realistic visuals, they have shown limited success in planning tasks. **JEPA world models** encode observations into compact abstract representations, with a predictor trained to forecast these latent states. JEPA models have proven beneficial in representation learning, demonstrated by I-JEPA [@assran2023self], IWM [@garrido2024learning], and V-JEPA [@bardes2024revisiting], and have facilitated MPC-based planning, exemplified by DINO-WM [@zhou2025dinowmworldmodelspretrained], V-JEPA2 [@assran2025vjepa2], and NWM [@bar2025navigation]. However, joint training of encoders and predictors poses challenges, notably the need for anti-collapse techniques such as EMA. Moreover, existing JEPA-based world models predominantly focus on low-level motion planning, and extending them to high-level action planning remains an open research challenge. **Language-based world models** exploit natural language as a high-level abstraction interface, offering interpretability and computational advantages over pixel-based reconstruction. Prior work has explored prompting LLMs as world models [@hao2023reasoning; @tang2024worldcoder; @wang2024can; @gu2024your] or training language-based world model in narrowed domains, such as web navigation [@chae2024web], text games [@lin2023learning; @wu2025rlvr], and in embodied environment [@wang2025world]. In contrast, our VLWM approach explicitly learns a world model directly from large-scale raw video data. # Conclusion {#sec:conclusion} In this work, we introduced the Vision Language World Model (VLWM), a foundation model that learns to represent and predict world dynamics directly in language space, enabling interpretable and efficient high-level planning. By compressing raw videos into hierarchical Trees of Captions and refining them into structured trajectories of goals, actions, and world state changes, VLWM bridges the gap between perception-driven VLMs and reasoning-oriented LLMs. Its dual-mode design supports both fast, reactive System-1 planning through direct policy decoding and reflective System-2 planning via cost minimization guided by a self-supervised critic, which allows the model to internally perform trial-and-error reasoning and select optimal plans. Trained on a large and diverse corpus of instructional and egocentric videos, VLWM establishes new state-of-the-art results on the Visual Planning for Assistance benchmark, demonstrates superior plan quality in PlannerArena human preference evaluations, and achieves top-tier performance on RoboVQA, all while producing interpretable action-state rollouts. Furthermore, the critic model independently excels in goal achievement detection and procedural planning benchmarks, highlighting the value of explicit semantic cost modeling for world-model-based reasoning. Taken together, these contributions show that by learning directly from large-scale natural videos and predicting in abstract, non-generative representation spaces rather than raw pixels, Vision Language World Model (VLWM) can provide a powerful interface for bridging perception, reasoning, and planning, pushing AI assistants beyond imitation toward reflective agents capable of robust, long-horizon decision making. ```{=latex} \clearpage ``` ```{=latex} \newpage ``` ```{=latex} \bibliographystyle{assets/plainnat} ``` ```{=latex} \clearpage ``` ```{=latex} \newpage ``` ```{=latex} \beginappendix ``` # PlannerArena Details ## Instructions & data To evaluate model-generated plans, we conducted a controlled human evaluation study using a custom-built streamlit application. Annotators were presented with (i) a short video context, (ii) a textual goal (e.g., Make a fish curry), and (iii) two alternative plans generated by different anonymous models. The task is to select the preferred plan to achieve the goal given the provided video context. The instruction shown to annotators is: ```{=latex} \begin{tcolorbox}[colback=gray!10!white, colframe=gray!80!black, title=PlannerArena Instruction]\textit{You will see a short video that sets the context, then see a goal sentence. Two alternative plans (Plan A and Plan B) generated by a model are shown. Your job is to select the plan you would prefer to follow in order to achieve the stated goal within the given video context.} \end{tcolorbox} ``` The evaluation setup is based on three datasets commonly used for procedural video planning and understanding: COIN, CrossTask, and EgoExo4D. For all datasets, the video context given to the annotators is the entire original video truncated right before the start of the first annotated step in order to prevent models from leveraging future visual information in their plan. This is similar to the Visual Planning for human Assistance (VPA) setup, but in order to evaluate human plan preference. For EgoExo4D, the exo point of view is given as video context to prevent any partial observation problems. We generate candidate plans with the other VLMs with zero-shot prompting, all models are provided with the same video context and were prompted with the following template: `\noindent`{=latex} ------------------------------------------------------------------------ {language=Markdown} You are provided with a context segment of a procedural video about {goal_formatted}. Generate the remaining actions (steps) to take from that context segment in order to reach the goal. The plan should be composed of high-level descriptions starting with a verb, and it should be clear and concise, including all essential information. There is no need to be overly descriptive. Generate only the action steps. `\noindent`{=latex} ------------------------------------------------------------------------ ## Pairs sampling & IAA Unlike ChatbotArena which relies on an Elo-based sampling method to balance the evaluation across a large number of models, we adopt a uniform uniform sampling strategy as we only have six models to compare. Specifically, we first sample an equal number of battle pairs from each dataset, then enforce balanced participation across models such that each model competed equally against others within each dataset. A "setup" is defined as a (dataset, model pair) combination, and each setup is represented equally in the sample pool, yielding 3500 unique battle setups for PlannerArena. Five annotators participated in the study. Prior to annotation, they completed a short warm-up consisting of five solved examples to familiarize themselves with the task. Inter-annotator agreement is computed over a shared subset of 100 samples with three annotators: the Fleiss' $K$ was 0.63, indicating substantial agreement, with a raw agreement percentage of 72.22%. ## Example ![PlannerArena interface. The sample shown here is from COIN, Plan A from the ground truth annotations and Plan B from Llama 4.](figures/planner_arena_eval.png){#fig:planner_arena_setup width="\\linewidth"} ```{=latex} \clearpage ``` ```{=latex} \newpage ``` # Prompts ## Meta Prompt for LLM [Self-Refine]{.smallcaps} `\noindent`{=latex} ------------------------------------------------------------------------ {language=Markdown} {TREE OF CAPTIONS} {ADDITIONAL VIDEO INFO} # Draft Here is a draft for structured data extraction: {PREVISOUS DRAFT} --- # Your task You carefully examine the draft above and identify problems. The requirements are listed below. Go through each point one by one and discuss aspects in the draft that doesn't meet the requirements. Be specific and constructive, avoid vague and generic comments or simply repeating the requirements. Quote the draft for detailed discussion. Provide concrete points of explicit actionable revisions that could help improve and enrich the draft. Make sure your revisions and the added information is grounded in the provided content. After extensive analysis and discussion, give the revision in the desired format. {REQUIREMENTS OF PALN EXTRACTION} # Output format ```yaml discussion: |- Free form chain-of-thought reasoning: analyze the draft, identify problems, and suggest actionable revisions or enrichments. plan: - action: state: |- start: xx.xx # float between and round to two decimal digits end: xx.xx # float between and round to two decimal digits - action: state: |- ... goal: interpretation: |- ``` Start your response with "```yaml\n..." and end with "\n```" ## Requirements of Plan Extraction for LLM [Self-Refine]{.smallcaps} `\noindent`{=latex} ------------------------------------------------------------------------ **Action Plan** 1. Identify a sequence of physical actions that meaningfully advance the task progress; Omit vague, redundant, or purely presentational steps. 2. Each action is one informative imperative sentence said from the actor's perspective. Avoid describing actions from the tutor's or demonstrator's voice. 3. Infer the span of each action according the provided timestamps. They must fall within and and do not overlap with each other. 4. Be selective - time in the video may be non-linear. For example, the final result may appear at the beginning of the video. Such actions should be skipped. **World State** 1. Explain how the action is performed according to the provided captions. Use imperative voice and instructional or tutoral style. 2. Provide elaborated discussion of the motivation, rationale, and purpose behind the action. 3. Discuss all relevant objects (can be both physical object or abstract concept, or the actor itself) whose states are changed by the action. 4. Cover various aspects, such as status, position, condition, temperature, etc. Highlight the causal relationship between actions and states. 5. Be logically coherent and semantically connected with neighboring steps. They are autoregressive and shouldn't conflict with each other. 6. Provide in-depth analysis. Perspectives may include (but are not limited to): * Implications of state changes * What the change enables; whether it is (or is not) ready for future steps * Whether and how the change advances or contributes to the overall goal * Whether it satisfy the desired final state for the activity, if not what is still required 7. Organize the discussion into a single coherent paragraph, it should be comprehensive and detailed, but also avoid redundency and ensure readibility. **Goal Identification** 1. Summarize the overall achievements by the actions during to (not the entire video). 2. Ensure comprehensive coverage. Feel free to use multiple sentences if appropriate. 3. Use imperative voice. But it should not be a simple concatenation of individual action names. 4. It summarize WHAT is achived (e.g., aggregation and abstraction of state changes) but not HOW it is achieved (e.g., "do x by doing y"). **Goal Interpretation** 1. Infer and describe the initial state of the environment before any action is taken. Only describe task-relevant aspects. Start with "Now, ..." 2. Interpret the goal in detail by discussing objects needs to be what state such that goal can be considered achieved. 3. Start with "To achieve the goal, ...". You can also include related technical specifications if applicable. 4. Description of the desired world state should be grounded in the provided context and aggregate all the state changes caused by the actions. 5. Discuss all objects, tools, materials, dependencies, etc. needed or invovled in the action steps and explain the functional rationale. 6. Use the tone as if you are now at the starting time of the video () and tasked to plan towards the given goal. You are preparing by thinking and analyzing the task. 7. Provide one paragraph and ensure its coherence and readibility. Importantly, you should avoid the leakage of any action plan information in this section. **Overall Requirements** 1. Maintain faithfulness to the provided video content; Do not hallucinate or infer based on commonsense knowledge. 2. The output must strictly follow the given YAML format. Timestamps should be in the same format as and . 3. Except for the start and end times of the action, don't mention exact timestamp anywhere in your output. 4. Don't use 'the video' / 'the segment' in any part of the output. Instead, refer to the actions, objects, and environements directly. 5. Use specific functional description when referring to objects. Ignore task-irrelevant information such as appearance which does not affect the task. 6. Ensure comprehensiveness and detail in your output, but also conherence and readibility. Avoid repetition and redundancy. ```{=latex} \newpage ``` # Tree-of-Captions Example ![image](figures/tree-of-caption-video.png){width="1\\linewidth"} ![**Structure of [Tree of captions]{.smallcaps} (bottom) extracted from video (top)**. Each box is associated with a corresponding video caption.](figures/tree_example.png){#fig:enter-label width="1\\linewidth"} **Tree-of-Captions formate by depth-first search (DFS)**: {language=Markdown} # 0.00s -> 164.53s (duration: 164.5s) The video features a view of a man repairing a bicycle tire and tube. The man is wearing black gloves, and there is a bicycle lift holding a blue bike in the background. In the background is another person wearing a gray shirt. A black tool chest and a wooden tool bench can also be seen ... ## Segment 1 - 0.00s -> 126.20s (duration: 126.2s) This video features a man showing a second man how to repair a tire. The second man stands center screen in a workshop. The man holding the tire is center screen and behind him is a large workbench. In the background are several other objects of various sizes and shapes ... ### Segment 1.1 - 0.00s -> 83.40s (duration: 83.4s) The video features a first-person view of someone in a well-lit workshop. The camera wearer is constantly seen touching objects in their environment while working with a tire and a tube, which is later put on a bicycle. The lighting is bright and focused. In the first shot, the camera wearer holds a tire and a tube and feels the thread with their hands, before passing the tire to the left hand ... #### Segment 1.1.1 - 0.00s -> 53.13s (duration: 53.1s) This video shows a busy workshop where two men are busy with their work. The man in this video removes an inner tube from a bicycle wheel using a tire lever, then peels off the tube using the thumb and fingers of his right hand. He then puts the removed tube back on the table ... **0.00s -> 23.27s (duration: 23.3s)**: The video features a person working on a bicycle tire inside a bike shop. They wear black gloves and pull a blue bicycle towards them. They take out a tire lever to remove the tire from the wheel, and the focus is entirely on the cycle maintenance process ... **23.27s -> 53.13s (duration: 29.9s)**: The video is set inside a workshop, where a bicycle is clearly visible at the left side of the frame. The person's point of view is through a stere Pipeline video camera mounted on a fishing rod, which caps off the end of his arm, so the video looks like it is shot through a fish eye lens ... #### Segment 1.1.2 - 53.13s -> 83.40s (duration: 30.3s) The video shows a ... ```{=latex} \newpage ``` ```{=latex} \twocolumn ``` # VLWM Planning Examples ## Full Trajectory {#app:vlwm_trajectory} {html} Cooking Tomato and Eggs ![image](figures/appendix_context_frames_tomato_eggs.png){width="1\\linewidth"} {html} Now, the kitchen is set up with necessary ingredients and tools. To achieve the goal, the eggs need to be cooked and mixed with tomatoes, and the mixture should be seasoned appropriately. The eggs should be whisked thoroughly to achieve a uniform texture. The tomatoes should be heated until they are soft and well coated with the eggs. The dish should be seasoned with salt and pepper to taste. The final state of the dish should be a well-cooked, flavorful mixture of eggs and tomatoes, ready for serving. The kitchen environment and the tools used, such as the stove, pan, and utensils, are crucial for achieving this goal. The ingredients, including eggs, tomatoes, oil, salt, and pepper, are essential for creating the desired dish. --- Preheat the skillet on the stove To preheat the skillet, turn on the stove to medium-high heat and wait for the skillet to heat up. The skillet is garnished with a little bit of oil to prevent the eggs from sticking. This action prepares the skillet for cooking the eggs by increasing its temperature. The state of the skillet changes from cold to hot, making it ready for cooking. The oil used for preheating prevents the eggs from sticking to the skillet, ensuring they cook evenly. --- Crack eggs into a bowl and whisk them together Crack eggs into a bowl by breaking the eggshells over the edge of the bowl and pouring the contents into the bowl. Use a spatula to mix the eggs in a circular motion, being careful not to over-beate them. This action changes the state of the eggs from whole to a mixed liquid, making them ready for cooking. The eggs are now in a state that can be seasoned and cooked. The whisking action ensures the eggs are well mixed, contributing to a uniform texture in the final dish. --- Season the eggs with salt Add salt to the eggs and mix well using a spatula. This action seasons the eggs, enhancing their flavor. The state of the eggs changes from unseasoned to seasoned, making them ready for cooking. The seasoning adds flavor to the eggs, which is crucial for the overall taste of the dish. --- Cook the eggs in the pan Pour the egg mixture into the preheated pan with oil. Use a spatula to stir the eggs, breaking them into curds. Continue cooking until the eggs are almost set. This action changes the state of the eggs from liquid to a partially cooked state. The eggs are now partially cooked and are being transformed into curds. The heat from the pan cooks the eggs, and the stirring action ensures even cooking. --- Remove the cooked eggs from the pan Use a spatula to scrape the cooked eggs out of the pan and onto a plate. This action removes the eggs from the pan, making way for cooking other ingredients. The state of the eggs changes from being in the pan to being on a plate. The eggs are now removed from the pan, allowing for the addition of other ingredients to be cooked. --- Cook the tomatoes in the pan Add tomatoes to the pan and stir them using a spatula. Continue cooking until the tomatoes are soft. This action changes the state of the tomatoes from raw to cooked, making them ready for mixing with the eggs. The tomatoes are now cooked and can be combined with the eggs. The heat from the pan cooks the tomatoes, and the stirring action ensures even cooking. --- Add the cooked eggs back into the pan with the tomatoes Add the cooked eggs back into the pan with the tomatoes. Use a spatula to stir the mixture, combining the eggs and tomatoes. This action mixes the eggs and tomatoes together, creating a uniform dish. The state of the mixture changes from separate components to a combined dish. The eggs and tomatoes are now mixed together, creating a cohesive and ready-to-serve dish. --- Season the dish with salt and pepper Add salt and pepper to the dish and stir well using a spatula. This action seasons the dish, enhancing its flavor. The state of the dish changes from unseasoned to seasoned, making it ready for serving. The seasoning adds flavor to the dish, ensuring it is palatable. --- Transfer the dish to a serving plate Use a spatula to transfer the cooked dish from the pan to a serving plate. This action changes the state of the dish from being in the pan to being on a plate, making it ready for serving. The dish is now presented in a serving plate, ready to be consumed. --- Present the final dish The final dish is presented on a white surface. The dish is now ready to be served. The state of the dish changes from being in the cooking vessel to being on a serving plate. --- ## Cost Min/maximizing Plans {language=Markdown} ### Goal: Make Beef Tacos Cost Minimizing Plan (cost = -4.86): 1. Saute chopped onions and garlic in oil 2. Add ground beef to the pan and break it up 3. Add seasoning to the beef mixture 4. Add tomato sauce to the beef mixture and stir 5. Prepare taco shells 6. Fill taco shells with the beef mixture 7. Add cheese to the tacos 8. Serve the tacos Cost Maximizing Plan (cost = 3.03): 1. Add959 existing ingredients to the slow cooker 2. Mix the ingredients in the slow cooker 3. Add taco seasoning to the slow cooker 4. Stir the contents of the slow cooker 5. Open the slow cooker lid 6. Serve the cooked beef mixture onto a tortilla 7. Add cheese to the taco 8. Fold the tortilla ### Goal: Make Kerala Fish Curry Cost Minimizing Plan (cost = -4.36): 1. Dry roast the chilies in a pan over medium heat 2. Add ginger and garlic to the pan and mix with the chilies 3. Add water and tamarind paste to the pan and mix 4. Add salt and sugar to the curry and stir 5. Add fish to the curry and cook until done 6. Add water to the curry and stir 7. Simmer the curry and stir occasionally 8. Garnish with cilantro 9. Serve the curry Cost Maximizing Plan (cost = -0.04): 1. Dry red chili 2. Add mustard seeds 3. Saute the onions 4. Add turmeric powder 5. AddTransition metal pieces 6. Add significance 7. Add curry leaves 8. Add chili powder 9. Add coriander powder 10. Add fish 11. Add salt 12. Add tamarind juice 13. Finish with coconut oil ### Goal: Use Tapping Gun Cost Minimizing Plan (cost = -7.58): 1. Insert the gun head into the Tapping gun 2. Remove the existing gun head from the Tapping gun 3. Handle the Tapping gun to prepare it for a new gun head 4. Clean the Tapping gun 5. Prepare the Tapping gun for use by loading it with a new gun head 6. Load the Tapping gun with a new gun head 7. Demonstrate the final preparation of the Tapping gun 8. Emphasize the completion of the task 9. Conclude the demonstration 10. Fill the Tapping gun with disposable scoops Cost Maximizing Plan (cost = 3.46): 1. Ignited the lighter to produce a flame 2. Insert the gun head into the tag brand 3. Load the Tapping gun with degrees 4. Insert the Tapping gun into the red compartment 5. Finish up by Aer 6. Aer.YEAR4 doesn't come off easily ### Goal: Replace SIM Card Cost Minimizing Plan (cost = -5.43): 1. Eject the SIM card tray using a SIM card ejector tool 2. Remove the SIM card from the tray 3. Place the SIM card into the tray 4. Reinsert the SIM card tray into the phone 5. Ensure the SIM card is properly seated and the tray is fully inserted 6. Adjust and finalize the SIM card tray insertion 7. Verify the SIM card tray is fully inserted and functional Cost Maximizing Plan (cost = 4.24): 1. Use the SIM card ejector tool to eject the SIM card tray 2. Remove the SIM card tray from the phone 3. Place the SIM card into the SIM card tray 4. Insert the SIM card tray back into the phone 5. Place the phone on a flat surface 6. Display the final result or a related message ### Goal: Use Rice Cooker To Cook Rice Cost Minimizing Plan (cost = -4.31): 1. Take out some rice 2. Wash the rice 3. Put the washed rice into the rice cooker 4. Measure the water level 5. Cook the rice using the rice cooker 6. Open the rice cooker and fluff the cooked rice 7. Serve the cooked rice Cost Maximizing Plan (cost = 3.91): 1. Take out some rice 2. Rinse the rice 3. Place the strainer over the rice cooker 4. Add water to the rice cooker 5. Start the rice cooker 6. Analytical balance assessment 7. Open the rice cooker to check the cooked rice 8. Scoop the cooked rice from the rice cooker ### Goal: Make Chinese Spring Rolls Cost Minimizing Plan (cost = -4.44): 1. Pour oil into a wok and heat it 2. Add chopped garlic and ginger to the heated oil in the wok 3. Cook chopped vegetables such as carrots and onions in the wok 4. Add noodles and various sauces to the wok and stir-fry 5. Prepare spring roll sheets and fill them with the cooked mixture 6. Fry the filled spring rolls in hot oil until they are golden brown 7. Serve the fried spring rolls on a plate Cost Maximizing Plan (cost = -0.59): 1. Pour vegetable oil into a wok 2. Add chopped garlic and ginger to the wok 3. Stir-fry chopped onions 4. Add chopped carrots and capsicum to the wok 5. Add noodles and purified water to the wok 6. Drain the cooked noodles 7. Mix the cooked noodles with chopped parsley, salt, and pepper 8. Prepare the cooking vessel for frying