Motion Attribution for Video Generation

Source

• Raw Markdown: paper_motive-2026.md
• PDF: paper_motive-2026.pdf
• Preprint: arXiv 2601.08828v2
• ICML record: OpenReview forum
• Official project page: NVIDIA Spatial Intelligence Lab
• Award record: ICML 2026 awards
• Trigger post: Gonzo ML 5711; local snapshot: papers/motive-2026/telegram-post-gonzo-ml-5711.md.
• Long-form trigger thread: Gonzo ML Podcasts 4404; local 4404–4407 snapshot: papers/motive-2026/telegram-thread-gonzo-ml-podcasts-4404-4407.md.
• Author narrative: Xindi Wu’s 1/10–10/10 X thread; local authenticated-API snapshots: papers/motive-2026/x-thread-cindy-x-wu-2013703939448074481.md and .json.
• Official code: no code repository is linked from the paper or official project page as of 2026-07-17.

Status And Credibility

The ingested artifact is arXiv v2, dated 2026-07-03 in the rendered paper. The official NVIDIA project page identifies the paper as an ICML 2026 Oral, and the ICML program chairs list it as an Outstanding Paper Honorable Mention. This is current, peer-reviewed work from a credible NVIDIA–Princeton–MIT–Michigan–Toronto–Vector Institute team, with evidence across three video-generation backbones and two candidate video corpora. The award and official project page make it high-confidence evidence for query-conditioned video-data attribution.

The main reproducibility caveat is artifact availability: the paper documents hyperparameters and runtime, but no official code link is visible. The quantitative tables also do not report uncertainty across random seeds.

Core Claim

Motive asks a targeted data question:

Given one or more query clips that specify desired temporal dynamics, which candidate fine-tuning clips most influence a video generator’s ability to reproduce those dynamics?

It answers with motion-weighted gradient similarity. Optical-flow masks suppress static appearance, a fixed diffusion timestep and shared noise reduce estimator cost and variance, and a structured projection compresses per-example gradients. Ranking candidate clips by their projected motion-gradient similarity yields a query-conditioned fine-tuning subset.

The strongest result is deliberately narrower than “less data is always better”: for ten preselected motion categories, Motive’s top 10% subsets improve the target motion metrics relative to random 10% subsets and, on dynamic degree, relative to the tested full-corpus fine-tuning baselines.

Method

For a candidate clip $v$ with conditioning $c$, Motive derives an optical-flow magnitude mask $W_v(f,h,w)$, downsamples it to the model’s latent grid, and reweights the per-location diffusion or flow-matching error:

$${\mathcal{L}}_{\mathrm{mot}}(\theta ;v,c)=\frac{1}{F_v}{\mathrm{mean}}_{f,\tilde{h},\tilde{w}}\left[{\tilde{W}}_{v,c}(f,\tilde{h},\tilde{w}){\tilde{\mathcal{L}}}_{\theta ,v,c}(f,\tilde{h},\tilde{w})\right].$$

It then computes a per-example motion gradient at one fixed timestep and shared noise draw, projects it to $D^{\prime }=512$ dimensions with a Fastfood Johnson–Lindenstrauss transform, and unit-normalizes it:

$${\tilde{g}}_{\mathrm{mot}}(\theta ,v)=\frac{P{\nabla }_{\theta }{\mathcal{L}}_{\mathrm{mot}}(\theta ;v,c)}{{\Vert P{\nabla }_{\theta }{\mathcal{L}}_{\mathrm{mot}}(\theta ;v,c)\Vert }_2}.$$

The influence score for training clip $v_n$ and query clip $\hat{v}$ is their projected-gradient inner product:

$$I_{\mathrm{mot}}(v_n,\hat{v})={\tilde{g}}_{\mathrm{mot}}(\theta ,v_n{)}^{\top }{\tilde{g}}_{\mathrm{mot}}(\theta ,\hat{v}).$$

For multiple queries, each query votes for candidates above a percentile threshold; the top-$K$ vote totals form the selected subset.

flowchart LR
  V["candidate video clip"] --> Flow["AllTracker displacement"]
  Flow --> Mask["motion mask on latent loss"]
  Mask --> Grad["one motion-weighted gradient"]
  Grad --> Proj["Fastfood projection, 512-D"]
  Q["query motion clips"] --> QGrad["projected query gradients"]
  Proj --> Sim["gradient cosine influence"]
  QGrad --> Sim
  Sim --> Vote["percentile votes across queries"]
  Vote --> Top["top 10% fine-tuning subset"]

Experimental Contract

• Candidate data: 10,000 clips sampled from each of VIDGEN-1M and 4DNeX-10M. The paper’s “100%” baseline means this 10,000-clip experimental pool, not either complete source corpus.
• Models: Wan2.1-T2V-1.3B, Wan2.2-TI2V-5B, and LTX-2B.
• Query set: 50 manually screened Veo-3 clips, five each for compress, bounce, roll, explode, float, free fall, slide, spin, stretch, and swing.
• Test set: the same ten motion categories with different visual appearances.
• Selection: top 10% of the candidate corpus; multi-query results aggregate query votes.
• Fine-tuning: DiT backbone only, with text encoder and VAE frozen; one epoch with the active fine-tuning dataset repeated 50 times.
• Evaluation: VBench motion and appearance metrics plus pairwise judgments from 17 participants over 50 generated videos.

Evidence And Results

Backbone	Dynamic degree: base	Full fine-tuning on 10,000	Random 1,000	Motive 1,000
Wan2.1-T2V-1.3B	39.6	42.0	41.3	47.6
Wan2.2-TI2V-5B	42.0	45.3	41.6	48.3
LTX-2B	36.5	38.9	37.8	45.1

Additional evidence:

• Human raters preferred Motive over the base model in 74.1% of pairwise judgments and over full fine-tuning in 53.1%; ties were 12.3% and 14.8%, respectively.
• Motion masking matters: on Wan2.1, dynamic degree rises from 43.8 without motion masking to 47.6 with it.
• The selected top 10% is not merely a highest-motion filter: its mean motion magnitude is 3.85 versus 3.69 for the bottom 10%.
• A 512-dimensional projection preserves the full-gradient ranking with Spearman $\rho =74.7\%$; 1,024 and 2,048 dimensions reach only 75.7% and 76.1%.
• One fixed timestep reaches $\rho =66\%$ against a ten-timestep reference. This makes the method much cheaper, but also quantifies approximation error.
• Cross-category overlap is similar across the two candidate corpora (mean 24.0% and 24.3%), suggesting some motion primitives share useful examples while others need specialized subsets.

What The “10% Beats 100%” Result Does And Does Not Mean

The headline is valid only as a comparison between fine-tuning arms. Starting from the same pretrained video generator, Motive fine-tuning on 1,000 selected clips beats fine-tuning on all 10,000 clips in the experimental candidate pool on dynamic degree for all three tested backbones and wins a narrow majority of human comparisons against that full-pool fine-tuning baseline.

The data accounting is:

Stage	Data processed	What the percentage means
Original model pretraining	Not rerun by this paper	Motive does not compare 10% versus 100% pretraining corpora.
Motive attribution	All 10,000 candidate clips	Every candidate receives a per-example gradient before ranking.
Motive fine-tuning	Selected 1,000 clips	This is the reported 10% subset.
Full fine-tuning baseline	All 10,000 candidate clips	This is the reported 100%; it is not all of VIDGEN-1M or 4DNeX-10M.

Therefore, “the model trained on 10% of the data beats the model trained on 100%” is not an acceptable unqualified summary. The method accesses and processes the whole candidate pool during attribution, then uses only the selected 10% for targeted post-training. The paper establishes useful-subset selection for a narrow adaptation objective, not data-efficient foundation-model pretraining.

The compute comparison is also not controlled tightly enough to infer a 90% end-to-end saving. The paper says that models train for one epoch with the active dataset repeated 50 times, but it does not tabulate optimizer-update counts per arm. Under the literal repeated-dataset reading, the selected arm exposes 50,000 clip instances while the full arm exposes 500,000; exact step matching cannot be verified without code or run logs. Regardless of that ambiguity, Motive adds an approximately 150 A100-hour attribution pass over all 10,000 candidates, so selected-data fraction and total compute fraction are different quantities.

Important boundaries:

• Selection is conditioned on 50 query clips covering the same ten motion categories used at evaluation.
• The comparison is pretrained-model fine-tuning, not foundation-model pretraining from scratch.
• “100%” is the sampled 10,000-clip candidate pool, not the complete upstream video dataset.
• “10%” is the final fine-tuning subset, not the fraction of candidate data inspected by the pipeline.
• The method targets temporal dynamics; broader visual capability can move differently. For example, Motive does not beat full fine-tuning on every imaging-quality column.
• The 150 A100-hour attribution pass for 10,000 Wan2.1 clips is additional upfront compute and must be amortized over queries or repeated uses; no 10-times total-compute reduction is demonstrated.
• Results are reported as point estimates without seed-level variance or confidence intervals.

Social-Source Provenance

The concise Gonzo ML post interprets Motive as evidence that intelligent data selection can make a small subset outperform the full candidate set and suggests relevance beyond video. The linked four-post Gonzo ML Podcasts thread accurately reconstructs the optical-flow masking, projected-gradient attribution, duration-bias correction, main quantitative results, runtime, segment dilution, and camera-motion limitations.

The thread also frames Motive as a step toward physically coherent world models. The authors make a similar but carefully modal claim in their official X thread: motion-level attribution “could inform” data curation for next-generation video world models. Neither source demonstrates an action-conditioned world model, planning, control input, intervention, or counterfactual rollout. In this wiki, that framing is therefore adjacent evidence for world-model data curation, not direct evidence that Motive learns a world model.

Relation To Dynamic Curriculum Learning

Motive is a strong static component for Dynamic Curriculum Learning For JEPA: it turns target temporal dynamics into a per-example gradient-alignment score and demonstrates that query-conditioned selection can beat random selection and full-candidate-pool fine-tuning under a fixed task contract.

It does not yet implement the dynamic loop or avoid processing the non-selected data. Gradients are computed once for every candidate clip, the ranking is one-shot, and selection is tied to a fixed set of motion queries. A curriculum extension would re-attribute as the model changes, preserve a sampling floor for the broader data distribution, and monitor whether target gains erase unrelated capabilities.

Foundation TSFM Relevance

Agenda slot	Verdict	Evidence	Missing pieces
Data diversity and curriculum	adjacent	Demonstrates query-conditioned influence scoring over temporal observation trajectories and a useful-subset fine-tuning result.	No multivariate time-series experiments, streaming updates, or support-preserving sampling policy.
Temporal representation	adjacent	Motion masks isolate dynamic regions from static appearance before measuring influence.	Numeric features and event streams need modality-appropriate change masks rather than optical flow.
Training efficiency	adjacent	Reusable projected gradients make later query ranking cheap and the selected fine-tuning arm uses fewer unique clips.	Upfront attribution scans all candidates and remains 150 A100-hours for only 10,000 clips; matched optimizer steps and end-to-end savings are not established.
World models and control	insufficient evidence	Authors motivate future video-world-model data curation.	No action, control input, intervention, latent transition model, planning, or counterfactual evaluation.

For multivariate time-series models, the closest translation is not “use optical flow.” It is to define a target trajectory or regime, construct a change-aware loss over relevant numeric features and event windows, and rank training windows by projected gradient alignment. That is a research hypothesis, not a result established by this paper.

Links Into The Wiki

• Motive
• Dynamic Curriculum Learning For JEPA
• Training Dynamics
• World Models
• Time-Series Benchmark Hygiene
• Vision Foundation Models

Limitations And Gotchas

• Per-example gradient extraction is expensive: about 150 A100-hours for 10,000 Wan2.1 clips, versus roughly 3–5.7 hours for the tested V-JEPA, motion-magnitude, and optical-flow baselines.
• The method must process all 10,000 candidate clips before selecting 1,000, so 10% fine-tuning data does not mean 10% candidate-data access.
• The paper does not give per-arm optimizer-step counts; its one-epoch, 50-repeat description leaves matched fine-tuning compute ambiguous.
• A clip is the attribution unit. Short, informative motion events can be diluted by long static or redundant segments.
• Optical-flow masks can overemphasize camera ego-motion; spatial-uniformity downweighting is a heuristic rather than full camera/object disentanglement.
• The attribution objective does not explicitly model classifier-free guidance, creating a training/inference mismatch.
• Targeted fine-tuning may trade away untargeted base capabilities; the paper flags this but does not provide a broad retention suite.
• The synthetic query set and evaluation set share motion categories, so broader open-world motion coverage remains untested.
• The projected and single-timestep estimators preserve rankings imperfectly.
• No official code link was visible at ingest time, and the main tables do not report repeated-run uncertainty.

Open Questions

• Does iterative re-attribution outperform one-shot selection once the fine-tuned model has moved?
• Can segment- or motion-event-level attribution reduce dilution without making storage and compute prohibitive?
• How should camera ego-motion and object motion be separated for data attribution?
• Can a support-preserving sampler retain general visual capability while concentrating compute on target dynamics?
• Do projected gradient-influence rankings transfer to multivariate time-series windows, especially rare regimes and intervention-response trajectories?
• Under matched total compute, when does query-conditioned selection beat loose filtering, no filtering, or diversity-constrained reweighting?